标签归档：kcbs_dump_adv_state

110T oracle故障恢复

发表于 2022 年 2 月 23 日 由 惜分飞

有客户一套110T的数据库由于存储控制器故障导致数据库无法正常启动,启动报错如下：

Wed Feb  2 23:16:13 2022
Recovery of Online Redo Log: Thread 1 Group 7 Seq 1647469 Reading mem 0
  Mem# 0: /dev/vgredo6/rredo7b
  Mem# 1: /dev/vgredo4/rredo7a
Wed Feb  2 23:16:14 2022
Errors in file /opt/oracle/admin/xifenfei/udump/xifenfei_ora_26754.trc:
ORA-07445: exception encountered: core dump [_memcpy()+7040] [SIGSEGV] [Address not mapped to object] [] [] []
Wed Feb  2 23:16:15 2022
Errors in file /opt/oracle/admin/xifenfei/udump/xifenfei_ora_26754.trc:
ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] []
ORA-07445: exception encountered: core dump [_memcpy()+7040] [SIGSEGV] [Address not mapped to object] [] [] []
Wed Feb  2 23:16:16 2022
Errors in file /opt/oracle/admin/xifenfei/udump/xifenfei_ora_26754.trc:
ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] []
ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] []
ORA-07445: exception encountered: core dump [_memcpy()+7040] [SIGSEGV] [Address not mapped to object] [] [] []
Wed Feb  2 23:16:16 2022
Errors in file /opt/oracle/admin/xifenfei/udump/xifenfei_ora_26754.trc:
ORA-00600: internal error code, arguments: [kghstack_free2], [], [], [], [], [], [], []
ORA-00607: Internal error occurred while making a change to a data block
ORA-00602: internal programming exception
ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] []
ORA-07445: exception encountered: core dump [kcbzdh()+560] [SIGSEGV] [Address not mapped to object] [] [] []
ORA-07445: exception encountered: core dump [_memcpy()+7040] [SIGSEGV] [Address not mapped to object] [] [] []
Wed Feb  2 23:16:26 2022
Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_pmon_26722.trc:
ORA-07445: exception encountered: core dump [kcbzre1()+6593] [SIGSEGV] [Address not mapped to object] [] [] []
Wed Feb  2 23:16:27 2022
Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_pmon_26722.trc:
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1200] [SIGSEGV][] []
ORA-07445: exception encountered: core dump [kcbzre1()+6593] [SIGSEGV] [Address not mapped to object] [] [] []
Wed Feb  2 23:16:27 2022
Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_pmon_26722.trc:
ORA-00602: internal programming exception
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1200] [SIGSEGV][] []
ORA-07445: exception encountered: core dump [kcbzre1()+6593] [SIGSEGV] [Address not mapped to object] [] [] []
Wed Feb  2 23:17:11 2022
PSP0: terminating instance due to error 472
Instance terminated by PSP0, pid = 26724

该错误原因是由于redo信息和数据文件block信息不匹配导致无法正常应用日志,从而出现异常,在后续的recover 中还出现以下错误

Fri Feb 18 16:09:59 2022
ALTER DATABASE RECOVER  datafile 609,610,611,612,613,614,615,602,603,604,605,606,607,608  
Fri Feb 18 16:09:59 2022
Media Recovery Start
 parallel recovery started with 16 processes
Fri Feb 18 16:10:00 2022
Recovery of Online Redo Log: Thread 1 Group 7 Seq 1647469 Reading mem 0
  Mem# 0: /dev/vgredo6/rredo7b
  Mem# 1: /dev/vgredo4/rredo7a
Fri Feb 18 16:12:17 2022
Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_p000_22509.trc:
ORA-00600: internal error code, arguments: [6101], [0], [42], [96], [], [], [], []
Fri Feb 18 16:18:51 2022
Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_p000_22509.trc:
ORA-10562: Error occurred while applying redo to data block (file# 602, block# 1693691)
ORA-10564: tablespace DBS_DCDL_PT
ORA-01110: data file 602: '/dev/vgora12/rdbs_dcdl_pt0155'
ORA-10561: block type 'TRANSACTION MANAGED INDEX BLOCK', data object# 33682645
ORA-00600: internal error code, arguments: [6101], [0], [42], [96], [], [], [], []
Fri Feb 18 16:18:55 2022
Media Recovery failed with error 12801

Fri Feb 18 18:23:59 2022
Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_dbw1_22483.trc:
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1200] [SIGSEGV][] []
ORA-07445: exception encountered: core dump [ksuitm()+2400] [SIGSEGV] [] [] [] []
ORA-00472: PMON  process terminated with error
Fri Feb 18 18:24:04 2022
DBW3: terminating instance due to error 472
Fri Feb 18 18:24:04 2022
Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_dbw3_22487.trc:
ORA-07445: exception encountered: core dump [ksuitm()+2400] [SIGSEGV] [] [] [] []
ORA-00472: PMON  process terminated with error
Fri Feb 18 18:24:04 2022
Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_dbw3_22487.trc:
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1200] [SIGSEGV] [] []
ORA-07445: exception encountered: core dump [ksuitm()+2400] [SIGSEGV] [] [] [] []
ORA-00472: PMON  process terminated with error
Fri Feb 18 18:24:09 2022
LGWR: terminating instance due to error 472
Fri Feb 18 18:24:09 2022
Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_lgwr_22489.trc:
ORA-07445: exception encountered: core dump [ksuitm()+2400] [SIGSEGV] [] [] [] []
ORA-00472: PMON  process terminated with error

从sqlplus中看到类似一些报错

SQL>  recover datafile 601;
ORA-03113: end-of-file on communication channel

SQL> recover datafile 1066;
ORA-00283: recovery session canceled due to errors
ORA-12801: error signaled in parallel query server P015
ORA-00600: internal error code, arguments: [2037], [207064103], [207064103],
[162], [6], [1], [1833009883], [1130705717]


SQL> recover datafile 1065;
ORA-00283: recovery session canceled due to errors
ORA-12801: error signaled in parallel query server P004
ORA-00600: internal error code, arguments: [kcbzpb_1], [142189139], [3], [0],
[], [], [], []


SQL> recover datafile 2042;
ORA-00283: recovery session canceled due to errors
ORA-12801: error signaled in parallel query server P014
ORA-00600: internal error code, arguments: [3020], [627], [3234156],
[2633062764], [], [], [], []
ORA-10567: Redo is inconsistent with data block

通过屏蔽一致性,强制open库成功

Sun Feb 20 21:20:06 2022
SMON: enabling tx recovery
Sun Feb 20 21:20:06 2022
Database Characterset is ZHS16GBK
Sun Feb 20 21:20:07 2022
ORACLE Instance xifenfei (pid = 38) - Error 376 encountered while recovering transaction (74, 17) on object 34131051.
Sun Feb 20 21:20:07 2022
Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc:
ORA-00376: file 1416 cannot be read at this time
ORA-01110: data file 1416: '/dev/vgora14/rdbs_icdl_pt116'
Sun Feb 20 21:20:08 2022
Stopping background process MMNL
Sun Feb 20 21:20:09 2022
ORACLE Instance xifenfei (pid = 38) - Error 376 encountered while recovering transaction (88, 36) on object 33514955.
Sun Feb 20 21:20:09 2022
Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc:
ORA-00376: file 1264 cannot be read at this time
ORA-01110: data file 1264: '/dev/vgora14/rdbs_icdl_pt102'
Sun Feb 20 21:20:09 2022
Stopping background process MMON
Starting background process MMON
Starting background process MMNL
MMON started with pid=46, OS id=1482
Sun Feb 20 21:20:10 2022
Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc:
ORA-01578: ORACLE data block corrupted (file # 652, block # 3767844)
ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205'
Sun Feb 20 21:20:10 2022
Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc:
ORA-01578: ORACLE data block corrupted (file # 652, block # 3767661)
ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205'
replication_dependency_tracking turned off (no async multimaster replication found)
Sun Feb 20 21:20:11 2022
Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc:
ORA-01578: ORACLE data block corrupted (file # 652, block # 3767661)
ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205'
Sun Feb 20 21:20:11 2022
Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc:
ORA-01578: ORACLE data block corrupted (file # 652, block # 3767661)
ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205'
Sun Feb 20 21:20:11 2022
LOGSTDBY: Validating controlfile with logical metadata
Sun Feb 20 21:20:11 2022
LOGSTDBY: Validation complete
Sun Feb 20 21:20:11 2022
Errors in file /opt/oracle/admin/xifenfei/bdump/xifenfei_smon_25140.trc:
ORA-01578: ORACLE data block corrupted (file # 652, block # 3767661)
ORA-01110: data file 652: '/dev/vgora13/rdbs_dcdl_pt0205'
Completed: alter database open

对于异常undo进行处理,数据库正常open
20220220225006
20220220224004
由于客户短期无法迁移数据,先对于一些坏块进行修复，暂时运行数据库后续有时间窗口进行迁移.

发表在 非常规恢复 | 标签为 , , , , , , , , , | 评论关闭

recover database 报kcbs_dump_adv_state恢复

发表于 2021 年 12 月 19 日 由 惜分飞

数据库recover database报ORA-600 kcbrsearchflist_2,kcbs_dump_adv_state,ORA-600 kdBlkCheckError等错

Thu Dec 16 23:57:39 2021
ALTER DATABASE RECOVER  database   
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 8 slaves
Thu Dec 16 23:57:40 2021
Recovery of Online Redo Log: Thread 1 Group 3 Seq 82476 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Thu Dec 16 23:57:41 2021
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr02_5628.trc  (incident=85419):
ORA-00600: internal error code, arguments: [kcbrsearchflist_2], [], [], [], [], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_85419\orcl_pr02_5628_i85419.trc
Thu Dec 16 23:57:42 2021
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr04_668.trc  (incident=85435):
ORA-00353: log corruption near block 9889 change 997219782 time 12/15/2021 17:48:50
ORA-00312: online log 2 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG'
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_85435\orcl_pr04_668_i85435.trc
Thu Dec 16 23:57:42 2021
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr03_5972.trc  (incident=85427):
ORA-00353: log corruption near block 9889 change 997219782 time 12/15/2021 17:48:50
ORA-00312: online log 2 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG'
ORA-10567: Redo is inconsistent with data block (file# 1, block# 2296, file offset is 18808832 bytes)
ORA-10564: tablespace SYSTEM
ORA-01110: data file 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'
ORA-10560: block type 'DATA SEGMENT HEADER - UNLIMITED'
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_85427\orcl_pr03_5972_i85427.trc
Thu Dec 16 23:57:43 2021
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr05_5280.trc  (incident=85443):
ORA-00353: log corruption near block 9889 change 997219782 time 12/15/2021 17:48:50
ORA-00312: online log 2 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG'
ORA-10567: Redo is inconsistent with data block (file# 3, block# 128, file offset is 1048576 bytes)
ORA-10564: tablespace UNDOTBS1
ORA-01110: data file 3: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\UNDOTBS01.DBF'
ORA-10560: block type 'KTU SMU HEADER BLOCK'
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_85443\orcl_pr05_5280_i85443.trc
Exception[type: ACCESS_VIOLATION, UNABLE_TO_READ][ADDR:0xFFFFFFFFFFFFFFFF][PC:0x1351BB9, kcbs_dump_adv_state()+1529]
Exception[type: ACCESS_VIOLATION, UNABLE_TO_READ][ADDR:0xFFFFFFFFFFFFFFFF][PC:0x1351BB9, kcbs_dump_adv_state()+1529]
ERROR: Unable to normalize symbol name for the following short stack (at offset 199):
dbgexProcessError()+193<-dbgeExecuteForError()+65<-dbgePostErrorKGE()+1726<-dbkePostKGE_kgsf()+75<-kgeade()+560
<-kgerev()+125<-kgerec5()+60<-sss_xcpt_EvalFilterEx()+1869<-sss_xcpt_EvalFilter()+174<-.1.4_5+59<-0000000077867958
<-000000007787812D<-000000007786855F<-000000007789BCB8<-kcbs_dump_adv_state()+1529<-kcbs_advice_dump()+214
<-dbkedDefDump()+16379<-ksedmp()+43<-ksfdmp()+87<-dbgexPhaseII()+1819<-dbgexProcessError()+2563
<-dbgeExecuteForError()+65<-dbgePostErrorKGE()+1726<-dbkePostKGE_kgsf()+75<-kgeade()+560<-kgerev()+125
<-kserec3()+111<-kcrcrl()+262<-krr_thread_read()+2718<-krr_read_buffer()+30<-krr_parse_redo()+2228
<-kcra_scan_redo()+10061<-kcra_dump_redo()+2246<-kcra_dump_redo_internal()+1752<-kcbr_mapply_change()+5953
<-kcbrapply()+2251<-kcbr_apply_pending()+2931<-kcbr_media_apply()+6901<-krp_slave_apply()+313<-krp_slave_main()+2545
<-ksvrdp()+2506<-opirip()+965<-opidrv()+909<-sou2o()+98<-opimai_real()+299<-opimai()+191
<-BackgroundThreadStart()+693<-00000000776459CD<-000000007787A561
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr03_5972.trc  (incident=85428):
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1529] [ACCESS_VIOLATION][ADDR:0xFFFFFFFFFFFFFFFF]
ORA-00353: log corruption near block 9889 change 997219782 time 12/15/2021 17:48:50
ORA-00312: online log 2 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG'
ORA-10567: Redo is inconsistent with data block (file# 1, block# 2296, file offset is 18808832 bytes)
ORA-10564: tablespace SYSTEM
ORA-01110: data file 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'
ORA-10560: block type 'DATA SEGMENT HEADER - UNLIMITED'
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_85428\orcl_pr03_5972_i85428.trc
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xFFFFFFFFFFFFFFFF] [PC:0x1351BB9, kcbs_dump_adv_state()+1529]
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xFFFFFFFFFFFFFFFF] [PC:0x1351BB9, kcbs_dump_adv_state()+1529]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr04_668.trc  (incident=85436):
ORA-07445: exception encountered:core dump [kcbs_dump_adv_state()+1529][ACCESS_VIOLATION][ADDR:0xFFFFFFFFFFFFFFFF]
ORA-00353: log corruption near block 9889 change 997219782 time 12/15/2021 17:48:50
ORA-00312: online log 2 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG'
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_85436\orcl_pr04_668_i85436.trc
Thu Dec 16 23:57:44 2021
Trace dumping is performing id=[cdmp_20211216235744]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr02_5628.trc  (incident=85420):
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1529] [ACCESS_VIOLATION] [ADDR:0xFFFFFFFFFFFFFFFF]
ORA-00600: internal error code, arguments: [kcbrsearchflist_2], [], [], [], [], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_85420\orcl_pr02_5628_i85420.trc
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr05_5280.trc  (incident=85444):
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+1529] [ACCESS_VIOLATION] [ADDR:0xFFFFFFFFFFFFFFFF]
ORA-00353: log corruption near block 9889 change 997219782 time 12/15/2021 17:48:50
ORA-00312: online log 2 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG'
ORA-10567: Redo is inconsistent with data block (file# 3, block# 128, file offset is 1048576 bytes)
ORA-10564: tablespace UNDOTBS1
ORA-01110: data file 3: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\UNDOTBS01.DBF'
ORA-10560: block type 'KTU SMU HEADER BLOCK'
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_85444\orcl_pr05_5280_i85444.trc
Thu Dec 16 23:57:45 2021
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0x29FFFFFFE] [PC:0x7473E40B, 000000007473E40B]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr06_4268.trc  (incident=85451):
ORA-07445: exception encountered: core dump [PC:0x7473E40B] [ACCESS_VIOLATION] [ADDR:0x29FFFFFFE] [PC:0x7473E40B] 
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_85451\orcl_pr06_4268_i85451.trc
Trace dumping is performing id=[cdmp_20211216235745]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_85451\orcl_pr06_4268_i85451.trc:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00602: internal programming exception
ORA-07445: exception encountered: core dump [PC:0x7473E40B] [ACCESS_VIOLATION] [ADDR:0x29FFFFFFE] [PC:0x7473E40B]
Process debug not enabled via parameter _debug_enable
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr04_668.trc:
ORA-00354: corrupt redo log block header
ORA-00353: log corruption near block 9889 change 997219782 time 12/15/2021 17:48:50
ORA-00312: online log 2 thread 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG'
Trace dumping is performing id=[cdmp_20211216235747]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr04_668.trc  (incident=85437):
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [2025], [6401], [], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_85437\orcl_pr04_668_i85437.trc
Recovery Slave PR02 died 
Trace dumping is performing id=[cdmp_20211216235748]
Slave exiting with ORA-10562 exception
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr04_668.trc:
ORA-10562: Error occurred while applying redo to data block (file# 1, block# 2025)
ORA-10564: tablespace SYSTEM
ORA-01110: data file 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [2025], [6401], [], [], [], [], [], [], [], []
Thu Dec 16 23:57:48 2021
Sweep [inc][85451]: completed
Sweep [inc][85444]: completed
Media Recovery failed with error 448
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr00_3528.trc:
ORA-00283: recovery session canceled due to errors
ORA-00448: normal completion of background process
Slave exiting with ORA-283 exception
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr00_3528.trc:
ORA-00283: recovery session canceled due to errors
ORA-00448: normal completion of background process
ORA-10878 signalled during: ALTER DATABASE RECOVER  database   ...

尝试recover datafile报ORA-07445 kcbsacc错

ALTER DATABASE RECOVER  datafile 5  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 3 Seq 82476 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Exception [type: ACCESS_VIOLATION, UNABLE_TO_READ] [ADDR:0xFFFFFFFFFFFFFFFF] [PC:0x13467BB, kcbsacc()+5913]
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_6180.trc  (incident=85398):
ORA-07445: 出现异常错误: 核心转储 [kcbsacc()+5913] [ACCESS_VIOLATION] [ADDR:0xFFFFFFFFFFFFFFFF] [PC:0x13467BB]
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_85398\orcl_ora_6180_i85398.trc

通过上述报错分析,基本上确认是由于异常断电导致redo和数据文件的block不一致引起,尝试强制open库,数据库报ORA-600 2662错误

Fri Dec 17 11:41:40 2021
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 997234250
Resetting resetlogs activation ID 1489370600 (0x58c5fde8)
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_7112.trc:
ORA-00367: 日志文件标头中的校验和错误
ORA-00322: 日志 1 (用于线程 1) 不是最新副本
ORA-00312: 联机日志 1 线程 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG'
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_7112.trc:
ORA-00367: 日志文件标头中的校验和错误
ORA-00322: 日志 2 (用于线程 1) 不是最新副本
ORA-00312: 联机日志 2 线程 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG'
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_7112.trc:
ORA-00367: 日志文件标头中的校验和错误
ORA-00322: 日志 3 (用于线程 1) 不是最新副本
ORA-00312: 联机日志 3 线程 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG'
Fri Dec 17 11:41:46 2021
Setting recovery target incarnation to 2
Fri Dec 17 11:41:46 2021
Assigning activation ID 1619342902 (0x60853636)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Successful open of redo thread 1
Fri Dec 17 11:41:46 2021
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Dec 17 11:41:47 2021
SMON: enabling cache recovery
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_7112.trc  (incident=89000):
ORA-00600: 内部错误代码, 参数: [2662], [0], [997234264], [0], [997238345], [12583088], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_89000\orcl_ora_7112_i89000.trc
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_7112.trc:
ORA-00600: 内部错误代码, 参数: [2662], [0], [997234264], [0], [997238345], [12583088], []
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_7112.trc:
ORA-00600: 内部错误代码, 参数: [2662], [0], [997234264], [0], [997238345], [12583088], []
Error 600 happened during db open, shutting down database
USER (ospid: 7112): terminating the instance due to error 600
Instance terminated by USER, pid = 7112
ORA-1092 signalled during: alter database open resetlogs...
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_7112.trc  (incident=89001):
ORA-00600: 内部错误代码, 参数: [2662], [0], [997234266], [0], [997238345], [12583088], [], [], [], [], [], []
ORA-01092: ORACLE 实例终止。强制断开连接
ORA-00600: 内部错误代码, 参数: [2662], [0], [997234264], [0], [997238345], [12583088], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_89001\orcl_ora_7112_i89001.trc
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_7112.trc  (incident=89002):
ORA-00600: 内部错误代码, 参数: [2662], [0], [997234267], [0], [997238345], [12583088], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [2662], [0], [997234266], [0], [997238345], [12583088], [], [], [], [], [], []
ORA-01092: ORACLE 实例终止。强制断开连接
ORA-00600: 内部错误代码, 参数: [2662], [0], [997234264], [0], [997238345], [12583088], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_89002\orcl_ora_7112_i89002.trc
Fri Dec 17 11:41:57 2021
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_7112.trc:
ORA-00600: 内部错误代码, 参数: [2662], [0], [997234267], [0], [997238345], [12583088], [], [], [], [], [], []
ORA-00600: 内部错误代码, 参数: [2662], [0], [997234266], [0], [997238345], [12583088], [], [], [], [], [], []
ORA-01092: ORACLE 实例终止。强制断开连接
ORA-00600: 内部错误代码, 参数: [2662], [0], [997234264], [0], [997238345], [12583088], [], [], [], [], [], []
Fri Dec 17 11:41:59 2021
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_7112.trc  (incident=90048):
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [0], [997234267], [0], [997238345], [12583088], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [2662], [0], [997234266], [0], [997238345], [12583088], [], [], [], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [0], [997234264], [0], [997238345], [12583088], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_90048\orcl_ora_7112_i90048.trc
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_90048\orcl_ora_7112_i90048.trc:
ORA-00603: ORACLE server session terminated by fatal error
ORA-00600: internal error code, arguments: [2662], [0], [997234267], [0], [997238345], [12583088], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [2662], [0], [997234266], [0], [997238345], [12583088], [], [], [], [], [], []
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [2662], [0], [997234264], [0], [997238345], [12583088], [], [], [], [], [], []

通过修改scn数据库open正常

Fri Dec 17 11:48:37 2021
alter database open 
Fri Dec 17 11:48:38 2021
SMON: enabling cache recovery
Database Characterset is ZHS16GBK
No Resource Manager plan active
Completed: alter database open

由于该库非一致性恢复,逻辑方式导出数据并导入新库,恢复完成

发表在 Oracle备份恢复 | 标签为 , , , | 评论关闭

ORA-01172 ORA-01151 故障恢复

发表于 2016 年 3 月 24 日 由 惜分飞

有客户存储异常断电,导致数据库启动报ORA-01172错,导致数据库无法open
数据库启动报ORA-01172错误

Wed Mar 23 14:16:23 2016
ALTER DATABASE OPEN
Wed Mar 23 14:16:24 2016
Beginning crash recovery of 1 threads
 parallel recovery started with 15 processes
Wed Mar 23 14:16:24 2016
Started redo scan
Wed Mar 23 14:16:25 2016
Completed redo scan
 62588 redo blocks read, 15 data blocks need recovery
Wed Mar 23 14:16:25 2016
Started redo application at
 Thread 1: logseq 15050, block 2, scn 2439828667
Wed Mar 23 14:16:25 2016
Recovery of Online Redo Log: Thread 1 Group 1 Seq 15050 Reading mem 0
  Mem# 0 errs 0: /oracle/oradata/orcl/redo01.log
Wed Mar 23 14:16:25 2016
Completed redo application
Wed Mar 23 14:16:25 2016
RECOVERY OF THREAD 1 STUCK AT BLOCK 26185 OF FILE 3
Wed Mar 23 14:16:25 2016
RECOVERY OF THREAD 1 STUCK AT BLOCK 69385 OF FILE 3
Wed Mar 23 14:16:25 2016
RECOVERY OF THREAD 1 STUCK AT BLOCK 566 OF FILE 2
Wed Mar 23 14:16:25 2016
RECOVERY OF THREAD 1 STUCK AT BLOCK 89 OF FILE 2
Wed Mar 23 14:16:25 2016
RECOVERY OF THREAD 1 STUCK AT BLOCK 53769 OF FILE 3
Wed Mar 23 14:16:26 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p012_6540.trc:
ORA-01172: recovery of thread 1 stuck at block 566 of file 2
ORA-01151: use media recovery to recover block, restore backup if needed
Wed Mar 23 14:16:26 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p008_6532.trc:
ORA-01172: recovery of thread 1 stuck at block 53769 of file 3
ORA-01151: use media recovery to recover block, restore backup if needed
Wed Mar 23 14:16:26 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p011_6538.trc:
ORA-01172: recovery of thread 1 stuck at block 69385 of file 3
ORA-01151: use media recovery to recover block, restore backup if needed
Wed Mar 23 14:16:26 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p005_6526.trc:
ORA-01172: recovery of thread 1 stuck at block 26185 of file 3
ORA-01151: use media recovery to recover block, restore backup if needed
Wed Mar 23 14:16:27 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p014_6544.trc:
ORA-01172: recovery of thread 1 stuck at block 89 of file 2
ORA-01151: use media recovery to recover block, restore backup if needed
Wed Mar 23 14:16:27 2016
Aborting crash recovery due to slave death, attempting serial crash recovery
Wed Mar 23 14:16:27 2016
Beginning crash recovery of 1 threads
Wed Mar 23 14:16:27 2016
Started redo scan
Wed Mar 23 14:16:27 2016
Completed redo scan
 62588 redo blocks read, 15 data blocks need recovery
Wed Mar 23 14:16:27 2016
Started redo application at
 Thread 1: logseq 15050, block 2, scn 2439828667
Wed Mar 23 14:16:27 2016
Recovery of Online Redo Log: Thread 1 Group 1 Seq 15050 Reading mem 0
  Mem# 0 errs 0: /oracle/oradata/orcl/redo01.log
RECOVERY OF THREAD 1 STUCK AT BLOCK 566 OF FILE 2
Wed Mar 23 14:16:27 2016
Aborting crash recovery due to error 1172
Wed Mar 23 14:16:27 2016
Errors in file /oracle/admin/orcl/udump/orcl_ora_6514.trc:
ORA-01172: recovery of thread 1 stuck at block 566 of file 2
ORA-01151: use media recovery to recover block, restore backup if needed
ORA-1172 signalled during: ALTER DATABASE OPEN...

ALTER DATABASE RECOVER datafile 1 报错
尝试recover datafile 1之后报ORA-600 kcbrapply_4,ORA-600 kcfrbd_3,ORA-600 kcbrapply_12等错误,从报错信息看,出现这些错误的原因,是由于断电导致坏块引起.

Thu Mar 24 21:50:18 2016
ALTER DATABASE RECOVER  datafile 1  
Thu Mar 24 21:50:18 2016
Media Recovery Start
 parallel recovery started with 15 processes
Thu Mar 24 21:50:18 2016
Recovery of Online Redo Log: Thread 1 Group 1 Seq 15050 Reading mem 0
  Mem# 0 errs 0: /oracle/oradata/orcl/redo01.log
Thu Mar 24 21:50:19 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p004_13391.trc:
ORA-00600: internal error code, arguments: [kcbrapply_4], [2], [], [], [], [], [], []
Thu Mar 24 21:50:19 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p010_13403.trc:
ORA-00600: internal error code, arguments: [kcbrapply_4], [0], [], [], [], [], [], []
Thu Mar 24 21:50:19 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p000_13383.trc:
ORA-00600: internal error code, arguments: [kcbrapply_4], [0], [], [], [], [], [], []
Thu Mar 24 21:50:19 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p009_13401.trc:
ORA-00600: internal error code, arguments: [kcbrapply_4], [3], [], [], [], [], [], []
Thu Mar 24 21:50:19 2016
Hex dump of (file 1, block 61562) in trace file /oracle/admin/orcl/bdump/orcl_p001_13385.trc
Corrupt block relative dba: 0x0040f07a (file 1, block 61562)
Bad header found during media recovery
Data in bad block:
 type: 0 format: 0 rdba: 0xf07a0000
 last change scn: 0x916c.dc4b0040 seq: 0x0 flg: 0x00
 spare1: 0x6 spare2: 0xa2 spare3: 0xb088
 consistency value in tail: 0x06010fc1
 check value in block header: 0x601
 block checksum disabled
Thu Mar 24 21:50:19 2016
Hex dump of (file 1, block 55706) in trace file /oracle/admin/orcl/bdump/orcl_p014_13411.trc
Corrupt block relative dba: 0x0040d99a (file 1, block 55706)
Bad header found during media recovery
Data in bad block:
 type: 0 format: 0 rdba: 0xd99a0000
 last change scn: 0x916c.e1ad0040 seq: 0x0 flg: 0x00
 spare1: 0x6 spare2: 0xa2 spare3: 0xa520
 consistency value in tail: 0x06012222
 check value in block header: 0x601
 block checksum disabled
Thu Mar 24 21:50:19 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p006_13395.trc:
ORA-00600: internal error code, arguments: [kcfrbd_3], [1], [3342335], [1], [0], [64000], [], []
Thu Mar 24 21:50:19 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p003_13389.trc:
ORA-00600: internal error code, arguments: [kcfrbd_3], [1], [3932159], [1], [0], [64000], [], []
Thu Mar 24 21:50:19 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p002_13387.trc:
ORA-00600: internal error code, arguments: [kcfrbd_3], [1], [2293759], [1], [0], [64000], [], []
Reread of rdba: 0x0040d99a (file 1, block 55706) found valid data
Thu Mar 24 21:50:19 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p014_13411.trc:
ORA-00600: internal error code, arguments: [kcbrapply_12], [], [], [], [], [], [], []
Thu Mar 24 21:50:19 2016
Reread of rdba: 0x0040f07a (file 1, block 61562) found valid data
Thu Mar 24 21:50:19 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p001_13385.trc:
ORA-00600: internal error code, arguments: [kcbrapply_12], [], [], [], [], [], [], []
Thu Mar 24 21:50:23 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p014_13411.trc:
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+450] [SIGSEGV] [Address not mapped to object] [0xB9782BF4] [] []
ORA-00600: internal error code, arguments: [kcbrapply_12], [], [], [], [], [], [], []
Thu Mar 24 21:50:23 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p006_13395.trc:
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+450] [SIGSEGV] [Address not mapped to object] [0xB9C82BF4] [] []
ORA-00600: internal error code, arguments: [kcfrbd_3], [1], [3342335], [1], [0], [64000], [], []
Thu Mar 24 21:50:23 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p009_13401.trc:
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+450] [SIGSEGV] [Address not mapped to object] [0xB9A02BF4] [] []
ORA-00600: internal error code, arguments: [kcbrapply_4], [3], [], [], [], [], [], []
Thu Mar 24 21:50:23 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p003_13389.trc:
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+450] [SIGSEGV] [Address not mapped to object] [0xB9F02AF4] [] []
ORA-00600: internal error code, arguments: [kcfrbd_3], [1], [3932159], [1], [0], [64000], [], []
Thu Mar 24 21:50:23 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p004_13391.trc:
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+450] [SIGSEGV] [Address not mapped to object] [0xBA182AF4] [] []
ORA-00600: internal error code, arguments: [kcbrapply_4], [2], [], [], [], [], [], []
Thu Mar 24 21:50:23 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p010_13403.trc:
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+450] [SIGSEGV] [Address not mapped to object] [0xBA402AF4] [] []
ORA-00600: internal error code, arguments: [kcbrapply_4], [0], [], [], [], [], [], []
Thu Mar 24 21:50:23 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p000_13383.trc:
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+450] [SIGSEGV] [Address not mapped to object] [0xB9282AF4] [] []
ORA-00600: internal error code, arguments: [kcbrapply_4], [0], [], [], [], [], [], []
Thu Mar 24 21:50:23 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p001_13385.trc:
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+450] [SIGSEGV] [Address not mapped to object] [0xB9C82AF4] [] []
ORA-00600: internal error code, arguments: [kcbrapply_12], [], [], [], [], [], [], []
Thu Mar 24 21:50:23 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p002_13387.trc:
ORA-10562: Error occurred while applying redo to data block (file# 1, block# 11042)
ORA-10564: tablespace SYSTEM
ORA-01110: data file 1: '/oracle/oradata/orcl/system01.dbf'
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kcfrbd_3], [1], [2293759], [1], [0], [64000], [], []

ALTER DATABASE RECOVER datafile 3 报错
该文件恢复主要报ORA-600 kcbrsearchflist_2,ORA-600 kdxlin:psno out of range,ORA-600 kcbs_dump_adv_state等错误

Thu Mar 24 21:52:04 2016
ALTER DATABASE RECOVER  datafile 3  
Thu Mar 24 21:52:04 2016
Media Recovery Start
 parallel recovery started with 15 processes
Thu Mar 24 21:52:04 2016
Recovery of Online Redo Log: Thread 1 Group 1 Seq 15050 Reading mem 0
  Mem# 0 errs 0: /oracle/oradata/orcl/redo01.log
Thu Mar 24 21:52:05 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p007_13462.trc:
ORA-00600: internal error code, arguments: [kdxlin:psno out of range], [], [], [], [], [], [], []
Thu Mar 24 21:52:05 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p001_13450.trc:
ORA-00600: internal error code, arguments: [kcbrsearchflist_2], [], [], [], [], [], [], []
Thu Mar 24 21:52:05 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p007_13462.trc:
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+464] [SIGSEGV] [Address not mapped to object] [0xB9F076F4] [] []
ORA-00600: internal error code, arguments: [kdxlin:psno out of range], [], [], [], [], [], [], []
Thu Mar 24 21:52:05 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p001_13450.trc:
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+464] [SIGSEGV] [Address not mapped to object] [0xB9C874F4] [] []
ORA-00600: internal error code, arguments: [kcbrsearchflist_2], [], [], [], [], [], [], []
Thu Mar 24 21:52:06 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p007_13462.trc:
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+464] [SIGSEGV] [Address not mapped to object] [0xB9F066F4] [] []
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+464] [SIGSEGV] [Address not mapped to object] [0xB9F076F4] [] []
ORA-00600: internal error code, arguments: [kdxlin:psno out of range], [], [], [], [], [], [], []
Thu Mar 24 21:52:06 2016
Errors in file /oracle/admin/orcl/bdump/orcl_p001_13450.trc:
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+464] [SIGSEGV] [Address not mapped to object] [0xB9C864F4] [] []
ORA-07445: exception encountered: core dump [kcbs_dump_adv_state()+464] [SIGSEGV] [Address not mapped to object] [0xB9C874F4] [] []
ORA-00600: internal error code, arguments: [kcbrsearchflist_2], [], [], [], [], [], [], []

恢复过程

SQL> startup mount
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size          1220432 bytes
Variable Size             369098928 bytes
Database Buffers         1761607680 bytes
Redo Buffers               15556608 bytes
Database mounted.
SQL> select file# from v$datafile;

     FILE#
----------
         1
         2
         3
         4
         5
         6

6 rows selected.

SQL> recover datafile 1;
ORA-03113: end-of-file on communication channel

SQL> startup mount
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size          1220432 bytes
Variable Size             369098928 bytes
Database Buffers         1761607680 bytes
Redo Buffers               15556608 bytes
Database mounted.

SQL> recover datafile 3;
ORA-03113: end-of-file on communication channel

SQL> startup mount
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size          1220432 bytes
Variable Size             369098928 bytes
Database Buffers         1761607680 bytes
Redo Buffers               15556608 bytes
Database mounted.
SQL> recover datafile 5;
Media recovery complete.
SQL> recover datafile 6;
Media recovery complete.
SQL> recover datafile 4;
Media recovery complete.
SQL> recover datafile 2;
Media recovery complete.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [4194], [57], [11], [], [], [], [],
[]

SQL> select open_mode from v$database;

OPEN_MODE
----------
READ WRITE

这次运气不错,system坏的是mon_mods$,undo异常可以重建,基本上可以说没有数据丢失,数据库恢复完成.
重要的库,通过open过程报错信息,分析可能的坏块所属对象,然后确定处理方法,以免造成永久性数据块损坏.

发表在 Oracle备份恢复 | 标签为 , , , , , , , | 评论关闭