分类目录归档：数据库

ORA-01092 ORA-00604 ORA-01558故障处理

发表于 2024 年 9 月 21 日由惜分飞

客户一个11.2.0.1的库,在重启之前报ORA-00604和ORA-01558: out of transaction ID’s in rollback segment SYSTEM错误

Sat Mar 16 05:51:44 2024
Errors in file /opt/oracle/app/diag/rdbms/xff/xff/trace/xff_smon_47709.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTEM
Sat Mar 16 05:51:55 2024
Errors in file /opt/oracle/app/diag/rdbms/xff/xff/trace/xff_smon_47709.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTEM

数据库关闭之后无法open,报ORA-01092 ORA-00604 ORA-01558错误

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTEM
Process ID: 44308
Session ID: 769 Serial number: 3

对应的alert日志信息

Fri Aug 30 20:05:40 2024
alter database open
Beginning crash recovery of 1 threads
Fri Aug 30 20:06:08 2024
 parallel recovery started with 32 processes
Started redo scan
Completed redo scan
 read 0 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 7583120, block 130906, scn 832944617073
Recovery of Online Redo Log: Thread 1 Group 6 Seq 7583120 Reading mem 0
  Mem# 0: /OracleData/db_mc/xff/redo06.log
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 7583120, block 130906, scn 832944637074
 0 data blocks read, 0 data blocks written, 0 redo k-bytes read
Fri Aug 30 20:06:09 2024
Thread 1 advanced to log sequence 7583121 (thread open)
Thread 1 opened at log sequence 7583121
  Current log# 4 seq# 7583121 mem# 0: /OracleData/db_mc/xff/redo04.log
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Aug 30 20:06:09 2024
SMON: enabling cache recovery
Errors in file /opt/oracle/app/diag/rdbms/xff/xff/trace/xff_ora_6066.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTEM
Errors in file /opt/oracle/app/diag/rdbms/xff/xff/trace/xff_ora_6066.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01558: out of transaction ID's in rollback segment SYSTEM
Error 604 happened during db open, shutting down database
USER (ospid: 6066): terminating the instance due to error 604
Instance terminated by USER, pid = 6066
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (6066) as a result of ORA-1092

通过对异常block进行dump

  index  state cflags  wrap#    uel         scn            dba            parent-xid    nub     stmt_num
  ------------------------------------------------------------------------------------------------
   0x00    9    0x00  0xfffffffe  0x0059  0x00c1.ee6ab58f  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x01    9    0x00  0xfffffffe  0x0008  0x00c1.ee6ab5a7  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x02    9    0x00  0xfffffffe  0x005b  0x00c1.ee6ab59f  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x03    9    0x00  0xfffffffe  0x0007  0x00c1.ee6ab5b5  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x04    9    0x00  0xfffffffe  0x005f  0x00c1.ee6ab593  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x05    9    0x00  0xfffffffe  0x000a  0x00c1.ee6ab5ad  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x06    9    0x00  0xfffffffe  0x0019  0x00c1.ee6ab5dd  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x07    9    0x00  0xfffffffe  0x0013  0x00c1.ee6ab5b7  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x08    9    0x00  0xfffffffe  0x0035  0x00c1.ee6ab5a9  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x09    9    0x00  0xfffffffe  0x0002  0x00c1.ee6ab59d  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x0a    9    0x00  0xfffffffe  0x003c  0x00c1.ee6ab5af  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x0b    9    0x00  0xfffffffe  0x0011  0x00c1.ee6ab5d7  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x0c    9    0x00  0xfffffffe  0x0026  0x00c1.ee6ab605  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x0d    9    0x00  0xfffffffe  0x000f  0x00c1.ee6ab5c5  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x0e    9    0x00  0xfffffffe  0x0014  0x00c1.ee6ab5d3  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x0f    9    0x00  0xfffffffe  0x0012  0x00c1.ee6ab5c7  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x10    9    0x00  0xfffffffe  0x0029  0x00c1.ee6ab5c1  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x11    9    0x00  0xfffffffe  0x0018  0x00c1.ee6ab5d9  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x12    9    0x00  0xfffffffe  0x001e  0x00c1.ee6ab5c9  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x13    9    0x00  0xfffffffe  0x001b  0x00c1.ee6ab5b9  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x14    9    0x00  0xfffffffe  0x000b  0x00c1.ee6ab5d5  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x15    9    0x00  0xfffffffe  0x0003  0x00c1.ee6ab5b3  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x16    9    0x00  0xfffffffe  0x0023  0x00c1.ee6ab5f1  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x17    9    0x00  0xfffffffe  0x0021  0x00c1.ee6ab5bd  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x18    9    0x00  0xfffffffe  0x0006  0x00c1.ee6ab5db  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x19    9    0x00  0xfffffffe  0x0049  0x00c1.ee6ab5df  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x1a    9    0x00  0xfffffffe  0x0043  0x00c1.ee6ab617  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x1b    9    0x00  0xfffffffe  0x0017  0x00c1.ee6ab5bb  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x1c    9    0x00  0xfffffffe  0x002e  0x00c1.ee6ab5e9  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x1d    9    0x00  0xfffffffe  0x001c  0x00c1.ee6ab5e7  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x1e    9    0x00  0xfffffffe  0x002a  0x00c1.ee6ab5cb  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x1f    9    0x00  0xffffff28  0x003e  0x00c1.ee6ab62d  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x20    9    0x00  0xfffffffe  0x0022  0x00c1.ee6ab5cf  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x21    9    0x00  0xfffffffe  0x0010  0x00c1.ee6ab5bf  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x22    9    0x00  0xfffffffe  0x000e  0x00c1.ee6ab5d1  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x23    9    0x00  0xfffffffe  0x0025  0x00c1.ee6ab5f3  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x24    9    0x00  0xfffffffe  0x0032  0x00c1.ee6ab5f7  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x25    9    0x00  0xfffffffd  0x0024  0x00c1.ee6ab5f5  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x26    9    0x00  0xfffffffe  0x003d  0x00c1.ee6ab607  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x27    9    0x00  0xfffffffe  0x0041  0x00c1.ee6ab63b  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x28    9    0x00  0xfffffffe  0x003f  0x00c1.ee6ab613  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x29    9    0x00  0xfffffffb  0x000d  0x00c1.ee6ab5c3  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x2a    9    0x00  0xfffffffe  0x0020  0x00c1.ee6ab5cd  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x2b    9    0x00  0xfffffffe  0x0037  0x00c1.ee6ab60d  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x2c    9    0x00  0xfffffffe  0x002d  0x00c1.ee6ab5ed  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x2d    9    0x00  0xfffffffe  0x0016  0x00c1.ee6ab5ef  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x2e    9    0x00  0xfffffffe  0x002c  0x00c1.ee6ab5eb  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x2f    9    0x00  0xfffffffe  0x0034  0x00c1.ee6ab5ff  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x30    9    0x00  0xfffffffe  0x001f  0x00c1.ee6ab62b  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x31    9    0x00  0xfffffffe  0x000c  0x00c1.ee6ab603  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x32    9    0x00  0xfffffffe  0x003b  0x00c1.ee6ab5f9  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x33    9    0x00  0xfffffffa  0x0038  0x00c1.ee6ab61b  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x34    9    0x00  0xfffffffe  0x0031  0x00c1.ee6ab601  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x35    9    0x00  0xfffffffd  0x0005  0x00c1.ee6ab5ab  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x36    9    0x00  0xfffffffe  0x0028  0x00c1.ee6ab611  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x37    9    0x00  0xfffffffe  0x0036  0x00c1.ee6ab60f  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x38    9    0x00  0xfffffffe  0x0048  0x00c1.ee6ab61d  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x39    9    0x00  0xfffffffe  0x0052  0x00c1.ee6ab64d  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x3a    9    0x00  0xfffffffe  0x002f  0x00c1.ee6ab5fd  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x3b    9    0x00  0xfffffffe  0x003a  0x00c1.ee6ab5fb  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x3c    9    0x00  0xfffffffd  0x0015  0x00c1.ee6ab5b1  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x3d    9    0x00  0xfffffffe  0x0061  0x00c1.ee6ab609  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x3e    9    0x00  0xfffffffe  0x004d  0x00c1.ee6ab62f  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x3f    9    0x00  0xfffffffe  0x001a  0x00c1.ee6ab615  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x40    9    0x00  0xfffffffd  0x001d  0x00c1.ee6ab5e5  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x41    9    0x00  0xfffffffe  0x0058  0x00c1.ee6ab63d  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x42    9    0x00  0xfffffffe  0x0027  0x00c1.ee6ab639  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x43    9    0x00  0xfffffffe  0x0033  0x00c1.ee6ab619  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x44    9    0x00  0xfffffffe  0x0050  0x00c1.ee6ab635  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x45    9    0x00  0xfffffffe  0x0044  0x00c1.ee6ab633  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x46    9    0x00  0xfffffffe  0x0057  0x00c1.ee6ab625  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x47    9    0x00  0xfffffffe  0x0030  0x00c1.ee6ab629  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x48    9    0x00  0xfffffffe  0x004c  0x00c1.ee6ab61f  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x49    9    0x00  0xfffffffd  0x004e  0x00c1.ee6ab5e1  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x4a    9    0x00  0xfffffffe  0x0054  0x00c1.ee6ab641  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x4b    9    0x00  0xfffffffe  0x0046  0x00c1.ee6ab623  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x4c    9    0x00  0xfffffffe  0x004b  0x00c1.ee6ab621  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x4d    9    0x00  0xfffffffe  0x0045  0x00c1.ee6ab631  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x4e    9    0x00  0xfffffffd  0x0040  0x00c1.ee6ab5e3  0x03000437  0x0000.000.00000000  0x00000001   0x00000000
   0x4f    9    0x00  0xfffffffd  0x005a  0x00c1.ee6ab599  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x50    9    0x00  0xfffffffe  0x0042  0x00c1.ee6ab637  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x51    9    0x00  0xfffffffe  0x0053  0x00c1.ee6ab645  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x52    9    0x00  0xfffffffe  0x0060  0x00c1.ee6ab64f  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x53    9    0x00  0xfffffffe  0x0056  0x00c1.ee6ab647  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x54    9    0x00  0xfffffffe  0x0051  0x00c1.ee6ab643  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x55    9    0x00  0xfffffffe  0x0039  0x00c1.ee6ab64b  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x56    9    0x00  0xfffffffe  0x0055  0x00c1.ee6ab649  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x57    9    0x00  0xfffffffe  0x0047  0x00c1.ee6ab627  0x03000438  0x0000.000.00000000  0x00000001   0x00000000
   0x58    9    0x00  0xfffffffe  0x004a  0x00c1.ee6ab63f  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x59    9    0x00  0xfffffffd  0x0004  0x00c1.ee6ab591  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x5a    9    0x00  0xfffffffd  0x0009  0x00c1.ee6ab59b  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x5b    9    0x00  0xfffffffd  0x005e  0x00c1.ee6ab5a1  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x5c    9    0x00  0xfffffffd  0x0001  0x00c1.ee6ab5a5  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x5d    9    0x00  0xfffffffd  0x004f  0x00c1.ee6ab597  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x5e    9    0x00  0xfffffffd  0x005c  0x00c1.ee6ab5a3  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x5f    9    0x00  0xfffffffd  0x005d  0x00c1.ee6ab595  0x03000436  0x0000.000.00000000  0x00000001   0x00000000
   0x60    9    0x00  0xfffffffe  0xffff  0x00c1.ee6ab651  0x03000439  0x0000.000.00000000  0x00000001   0x00000000
   0x61    9    0x00  0xfffffdb4  0x002b  0x00c1.ee6ab60b  0x03000438  0x0000.000.00000000  0x00000001   0x00000000

确实是Wrap#达到了极限值,通过bbed也可以进一步确认

BBED> p ktuxe[1]
struct ktuxe[1], 40 bytes                   @4292
   ub4 ktuxexid                             @4292     0xfffffffe
   ub4 ktuxebrb                             @4296     0x03000436
   struct ktuxescn, 8 bytes                 @4300
      ub4 kscnbas                           @4300     0xee6ab5a7
      ub2 kscnwrp                           @4304     0x00c1
   sb4 ktuxesta                             @4308     524297 (KTU_SCO, KTU_DEAD)
   ub1 ktuxecfl                             @4309     0x00
   sb2 ktuxeuel                             @4310     8

BBED> p ktuxe[2]
struct ktuxe[2], 40 bytes                   @4332
   ub4 ktuxexid                             @4332     0xfffffffe
   ub4 ktuxebrb                             @4336     0x03000436
   struct ktuxescn, 8 bytes                 @4340
      ub4 kscnbas                           @4340     0xee6ab59f
      ub2 kscnwrp                           @4344     0x00c1
   sb4 ktuxesta                             @4348     5963785 (KTU_SCO, KTU_DEAD)
   ub1 ktuxecfl                             @4349     0x00
   sb2 ktuxeuel                             @4350     91

处理方法,通过bbed对异常的block进行编辑,修改Wrap#的值,重新dumpblock进行确认

  index  state cflags  wrap#    uel         scn            dba            parent-xid    nub     stmt_num
  ------------------------------------------------------------------------------------------------
   0x00    9    0x00  0x0015  0x0003  0x0000.11e1a313  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x01    9    0x00  0x0015  0x0007  0x0000.11e1a317  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x02    9    0x00  0x0015  0x000c  0x0000.11e32087  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x03    9    0x00  0x0015  0x005a  0x0000.11e1a314  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x04    9    0x00  0x0015  0x0060  0x0000.11e3206e  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x05    9    0x00  0x0015  0x005d  0x0000.11e1a311  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x06    9    0x00  0x0015  0x0015  0x0000.11e3208b  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x07    9    0x00  0x0015  0x0004  0x0000.11e1a318  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x08    9    0x00  0x0015  0x0014  0x0000.11e32084  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x09    9    0x00  0x0015  0x0001  0x0000.11e1a316  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x0a    9    0x00  0x0015  0x0013  0x0000.11e3207d  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x0b    9    0x00  0x0015  0x0011  0x0000.11e3207f  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x0c    9    0x00  0x0015  0x0059  0x0000.11e32089  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x0d    9    0x00  0x0015  0x000a  0x0000.11e3207c  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x0e    9    0x00  0x0015  0x0017  0x0000.11e32082  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x0f    9    0x00  0x0015  0x002a  0x0000.120964dc  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x10    9    0x00  0x0015  0x001a  0x0000.11e3208d  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x11    9    0x00  0x0015  0x005f  0x0000.11e32080  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x12    9    0x00  0x0015  0x0028  0x0000.120964d9  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x13    9    0x00  0x0015  0x000b  0x0000.11e3207e  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x14    9    0x00  0x0015  0x0018  0x0000.11e32085  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x15    9    0x00  0x0015  0x0010  0x0000.11e3208c  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x16    9    0x00  0x0015  0x0032  0x0000.120964e2  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x17    9    0x00  0x0015  0x0008  0x0000.11e32083  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x18    9    0x00  0x0015  0x0002  0x0000.11e32086  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x19    9    0x00  0x0015  0x0020  0x0000.120964d3  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x1a    9    0x00  0x0015  0x0025  0x0000.120964c5  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x1b    9    0x00  0x0015  0x0021  0x0000.120964d6  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x1c    9    0x00  0x0015  0x001b  0x0000.120964d5  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x1d    9    0x00  0x0015  0x0019  0x0000.120964d2  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x1e    9    0x00  0x0015  0x0030  0x0000.1210ab57  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x1f    9    0x00  0x0015  0x000f  0x0000.120964db  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x20    9    0x00  0x0015  0x001c  0x0000.120964d4  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x21    9    0x00  0x0015  0x0012  0x0000.120964d8  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x22    9    0x00  0x0015  0x0029  0x0000.120964de  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x23    9    0x00  0x0015  0x0031  0x0000.120964e4  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x24    9    0x00  0x0015  0x001d  0x0000.120964d1  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x25    9    0x00  0x0014  0x0024  0x0000.120964cf  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x26    9    0x00  0x0015  0x003b  0x0000.1210ab5b  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x27    9    0x00  0x0014  0x0058  0x0000.0ec9fb55  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x28    9    0x00  0x0015  0x001f  0x0000.120964da  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x29    9    0x00  0x0015  0x002b  0x0000.120964df  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x2a    9    0x00  0x0015  0x0022  0x0000.120964dd  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x2b    9    0x00  0x0015  0x002d  0x0000.120964e0  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x2c    9    0x00  0x0015  0x003d  0x0000.1210ab60  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x2d    9    0x00  0x0015  0x0016  0x0000.120964e1  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x2e    9    0x00  0x0014  0x003c  0x0000.0ec9fb51  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x2f    9    0x00  0x0015  0x0045  0x0000.1210ab65  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x30    9    0x00  0x0015  0x0034  0x0000.1210ab58  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x31    9    0x00  0x0015  0x0036  0x0000.1210ab4b  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x32    9    0x00  0x0015  0x0023  0x0000.120964e3  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x33    9    0x00  0x0015  0x0041  0x0000.1210ab67  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x34    9    0x00  0x0015  0x003a  0x0000.1210ab59  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x35    9    0x00  0x0015  0x002c  0x0000.1210ab5e  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x36    9    0x00  0x0015  0x001e  0x0000.1210ab55  0x00400225  0x0000.000.00000000  0x00000001   0x00000000
   0x37    9    0x00  0x0014  0x0050  0x0000.0ec9fb53  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x38    9    0x00  0x0015  0x003f  0x0000.1210ab62  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x39    9    0x00  0x0015  0x0035  0x0000.1210ab5d  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x3a    9    0x00  0x0015  0x0026  0x0000.1210ab5a  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x3b    9    0x00  0x0015  0x0039  0x0000.1210ab5c  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x3c    9    0x00  0x0014  0x0037  0x0000.0ec9fb52  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x3d    9    0x00  0x0015  0x0038  0x0000.1210ab61  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x3e    9    0x00  0x0014  0x0040  0x0000.0ec9fb49  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x3f    9    0x00  0x0015  0x0044  0x0000.1210ab63  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x40    9    0x00  0x0014  0x004b  0x0000.0ec9fb4a  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x41    9    0x00  0x0015  0x0043  0x0000.1210ab68  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x42    9    0x00  0x0015  0xffff  0x0000.1210ab6b  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x43    9    0x00  0x0015  0x0042  0x0000.1210ab69  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x44    9    0x00  0x0015  0x002f  0x0000.1210ab64  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x45    9    0x00  0x0015  0x0033  0x0000.1210ab66  0x00400226  0x0000.000.00000000  0x00000001   0x00000000
   0x46    9    0x00  0x0014  0x004e  0x0000.0ec9fb4e  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x47    9    0x00  0x0014  0x004f  0x0000.0ec9fb4c  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x48    9    0x00  0x0014  0x0054  0x0000.11e1a306  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x49    9    0x00  0x0014  0x0055  0x0000.0ec9fd5b  0x00400223  0x0000.000.00000000  0x00000002   0x00000000
   0x4a    9    0x00  0x0014  0x003e  0x0000.0ec9fb48  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x4b    9    0x00  0x0014  0x0047  0x0000.0ec9fb4b  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x4c    9    0x00  0x0014  0x000d  0x0000.11e3207b  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x4d    9    0x00  0x0014  0x0061  0x0000.11e1a30f  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x4e    9    0x00  0x0014  0x002e  0x0000.0ec9fb4f  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x4f    9    0x00  0x0014  0x0046  0x0000.0ec9fb4d  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x50    9    0x00  0x0014  0x0027  0x0000.0ec9fb54  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x51    9    0x00  0x0014  0x0053  0x0000.11e1a30a  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x52    9    0x00  0x0014  0x005b  0x0000.11e1a30c  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x53    9    0x00  0x0014  0x0052  0x0000.11e1a30b  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x54    9    0x00  0x0014  0x0057  0x0000.11e1a307  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x55    9    0x00  0x0014  0x0048  0x0000.11e1a304  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x56    9    0x00  0x0014  0x0051  0x0000.11e1a309  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x57    9    0x00  0x0014  0x0056  0x0000.11e1a308  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x58    9    0x00  0x0014  0x0049  0x0000.0ec9fb56  0x00400222  0x0000.000.00000000  0x00000001   0x00000000
   0x59    9    0x00  0x0014  0x0006  0x0000.11e3208a  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x5a    9    0x00  0x0014  0x0009  0x0000.11e1a315  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x5b    9    0x00  0x0014  0x005e  0x0000.11e1a30d  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x5c    9    0x00  0x0014  0x004c  0x0000.11e3207a  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x5d    9    0x00  0x0014  0x0000  0x0000.11e1a312  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x5e    9    0x00  0x0014  0x004d  0x0000.11e1a30e  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x5f    9    0x00  0x0014  0x000e  0x0000.11e32081  0x00400224  0x0000.000.00000000  0x00000001   0x00000000
   0x60    9    0x00  0x0014  0x005c  0x0000.11e32078  0x00400223  0x0000.000.00000000  0x00000001   0x00000000
   0x61    9    0x00  0x0014  0x0005  0x0000.11e1a310  0x00400223  0x0000.000.00000000  0x00000001   0x00000000

直接启动数据库成功

[oracle@www.xifenfei.com ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Sat Sep 21 13:45:23 2024

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount pfile='/tmp/pfile';
ORACLE instance started.

Total System Global Area 8551575552 bytes
Fixed Size		    2215064 bytes
Variable Size		 2483028840 bytes
Database Buffers	 6039797760 bytes
Redo Buffers		   26533888 bytes
Database mounted.
SQL> recover database;
Media recovery complete.
SQL> alter database open;       

Database altered.

由于客户遇到故障之后,第一时间保护了现场,没有进行二次破坏,使用bbed进行修改block,实现完美恢复.
以前有过一次在11.2.0.4的环境中恢复的同样案例:ORA-01558: out of transaction ID’s in rollback segment SYSTEM
通过查询mos,确认类似Bug 19700135 – ORA-600 [4187] when the undo segment wrap# is close to the max value of 0xffffffff (Doc ID 19700135.8)
bug-19700135

Description
ORA-600 [4187] can occur for undo segments where wrap# is close to the max value of 0xffffffff (KSQNMAXVAL).
This normally affects databases with high transaction rate that have existed for a relatively long time.
 
To identify undo segments causing the above error and others that may potentially cause it 
in the future, run the next query:
 
 select b.segment_name, b.tablespace_name 
         ,a.ktuxeusn "Undo Segment Number"
         ,a.ktuxeslt "Slot"
         ,a.ktuxesqn "Wrap#"
   from  x$ktuxe a, dba_rollback_segs b
   where a.ktuxesqn > -429496730 and a.ktuxesqn < 0
       and a.ktuxeusn = b.segment_id;
 
Then drop the undo segments or the undo tablespace from the output above.
 
With this fix in place an error ORA-1558 is eventually produced for the affected undo segment
which still requires dropping the undo segment:
  ORA-1558 "out of transaction ID's in rollback segment %s"
   Cause: All the available transaction id's have been used
   Action: Shutdown the instance and restart using other rollback segment(s),
                then drop the rollback segment that has no more transaction id's.
 
Workaround
 Once the ORA-600 [4187] is produced, drop the affected undo segment or undo tablespace.

由于本次故障是在system的rollback回滚段上,无法删除该回滚段.所以处理起来比较棘手.

发表在非常规恢复 | 标签为 ORA-01558, ORA-600 4187 | 评论关闭

ORA-65088: database open should be retried

发表于 2024 年 9 月 18 日由惜分飞

在12.2以及后续的cdb版本中,如果重建ctl并且resetlogs库,很可能会遇到ORA-65088: database open should be retried错误

SQL> startup nomount force pfile='/<path>/<filename>.ora';
ORACLE instance started.

Total System Global Area 1593835520 bytes
Fixed Size                  8793256 bytes
Variable Size             402654040 bytes
Database Buffers         1174405120 bytes
Redo Buffers                7983104 bytes
SQL> !vi ctl.sql

SQL> @ctl.sql

Control file created.

SQL> select count(*) ,fhsta from x$kcvfh group by fhsta;

  COUNT(*)      FHSTA
---------- ----------
        11      32768
         4      40960

SQL> select count(*) ,FHSCN from x$kcvfh group by FHSCN;

  COUNT(*) FHSCN
---------- --------------------
         3 1820866
         4 2281969
         4 2281978
         4 2281982

SQL> select file#,error from v$datafile_header where length(error)>=1;

no rows selected

SQL> select count(*) ,fhrba_seq from x$kcvfh group by fhrba_seq;

  COUNT(*)  FHRBA_SEQ
---------- ----------
         3         20
        12         32

SQL> recover database using backup controlfile until cancel;
ORA-00279: change 2281978 generated at 09/19/2018 00:52:00 needed for thread 1
ORA-00289: suggestion : /<path>/1_32_981800889.dbf
ORA-00280: change 2281978 for thread 1 is in sequence #32


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/<path>/1_32_981800889.dbf
ORA-00279: change 2282008 generated at 09/19/2018 00:52:13 needed for thread 1
ORA-00289: suggestion : /<path>/1_33_981800889.dbf
ORA-00280: change 2282008 for thread 1 is in sequence #33
ORA-00278: log file '/<path>/1_32_981800889.dbf' no longer needed for this recovery


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
Media recovery cancelled.    << Expected message "Media recovery complete." !!
SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       MOUNTED
         3 _###_UNKNOWN_PDB_#_3           MOUNTED
         4 _###_UNKNOWN_PDB_#_4           MOUNTED
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-65088: database open should be retried
Process ID: 32688
Session ID: 10 Serial number: 38416

alert日志类似错误

 Dictionary check beginning
 Pluggable Database <pdb_name_1> (#3) found in data dictionary,
 but not in the control file. Adding it to control file.
 Pluggable Database <pdb_name_2> (#4) found in data dictionary,
 but not in the control file. Adding it to control file.
 Tablespace '<tablespace_name>' #3 found in data dictionary,
 but not in the controlfile. Adding to controlfile.
 --
 File 8 not verified due to error ORA-01122
 File 9 not verified due to error ORA-01122
 File 11 not verified due to error ORA-01122
 File 16 not verified due to error ORA-01122
 File 17 not verified due to error ORA-01122
 File 18 not verified due to error ORA-01122
 File 19 not verified due to error ORA-01122
 File 20 not verified due to error ORA-01122
  --
 ORA-65088: database open should be retried
 2018-09-19T01:00:54.083814+05:30
 Errors in file /<path>/trace/<oracle_sid>_ora_12412.trc:
 ORA-65088: database open should be retried
 Error 65088 happened during db open, shutting down database
 Errors in file /<path>/trace/<oracle_sid>_ora_12412.trc  (incident=12289) (PDBNAME=CDB$ROOT):
 ORA-00603: ORACLE server session terminated by fatal error
 ORA-01092: ORACLE instance terminated. Disconnection forced
 ORA-65088: database open should be retried

出现这类故障的原因是由于:
we see that the created controlfile is not aware of PDB and open resetlogs process trying to add information in newly created file . Hence, recovery process ,in newly created controlfile didn’t applied the archives to datafiles part of PDB which says later it will ask for recovery once controlfile is aware of PDB files During the resetlogs process, its pushing the required information to controlfile and shutting the database with suggestion to re-try opening the DB.

$ sqlplus "/as sysdba"

SQL*Plus: Release 12.2.0.1.0 Production on Wed Sep 19 01:34:01 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount force pfile='/<path>/<filename>.ora';
ORACLE instance started.

Total System Global Area 1593835520 bytes
Fixed Size                  8793256 bytes
Variable Size             402654040 bytes
Database Buffers         1174405120 bytes
Redo Buffers                7983104 bytes
SQL> alter database mount;

Database altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       MOUNTED
         3 PDB1                           MOUNTED
         4 APDB                           MOUNTED
        
SQL> select count(*) ,FHSCN from x$kcvfh group by FHSCN;

  COUNT(*) FHSCN
---------- --------------------
         3 1820866
         4 2281969
         4 2281982
         4 2282012        
        
//* Here , we see controlfile is aware of PDB



$ sqlplus "/as sysdba"

SQL*Plus: Release 12.2.0.1.0 Production on Wed Sep 19 01:02:13 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL>  startup nomount force pfile='/<path>/<filename>.ora';
ORACLE instance started.

Total System Global Area 1593835520 bytes
Fixed Size                  8793256 bytes
Variable Size             402654040 bytes
Database Buffers         1174405120 bytes
Redo Buffers                7983104 bytes
SQL> alter database mount;

Database altered.

SQL> recover database;
ORA-00279: change 2281969 generated at 09/19/2018 00:51:35 needed for thread 1
ORA-00289: suggestion : /<path>/1_32_981800889.dbf
ORA-00280: change 2281969 for thread 1 is in sequence #32


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/<path>/1_32_981800889.dbf
Log applied.
Media recovery complete.
SQL> alter database open;

Database altered.

SQL>

官方的进一步解释:
We clearly see that the recovery steps applies the same archivelog file twice. When a controlfile is recreated, the recovery initiated will apply archivelog files to only the CDB datafiles, not to the PDB. Once the database open returns the ORA-65088 error, the next database re-start will apply the archivelog files to the PDB for the sake of database consistency.This should explain why Oracle is looking to apply the same archivelog sequence a second time. The following bugs report similar issues. They have both been closed as ‘not a bug’ as this is expected behavior:
BUG 24951417 – ERROR OPENING DATABASE WITH RESETLOGS AFTER CREATE CONTROLFILE
BUG 25172530 – MULTITENANT RESTORE FAILED WITH ORA-65088: DATABASE OPEN
参考:ORA-65088 while opening DB with resetlogs for multi-tenant DB in 12.2 (Doc ID 2449591.1)

发表在 ORA-xxxxx | 标签为 ORA-65088 | 评论关闭

Oracle 19c异常恢复—ORA-01209/ORA-65088

发表于 2024 年 9 月 17 日由惜分飞

由于raid卡bug故障,导致文件系统异常,从而使得数据库无法正常启动,客户找到我之前已经让多人分析,均未恢复成功,查看alert日志,发现他们恢复的时候尝试resetlogs库,然后报ORA-600 kcbzib_kcrsds_1错误

2024-09-15T17:07:32.553215+08:00
alter database open resetlogs
2024-09-15T17:07:32.569110+08:00
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
RESETLOGS after incomplete recovery UNTIL CHANGE 274757454692 time 
.... (PID:8074): Clearing online redo logfile 1 /opt/oracle/oradata/XFF/onlinelog/o1_mf_1_j3k201g9_.log
.... (PID:8074): Clearing online redo logfile 2 /opt/oracle/oradata/XFF/onlinelog/o1_mf_2_j3k201h3_.log
.... (PID:8074): Clearing online redo logfile 3 /opt/oracle/oradata/XFF/onlinelog/o1_mf_3_j3k201hk_.log
Clearing online log 1 of thread 1 sequence number 0
Clearing online log 2 of thread 1 sequence number 0
Clearing online log 3 of thread 1 sequence number 0
2024-09-15T17:07:34.939550+08:00
.... (PID:8074): Clearing online redo logfile 1 complete
.... (PID:8074): Clearing online redo logfile 2 complete
.... (PID:8074): Clearing online redo logfile 3 complete
Online log /opt/oracle/oradata/XFF/onlinelog/o1_mf_1_j3k201g9_.log: Thread 1 Group 1 was previously cleared
Online log /opt/oracle/fast_recovery_area/XFF/onlinelog/o1_mf_1_j3k201l4_.log: Thread 1 Group 1 was previously cleared
Online log /opt/oracle/oradata/XFF/onlinelog/o1_mf_2_j3k201h3_.log: Thread 1 Group 2 was previously cleared
Online log /opt/oracle/fast_recovery_area/XFF/onlinelog/o1_mf_2_j3k201kw_.log: Thread 1 Group 2 was previously cleared
Online log /opt/oracle/oradata/XFF/onlinelog/o1_mf_3_j3k201hk_.log: Thread 1 Group 3 was previously cleared
Online log /opt/oracle/fast_recovery_area/XFF/onlinelog/o1_mf_3_j3k201mt_.log: Thread 1 Group 3 was previously cleared
2024-09-15T17:07:34.966674+08:00
Setting recovery target incarnation to 2
2024-09-15T17:07:34.992357+08:00
Ping without log force is disabled:
  instance mounted in exclusive mode.
Buffer Cache Full DB Caching mode changing from FULL CACHING DISABLED to FULL CACHING ENABLED 
2024-09-15T17:07:34.994329+08:00
Crash Recovery excluding pdb 2 which was cleanly closed.
2024-09-15T17:07:34.994390+08:00
Crash Recovery excluding pdb 3 which was cleanly closed.
2024-09-15T17:07:34.994433+08:00
Crash Recovery excluding pdb 4 which was cleanly closed.
2024-09-15T17:07:34.994474+08:00
Crash Recovery excluding pdb 5 which was cleanly closed.
Initializing SCN for created control file
Database SCN compatibility initialized to 3
Endian type of dictionary set to little
2024-09-15T17:07:35.001752+08:00
Assigning activation ID 2966012017 (0xb0c9c071)
Redo log for group 1, sequence 1 is not located on DAX storage
2024-09-15T17:07:35.015921+08:00
TT00 (PID:8113): Gap Manager starting
2024-09-15T17:07:35.034047+08:00
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /opt/oracle/oradata/XFF/onlinelog/o1_mf_1_j3k201g9_.log
  Current log# 1 seq# 1 mem# 1: /opt/oracle/fast_recovery_area/XFF/onlinelog/o1_mf_1_j3k201l4_.log
Successful open of redo thread 1
2024-09-15T17:07:35.034573+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2024-09-15T17:07:35.063726+08:00
TT03 (PID:8119): Sleep 5 seconds and then try to clear SRLs in 2 time(s)
2024-09-15T17:07:35.129748+08:00
Undo initialization recovery: Parallel FPTR failed: start:2528681 end:2528684 diff:3 ms (0.0 seconds)
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_8074.trc  (incident=146455) (PDBNAME=CDB$ROOT):
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /opt/oracle/diag/rdbms/xff/XFF/incident/incdir_146455/XFF_ora_8074_i146455.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Please look for redo dump in pinned buffers history in incident trace file, if not dumped for what so ever reason,
use the following command to dump it at the earliest. ALTER SYSTEM DUMP REDO DBA MIN 4 128 DBA MAX 4 128 SCN MIN 1;
*****************************************************************
An internal routine has requested a dump of selected redo.
This usually happens following a specific internal error, when
analysis of the redo logs will help Oracle Support with the
diagnosis.
It is recommended that you retain all the redo logs generated (by
all the instances) during the past 12 hours, in case additional
redo dumps are required to help with the diagnosis.
*****************************************************************
Undo initialization recovery: err:600 start: 2528681 end: 2529341 diff: 660 ms (0.7 seconds)
2024-09-15T17:07:35.786923+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_8074.trc:
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
2024-09-15T17:07:35.786967+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_8074.trc:
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_8074.trc  (incident=146456) (PDBNAME=CDB$ROOT):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kcbzib_kcrsds_1], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /opt/oracle/diag/rdbms/xff/XFF/incident/incdir_146456/XFF_ora_8074_i146456.trc
2024-09-15T17:07:36.291884+08:00
opiodr aborting process unknown ospid (8074) as a result of ORA-603
2024-09-15T17:07:36.299928+08:00
ORA-603 : opitsk aborting process
License high water mark = 4
USER(prelim) (ospid: 8074): terminating the instance due to ORA error 600

然后他们又重建了ctl,通过Oracle数据库异常恢复检查脚本(Oracle Database Recovery Check)检查,发现几个问题:
1. PDB$SEED不在该库记录中(由于该pdb中无业务数据,可以忽略)
pdb

2. 部分文件resetlogs 信息不正确(应该是对部分文件offline或者重建ctl的时候没有带上他们)
resetlogs-scn

接手该库进行恢复,尝试resetlogs该库

[oracle@localhost check_db]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Tue Sep 17 11:29:28 2024
Version 19.9.0.0.0

Copyright (c) 1982, 2020, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.9.0.0.0

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-65088: database open should be retried
Process ID: 101712
Session ID: 105 Serial number: 4711

对应的alert日志报错

Endian type of dictionary set to little
2024-09-17T11:29:46.691904+08:00
Assigning activation ID 2966261119 (0xb0cd8d7f)
Redo log for group 1, sequence 1 is not located on DAX storage
2024-09-17T11:29:46.714594+08:00
TT00 (PID:101731): Gap Manager starting
2024-09-17T11:29:46.735407+08:00
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /opt/oracle/oradata/XFF/onlinelog/o1_mf_1_j3k201g9_.log
Successful open of redo thread 1
2024-09-17T11:29:46.736182+08:00
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
stopping change tracking
2024-09-17T11:29:46.774207+08:00
TT03 (PID:101737): Sleep 5 seconds and then try to clear SRLs in 2 time(s)
2024-09-17T11:29:46.793381+08:00
Undo initialization recovery: Parallel FPTR complete: start:99831350 end:99831351 diff:1 ms (0.0 seconds)
Undo initialization recovery: err:0 start: 99831349 end: 99831351 diff: 2 ms (0.0 seconds)
Undo initialization online undo segments: err:0 start: 99831351 end: 99831353 diff: 2 ms (0.0 seconds)
Undo initialization finished serial:0 start:99831349 end:99831356 diff:7 ms (0.0 seconds)
Dictionary check beginning
2024-09-17T11:29:46.817810+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_101712.trc:
ORA-65106: Pluggable database #2 (PDB$SEED) is in an invalid state.
Pluggable Database PDB$SEED (#2) found in data dictionary,
but not in the control file. Adding it to control file.
Pluggable Database PDB1 (#3) found in data dictionary,
but not in the control file. Adding it to control file.
Pluggable Database PDB2 (#4) found in data dictionary,
but not in the control file. Adding it to control file.
Pluggable Database PDB3 (#5) found in data dictionary,
but not in the control file. Adding it to control file.
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
2024-09-17T11:29:46.878684+08:00
Read of datafile '/opt/oracle/oradata/XFF/PDB/datafile/o1_mf_system_j3kc9hl0_.dbf'(fno 9)header failed with ORA-01209
Rereading datafile 9 header failed with ORA-01209
2024-09-17T11:29:46.921314+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_dbw0_100632.trc:
ORA-01186: file 9 failed verification tests
ORA-01122: database file 9 failed verification check
ORA-01110: data file 9: '/opt/oracle/oradata/XFF/PDB/datafile/o1_mf_system_j3kc9hl0_.dbf'
ORA-01209: data file is from before the last RESETLOGS
File 9 not verified due to error ORA-01122
…………
Read of datafile '/opt/oracle/oradata/XFF/datafile/users07.dbf' (fno 39) header failed with ORA-01209
Rereading datafile 39 header failed with ORA-01209
2024-09-17T11:29:46.983955+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_dbw0_100632.trc:
ORA-01186: file 39 failed verification tests
ORA-01122: database file 39 failed verification check
ORA-01110: data file 39: '/opt/oracle/oradata/XFF/datafile/users07.dbf'
ORA-01209: data file is from before the last RESETLOGS
File 39 not verified due to error ORA-01122
2024-09-17T11:29:46.987947+08:00
Dictionary check complete
Verifying minimum file header compatibility for tablespace encryption for pdb 1..
Verifying file header compatibility for tablespace encryption completed for pdb 1
*********************************************************************
WARNING: The following temporary tablespaces in container(CDB$ROOT)
         contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is AL32UTF8
2024-09-17T11:29:47.059806+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_mz00_101739.trc:
ORA-01110: data file 9: '/opt/oracle/oradata/XFF/PDB/datafile/o1_mf_system_j3kc9hl0_.dbf'
ORA-01209: data file is from before the last RESETLOGS
…………
**********************************************************
WARNING: Files may exists in db_recovery_file_dest
that are not known to the database. Use the RMAN command
CATALOG RECOVERY AREA to re-catalog any such files.
If files cannot be cataloged, then manually delete them
using OS command.
One of the following events caused this:
1. A backup controlfile was restored.
2. A standby controlfile was restored.
3. The controlfile was re-created.
4. db_recovery_file_dest had previously been enabled and
   then disabled.
**********************************************************
Starting background process IMCO
2024-09-17T11:29:47.340660+08:00
2024-09-17T11:29:47.382153+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_mz00_101739.trc:
ORA-01110: data file 13: '/opt/oracle/oradata/XFF/PDB/datafile/o1_mf_users_j3kckos2_.dbf'
ORA-01209: data file is from before the last RESETLOGS
replication_dependency_tracking turned off (no async multimaster replication found)
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
2024-09-17T11:29:47.464233+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_mz00_101739.trc:
ORA-01110: data file 14: '/opt/oracle/oradata/XFF/PDB/datafile/o1_mf_users_j3kckqfx_.dbf'
ORA-01209: data file is from before the last RESETLOGS
AQ Processes can not start in restrict mode
Could not open PDB$SEED error=65106
2024-09-17T11:29:47.522825+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_101712.trc:
ORA-65106: Pluggable database #2 (PDB$SEED) is in an invalid state.
ORA-65106: Pluggable database #2 (PDB$SEED) is in an invalid state.
2024-09-17T11:29:47.525249+08:00
db_recovery_file_dest_size of 65536 MB is 0.05% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
2024-09-17T11:29:47.529134+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_101712.trc:
ORA-65088: database open should be retried
2024-09-17T11:29:47.529202+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_101712.trc:
ORA-65088: database open should be retried
2024-09-17T11:29:47.529253+08:00
Error 65088 happened during db open, shutting down database
2024-09-17T11:29:47.545440+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_mz00_101739.trc:
ORA-01110: data file 15: '/opt/oracle/oradata/XFF/PDB/datafile/o1_mf_users_j3kckstd_.dbf'
ORA-01209: data file is from before the last RESETLOGS
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_ora_101712.trc(incident=775863)(PDBNAME=CDB$ROOT):
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-65088: database open should be retried
2024-09-17T11:29:48.046698+08:00
Errors in file /opt/oracle/diag/rdbms/xff/XFF/trace/XFF_mz00_101739.trc:
ORA-01110: data file 21: '/opt/oracle/oradata/XFF/PDB2/datafile/o1_mf_users_j45x90oq_.dbf'
ORA-01209: data file is from before the last RESETLOGS
2024-09-17T11:29:48.073328+08:00
opiodr aborting process unknown ospid (101712) as a result of ORA-603
2024-09-17T11:29:48.081576+08:00
ORA-603 : opitsk aborting process
License high water mark = 122
USER(prelim) (ospid: 101712): terminating the instance due to ORA error 65088
2024-09-17T11:29:49.104770+08:00
Instance terminated by USER(prelim), pid = 101712

主要错误有两个
ORA-01209: data file is from before the last RESETLOGS 和
ORA-65088: database open should be retried
通过分析这两个错误

[oracle@ora19c:/home/oracle]$ oerr ora 65088
65088, 00000, "database open should be retried"
// *Cause:   An inconsistency between the control file and the data dictionary
//           was found and fixed during the database open. The database open
//           needs to be executed again.
// *Action:  Retry the database open.
//
[oracle@ora19c:/home/oracle]$ oerr ora 01209
01209, 00000, "data file is from before the last RESETLOGS"   
// *Cause:  The reset log data in the file header does not match the   
//         control file. If the database is closed or the file is offline,  
//         the backup is old because it was taken before the last ALTER   
//         DATABASE OPEN RESETLOGS command. If opening a database that is   
//         open already by another instance, or if another instance just   
//         brought this file online, the file accessed by this instance is 
//         probably a different version. Otherwise, a backup of the file 
//         probably was restored while the file was in use.   
// *Action: Make the correct file available to the database. Then, either open
//         the database, or execute ALTER SYSTEM CHECK DATAFILES.

ORA-65088参见官方:ORA-65088 while opening DB with resetlogs for multi-tenant DB in 12.2 (Doc ID 2449591.1),应该不是一个技术问题(由于重建ctl+resetlogs导致)
ORA-01209: data file is from before the last RESETLOGS 这个错误,可以简单理解resetlogs的信息比数据文件的checkpoint信息新,对于这种情况,以及结合上述的部分文件resetlogs信息不一致问题,索性直接使用m_scn小工具对其进行批量
m_scn