标签归档:tab$恢复

2025年第一起ORA-600 16703故障恢复

又有一个客户数据库启动报ORA-600 16703错误
ora-600-16703


查看alert日志信息

Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_ora_8784.trc  (incident=20617):
ORA-00600: 内部错误代码, 参数: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Incident details in: D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\incident\incdir_20617\xff_ora_8784_i20617.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_ora_8784.trc:
ORA-00704: 引导程序进程失败
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Errors in file D:\APP\ADMINISTRATOR\diag\rdbms\xff\xff\trace\xff_ora_8784.trc:
ORA-00704: 引导程序进程失败
ORA-00704: 引导程序进程失败
ORA-00600: 内部错误代码, 参数: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 8784): terminating the instance due to error 704
Instance terminated by USER, pid = 8784
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (8784) as a result of ORA-1092

这个故障一般是由于oracle安装介质被注入了恶意脚本,在数据库运行300天之后重启数据库就会遭遇该问题:
警告:互联网中有oracle介质被注入恶意程序导致—ORA-600 16703
对于这个问题,我们这边经过多年的恢复经验积累和研究,可以实现直接open库并且不用做逻辑迁移,完美恢复之后业务直接使用,最大限度恢复数据和最小限度减少停机时间
以往类似恢复case汇总:
tab$恢复错误汇总
ORA-600 16703故障再现
tab$异常被处理之后报ORA-600 13304故障处理
ORA-600 16703直接把orachk备份表插入到tab$恢复
最近遇到几起ORA-600 16703故障(tab$被清空),请引起重视
ORA-00600: internal error code, arguments: [16703], [1403], [32]
aix平台tab$被删除可能出现ORA-600 [16703], [1403], [28]错误
ORA-00600: internal error code, arguments: [16703], [1403], [4] 原因
ORA-00600: internal error code, arguments: [16703], [1403], [4] 故障处理

ORA-600 16703故障,客户找人恢复数据库,数据库被进一步恶意破坏—ORA-00704 ORA-00922

发表在 Oracle备份恢复 | 标签为 , , , | 评论关闭

ORA-00600: internal error code, arguments: [16703], [1403], [4] 故障处理

有一个客户数据库遭遇ORA-600 16703错误,故障原因见:警告:互联网中有oracle介质被注入恶意程序导致—ORA-600 16703,这种故障已经恢复比较多,在这次的恢复中遇到一个新错误,进行记录
接手给客户报错情况ORA-00600: internal error code, arguments: [16703], [1403], [20]
20210515011736


Thu May 13 22:36:11 2021
SMON: enabling cache recovery
Thu May 13 22:36:11 2021
NSA2 started with pid=61, OS id=6261 
Archived Log entry 90224 added for thread 1 sequence 50454 ID 0x19ae1c6c dest 1:
Errors in file /oracle/diag/rdbms/xff/xff/trace/xff_ora_5931.trc  (incident=741052):
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Incident details in: /oracle/diag/rdbms/xff/xff/incident/incdir_741052/xff_ora_5931_i741052.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /oracle/diag/rdbms/xff/xff/trace/xff_ora_5931.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Errors in file /oracle/diag/rdbms/xff/xff/trace/xff_ora_5931.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 5931): terminating the instance due to error 704
Instance terminated by USER, pid = 5931
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (5931) as a result of ORA-1092
Thu May 13 22:36:13 2021
ORA-1092 : opitsk aborting process

这种故障,由于是恶意脚本在数据库启动的时候清空tab$所致,使用bbed对tab$进行反向删除操作,实现数据库打开.
在这次的恢复过程中遇到ORA-600 16703 1403 4的错误

SQL> startup mount pfile='/tmp/pfile';
ORACLE instance started.

Total System Global Area 7.0818E+10 bytes
Fixed Size                  2260448 bytes
Variable Size            1.3422E+10 bytes
Database Buffers         5.7177E+10 bytes
Redo Buffers              217030656 bytes
Database mounted.
SQL> alter database open ;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [4], [], [], [],
[], [], [], [], [], []
Process ID: 51886
Session ID: 3497 Serial number: 3

根据ora-600 16703 1403 4基本上可以判断是由于tab$这个表中缺少obj#=4的对象导致,通过查询正常库,确认是该对象为tab$,也就是说由于tab$对象中少了tab$记录.通过bbed分析确认

SQL> SELECT a.OBJ#
  2        ,TAB#
  3        ,a.DATAOBJ#
  4        ,BOBJ#
  5        ,NAME
  6        ,DBMS_ROWID.ROWID_RELATIVE_FNO (a.ROWID) FILE_ID
  7        ,DBMS_ROWID.ROWID_BLOCK_NUMBER (a.ROWID) BLOCK_ID
  8    FROM TAB$ a, obj$ b
  9   WHERE     a.obj# = b.obj#
 10         AND A.OBJ# IN (4);

      OBJ#       TAB#   DATAOBJ#      BOBJ# NAME
---------- ---------- ---------- ---------- ------------------------------
   FILE_ID   BLOCK_ID
---------- ----------
         4          1          2          2 TAB$
         1        147

BBED> set dba 1,147
        DBA             0x00400093 (4194451 1,147)

BBED> x /rnnnnnnnnnnnnncnnnnnnnntnnnnnnnnnncct  *kdbr[14]
rowdata[6848]                               @7349
-------------
flag@7349: 0x20 (KDRHFH)
lock@7350: 0x02
cols@7351:    0
nrid@7352:0x00407b09.1
BBED> set dba 0x00407b09
        DBA             0x00407b09 (4225801 1,31497)

BBED> p kdbt[1]
struct kdbt[1], 4 bytes                     @110
   sb2 kdbtoffs                             @110      10
   sb2 kdbtnrow                             @112      2

BBED> x /rnnnnnnnnnnnnncnnnnnnnntnnnnnnnnnncct  *kdbr[11]
rowdata[815]                                @4436
------------
flag@4436: 0x5c (KDRHFL, KDRHFF, KDRHFD, KDRHFC)
lock@4437: 0x02
cols@4438:    0
ckix@4439:    8

BBED> x /rn  *kdbr[8]
rowdata[950]                                @4571
------------
flag@4571: 0xac (KDRHFL, KDRHFF, KDRHFH, KDRHFK)
lock@4572: 0x00
cols@4573:    1
kref@4574:    1
hrid@4576:0x00400093.8
nrid@4582:0x00400094.0
col    0[2] @4590: 4

确认该记录发生了行迁移导致该问题,对其对应的block进行修复,数据库正常打开.

发表在 非常规恢复 | 标签为 , , , , , , | 评论关闭