标签归档：ORA-600 16703

12C数据库遭遇ORA-600 16703

发表于 2019 年 3 月 11 日由惜分飞

关于tab$异常的问题,我们在10G(10g数据库遭遇ORA-600 16703),11G(警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703)中有发现,近期在12C版本中也发现该问题.再次提醒请注意检查安装介质的md5值和现在运行的库中是否有相关恶意程序
数据库版本

alert日志报错

************************************************************
Warning: The SCN headroom for this database is only 45 days!
************************************************************
************************************************************
Warning: The SCN headroom for this database is only 45 days!
************************************************************
Picked broadcast on commit scheme to generate SCNs
Fri Jan 25 11:34:55 2019
Starting background process TMON
Fri Jan 25 11:34:55 2019
TMON started with pid=104, OS id=47282
Fri Jan 25 11:34:55 2019
Thread 1 advanced to log sequence 11942 (thread open)
Thread 1 opened at log sequence 11942
Current log# 5 seq# 11942 mem# 0: +DATA/xifenfei/ONLINELOG/redo01_01.log
Successful open of redo thread 1
Fri Jan 25 11:34:55 2019
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan 25 11:34:55 2019
Network Resource Management enabled for Process LG00 (pid 451491) for Exadata I/O
Fri Jan 25 11:34:55 2019
SMON: enabling cache recovery
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_46443.trc (incident=394486):
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Fri Jan 25 11:34:56 2019
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_46443.trc:
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Fri Jan 25 11:34:56 2019
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_46443.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Fri Jan 25 11:34:56 2019
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_46443.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []
Fri Jan 25 11:34:56 2019
Error 704 happened during db open, shutting down database
USER (ospid: 46443): terminating the instance due to error 704
Fri Jan 25 11:34:57 2019
Instance terminated by USER, pid = 46443

处理思路基本上和其他两个版本一样,都是对tab$进行修复,然后直接open数据库,但是在这次的恢复中遇到ORA-00600 kdfReserveSingle_1错误
ORA-600 kdfReserveSingle_1

SQL> alter database open ;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [kdfReserveSingle_1], [0], [65280],
[], [], [], [], [], [], [], [], []
Process ID: 319852
Session ID: 2401 Serial number: 24555

---alert日志
Starting background process TMON
Fri Jan 25 20:51:41 2019
TMON started with pid=94, OS id=322554
Fri Jan 25 20:51:41 2019
Thread 1 advanced to log sequence 11943 (thread open)
Thread 1 opened at log sequence 11943
Current log# 6 seq# 11943 mem# 0: +DATA/xifenfei/ONLINELOG/redo02_01.log
Successful open of redo thread 1
Fri Jan 25 20:51:41 2019
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan 25 20:51:41 2019
Network Resource Management enabled for Process LG00 (pid 319754) for Exadata I/O
Fri Jan 25 20:51:41 2019
SMON: enabling cache recovery
Fri Jan 25 20:51:41 2019
[319852] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:88159754 end:88160264 diff:510 ms (0.5 seconds)
Verifying minimum file header compatibility (11g) for tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
Fri Jan 25 20:51:41 2019
SMON: enabling tx recovery
Fri Jan 25 20:51:41 2019
Database Characterset is ZHS16GBK
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_319852.trc (incident=466446):
ORA-00600: internal error code, arguments: [kdfReserveSingle_1], [0], [65280], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Fri Jan 25 20:51:42 2019
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_319852.trc:
ORA-00600: internal error code, arguments: [kdfReserveSingle_1], [0], [65280], [], [], [], [], [], [], [], [], []
Fri Jan 25 20:51:42 2019
Errors in file /u01/oracle/app/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_319852.trc:
ORA-00600: internal error code, arguments: [kdfReserveSingle_1], [0], [65280], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 319852): terminating the instance due to error 600
Fri Jan 25 20:51:43 2019
Instance terminated by USER, pid = 319852
ORA-1092 signalled during: alter database open…
opiodr aborting process unknown ospid (319852) as a result of ORA-1092
Fri Jan 25 20:51:49 2019
ORA-1092 : opitsk aborting process

这个错误主要是由于12c的tab$和以前版本比有一些特殊,导致开始恢复不全,通过继续修复tab$之后数据库正常open.
相关文章汇总
tab$恢复错误汇总
 12C数据库遭遇ORA-600 16703
10G数据库遭遇ORA-600 16703
ORA-600 16703故障解析—tab$表被清空
 警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703

发表在 Oracle | 标签为 12c tab$, delete tab$, kdfReserveSingle_1, ORA-600 16703, ORA-600 kdfReserveSingle_1, tab$清空 | 评论关闭

在以前多以前发现的tab$被恶意脚本删除的问题(ORA-600 16703故障解析—tab$表被清空,警告：互联网中有oracle介质被注入恶意程序导致—ORA-600 16703),虽然多次强调注意Oracle安装介质安全,但是很不幸,还是大量客户中招.我们这一年多对于tab$的故障进行了大量case处理,拯救了大量客户的核心数据,也积累了一些常见的可能遭遇的错误.主要恢复思路是使用bbed处理异常block,让数据库open起来.
ORA-00704 ORA-39700
有核心基表处理异常导致

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Process ID: 1603
Session ID: 1 Serial number: 5

Sun Jan 06 21:30:14 2019
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1603.trc:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1603.trc:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Error 704 happened during db open, shutting down database
USER (ospid: 1603): terminating the instance due to error 704
Instance terminated by USER, pid = 1603
ORA-1092 signalled during: alter database open...
opiodr aborting process unknown ospid (1603) as a result of ORA-1092
Sun Jan 06 21:30:14 2019
ORA-1092 : opitsk aborting process

ora-704 ora-604 ora-01555
由于scn异常导致

SQL> alter database open upgrade;
alter database open upgrade
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name
"_SYSSMU2_2996391332$" too small
Process ID: 26520
Session ID: 1 Serial number: 5

Sun Jan 06 19:49:12 2019
SMON: enabling cache recovery
ORA-01555 caused by SQL statement below (SQL ID: bqbdby3c400p7, SCN: 0x0022.1117ef75):
select rowcnt,blkcnt,empcnt,avgspc,chncnt,avgrln,nvl(degree,1), nvl(instances,1) from tab$ where obj# = :1
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_26520.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name "_SYSSMU2_2996391332$" too small
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_26520.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-01555: snapshot too old: rollback segment number 2 with name "_SYSSMU2_2996391332$" too small
Error 704 happened during db open, shutting down database
USER (ospid: 26520): terminating the instance due to error 704
Instance terminated by USER, pid = 26520
ORA-1092 signalled during: alter database open upgrade...
opiodr aborting process unknown ospid (26520) as a result of ORA-1092

ORA-600 13304
有核心基表处理异常导致

SQL> startup mount;
ORACLE instance started.

Total System Global Area  521936896 bytes
Fixed Size                  2254824 bytes
Variable Size             352323608 bytes
Database Buffers          163577856 bytes
Redo Buffers                3780608 bytes
Database mounted.
SQL> alter database open upgrade;
alter database open upgrade
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [],
[], [], [], []
Process ID: 1724
Session ID: 1 Serial number: 5

Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Sun Jan 06 21:31:04 2019
SMON: enabling cache recovery
[1724] Successfully onlined Undo Tablespace 2.
Undo initialization finished serial:0 start:2239884804 end:2239884864 diff:60 (0 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is ZHS16GBK
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc  (incident=61755):
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl/incident/incdir_61755/orcl_ora_1724_i61755.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc:
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl/trace/orcl_ora_1724.trc:
ORA-00600: internal error code, arguments: [13304], [], [], [], [], [], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 1724): terminating the instance due to error 600
Instance terminated by USER, pid = 1724
ORA-1092 signalled during: alter database open upgrade...
opiodr aborting process unknown ospid (1724) as a result of ORA-1092
Sun Jan 06 21:31:06 2019
ORA-1092 : opitsk aborting process

ORA-00704 ORA-600 kdBlkCheckError
恢复的block有逻辑坏块

SQL> startup         
ORACLE instance started.

Total System Global Area 3056513024 bytes
Fixed Size                  2257152 bytes
Variable Size             704646912 bytes
Database Buffers         2332033024 bytes
Redo Buffers               17575936 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497],
[6121], [], [], [], [], [], [], [], []
Process ID: 76932
Session ID: 191 Serial number: 3

Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
SMON: enabling cache recovery
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc  (incident=6153):
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/incident/incdir_6153/xifenfei_ora_76932_i6153.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei/trace/xifenfei_ora_76932.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-00607: Internal error occurred while making a change to a data block
ORA-00600: internal error code, arguments: [kdBlkCheckError], [1], [31497], [6121], [], [], [], [], [], [], [], []
Error 704 happened during db open, shutting down database
USER (ospid: 76932): terminating the instance due to error 704
Instance terminated by USER, pid = 76932
ORA-1092 signalled during: ALTER DATABASE OPEN...
opiodr aborting process unknown ospid (76932) as a result of ORA-1092
Sat Feb 22 11:04:19 2014
ORA-1092 : opitsk aborting process

发表在非常规恢复 | 标签为 bbed tab$, kdBlkCheckError, ora 600 1555, ORA-00704 ORA-39700, ORA-600 13304, ORA-600 16703, ORA-600 kdBlkCheckError, ORA-704 ORA-39700, tab$ 删除, tab$ 恢复, tab$清空 | 评论关闭

10g数据库遭遇ORA-600 16703

发表于 2019 年 1 月 5 日由惜分飞

ORA-600 16703
有客户反馈10g数据库启动报ORA-00704 ORA-00600,具体如下

Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bi
PL/SQL Release 10.2.0.3.0 - Production
CORE	10.2.0.3.0	Production
TNS for IBM/AIX RISC System/6000: Version 10.2.0.3.0 - Productio
NLSRTL Version 10.2.0.3.0 - Production

Fri Jan  4 10:11:05 2019
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan  4 10:11:05 2019
SMON: enabling cache recovery
Fri Jan  4 10:11:05 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1826952.trc:
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], []
Fri Jan  4 10:11:06 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1826952.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], []
Fri Jan  4 10:11:06 2019
Error 704 happened during db open, shutting down database
USER: terminating instance due to error 704
Instance terminated by USER, pid = 1826952
ORA-1092 signalled during: alter database open...

看到这个错误第一想到以前的ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []，但是有几个地方不同:1)根据以往恢复经验这个错误都出现在11.2.0.4版本数据库中;2)以往的ORA-600 16703错误最后值是20,而这次是28

分析数据库故障过程
节点一因为其他错误重启,重启之后报大量错误,然后abort掉.

--节点1
Fri Jan  4 06:11:00 2019
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=29, OS id=2883786
Fri Jan  4 06:11:07 2019
SMON: Parallel transaction recovery tried
Fri Jan  4 06:11:07 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2769106.trc:
ORA-00600: internal error code, arguments: [16607], [0x70000020E4E7258], [257], [0], [], [], [], []
Fri Jan  4 06:11:09 2019
ORA-600 signalled during: ALTER DATABASE OPEN...
Fri Jan  4 06:11:09 2019
Trace dumping is performing id=[cdmp_20190104061109]
Fri Jan  4 06:11:30 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2240550.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-20001: the IP can not logon
ORA-06512: at line 36
Fri Jan  4 06:12:35 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2793508.trc:
ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [65], [], [], []
Fri Jan  4 06:12:40 2019
Trace dumping is performing id=[cdmp_20190104061240]
Fri Jan  4 06:16:10 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1933540.trc:
ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], []
Fri Jan  4 06:16:10 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2793546.trc:
ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], []
Fri Jan  4 06:16:11 2019
Trace dumping is performing id=[cdmp_20190104061611]
Fri Jan  4 06:16:15 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_smon_3055636.trc:
ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], []
Fri Jan  4 06:16:16 2019
Non-fatal internal error happenned while SMON was doing logging scn->time mapping.
SMON encountered 1 out of maximum 100 non-fatal internal errors.
Fri Jan  4 06:17:28 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2035774.trc:
ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], []
…………
Fri Jan  4 06:56:47 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_smon_3055636.trc:
ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], []
Fri Jan  4 06:56:49 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_3072030.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []
ORA-06544: PL/SQL: internal error, arguments: [], [interpreter cannot interpret pcode], [], [], [], [], [], []
ORA-06544: PL/SQL: internal error, arguments: [], [interpreter cannot interpret pcode], [], [], [], [], [], []
Fri Jan  4 06:56:50 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_pmon_3092524.trc:
ORA-00474: SMON process terminated with error
Fri Jan  4 06:56:50 2019
PMON: terminating instance due to error 474
Fri Jan  4 06:56:50 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_lms0_3178590.trc:
ORA-00474: SMON process terminated with error
Fri Jan  4 06:56:50 2019
System state dump is made for local instance
System State dumped to trace file /oracle/admin/xifenfei/bdump/xifenfei1_diag_434192.trc
Fri Jan  4 06:56:51 2019
Shutting down instance (abort)

节点一重启触发故障之后，节点二也开始报错，然后数据库直接挂掉

--节点2
Fri Jan  4 06:21:19 2019
Trace dumping is performing id=[cdmp_20190104062118]
Fri Jan  4 06:21:22 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j000_1253440.trc:
ORA-00600: internal error code, arguments: [kghGetHpSz1], [0x7000001DF886220], [], [], [], [], [], []
Fri Jan  4 06:21:24 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_1835262.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-20001: the IP can not logon
ORA-06512: at line 36
Fri Jan  4 06:21:56 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j002_1777888.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []
Fri Jan  4 06:24:38 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j003_1589340.trc:
ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [97952], [], [], []
Fri Jan  4 06:24:44 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j003_1589340.trc:
ORA-12012: error on auto execute of job 351
ORA-12008: error in materialized view refresh path
ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [97952], [], [], []
Fri Jan  4 06:28:48 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j001_1417456.trc:
ORA-00600: internal error code, arguments: [kkzufst], [18446744073709551615], [], [], [], [], [], []
Fri Jan  4 06:28:49 2019
ORA-00600: internal error code, arguments: [kghstack_underflow_internal_2], [0x1104B2580], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], []
Fri Jan  4 07:30:25 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j002_2121800.trc:
ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kggfaAllocFunc1], [], [], [], [], [], [], []
…………
Fri Jan  4 07:31:23 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_pmon_634956.trc:
ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], []
Fri Jan  4 07:31:24 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_pmon_634956.trc:
ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], []
Fri Jan  4 07:31:24 2019
PMON: terminating instance due to error 472
Fri Jan  4 07:31:24 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_lms0_569432.trc:
ORA-00472: PMON  process terminated with error

两个节点再次启动报ORA-600 16703错误

Fri Jan  4 07:31:50 2019
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan  4 07:31:50 2019
SMON: enabling cache recovery
Fri Jan  4 07:31:50 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_3813516.trc:
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], []
Fri Jan  4 07:31:52 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_3813516.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], []
Fri Jan  4 07:31:52 2019
Error 704 happened during db open, shutting down database
USER: terminating instance due to error 704
Fri Jan  4 07:31:52 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_lms0_3780764.trc:
ORA-00704: bootstrap process failure
Instance terminated by USER, pid = 3813516
ORA-1092 signalled during: ALTER DATABASE OPEN...

10046跟踪启动过程

分析故障原因
通过分析,确认该数据库被注入了恶意脚本,当发生重启之后导致数据库核心基表被破坏,从而使得数据库无法正常启动

procedure     DBMS_DBMONITOR wrapped
a000000
369
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
7
15a 171
VKGnETw5GkgNlUvXadIYpQ2thwAwgzKur9zWZ47SR+pHN0bgoPvQ8ezdufYxFkzCmAQA8xT2
IQkFph+2IhlEBrPrXe14giDow/HzZI43FwLiMlynCYjnzQh3aXRSIVOalcwGAfUvgCip6Eng
OWA8Vq49YJ38WCPZjq2P5Dc428Wa1ZOPMb+E7GPs8ZOWM7RWsQdMzx/pqFncbX/tLwp0NY5E
Uu4E54MZ34yVtDQybwljVqp06KHqWN/ZwZJpvT+2gO4hRNX2UyE7laWCXzM2IR05BTGf2yoZ
+E7eIn6kciinFmhcUiBuszxE0pykt+moWZuuDuj9ebUXmj+0Mx7A+eQc6tp7wLHRCHYb6p0D
VtDc4f6

通过对损坏的字典进行恢复,实现数据0丢失