标签归档：ORA-600 17147

ORA-07445 opiaba—绑定变量超过65535导致实例crash

发表于 2023 年 3 月 15 日 由 惜分飞

数据库异常报ORA-07445 opiaba，ORA-00600 17147错,导致实例crash

Wed Mar 15 09:48:06 2023
Exception [type: SIGSEGV, SI_KERNEL(general_protection)] [ADDR:0x0] [PC:0x187B953, opiaba()+639] [flags: 0x0, count: 1]
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei3/trace/xifenfei3_ora_169909.trc  (incident=446531):
ORA-07445: 出现异常错误: 核心转储 [opiaba()+639] [SIGSEGV] [ADDR:0x0] [PC:0x187B953] [SI_KERNEL(general_protection)] []
Incident details in: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei3/incident/incdir_446531/xifenfei3_ora_169909_i446531.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Wed Mar 15 09:48:07 2023
Dumping diagnostic data in directory=[cdmp_20230315094807], requested by (instance=3, osid=169909), summary=[incident=446531].
Wed Mar 15 09:48:08 2023
Sweep [inc][446531]: completed
Sweep [inc2][446531]: completed
Wed Mar 15 09:48:33 2023
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei3/trace/xifenfei3_pmon_161557.trc  (incident=440035):
ORA-00600: internal error code, arguments: [17147], [0x4AFC25D0C8], [], [], [], [], [], [], [], [], [], []
Incident details in: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei3/incident/incdir_440035/xifenfei3_pmon_161557_i440035.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Wed Mar 15 09:48:33 2023
Dumping diagnostic data in directory=[cdmp_20230315094833], requested by (instance=3, osid=161557 (PMON)), summary=[incident=440035].
Errors in file /u01/app/oracle/diag/rdbms/xifenfei/xifenfei3/trace/xifenfei3_pmon_161557.trc:
ORA-00600: internal error code, arguments: [17147], [0x4AFC25D0C8], [], [], [], [], [], [], [], [], [], []
PMON (ospid: 161557): terminating the instance due to error 472
Wed Mar 15 09:48:34 2023
opiodr aborting process unknown ospid (170089) as a result of ORA-1092

对应的trace文件信息

*** 2023-03-15 09:09:34.862
*** SESSION ID:(1858.63187) 2023-03-15 09:09:34.862
*** CLIENT ID:() 2023-03-15 09:09:34.862
*** SERVICE NAME:(xifenfei) 2023-03-15 09:09:34.862
*** MODULE NAME:(JDBC Thin Client) 2023-03-15 09:09:34.862
*** ACTION NAME:() 2023-03-15 09:09:34.862
 
Dump continued from file: /u01/app/oracle/diag/rdbms/xifenfei/xifenfei3/trace/xifenfei3_ora_116886.trc
ORA-07445: 出现异常错误: 核心转储 [opiaba()+639] [SIGSEGV] [ADDR:0x0] [PC:0x187B953] [SI_KERNEL(general_protection)] []

========= Dump for incident 326049 (ORA 7445 [opiaba()+639]) ========
----- Beginning of Customized Incident Dump(s) -----
Exception [type: SIGSEGV, SI_KERNEL(general_protection)] [ADDR:0x0] [PC:0x187B953, opiaba()+639] [flags: 0x0, count: 1]
Registers:
%rax: 0x00000046fa212588 %rbx: 0x00000046762ee798 %rcx: 0x0000000000003200
%rdx: 0x0000000000000000 %rdi: 0x0000000000007340 %rsi: 0x0000000000007340
%rsp: 0x00007fffffff41d0 %rbp: 0x00007fffffff4240  %r8: 0x00000044a8599f00
 %r9: 0x0000000000000099 %r10: 0x0000000000000b34 %r11: 0x00000046fa211720
%r12: 0x00000046fa5c7488 %r13: 0x0000000000000000 %r14: 0x0200000046fa2125
%r15: 0x0000000000000005 %rip: 0x000000000187b953 %efl: 0x0000000000010202
  opiaba()+625 (0x187b945) jmp 0x187b982
  opiaba()+627 (0x187b947) movzbl 0x2c(%rbx),%r15d
  opiaba()+632 (0x187b94c) mov %r14,-0x60(%rbp)
  opiaba()+636 (0x187b950) mov %rax,%r14
> opiaba()+639 (0x187b953) movswq 0xa(%r14),%rcx
  opiaba()+644 (0x187b958) cmp %ecx,%r15d
  opiaba()+647 (0x187b95b) jne 0x187b976
  opiaba()+649 (0x187b95d) mov 0x18(%rbx),%rdi
  opiaba()+653 (0x187b961) lea 0xc(%r14),%rsi

*** 2023-03-15 09:09:34.863
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x3, level=3, mask=0x0)
----- Current SQL Statement for this session (sql_id=5cvg8vqsbr8j6) -----
BEGIN  
            UPDATE
             t_xifenfei 
             SET valid_param=:1 , comm_addr1 = :2 ,
                password=:3 , customer_type = :4 ,
                comm_no=:5 , date_grade_no = :6 ,
                date_grade_flag=:7 , curve_config_no = :8 ,
                curve_config_flag=:9 , baudrate = :10 ,
                meas_order=:11 , meter_no = :12 ,
                kind_no=:13 , tariff_count = :14 ,
                meter_digits = :15 , import_user=:16 ,
                readmeter_flag = :17 ,status_code=:18  
            WHERE meter_id = :19 
         ; 
…………
         ; 
            UPDATE
             t_xifenfei 
             SET valid_param=:94982 , comm_addr1 = :94983 ,
                password=:94984 , customer_type = :94985 ,
                comm_no=:94986 , date_grade_no = :94987 ,
                date_grade_flag=:94988 , curve_config_no = :94989 ,
                curve_config_flag=:94990 , baudrate = :94991 ,
                meas_order=:94992 , meter_no = :94993 ,
                kind_no=:94994 , tariff_count = :94995 ,
                meter_digits = :94996 , import_user=:94997 ,
                readmeter_flag = :94998 ,status_code=:94999  
            WHERE meter_id = :95000 
         ;END;

写了一个begin end,里面对于同一个update语句进行多次绑定变量实现批量提交功能,绑定变量的数量达到95000个,远超oracle官方限制的65535的极限,触发类似Bug 12578873 ORA-7445 [opiaba] when using more than 65535 bind variables
20230315132517

处理方法减少绑定变量数量,不能超过65535个

发表在 ORA-xxxxx | 标签为 , | 评论关闭

10g数据库遭遇ORA-600 16703

发表于 2019 年 1 月 5 日 由 惜分飞

ORA-600 16703
有客户反馈10g数据库启动报ORA-00704 ORA-00600,具体如下

Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bi
PL/SQL Release 10.2.0.3.0 - Production
CORE	10.2.0.3.0	Production
TNS for IBM/AIX RISC System/6000: Version 10.2.0.3.0 - Productio
NLSRTL Version 10.2.0.3.0 - Production

Fri Jan  4 10:11:05 2019
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan  4 10:11:05 2019
SMON: enabling cache recovery
Fri Jan  4 10:11:05 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1826952.trc:
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], []
Fri Jan  4 10:11:06 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1826952.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], []
Fri Jan  4 10:11:06 2019
Error 704 happened during db open, shutting down database
USER: terminating instance due to error 704
Instance terminated by USER, pid = 1826952
ORA-1092 signalled during: alter database open...

看到这个错误第一想到以前的ORA-00600: internal error code, arguments: [16703], [1403], [20], [], [], [], [], [], [], [], [], []，但是有几个地方不同:1)根据以往恢复经验这个错误都出现在11.2.0.4版本数据库中;2)以往的ORA-600 16703错误最后值是20,而这次是28

分析数据库故障过程
节点一因为其他错误重启,重启之后报大量错误,然后abort掉.

--节点1
Fri Jan  4 06:11:00 2019
Database Characterset is ZHS16GBK
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=29, OS id=2883786
Fri Jan  4 06:11:07 2019
SMON: Parallel transaction recovery tried
Fri Jan  4 06:11:07 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2769106.trc:
ORA-00600: internal error code, arguments: [16607], [0x70000020E4E7258], [257], [0], [], [], [], []
Fri Jan  4 06:11:09 2019
ORA-600 signalled during: ALTER DATABASE OPEN...
Fri Jan  4 06:11:09 2019
Trace dumping is performing id=[cdmp_20190104061109]
Fri Jan  4 06:11:30 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2240550.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-20001: the IP can not logon
ORA-06512: at line 36
Fri Jan  4 06:12:35 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2793508.trc:
ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [65], [], [], []
Fri Jan  4 06:12:40 2019
Trace dumping is performing id=[cdmp_20190104061240]
Fri Jan  4 06:16:10 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_1933540.trc:
ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], []
Fri Jan  4 06:16:10 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2793546.trc:
ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], []
Fri Jan  4 06:16:11 2019
Trace dumping is performing id=[cdmp_20190104061611]
Fri Jan  4 06:16:15 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_smon_3055636.trc:
ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], []
Fri Jan  4 06:16:16 2019
Non-fatal internal error happenned while SMON was doing logging scn->time mapping.
SMON encountered 1 out of maximum 100 non-fatal internal errors.
Fri Jan  4 06:17:28 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_2035774.trc:
ORA-00600: internal error code, arguments: [16607], [0x70000020D957260], [1793], [0], [], [], [], []
…………
Fri Jan  4 06:56:47 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_smon_3055636.trc:
ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], []
Fri Jan  4 06:56:49 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei1_ora_3072030.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []
ORA-06544: PL/SQL: internal error, arguments: [], [interpreter cannot interpret pcode], [], [], [], [], [], []
ORA-06544: PL/SQL: internal error, arguments: [], [interpreter cannot interpret pcode], [], [], [], [], [], []
Fri Jan  4 06:56:50 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_pmon_3092524.trc:
ORA-00474: SMON process terminated with error
Fri Jan  4 06:56:50 2019
PMON: terminating instance due to error 474
Fri Jan  4 06:56:50 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei1_lms0_3178590.trc:
ORA-00474: SMON process terminated with error
Fri Jan  4 06:56:50 2019
System state dump is made for local instance
System State dumped to trace file /oracle/admin/xifenfei/bdump/xifenfei1_diag_434192.trc
Fri Jan  4 06:56:51 2019
Shutting down instance (abort)

节点一重启触发故障之后，节点二也开始报错，然后数据库直接挂掉

--节点2
Fri Jan  4 06:21:19 2019
Trace dumping is performing id=[cdmp_20190104062118]
Fri Jan  4 06:21:22 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j000_1253440.trc:
ORA-00600: internal error code, arguments: [kghGetHpSz1], [0x7000001DF886220], [], [], [], [], [], []
Fri Jan  4 06:21:24 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_1835262.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-20001: the IP can not logon
ORA-06512: at line 36
Fri Jan  4 06:21:56 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j002_1777888.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []
Fri Jan  4 06:24:38 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j003_1589340.trc:
ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [97952], [], [], []
Fri Jan  4 06:24:44 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j003_1589340.trc:
ORA-12012: error on auto execute of job 351
ORA-12008: error in materialized view refresh path
ORA-00600: internal error code, arguments: [16659], [kqldtu], [D], [0], [97952], [], [], []
Fri Jan  4 06:28:48 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j001_1417456.trc:
ORA-00600: internal error code, arguments: [kkzufst], [18446744073709551615], [], [], [], [], [], []
Fri Jan  4 06:28:49 2019
ORA-00600: internal error code, arguments: [kghstack_underflow_internal_2], [0x1104B2580], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [insSetColumnInfo_1], [8], [8], [], [], [], [], []
Fri Jan  4 07:30:25 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_j002_2121800.trc:
ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], []
ORA-00600: internal error code, arguments: [kggfaAllocFunc1], [], [], [], [], [], [], []
…………
Fri Jan  4 07:31:23 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_pmon_634956.trc:
ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], []
Fri Jan  4 07:31:24 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_pmon_634956.trc:
ORA-00600: internal error code, arguments: [17147], [0x7000001D291D230], [], [], [], [], [], []
Fri Jan  4 07:31:24 2019
PMON: terminating instance due to error 472
Fri Jan  4 07:31:24 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_lms0_569432.trc:
ORA-00472: PMON  process terminated with error

两个节点再次启动报ORA-600 16703错误

Fri Jan  4 07:31:50 2019
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Jan  4 07:31:50 2019
SMON: enabling cache recovery
Fri Jan  4 07:31:50 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_3813516.trc:
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], []
Fri Jan  4 07:31:52 2019
Errors in file /oracle/admin/xifenfei/udump/xifenfei2_ora_3813516.trc:
ORA-00704: bootstrap process failure
ORA-00704: bootstrap process failure
ORA-00600: internal error code, arguments: [16703], [1403], [28], [], [], [], [], []
Fri Jan  4 07:31:52 2019
Error 704 happened during db open, shutting down database
USER: terminating instance due to error 704
Fri Jan  4 07:31:52 2019
Errors in file /oracle/admin/xifenfei/bdump/xifenfei2_lms0_3780764.trc:
ORA-00704: bootstrap process failure
Instance terminated by USER, pid = 3813516
ORA-1092 signalled during: ALTER DATABASE OPEN...

10046跟踪启动过程
ora-600-16703-10046

分析故障原因
通过分析,确认该数据库被注入了恶意脚本,当发生重启之后导致数据库核心基表被破坏,从而使得数据库无法正常启动

procedure     DBMS_DBMONITOR wrapped
a000000
369
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
abcd
7
15a 171
VKGnETw5GkgNlUvXadIYpQ2thwAwgzKur9zWZ47SR+pHN0bgoPvQ8ezdufYxFkzCmAQA8xT2
IQkFph+2IhlEBrPrXe14giDow/HzZI43FwLiMlynCYjnzQh3aXRSIVOalcwGAfUvgCip6Eng
OWA8Vq49YJ38WCPZjq2P5Dc428Wa1ZOPMb+E7GPs8ZOWM7RWsQdMzx/pqFncbX/tLwp0NY5E
Uu4E54MZ34yVtDQybwljVqp06KHqWN/ZwZJpvT+2gO4hRNX2UyE7laWCXzM2IR05BTGf2yoZ
+E7eIn6kciinFmhcUiBuszxE0pykt+moWZuuDuj9ebUXmj+0Mx7A+eQc6tp7wLHRCHYb6p0D
VtDc4f6

通过对损坏的字典进行恢复,实现数据0丢失
open-database

发表在 非常规恢复 | 标签为 , , , , , , , | 评论关闭