-
加我微信
加我QQ
标签云
asm恢复 bbed bootstrap$ dul In Memory kcbzib_kcrsds_1 kccpb_sanity_check_2 kfed MySQL恢复 ORA-00312 ORA-00607 ORA-00704 ORA-01110 ORA-01555 ORA-01578 ORA-08103 ORA-600 2131 ORA-600 2662 ORA-600 2663 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-600 kcbzib_kcrsds_1 ORA-600 KCLCHKBLK_4 ORA-15042 ORA-15196 ORACLE 12C oracle dul ORACLE PATCH Oracle Recovery Tools oracle加密恢复 oracle勒索 oracle勒索恢复 oracle异常恢复 Oracle 恢复 ORACLE恢复 ORACLE数据库恢复 oracle 比特币 OSD-04016 YOUR FILES ARE ENCRYPTED 勒索恢复 比特币加密文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (102)
- 数据库 (1,670)
- DB2 (22)
- MySQL (73)
- Oracle (1,532)
- Data Guard (52)
- EXADATA (8)
- GoldenGate (21)
- ORA-xxxxx (159)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (14)
- ORACLE 21C (3)
- Oracle 23ai (7)
- Oracle ASM (65)
- Oracle Bug (8)
- Oracle RAC (52)
- Oracle 安全 (6)
- Oracle 开发 (28)
- Oracle 监听 (28)
- Oracle备份恢复 (560)
- Oracle安装升级 (91)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (78)
- PostgreSQL (18)
- PostgreSQL恢复 (6)
- SQL Server (27)
- SQL Server恢复 (8)
- TimesTen (7)
- 达梦数据库 (2)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (37)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (20)
-
最近发表
- ORA-600 krse_arc_complete.4
- Oracle 19c 202410补丁（RUs+OJVM）
- ntfs MFT损坏(ntfs文件系统故障)导致oracle异常恢复
- .mkp扩展名oracle数据文件加密恢复
- 清空redo,导致ORA-27048: skgfifi: file header information is invalid
- A_H_README_TO_RECOVER勒索恢复
- 通过alert日志分析客户自行对一个数据库恢复的来龙去脉和点评
- ORA-12514: TNS: 监听进程不能解析在连接描述符中给出的SERVICE_NAME
- ORA-01092 ORA-00604 ORA-01558故障处理
- ORA-65088: database open should be retried
- Oracle 19c异常恢复—ORA-01209/ORA-65088
- ORA-600 16703故障再现
- 数据库启动报ORA-27102 OSD-00026 O/S-Error: (OS 1455)
- .[metro777@cock.li].Elbie勒索病毒加密数据库恢复
- 应用连接错误,初始化mysql数据库恢复
- RAC默认服务配置优先节点
- Oracle 19c RAC 替换私网操作
- 监听报TNS-12541 TNS-12560 TNS-00511错误
- drop tablespace xxx including contents恢复
- Linux 8 修改网卡名称
标签归档：Controlfile sequence number
Controlfile sequence number in file header is different from the one in memory
最近一段时间遇到多个客户类似如下错误,由于某种原因导致rac两个节点oracle实例一段时间之后就crash,然后重启,系统非常不稳定,严重影响业务使用,一个节点alert日志有类似日志
Sat Oct 22 13:01:59 2022
Instance recovery: looking for dead threads
********************* ATTENTION: ********************
The controlfile header block returned by the OS
has a sequence number that is too old.
The controlfile might be corrupted.
PLEASE DO NOT ATTEMPT TO START UP THE INSTANCE
without following the steps below.
RE-STARTING THE INSTANCE CAN CAUSE SERIOUS DAMAGE
TO THE DATABASE, if the controlfile is truly corrupted.
In order to re-start the instance safely,
please do the following:
(1) Save all copies of the controlfile for later
analysis and contact your OS vendor and Oracle support.
(2) Mount the instance and issue:
ALTER DATABASE BACKUP CONTROLFILE TO TRACE;
(3) Unmount the instance.
(4) Use the script in the trace file to
RE-CREATE THE CONTROLFILE and open the database.
*****************************************************
Submitted all GCS remote-cache requests
Post SMON to start 1st pass IR
Fix write in gcs resources
Reconfiguration complete
SMON (ospid: 31905): terminating the instance
Sat Oct 22 13:02:03 2022
System state dump requested by (instance=1, osid=31905 (SMON)), summary=[abnormal instance termination].
System State dumped to trace file
/u02/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_diag_31863_20221022130203.trc
Sat Oct 22 13:02:04 2022
ORA-1092 : opitsk aborting process
Instance terminated by SMON, pid = 31905
另外的一个节点错误日志如下:
Sat Oct 22 13:02:41 2022
[24610] Successfully onlined Undo Tablespace 5.
Undo initialization finished serial:0 start:152928776 end:152930636 diff:1860 (18 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Sat Oct 22 13:03:10 2022
Error: Controlfile sequence number in file header is different from the one in memory
Please check that the correct mount options are used if controlfile is located on NFS
USER (ospid: 24610): terminating the instance
Sat Oct 22 13:03:10 2022
System state dump requested by (instance=2, osid=24610), summary=[abnormal instance termination].
System State dumped to trace file
/u02/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_diag_24486_20221022130310.trc
Dumping diagnostic data in directory=[cdmp_20221022130310],
requested by (instance=2, osid=24610), summary=[abnormal instance termination].
Instance terminated by USER, pid = 24610
通过节点的日志基本上可以确定是由于Controlfile sequence number异常导致,官方也有类似的文档描述:
Instance Crashed With “Controlfile sequence number in file header is different from the onein memory” (Doc ID 2884958.1)
ORA-600 kcvent_internal_02故障处理
数据库启动报ORA-00600: internal error code, arguments: [kcvent_internal_02]错,无法正常open
Reconfiguration complete parallel recovery started with 32 processes Started redo scan Completed redo scan read 22775 KB redo, 5055 data blocks need recovery Started redo application at Thread 2: logseq 166395, block 88 Recovery of Online Redo Log: Thread 2 Group 3 Seq 166395 Reading mem 0 Mem# 0: +DATA/orcl/onlinelog/group_3.283.1036687245 Mem# 1: +FLASH/orcl/onlinelog/group_3.264.1036687257 Recovery of Online Redo Log: Thread 2 Group 4 Seq 166396 Reading mem 0 Mem# 0: +DATA/orcl/onlinelog/group_4.284.1036687257 Mem# 1: +FLASH/orcl/onlinelog/group_4.265.1036687257 Completed redo application of 15.97MB Completed instance recovery at Thread 2: logseq 166396, block 15854, scn 27533037896 5055 data blocks read, 5055 data blocks written, 22775 redo k-bytes read Thread 2 advanced to log sequence 166397 (thread recovery) Redo thread 2 internally disabled at seq 166397 Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_35652472.trc (incident=195549): ORA-00600: internal error code, arguments: [kcvent_internal_02], [], [], [], [], [], [], [], [], [], [], [] Incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_195549/orcl1_ora_35652472_i195549.trc
对应的trace文件信息
Dump continued from file: /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_35652472.trc
ORA-00600: internal error code, arguments: [kcvent_internal_02], [], [], [], [], [], [], [], [], [], [], []
========= Dump for incident 195549 (ORA 600 [kcvent_internal_02]) ========
*** 2022-06-06 22:17:48.743
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0)
----- Current SQL Statement for this session (sql_id=5fmpzya54p4hf) -----
ALTER DATABASE OPEN /* db agent *//* {1:38339:2} */
----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
skdstdst()+40 bl 0000000109B1E77C 000000000 ? 000000001 ?
000000003 ? 000000000 ?
000000000 ? 000000001 ?
000000003 ? 000000000 ?
ksedst1()+112 call skdstdst() 16F60DC8B26FAB02 ?
4846284100000000 ?
FFFFFFFFFFE46D0 ?
283C6E7C6A9A6 ? 10A6B923C ?
000000000 ? 110737880 ?
2050033FFFE46D8 ?
ksedst()+40 call ksedst1() 000000000 ? 00000000A ?
07FFFFFFF ? 700000000003670 ?
000000000 ? 000000000 ?
000002004 ? 000000001 ?
dbkedDefDump()+1516 call ksedst() 000000000 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ? 300000003 ?
ksedmp()+72 call dbkedDefDump() 310737880 ? 110000D40 ?
FFFFFFFFFFE4EE0 ? 1106AB740 ?
100124BB8 ? 000000000 ?
700011D7387FF08 ? 1106AB740 ?
ksfdmp()+100 call ksedmp() 000000002 ? 000000000 ?
000000002 ? 10AF01CA8 ?
10A041C38 ? 000000000 ?
11073C760 ? 110737880 ?
dbgexPhaseII()+1904 call ksfdmp() 000000000 ? 00000000A ?
000000002 ? 000000000 ?
000000002 ? 10A041C30 ?
000000000 ? 001050005 ?
dbgexProcessError() call dbgexPhaseII() 110737880 ? 11073A970 ?
+1556 00002FBDD ? 200000000 ?
FFFFFFFFFFE5DF8 ? 00000006C ?
200000000 ? 1000000000 ?
dbgeExecuteForError call dbgexProcessError() 110737880 ? 11073C760 ?
()+72 100000703 ? 000004000 ?
000000000 ? FFFFFFFFFFE9608 ?
000000001 ? 11073E4A8 ?
dbgePostErrorKGE()+ call dbgeExecuteForError FFFFFFFFFFE92B0 ?
2044 () 700011D61558BB8 ? 102878B5C ?
000000000 ? 000000000 ?
FFFFFFFFFFE9608 ? 000000000 ?
000000000 ?
dbkePostKGE_kgsf()+ call dbgePostErrorKGE() 07FFFFFFF ? 700000000003670 ?
68 25800000001 ? 109E4A618 ?
000000000 ? 000000000 ?
FFFFFFFFFFEA0B0 ? 1109C0040 ?
kgeadse()+380 call dbkePostKGE_kgsf() 000000000 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ? 069186EAB ?
kgerinv_internal()+ call kgeadse() 000000002 ? 000000002 ?
48 000000001 ? FFFFFFFFFFEAB58 ?
10A4E02F0 ? 000000002 ?
FFFFFFFFFFE9FE0 ? 000000000 ?
kgerinv()+48 call kgerinv_internal() 200000002 ? 000000002 ?
FFFFFFFFFFEA060 ? 000000000 ?
102860EB0 ? FFFFFFFFFFEA458 ?
10285CE74 ? FFFFFFFFFFEA358 ?
kgeasnmierr()+72 call kgerinv() 38400000001 ? 000000000 ?
10A4E0D20 ? 497F0A29CAE0 ?
000000001 ? FFFFFFFFFFEA1C0 ?
10A4E0D20 ? 110000D78 ?
kcvent_internal()+1 call kgeasnmierr() FFFFFFFFFFEA1C0 ? 200000002 ?
532 1F0410001F041 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ? 000000004 ?
kctenb_internal()+2 call kcvent_internal() FFFFFFFFFFEB378 ? 200000002 ?
772 FFFFFFFFFFEB448 ?
FFFFFFFFFFEB2E8 ?
41F6C57900000000 ?
000000000 ? FFFFFFFFFFEB330 ?
1106AB740 ?
kcfopd()+1508 call kctenb_internal() 07FFFFFFF ? 000000000 ?
000000018 ? FFFFFFFFFFEC380 ?
000000000 ? 110A39050 ?
FFFFFFFFFFEC390 ? 000000000 ?
adbdrv()+8028 call kcfopd() 081F0AD00 ? 00000000F ?
0FFFED4C0 ? 000000000 ?
FFFFFFFFFFED548 ? 100000000 ?
000000000 ? 1000100000000 ?
opiexe()+16048 call adbdrv() 2300000023 ? 100000001 ?
000000000 ? FFFFFFFFFFF6960 ?
000000000 ? FFFFFFFFFFF6B60 ?
FFFFFFFFFFF6A98 ? 200000002 ?
opiosq0()+3984 call opiexe() 700011E117B3B20 ? 000000000 ?
FFFFFFFFFFF7ED8 ? 110000D78 ?
000000001 ? 1109FA438 ?
FFFFFFFFFFF7E70 ?
2216414400000001 ?
kpooprx()+316 call opiosq0() 300000000 ? 000000000 ?
000000000 ? A4000000000000 ?
000000000 ? FFFFFFFFFFF87F0 ?
28104221FFFF86F0 ?
1109FAB08 ?
kpoal8()+872 call kpooprx() 1000CE68C ? 000000001 ?
FFFFFFFFFFFAD14 ? 100000001 ?
000000000 ? A40000000000A4 ?
109EB6D00 ? 000000000 ?
opiodr()+908 call kpoal8() 100000000 ? 9001000A0091108 ?
000000FFF ? 07FFFFFF8 ?
FFFFFFFFFFF8F10 ? 000000018 ?
000000000 ? 000072FFF ?
ttcpip()+1028 call opiodr() 5EFFFFA480 ? 1C00200048 ?
FFFFFFFFFFFA9F8 ? 000530058 ?
1108BEE30 ? 000000028 ?
FFFFFFFFFFFA3A0 ? 1108BEC70 ?
opitsk()+1612 call ttcpip() 110135440 ? 000002078 ?
000000000 ? 110000D78 ?
110005210 ? 000000000 ?
FFFFFFFFFFFAA20 ?
2222208009EF13C0 ?
opiino()+940 call opitsk() 110024C58 ? 000000000 ?
11079B550 ? 1107A0850 ?
110737880 ? FFFFFFFFFFFCAE0 ?
FFFFFFFFFFFEB3C ? 000000101 ?
opiodr()+908 call opiino() 3C006C787C ?
BFF0000000000000 ?
FFFFFFFFFFFEF60 ?
FFFFFFFFFFFD5E9 ?
FFFFFFFFFFFD630 ? 1106AB740 ?
FFFFFFFFFFFD650 ?
9FFFFFFF000E608 ?
opidrv()+1132 call opiodr() 3C0AFBC600 ? 410134340 ?
FFFFFFFFFFFEF60 ? 07530312F ?
108820CE4 ? 1106AB740 ?
7264626D732F6F72 ?
1106AB740 ?
sou2o()+136 call opidrv() 3C0882A9D0 ? 41170031F ?
FFFFFFFFFFFEF60 ?
110017002A0000 ? 0E0DDF00D ?
1106AB740 ?
BADC0FFEE0DDF00D ?
BADC0FFEE0DDF00D ?
opimai_real()+560 call sou2o() FFFFFFFFFFFEFD0 ?
BADC0FFEE0DDF00D ?
90000000008BE3C ?
BADC0FFEE0DDF00D ?
000000002 ? 9001000A0091108 ?
A0000000A000000 ? 10B671248 ?
ssthrdmain()+276 call opimai_real() 10B6B1D74 ? 9001000A0095260 ?
FFFFFFFFFFFF0B0 ? 10B6B1598 ?
FFFFFFFFFFFF0D0 ?
FFFFFFFFFFFF428 ?
900000000100968 ?
9001000A0091108 ?
main()+204 call ssthrdmain() 240000000 ? FFFFFFFFFFFF418 ?
8FFFFFFF0000090 ? 000000000 ?
000000000 ? 000000000 ?
BADC0FFEE0DDF00D ?
BADC0FFEE0DDF00D ?
__start()+112 call main() 000000000 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ? 000000000 ?
--------------------- Binary Stack Dump ---------------------
该错误在mos,互联网上没有任何信息,不过在alert日志中发现类似信息
Mon Jun 06 23:03:58 2022
Error: Controlfile sequence number in file header is different from the one in memory
Please check that the correct mount options are used if controlfile is located on NFS
初步判断可能和这个错误有关系,解决相关问题后,尝试open库
SQL> recover database;
ORA-00279: change 27533037896 generated at 06/06/2022 22:17:46 needed for
thread 2
ORA-00289: suggestion :
+FLASH/orcl/archivelog/2022_06_06/thread_2_seq_166396.6532.1106691471
ORA-00280: change 27533037896 for thread 2 is in sequence #166396
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
Log applied.
Media recovery complete.
SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01216: thread 2 is expected to be disabled after CREATE CONTROLFILE
SQL> !oerr ora 01216
01216, 00000, "thread %s is expected to be disabled after CREATE CONTROLFILE"
// *Cause: A thread that was given during CREATE CONTROLFILE is enabled, but
// the datafiles indicate that it should be disabled. This is
// probably because the logs supplied to the CREATE CONTROLFILE
// are old (from before the disabling of the thread).
// *Action: This thread is not required to run the database. The CREATE
// CONTROLFILE statement can be reissued without the problem thread,
// and, if desired, the thread can be recreated after the database
// is open.
ORA-01216这个错误比较也比较少见,但是感觉和thread有关系,大概的意思是thread 被disable了
SQL> select thread#,STATUS FROM V$THREAD;
THREAD# STATUS
---------- ------------------
1 CLOSED
2 CLOSED
通过人工强制把thread个open,然后数据库启动成功
SQL> select thread#,status from v$thread;
THREAD# STATUS
---------- ------------------
1 OPEN
2 CLOSED
SQL> alter database open;
Database altered.
然后启动thread 2,open 第二个节点
--需要open节点 QL> startup ORACLE instance started. Total System Global Area 1.2961E+11 bytes Fixed Size 2262400 bytes Variable Size 3.3018E+10 bytes Database Buffers 9.6368E+10 bytes Redo Buffers 221818880 bytes ORA-01618: redo thread 2 is not enabled - cannot mount --已经open节点 SQL> ALTER DATABASE ENABLE THREAD 2; Database altered. --需要open节点 SQL> ALTER DATABASE MOUNT; Database altered. SQL> ALTER DATABASE OPEN; Database altered.
xifenfei1:/home/grid$crsctl status res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE xifenfei1
ONLINE ONLINE xifenfei2
ora.FLASH.dg
ONLINE ONLINE xifenfei1
ONLINE ONLINE xifenfei2
ora.LISTENER.lsnr
ONLINE ONLINE xifenfei1
ONLINE ONLINE xifenfei2
ora.OCR.dg
ONLINE ONLINE xifenfei1
ONLINE ONLINE xifenfei2
ora.asm
ONLINE ONLINE xifenfei1 Started
ONLINE ONLINE xifenfei2 Started
ora.gsd
OFFLINE OFFLINE xifenfei1
OFFLINE OFFLINE xifenfei2
ora.net1.network
ONLINE ONLINE xifenfei1
ONLINE ONLINE xifenfei2
ora.ons
ONLINE ONLINE xifenfei1
ONLINE ONLINE xifenfei2
ora.registry.acfs
ONLINE ONLINE xifenfei1
ONLINE ONLINE xifenfei2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE xifenfei1
ora.cvu
1 OFFLINE OFFLINE
ora.xifenfei1.vip
1 ONLINE ONLINE xifenfei1
ora.xifenfei2.vip
1 ONLINE ONLINE xifenfei2
ora.oc4j
1 ONLINE ONLINE xifenfei2
ora.orcl.db
1 ONLINE ONLINE xifenfei1 Open
2 ONLINE ONLINE xifenfei2 Open
ora.scan1.vip
1 ONLINE ONLINE xifenfei1