联系：手机/微信(+86 17813235971) QQ(107644445)
标题：ORA-15196: invalid ASM block header [kfc.c:26368]故障恢复
作者：惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]
有客户对asm的data磁盘组增加磁盘进行扩容,在做reblance的过程中重启了主机,结果导致data磁盘组mount之后自动dismount
Fri Oct 09 20:48:06 2020 NOTE: PST enabling heartbeating (grp 1) Fri Oct 09 20:48:06 2020 NOTE: ASM did background COD recovery for group 1/0x739536c (DATA) NOTE: starting rebalance of group 1/0x739536c (DATA) at power 10 Starting background process ARB0 Fri Oct 09 20:48:07 2020 ARB0 started with pid=28, OS id=39278 NOTE: assigning ARB0 to group 1/0x739536c (DATA) with 10 parallel I/Os cellip.ora not found. WARNING:cache read a corrupt block:group=1(DATA) dsk=8 blk=7 disk=8(DATA_0008)incarn=3916014506 au=0 blk=7 count=1 Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_39278.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] NOTE: a corrupted block from group DATA was dumped to /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_39278.trc WARNING:cache read(retry)a corrupt block:group=1(DATA) dsk=8 blk=7 disk=8(DATA_0008)incarn=3916014506 au=0 blk=7 count=1 Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_39278.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] ERROR: cache failed to read group=1(DATA) dsk=8 blk=7 from disk(s): 8(DATA_0008) Fri Oct 09 20:48:13 2020 NOTE: GroupBlock outside rolling migration privileged region ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] NOTE: requesting all-instance membership refresh for group=1 NOTE: cache initiating offline of disk 8 group DATA NOTE: process _arb0_+asm1 (39278) initiating offline of disk 8.3916014506 (DATA_0008) with mask 0x7e in group 1 NOTE: initiating PST update: grp = 1, dsk = 8/0xe969a3aa, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 7 for pid 28, osid 39278 ERROR: Disk 8 cannot be offlined, since diskgroup has external redundancy. ERROR: too many offline disks in PST (grp 1) Fri Oct 09 20:48:13 2020 NOTE: cache dismounting (not clean) group 1/0x0739536C (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 39346, image: oracle@rac1 (B000) Fri Oct 09 20:48:13 2020 NOTE: halting all I/Os to diskgroup 1 (DATA) Fri Oct 09 20:48:13 2020 NOTE: LGWR doing non-clean dismount of group 1 (DATA) NOTE: LGWR sync ABA=32.4749 last written ABA 32.4749 WARNING: Offline for disk DATA_0008 in mode 0x7f failed. Fri Oct 09 20:48:13 2020 kjbdomdet send to inst 2 detach from dom 1, sending detach message to inst 2 Fri Oct 09 20:48:13 2020 List of instances: 1 2 Dirty detach reconfiguration started (new ddet inc 2, cluster inc 4) Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_39278.trc (incident=337185): ORA-15335: ASM metadata corruption detected in disk group 'DATA' ORA-15130: diskgroup "DATA" is being dismounted ORA-15066: offlining disk "DATA_0008" in group "DATA" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] ORA-15196: invalid ASM block header [kfc.c:26368] [check_kfbh] [2147483656] [7] [2182009786 != 2190395015] Incident details in: /u01/grid/diag/asm/+asm/+ASM1/incident/incdir_337185/+ASM1_arb0_39278_i337185.trc Global Resource Directory partially frozen for dirty detach * dirty detach - domain 1 invalid = TRUE 2341 GCS resources traversed, 0 cancelled Dirty Detach Reconfiguration complete freeing rdom 1 Fri Oct 09 20:48:13 2020 WARNING: dirty detached from domain 1 NOTE: cache dismounted group 1/0x0739536C (DATA)
错误信息比较明显dsk=8 blk=7 au=0 blk=7 的check值不对,本来应该是2190395015现在变为了2182009786,通过kfed分析确实如此
C:\Users\Administrator>kfed read f:/temp/xff/2.dd blkn=7|more kfbh.endian: 1 ; 0x000: 0x01 kfbh.hard: 130 ; 0x001: 0x82 kfbh.type: 3 ; 0x002: KFBTYP_ALLOCTBL kfbh.datfmt: 2 ; 0x003: 0x02 kfbh.block.blk: 7 ; 0x004: blk=7 kfbh.block.obj: 2147483656 ; 0x008: disk=8 kfbh.check: 2182009786 ; 0x00c: 0x820ed3ba kfbh.fcn.base: 2711248 ; 0x010: 0x00295ed0 kfbh.fcn.wrap: 0 ; 0x014: 0x00000000 kfbh.spare1: 0 ; 0x018: 0x00000000 kfbh.spare2: 0 ; 0x01c: 0x00000000 kfdatb.aunum: 2240 ; 0x000: 0x000008c0 kfdatb.shrink: 448 ; 0x004: 0x01c0 kfdatb.ub2pad: 0 ; 0x006: 0x0000 kfdatb.auinfo[0].link.next: 8 ; 0x008: 0x0008 kfdatb.auinfo[0].link.prev: 8 ; 0x00a: 0x0008 kfdatb.auinfo[1].link.next: 12 ; 0x00c: 0x000c kfdatb.auinfo[1].link.prev: 12 ; 0x00e: 0x000c kfdatb.auinfo[2].link.next: 16 ; 0x010: 0x0010 kfdatb.auinfo[2].link.prev: 16 ; 0x012: 0x0010 kfdatb.auinfo[3].link.next: 20 ; 0x014: 0x0014 kfdatb.auinfo[3].link.prev: 20 ; 0x016: 0x0014 kfdatb.auinfo[4].link.next: 24 ; 0x018: 0x0018 kfdatb.auinfo[4].link.prev: 24 ; 0x01a: 0x0018 kfdatb.auinfo[5].link.next: 28 ; 0x01c: 0x001c kfdatb.auinfo[5].link.prev: 28 ; 0x01e: 0x001c kfdatb.auinfo[6].link.next: 32 ; 0x020: 0x0020 kfdatb.auinfo[6].link.prev: 32 ; 0x022: 0x0020 kfdatb.spare: 0 ; 0x024: 0x00000000
修改该值之后,再次mount data磁盘组,报错如下
Sat Oct 10 13:49:22 2020 ARB0 started with pid=28, OS id=10329 NOTE: assigning ARB0 to group 1/0x3759521c (DATA) with 10 parallel I/Os cellip.ora not found. Sat Oct 10 13:49:26 2020 NOTE: GroupBlock outside rolling migration privileged region NOTE: requesting all-instance membership refresh for group=1 WARNING: cache read a corrupt block: group=1(DATA) dsk=8 blk=8 disk=8 (DATA_0008) incarn=3916014011 au=0 blk=8 count=1 Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_10329.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] NOTE: a corrupted block from group DATA was dumped to /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_10329.trc WARNING:cache read(retry)a corrupt block: group=1(DATA)dsk=8 blk=8 disk=8(DATA_0008)incarn=3916014011 au=0 blk=8 count=1 Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_10329.trc: ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] ERROR: cache failed to read group=1(DATA) dsk=8 blk=8 from disk(s): 8(DATA_0008) ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] NOTE: cache initiating offline of disk 8 group DATA NOTE: process _arb0_+asm1 (10329) initiating offline of disk 8.3916014011 (DATA_0008) with mask 0x7e in group 1 NOTE: initiating PST update: grp = 1, dsk = 8/0xe969a1bb, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 64 for pid 28, osid 10329 ERROR: Disk 8 cannot be offlined, since diskgroup has external redundancy. ERROR: too many offline disks in PST (grp 1) Sat Oct 10 13:49:28 2020 NOTE: cache dismounting (not clean) group 1/0x3759521C (DATA) WARNING: Offline for disk DATA_0008 in mode 0x7f failed. Sat Oct 10 13:49:28 2020 NOTE: halting all I/Os to diskgroup 1 (DATA) NOTE: messaging CKPT to quiesce pins Unix process pid: 10346, image: oracle@rac1 (B000) Errors in file /u01/grid/diag/asm/+asm/+ASM1/trace/+ASM1_arb0_10329.trc (incident=363107): ORA-15335: ASM metadata corruption detected in disk group 'DATA' ORA-15130: diskgroup "DATA" is being dismounted ORA-15066: offlining disk "DATA_0008" in group "DATA" may result in a data loss ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] ORA-15196: invalid ASM block header [kfc.c:26368] [endian_kfbh] [2147483656] [8] [0 != 1] Incident details in: /u01/grid/diag/asm/+asm/+ASM1/incident/incdir_363107/+ASM1_arb0_10329_i363107.trc
该报错为:dsk=8 blk=7 au=0 blk=8异常,通过kfed查看发现
C:\Users\Administrator>kfed read f:/temp/xff/2.dd blkn=8
kfbh.endian: 0 ; 0x000: 0x00
kfbh.hard: 0 ; 0x001: 0x00
kfbh.type: 0 ; 0x002: KFBTYP_INVALID
kfbh.datfmt: 0 ; 0x003: 0x00
kfbh.block.blk: 0 ; 0x004: blk=0
kfbh.block.obj: 0 ; 0x008: file=0
kfbh.check: 0 ; 0x00c: 0x00000000
kfbh.fcn.base: 0 ; 0x010: 0x00000000
kfbh.fcn.wrap: 0 ; 0x014: 0x00000000
kfbh.spare1: 0 ; 0x018: 0x00000000
kfbh.spare2: 0 ; 0x01c: 0x00000000
006BE8C00 00000000 00000000 00000000 00000000 [................]
Repeat 31 times
006BE8E00 012C0000 04AFFC07 003BFFCD 03F15BD0 [..,.......;..[..]
006BE8E10 012BFC30 00000000 00000002 00000002 [0.+.............]
006BE8E20 00008000 00008000 00002000 564F22AF [......... ..."OV]
006BE8E30 5F805293 FFFF0002 0001EF53 00000001 [.R._....S.......]
006BE8E40 545AE384 00000000 00000000 00000001 [..ZT............]
006BE8E50 00000000 0000000B 00000100 0000003C [............<...]
006BE8E60 00000242 0000007B 52438BA0 C44FFA90 [B...{.....CR..O.]
006BE8E70 33B6F381 919E2DBA 00000000 00000000 [...3.-..........]
006BE8E80 00000000 00000000 6361622F 0070756B [......../backup.]
006BE8E90 00000000 00000000 00000000 00000000 [................]
Repeat 2 times
006BE8EC0 00000000 00000000 00000000 03ED0000 [................]
006BE8ED0 00000000 00000000 00000000 00000000 [................]
006BE8EE0 00000008 00000000 00000000 AC3C87D6 [..............<.]
006BE8EF0 F1401174 F4F036BD 274FB92F 00000101 [t.@..6../.O'....]
006BE8F00 0000000C 00000000 545AE384 0002F30A [..........ZT....]
006BE8F10 00000004 00000000 00000000 00007FFF [................]
006BE8F20 02508000 00007FFF 00000001 0250FFFF [..P...........P.]
006BE8F30 00000000 00000000 00000000 00000000 [................]
006BE8F40 00000000 00000000 00000000 08000000 [................]
006BE8F50 00000000 00000000 00000000 001C001C [................]
006BE8F60 00000001 00000000 00000000 00000000 [................]
006BE8F70 00000000 00000004 A9AF72B9 0000003B [.........r..;...]
006BE8F80 00000000 00000000 00000000 00000000 [................]
Repeat 167 times
006BE9A00 00001CC4 00800101 00001CC9 00800101 [................]
006BE9A10 00001CCD 00800101 00001CD2 00800101 [................]
006BE9A20 00001CD7 00800101 00001CDE 00800101 [................]
006BE9A30 00001CE3 00800101 00001CE8 00800101 [................]
006BE9A40 00001CEC 00800101 00000000 00000000 [................]
006BE9A50 00000000 00000000 00000000 00000000 [................]
Repeat 26 times
KFED-00322:Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
该block完全损坏,基本上无直接修复的可能,通过对data 磁盘组进行patch操作,让其mount之后不再dismount
NOTE: GMON heartbeating for grp 1 GMON querying group 1 at 76 for pid 27, osid 14466 NOTE: cache opening disk 0 of grp 1: DATA_0000 path:/dev/emcpowere NOTE: F1X0 found on disk 0 au 2 fcn 0.2708382 NOTE: cache opening disk 1 of grp 1: DATA_0001 path:/dev/emcpowerf NOTE: cache opening disk 2 of grp 1: DATA_0002 path:/dev/emcpowerg NOTE: cache opening disk 3 of grp 1: DATA_0003 path:/dev/emcpowerh NOTE: cache opening disk 4 of grp 1: DATA_0004 path:/dev/emcpoweri NOTE: cache opening disk 5 of grp 1: DATA_0005 path:/dev/emcpowerj NOTE: cache opening disk 6 of grp 1: DATA_0006 path:/dev/emcpowerk NOTE: cache opening disk 7 of grp 1: DATA_0007 path:/dev/emcpowerl NOTE: cache opening disk 8 of grp 1: DATA_0008 path:/dev/emcpowerc NOTE: cache mounting (first) external redundancy group 1/0x47495222 (DATA) Sat Oct 10 13:59:38 2020 * allocate domain 1, invalid = TRUE Sat Oct 10 13:59:38 2020 NOTE: attached to recovery domain 1 NOTE: starting recovery of thread=1 ckpt=53.6778 group=1 (DATA) NOTE: advancing ckpt for group 1 (DATA) thread=1 ckpt=53.6778 NOTE: cache recovered group 1 to fcn 0.2961429 NOTE: redo buffer size is 256 blocks (1053184 bytes) Sat Oct 10 13:59:38 2020 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (DATA) NOTE: LGWR found thread 1 closed at ABA 53.6777 NOTE: LGWR mounted thread 1 for diskgroup 1 (DATA) NOTE: LGWR opening thread 1 at fcn 0.2961429 ABA 54.6778 NOTE: cache mounting group 1/0x47495222 (DATA) succeeded NOTE: cache ending mount (success) of group DATA number=1 incarn=0x47495222 NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1 SUCCESS: diskgroup DATA was mounted SUCCESS: alter diskgroup data mount
然后通过rman备份数据库,删除老磁盘组,创建新磁盘组,恢复数据,实现数据库完美恢复,数据0丢失.
加我QQ
