标签云
asm恢复 bbed bootstrap$ dul In Memory kcbzib_kcrsds_1 kccpb_sanity_check_2 kfed MySQL恢复 ORA-00312 ORA-00607 ORA-00704 ORA-01110 ORA-01555 ORA-01578 ORA-08103 ORA-600 2131 ORA-600 2662 ORA-600 2663 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-600 kcbzib_kcrsds_1 ORA-600 KCLCHKBLK_4 ORA-15042 ORA-15196 ORACLE 12C oracle dul ORACLE PATCH Oracle Recovery Tools oracle加密恢复 oracle勒索 oracle勒索恢复 oracle异常恢复 Oracle 恢复 ORACLE恢复 ORACLE数据库恢复 oracle 比特币 OSD-04016 YOUR FILES ARE ENCRYPTED 勒索恢复 比特币加密文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (102)
- 数据库 (1,655)
- DB2 (22)
- MySQL (71)
- Oracle (1,519)
- Data Guard (51)
- EXADATA (8)
- GoldenGate (21)
- ORA-xxxxx (158)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (14)
- ORACLE 21C (3)
- Oracle 23ai (7)
- Oracle ASM (65)
- Oracle Bug (8)
- Oracle RAC (52)
- Oracle 安全 (6)
- Oracle 开发 (28)
- Oracle 监听 (28)
- Oracle备份恢复 (553)
- Oracle安装升级 (90)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (76)
- PostgreSQL (18)
- PostgreSQL恢复 (6)
- SQL Server (27)
- SQL Server恢复 (8)
- TimesTen (7)
- 达梦数据库 (2)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (37)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (20)
-
最近发表
- RAC默认服务配置优先节点
- Oracle 19c RAC 替换私网操作
- 监听报TNS-12541 TNS-12560 TNS-00511错误
- drop tablespace xxx including contents恢复
- Linux 8 修改网卡名称
- 如何修改集群的公网信息(包括 VIP) (Doc ID 1674442.1)
- 如何在 oracle 集群环境下修改私网信息 (Doc ID 2103317.1)
- ORA-600 [kcvfdb_pdb_set_clean_scn: cleanckpt] 相关bug
- ORA-600 krhpfh_03-1210故障处理
- 19c库启动报ORA-600 kcbzib_kcrsds_1
- DBMS_SESSION.set_context提示ORA-01031问题解决
- redo写丢失导致ORA-600 kcrf_resilver_log_1故障
- 硬件故障导致ORA-01242 ORA-01122等错误
- 200T 数据库非归档无备份恢复
- 利用flashback快速恢复failover 的备库
- [comingback2022@cock.li].eking和[tsai.shen@mailfence.com].faust扩展名勒索病毒数据库可以完美恢复
- opatch auto 出现unable to get oracle owner for 错误
- Oracle 23ai 表和视图的列最多支持到4096个
- 断电引起redo和数据文件不一致故障恢复
- ORA-03113: 通信通道的文件结尾
标签归档:exadata坏盘恢复
Exadata磁盘损坏导致磁盘组无法mount恢复(oracle一体机磁盘组异常恢复)
Oracle Exadata客户,在换盘过程中,cell节点又一块磁盘损坏,导致datac1磁盘组(该磁盘组是normal方式冗余)无法mount
Thu Jul 20 22:01:21 2023 SQL> alter diskgroup datac1 mount force NOTE: cache registered group DATAC1 number=1 incarn=0x0728ad12 NOTE: cache began mount (first) of group DATAC1 number=1 incarn=0x0728ad12 NOTE: Assigning number (1,35) to disk (o/192.168.10.9;192.168.10.10/DATAC1_CD_11_dm01celadm03) NOTE: Assigning number (1,31) to disk (o/192.168.10.9;192.168.10.10/DATAC1_CD_07_dm01celadm03) NOTE: Assigning number (1,24) to disk (o/192.168.10.9;192.168.10.10/DATAC1_CD_00_dm01celadm03) NOTE: Assigning number (1,25) to disk (o/192.168.10.9;192.168.10.10/DATAC1_CD_01_dm01celadm03) NOTE: Assigning number (1,27) to disk (o/192.168.10.9;192.168.10.10/DATAC1_CD_03_dm01celadm03) NOTE: Assigning number (1,33) to disk (o/192.168.10.9;192.168.10.10/DATAC1_CD_09_dm01celadm03) NOTE: Assigning number (1,30) to disk (o/192.168.10.9;192.168.10.10/DATAC1_CD_06_dm01celadm03) NOTE: Assigning number (1,28) to disk (o/192.168.10.9;192.168.10.10/DATAC1_CD_04_dm01celadm03) NOTE: Assigning number (1,26) to disk (o/192.168.10.9;192.168.10.10/DATAC1_CD_02_dm01celadm03) NOTE: Assigning number (1,1) to disk (o/192.168.10.9;192.168.10.10/DATAC1_CD_08_dm01celadm03) NOTE: Assigning number (1,34) to disk (o/192.168.10.9;192.168.10.10/DATAC1_CD_10_dm01celadm03) NOTE: Assigning number (1,29) to disk (o/192.168.10.9;192.168.10.10/DATAC1_CD_05_dm01celadm03) NOTE: Assigning number (1,3) to disk (o/192.168.10.7;192.168.10.8/DATAC1_CD_07_dm01celadm02) NOTE: Assigning number (1,4) to disk (o/192.168.10.7;192.168.10.8/DATAC1_CD_06_dm01celadm02) NOTE: Assigning number (1,5) to disk (o/192.168.10.7;192.168.10.8/DATAC1_CD_00_dm01celadm02) NOTE: Assigning number (1,6) to disk (o/192.168.10.7;192.168.10.8/DATAC1_CD_10_dm01celadm02) NOTE: Assigning number (1,7) to disk (o/192.168.10.7;192.168.10.8/DATAC1_CD_08_dm01celadm02) NOTE: Assigning number (1,8) to disk (o/192.168.10.7;192.168.10.8/DATAC1_CD_03_dm01celadm02) NOTE: Assigning number (1,9) to disk (o/192.168.10.7;192.168.10.8/DATAC1_CD_11_dm01celadm02) NOTE: Assigning number (1,10) to disk (o/192.168.10.7;192.168.10.8/DATAC1_CD_01_dm01celadm02) NOTE: Assigning number (1,11) to disk (o/192.168.10.7;192.168.10.8/DATAC1_CD_04_dm01celadm02) NOTE: Assigning number (1,21) to disk (o/192.168.10.7;192.168.10.8/DATAC1_CD_05_dm01celadm02) NOTE: Assigning number (1,43) to disk (o/192.168.10.7;192.168.10.8/DATAC1_CD_02_dm01celadm02) NOTE: Assigning number (1,36) to disk (o/192.168.10.5;192.168.10.6/DATAC1_CD_07_dm01celadm01) NOTE: Assigning number (1,37) to disk (o/192.168.10.5;192.168.10.6/DATAC1_CD_09_dm01celadm01) NOTE: Assigning number (1,38) to disk (o/192.168.10.5;192.168.10.6/DATAC1_CD_11_dm01celadm01) NOTE: Assigning number (1,0) to disk (o/192.168.10.5;192.168.10.6/DATAC1_CD_08_dm01celadm01) NOTE: Assigning number (1,40) to disk (o/192.168.10.5;192.168.10.6/DATAC1_CD_00_dm01celadm01) NOTE: Assigning number (1,41) to disk (o/192.168.10.5;192.168.10.6/DATAC1_CD_03_dm01celadm01) NOTE: Assigning number (1,42) to disk (o/192.168.10.5;192.168.10.6/DATAC1_CD_06_dm01celadm01) NOTE: Assigning number (1,44) to disk (o/192.168.10.5;192.168.10.6/DATAC1_CD_05_dm01celadm01) NOTE: Assigning number (1,45) to disk (o/192.168.10.5;192.168.10.6/DATAC1_CD_01_dm01celadm01) NOTE: Assigning number (1,46) to disk (o/192.168.10.5;192.168.10.6/DATAC1_CD_02_dm01celadm01) NOTE: Assigning number (1,47) to disk (o/192.168.10.5;192.168.10.6/DATAC1_CD_10_dm01celadm01) NOTE: Assigning number (1,2) to disk (o/192.168.10.5;192.168.10.6/DATAC1_CD_04_dm01celadm01) Thu Jul 20 22:01:28 2023 NOTE: GMON heartbeating for grp 1 GMON querying group 1 at 450 for pid 30, osid 171838 NOTE: Assigning number (1,32) to disk () NOTE: Assigning number (1,39) to disk () GMON querying group 1 at 451 for pid 30, osid 171838 NOTE: cache closing disk 32 of grp 1: (not open) NOTE: process _user171838_+asm1 (171838) initiating offline of disk 39.3915945266 () with mask 0x7e[0x7f] in group 1 NOTE: initiating PST update: grp = 1, dsk = 39/0xe9689532, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 452 for pid 30, osid 171838 NOTE: cache closing disk 32 of grp 1: (not open) ERROR: Disk 39 cannot be offlined, since all the disks [39, 32] with mirrored data would be offline. ERROR: too many offline disks in PST (grp 1) WARNING: Offline for disk in mode 0x7f failed. NOTE: cache dismounting (not clean) group 1/0x0728AD12 (DATAC1) NOTE: messaging CKPT to quiesce pins Unix process pid: 171838, image: oracle@dm01dbadm01.gyzq.cn (TNS V1-V3) NOTE: dbwr not being msg'd to dismount NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 1/0x0728AD12 (DATAC1) NOTE: cache ending mount (fail) of group DATAC1 number=1 incarn=0x0728ad12 NOTE: cache deleting context for group DATAC1 1/0x0728ad12 NOTE: cache closing disk 32 of grp 1: (not open) GMON dismounting group 1 at 453 for pid 30, osid 171838 NOTE: Disk DATAC1_CD_08_DM01CELADM01 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_08_DM01CELADM03 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_04_DM01CELADM01 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_07_DM01CELADM02 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_06_DM01CELADM02 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_00_DM01CELADM02 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_10_DM01CELADM02 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_08_DM01CELADM02 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_03_DM01CELADM02 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_11_DM01CELADM02 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_01_DM01CELADM02 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_04_DM01CELADM02 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_05_DM01CELADM02 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_00_DM01CELADM03 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_01_DM01CELADM03 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_02_DM01CELADM03 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_03_DM01CELADM03 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_04_DM01CELADM03 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_05_DM01CELADM03 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_06_DM01CELADM03 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_07_DM01CELADM03 in mode 0x7f marked for de-assignment NOTE: Disk in mode 0x1 marked for de-assignment NOTE: Disk DATAC1_CD_09_DM01CELADM03 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_10_DM01CELADM03 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_11_DM01CELADM03 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_07_DM01CELADM01 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_09_DM01CELADM01 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_11_DM01CELADM01 in mode 0x7f marked for de-assignment NOTE: Disk in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_00_DM01CELADM01 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_03_DM01CELADM01 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_06_DM01CELADM01 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_02_DM01CELADM02 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_05_DM01CELADM01 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_01_DM01CELADM01 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_02_DM01CELADM01 in mode 0x7f marked for de-assignment NOTE: Disk DATAC1_CD_10_DM01CELADM01 in mode 0x7f marked for de-assignment ERROR: diskgroup DATAC1 was not mounted ORA-15032: not all alterations performed ORA-15040: diskgroup is incomplete ORA-15066: offlining disk "39" in group "DATAC1" may result in a data loss ORA-15042: ASM disk "39" is missing from group number "1" ORA-15042: ASM disk "32" is missing from group number "1" ERROR: alter diskgroup datac1 mount force
故障原因是由于asm disk 32还已经损坏在换盘过程中(数据没有reblance完成),又损坏了asm disk 39,而这两份磁盘中有数据互为镜像,因此磁盘组无法正常mount起来.
检查cell节点celldisk和griddisk情况,确认底层磁盘损坏
对于这种情况,因为normal冗余的两份数据都有部分丢失,无法直接恢复数据,通过底层磁盘级别恢复(参考以前一次的Oracle exadata故障恢复:Oracle Exadata坏盘导致磁盘组无法mount恢复),然后比较顺利恢复数据,实现业务数据0丢失
SQL> alter datac1 mount; Diskgroup altered. SQL> alter diskgroup datac1 check all; Diskgroup altered.
在实际恢复过程中由于客户进行了各种尝试,直接新镜像盘然后插入新盘,强制拉磁盘组drop异常disk操作等,导致第一现场发生一些破坏,增加了恢复难道,但是最终通过各种方法弥补,实现了预期的恢复效果(业务数据0丢失)
发表在 Oracle备份恢复
标签为 exadata mount, exadata坏盘恢复, exadata恢复, exadata磁盘组恢复, ORA-15040, ORA-15042, ORA-15066, xd坏盘恢复, xd恢复, 一体机数据恢复
评论关闭
Oracle Exadata坏盘导致磁盘组无法mount恢复
接到朋友求救有客户oracle exadata一体机 的 asm磁盘组无法mount,希望我们提供恢复支持服务
经过分析和了解,大致问题是:磁盘空间已经超容量使用(部分数据不能完成ASM镜像),最近又损坏一块盘,导致asm 磁盘组无法mount。我们分析后,通过重构exadata celldisk数据,将asm 磁盘组 mount成功后,实现五套数据库全部open成功(由于底层磁盘部分数据损坏,导致部分数据访问报错,需要在oracle层面进行处理)。
本次问题的具体分析和处理如下:
存放数据库文件的磁盘组不能mount
Wed Dec 12 21:29:04 2018 SQL> alter diskgroup DATA_XFF mount force NOTE: cache registered group DATA_XFF number=1 incarn=0x5fe882cb NOTE: cache began mount (first) of group DATA_XFF number=1 incarn=0x5fe882cb NOTE: Assigning number (1,36) to disk (o/192.168.10.5/DATA_XFF_CD_11_XFFCEL03) NOTE: Assigning number (1,34) to disk (o/192.168.10.5/DATA_XFF_CD_10_XFFCEL03) NOTE: Assigning number (1,37) to disk (o/192.168.10.5/DATA_XFF_CD_04_XFFCEL03) NOTE: Assigning number (1,38) to disk (o/192.168.10.5/DATA_XFF_CD_00_XFFCEL03) NOTE: Assigning number (1,39) to disk (o/192.168.10.5/DATA_XFF_CD_03_XFFCEL03) NOTE: Assigning number (1,40) to disk (o/192.168.10.5/DATA_XFF_CD_05_XFFCEL03) NOTE: Assigning number (1,41) to disk (o/192.168.10.5/DATA_XFF_CD_08_XFFCEL03) NOTE: Assigning number (1,42) to disk (o/192.168.10.5/DATA_XFF_CD_01_XFFCEL03) NOTE: Assigning number (1,43) to disk (o/192.168.10.5/DATA_XFF_CD_09_XFFCEL03) NOTE: Assigning number (1,44) to disk (o/192.168.10.5/DATA_XFF_CD_06_XFFCEL03) NOTE: Assigning number (1,45) to disk (o/192.168.10.5/DATA_XFF_CD_07_XFFCEL03) NOTE: Assigning number (1,46) to disk (o/192.168.10.5/DATA_XFF_CD_02_XFFCEL03) NOTE: Assigning number (1,22) to disk (o/192.168.10.4/DATA_XFF_CD_10_XFFCEL02) NOTE: Assigning number (1,18) to disk (o/192.168.10.4/DATA_XFF_CD_06_XFFCEL02) NOTE: Assigning number (1,19) to disk (o/192.168.10.4/DATA_XFF_CD_07_XFFCEL02) NOTE: Assigning number (1,15) to disk (o/192.168.10.4/DATA_XFF_CD_03_XFFCEL02) NOTE: Assigning number (1,20) to disk (o/192.168.10.4/DATA_XFF_CD_08_XFFCEL02) NOTE: Assigning number (1,17) to disk (o/192.168.10.4/DATA_XFF_CD_05_XFFCEL02) NOTE: Assigning number (1,16) to disk (o/192.168.10.4/DATA_XFF_CD_04_XFFCEL02) NOTE: Assigning number (1,23) to disk (o/192.168.10.4/DATA_XFF_CD_11_XFFCEL02) NOTE: Assigning number (1,12) to disk (o/192.168.10.4/DATA_XFF_CD_00_XFFCEL02) NOTE: Assigning number (1,21) to disk (o/192.168.10.4/DATA_XFF_CD_09_XFFCEL02) NOTE: Assigning number (1,13) to disk (o/192.168.10.4/DATA_XFF_CD_01_XFFCEL02) NOTE: Assigning number (1,14) to disk (o/192.168.10.4/DATA_XFF_CD_02_XFFCEL02) NOTE: Assigning number (1,1) to disk (o/192.168.10.3/DATA_XFF_CD_05_XFFCEL01) NOTE: Assigning number (1,2) to disk (o/192.168.10.3/DATA_XFF_CD_03_XFFCEL01) NOTE: Assigning number (1,3) to disk (o/192.168.10.3/DATA_XFF_CD_06_XFFCEL01) NOTE: Assigning number (1,4) to disk (o/192.168.10.3/DATA_XFF_CD_09_XFFCEL01) NOTE: Assigning number (1,5) to disk (o/192.168.10.3/DATA_XFF_CD_04_XFFCEL01) NOTE: Assigning number (1,6) to disk (o/192.168.10.3/DATA_XFF_CD_07_XFFCEL01) NOTE: Assigning number (1,7) to disk (o/192.168.10.3/DATA_XFF_CD_11_XFFCEL01) NOTE: Assigning number (1,8) to disk (o/192.168.10.3/DATA_XFF_CD_01_XFFCEL01) NOTE: Assigning number (1,9) to disk (o/192.168.10.3/DATA_XFF_CD_00_XFFCEL01) NOTE: Assigning number (1,10) to disk (o/192.168.10.3/DATA_XFF_CD_10_XFFCEL01) NOTE: Assigning number (1,11) to disk (o/192.168.10.3/DATA_XFF_CD_08_XFFCEL01) Wed Dec 12 21:29:10 2018 NOTE: GMON heartbeating for grp 1 GMON querying group 1 at 101 for pid 27, osid 62541 NOTE: Assigning number (1,0) to disk () GMON querying group 1 at 102 for pid 27, osid 62541 NOTE: process _user62541_+asm2 (62541) initiating offline of disk 0.3915937355 () with mask 0x7e[0x7f] in group 1 NOTE: initiating PST update: grp = 1, dsk = 0/0xe968764b, mask = 0x6a, op = clear GMON updating disk modes for group 1 at 103 for pid 27, osid 62541 ERROR: Disk 0 cannot be offlined, since all the disks [0, 25] with mirrored data would be offline. ERROR: too many offline disks in PST (grp 1) WARNING: Offline of disk 0 () in group 1 and mode 0x7f failed on ASM inst 2 NOTE: cache dismounting (not clean) group 1/0x5FE882CB (DATA_XFF) NOTE: dbwr not being msg'd to dismount NOTE: lgwr not being msg'd to dismount NOTE: cache dismounted group 1/0x5FE882CB (DATA_XFF) NOTE: cache ending mount (fail) of group DATA_XFF number=1 incarn=0x5fe882cb NOTE: cache deleting context for group DATA_XFF 1/0x5fe882cb GMON dismounting group 1 at 104 for pid 27, osid 62541 ERROR: diskgroup DATA_XFF was not mounted ORA-15032: not all alterations performed ORA-15040: diskgroup is incomplete ORA-15066: offlining disk "0" in group "DATA_XFF" may result in a data loss ORA-15042: ASM disk "0" is missing from group number "1" ERROR: alter diskgroup DATA_XFF mount force
检查底层损坏情况
CellCLI> list physicaldisk 20:0 KN3VZL normal 20:1 KNAWLL normal 20:2 KN4E4L warning - predictive failure, poor performance 20:3 KNAN5L normal 20:4 KMJKYL normal 20:5 KN5DGL normal 20:6 KMDLWL normal 20:7 KMDKPL normal 20:8 KMDA7L normal 20:9 KN1YJL normal 20:10 KMH1YL normal 20:11 KMVHAL normal CellCLI> list griddisk DATA_XFF_CD_00_XFFCEL01 active DATA_XFF_CD_01_XFFCEL01 active DATA_XFF_CD_02_XFFCEL01 proactive failure DATA_XFF_CD_03_XFFCEL01 active DATA_XFF_CD_04_XFFCEL01 active DATA_XFF_CD_05_XFFCEL01 active DATA_XFF_CD_06_XFFCEL01 active DATA_XFF_CD_07_XFFCEL01 active DATA_XFF_CD_08_XFFCEL01 active DATA_XFF_CD_09_XFFCEL01 active DATA_XFF_CD_10_XFFCEL01 active DATA_XFF_CD_11_XFFCEL01 active
在db节点无法发现异常磁盘的asm disk
[grid@ycdwdb01 grid]$ kfod disk=all -------------------------------------------------------------------------------- Disk Size Path User Group ============================================================ 1: 433152 Mb o/192.168.10.3/DATA_XFF_CD_00_XFFCEL01 <unknown> <unknown> 2: 433152 Mb o/192.168.10.3/DATA_XFF_CD_01_XFFCEL01 <unknown> <unknown> 3: 433152 Mb o/192.168.10.3/DATA_XFF_CD_03_XFFCEL01 <unknown> <unknown> 4: 433152 Mb o/192.168.10.3/DATA_XFF_CD_04_XFFCEL01 <unknown> <unknown> 5: 433152 Mb o/192.168.10.3/DATA_XFF_CD_05_XFFCEL01 <unknown> <unknown> 6: 433152 Mb o/192.168.10.3/DATA_XFF_CD_06_XFFCEL01 <unknown> <unknown> 7: 433152 Mb o/192.168.10.3/DATA_XFF_CD_07_XFFCEL01 <unknown> <unknown> 8: 433152 Mb o/192.168.10.3/DATA_XFF_CD_08_XFFCEL01 <unknown> <unknown> 9: 433152 Mb o/192.168.10.3/DATA_XFF_CD_09_XFFCEL01 <unknown> <unknown> 10: 433152 Mb o/192.168.10.3/DATA_XFF_CD_10_XFFCEL01 <unknown> <unknown> 11: 433152 Mb o/192.168.10.3/DATA_XFF_CD_11_XFFCEL01 <unknown> <unknown>
根据客户的反馈该磁盘组几乎全部被使用,asmcmd lsdg看到Usable_file_MB已经出现负值.证明该磁盘组本身的normal没有完全存储两份数据,在这样的情况下,继续坏盘会导致部分数据只有一份,因此也就出现了这里的磁盘组无法正常mount成功.
通过底层修复celldisk之后
CellCLI> list griddisk DATA_XFF_CD_00_XFFCEL01 active DATA_XFF_CD_01_XFFCEL01 active DATA_XFF_CD_02_XFFCEL01 active DATA_XFF_CD_03_XFFCEL01 active DATA_XFF_CD_04_XFFCEL01 active DATA_XFF_CD_05_XFFCEL01 active DATA_XFF_CD_06_XFFCEL01 active DATA_XFF_CD_07_XFFCEL01 active DATA_XFF_CD_08_XFFCEL01 active DATA_XFF_CD_09_XFFCEL01 active DATA_XFF_CD_10_XFFCEL01 active DATA_XFF_CD_11_XFFCEL01 active [grid@ycdwdb01 grid]$ kfod disk=all -------------------------------------------------------------------------------- Disk Size Path User Group ============================================================ 1: 433152 Mb o/192.168.10.3/DATA_XFF_CD_00_XFFCEL01 <unknown> <unknown> 2: 433152 Mb o/192.168.10.3/DATA_XFF_CD_01_XFFCEL01 <unknown> <unknown> 3: 433152 Mb o/192.168.10.3/DATA_XFF_CD_02_XFFCEL01 <unknown> <unknown> 4: 433152 Mb o/192.168.10.3/DATA_XFF_CD_03_XFFCEL01 <unknown> <unknown> 5: 433152 Mb o/192.168.10.3/DATA_XFF_CD_04_XFFCEL01 <unknown> <unknown> 6: 433152 Mb o/192.168.10.3/DATA_XFF_CD_05_XFFCEL01 <unknown> <unknown> 7: 433152 Mb o/192.168.10.3/DATA_XFF_CD_06_XFFCEL01 <unknown> <unknown> 8: 433152 Mb o/192.168.10.3/DATA_XFF_CD_07_XFFCEL01 <unknown> <unknown> 9: 433152 Mb o/192.168.10.3/DATA_XFF_CD_08_XFFCEL01 <unknown> <unknown> 10: 433152 Mb o/192.168.10.3/DATA_XFF_CD_09_XFFCEL01 <unknown> <unknown> 11: 433152 Mb o/192.168.10.3/DATA_XFF_CD_10_XFFCEL01 <unknown> <unknown> 12: 433152 Mb o/192.168.10.3/DATA_XFF_CD_11_XFFCEL01 <unknown> <unknown>
data磁盘组直接mount成功
Fri Dec 14 14:04:59 2018 SQL> alter diskgroup DATA_XFF mount NOTE: cache registered group DATA_XFF number=1 incarn=0x78a886e7 NOTE: cache began mount (not first) of group DATA_XFF number=1 incarn=0x78a886e7 NOTE: Assigning number (1,36) to disk (o/192.168.10.5/DATA_XFF_CD_11_XFFCEL03) NOTE: Assigning number (1,34) to disk (o/192.168.10.5/DATA_XFF_CD_10_XFFCEL03) NOTE: Assigning number (1,37) to disk (o/192.168.10.5/DATA_XFF_CD_04_XFFCEL03) NOTE: Assigning number (1,38) to disk (o/192.168.10.5/DATA_XFF_CD_00_XFFCEL03) NOTE: Assigning number (1,39) to disk (o/192.168.10.5/DATA_XFF_CD_03_XFFCEL03) NOTE: Assigning number (1,40) to disk (o/192.168.10.5/DATA_XFF_CD_05_XFFCEL03) NOTE: Assigning number (1,41) to disk (o/192.168.10.5/DATA_XFF_CD_08_XFFCEL03) NOTE: Assigning number (1,42) to disk (o/192.168.10.5/DATA_XFF_CD_01_XFFCEL03) NOTE: Assigning number (1,43) to disk (o/192.168.10.5/DATA_XFF_CD_09_XFFCEL03) NOTE: Assigning number (1,44) to disk (o/192.168.10.5/DATA_XFF_CD_06_XFFCEL03) NOTE: Assigning number (1,45) to disk (o/192.168.10.5/DATA_XFF_CD_07_XFFCEL03) NOTE: Assigning number (1,46) to disk (o/192.168.10.5/DATA_XFF_CD_02_XFFCEL03) NOTE: Assigning number (1,22) to disk (o/192.168.10.4/DATA_XFF_CD_10_XFFCEL02) NOTE: Assigning number (1,18) to disk (o/192.168.10.4/DATA_XFF_CD_06_XFFCEL02) NOTE: Assigning number (1,19) to disk (o/192.168.10.4/DATA_XFF_CD_07_XFFCEL02) NOTE: Assigning number (1,15) to disk (o/192.168.10.4/DATA_XFF_CD_03_XFFCEL02) NOTE: Assigning number (1,20) to disk (o/192.168.10.4/DATA_XFF_CD_08_XFFCEL02) NOTE: Assigning number (1,17) to disk (o/192.168.10.4/DATA_XFF_CD_05_XFFCEL02) NOTE: Assigning number (1,16) to disk (o/192.168.10.4/DATA_XFF_CD_04_XFFCEL02) NOTE: Assigning number (1,23) to disk (o/192.168.10.4/DATA_XFF_CD_11_XFFCEL02) NOTE: Assigning number (1,12) to disk (o/192.168.10.4/DATA_XFF_CD_00_XFFCEL02) NOTE: Assigning number (1,21) to disk (o/192.168.10.4/DATA_XFF_CD_09_XFFCEL02) NOTE: Assigning number (1,13) to disk (o/192.168.10.4/DATA_XFF_CD_01_XFFCEL02) NOTE: Assigning number (1,14) to disk (o/192.168.10.4/DATA_XFF_CD_02_XFFCEL02) NOTE: Assigning number (1,1) to disk (o/192.168.10.3/DATA_XFF_CD_05_XFFCEL01) NOTE: Assigning number (1,2) to disk (o/192.168.10.3/DATA_XFF_CD_03_XFFCEL01) NOTE: Assigning number (1,3) to disk (o/192.168.10.3/DATA_XFF_CD_06_XFFCEL01) NOTE: Assigning number (1,4) to disk (o/192.168.10.3/DATA_XFF_CD_09_XFFCEL01) NOTE: Assigning number (1,5) to disk (o/192.168.10.3/DATA_XFF_CD_04_XFFCEL01) NOTE: Assigning number (1,6) to disk (o/192.168.10.3/DATA_XFF_CD_07_XFFCEL01) NOTE: Assigning number (1,7) to disk (o/192.168.10.3/DATA_XFF_CD_11_XFFCEL01) NOTE: Assigning number (1,8) to disk (o/192.168.10.3/DATA_XFF_CD_01_XFFCEL01) NOTE: Assigning number (1,9) to disk (o/192.168.10.3/DATA_XFF_CD_00_XFFCEL01) NOTE: Assigning number (1,10) to disk (o/192.168.10.3/DATA_XFF_CD_10_XFFCEL01) NOTE: Assigning number (1,11) to disk (o/192.168.10.3/DATA_XFF_CD_08_XFFCEL01) NOTE: Assigning number (1,0) to disk (o/192.168.10.3/DATA_XFF_CD_02_XFFCEL01) Fri Dec 14 14:04:59 2018 GMON querying group 1 at 78 for pid 28, osid 76016 NOTE: Assigning number (1,24) to disk () NOTE: Assigning number (1,25) to disk () NOTE: Assigning number (1,26) to disk () NOTE: Assigning number (1,27) to disk () NOTE: Assigning number (1,28) to disk () NOTE: Assigning number (1,29) to disk () NOTE: Assigning number (1,30) to disk () NOTE: Assigning number (1,31) to disk () NOTE: Assigning number (1,32) to disk () NOTE: Assigning number (1,33) to disk () NOTE: Assigning number (1,35) to disk () GMON querying group 1 at 79 for pid 28, osid 76016 NOTE: cache opening disk 0 of grp 1: DATA_XFF_CD_02_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_02_XFFCEL01 NOTE: cache opening disk 1 of grp 1: DATA_XFF_CD_05_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_05_XFFCEL01 NOTE: cache opening disk 2 of grp 1: DATA_XFF_CD_03_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_03_XFFCEL01 NOTE: F1X0 found on disk 2 au 5 fcn 0.15948262 NOTE: cache opening disk 3 of grp 1: DATA_XFF_CD_06_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_06_XFFCEL01 NOTE: cache opening disk 4 of grp 1: DATA_XFF_CD_09_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_09_XFFCEL01 NOTE: cache opening disk 5 of grp 1: DATA_XFF_CD_04_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_04_XFFCEL01 NOTE: cache opening disk 6 of grp 1: DATA_XFF_CD_07_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_07_XFFCEL01 NOTE: cache opening disk 7 of grp 1: DATA_XFF_CD_11_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_11_XFFCEL01 NOTE: cache opening disk 8 of grp 1: DATA_XFF_CD_01_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_01_XFFCEL01 NOTE: cache opening disk 9 of grp 1: DATA_XFF_CD_00_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_00_XFFCEL01 NOTE: cache opening disk 10 of grp 1: DATA_XFF_CD_10_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_10_XFFCEL01 NOTE: cache opening disk 11 of grp 1: DATA_XFF_CD_08_XFFCEL01 path:o/192.168.10.3/DATA_XFF_CD_08_XFFCEL01 NOTE: cache opening disk 12 of grp 1: DATA_XFF_CD_00_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_00_XFFCEL02 NOTE: cache opening disk 13 of grp 1: DATA_XFF_CD_01_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_01_XFFCEL02 NOTE: cache opening disk 14 of grp 1: DATA_XFF_CD_02_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_02_XFFCEL02 NOTE: cache opening disk 15 of grp 1: DATA_XFF_CD_03_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_03_XFFCEL02 NOTE: cache opening disk 16 of grp 1: DATA_XFF_CD_04_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_04_XFFCEL02 NOTE: cache opening disk 17 of grp 1: DATA_XFF_CD_05_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_05_XFFCEL02 NOTE: cache opening disk 18 of grp 1: DATA_XFF_CD_06_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_06_XFFCEL02 NOTE: cache opening disk 19 of grp 1: DATA_XFF_CD_07_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_07_XFFCEL02 NOTE: cache opening disk 20 of grp 1: DATA_XFF_CD_08_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_08_XFFCEL02 NOTE: cache opening disk 21 of grp 1: DATA_XFF_CD_09_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_09_XFFCEL02 NOTE: F1X0 found on disk 21 au 2 fcn 0.15948262 NOTE: cache opening disk 22 of grp 1: DATA_XFF_CD_10_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_10_XFFCEL02 NOTE: cache opening disk 23 of grp 1: DATA_XFF_CD_11_XFFCEL02 path:o/192.168.10.4/DATA_XFF_CD_11_XFFCEL02 NOTE: cache opening disk 36 of grp 1: DATA_XFF_CD_11_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_11_XFFCEL03 NOTE: cache opening disk 37 of grp 1: DATA_XFF_CD_04_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_04_XFFCEL03 NOTE: cache opening disk 38 of grp 1: DATA_XFF_CD_00_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_00_XFFCEL03 NOTE: cache opening disk 39 of grp 1: DATA_XFF_CD_03_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_03_XFFCEL03 NOTE: cache opening disk 40 of grp 1: DATA_XFF_CD_05_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_05_XFFCEL03 NOTE: cache opening disk 41 of grp 1: DATA_XFF_CD_08_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_08_XFFCEL03 NOTE: cache opening disk 42 of grp 1: DATA_XFF_CD_01_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_01_XFFCEL03 NOTE: cache opening disk 43 of grp 1: DATA_XFF_CD_09_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_09_XFFCEL03 NOTE: cache opening disk 44 of grp 1: DATA_XFF_CD_06_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_06_XFFCEL03 NOTE: F1X0 found on disk 44 au 2 fcn 0.15948262 NOTE: cache opening disk 45 of grp 1: DATA_XFF_CD_07_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_07_XFFCEL03 NOTE: cache opening disk 46 of grp 1: DATA_XFF_CD_02_XFFCEL03 path:o/192.168.10.5/DATA_XFF_CD_02_XFFCEL03 NOTE: cache mounting (not first) normal redundancy group 1/0x78A886E7 (DATA_XFF) Fri Dec 14 14:04:59 2018 kjbdomatt send to inst 2 Fri Dec 14 14:04:59 2018 NOTE: attached to recovery domain 1 NOTE: redo buffer size is 512 blocks (2101760 bytes) Fri Dec 14 14:04:59 2018 NOTE: LGWR attempting to mount thread 2 for diskgroup 1 (DATA_XFF) NOTE: LGWR found thread 2 closed at ABA 98.4672 NOTE: LGWR mounted thread 2 for diskgroup 1 (DATA_XFF) NOTE: LGWR opening thread 2 at fcn 0.18931129 ABA 99.4673 NOTE: cache mounting group 1/0x78A886E7 (DATA_XFF) succeeded NOTE: cache ending mount (success) of group DATA_XFF number=1 incarn=0x78a886e7 GMON querying group 1 at 80 for pid 19, osid 9805 Fri Dec 14 14:04:59 2018 NOTE: Instance updated compatible.asm to 11.2.0.3.0 for grp 1 SUCCESS: diskgroup DATA_XFF was mounted SUCCESS: alter diskgroup DATA_XFF mount
恢复后的asm磁盘状态
ASMCMD> lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED NORMAL Y 512 4096 4194304 15160320 4776184 5197824 -210820 12 N DATA_XFF/ MOUNTED NORMAL N 512 4096 4194304 864896 863400 298240 282580 0 Y DBFS_DG/ MOUNTED NORMAL N 512 4096 4194304 3787840 2157232 1298688 429272 0 N RECO_XFF/
后续数据库open成功,有部分坏块通过技术手段进行二次处理,至此数据库恢复完成,成功抢救了客户Oracle Exadata中的绝大部分数据.如果有类似xd故障恢复,无法自行解决,需要恢复支持请联系我们
Phone:17813235971 Q Q:107644445 E-Mail:dba@xifenfei.com
发表在 非常规恢复
标签为 exadata mount, exadata坏盘恢复, exadata恢复, exadata磁盘组恢复, ORA-15040, ORA-15042, ORA-15066, xd坏盘恢复, xd恢复
评论关闭