标签云
asm恢复 bbed bootstrap$ dul In Memory kcbzib_kcrsds_1 kccpb_sanity_check_2 kfed MySQL恢复 ORA-00312 ORA-00607 ORA-00704 ORA-01110 ORA-01555 ORA-01578 ORA-08103 ORA-600 2131 ORA-600 2662 ORA-600 2663 ORA-600 3020 ORA-600 4000 ORA-600 4137 ORA-600 4193 ORA-600 4194 ORA-600 16703 ORA-600 kcbzib_kcrsds_1 ORA-600 KCLCHKBLK_4 ORA-15042 ORA-15196 ORACLE 12C oracle dul ORACLE PATCH Oracle Recovery Tools oracle加密恢复 oracle勒索 oracle勒索恢复 oracle异常恢复 Oracle 恢复 ORACLE恢复 ORACLE数据库恢复 oracle 比特币 OSD-04016 YOUR FILES ARE ENCRYPTED 勒索恢复 比特币加密文章分类
- Others (2)
- 中间件 (2)
- WebLogic (2)
- 操作系统 (102)
- 数据库 (1,670)
- DB2 (22)
- MySQL (73)
- Oracle (1,532)
- Data Guard (52)
- EXADATA (8)
- GoldenGate (21)
- ORA-xxxxx (159)
- ORACLE 12C (72)
- ORACLE 18C (6)
- ORACLE 19C (14)
- ORACLE 21C (3)
- Oracle 23ai (7)
- Oracle ASM (65)
- Oracle Bug (8)
- Oracle RAC (52)
- Oracle 安全 (6)
- Oracle 开发 (28)
- Oracle 监听 (28)
- Oracle备份恢复 (560)
- Oracle安装升级 (91)
- Oracle性能优化 (62)
- 专题索引 (5)
- 勒索恢复 (78)
- PostgreSQL (18)
- PostgreSQL恢复 (6)
- SQL Server (27)
- SQL Server恢复 (8)
- TimesTen (7)
- 达梦数据库 (2)
- 生活娱乐 (2)
- 至理名言 (11)
- 虚拟化 (2)
- VMware (2)
- 软件开发 (37)
- Asp.Net (9)
- JavaScript (12)
- PHP (2)
- 小工具 (20)
-
最近发表
- ORA-600 krse_arc_complete.4
- Oracle 19c 202410补丁(RUs+OJVM)
- ntfs MFT损坏(ntfs文件系统故障)导致oracle异常恢复
- .mkp扩展名oracle数据文件加密恢复
- 清空redo,导致ORA-27048: skgfifi: file header information is invalid
- A_H_README_TO_RECOVER勒索恢复
- 通过alert日志分析客户自行对一个数据库恢复的来龙去脉和点评
- ORA-12514: TNS: 监听进程不能解析在连接描述符中给出的SERVICE_NAME
- ORA-01092 ORA-00604 ORA-01558故障处理
- ORA-65088: database open should be retried
- Oracle 19c异常恢复—ORA-01209/ORA-65088
- ORA-600 16703故障再现
- 数据库启动报ORA-27102 OSD-00026 O/S-Error: (OS 1455)
- .[metro777@cock.li].Elbie勒索病毒加密数据库恢复
- 应用连接错误,初始化mysql数据库恢复
- RAC默认服务配置优先节点
- Oracle 19c RAC 替换私网操作
- 监听报TNS-12541 TNS-12560 TNS-00511错误
- drop tablespace xxx including contents恢复
- Linux 8 修改网卡名称
标签归档:ORA-04031
多cpu环境中运行root.sh失败,asm报ORA-04031
有朋友和我反馈,说他们在装linux 6.5上面装11.2.0.3的rac出现异常,root.sh在第一个节点执行就失败了,请求帮助
根据上面记录,查看asmca日志
[main] [ 2015-07-24 12:49:35.885 CST ] [SQLEngine.reInitialize:738] Reinitializing SQLEngine... [main] [ 2015-07-24 12:49:35.885 CST ] [OracleHome.getVersion:889] OracleHome.getVersion called. Current Version: 11.2.0.3.0 [main] [ 2015-07-24 12:49:35.885 CST ] [OracleHome.getVersion:957] Current Version From Inventory: 11.2.0.3.0 [main] [ 2015-07-24 12:49:35.885 CST ] [OracleHome.getVersion:889] OracleHome.getVersion called. Current Version: 11.2.0.3.0 [main] [ 2015-07-24 12:49:35.886 CST ] [OracleHome.getVersion:957] Current Version From Inventory: 11.2.0.3.0 [main] [ 2015-07-24 12:49:35.886 CST ] [OracleHome.getVersion:889] OracleHome.getVersion called. Current Version: 11.2.0.3.0 [main] [ 2015-07-24 12:49:35.886 CST ] [OracleHome.getVersion:957] Current Version From Inventory: 11.2.0.3.0 [main] [ 2015-07-24 12:49:35.886 CST ] [SQLPlusEngine.getCmmdParams:222] m_home 11.2.0.3.0 [main] [ 2015-07-24 12:49:35.887 CST ] [SQLPlusEngine.getCmmdParams:223] version > 112 true [main] [ 2015-07-24 12:49:35.887 CST ] [SQLEngine.getEnvParams:555] Default NLS_LANG: AMERICAN_AMERICA.AL32UTF8 [main] [ 2015-07-24 12:49:35.887 CST ] [SQLEngine.getEnvParams:565] NLS_LANG: AMERICAN_AMERICA.AL32UTF8 [main] [ 2015-07-24 12:49:35.888 CST ] [SQLEngine.initialize:325] Execing SQLPLUS/SVRMGR process... [main] [ 2015-07-24 12:49:35.900 CST ] [SQLEngine.initialize:362] m_bReaderStarted: false [main] [ 2015-07-24 12:49:35.900 CST ] [SQLEngine.initialize:366] Starting Reader Thread... [main] [ 2015-07-24 12:49:35.901 CST ] [SQLEngine.initialize:415] Waiting for m_bReaderStarted to be true [main] [ 2015-07-24 12:49:35.972 CST ] [SQLEngine.done:2189] Done called [main] [ 2015-07-24 12:49:35.972 CST ] [UsmcaLogger.logException:173] SEVERE:method oracle.sysman.assistants.usmca.backend.USMInstance:configureLocalASM [main] [ 2015-07-24 12:49:35.973 CST ] [UsmcaLogger.logException:174] ORA-01012: not logged on [main] [ 2015-07-24 12:49:35.973 CST ] [UsmcaLogger.logException:175] oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-01012: not logged on oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeImpl(SQLEngine.java:1658) oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeQuery(SQLEngine.java:831) oracle.sysman.assistants.usmca.backend.USMInstance.configureLocalASM(USMInstance.java:3036) oracle.sysman.assistants.usmca.service.UsmcaService.configureLocalASM(UsmcaService.java:1049) oracle.sysman.assistants.usmca.model.UsmcaModel.performConfigureLocalASM(UsmcaModel.java:944) oracle.sysman.assistants.usmca.model.UsmcaModel.performOperation(UsmcaModel.java:797) oracle.sysman.assistants.usmca.Usmca.execute(Usmca.java:174) oracle.sysman.assistants.usmca.Usmca.main(Usmca.java:369) [main] [ 2015-07-24 12:49:35.989 CST ] [UsmcaLogger.logException:173] SEVERE:method oracle.sysman.assistants.usmca.backend.USMInstance:configureLocalASM [main] [ 2015-07-24 12:49:35.989 CST ] [UsmcaLogger.logException:174] ORA-03113: end-of-file on communication channel [main] [ 2015-07-24 12:49:35.989 CST ] [UsmcaLogger.logException:175] oracle.sysman.assistants.util.sqlEngine.SQLFatalErrorException: ORA-03113: end-of-file on communication channel
这里可以看出来,asm实例无法登陆(ORA-01012和ORA-03113),根据这样的错误,分析asm日志
Reconfiguration complete Fri Jul 24 12:49:29 2015 LCK0 started with pid=22, OS id=46913 Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_lmd0_46887.trc (incident=81): ORA-04031: unable to allocate 7072 bytes of shared memory ("shared pool","unknown object","sga heap(1,1)","ges resource ") Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_81/+ASM1_lmd0_46887_i81.trc Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_lck0_46913.trc (incident=177): ORA-04031: unable to allocate 760 bytes of shared memory ("shared pool","unknown object","KKSSP^1343","kglss") Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_177/+ASM1_lck0_46913_i177.trc Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_lmon_46885.trc (incident=73): ORA-04031: unable to allocate 632 bytes of shared memory ("shared pool","unknown object","sga heap(1,1)","name-service ") Incident details in: /u01/app/grid/diag/asm/+asm/+ASM1/incident/incdir_73/+ASM1_lmon_46885_i73.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_lck0_46913.trc: ORA-04031: unable to allocate 760 bytes of shared memory ("shared pool","unknown object","KKSSP^1343","kglss") System state dump requested by (instance=1, osid=46913 (LCK0)), summary=[abnormal instance termination]. System State dumped to trace file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_diag_46879.trc Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. LCK0 (ospid: 46913): terminating the instance due to error 4031 Fri Jul 24 12:49:35 2015 ORA-1092 : opitsk aborting process Instance terminated by LCK0, pid = 46913
进一步分析asm日志,发现是大家熟悉的asm的ORA-4031问题,那就是说明数据库在执行root.sh的时候使用默认参数文件启动asm的时候shared pool不够大(根据ORACLE最佳实践,建议memory_target=1536M及其以上值),从而出现该问题。类似Bug 14292825 ORA-4031 in ASM as default memory parameters values for 11.2 ASM instances low,根据官方描述该问题在11.2.0.4中修复
通过asm日志发现相关默认值配置
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options. ORACLE_HOME = /u01/app/11.2.0/grid System name: Linux Node name: RAC01 Release: 2.6.32-358.el6.x86_64 Version: #1 SMP Tue Jan 29 11:47:41 EST 2013 Machine: x86_64 Using parameter settings in client-side pfile /u01/app/11.2.0/grid/dbs/init+ASM1.ora on machine RAC01 System parameters with non-default values: large_pool_size = 16M instance_type = "asm" remote_login_passwordfile= "EXCLUSIVE" asm_power_limit = 1 diagnostic_dest = "/u01/app/grid" Cluster communication is configured to use the following interface(s) for this instance 10.10.10.31 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Fri Jul 24 12:49:27 2015
通过查询/proc/cpuinfo,检查cpu数量
processor : 191 vendor_id : GenuineIntel cpu family : 6 model : 62 model name : Intel(R) Xeon(R) CPU E7-8850 v2 @ 2.30GHz stepping : 7 cpu MHz : 1200.000 cache size : 24576 KB physical id : 7 siblings : 24 core id : 13 cpu cores : 12 apicid : 251 initial apicid : 251 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes
而根据How To Determine The Default Number Of Subpools Allocated During Startup (Doc ID 455179.1)中描述
最多7个subpool(这里一共有192个cpu,因此subpool数量为7)
每个suppool最少512m内存,因此shared pool最小需要3.5G(而默认值几百M,远远不够)
由于cpu多,导致shared pool的Subpools 更加多,使得shared pool的需求量更加大。至此本次故障原因可以总结:
由于cpu较多,需要更多的shared pool,而11.2.0.3中由于asm默认内存分配较少,导致在asm启动之时出现shared pool不足(本身默认值小,而且shared pool需求大,从而出现了ORA-04031就不奇怪了),因为运行root.sh过程中asm无法正常启动,从而使得root.sh运行失败。
处理办法:临时disable部分cpu,然后重新执行root.sh,修改asm内存分配,再enable cpu.
特别说明:此故障acs的兄弟遇到过,所以这次我能够快速反应,感谢acs兄弟们的帮忙,另外有权限的朋友可以看看:3-10479952701和3-7976215751等sr描述
查询v$session报ORA-04031错误
客户的数据库在出账期间有工具登录Oracle数据库偶尔性报ORA-04031,经过分析是因为该工具需要查询v$session,经过分析确定是Bug 12808696 – Shared pool memory leak of “hng: All sessi” memory (Doc ID 12808696.8),重现错误如下
节点1进行查询报ORA-4031
SQL> select count(*) from v$session; COUNT(*) ---------- 1536 SQL> select count(*) from gv$session; COUNT(*) ---------- 2089 SQL> select /*+ full(t) */ count(*) from gv$session t; COUNT(*) ---------- 2053 SQL> select * from gv$session; select * from gv$session * ERROR at line 1: ORA-12801: error signaled in parallel query server PZ93, instance ocs_db_2:zjocs2 (2) ORA-04031: unable to allocate 308448 bytes of shared memory ("shared pool","unknown object","sga heap(1,0)","hng: All sessions data for API.")
节点2进行查询报ORA-04031
SQL> select * from gv$session; select * from gv$session * ERROR at line 1: ORA-12801: error signaled in parallel query server PZ95, instance ocs_db_2:zjocs2 (2) ORA-04031: unable to allocate 308448 bytes of shared memory ("shared pool","unknown object","sga heap(6,0)","hng: All sessions data for API.") SQL> select * from v$session; select * from v$session * ERROR at line 2: ORA-04031: unable to allocate 308448 bytes of shared memory ("shared pool","unknown object","sga heap(7,0)","hng: All sessions data for API.")
通过上述分析:确认是节点2的v$session遭遇到Bug 12808696,导致在该节点中中查询v$session和Gv$session报ORA-04031,而在节点1中查询v$session正常,查询Gv$session报ORA-04031.
该bug在11.1.0.6中修复,所有的10g版本中未修复,只能通过临时重启来暂时避免,注意该bug通过flash shared_pool无法解决
如果您有权限可以进步一查询SR 3-7670890781: 查询v$session的BLOCKING_SESSION字段时,出现ora-04031错误
因asm sga_target设置不当导致11gr2 rac无法正常启动
2014年第一个故障排查和解决:同事反馈给我说solaris 11.2 两节点rac无法启动,让我帮忙看下。通过分析是因为sga_target参数设置不合理导致asm无法正常启动
GI无法正常启动
grid@zwq-rpt1:~$crsctl status resource -t CRS-4535: Cannot communicate with Cluster Ready Services CRS-4000: Command Status failed, or completed with errors. grid@zwq-rpt1:~$crsctl status resource -t -init -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.asm 1 ONLINE OFFLINE Instance Shutdown ora.cluster_interconnect.haip 1 ONLINE ONLINE zwq-rpt1 ora.crf 1 ONLINE ONLINE zwq-rpt1 ora.crsd 1 ONLINE OFFLINE ora.cssd 1 ONLINE ONLINE zwq-rpt1 ora.cssdmonitor 1 ONLINE ONLINE zwq-rpt1 ora.ctssd 1 ONLINE ONLINE zwq-rpt1 ACTIVE:0 ora.diskmon 1 OFFLINE OFFLINE ora.evmd 1 ONLINE INTERMEDIATE zwq-rpt1 ora.gipcd 1 ONLINE ONLINE zwq-rpt1 ora.gpnpd 1 ONLINE ONLINE zwq-rpt1 ora.mdnsd 1 ONLINE ONLINE zwq-rpt1
asm未正常启动
GI日志报错
2014-01-01 00:40:47.708 [cssd(1418)]CRS-1605:CSSD voting file is online: /dev/rdsk/emcpower0a; details in /export/home/app/grid/log/zwq-rpt1/cssd/ocssd.log. 2014-01-01 00:40:53.234 [cssd(1418)]CRS-1601:CSSD Reconfiguration complete. Active nodes are zwq-rpt1 zwq-rpt2 . 2014-01-01 00:40:56.659 [ctssd(1483)]CRS-2407:The new Cluster Time Synchronization Service reference node is host zwq-rpt2. 2014-01-01 00:40:56.661 [ctssd(1483)]CRS-2401:The Cluster Time Synchronization Service started on host zwq-rpt1. 2014-01-01 00:41:02.016 [ctssd(1483)]CRS-2408:The clock on host zwq-rpt1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time. 2014-01-01 00:43:23.874 [/export/home/app/grid/bin/oraagent.bin(1348)]CRS-5019:All OCR locations are on ASM disk groups [], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/export/home/app/grid/log/zwq-rpt1/agent/ohasd/oraagent_grid/oraagent_grid.log". 2014-01-01 00:45:42.837 [/export/home/app/grid/bin/oraagent.bin(1348)]CRS-5019:All OCR locations are on ASM disk groups [], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/export/home/app/grid/log/zwq-rpt1/agent/ohasd/oraagent_grid/oraagent_grid.log". 2014-01-01 00:48:02.087 [/export/home/app/grid/bin/oraagent.bin(1348)]CRS-5019:All OCR locations are on ASM disk groups [], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/export/home/app/grid/log/zwq-rpt1/agent/ohasd/oraagent_grid/oraagent_grid.log". 2014-01-01 00:48:18.836 [ohasd(1083)]CRS-2807:Resource 'ora.asm' failed to start automatically. 2014-01-01 00:48:18.837 [ohasd(1083)]CRS-2807:Resource 'ora.crsd' failed to start automatically. 2014-01-01 01:05:15.396 [/export/home/app/grid/bin/oraagent.bin(1348)]CRS-5019:All OCR locations are on ASM disk groups [CRSDG], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/export/home/app/grid/log/zwq-rpt1/agent/ohasd/oraagent_grid/oraagent_grid.log". 2014-01-01 01:05:45.101 [/export/home/app/grid/bin/oraagent.bin(1348)]CRS-5019:All OCR locations are on ASM disk groups [CRSDG], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/export/home/app/grid/log/zwq-rpt1/agent/ohasd/oraagent_grid/oraagent_grid.log". 2014-01-01 01:06:15.104 [/export/home/app/grid/bin/oraagent.bin(1348)]CRS-5019:All OCR locations are on ASM disk groups [CRSDG], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/export/home/app/grid/log/zwq-rpt1/agent/ohasd/oraagent_grid/oraagent_grid.log".
这里较为明显的看到,因为asm磁盘组异常导致ocr无法被访问导致crs无法正常启动
ORAAGENT日志
2014-01-01 00:43:23.870: [ora.asm][9] {0:0:2} [start] InstConnection::connectInt (2) Exception OCIException 2014-01-01 00:43:23.870: [ora.asm][9] {0:0:2} [start] InstConnection:connect:excp OCIException OCI error 604 2014-01-01 00:43:23.870: [ora.asm][9] {0:0:2} [start] DgpAgent::queryDgStatus excp ORA-00604: error occurred at recursive SQL level 1 ORA-04031: unable to allocate 32 bytes of shared memory ("shared pool","unknown object","KGLH0^34f764db","kglHeapInitialize:temp")
报了较为清晰的ORA-4031错误,检查asm日志
ASM日志报错
Wed Jan 01 00:47:33 2014 ORACLE_BASE not set in environment. It is recommended that ORACLE_BASE be set in the environment Reusing ORACLE_BASE from an earlier startup = /export/home/app/oracle Wed Jan 01 00:47:39 2014 Errors in file /export/home/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_1728.trc (incident=291447): ORA-04031: unable to allocate 32 bytes of shared memory ("shared pool","unknown object","KGLH0^34f764db","kglHeapInitialize:temp") Incident details in: /export/home/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_291447/+ASM1_ora_1728_i291447.trc Wed Jan 01 00:47:48 2014 Dumping diagnostic data in directory=[cdmp_20140101004748], requested by (instance=1, osid=1728), summary=[incident=291447]. Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Wed Jan 01 00:47:53 2014 Errors in file /export/home/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_1730.trc (incident=291448): ORA-04031: unable to allocate 32 bytes of shared memory ("shared pool","unknown object","KGLH0^34f764db","kglHeapInitialize:temp") Incident details in: /export/home/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_291448/+ASM1_ora_1730_i291448.trc Wed Jan 01 00:48:01 2014 Dumping diagnostic data in directory=[cdmp_20140101004801], requested by (instance=1, osid=1730), summary=[incident=291448]. Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Wed Jan 01 00:48:07 2014 Errors in file /export/home/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_1732.trc (incident=291449): ORA-04031: unable to allocate 32 bytes of shared memory ("shared pool","unknown object","KGLH0^34f764db","kglHeapInitialize:temp") Incident details in: /export/home/app/oracle/diag/asm/+asm/+ASM1/incident/incdir_291449/+ASM1_ora_1732_i291449.trc Wed Jan 01 00:48:16 2014 Dumping diagnostic data in directory=[cdmp_20140101004816], requested by (instance=1, osid=1732), summary=[incident=291449]. Use ADRCI or Support Workbench to package the incident. See Note 411.1 at My Oracle Support for error and packaging details. Wed Jan 01 00:48:16 2014 License high water mark = 1 USER (ospid: 1736): terminating the instance Instance terminated by USER, pid = 1736
这里可以清晰的看到,因为shared pool不足,导致asm报ora-4031错误,从而使得asm无法正常启动
分析原因
Starting up: Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options. ORACLE_HOME = /export/home/app/grid System name: SunOS Node name: zwq-rpt1 Release: 5.11 Version: 11.1 Machine: sun4v Using parameter settings in server-side spfile +CRSDG/zwq-rpt-cluster/asmparameterfile/registry.253.823992831 System parameters with non-default values: sga_max_size = 2G large_pool_size = 16M instance_type = "asm" sga_target = 0 remote_login_passwordfile= "EXCLUSIVE" asm_diskstring = "/dev/rdsk/*" asm_diskgroups = "FRADG" asm_diskgroups = "DATADG" asm_power_limit = 1 diagnostic_dest = "/export/home/app/oracle"
这里可以看到sga_target被设置为了0,而shared pool又未被配置,这里因为shared pool不足从而出现了ORA-4031,从而导致crs在启动asm的过程失败,从而使得ocr不能被访问,进而使得crs不能正常启动.
处理方法
1.编辑pfile
grid@zwq-rpt1:/export/home/app/oracle/diag/asm/+asm/+ASM1/trace$vi /tmp/asm.pfile memory_target = 2G large_pool_size = 16M instance_type = "asm" sga_target = 0 remote_login_passwordfile= "EXCLUSIVE" asm_diskstring = "/dev/rdsk/*" asm_diskgroups = "FRADG" asm_diskgroups = "DATADG" asm_power_limit = 1 diagnostic_dest = "/export/home/app/oracle"
2.启动asm
grid@zwq-rpt1:/export/home/app/oracle/diag/asm/+asm/+ASM1/trace$sqlplus / as sysasm SQL*Plus: Release 11.2.0.3.0 Production on Wed Jan 1 01:04:10 2014 Copyright (c) 1982, 2011, Oracle. All rights reserved. Connected to an idle instance. SQL> startup pfile='/tmp/asm.pfile' ASM instance started Total System Global Area 2138521600 bytes Fixed Size 2161024 bytes Variable Size 2102806144 bytes ASM Cache 33554432 bytes ASM diskgroups mounted
3. 创建spfile
SQL> create spfile='+CRSDG' FROM PFILE='/tmp/asm.pfile'; File created. --asm alert日志 Wed Jan 01 01:08:59 2014 NOTE: updated gpnp profile ASM SPFILE to NOTE: updated gpnp profile ASM diskstring: /dev/rdsk/* NOTE: updated gpnp profile ASM diskstring: /dev/rdsk/* NOTE: updated gpnp profile ASM SPFILE to +CRSDG/zwq-rpt-cluster/asmparameterfile/registry.253.835664939
4. 关闭asm
SQL> shutdown immediate ORA-15097: cannot SHUTDOWN ASM instance with connected client (process 1971) SQL> shutdown abort ASM instance shutdown
5. 重启crs
root@zwq-rpt1:~# crsctl stop crs -f root@zwq-rpt1:~# crsctl start crs
6. 重启其他节点crs
root@zwq-rpt2:~# crsctl stop crs -f root@zwq-rpt2:~# crsctl start crs
7. 检查结果
root@zwq-rpt1:~# crsctl status res -t -------------------------------------------------------------------------------- NAME TARGET STATE SERVER STATE_DETAILS -------------------------------------------------------------------------------- Local Resources -------------------------------------------------------------------------------- ora.CRSDG.dg ONLINE ONLINE zwq-rpt1 ONLINE ONLINE zwq-rpt2 ora.DATADG.dg ONLINE ONLINE zwq-rpt1 ONLINE ONLINE zwq-rpt2 ora.FRADG.dg ONLINE ONLINE zwq-rpt1 ONLINE ONLINE zwq-rpt2 ora.LISTENER.lsnr ONLINE ONLINE zwq-rpt1 ONLINE ONLINE zwq-rpt2 ora.asm ONLINE ONLINE zwq-rpt1 Started ONLINE ONLINE zwq-rpt2 Started ora.gsd OFFLINE OFFLINE zwq-rpt1 OFFLINE OFFLINE zwq-rpt2 ora.net1.network ONLINE ONLINE zwq-rpt1 ONLINE ONLINE zwq-rpt2 ora.ons ONLINE ONLINE zwq-rpt1 ONLINE ONLINE zwq-rpt2 -------------------------------------------------------------------------------- Cluster Resources -------------------------------------------------------------------------------- ora.LISTENER_SCAN1.lsnr 1 ONLINE ONLINE zwq-rpt1 ora.cvu 1 ONLINE ONLINE zwq-rpt1 ora.oc4j 1 ONLINE ONLINE zwq-rpt1 ora.rptdb.db 1 ONLINE ONLINE zwq-rpt1 Open 2 ONLINE ONLINE zwq-rpt2 Open ora.scan1.vip 1 ONLINE ONLINE zwq-rpt1 ora.zwq-rpt1.vip 1 ONLINE ONLINE zwq-rpt1 ora.zwq-rpt2.vip 1 ONLINE ONLINE zwq-rpt2
至此恢复正常,2014年第一个故障顺利解决