分类目录归档：Oracle RAC

CSSD signal 11 in thread clssnmRcfgMgrThread故障处理

发表于 2025 年 2 月 21 日 由 惜分飞

一个客户,集群无法启动,只能启动到如下状态
11

查看cssd日志有CSSD signal 11 in thread clssnmRcfgMgrThread报错

2025-02-21 18:21:25.500: [    CSSD][2788693760]clssnmDoSyncUpdate: node(2) is transitioning from joining state to active state
2025-02-21 18:21:25.500: [    CSSD][2788693760]clssnmDoSyncUpdate: Wait for 0 vote ack(s)
2025-02-21 18:21:25.500: [    CSSD][2788693760]clssnmDoSyncUpdate: waiting to update states on disk
2025-02-21 18:21:25.700: [    CSSD][2788693760]clssnmDoSyncUpdate: waiting to update states on disk
2025-02-21 18:21:25.901: [    CSSD][2788693760]clssnmDoSyncUpdate: waiting to update states on disk
2025-02-21 18:21:25.995: [    CSSD][2801538816]clssnmvDiskPing: Writing with status 0x2, timestamp 1740133285/5870104
2025-02-21 18:21:25.997: [    CSSD][2799818496]clssnmvDiskKillCheck: not evicted, file /dev/dm-4 flags 0x00000000,
                                                                          kill block unique 0, my unique 1740133265
2025-02-21 18:21:26.000: [    CSSD][2793424640]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 2 waited 500
2025-02-21 18:21:26.101: [    CSSD][2788693760]clssnmDoSyncUpdate: waiting to update states on disk
2025-02-21 18:21:26.302: [    CSSD][2788693760]clssnmDoSyncUpdate: waiting to update states on disk
2025-02-21 18:21:26.497: [    CSSD][2801538816]clssnmvDiskPing: Writing with status 0x2, timestamp 1740133286/5870604
2025-02-21 18:21:26.502: [    CSSD][2788693760]clssnmDoSyncUpdate: waiting to update states on disk
2025-02-21 18:21:26.702: [    CSSD][2788693760]clssnmDoSyncUpdate: waiting to update states on disk
2025-02-21 18:21:26.902: [    CSSD][2788693760]clssnmDoSyncUpdate: waiting to update states on disk
2025-02-21 18:21:26.997: [    CSSD][2799818496]clssnmvDiskKillCheck: not evicted, file /dev/dm-4 flags 0x00000000,
                                                                      kill block unique 0, my unique 1740133265
2025-02-21 18:21:26.997: [    CSSD][2801538816]clssnmvDiskPing: Writing with status 0x2, timestamp 1740133286/5871114
2025-02-21 18:21:27.000: [    CSSD][2793424640]clssgmWaitOnEventValue: after CmInfo State  val 3, eval 2 waited 0
2025-02-21 18:21:27.102: [    CSSD][2788693760]clssnmCheckDskInfo: Checking disk info...
2025-02-21 18:21:27.102: [    CSSD][2788693760]clssnmCheckDskInfo: diskTimeout set to (200000)ms
2025-02-21 18:21:27.103: [    CSSD][2788693760]###################################
2025-02-21 18:21:27.103: [    CSSD][2788693760]clssscExit: CSSD signal 11 in thread clssnmRcfgMgrThread
2025-02-21 18:21:27.103: [    CSSD][2788693760]###################################
2025-02-21 18:21:27.103: [    CSSD][2788693760](:CSSSC00012:)clssscExit: A fatal error occurred and the CSS daemon is terminating abnormally
2025-02-21 18:21:27.103: [    CSSD][2788693760]

----- Call Stack Trace -----
2025-02-21 18:21:27.103: [    CSSD][2788693760]calling              call     entry                argument values in hex      
2025-02-21 18:21:27.103: [    CSSD][2788693760]location             type     point                (? means dubious value)     
2025-02-21 18:21:27.103: [    CSSD][2788693760]-------------------- -------- -------------------- ----------------------------
2025-02-21 18:21:27.109: [    CSSD][2788693760]clssscExit()+745     call     kgdsdst()            000000000 ? 000000000 ?
2025-02-21 18:21:27.109: [    CSSD][2788693760]                                                   7F9EA637A650 ? 7F9EA637A728 ?
2025-02-21 18:21:27.109: [    CSSD][2788693760]                                                   7F9EA637F1D0 ? 000000003 ?
2025-02-21 18:21:27.109: [    CSSD][2788693760]s0clsssc_sighandler  call     clssscExit()         001FB9FA0 ? 000000002 ?
2025-02-21 18:21:27.109: [    CSSD][2788693760]()+616                                             7F9EA637A650 ? 7F9EA637A728 ?
2025-02-21 18:21:27.109: [    CSSD][2788693760]                                                   7F9EA637F1D0 ? 000000003 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]__sighandler()       call     s0clsssc_sighandler  00000000B ? 000000002 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                              ()                   7F9EA637A650 ? 7F9EA637A728 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   7F9EA637F1D0 ? 000000003 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]clssnmCheckSplit()+  signal   __sighandler()       001BEE8A8 ? 000000000 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]378                                                002039A80 ? 000000001 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   0004D2B40 ? 7F9EA63803C0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]clssnmCheckDskInfo(  call     clssnmCheckSplit()   001FB9FA0 ? 001DC83F0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760])+387                                              000030D40 ? 000000001 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   0004D2B40 ? 7F9EA63803C0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]clssnmDoSyncUpdate(  call     clssnmCheckDskInfo(  001FB9FA0 ? 001DC83F0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760])+4692                        )                    000000001 ? 000000001 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   0004D2B40 ? 7F9EA63803C0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]clssnmLocalJoinEven  call     clssnmDoSyncUpdate(  001FB9FA0 ? 001DC83F0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]t()+3992                      )                    FFFFFFFFFFFFFFFF ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   000000001 ? 7F9EA6380D20 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   7F9EA63803C0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]clssnmRcfgMgrThread  call     clssnmLocalJoinEven  001FB9FA0 ? 001DC83F0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]()+2290                       t()                  FFFFFFFFFFFFFFFF ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   000000001 ? 7F9EA6380D20 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   7F9EA63803C0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]clssscthrdmain()+25  call     clssnmRcfgMgrThread  001FB9FA0 ? 001DC83F0 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]3                             ()                   FFFFFFFFFFFFFFFF ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   000000001 ? 7F9EA6380D20 ?
2025-02-21 18:21:27.110: [    CSSD][2788693760]                                                   7F9EA63803C0 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]start_thread()+209   call     clssscthrdmain()     001FB9FA0 ? 001DC83F0 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   FFFFFFFFFFFFFFFF ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   000000001 ? 7F9EA6380D20 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   7F9EA63803C0 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]clone()+109          call     start_thread()       7F9EA6381700 ? 001DC83F0 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   FFFFFFFFFFFFFFFF ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   000000001 ? 7F9EA6380D20 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   7F9EA63803C0 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]0000000000000000     call     clone()              7F9EA6381700 ? 001DC83F0 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   FFFFFFFFFFFFFFFF ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   000000001 ? 7F9EA6380D20 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760]                                                   7F9EA63803C0 ?
2025-02-21 18:21:27.111: [    CSSD][2788693760] 
2025-02-21 18:21:27.111: [    CSSD][2788693760]--------------------- Binary Stack Dump ---------------------

这里提示表决盘超时,尝试启动nocrs貌似,在表决盘存在的情况下,启动依旧失败,通过处理让启动过程不读表决盘,启动nocrs模式成功,并mount其他业务磁盘组
22
33
44

确认其他磁盘没有问题,重建crs磁盘组

SQL> create diskgroup OCR  external redundancy disk '/dev/dm-4' force  attribute 'COMPATIBLE.ASM' = '11.2.0';
# ocrconfig -restore /u01/app/11.2.0.3/grid/cdata/scan/backup00.ocr
# crsctl replace votedisk +OCR
SQL> create spfile from pfile='/tmp/pfile.asm';

然后重启crs恢复正常

发表在 Oracle RAC | 标签为 | 留下评论

手工删除19c rac

发表于 2024 年 12 月 15 日 由 惜分飞

在某些时候,需要删除掉手工删除19c RAC,重新安装,以下是比较简便的操作(root用户操作)

source /home/grid/.bash_profile
crsctl stop crs
systemctl disable oracle-ohasd
systemctl stop oracle-ohasd
systemctl disable oracle-tfa.service
systemctl stop oracle-tfa

rm -rf /etc/oracle
rm -rf /etc/ora*
rm -rf /u01
rm -rf /tmp/CVU*
rm -rf /tmp/.oracle
rm -rf /var/tmp/.oracle
rm -f /etc/init.d/init.ohasd
rm -f /etc/systemd/system/oracle-ohasd.service
rm -f /etc/systemd/system/oracle-tfa.service

dd if=/dev/zero of=/dev/asm_ocr1 bs=1024 count=1
dd if=/dev/zero of=/dev/asm_ocr2 bs=1024 count=1
dd if=/dev/zero of=/dev/asm_ocr3 bs=1024 count=1
发表在 ORACLE 19C, Oracle RAC | 评论关闭

RAC默认服务配置优先节点

发表于 2024 年 9 月 7 日 由 惜分飞

在某些rac情况下需要,需要对数据库默认的db_name对应的服务名进行修改,实现优先节点访问的效果.分析了下在默认值情况下,db_name影响到db_unique_name,然后决定了数据库的service_names.现有库的db_name无法修改,那就只能在db_unique_name上做手脚(只是修改service_names,对应的db_unique_name还是会创建默认服务,这样该服务依旧可以连接).但是在rac环境中db_unique_name记录到了crs资源之中,无法直接在数据库级别修改(修改会报ORA-32017 ORA-65500错误)

SQL> alter system set db_unique_name='nxifenfei' sid='*' scope=spfile;
alter system set db_unique_name='nxifenfei' sid='*' scope=spfile
*
ERROR at line 1:
ORA-32017: failure in updating SPFILE
ORA-65500: could not modify DB_UNIQUE_NAME, resource exists

只能先删除crs中关于db的资源,然后再进行修改服务名,再增加db资源

[oracle@xffdb1 ~]$ srvctl remove database -d xifenfei  -f

SQL> alter system set db_unique_name='nxifenfei' sid='*' scope=spfile;

[oracle@xffdb1 ~]$ srvctl add database -d nxifenfei -o /u01/app/oracle/product/19c/db_1 -p \
  +DATADG/XIFENFEI/PARAMETERFILE/spfile.271.1174153165 -pwfile +DATADG/XIFENFEI/PASSWORD/pwdxifenfei.256.1174152463
[oracle@xffdb1 ~]$ srvctl add instance -d nxifenfei -i xifenfei1 -n xffdb1
[oracle@xffdb1 ~]$ srvctl add instance -d nxifenfei -i xifenfei2 -n xffdb2
[oracle@xffdb1 ~]$ srvctl add instance -d nxifenfei -i xifenfei3 -n xffdb3

创建新服务(和db_name同名,和现在的db_unique_name不同名)

[oracle@xffdb1 ~]$ srvctl add service -db nxifenfei -service xifenfei -r xifenfei2 -a xifenfei1,xifenfei3 \
  -failovertype SESSION -failovermethod BASIC -failoverdelay 10 -failoverretry 3 -failback YES
[oracle@xffdb1 ~]$ srvctl start service -db nxifenfei -service xifenfei

[oracle@xffdb1 ~]$ srvctl config service -d nxifenfei -service xifenfei
Service name: xifenfei
Server pool:
Cardinality: 1
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Global: false
Commit Outcome: false
Failover type: SESSION
Failover method: BASIC
Failover retries: 3
Failover delay: 10
Failover restore: NONE
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Pluggable database name:
Hub service:
Maximum lag time: ANY
SQL Translation Profile:
Retention: 86400 seconds
Failback :  yes
Replay Initiation Time: 300 seconds
Drain timeout:
Stop option:
Session State Consistency: DYNAMIC
GSM Flags: 0
Service is enabled
Preferred instances: xifenfei2
Available instances: xifenfei1,xifenfei3
CSS critical: no
Service uses Java: false
[grid@xffdb1 ~]$

服务的其他操作

--调整服务的优先节点
srvctl modify service -db nxifenfei -service xifenfei -modifyconfig -preferred "xifenfei1" -available "xifenfei2,xifenfei3"
srvctl stop service -db nxifenfei -service xifenfei 
srvctl start service -db nxifenfei -service xifenfei 

--切换服务所在节点
srvctl relocate service -db nxifenfei -service xifenfei -oldinst xifenfei2 -newinst xifenfei1

--删除服务
srvctl stop service -db nxifenfei -service xifenfei
srvctl remove service -db nxifenfei -service xifenfei
发表在 Oracle RAC | 标签为 , | 评论关闭