11.2.0.3 adg库因 bug 16427872 导致smon占用大量cpu

《11.2.0.3 adg库因 bug 16427872 导致smon占用大量cpu》有 1 条评论

1. 惜分飞 说：

   2014 年 1 月 8 日14:49

   Bug 16427872 : SMON STUCK ON LATCH FREE ON ACTIVE DATA GUARD AFTER ADDING A DATAFILE IN PRIMARY

   Hdr: 16427872 11.2.0.3 RDBMS 11.2.0.3 SPACE PRODID-5 PORTID-226
   Abstract: SMON STUCK ON LATCH FREE ON ACTIVE DATA GUARD AFTER ADDING A DATAFILE IN PRIMARY
    
   *** 03/03/13 05:24 pm *** (ADD: Impact/Symptom->WRONG RESULT - INCO...)
   *** 03/03/13 05:24 pm ***
    
   BUG TYPE CHOSEN
   ===============
   Code
    
   == Component: RDBMS ==
   ======================
   DETAILED PROBLEM DESCRIPTION
   ============================
   The SMON process appears stuck on Latch Free on Active Data Guard after a
   datafile was added to the sysaux tablespace in the primary database.  The
   SMON is moving between these few functions: ktm_back_to_smon, ktm_set_curfun,
   ktmmon, ktmSmonMain.
    
   DIAGNOSTIC ANALYSIS
   ===================
   We obtained the pstack outputs of the smon process
       ps -ef | grep smon
       script /tmp/stack.<pid of smon>
       date
       pstack <pid of smon>
       -- wait for 15 seconds
       date
       pstack <pid of smon>
       -- wait for 15 seconds
       date
       pstack <pid of smon>
       exit
    
   In stack12124.htm we can find the pstack of smon as follow:
    
   876:skywp_nwk-p20adm-lapd04.corp.apple.com:/home/oracle/skywp/admin/backdump>
   date
   Thu Feb 28 23:22:57 GMT 2013
   877:skywp_nwk-p20adm-lapd04.corp.apple.com:/home/oracle/skywp/admin/backdump>
   pstack 12124
   #0  0x000000339f698b17 in times () from /lib64/libc.so.6
   #1  0x00000000092cde1f in sltrgatime64 ()
   #2  0x000000000107967a in ktm_set_curfun ()
   #3  0x00000000010744c3 in ktmmon ()
   #4  0x0000000001073198 in ktmSmonMain ()
   #5  0x00000000023343b3 in ksbrdp ()
   #6  0x000000000255a4ff in opirip ()
   #7  0x000000000180c839 in opidrv ()
   #8  0x0000000001e01f9b in sou2o ()
   #9  0x0000000000a0b3ea in opimai_real ()
   #10 0x0000000001e07f0c in ssthrdmain ()
   #11 0x0000000000a0b2d1 in main ()
   878:skywp_nwk-p20adm-lapd04.corp.apple.com:/home/oracle/skywp/admin/backdump>
   date
   Thu Feb 28 23:23:35 GMT 2013
   880:skywp_nwk-p20adm-lapd04.corp.apple.com:/home/oracle/skywp/admin/backdump>
   pstack 12124
   #0  0x000000339f698b17 in times () from /lib64/libc.so.6
   #1  0x00000000092cde1f in sltrgatime64 ()
   #2  0x00000000010798ef in ktm_back_to_smon ()
   #3  0x00000000010744e8 in ktmmon ()
   #4  0x0000000001073198 in ktmSmonMain ()
   #5  0x00000000023343b3 in ksbrdp ()
   #6  0x000000000255a4ff in opirip ()
   #7  0x000000000180c839 in opidrv ()
   #8  0x0000000001e01f9b in sou2o ()
   #9  0x0000000000a0b3ea in opimai_real ()
   #10 0x0000000001e07f0c in ssthrdmain ()
   #11 0x0000000000a0b2d1 in main ()
   881:skywp_nwk-p20adm-lapd04.corp.apple.com:/home/oracle/skywp/admin/backdump>
   pstack 12124
   #0  0x000000339f698b17 in times () from /lib64/libc.so.6
   #1  0x00000000092cde1f in sltrgatime64 ()
   #2  0x000000000107967a in ktm_set_curfun ()
   #3  0x0000000001074425 in ktmmon ()
   #4  0x0000000001073198 in ktmSmonMain ()
   #5  0x00000000023343b3 in ksbrdp ()
   #6  0x000000000255a4ff in opirip ()
   #7  0x000000000180c839 in opidrv ()
   #8  0x0000000001e01f9b in sou2o ()
   #9  0x0000000000a0b3ea in opimai_real ()
   #10 0x0000000001e07f0c in ssthrdmain ()
   #11 0x0000000000a0b2d1 in main ()
   883:skywp_nwk-p20adm-lapd04.corp.apple.com:/home/oracle/skywp/admin/backdump>
   exit
    
   This indicates that smon is moving between these few functions:
   ktm_back_to_smon, ktm_set_curfun, ktmmon and ktmSmonMain.  This is matched to
   the errorstack level 3.
    
   A repeated query from v$session and v$process shows:
   TIME			 SID	   SEQ# PNAME	      P1	 P2	    P3	WAIT_TIME SECONDS_IN_WAIT STATE
   	      EVENT
   ----------------- ---------- ---------- ----- ---------- ----------
   ---------- ---------- --------------- -------------------
   ----------------------------------------------------------------
   13-03-01:04:03:34	1639	   3702 SMON  1610802608	259	     0	       -1	   
   25277 WAITED SHORT TIME   latch free 
    
   TIME			 SID	   SEQ# PNAME	      P1	 P2	    P3	WAIT_TIME SECONDS_IN_WAIT STATE
   	      EVENT
   ----------------- ---------- ---------- ----- ---------- ----------
   ---------- ---------- --------------- -------------------
   ----------------------------------------------------------------
   13-03-01:04:03:22	1639	   3702 SMON  1610802608	259	     0	       -1	   
   25325 WAITED SHORT TIME   latch free 
    
   TIME			 SID	   SEQ# PNAME	      P1	 P2	    P3	WAIT_TIME SECONDS_IN_WAIT STATE
   	      EVENT
   ----------------- ---------- ---------- ----- ---------- ----------
   ---------- ---------- --------------- -------------------
   ----------------------------------------------------------------
   13-03-01:04:03:09	1639	   3702 SMON  1610802608	259	     0	       -1	   
   25433 WAITED SHORT TIME   latch free
    
   WORKAROUND?
   ===========
   No
    
   TECHNICAL IMPACT
   ================
   Standby database unusable.
    
   RELATED ISSUES (bugs, forums, RFAs)
   ===================================
   Few bugs found:
   Defect 13712579 SMON CPU SPIKES ON ADG PHYSICAL STANDBY
   Defect 13060518 ADG 11.2.0.2 - SMON TAKES 100% CPU ON STANDBY SINCE ADDING A
   DATAFILE ON PRIMARY
   Defect 16092439 CDB EXIT : SMON IN ADG AT 100% CPU IN KTM
   All of them are not closed.
    
    
   We have set the following event:
   alter session set tracefile_identifier='SMON';
   alter system set events ' 10500 level 15 ';
   After few minutes got a 1GB trace file smon.trc.gz, in it:
   ...
       0: waited for 'latch free'
           address=0x6002e5b0, number=0x103, tries=0x0
           wait_id=3701 seq_num=3702 snap_id=1
           wait times: snap=0.000069 sec, exc=0.000069 sec, total=0.000069 sec
           wait times: max=infinite
           wait counts: calls=0 os=0
           occurred after 287 min 5 sec of elapsed time
        1: waited for 'smon timer'
           sleep time=0x12c, failed=0x0, =0x0
           wait_id=3700 seq_num=3701 snap_id=1
           wait times: snap=4 min 43 sec, exc=4 min 43 sec, total=4 min 43 sec
           wait times: max=5 min 0 sec
           wait counts: calls=1 os=95
           occurred after 0.003400 sec of elapsed time
        2: waited for 'smon timer'
           sleep time=0x12c, failed=0x0, =0x0
           wait_id=3699 seq_num=3700 snap_id=1
           wait times: snap=5 min 0 sec, exc=5 min 0 sec, total=5 min 0 sec
           wait times: max=5 min 0 sec
           wait counts: calls=1 os=99
           occurred after 0.000032 sec of elapsed time
   ...
   The above looks very similar to defect 13060518 "ADG 11.2.0.2 - SMON TAKES
   100% CPU ON STANDBY SINCE ADDING A DATAFILE ON PRIMARY" (status 11).
    
   HOW OFTEN DOES THE ISSUE REPRODUCE AT CUSTOMER SITE?
   ====================================================
   Intermittent
    
   DOES THE ISSUE REPRODUCE INTERNALLY?
   ====================================
   No
    
   IS A TESTCASE AVAILABLE?
   ========================
   No
    
   Link to IPS Package:
   ====================
   "not available"