ORA-65088: database open should be retried

在12.2以及后续的cdb版本中,如果重建ctl并且resetlogs库,很可能会遇到ORA-65088: database open should be retried错误

SQL> startup nomount force pfile='/<path>/<filename>.ora';
ORACLE instance started.

Total System Global Area 1593835520 bytes
Fixed Size                  8793256 bytes
Variable Size             402654040 bytes
Database Buffers         1174405120 bytes
Redo Buffers                7983104 bytes
SQL> !vi ctl.sql

SQL> @ctl.sql

Control file created.

SQL> select count(*) ,fhsta from x$kcvfh group by fhsta;

  COUNT(*)      FHSTA
---------- ----------
        11      32768
         4      40960

SQL> select count(*) ,FHSCN from x$kcvfh group by FHSCN;

  COUNT(*) FHSCN
---------- --------------------
         3 1820866
         4 2281969
         4 2281978
         4 2281982

SQL> select file#,error from v$datafile_header where length(error)>=1;

no rows selected

SQL> select count(*) ,fhrba_seq from x$kcvfh group by fhrba_seq;

  COUNT(*)  FHRBA_SEQ
---------- ----------
         3         20
        12         32

SQL> recover database using backup controlfile until cancel;
ORA-00279: change 2281978 generated at 09/19/2018 00:52:00 needed for thread 1
ORA-00289: suggestion : /<path>/1_32_981800889.dbf
ORA-00280: change 2281978 for thread 1 is in sequence #32


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/<path>/1_32_981800889.dbf
ORA-00279: change 2282008 generated at 09/19/2018 00:52:13 needed for thread 1
ORA-00289: suggestion : /<path>/1_33_981800889.dbf
ORA-00280: change 2282008 for thread 1 is in sequence #33
ORA-00278: log file '/<path>/1_32_981800889.dbf' no longer needed for this recovery


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
cancel
Media recovery cancelled.    << Expected message "Media recovery complete." !!
SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       MOUNTED
         3 _###_UNKNOWN_PDB_#_3           MOUNTED
         4 _###_UNKNOWN_PDB_#_4           MOUNTED
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00603: ORACLE server session terminated by fatal error
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-65088: database open should be retried
Process ID: 32688
Session ID: 10 Serial number: 38416

alert日志类似错误

 Dictionary check beginning
 Pluggable Database <pdb_name_1> (#3) found in data dictionary,
 but not in the control file. Adding it to control file.
 Pluggable Database <pdb_name_2> (#4) found in data dictionary,
 but not in the control file. Adding it to control file.
 Tablespace '<tablespace_name>' #3 found in data dictionary,
 but not in the controlfile. Adding to controlfile.
 --
 File 8 not verified due to error ORA-01122
 File 9 not verified due to error ORA-01122
 File 11 not verified due to error ORA-01122
 File 16 not verified due to error ORA-01122
 File 17 not verified due to error ORA-01122
 File 18 not verified due to error ORA-01122
 File 19 not verified due to error ORA-01122
 File 20 not verified due to error ORA-01122
  --
 ORA-65088: database open should be retried
 2018-09-19T01:00:54.083814+05:30
 Errors in file /<path>/trace/<oracle_sid>_ora_12412.trc:
 ORA-65088: database open should be retried
 Error 65088 happened during db open, shutting down database
 Errors in file /<path>/trace/<oracle_sid>_ora_12412.trc  (incident=12289) (PDBNAME=CDB$ROOT):
 ORA-00603: ORACLE server session terminated by fatal error
 ORA-01092: ORACLE instance terminated. Disconnection forced
 ORA-65088: database open should be retried

出现这类故障的原因是由于:
we see that the created controlfile is not aware of PDB and open resetlogs process trying to add information in newly created file . Hence, recovery process ,in newly created controlfile didn’t applied the archives to datafiles part of PDB which says later it will ask for recovery once controlfile is aware of PDB files During the resetlogs process, its pushing the required information to controlfile and shutting the database with suggestion to re-try opening the DB.

$ sqlplus "/as sysdba"

SQL*Plus: Release 12.2.0.1.0 Production on Wed Sep 19 01:34:01 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount force pfile='/<path>/<filename>.ora';
ORACLE instance started.

Total System Global Area 1593835520 bytes
Fixed Size                  8793256 bytes
Variable Size             402654040 bytes
Database Buffers         1174405120 bytes
Redo Buffers                7983104 bytes
SQL> alter database mount;

Database altered.

SQL> show pdbs

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       MOUNTED
         3 PDB1                           MOUNTED
         4 APDB                           MOUNTED
        
SQL> select count(*) ,FHSCN from x$kcvfh group by FHSCN;

  COUNT(*) FHSCN
---------- --------------------
         3 1820866
         4 2281969
         4 2281982
         4 2282012        
        
//* Here , we see controlfile is aware of PDB



$ sqlplus "/as sysdba"

SQL*Plus: Release 12.2.0.1.0 Production on Wed Sep 19 01:02:13 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL>  startup nomount force pfile='/<path>/<filename>.ora';
ORACLE instance started.

Total System Global Area 1593835520 bytes
Fixed Size                  8793256 bytes
Variable Size             402654040 bytes
Database Buffers         1174405120 bytes
Redo Buffers                7983104 bytes
SQL> alter database mount;

Database altered.

SQL> recover database;
ORA-00279: change 2281969 generated at 09/19/2018 00:51:35 needed for thread 1
ORA-00289: suggestion : /<path>/1_32_981800889.dbf
ORA-00280: change 2281969 for thread 1 is in sequence #32


Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
/<path>/1_32_981800889.dbf
Log applied.
Media recovery complete.
SQL> alter database open;

Database altered.

SQL>

官方的进一步解释:
We clearly see that the recovery steps applies the same archivelog file twice. When a controlfile is recreated, the recovery initiated will apply archivelog files to only the CDB datafiles, not to the PDB. Once the database open returns the ORA-65088 error, the next database re-start will apply the archivelog files to the PDB for the sake of database consistency.This should explain why Oracle is looking to apply the same archivelog sequence a second time. The following bugs report similar issues. They have both been closed as ‘not a bug’ as this is expected behavior:
BUG 24951417 – ERROR OPENING DATABASE WITH RESETLOGS AFTER CREATE CONTROLFILE
BUG 25172530 – MULTITENANT RESTORE FAILED WITH ORA-65088: DATABASE OPEN
参考:ORA-65088 while opening DB with resetlogs for multi-tenant DB in 12.2 (Doc ID 2449591.1)