PostgreSQL恢复工具—pdu工具介绍

张晨同学开发了一个PostgreSQL数据恢复工具PDU(PDU: Postgresql Data Rescue Tool),我这边配合做一些测试
使用帮助命令

PDU.public=# ;

PDU数据拯救工具 | 命令帮助
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
**基础操作**
b;                                      │ 初始化数据库元信息
exit;                                   │ 退出工具

**数据库切换**
use <db>;                               │ 指定目标数据库 (例: use logs;)
set <schema>;                           │ 指定操作模式 (例: set recovery;)

**元数据展示**
\l;                                     │ 列出所有数据库
\dn;                                    │ 显示当前数据库模式
\dt;                                    │ 列出当前模式下的表
\d+ <table>;                            │ 查看表结构详情 (例: \d+ users;)
\d <table>;                             │ 查看表列类型 (例: \d users;)

**数据导出**
unload <table>;                         │ 导出表数据 → ./<表名>.csv (例: unload orders;)
unload SCH;                             │ 导出当前模式所有数据
unload DDL;                             │ 生成模式结构定义文件
unload COPY;                            │ 生成PSQL COPY语句脚本

**误删数据恢复**
scan t1;                                │ 扫描被误删的表
restore del <Tx Number>;                │ 通过事务号恢复被误删的数据
--------------------------------------------------------------------------------------
scan drop;                              │ 扫描wal日志中的drop事务
restore drop <Tx Number>;               │ 通过事务号恢复被drop的表
add <oid> <tablename> <attibutes>;      │ 将表信息手动添加到restore库中
例如: <add 12345 t1 varchar,varchar,timestamp,varchar,numeric,varchar,varchar,varchar,numeric;>

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
语法规则
◈ 指令后缀必须带 `;`

加载PostgreSQL元数据

PDU.public=# b;
开始初始化...
 -pg_database:</var/lib/pgsql/12/data/global/1262>
    【postgres】
      -pg_schema:</var/lib/pgsql/12/data/base/14399/2615>
      -pg_class:</var/lib/pgsql/12/data/base/14399/1259>,共55行
      -pg_attribute:</var/lib/pgsql/12/data/base/14399/1249>,共2913行
      模式:
        -->public,2张表
    【his5_dms】
      -pg_schema:</var/lib/pgsql/12/data/base/16386/2615>
      -pg_class:</var/lib/pgsql/12/data/base/16386/1259>,共793行
      -pg_attribute:</var/lib/pgsql/12/data/base/16386/1249>,共31329行
      模式:
        -->public,660张表
        -->hiscrm,55张表
        -->pgagent,8张表
        -->report,7张表
        -->statistics,10张表

查看当前有哪些库

PDU.public=# \l;
|------------------|
|     数据库名     |
|------------------|
|    postgres      |
|    template1     |
|    template0     |
|    his5_dms      |
|    restore       |
|------------------|

      5 rows selected

进入某个库

PDU.public=# use his5_dms;
|----------------------------------------|
|          模式             |  表数量    |
|----------------------------------------|
|    public                 |  660       |
|    hiscrm                 |  55        |
|    pgagent                |  8         |
|    report                 |  7         |
|    statistics             |  10        |
|----------------------------------------|

进入某个模式

his5_dms.public=# set hiscrm;
|--------------------------------------------------|
|               表名                  |  表大小    |
|--------------------------------------------------|
|    t_patient_other                  |  600.00 KB |
|    t_sys_oper_log                   |  88.00 KB  |
|    t_auth                           |  88.00 KB  |
|    t_setting_user                   |  56.00 KB  |
|    t_field_define                   |  32.00 KB  |
|    t_oper_log                       |  16.00 KB  |
|    t_role                           |  16.00 KB  |
|    t_sys_login_log                  |  8.00 KB   |
|    t_appointment_item               |  8.00 KB   |
|    t_clinic                         |  8.00 KB   |
|    t_dept                           |  8.00 KB   |
|    t_employee                       |  8.00 KB   |
|    t_menu                           |  8.00 KB   |
|    t_patient_label                  |  8.00 KB   |
|    t_patient_label_detail_tpl       |  8.00 KB   |
|    t_patient_source_ref             |  8.00 KB   |
|    t_return_visit_tpl               |  8.00 KB   |
|    t_setting_clinic                 |  8.00 KB   |
|    t_setting_notify                 |  8.00 KB   |
|    t_sms_template_category          |  8.00 KB   |
|--------------------------------------------------|

        仅显示表大小排名前 50 的表名

显示部分表

his5_dms.hiscrm=# \dt;
|--------------------------------------------------|
|               表名                  |  表大小    |
|--------------------------------------------------|
|    t_patient_other                  |  600.00 KB |
|    t_sys_oper_log                   |  88.00 KB  |
|    t_auth                           |  88.00 KB  |
|    t_setting_user                   |  56.00 KB  |
|    t_field_define                   |  32.00 KB  |
|    t_oper_log                       |  16.00 KB  |
|    t_role                           |  16.00 KB  |
|    t_sys_login_log                  |  8.00 KB   |
|    t_appointment_item               |  8.00 KB   |
|    t_clinic                         |  8.00 KB   |
|    t_dept                           |  8.00 KB   |
|    t_employee                       |  8.00 KB   |
|    t_menu                           |  8.00 KB   |
|    t_patient_label                  |  8.00 KB   |
|    t_patient_label_detail_tpl       |  8.00 KB   |
|    t_patient_source_ref             |  8.00 KB   |
|    t_return_visit_tpl               |  8.00 KB   |
|    t_setting_clinic                 |  8.00 KB   |
|    t_setting_notify                 |  8.00 KB   |
|    t_sms_template_category          |  8.00 KB   |
|--------------------------------------------------|

        仅显示表大小排名前 50 的表名

显示某个表的信息

his5_dms.hiscrm=# \d+ t_auth;
----------------------------------------------------------------
|                            建表语句                           |
----------------------------------------------------------------
   CREATE TABLE t_auth (
        id                       bigint,
        clinic_id                bigint,
        group_id                 bigint,
        parient_id               varchar(64),
        menu_id                  varchar(64),
        auth_key                 varchar(60),
        auth_name                varchar(64),
        uris                     varchar,
        rely                     varchar(255),
        state                    bigint,
        sort                     bigint,
        tag                      bigint,
        explain                  varchar(255),
        desc                     varchar(255)
   );
----------------------------------------------------------------
|                                                              |
----------------------------------------------------------------
his5_dms.hiscrm=# \d t_auth;
----------------------------------------------------------------
|                            列类型                             |
----------------------------------------------------------------
bigint,bigint,bigint,varchar,varchar,varchar,varchar,varchar,varchar,bigint,bigint,bigint,varchar,varchar

恢复表数据

his5_dms.hiscrm=# unload t_auth;
正在解析表 <t_auth>. 已解析数据页: 11, 已解析数据: 492 条
<t_auth>-</var/lib/pgsql/12/data/base/16386/16895> 解析完成, 12 个数据页 ,共计 492 条数据. 成功 492 条; 失败【0】条 
 COPY文件路径为:<his5_dms/hiscrm/t_auth.csv>

确认恢复表的数据情况

[root@xifenfeidg hiscrm]# wc -l t_auth.csv
492 t_auth.csv

QQ20250228-224837


PostgreSQL中查询表实际数据情况

his5_dms=# select count(1) from hiscrm.t_auth;
 count 
-------
   492
(1 row)
is5_dms=# \d hiscrm.t_auth;
                                Table "hiscrm.t_auth"
   Column   |          Type          | Collation | Nullable |         Default         
------------+------------------------+-----------+----------+-------------------------
 id         | bigint                 |           | not null | 
 clinic_id  | bigint                 |           |          | 
 group_id   | bigint                 |           |          | 
 parient_id | character varying(64)  |           | not null | 
 menu_id    | character varying(64)  |           | not null | 
 auth_key   | character varying(60)  |           | not null | 
 auth_name  | character varying(64)  |           | not null | 
 uris       | text                   |           | not null | 
 rely       | character varying(255) |           |          | NULL::character varying
 state      | bigint                 |           | not null | 
 sort       | bigint                 |           | not null | 
 tag        | bigint                 |           |          | '0'::bigint
 explain    | character varying(255) |           |          | NULL::character varying
 desc       | character varying(255) |           |          | NULL::character varying

truncate表

his5_dms=# truncate table hiscrm.t_auth;
TRUNCATE TABLE
his5_dms=# select count(1) from hiscrm.t_auth;
 count 
-------
     0
(1 row)

导入数据,并验证恢复效果

his5_dms=# \i /tmp/pdu/his5_dms/COPY/hiscrm_copy.sql 
SET
COPY 492
his5_dms=# select count(1) from  hiscrm.t_auth;
 count 
-------
   492
(1 row)

通过验证pdu可以在数据库离线的情况下,恢复PostgreSQL数据库中表的数据,更加方便和灵活的实现替代pg_filedump功能,而且pdu还在不断完善和新功能更新中

发表在 PostgreSQL恢复 | 标签为 , , , , | 留下评论

近1万个数据文件的恢复case

朋友介绍一个恢复case,数据库发生过硬件故障,做过硬件恢复之后,数据库无法正常启动.我恢复的已经不是第一现场,客户和我反馈说找过三批人进行恢复,都没有正常打开数据库.数据库整体不大(1T左右),但是数据文件近1万个(9895个数据文件),我看了下alert日志,主要报错有:
ORA-600 kcratr_scan_lastbwr,该错误比较常见,一般是由于坏块或者redo和数据文件不匹配导致,在某些情况下recover下就可以解决,有些时候不行,看人品

Mon Feb 17 15:51:15 2025
Started redo scan
Hex dump of (file 3, block 240) in trace file F:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_10508.trc
Reading datafile 'F:\ORACLEDATA\ORCL\UNDOTBS01.DBF' for corruption at rdba: 0x00c000f0 (file 3, block 240)
Reread (file 3, block 240) found same corrupt data (logically corrupt)
Write verification failed for File 3 Block 240 (rdba 0xc000f0)
Errors in file F:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_10508.trc  (incident=293029):
ORA-00600: 内部错误代码, 参数: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
Incident details in: F:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\incident\incdir_293029\orcl_ora_10508_i293029.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Aborting crash recovery due to error 600
Errors in file F:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_10508.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []
Mon Feb 17 15:51:22 2025
Sweep [inc2][293029]: completed
Mon Feb 17 15:51:25 2025
Errors in file F:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_10508.trc:
ORA-00600: 内部错误代码, 参数: [kcratr_scan_lastbwr], [], [], [], [], [], [], [], [], [], [], []

ORA-600 krr_parse_3错误,官方没有查询到资料,但是从报错的位置分析,应该和redo的应用有直接关系

Thu Feb 20 11:45:03 2025
ALTER DATABASE RECOVER  datafile 116  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2084282 Reading mem 0
  Mem# 0: F:\ORACLEDATA\ORCL\REDO02.LOG
Errors in file F:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_10840.trc  (incident=321616):
ORA-00600: 内部错误代码, 参数: [krr_parse_3], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Media Recovery failed with error 600
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 116  ...
ALTER DATABASE RECOVER  datafile 1168  
Media Recovery Start
Serial Media Recovery started
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2084282 Reading mem 0
  Mem# 0: F:\ORACLEDATA\ORCL\REDO02.LOG
Errors in file F:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_ora_10840.trc  (incident=321617):
ORA-00600: 内部错误代码, 参数: [krr_parse_3], [], [], [], [], [], [], [], [], [], [], []
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Media Recovery failed with error 600
ORA-283 signalled during: ALTER DATABASE RECOVER  datafile 1168  ...

上述两个错误,由于数据库部分文件被offline,而且屏蔽一致性打开等操作,绕过了上述的两个ORA-600错误,现在停留在ORA-00604 ORA-00376 ORA-01110故障导致数据库无法打开的情况,该错误是由于数据库启动过程中有事务,需要使用被offline的undo文件.

Fri Feb 21 07:42:37 2025
minact-scn: got error during useg scan e:376 usn:1
minact-scn: useg scan erroring out with error e:376
Fri Feb 21 07:44:02 2025
Errors in file F:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_m007_11464.trc:
ORA-51106: 由于出错, 检查无法完成。请查看下面的错误
ORA-48223: 已请求中断 - 提取已中止 - 返回代码 [12751] [HM_FINDING]
Fri Feb 21 07:45:12 2025
Errors in file F:\APP\ADMINISTRATOR\diag\rdbms\orcl\orcl\trace\orcl_smon_14108.trc:
ORA-00604: 递归 SQL 级别 1 出现错误
ORA-00376: 此时无法读取文件 3
ORA-01110: 数据文件 3: 'F:\ORACLEDATA\ORCL\UNDOTBS01.DBF'

分析数据库文件状态,有25个数据文件被offline,而且这些文件的resetlogs信息均不对(截取了部分文件)

SQL> set lines 150
SQL> set numw 16
SQL> col CHECKPOINT_TIME for a40
SQL> set lines 150
SQL> set pages 1000
SQL> SELECT status,
  2  to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss') checkpoint_time,FUZZY,checkpoint_change#,
  3  count(*) ROW_NUM
  4  FROM v$datafile_header
  5  GROUP BY status, checkpoint_change#, to_char(checkpoint_time,'yyyy-mm-dd hh24:mi:ss'),fuzzy
  6  ORDER BY status, checkpoint_change#, checkpoint_time;

STATUS         CHECKPOINT_TIME                          FUZZY  CHECKPOINT_CHANGE#          ROW_NUM
-------------- ---------------------------------------- ------ ------------------ ----------------
OFFLINE        2025-02-11 15:27:00                      YES            1909526545               22
OFFLINE        2025-02-17 17:24:14                      YES            1909551234                2
OFFLINE        2025-02-17 17:27:35                      NO             1909551234                1
ONLINE         2025-02-22 17:29:25                      YES            2095190672             9869

wrong-resetlogs
offline
对于这种情况,最简单的解决方法就是使用开发的小工具Oracle Recovery Tools(Oracle Recovery Tools工具一键解决ORA-00376 ORA-01110故障(文件offline)),对这些offline的文件头信息进行修改
ora-tool
对于这类缺少归档数据文件offline的故障Oracle Recovery Tools可以快速傻瓜式恢复
尝试直接open数据库

SQL> STARTUP MOUNT PFILE='D:/PFILE.TXT'
ORACLE 例程已经启动。

Total System Global Area      82309009408 bytes
Fixed Size                        2290160 bytes
Variable Size                 12884905488 bytes
Database Buffers              69256347648 bytes
Redo Buffers                    165466112 bytes
数据库装载完毕。
SQL> RECOVER DATAFILE 3;
完成介质恢复。
SQL> RECOVER datafile 6601,7043,7044,7045,7050,
   7053,7054,7055,7056,7059,7060,7061,7062,7063,7064,7071,7072,7187
  ,7188,7190,7191,7192,7244,9501 ;
完成介质恢复。
SQL> alter database datafile 3,6601,7043,7044,7045,7050,
  7053,7054,7055,7056,7059,7060,7061,7062,7063,7064,7071,7072,7187
 ,7188,7190,7191,7192,7244,9501 online;
SQL> ALTER DATABASE OPEN;

数据库已更改。

QQ20250222-184800

Sat Feb 22 18:38:26 2025
alter database mount exclusive
Sat Feb 22 18:38:26 2025
MMNL started with pid=25, OS id=7524 
Successful mount of redo thread 1, with mount id 3367723362
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: alter database mount exclusive
alter database open
Sat Feb 22 18:42:34 2025
Thread 1 opened at log sequence 5
  Current log# 2 seq# 5 mem# 0: F:\ORACLEDATA\ORCL\REDO02.LOG
Successful open of redo thread 1
Sat Feb 22 18:42:34 2025
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Sat Feb 22 18:42:34 2025
SMON: enabling cache recovery
[7960] Successfully onlined Undo Tablespace 12273.
Undo initialization finished serial:0 start:98760972 end:98761612 diff:640 (6 seconds)
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is AL32UTF8
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Sat Feb 22 18:42:41 2025
QMNC started with pid=29, OS id=8116 
Sat Feb 22 18:42:45 2025
Completed: alter database open
Sat Feb 22 18:42:47 2025
Starting background process CJQ0
Sat Feb 22 18:42:47 2025
CJQ0 started with pid=31, OS id=3264 
Sat Feb 22 18:42:47 2025
db_recovery_file_dest_size of 4977 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.

数据库已经open,后续收尾工作比较简单,不再累赘.
对于这类缺少归档数据文件offline的故障Oracle Recovery Tools可以快速傻瓜式恢复,还是比较方便的
软件下载:OraRecovery下载
使用说明:使用说明

发表在 非常规恢复 | 标签为 , , , , | 留下评论

不当使用_allow_resetlogs_corruption参数引起ORA-600 2662错误

有一个数据库由于机房掉电,导致数据库无法启动,由于不当恢复导致open过程报ORA-600 2662等错误客户无法自行解决,让我们协助.幸运的是客户对现场做了备份,我接手恢复之后,让客户还原现场,然后进行恢复,比较顺利的直接open数据库,实现数据0丢失原库直接可用,避免了一次因为不当操作而引起少量数据丢失的风险,和业务的快速恢复(避免的因为强制拉库引起的不一致性问题而要做数据库逻辑迁移).通过日志回顾第一次现场恢复经历
最初故障数据库mount报错

SQL> startup mount pfile='d:/pfile.txt'
ORACLE 例程已经启动。

Total System Global Area 1071333376 bytes
Fixed Size                  1375792 bytes
Variable Size             754975184 bytes
Database Buffers          310378496 bytes
Redo Buffers                4603904 bytes
ORA-03113: 通信通道的文件结尾
进程 ID: 964
会话 ID: 1145 序列号: 1

无法mount,大部分情况是由于控制文件损坏,然后客户选择了重建controlfile

Thu Feb 20 10:20:45 2025
Successful mount of redo thread 1, with mount id 1721384698
Completed: CREATE CONTROLFILE REUSE DATABASE "ORCL" RESETLOGS ARCHIVELOG
MAXLOGFILES 32
MAXLOGMEMBERS 2
MAXDATAFILES 254
MAXINSTANCES 1
MAXLOGHISTORY 226
LOGFILE
GROUP 1 'D:\app\Administrator\oradata\orcl\redo01.log' size 50M,
GROUP 2 'D:\app\Administrator\oradata\orcl\redo02.log' size 50M,
GROUP 3 'D:\app\Administrator\oradata\orcl\redo03.log' size 50M
DATAFILE
'D:\app\Administrator\oradata\orcl\XFF.DBF',
'D:\app\Administrator\oradata\orcl\XIFENFEI.DBF',
'D:\app\Administrator\oradata\orcl\SYSAUX01.DBF',
'D:\app\Administrator\oradata\orcl\SYSTEM01.DBF',
'D:\app\Administrator\oradata\orcl\UNDOTBS01.DBF',
'D:\app\Administrator\oradata\orcl\USERS01.DBF'
CHARACTER SET ZHS16GBK

重建ctl使用了resetlogs模式,然后下一步客户进行恢复使用命令为:ALTER DATABASE RECOVER DATABASE UNTIL CANCEL USING BACKUP CONTROLFILE

Thu Feb 20 10:22:05 2025
ALTER DATABASE RECOVER  DATABASE UNTIL CANCEL USING BACKUP CONTROLFILE  
Media Recovery Start
 started logmerger process
Thu Feb 20 10:22:05 2025
WARNING! Recovering data file 1 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 2 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 3 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 4 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 5 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
WARNING! Recovering data file 6 from a fuzzy file. If not the current file
it might be an online backup taken without entering the begin backup command.
Parallel Media Recovery started with 4 slaves
ORA-279 signalled during: ALTER DATABASE RECOVER  DATABASE UNTIL CANCEL USING BACKUP CONTROLFILE  ...
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log D:\APP\ADMINISTRATOR\FLASH_RECOVERY_AREA\ORCL\ARCHIVELOG\2025_02_20\O1_MF_1_9087_%U_.ARC
Errors with log D:\APP\ADMINISTRATOR\FLASH_RECOVERY_AREA\ORCL\ARCHIVELOG\2025_02_20\O1_MF_1_9087_%U_.ARC
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr00_3044.trc:
ORA-00308: cannot open archived log 
        'D:\APP\ADMINISTRATOR\FLASH_RECOVERY_AREA\ORCL\ARCHIVELOG\2025_02_20\O1_MF_1_9087_%U_.ARC'
ORA-27041: unable to open file
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件。
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER    CONTINUE DEFAULT  
Media Recovery Log D:\APP\ADMINISTRATOR\FLASH_RECOVERY_AREA\ORCL\ARCHIVELOG\2025_02_20\O1_MF_1_9087_%U_.ARC
Errors with log D:\APP\ADMINISTRATOR\FLASH_RECOVERY_AREA\ORCL\ARCHIVELOG\2025_02_20\O1_MF_1_9087_%U_.ARC
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr00_3044.trc:
ORA-00308: cannot open archived log 
        'D:\APP\ADMINISTRATOR\FLASH_RECOVERY_AREA\ORCL\ARCHIVELOG\2025_02_20\O1_MF_1_9087_%U_.ARC'
ORA-27041: unable to open file
OSD-04002: 无法打开文件
O/S-Error: (OS 2) 系统找不到指定的文件。
ORA-308 signalled during: ALTER DATABASE RECOVER    CONTINUE DEFAULT  ...
ALTER DATABASE RECOVER CANCEL 
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_pr00_3044.trc:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: 'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF'
ORA-10879 signalled during: ALTER DATABASE RECOVER CANCEL ...

直接提示需要找归档日志seq为9087的,但是由于该库为非归档模式,客户直接输入了auto,无法找到对应的日志.然后尝试直接resetlogs打开库

Thu Feb 20 10:22:58 2025
ALTER DATABASE OPEN RESETLOGS
ORA-1194 signalled during: ALTER DATABASE OPEN RESETLOGS...
ALTER DATABASE OPEN
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_4836.trc:
ORA-01589: ??????????? RESETLOGS ? NORESETLOGS ??
ORA-1589 signalled during: ALTER DATABASE OPEN...
ALTER DATABASE OPEN RESETLOGS
ORA-1194 signalled during: ALTER DATABASE OPEN RESETLOGS...

由于数据文件不一致(前面recover没有成功),导致直接reseltogs方式打开库失败,然后设置一些参数

Thu Feb 20 10:27:24 2025
ALTER SYSTEM SET _allow_error_simulation=TRUE SCOPE=SPFILE;
ALTER SYSTEM SET _allow_terminal_recovery_corruption=TRUE SCOPE=SPFILE;
Thu Feb 20 10:27:38 2025
ALTER SYSTEM SET _allow_resetlogs_corruption=TRUE SCOPE=SPFILE;

重启库之后强制拉库

Completed: alter database mount
Thu Feb 20 10:29:08 2025
alter database open resetlogs
RESETLOGS is being done without consistancy checks. This may result
in a corrupted database. The database should be recreated.
Thu Feb 20 10:29:19 2025
Archived Log entry 1 added for thread 1 sequence 9088 ID 0x5d904d0a dest 1:
Archived Log entry 2 added for thread 1 sequence 9086 ID 0x5d904d0a dest 1:
Thu Feb 20 10:29:31 2025
Archived Log entry 3 added for thread 1 sequence 9087 ID 0x5d904d0a dest 1:
RESETLOGS after incomplete recovery UNTIL CHANGE 223770120
Thu Feb 20 10:29:38 2025
Setting recovery target incarnation to 2
Thu Feb 20 10:29:39 2025
Assigning activation ID 1721340650 (0x669992ea)
LGWR: STARTING ARCH PROCESSES
Thu Feb 20 10:29:39 2025
ARC0 started with pid=20, OS id=2924 
ARC0: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Thu Feb 20 10:29:40 2025
ARC1 started with pid=21, OS id=1832 
Thu Feb 20 10:29:40 2025
ARC2 started with pid=22, OS id=3668 
ARC1: Archival started
Thu Feb 20 10:29:40 2025
ARC3 started with pid=23, OS id=5104 
ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Successful open of redo thread 1
Thu Feb 20 10:29:41 2025
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Thu Feb 20 10:29:41 2025
SMON: enabling cache recovery
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_1652.trc  (incident=1006385):
ORA-00600: ??????, ??: [2662], [0], [223770128], [0], [223811777], [12583040], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_1006385\orcl_ora_1652_i1006385.trc
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_1652.trc:
ORA-00600: ??????, ??: [2662], [0], [223770128], [0], [223811777], [12583040], [], [], [], [], [], []
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_1652.trc:
ORA-00600: ??????, ??: [2662], [0], [223770128], [0], [223811777], [12583040], [], [], [], [], [], []
Error 600 happened during db open, shutting down database
USER (ospid: 1652): terminating the instance due to error 600
Instance terminated by USER, pid = 1652

数据库报ORA-600 2662错误,导致强制拉库失败.
后面客户进行了一系列折腾,导致出现其他错误,比如:
ORA-01595/ORA-600 4194

Fri Feb 21 04:52:19 2025
Trace dumping is performing id=[cdmp_20250221045219]
Doing block recovery for file 3 block 472
Resuming block recovery (PMON) for file 3 block 472
Block recovery from logseq 2, block 51 to scn 223837038
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
Block recovery stopped at EOT rba 2.54.16
Block recovery completed at rba 2.54.16, scn 0.223837035
Doing block recovery for file 3 block 144
Resuming block recovery (PMON) for file 3 block 144
Block recovery from logseq 2, block 51 to scn 223837033
Recovery of Online Redo Log: Thread 1 Group 2 Seq 2 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
Block recovery completed at rba 2.52.16, scn 0.223837034
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_smon_2124.trc:
ORA-01595: error freeing extent (3) of rollback segment (2))
ORA-00600: internal error code, arguments: [4194], [], [                                      

ORA-600 2256/ORA-600 4194

Fri Feb 21 05:02:43 2025
SMON: enabling cache recovery
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_2184.trc  (incident=1134285):
ORA-00600: 内部错误代码, 参数: [2256], [0], [1073741824], [0], [1073761870], [], [], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_1134285\orcl_ora_2184_i1134285.trc
Successfully onlined Undo Tablespace 2.
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is ZHS16GBK
No Resource Manager plan active
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_smon_648.trc  (incident=1134237):
ORA-00600: internal error code, arguments: [4194], [], [
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_1134237\orcl_smon_648_i1134237.trc
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_ora_2184.trc  (incident=1134286):
ORA-00600: 内部错误代码, 参数: [4194], [0], [
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_1134286\orcl_ora_2184_i1134286.trc

ORA-600 3712

Fri Feb 21 05:24:51 2025
Assigning activation ID 1721440698 (0x669b19ba)
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_lgwr_3620.trc  (incident=1150206):
ORA-00600: internal error code, arguments: [3712], [1], [1], [3584], [3], [3584], [1], [], [], [], [], []
Incident details in: d:\app\administrator\diag\rdbms\orcl\orcl\incident\incdir_1150206\orcl_lgwr_3620_i1150206.trc
Errors in file d:\app\administrator\diag\rdbms\orcl\orcl\trace\orcl_lgwr_3620.trc:
ORA-00600: internal error code, arguments: [3712], [1], [1], [3584], [3], [3584], [1], [], [], [], [], []
LGWR (ospid: 3620): terminating the instance due to error 470

基于客户本身库不大,而且在操作之前备份了现场,然后客户还原故障现场备份,我开始接手恢复
重建控制文件

Fri Feb 21 21:57:19 2025
Successful mount of redo thread 1, with mount id 1721495486
Completed: CREATE CONTROLFILE REUSE DATABASE "ORCL" NORESETLOGS  NOARCHIVELOG
       MAXLOGFILES 50
       MAXLOGMEMBERS 5
       MAXDATAFILES 5000
       MAXINSTANCES 8
       MAXLOGHISTORY 2920
LOGFILE
       group 1   'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG' size 50M,
       group 3   'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG' size 50M,
       group 2   'D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG' size 50M
DATAFILE
        'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSTEM01.DBF',
        'D:\APP\ADMINISTRATOR\ORADATA\ORCL\SYSAUX01.DBF',
        'D:\APP\ADMINISTRATOR\ORADATA\ORCL\UNDOTBS01.DBF',
        'D:\APP\ADMINISTRATOR\ORADATA\ORCL\USERS01.DBF',
        'D:\APP\ADMINISTRATOR\ORADATA\ORCL\XIFENFEI.DBF',
        'D:\APP\ADMINISTRATOR\ORADATA\ORCL\XFF.DBF'
CHARACTER SET  ZHS16GBK 

这里重建控制文件使用的是noresetlogs模式,然后尝试recover database

ALTER DATABASE RECOVER  database  
Media Recovery Start
 started logmerger process
Parallel Media Recovery started with 4 slaves
Fri Feb 21 21:57:25 2025
Recovery of Online Redo Log: Thread 1 Group 3 Seq 9087 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO03.LOG
Recovery of Online Redo Log: Thread 1 Group 1 Seq 9088 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Fri Feb 21 21:57:30 2025
Completed: ALTER DATABASE RECOVER  database  

数据库直接recover成功,然后尝试正常open库

Fri Feb 21 21:58:17 2025
alter database open
Beginning crash recovery of 1 threads
 parallel recovery started with 3 processes
Started redo scan
Completed redo scan
 read 12019 KB redo, 0 data blocks need recovery
Started redo application at
 Thread 1: logseq 9088, block 2, scn 223805802
Recovery of Online Redo Log: Thread 1 Group 1 Seq 9088 Reading mem 0
  Mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO01.LOG
Completed redo application of 0.00MB
Completed crash recovery at
 Thread 1: logseq 9088, block 24040, scn 223836931
 0 data blocks read, 0 data blocks written, 12019 redo k-bytes read
Fri Feb 21 21:58:17 2025
Thread 1 advanced to log sequence 9089 (thread open)
Thread 1 opened at log sequence 9089
  Current log# 2 seq# 9089 mem# 0: D:\APP\ADMINISTRATOR\ORADATA\ORCL\REDO02.LOG
Successful open of redo thread 1
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Feb 21 21:58:17 2025
SMON: enabling cache recovery
Dictionary check beginning
Tablespace 'TEMP' #3 found in data dictionary,
but not in the controlfile. Adding to controlfile.
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
*********************************************************************
WARNING: The following temporary tablespaces contain no files.
         This condition can occur when a backup controlfile has
         been restored.  It may be necessary to add files to these
         tablespaces.  That can be done using the SQL statement:
 
         ALTER TABLESPACE <tablespace_name> ADD TEMPFILE
 
         Alternatively, if these temporary tablespaces are no longer
         needed, then they can be dropped.
           Empty temporary tablespace: TEMP
*********************************************************************
Database Characterset is ZHS16GBK
No Resource Manager plan active
**********************************************************
WARNING: Files may exists in db_recovery_file_dest
that are not known to the database. Use the RMAN command
CATALOG RECOVERY AREA to re-catalog any such files.
If files cannot be cataloged, then manually delete them
using OS command.
One of the following events caused this:
1. A backup controlfile was restored.
2. A standby controlfile was restored.
3. The controlfile was re-created.
4. db_recovery_file_dest had previously been enabled and
   then disabled.
**********************************************************
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Fri Feb 21 21:58:20 2025
QMNC started with pid=22, OS id=524 
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Completed: alter database open

数据库完美打开,然后增加tempfile,检查/导出数据均没有任何问题,业务可以直接使用本库,不用逻辑迁移
其实这是一个比较小的故障,由于断电导致控制文件损坏,然后客户重建控制文件使用resetlogs方式,然后recover没有正确指定redo,来完成数据库实例恢复,直接使用_allow_resetlogs_corruption参数强制拉库,然后出现了ORA-600 2662/ORA-600 4194/ORA-600 2256等大家熟悉的错误.这个库不大,也做了故障现场保护,如果是一个10T以上大库,发生这样的故障,后果还是比较麻烦的(库可以打开,但是redo中的数据肯定丢失导致数据库不一致,后续可能有很多不一致性问题,可能涉及逻辑迁移),增加企业业务不可用时间成本和业务数据丢失风险,再次提醒对于Oracle隐含参数,还是需要慎重,做好专业的评估再使用.

发表在 Oracle备份恢复 | 标签为 , , | 留下评论