How to use RMAN incremental backup to synchronize out of sync Disaster Recovery [DR] site or Standby Database

In many ways your standby database can become out of sync with primary database. It may happen due to following reasons.

• Due to network outage or slow network
• Sudden high archive log generation in primary site etc

If unfortunately you may have deleted the archive log from primary [or like in our case 500 GB archive log needs to be transferred to DR site, which would have taken lots of time], you need to use RMAN incremental backup to sync the DR site.

The following is a step wise document. Also I tried to show the issues that we have faced during recovery of DR site.  


Step 1 
On the standby database, stop the managed recovery process (MRP) by using following command.
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

Step 2 
On the standby database, find the SCN which will be used for the incremental backup at the primary database.
SQL> SELECT CURRENT_SCN FROM V$DATABASE;

Step 3 
In the primary database RMAN, start to take an incremental backup from the SCN that we have found from previous step 2.
$ rman target /
RMAN> BACKUP INCREMENTAL FROM SCN DATABASE FORMAT '/backup/Standby_%U' tag 'STANDBY'  FILESPERSET 1;

or

spool msglog to '/home/oraprod/rman_2sep13.log'
crosscheck archivelog all;
run
{
allocate channel c1 type disk;
allocate channel c2 type disk;
BACKUP INCREMENTAL FROM SCN 5989520905001 DATABASE FORMAT '/backup/Standby_%U' tag 'FORSTANDBY' FILESPERSET 1;
RELEASE CHANNEL c1;
RELEASE CHANNEL c2;
}


Step 4 
Transfer the backup to the standby server. You may use scp or rsync command. Also you can compress the backup using "gzip" command to minimize the transfer time. Say in our case we have moved all the backups to /d01/backup/ directory

Step 5 
Connect to the standby database as the RMAN target, and catalog all incremental backup pieces.
RMAN> CATALOG START WITH '/d01/backup/Standby';


Step 6 
Recover the standby database with the incremental backup.
RMAN> RECOVER DATABASE NOREDO;

Step 7 
On the Primary database create a standby control file backup.
RMAN> BACKUP CURRENT CONTROLFILE FOR STANDBY FORMAT  '/backup/StandbyCTRL.bck';
Copy the standby control file backup to the standby system.

Step 8 
In the standby database, restore the standby control file as following way.

RMAN> SHUTDOWN;
RMAN> STARTUP NOMOUNT;

RMAN> RESTORE STANDBY CONTROLFILE FROM '/d01/backup/StandbyCTRL.bck';

RMAN> SHUTDOWN;
RMAN> STARTUP MOUNT;

Step 9 
If the primary and standby database data file directories are identical, skip to step10. 
In RMAN, connect to the standby database, catalog the standby data files, and switch the standby database to use the just cataloged data files. For example.
RMAN> CATALOG START WITH '+DATA_1/LONDAN/DATAFILE/';
RMAN> SWITCH DATABASE TO COPY;

Otherwise, on the standby database, use an OS utility or the asmcmd utility (if it is an ASM-managed database) to remove all online and standby redo logs from the standby directories and ensure that the LOG_FILE_NAME_CONVERT parameter is properly defined to translate log directory paths. For example, LOG_FILE_NAME_CONVERT=’/PARIS/’,’/LONDAN/’.

Step 10 
[Optional] On the standby database, clear all standby redo log groups.
SQL> ALTER DATABASE CLEAR LOGFILE GROUP 1;
SQL> ALTER DATABASE CLEAR LOGFILE GROUP 2;
SQL> ALTER DATABASE CLEAR LOGFILE GROUP 3;

Step 11 
[Optional] On the standby database, restart Flashback Database.
SQL> ALTER DATABASE FLASHBACK OFF;
SQL> ALTER DATABASE FLASHBACK ON;

Step 12 
On the standby database, restart MRP services.
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;

or 

restart the services with following command.

SQL>  startup nomount pfile=initLONDAN.ora;
SQL>  alter database mount standby database ;
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;


After this check whether the archive is coming and applying correctly or not using following command.
SELECT PROCESS, STATUS, THREAD#, SEQUENCE#,  BLOCK#, BLOCKS FROM V$MANAGED_STANDBY;

===========================

But, after this we faced some problem as some datafiles have been added in the primary database. So, archival process has not been started. To resolve this issue we need to take separate backup of the new datafiles and add it to the standby database using RMAN.

first backup the missing datafile using RMAN in primary database.
RMAN> backup datafile 42 FORMAT '/backup/Fordatafile_%U' tag 'FORDATAFILE' FILESPERSET 1;

then transfer this backup in standby. And then add the backup in standby using RMAN.
RMAN> CATALOG START WITH '/d01/backup/Fordatafile';
RMAN> RESTORE DATAFILE 42;


After this start recovering the standby database.

SQL> startup nomount;
SQL> recover standby database;

Once asked give the location of  asked archive log file location. You can use "AUTO" mode to apply the archive log files automatically. The log will be applied till to the point to which archives are available.

After this point we have started the automatic archive applying process using following commands and database came into sync with primary in some times. 
SQL>  startup nomount pfile=initLONDAN.ora;
SQL>  alter database mount standby database ;
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;