Thursday, April 10, 2014

Registering database with Recovery catalog

In order for oracle RMAN catalog to hold the backup information we need to register the database with recovery catalog and that can be done using the steps below:

  1. Create tns entry under tnsnames.ora on database server for which need registration.
  2. connect to catalog using catalog owner from database server.
  3. register database;
Step 1:
Added the entry below on the database server

RCAT =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = rcat1)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = RCAT1.localdomain)
    )
  )
Step 2:
[oracle@dbs1 ~]$ rman target /
Recovery Manager: Release 11.2.0 - Production on
Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.
connected to target database: ORCL (DBID=1372201017)
RMAN> connect catalog catowner@rcat
recovery catalog database Password:
connected to recovery catalog database
RMAN> 
Step 3:
RMAN> register database;
database registered in recovery catalog
starting full resync of recovery catalog
full resync complete
RMAN>




Thursday, April 3, 2014

Check status of FRA

FRA usage can be checked from V$FLASH_RECOVERY_AREA_USAGE

SQL> SELECT * FROM V$FLASH_RECOVERY_AREA_USAGE;

FILE_TYPE            PERCENT_SPACE_USED PERCENT_SPACE_RECLAIMABLE NUMBER_OF_FILES
-------------------- ------------------ ------------------------- ---------------
CONTROL FILE                          0                         0               0
REDO LOG                           2.49                         0               6
ARCHIVED LOG                      45.66                     45.31             171
BACKUP PIECE                      25.29                         0              11
IMAGE COPY                            0                         0               0
FLASHBACK LOG                      4.98                         0              12
FOREIGN ARCHIVED LOG                  0                         0               0
7 rows selected.
SQL>

Cluster voting disk and ocr

The location of voting disk can be found using command below:
[root@rac1 ~]# crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   972fdbaaabbf4f4fbfa9a8a2b96fedf7 (ORCL:ASM1) [DATA]
Located 1 voting disk(s).
[root@rac1 ~]# 
OCR file location:
[root@rac1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       3244
         Available space (kbytes) :     258876
         ID                       :  488817485
         Device/File Name         :      +DATA
                                    Device/File integrity check succeeded
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
         Cluster registry integrity check succeeded
         Logical corruption check succeeded
[root@rac1 ~]# 
Location of ocr backup
[root@rac1 ~]# ocrconfig -showbackup
rac1     2014/04/03 12:23:31     /u01/app/11.2.0/grid/cdata/scan/backup00.ocr
rac1     2014/04/03 08:23:31     /u01/app/11.2.0/grid/cdata/scan/backup01.ocr
rac1     2014/04/03 04:23:29     /u01/app/11.2.0/grid/cdata/scan/backup02.ocr
rac1     2014/04/02 02:41:12     /u01/app/11.2.0/grid/cdata/scan/day.ocr
rac1     2014/03/26 02:40:09     /u01/app/11.2.0/grid/cdata/scan/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available
[root@rac1 ~]# 

Oracle Voting Disk Explained

What is oracle voting disk and why we use it and how nodes use it ...

There can be a lot of scenarios for oracle RAC Nodes and Voting disks, but I am trying to explain the logic behind the voting disk and its use.

In simple terms Oracle voting disk is a file which hold at least two types of information. One if node can see its instance running, second if node can see other instances running.

Every second the node will send out an heart beat to other nodes and voting disk to check if they can see them and report in voting disk. Lets divide this into two main scenarios one where interconnect breaks down for a node and second where node can not write to voting disk.

Scenario 1 understanding interconnect breakdown scenario. Image 1 shows when all is good

Image 1
Image 1
In image 1 each node can see each other node and there are no issues. let's see what happens if interconnect for node 2 breaks as shown in image 2
Image 2
In case of image 2 because of break down of interconnect for node 2 now there are two cluster groups. One with node2 and other with node1, node3, node4. This situation is called split brain. In this case both sub cluster can work independently and update same block to cause corruption.

To avoid this oracle's two cluster with highest number of nodes will elect a master and get hold of control file then the newly elected master will see voting disk table to see which node should be poisoned and will put poison bit for that node and will evict that node from, cluster. In this case it will be node 2 which will be evicted.

Scenario 2 where nodes break communication with voting disks.

Suppose we are using more than one voting disk like three in example blow thinks are going smooth
Image 3
Now if because of some load or any other reason node 3 is not able to write to voting disk 1 and 2, please note that interconnect is all good here.
Image 4
If the node is not able to communicate with the voting disks then the condition is the node which can talk to more than half of the voting disks will survive. In the case of image 4 there is again split brain between two sub clusters. One with node3 and other with node1, node2, node4. Now one of the node from cluster withnode1, node2 and node4 should be able to elect a master and get hold of control file and then update voting disk to evict node 3 and node 3 will be evicted.

Lets discuss two scenarios which will make this exercise rather interesting.

Image 4 shows that on a perfectly working cluster somehow node3 can not communicate with votingdisk2. Now you remember if communication between node and voting disk is breaking down then node eviction will happen only for the nodes which access less then half of the voting disks. That means if the node is accessing only one voting disk that node will be evicted. Hence in this case the eviction will not occur but you will notice some error message in the alter log.

Image 4


Why I chose 4 node cluster for my example. I wanted to tell what will happen if some how there are two su clusters with equal number of nodes in it like in image 5

image 5
Interesting, what will happen now...
Because the breakdown of the interconnect caused a situation of split brain. Which means from either of the cluster one node will try to be the master and evict other. Usually the surviving sub cluster would be the one which has maximum number of nodes. But in this case both sub clusters have equal number. This issue is fixed by choosing the cluster with minimum node numbers in it. Hence the surviving sub cluster will be one with node 1 and node 2 and it will choose the master node and then master node will send the poison bit for other nodes in voting disk and other nodes will be evicted.