ORA-01506: missing or illegal database name

I am trying to duplicate database from one database to another and all went well. But after creating the spfile when I tried starting the database using srvctl I started getting error

ORA-01506: missing or illegal database name

Did the following steps.
1. Creates spfile from pfile.
2. Start database using srvctl and received the error.


SQL> create spfile='+DATA/RAC/spfileRAC.ora' from pfile;

File created.

SQL>


[oracle@rac01 (rac1) dbs]$ srvctl start database -d rac
PRCR-1079 : Failed to start resource ora.rac.db
CRS-5017: The resource action "ora.rac.db start" encountered the following error:
ORA-01506: missing or illegal database name
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/rac02/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.rac.db' on 'rac02' failed
CRS-2632: There are no more servers to try to place resource 'ora.rac.db' on that would satisfy its placement policy
CRS-5017: The resource action "ora.rac.db start" encountered the following error:
ORA-01506: missing or illegal database name
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/rac01/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.rac.db' on 'rac01' failed
[oracle@rac01 (rac1) dbs]$


What is happening. If the environment is the standalone environment then it is okay the spfile is created alright. But in case of RAC environment the best is if we specify the pfile to create spfile and it will work.

So solution is try creating the spfile from pfile as below:


SQL> create spfile='+DATA/RAC/spfileRAC.ora' from pfile ='/u01/app/oracle/product/11.2.0/dbs/init_RAC_refresh.ora';

File created.

SQL>

[oracle@rac01 (rac1) dbs]$ srvctl start database -d rac
[oracle@rac01 (rac1) dbs]$

Oracle 12c Change system password

I am connected to pdb and getting the error while trying to change the password of system user


PDB1@ORCL> conn system/oracle
ERROR:
ORA-28002: the password will expire within 7 days


Connected.
PDB1@ORCL>


What I did is connected to root container and changed the password there


PDB1@ORCL> alter user SYSTEM identified by oracle;
alter user SYSTEM identified by oracle
*
ERROR at line 1:
ORA-65066: The specified changes must apply to all containers


PDB1@ORCL> alter user SYSTEM identified by oracle container=all;
alter user SYSTEM identified by oracle container=all
*
ERROR at line 1:
ORA-65050: Common DDLs only allowed in CDB$ROOT


PDB1@ORCL> select con_id, dbid, name, open_mode from v$containers;

    CON_ID       DBID NAME                           OPEN_MODE
---------- ---------- ------------------------------ ----------
         3 3345156736 PDB1                           READ WRITE

PDB1@ORCL>

PDB1@ORCL> show con_id

CON_ID
------------------------------
3
PDB1@ORCL>

PDB1@ORCL> alter session set container=CDB$ROOT;

Session altered.

PDB1@ORCL> show con_id

CON_ID
------------------------------
1

PDB1@ORCL> alter user SYSTEM identified by oracle container=all;

User altered.

PDB1@ORCL> alter session set container=pdb1;

Session altered.

PDB1@ORCL> show con_id

CON_ID
------------------------------
3
PDB1@ORCL> conn system/oracle
Connected.
PDB1@ORCL>

ORA-15031: disk specification '/dev/oracleasm/disks/ASM5' matches no disks

Getting ORA-15031 while creating the diskgroup and or adding disk to diskgroup.

The reason behind this is parameter asm_diskstring it should be set to ORCL:* or appropriate to your disk names.

For details below see below:


SQL> alter diskgroup FRA add disk '/dev/oracleasm/disks/ASM5';
alter diskgroup FRA add disk '/dev/oracleasm/disks/ASM5'
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15031: disk specification '/dev/oracleasm/disks/ASM5' matches no disks
ORA-15014: path '/dev/oracleasm/disks/ASM6' is not in the discovery set


SQL>


SQL> show parameter asm

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups                       string      CRS
asm_diskstring                       string
asm_power_limit                      integer     1
asm_preferred_read_failure_groups    string
SQL>

SQL> alter system set asm_diskstring='ORCL:*';

System altered.

SQL> create diskgroup FRA external redundancy disk 'ORCL:ASM5';

Diskgroup created.

SQL> set lines 270 pages 10000
SQL> col diskgroup for a10
SQL> col DISKNAME for a30
SQL> select dg.NAME diskgroup, dg.TYPE, dg.VOTING_FILES,
  2  ds.NAME diskname, ds.TOTAL_MB, ds.FREE_MB, dg.USABLE_FILE_MB
  3  from V$ASM_DISKGROUP dg, V$asm_disk_stat ds
  4  where dg.GROUP_NUMBER = ds.GROUP_NUMBER
  5  order by diskgroup, diskname;

DISKGROUP  TYPE   V DISKNAME                         TOTAL_MB    FREE_MB USABLE_FILE_MB
---------- ------ - ------------------------------ ---------- ---------- --------------
CRS        EXTERN Y SYSCONF1                             2047       1855           3739
CRS        EXTERN Y SYSCONF2                             2047       1884           3739
DATA       EXTERN N ASM1                                 5114       4568          25600
DATA       EXTERN N ASM2                                 5114       4567          25600
DATA       EXTERN N ASM3                                 5114       4570          25600
DATA       EXTERN N ASM4                                 5114       4570          25600
DATA       EXTERN N ASM7                                 8189       7325          25600
FRA        EXTERN N ASM5                                 5114       5064           5064

8 rows selected.

SQL>
SQL>  alter diskgroup FRA add disk 'ORCL:ASM8';

Diskgroup altered.

SQL> set lines 270 pages 10000
SQL> col diskgroup for a10
SQL> col DISKNAME for a30
SQL> select dg.NAME diskgroup, dg.TYPE, dg.VOTING_FILES,
  2  ds.NAME diskname, ds.TOTAL_MB, ds.FREE_MB, dg.USABLE_FILE_MB
  3  from V$ASM_DISKGROUP dg, V$asm_disk_stat ds
  4  where dg.GROUP_NUMBER = ds.GROUP_NUMBER
  5  order by diskgroup, diskname;

DISKGROUP  TYPE   V DISKNAME                         TOTAL_MB    FREE_MB USABLE_FILE_MB
---------- ------ - ------------------------------ ---------- ---------- --------------
CRS        EXTERN Y SYSCONF1                             2047       1855           3739
CRS        EXTERN Y SYSCONF2                             2047       1884           3739
DATA       EXTERN N ASM1                                 5114       4568          25600
DATA       EXTERN N ASM2                                 5114       4567          25600
DATA       EXTERN N ASM3                                 5114       4570          25600
DATA       EXTERN N ASM4                                 5114       4570          25600
DATA       EXTERN N ASM7                                 8189       7325          25600
FRA        EXTERN N ASM5                                 5114       5065          13251
FRA        EXTERN N ASM8                                 8189       8186          13251

9 rows selected.

SQL>

Change the redundancy of FRA under DR site in active dataguard

I have noticed that my mistake FRA on DR site is set to wrong redundancy, it was suppose to be external but set to NORMAL.
Now task is change the redundancy to EXTERNAL for FRA.

The solution is simple.

1. Stop database on DR
2. Drop the diskgroup
3. Recreate diskgroup with correct redundancy.
4. Start the database


ASMCMD> lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576      4094     3739                0            3739              0             Y  CRS/
MOUNTED  EXTERN  N         512   4096  1048576     28645    25600                0           25600              0             N  DATA/
MOUNTED  NORMAL  N         512   4096  1048576     13303     5753                0            2876              0             N  FRA/
ASMCMD>


DISKGROUP  TYPE   V DISKNAME     TOTAL_MB    FREE_MB USABLE_FILE_MB
---------- ------ - ---------- ---------- ---------- --------------
CRS        EXTERN Y SYSCONF1         2047       1855           3739
CRS        EXTERN Y SYSCONF2         2047       1884           3739
DATA       EXTERN N ASM1             5114       4568          25600
DATA       EXTERN N ASM2             5114       4567          25600
DATA       EXTERN N ASM3             5114       4570          25600
DATA       EXTERN N ASM4             5114       4570          25600
DATA       EXTERN N ASM7             8189       7325          25600
FRA        NORMAL N ASM5             5114       1339           2876
FRA        NORMAL N ASM8             8189       4414           2876


[oracle@racdr1 ~]$ srvctl stop database -d RAC_STBY



[oracle@racdr1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.0 Production

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> drop diskgroup FRA including contents;

Diskgroup dropped.

SQL>
SQL> col diskgroup for a10
SQL> col DISKNAME for a30
SQL> select dg.NAME diskgroup, dg.TYPE, dg.VOTING_FILES,
  2  ds.NAME diskname, ds.TOTAL_MB, ds.FREE_MB, dg.USABLE_FILE_MB
  3  from V$ASM_DISKGROUP dg, V$asm_disk_stat ds
  4  where dg.GROUP_NUMBER = ds.GROUP_NUMBER
  5  order by diskgroup, diskname;

DISKGROUP  TYPE   V DISKNAME                         TOTAL_MB    FREE_MB USABLE_FILE_MB
---------- ------ - ------------------------------ ---------- ---------- --------------
CRS        EXTERN Y SYSCONF1                             2047       1855           3739
CRS        EXTERN Y SYSCONF2                             2047       1884           3739
DATA       EXTERN N ASM1                                 5114       4568          25600
DATA       EXTERN N ASM2                                 5114       4567          25600
DATA       EXTERN N ASM3                                 5114       4570          25600
DATA       EXTERN N ASM4                                 5114       4570          25600
DATA       EXTERN N ASM7                                 8189       7325          25600

7 rows selected.

SQL>
SQL> show parameter asm

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_diskgroups                       string      CRS
asm_diskstring                       string
asm_power_limit                      integer     1
asm_preferred_read_failure_groups    string
SQL>

SQL> alter system set asm_diskstring='ORCL:*';

System altered.

SQL> create diskgroup FRA external redundancy disk 'ORCL:ASM5';

Diskgroup created.

SQL> set lines 270 pages 10000
SQL> col diskgroup for a10
SQL> col DISKNAME for a30
SQL> select dg.NAME diskgroup, dg.TYPE, dg.VOTING_FILES,
  2  ds.NAME diskname, ds.TOTAL_MB, ds.FREE_MB, dg.USABLE_FILE_MB
  3  from V$ASM_DISKGROUP dg, V$asm_disk_stat ds
  4  where dg.GROUP_NUMBER = ds.GROUP_NUMBER
  5  order by diskgroup, diskname;

DISKGROUP  TYPE   V DISKNAME                         TOTAL_MB    FREE_MB USABLE_FILE_MB
---------- ------ - ------------------------------ ---------- ---------- --------------
CRS        EXTERN Y SYSCONF1                             2047       1855           3739
CRS        EXTERN Y SYSCONF2                             2047       1884           3739
DATA       EXTERN N ASM1                                 5114       4568          25600
DATA       EXTERN N ASM2                                 5114       4567          25600
DATA       EXTERN N ASM3                                 5114       4570          25600
DATA       EXTERN N ASM4                                 5114       4570          25600
DATA       EXTERN N ASM7                                 8189       7325          25600
FRA        EXTERN N ASM5                                 5114       5064           5064

8 rows selected.

SQL>
SQL>  alter diskgroup FRA add disk 'ORCL:ASM8';

Diskgroup altered.

SQL> set lines 270 pages 10000
SQL> col diskgroup for a10
SQL> col DISKNAME for a30
SQL> select dg.NAME diskgroup, dg.TYPE, dg.VOTING_FILES,
  2  ds.NAME diskname, ds.TOTAL_MB, ds.FREE_MB, dg.USABLE_FILE_MB
  3  from V$ASM_DISKGROUP dg, V$asm_disk_stat ds
  4  where dg.GROUP_NUMBER = ds.GROUP_NUMBER
  5  order by diskgroup, diskname;

DISKGROUP  TYPE   V DISKNAME                         TOTAL_MB    FREE_MB USABLE_FILE_MB
---------- ------ - ------------------------------ ---------- ---------- --------------
CRS        EXTERN Y SYSCONF1                             2047       1855           3739
CRS        EXTERN Y SYSCONF2                             2047       1884           3739
DATA       EXTERN N ASM1                                 5114       4568          25600
DATA       EXTERN N ASM2                                 5114       4567          25600
DATA       EXTERN N ASM3                                 5114       4570          25600
DATA       EXTERN N ASM4                                 5114       4570          25600
DATA       EXTERN N ASM7                                 8189       7325          25600
FRA        EXTERN N ASM5                                 5114       5065          13251
FRA        EXTERN N ASM8                                 8189       8186          13251

9 rows selected.

SQL>

[oracle@racdr1 ~]$ srvctl start database -d RAC_STBY

Change the location of asmparameterfile

We want to change the location of asmparameterfile, at the moment it is under +DATA and want to move to +CRS

Log in to asm as sysasm

SQL> select name,state from v$asm_diskgroup;

NAME                           STATE
------------------------------ -----------
CRS                            MOUNTED
DATA                           MOUNTED
FRA                            MOUNTED

SQL>
SQL> show parameter spfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +DATA/scandr/asmparameterfile/
                                                 registry.253.866558147
SQL>

Now create pfile from spfile and from that file create the spfile.

SQL> create pfile='/home/oracle/temp/initASM.ora' from spfile;

File created.

SQL> !ls -lah /home/oracle/temp/initASM.ora
-rw-r--r--. 1 oracle oinstall 186 Jan 12 19:16 /home/oracle/temp/initASM.ora

SQL> show parameter spfile;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +DATA/scandr/asmparameterfile/
                                                 registry.253.866558147
SQL>
SQL> create spfile='+CRS' from pfile='/home/oracle/temp/initASM.ora';

File created.

SQL>

Stop HAS and start it this will use spfile to start it and it will be new location.

[root@racdr1 ~]# . ./grid_env
[root@racdr1 ~]# crsclt stop has
-bash: crsclt: command not found
[root@racdr1 ~]# crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'racdr1'
CRS-2673: Attempting to stop 'ora.crsd' on 'racdr1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'racdr1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'racdr1'
CRS-2673: Attempting to stop 'ora.cvu' on 'racdr1'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'racdr1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'racdr1'
CRS-2673: Attempting to stop 'ora.rac_stby.db' on 'racdr1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'racdr1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'racdr1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'racdr1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'racdr1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'racdr1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.racdr1.vip' on 'racdr1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'racdr1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'racdr1'
CRS-2677: Stop of 'ora.cvu' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.rac_stby.db' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'racdr1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'racdr1'
CRS-2677: Stop of 'ora.scan1.vip' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.registry.acfs' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.scan2.vip' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.racdr1.vip' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'racdr1' succeeded
CRS-2679: Attempting to clean 'ora.DATA.dg' on 'racdr1'
CRS-2681: Clean of 'ora.DATA.dg' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'racdr1'
CRS-2677: Stop of 'ora.asm' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'racdr1'
CRS-2677: Stop of 'ora.ons' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'racdr1'
CRS-2677: Stop of 'ora.net1.network' on 'racdr1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'racdr1' has completed
CRS-2677: Stop of 'ora.crsd' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'racdr1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'racdr1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'racdr1'
CRS-2673: Attempting to stop 'ora.evmd' on 'racdr1'
CRS-2673: Attempting to stop 'ora.asm' on 'racdr1'
CRS-2677: Stop of 'ora.evmd' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.asm' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'racdr1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'racdr1'
CRS-2677: Stop of 'ora.cssd' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'racdr1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'racdr1' succeeded
CRS-2677: Stop of 'ora.crf' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'racdr1'
CRS-2677: Stop of 'ora.gipcd' on 'racdr1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'racdr1'
CRS-2677: Stop of 'ora.gpnpd' on 'racdr1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'racdr1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@racdr1 ~]#
[root@racdr1 ~]#


[root@racdr1 ~]# crsctl start has
CRS-4123: Oracle High Availability Services has been started.
[root@racdr1 ~]#


SQL> show parameter spfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +CRS/scandr/asmparameterfile/r
                                                 egistry.253.868799821
SQL>

ASMCMD> spget
+CRS/scandr/asmparameterfile/registry.253.868799821
ASMCMD>
ASMCMD>
ASMCMD> rm +DATA/scandr/asmparameterfile/registry.253.866558147

OCR and HAS status after node restart

In case you need to check what will happen to oracle cluster after the restart of the node then it is detailed in below files and location.

[root@rac1 root]# pwd
/etc/oracle/scls_scr/rac1/root
[root@rac1 root]# ls -l
total 12
-rw-r--r--. 1 root root     7 Dec 15 15:40 crsstart
-rw-r--r--. 1 root oinstall 8 Jan 12 17:38 ohasdrun
-rw-r--r--. 1 root oinstall 7 Jan 12 17:40 ohasdstr
[root@rac1 root]#
[root@rac1 root]#
[root@rac1 root]# cat crsstart
enable
[root@rac1 root]#
[root@rac1 root]# cat ohasdrun
restart
[root@rac1 root]#
[root@rac1 root]# cat ohasdstr
enable
[root@rac1 root]#
[root@rac1 root]# 

Change the location of ocr registry and voting disk

If you are looking to change the location of ocr registry file and votedisk location then it can be done as 

[root@rac1 ~]# 

[root@rac1 ~]# ocrcheck

Status of Oracle Cluster Registry is as follows :

         Version                  :          3

         Total space (kbytes)     :     262120

         Used space (kbytes)      :       3156

         Available space (kbytes) :     258964

         ID                       : 1353498280

         Device/File Name         :      +DATA

                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

[root@rac1 ~]# 

[root@rac1 ~]# 

[root@rac1 ~]# ocrconfig -add +CRS

[root@rac1 ~]# 

[root@rac1 ~]# 

[root@rac1 ~]# ocrcheck

Status of Oracle Cluster Registry is as follows :

         Version                  :          3

         Total space (kbytes)     :     262120

         Used space (kbytes)      :       3172

         Available space (kbytes) :     258948

         ID                       : 1353498280

         Device/File Name         :      +DATA

                                    Device/File integrity check succeeded

         Device/File Name         :       +CRS

                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

[root@rac1 ~]# 

[root@rac1 ~]# ocrconfig -delete +DATA

[root@rac1 ~]# 

[root@rac1 ~]# 

[root@rac1 ~]# ocrcheck

Status of Oracle Cluster Registry is as follows :

         Version                  :          3

         Total space (kbytes)     :     262120

         Used space (kbytes)      :       3172

         Available space (kbytes) :     258948

         ID                       : 1353498280

         Device/File Name         :       +CRS

                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

         Cluster registry integrity check succeeded

         Logical corruption check succeeded

[root@rac1 ~]# 

[root@rac1 ~]# 

[root@rac1 ~]# 

[root@rac1 ~]# crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------

 1. ONLINE   03f7c050af454f7bbf3eef9fc48babb1 (ORCL:ASM1) [DATA]

Located 1 voting disk(s).

[root@rac1 ~]# 

[root@rac1 ~]# 

[root@rac1 ~]# 

[root@rac1 ~]# crsctl replace votedisk +CRS

Successful addition of voting disk 6805336bdbb14fdbbfcc4ab99062b4bc.

Successful deletion of voting disk 03f7c050af454f7bbf3eef9fc48babb1.

Successfully replaced voting disk group with +CRS.

CRS-4266: Voting file(s) successfully replaced

[root@rac1 ~]# 

[root@rac1 ~]# 

[root@rac1 ~]# 

[root@rac1 ~]#  crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------

 1. ONLINE   6805336bdbb14fdbbfcc4ab99062b4bc (ORCL:CRS1) [CRS]

Located 1 voting disk(s).

[root@rac1 ~]# 

oracle database switchover and getting ORA-01017: invalid username/password; logon denied

Getting and error while doing the switchover to standby 

ORA-01017: invalid username/password; logon denied

[oracle@racdr1 ~]$ dgmgrl system/********
DGMGRL for Linux: Version 11.2.0.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected.
DGMGRL> show configuration

Configuration - racdg

  Protection Mode: MaxPerformance
  Databases:
    RAC      - Primary database
    RAC_STBY - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> 
DGMGRL> 
DGMGRL> switchover to 'RAC_STBY'
Performing switchover NOW, please wait...
New primary database "RAC_STBY" is opening...
Operation requires startup of instance "RAC2" on database "RAC"
Starting instance "RAC2"...
ORA-01017: invalid username/password; logon denied

Warning: You are no longer connected to ORACLE.

Please complete the following steps to finish switchover:
        start up instance "RAC2" of database "RAC"

DGMGRL> 

it is little annoying but looking carefully we can figure out the issue and that is I am tying to switchover with user system and that is not present in password file across primary and dr.

SQL> select * from v$pwfile_users;

USERNAME                       SYSDB SYSOP SYSAS
------------------------------ ----- ----- -----
SYS                            TRUE  TRUE  FALSE


SQL>

So the solution is try it with sys user as it is present in password file

[oracle@rac1 dbs]$ dgmgrl sys/********@RAC
DGMGRL for Linux: Version 11.2.0.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected.
DGMGRL> 
DGMGRL> show configuration

Configuration - racdg

  Protection Mode: MaxPerformance
  Databases:
    RAC_STBY - Primary database
    RAC      - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> 
DGMGRL> 
DGMGRL> switchover to 'RAC'
Performing switchover NOW, please wait...
Operation requires a connection to instance "RAC2" on database "RAC"
Connecting to instance "RAC2"...
Connected.
New primary database "RAC" is opening...
Operation requires startup of instance "RAC_STBY1" on database "RAC_STBY"
Starting instance "RAC_STBY1"...
ORACLE instance started.
Database mounted.
Database opened.
Switchover succeeded, new primary is "RAC"
DGMGRL> 
DGMGRL> 
DGMGRL> show configuration

Configuration - racdg

  Protection Mode: MaxPerformance
  Databases:
    RAC      - Primary database
    RAC_STBY - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS


DGMGRL> 

TNS-01201: Listener cannot find executable /u01/app/oracle/product/11.2.0/bin/oracle for SID RAC1

getting error while trying to start listener on rac node

TNS-01201: Listener cannot find executable /u01/app/oracle/product/11.2.0/bin/oracle for SID RAC1

[oracle@rac1 admin]$ lsnrctl start listener
LSNRCTL for Linux: Version 11.2.0.0 - Production
Copyright (c) 1991, 2013, Oracle.  All rights reserved.
Starting /u01/app/11.2.0/grid/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 11.2.0.0 - ProductionSystem parameter file is /u01/app/11.2.0/grid/network/admin/listener.oraLog messages written to /u01/app/oracle/diag/tnslsnr/rac1/listener/alert/log.xmlListening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))TNS-01201: Listener cannot find executable /u01/app/oracle/product/11.2.0/bin/oracle for SID RAC1
Listener failed to start. See the error message(s) above...
[oracle@rac1 admin]$ 

Looks like oracle can not find listener executable under ORACLE_HOME mentioned in listener.ora so had a look at the file and below was the content of the file

SID_LIST_LISTENER =(SID_LIST =        (SID_DESC =                (GLOBAL_DBNAME = RAC_DGMGRL.localdomain)                (ORACLE_HOME = /u01/app/oracle/product/11.2.0)                (SID_NAME = RAC1)   ) )

I have updated the listener.ora file with correct ORACLE_HOME and then tried starting the listener and all went well.

 SID_LIST_LISTENER = (SID_LIST =         (SID_DESC =                 (GLOBAL_DBNAME = RAC_DGMGRL.localdomain)                 (ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1)                 (SID_NAME = RAC1)    ) )

[oracle@rac1 admin]$ lsnrctl start listener
LSNRCTL for Linux: Version 11.2.0.0 - Production 
Copyright (c) 1991, 2013, Oracle.  All rights reserved.
Starting /u01/app/11.2.0/grid/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 11.2.0.0 - ProductionSystem parameter file is /u01/app/11.2.0/grid/network/admin/listener.oraLog messages written to /u01/app/oracle/diag/tnslsnr/rac1/listener/alert/log.xmlListening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))STATUS of the LISTENER------------------------Alias                     listenerVersion                   TNSLSNR for Linux: Version 11.2.0.0 - ProductionStart Date                Uptime                    0 days 0 hr. 0 min. 0 secTrace Level               offSecurity                  ON: Local OS AuthenticationSNMP                      OFFListener Parameter File   /u01/app/11.2.0/grid/network/admin/listener.oraListener Log File         /u01/app/oracle/diag/tnslsnr/rac1/listener/alert/log.xmlListening Endpoints Summary...  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))Services Summary...Service "RAC_DGMGRL.localdomain" has 1 instance(s).  Instance "RAC1", status UNKNOWN, has 1 handler(s) for this service...The command completed successfully[oracle@rac1 admin]$  

How data under V$DATABASE_BLOCK_CORRUPTION get refreshed

If there are any block corruption under oracle then an entry is get created under V$DATABASE_BLOCK_CORRUPTION. 

SQL> select count(*) from V$DATABASE_BLOCK_CORRUPTION;
    COUNT(*)------------         200
SQL>

Above sql shows that there are 200 block corruption in my database. I have discovered that few of those blocks are related to indexes. What I have done is dropped and created the index back. This way the index is created again and the corrupt blocks related to indexes are fixed.

After this if I run the count(*) query on V$DATABASE_BLOCK_CORRUPTION, I will still get the same results.

Reason:

V$DATABASE_BLOCK_CORRUPTION get updated when oracle reads a block and finds that there are corruption under that block. The block could remain corrupt for years if it was never queried.

So how to get the current picture and update V$DATABASE_BLOCK_CORRUPTION.

The easiest way is go thought and check the blocks and that can be done by RMAN with following command

RMAN> backup validate check logical database;

Above command will go though each and every block of the database and check if there is any corruption and update V$DATABASE_BLOCK_CORRUPTION, it will also remove previous entries which says there is a corruption in index and that was fixed. After running the validation when I again select from V$DATABASE_BLOCK_CORRUPTION blow were the results.

SQL> select count(*) from V$DATABASE_BLOCK_CORRUPTION;
    COUNT(*)------------         40SQL>