CRS-2632: There are no more servers to try to place resource 'ora.qa2tesp.db' on that would satisfy its placement policy

Trying to start the RAC database after cloning from other RAC database and getting the error below:

[oracle@ract01n01 (qa2tesp1) dbs]$ srvctl start database -d qa2tesp
PRCR-1079 : Failed to start resource ora.qa2tesp.db
CRS-5017: The resource action "ora.qa2tesp.db start" encountered the following error:
ORA-01102: cannot mount database in EXCLUSIVE mode
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/ract01n02/crs/trace/crsd_oraagent_oracle.trc".
CRS-2674: Start of 'ora.qa2tesp.db' on 'ract01n02' failed
CRS-2632: There are no more servers to try to place resource 'ora.qa2tesp.db' on that would satisfy its placement policy
[oracle@ract01n01 (qa2tesp1) dbs]$ 

After that error I have noticed that the database is running on n01 but not on n02

oracle@ract01n01 (qa2tesp1) dbs]$ srvctl status database -d qa2tesp
Instance qa2tesp1 is running on node ract01n01
Instance qa2tesp2 is not running on node ract01n02
[oracle@ract01n01 (qa2tesp1) dbs]$ 

That gave me a idea that some how there is an issue with the cluster_database parameter

SQL> show parameter cluster_dat
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
cluster_database                     boolean     FALSE
cluster_database_instances           integer     1
SQL> 

but when I had a look at the init file which was used to create the spfile it says cluster_database=TRUE

That is interesting, it looks like the correct spfile is not in use. But looking at the $ORACLE_HOME/dbs directory it looks like I forgot to move the spfile from the directory after the restore and recovery

[oracle@ract01n01 (qa2tesp1) dbs]$ ls -lah *qa2tes*
-rw-r--r-- 1 oracle oinstall   63 Nov 21 09:40 initqa2tesp1.ora
-rw-r----- 1 oracle oinstall 2.5K Nov 17 14:31 orapwqa2tesp1
-rw-r----- 1 oracle asmadmin  14K Nov 21 09:40 spfileqa2tesp1.ora
[oracle@ract01n01 (qa2tesp1) dbs]$ 

Solution:

I have moved the spfile away from that folder and tried starting the database and it worked

[oracle@ract01n01 (qa2tesp1) dbs]$ mv spfileqa2tesp1.ora ./backup/
[oracle@ract01n01 (qa2tesp1) dbs]$
[oracle@ract01n01 (qa2tesp1) dbs]$ srvctl start database -d qa2tesp
[oracle@ract01n01 (qa2tesp1) dbs]$ 

ORA-19809: limit exceeded for recovery files

While trying to duplicate the database using RMAN we are encountering error below:


ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01152: file 1 was not restored from a sufficiently old backup
ORA-01110: data file 1: '+DATA/qa1tst/datafile/system.593.959955191'

released channel: c1
released channel: c2
released channel: c3
released channel: c4
released channel: c5
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 11/13/2017 14:24:49
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
ORA-19870: error while restoring backup piece 933347_PRD1DAD_i7sjseoh_1_1
ORA-19809: limit exceeded for recovery files
ORA-19804: cannot reclaim 65011712 bytes disk space from 128849018880 limit

by looking at the alert logs for the instance we noticed the error that db_recovery_file_dest_size is fully used.

ORA-19815: WARNING: db_recovery_file_dest_size of 128849018880 bytes is 100.00% used, and has 0 remaining bytes available.

That is the reason for this error.

We have increased the size in init parameter while doing recovery and recovery has worked this time.