Dirty Cow vulnerability on linux

I just wanted to test what is this dirty cow and how does this work. I am not going to share how this works as you can search on google but below is what it can do.

Change the content of read only file with 0404 privilege

pi@raspberrypi:~/hdhillon/cow$ ls -l
total 16
-rwxr-xr-x 1 pi   pi   7653 Nov 10 15:33 harveyCOW
-rw-r--r-- 1 pi   pi   2408 Nov 10 15:33 harveyCOW.c
-r-----r-- 1 root root   22 Nov 10 15:40 myReadOnlyFile.txt
pi@raspberrypi:~/hdhillon/cow$
pi@raspberrypi:~/hdhillon/cow$
pi@raspberrypi:~/hdhillon/cow$ cat myReadOnlyFile.txt
readOnly file content
pi@raspberrypi:~/hdhillon/cow$
pi@raspberrypi:~/hdhillon/cow$ ./harveyCOW myReadOnlyFile.txt mOOOmOOOOmOOOmOOOOmOOOOO
mmap b6fae000
^C
pi@raspberrypi:~/hdhillon/cow$
pi@raspberrypi:~/hdhillon/cow$ cat myReadOnlyFile.txt
mOOOmOOOOmOOOmOOOOmOOOpi@raspberrypi:~/hdhillon/cow$
pi@raspberrypi:~/hdhillon/cow$
pi@raspberrypi:~/hdhillon/cow$ ls -l
total 16
-rwxr-xr-x 1 pi   pi   7653 Nov 10 15:33 harveyCOW
-rw-r--r-- 1 pi   pi   2408 Nov 10 15:33 harveyCOW.c
-r-----r-- 1 root root   22 Nov 10 15:40 myReadOnlyFile.txt
pi@raspberrypi:~/hdhillon/cow$

Target Manager failed at Startup: targets.xml was rejected: loaded with a wrong agent token

Trying to start the oracle management agent on client and it is failing to start and getting the error below:

$ ./emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting agent .............. failed.
Target Manager failed at Startup: targets.xml was rejected: loaded with a wrong agent token
Consult emctl.log and emagent.nohup in: /u01/agent12c/agent_inst/sysman/log



Solution :

Go to /u01/agent12c/agent_inst/sysman/emd and
mv targets.xml target.xml.backup

create new targets.xml with below content


<Targets>

</Targets>

Trying starting the agent with

emctl start agent

go back to your targets.xml file and cat it to get value of AGENT_TOKEN

$ cat targets.xml
<Targets AGENT_TOKEN="B7579B0E4405142405C6FCA2F86B2D2440FC93784BBA17229895F56E933ACA4F"/>
$

-- Stop agent

emctl stop agent

go to targets.xml.backup and modify the AGENT_TOKEN with current value.

remove  targets.xml which has got

<Targets AGENT_TOKEN="B7579B0E4405142405C6FCD2440FC93784BBA17229895F56E9334F"/>

Rename targets.xml.backup to targets.xml

start agent

Wallah it works !!!

RMAN-06726: could not locate archived log

oracle backup of logfiles is failing with error

RMAN-06726: could not locate archived log


Issue is that there are some expired logs which still have references in catalogue. To fix this issue the cross check

RMAN> crosscheck archivelog all;

You would notice that there are some validations of some of the files fail with  message

validation failed for archived log

After that check all expired logs

RMAN> list expired archivelog all;

and then delete all expired logs

RMAN> delete expired archivelog all;

This will ask to delete them, type yes and hit enter

Boom !!!

Issue is fixed.

Re-run the backup and it is all good.

ORA-00845: MEMORY_TARGET not supported on this system

Getting the error when trying to start database.

[oracle@rac1 ~]$ srvctl start database -d cdbracp
PRCR-1079 : Failed to start resource ora.cdbracp.db
CRS-5017: The resource action "ora.cdbracp.db start" encountered the following error:
ORA-00845: MEMORY_TARGET not supported on this system
. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/rac1/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.cdbracp.db' on 'rac1' failed
CRS-2632: There are no more servers to try to place resource 'ora.cdbracp.db' on that would satisfy its placement policy
CRS-5017: The resource action "ora.cdbracp.db start" encountered the following error:
ORA-00845: MEMORY_TARGET not supported on this system
. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/rac2/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.cdbracp.db' on 'rac2' failed
[oracle@rac1 ~]$

Annoying !!!!

Check the /dev/shm available space

[oracle@rac1 ~]$ df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs                 2.4G  635M  1.8G  27% /dev/shm

Look like 1.8G and I know that my database instance has memory_target set up to 2G. Means we do not have enough space for this oracle instance in tmpfs

to fix this what I have done is modified /etc/fstab and set the value of tempfs to 5G. Please note that this value has to be less than available memory on host.

There is no need to restart after this, just remount the volume.

[root@rac1 ~]# mount -o remount /dev/shm
[root@rac1 ~]# df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
tmpfs                 5.0G  635M  4.4G  13% /dev/shm

Now I have 4.4G free space which means I should be able to start the instance with 2G of memory_target.

[oracle@rac1 ~]$ srvctl start database -d cdbracp
PRCR-1079 : Failed to start resource ora.cdbracp.db
CRS-5017: The resource action "ora.cdbracp.db start" encountered the following error:
ORA-00845: MEMORY_TARGET not supported on this system
. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/rac2/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.cdbracp.db' on 'rac2' failed
CRS-2632: There are no more servers to try to place resource 'ora.cdbracp.db' on that would satisfy its placement policy
[oracle@rac1 ~]$


It only failed on node 2 as I am a dum ass and forgot to remount on node2

[oracle@rac1 ~]$ srvctl status database -d cdbracp
Instance cdbracp1 is running on node rac1
Instance cdbracp2 is not running on node rac2
[oracle@rac1 ~]$



after remounting on node2 I could start both database instances.

[root@rac2 ~]# mount -o remount /dev/shm
[root@rac2 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
tmpfs                 5.0G  635M  4.4G  13% /dev/shm

[oracle@rac1 ~]$ srvctl start instance -d cdbracp -i cdbracp2
[oracle@rac1 ~]$
[oracle@rac1 ~]$ srvctl status database -d cdbracp
Instance cdbracp1 is running on node rac1
Instance cdbracp2 is running on node rac2
[oracle@rac1 ~]$

ORA-15046: ASM file name '+FRA/CDBRAC/CONTROLFILE/Current.261.909834683' is not in single-file creation form

tried copying the control file from +data to +fra and encountered the error.

ASMCMD> cd +DATA/CDBRAC/CONTROLFILE
ASMCMD> ls -l
Type         Redund  Striped  Time             Sys  Name
CONTROLFILE  UNPROT  FINE     OCT 20 15:00:00  Y    Current.261.909834683
CONTROLFILE  UNPROT  FINE     OCT 20 15:00:00  N    Current.261.909834683.copy => +DATA/ASM/CONTROLFILE/Current.261.909834683.copy.277.925746471
ASMCMD>
ASMCMD>
ASMCMD> cp Current.261.909834683 +FRA/CDBRAC/CONTROLFILE/Current.256.909834683
copying +DATA/CDBRAC/CONTROLFILE/Current.261.909834683 -> +FRA/CDBRAC/CONTROLFILE/Current.256.909834683
ASMCMD-8016: copy source '+DATA/CDBRAC/CONTROLFILE/Current.261.909834683' and target '+FRA/CDBRAC/CONTROLFILE/Current.256.909834683' failed
ORA-15056: additional error message
ORA-15046: ASM file name '+FRA/CDBRAC/CONTROLFILE/Current.256.909834683' is not in single-file creation form
ORA-06512: at "SYS.X$DBMS_DISKGROUP", line 486
ORA-06512: at line 3 (DBD ERROR: OCIStmtExecute)
ASMCMD>


Please note that it will only work if we specify the file name on target and do not specify the number or incarnation. That is something that asm will choose

ASMCMD> cp Current.261.909834683 +FRA/CDBRAC/CONTROLFILE
copying +DATA/CDBRAC/CONTROLFILE/Current.261.909834683 -> +FRA/CDBRAC/CONTROLFILE/Current.261.909834683
ASMCMD-8016: copy source '+DATA/CDBRAC/CONTROLFILE/Current.261.909834683' and target '+FRA/CDBRAC/CONTROLFILE/Current.261.909834683' failed
ORA-15056: additional error message
ORA-15046: ASM file name '+FRA/CDBRAC/CONTROLFILE/Current.261.909834683' is not in single-file creation form
ORA-06512: at "SYS.X$DBMS_DISKGROUP", line 486
ORA-06512: at line 3 (DBD ERROR: OCIStmtExecute)
ASMCMD>


-- Below is how it should be done.

ASMCMD>
ASMCMD> cp Current.261.909834683 +FRA/CDBRAC/CONTROLFILE/Current
copying +DATA/CDBRAC/CONTROLFILE/Current.261.909834683 -> +FRA/CDBRAC/CONTROLFILE/Current
ASMCMD>


ASMCMD> cd +FRA/CDBRAC/CONTROLFILE/      
ASMCMD> ls -l
Type         Redund  Striped  Time             Sys  Name
CONTROLFILE  UNPROT  FINE     OCT 20 15:00:00  N    Current => +FRA/ASM/CONTROLFILE/Current.488.925746873
CONTROLFILE  UNPROT  FINE     OCT 20 15:00:00  Y    Current.256.909834683
CONTROLFILE  UNPROT  FINE     OCT 20 15:00:00  N    Current.256.909834683.copyFra => +FRA/ASM/CONTROLFILE/Current.256.909834683.copyFra.487.925746605
ASMCMD>

Getting oracle parameter value and description

Getting the parameter values and description is very important. Some times show parameter will not show every thing link in this case of underscore parameter

SQL> show parameter _gc_read_mostly_locking
SQL>


to get the values best is running the command below and passing the name of the parameter.


col Parameter FOR a30
col Instance FOR a10
col Description FOR a100 word_wrapped

SELECT a.ksppinm  "Parameter", c.ksppstvl "Instance", a.ksppdesc "Description"
FROM x$ksppi a, x$ksppcv b, x$ksppsv c, v$parameter p
WHERE a.indx = b.indx
AND a.indx = c.indx
AND p.name(+) = a.ksppinm
AND UPPER(a.ksppinm) LIKE UPPER('%&parameter%')
ORDER BY a.ksppinm;



Example:

SQL> col Parameter FOR a30
SQL> col Instance FOR a10
SQL> col Description FOR a100 word_wrapped
SQL>
SQL> SELECT a.ksppinm  "Parameter", c.ksppstvl "Instance",       a.ksppdesc "Description"
  2  FROM x$ksppi a, x$ksppcv b, x$ksppsv c, v$parameter p
  3  WHERE a.indx = b.indx
  4  AND a.indx = c.indx
  5  AND p.name(+) = a.ksppinm
  6  AND UPPER(a.ksppinm) LIKE UPPER('%&parameter%')
  7  ORDER BY a.ksppinm;
Enter value for parameter: _gc_read_mostly_lock
old   6: AND UPPER(a.ksppinm) LIKE UPPER('%&parameter%')
new   6: AND UPPER(a.ksppinm) LIKE UPPER('%_gc_read_mostly_lock%')

Parameter                      Instance   Description
------------------------------ ---------- -----------------------------------------------------------------
_gc_read_mostly_locking        TRUE       if TRUE, enable read-mostly locking

SQL> 

ORA-00230: operation disallowed: snapshot control file enqueue unavailable

Received error

ORA-00230: operation disallowed: snapshot control file enqueue unavailable

in alert log of newly created primary after the switchover from DR and tracefile has following content

*** 2016-10-18 08:58:53.795
*** SESSION ID:(1998.27) 2016-10-18 08:58:53.795
*** CLIENT ID:() 2016-10-18 08:58:53.795
*** SERVICE NAME:(SYS$BACKGROUND) 2016-10-18 08:58:53.795
*** MODULE NAME:(MMON_SLAVE) 2016-10-18 08:58:53.795
*** ACTION NAME:(Autobackup Control File) 2016-10-18 08:58:53.795

Starting control autobackup
Got error: 230
********************  WARNING ***************************
The errors during Server autobackup are not fatal, as it
is attempted after sucessful completion of the command.
However, it is recomended to take an RMAN control file
backup as soon as possible because the Autobackup failed
with the following error:
ORA-00230: operation disallowed: snapshot control file enqueue unavailable
********************  END OF WARNING *******************
ORA-00230: operation disallowed: snapshot control file enqueue unavailable


Solution:

It looks like starting control file autobackup had some issues and resulted in creating

ORA-00230: operation disallowed: snapshot control file enqueue unavailable


Log in on the database and run the command below:

col CLIENT_INFO for a20
col EVENT for a35
Select sysdate, inst_id, sid, client_info,
event, seq#,p1,p2,p3, wait_time, seconds_in_wait,
wait_time_micro, time_since_last_wait_micro, state
from gv$session
where program like '%rman%'
and client_info like '%channel%';


or

SELECT s.sid, username AS "user" , program, module, action, logon_time as "Logon" , l.*   FROM v$session s, v$enqueue_lock l   WHERE l.sid = s.sid and l.type = 'CF' AND l.id1 = 0 and l.id2 = 2;



If you are getting some output with state waiting that means there is already and operation going on causing new operation to fail and showing the error in alert log.


Sometimes the process could be sitting there for ages, which means it is hung. In order to kill the process we need to find its UNIX process id to kill it.


COLUMN EVENT FORMAT a10      
COLUMN SECONDS_IN_WAIT FORMAT 999
COLUMN STATE FORMAT a20      
COLUMN CLIENT_INFO FORMAT a30    

SELECT p.SPID, EVENT, SECONDS_IN_WAIT AS SEC_WAIT,    
STATE, CLIENT_INFO      
FROM V$SESSION_WAIT sw,
V$SESSION s, V$PROCESS p      
WHERE sw.EVENT LIKE 'sbt%'
AND s.SID=sw.SID      
AND s.PADDR=p.ADDR;


once you have the SPID which is server process id or ID of UNIX process

then simply issue

kill -9 <SPID>

It should fix the issue.

Check if MRP process is running on DR instance.

Sometime we need to check if the MRP process is running in our DR database.

Solution:

This can be done using one simple SQL


SELECT PROCESS, DELAY_MINS
FROM V$MANAGED_STANDBY
WHERE PROCESS like 'MRP%';

If the above SQL returns nothing that means MRP is not running. How ever if it returns something that means MRP is running.

SQL> SELECT PROCESS, DELAY_MINS FROM V$MANAGED_STANDBY WHERE PROCESS like 'MRP%';
no rows selected
SQL>

-- MRP is not running 

-- To start MPR issue the SQL below

SQL>  ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;
Database altered.
SQL>
SQL> SELECT PROCESS, DELAY_MINS FROM V$MANAGED_STANDBY WHERE PROCESS like 'MRP%';
PROCESS   DELAY_MINS
--------- ----------
MRP0               0
SQL>

-- MRP process is running now.

Tip: if you want standby logs to be applied instantly then issue

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;

Warning: ORA-16829: fast-start failover configuration is lagging

Automatic failover caused DR to become primary and primary to become DR, but when checked the dgmgrl configuration there were some errors.

$ dgmgrl
DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys
Password:
Connected.
DGMGRL> show configuration

Configuration - db1tes_dg

  Protection Mode: MaxPerformance
  Databases:
    db1tesd - Primary database
      Warning: ORA-16829: fast-start failover configuration is lagging

    db1tesp - (*) Physical standby database
      Error: ORA-16810: multiple errors or warnings detected for the database

Fast-Start Failover: ENABLED

Configuration Status:
ERROR

DGMGRL>

DGMGRL> show database db1tesd

Database - db1tesd

  Role:            PRIMARY
  Intended State:  TRANSPORT-ON
  Instance(s):
    db1tesd1

  Database Warning(s):
    ORA-16829: fast-start failover configuration is lagging

Database Status:
WARNING

DGMGRL>
DGMGRL>
DGMGRL>
DGMGRL> show database db1tesp

Database - db1tesp

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   (unknown)
  Apply Lag:       3 hours 20 minutes 3 seconds
  Real Time Query: ON
  Instance(s):
    db1tesp1 (apply instance)
    db1tesp2

  Database Warning(s):
    ORA-16826: apply service state is inconsistent with the DelayMins property
    ORA-16829: fast-start failover configuration is lagging

Database Status:
WARNING

DGMGRL>

Fix:

What I did in our case was simply restarted the the new DR and it fixed the issue.

$ srvctl stop database -d db1tesp

$ srvctl start database -d db1tesp

$ dgmgrl 

DGMGRL for Linux: Version 11.2.0.3.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.

DGMGRL> connect sys

Password:

Connected.

DGMGRL> show configuration

Configuration - db1tes_dg

  Protection Mode: MaxPerformance

  Databases:

    db1tesd - Primary database

    db1tesp - (*) Physical standby database

Fast-Start Failover: ENABLED

Configuration Status:

SUCCESS

DGMGRL> 

DGMGRL> 

DGMGRL> show database db1tesp

Database - db1tesp

  Role:            PHYSICAL STANDBY

  Intended State:  APPLY-ON

  Transport Lag:   0 seconds

  Apply Lag:       0 seconds

  Real Time Query: ON

  Instance(s):

    db1tesp1 (apply instance)

    db1tesp2

Database Status:

SUCCESS

DGMGRL> show database db1tesd

Database - db1tesd

  Role:            PRIMARY

  Intended State:  TRANSPORT-ON

  Instance(s):

    db1tesd1

Database Status:

SUCCESS

DGMGRL> 

Getting time with the ping command

Problem:
Not a big problem but more like created by me. I just wanted to get more information when performing the ping. Sometime it is hand to get the time stamp when ping was performed. We get timestamp using -T in ping but we have to convert it before we can read it.

Fix:
Not a big deal but it can be done as below:

[root@rac1 ~]# ping rac2 | while read cmdoutput; do echo "$(date): $cmdoutput"; done
Mon Oct 17 11:25:28 AEDT 2016: PING rac2.localdomain (192.168.56.102) 56(84) bytes of data.
Mon Oct 17 11:25:28 AEDT 2016: 64 bytes from rac2.localdomain (192.168.56.102): icmp_seq=1 ttl=64 time=0.317 ms
Mon Oct 17 11:25:29 AEDT 2016: 64 bytes from rac2.localdomain (192.168.56.102): icmp_seq=2 ttl=64 time=0.584 ms
Mon Oct 17 11:25:30 AEDT 2016: 64 bytes from rac2.localdomain (192.168.56.102): icmp_seq=3 ttl=64 time=0.405 ms

DGM-17016: failed to retrieve status for database "db1tesd"

Problem:

Dataguard is show multiple error:
ORA-16810 ORA-16737 ORA-16662 DGM-17016
Error: ORA-16810: multiple errors or warnings detected for the database
Error: ORA-16662: network timeout when contacting a database
DGM-17016: failed to retrieve status for database "db1tesd"
DGMGRL> connect sys
Password:
Connected as SYSDG.
DGMGRL> show configuration
Configuration - db1tes_dg
  Protection Mode: MaxPerformance
  Members:
  db1tesp - Primary database
    Error: ORA-16810: multiple errors or warnings detected for the database
    db1tesd - Physical standby database
      Error: ORA-16662: network timeout when contacting a database
Fast-Start Failover: DISABLED
Configuration Status:
ERROR   (status updated 428 seconds ago)
DGMGRL>
DGMGRL> show database db1tesp
Database - db1tesp
  Role:               PRIMARY
  Intended State:     TRANSPORT-ON
  Instance(s):
    db1tesp1
      Error: ORA-16737: the redo transport service for standby database "db1tesd" has an error
    db1tesp2
      Error: ORA-16737: the redo transport service for standby database "db1tesd" has an error
Database Status:
ERROR
DGMGRL>
DGMGRL>
DGMGRL> show database db1tesd
Database - db1tesd
  Role:               PHYSICAL STANDBY
  Intended State:     APPLY-ON
  Transport Lag:      (unknown)
  Apply Lag:          (unknown)
  Average Apply Rate: (unknown)
  Real Time Query:    OFF
  Instance(s):
    db1tesd1
Database Status:
DGM-17016: failed to retrieve status for database "db1tesd"
ORA-16662: network timeout when contacting a database
DGMGRL>

Cause:

Reason in my case is that the DR host is shutdown and primary can not connect to it.


Fix:

Start the DR host and things start looking normal again.

ORA-19554: error allocating device, device type: SBT_TAPE, device name:

Problem:

Getting an error ORA-19554: error allocating device, device type: SBT_TAPE, device name: when trying to duplicate the database to standby database

RMAN> duplicate target database for standby from active database
2> spfile
3> set audit_file_dest='/u01/app/oracle/admin/db1tesd/adump'
4> set db_unique_name='db1tesd'
5> set fal_server='db1tesp1','db1tesp2'
6> set instance_number='1'
7> NOFILENAMECHECK;
Starting Duplicate Db at 17-OCT-16
using target database control file instead of recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 10/17/2016 02:30:46
RMAN-05501: aborting duplication of target database
RMAN-03009: failure of allocate command on ORA_AUX_SBT_TAPE_1 channel at 10/17/2016 02:30:46
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27211: Failed to load Media Management Library
Additional information: 2
RMAN>

Issue1:

The issue here is that SBT_TAPE variable is set and causing the problem.

CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 1 BACKUP TYPE TO BACKUPSET;

Fix 1:

Check the current value of SBT_TAPE

$ rman target /
RMAN> show DEVICE TYPE;
RMAN configuration parameters for database with db_unique_name DB1TESP are:
CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 1 BACKUP TYPE TO BACKUPSET;
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
RMAN>
RMAN> CONFIGURE DEVICE TYPE 'SBT_TAPE' clear;
old RMAN configuration parameters:
CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 1 BACKUP TYPE TO BACKUPSET;
RMAN configuration parameters are successfully reset to default value
RMAN>
RMAN> show DEVICE TYPE;
RMAN configuration parameters for database with db_unique_name DB1TESP are:
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
RMAN> 

Issue 2:

RMAN had below setting
CONFIGURE DEFAULT DEVICE TYPE TO 'SBT_TAPE';

Fix 2: 

Just clear this setting and try again
RMAN> CONFIGURE DEFAULT DEVICE TYPE clear;

old RMAN configuration parameters:
CONFIGURE DEFAULT DEVICE TYPE TO 'SBT_TAPE';
RMAN configuration parameters are successfully reset to default value

RMAN> show DEVICE TYPE;

RMAN configuration parameters for database with db_unique_name QA1CSMP are:
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default

RMAN>

RMAN>

RMAN> show default DEVICE TYPE;

RMAN configuration parameters for database with db_unique_name QA1CSMP are:
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default

RMAN>

Now you will not get the error !!! Happy Days :)

ORA-39701: database must be mounted EXCLUSIVE for UPGRADE or DOWNGRADE

I am trying to start the database in upgrade mode and received the error as below:

SQL> startup upgrade
ORACLE instance started.
Total System Global Area 3758096384 bytes
Fixed Size                  4503952 bytes
Variable Size            3103786608 bytes
Database Buffers          637534208 bytes
Redo Buffers               12271616 bytes
Database mounted.
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-39701: database must be mounted EXCLUSIVE for UPGRADE or DOWNGRADE
Process ID: 10523
Session ID: 2092 Serial number: 36861

SQL>

Cause:

The reason for getting this error message in my case was that I was trying to issue this command on RAC database which had two instances.

SQL> show parameter cluster_database
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
cluster_database                     boolean     TRUE
cluster_database_instances           integer     2
SQL>

Fix:

Just turn the database clustering off and try starting it again in upgrade mode

Start database as normal and alter the parameter as below:

SQL> alter system set cluster_database=FALSE scope=spfile sid='*' ;
System altered.
SQL>

Now shutdown the database and then try the startup upgrade

SQL> startup upgrade
ORACLE instance started.
Total System Global Area 3758096384 bytes
Fixed Size                  4503952 bytes
Variable Size            3003123312 bytes
Database Buffers          738197504 bytes
Redo Buffers               12271616 bytes
Database mounted.
Database opened.
SQL>

Hurray!!! about to hit 100,000 visitors mark

The blog was started with the intention that I would use this for my notes and share my learning with the word and it looks like that I am achieving the goal very well. So far my blog is able to attract about 100K clicks and I believe that these are 100K moments when someone got some sort of assistance from my blog.


Thanks for coming to my blog and giving me a opportunity to share my knowledge, experience and thought on databases and other technologies.

I have got a keen interest in mobile development and raspberry pi and from time to time I will keep on sharing information related to those areas as well.

This year I am trying to contribute maximum so far and planning to create the most posts compared to other years. I love your feedback and suggestions because those encourage me gives me direction to move ahead. Please keep on posting comments as that is one of the way I can interact with you guys.

Over the next year I would like to explore more on SQL Server, Big Data, MySQL and other databases.


Below are some of the statistics collected from Google Analytics over 6 years and those clearly shows what are the big markets for oracle database and related technologies and most of the search queries / sessions come from those top 5 countries. Based on the data I believe that oracle is very popular in US or may be people in US has more questions about oracle than any other country in the world. But I am glad to write that I was able to create an online information material which could have been a help to most of you.

I would like to state again that it will be great if you can please keep on commenting on my posts so that I can learn from your experiences as well and we all together can make the experiences more valuable and great for other community member.

I would be looking forward to achieve another 100K visits far quicker this time and I promise to keep on sharing my knowledge and experiences.

Manually configure target using emcli oracle 12c

If you want to use emcli and setup the target from command line then do the below:

Log in to OMS server in my case it is unix box and the go to OMS home and bin directory and in my case it is

/u01/app/oracle/product/oem12c/middleware1/oms/bin

then log in using sysman and do a sync

$ ./emcli login -username=sysman
Enter password :

Login successful

$ ./emcli sync
Synchronized successfully

$

after sync issue the command below to add each instance and then add it to the cluster.


$ ./emcli add_target -name=db2datp1 -type=oracle_database -host="host1" -credentials="UserName:dbsnmp;password:*********;Role:Normal" -properties="SID:db2datp1;Port:1521;OracleHome:/u01/app/oracle/product/12.1.0.2;MachineName:host1.example.com"

$ ./emcli add_target -name=db2datp2 -type=oracle_database -host="host2" -credentials="UserName:dbsnmp;password:*********;Role:Normal" -properties="SID:db2datp2;Port:1521;OracleHome:/u01/app/oracle/product/12.1.0.2;MachineName:host2.example.com"

$ ./emcli add_target -name="db2datp.example.com" -type="rac_database" -host="host1" -monitor_mode="1" -properties="ServiceName:db2dat_ha;ClusterName:rachost1" -instances="db2datp1:oracle_database;db2datp2:oracle_database"

Oracle 12c find the current Cluster Health Monitor (CHM) Repository master node in RAC

Log in as Grid user or the OS user use to manage GI and run the command below:


[grid@rac01 ~]$ oclumon manage -get master

Master = rac02
[grid@rac01 ~]$


[grid@rac01 ~]$ which oclumon
/u01/app/12.1.0/grid/bin/oclumon
[grid@rac01 ~]$

pse util to get the proccsses

Create the .aliases file

Added below lines in .aliases at oracle home directory in my case it was /home/oracle

alias pse='ps -ef | grep -v grep | grep '
alias psg='ps -ef | grep -v grep | grep '

Modify .bashrc

Modified the .bashrc file so that new .aliases is there. My file is 

[oracle@o12c (cdb1) ~]$ cat .bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
# User specific aliases and functions
if [ -f $HOME/.aliases ]
then
  . $HOME/.aliases
fi
[oracle@o12c (cdb1) ~]$ 

Now simply run pse command with process name it saves some time.

[oracle@o12c (cdb1) ~]$ pse tns
root        13     2  0 16:03 ?        00:00:00 [netns]
oracle    2758     1  0 16:05 ?        00:00:00 /u01/app/oracle/product/12.1.0.2/db_1/bin/tnslsnr listener -inherit
[oracle@o12c (cdb1) ~]$ 
[oracle@o12c (cdb1) ~]$ pse smon
oracle    2507     1  0 16:04 ?        00:00:00 ora_smon_cdb1

[oracle@o12c (cdb1) ~]$ 

Making up arrow key work for sqlplus, rman, dgmgrl, adrci

It is nice to have up arrow key working in sqlplus, rman, dgmgrl, adrci. But can this work.

YES !!! of course it can, I have tried it on Redhat.

Below are the steps to do this.

1. Download the rlwrap package 
2. Install the package
3. Create the .aliases file
4. Modify .bashrc
5. WALLAH !!!

Download the rlwrap package 

I have downloaded rlwrap-0.42.tar.gz from http://utopia.knoware.nl/~hlub/uck/rlwrap/#rlwrap
or download the file by clicking link below

rlwrap-0.42.tar.gz

Install the package

the installation is done using 

su -
gunzip rlwrap*.gz
tar -xvf rlwrap*.tar
cd rlwrap*
./configure
make
make check
make install

Create the .aliases file

Added below lines in .aliases at oracle home directory in my case it was /home/oracle

alias adrci='/usr/local/bin/rlwrap $ORACLE_HOME/bin/adrci'
alias sqlplus='/usr/local/bin/rlwrap $ORACLE_HOME/bin/sqlplus'
alias dgmgrl='/usr/local/bin/rlwrap $ORACLE_HOME/bin/dgmgrl'
alias rman='/usr/local/bin/rlwrap $ORACLE_HOME/bin/rman'

Modify .bashrc

Modified the .bashrc file so that new .aliases is there. My file is 

[oracle@o12c (cdb1) ~]$ cat .bashrc
# .bashrc
# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
fi
# User specific aliases and functions
if [ -f $HOME/.aliases ]
then
  . $HOME/.aliases
fi
[oracle@o12c (cdb1) ~]$ 

WALLAH !!!

Once above is done then create a new oracle connection to unix server and you are done. The up arrow key will start working.

Adding swap file

In case we need to increase the size of the swap we can add a file to the file system and use it as a swap

[root@o12c ~]# swapon -s
Filename                                Type            Size    Used    Priority
/dev/dm-1                               partition       4620284 0       -1
[root@o12c ~]#
[root@o12c ~]#
[root@o12c ~]# cat /proc/swaps
Filename                                Type            Size    Used    Priority
/dev/dm-1                               partition       4620284 0       -1
[root@o12c ~]#
[root@o12c ~]# cat /etc/fstab|grep swap
/dev/mapper/vg_o12c-lv_swap swap                    swap    defaults        0 0
[root@o12c ~]# 

[root@o12c ~]#
[root@o12c ~]# dd if=/dev/zero of=/root/swapfile count=1024 bs=4194304
1024+0 records in
1024+0 records out
4294967296 bytes (4.3 GB) copied, 33.7263 s, 127 MB/s
[root@o12c ~]# mkswap -c /root/swapfile
mkswap: /root/swapfile: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.
Setting up swapspace version 1, size = 4194300 KiB
no label, UUID=92497138-d397-412d-b827-f3c73c91047d
[root@o12c ~]# swapon /root/swapfile
[root@o12c ~]# swapon -s
Filename                                Type            Size    Used    Priority
/dev/dm-1                               partition       4620284 0       -1
/root/swapfile                          file            4194300 0       -2
[root@o12c ~]# 

xauth: file /home/oracle/.Xauthority does not exist

Another issue with xauth. :(

But good thing that I have got the solution.

Logged in as oracle buy xclock is not working

[oracle@o12c sf_vshare]$ xclock
SSH gateway: X11 authentication failed. Error: Can't open display: localhost:11.0
[oracle@o12c sf_vshare]$ echo $DISPLAY
localhost:11.0
[oracle@o12c sf_vshare]$ DISPLAY=localhost:10.0
[oracle@o12c sf_vshare]$ echo $DISPLAY
localhost:10.0
[oracle@o12c sf_vshare]$ xclock
SSH gateway: X11 authentication failed. Error: Can't open display: localhost:10.0
[oracle@o12c sf_vshare]$
[oracle@o12c sf_vshare]$ xauth list
xauth:  file /home/oracle/.Xauthority does not exist
[oracle@o12c sf_vshare]$

But it is working as root. let's see what are the values of display and xauth there

[root@o12c ~]# echo $DISPLAY
localhost:10.0
[root@o12c ~]#
[root@o12c ~]#
[root@o12c ~]# xauth list
o12c.localdomain/unix:12  MIT-MAGIC-COOKIE-1  fc6f5640fe846f9825cb0e121a303517
o12c.localdomain/unix:10  MIT-MAGIC-COOKIE-1  0a9f61e7313ebd9919b03dabe07f9697
o12c.localdomain/unix:11  MIT-MAGIC-COOKIE-1  d34c3a647ff47405f452c4910e7d9a39
[root@o12c ~]#

Now log back to oracle and set the following

[oracle@o12c ~]$ xauth add o12c.localdomain/unix:10  MIT-MAGIC-COOKIE-1  0a9f61e7313ebd9919b03dabe07f9697
xauth:  file /home/oracle/.Xauthority does not exist
[oracle@o12c ~]$ ls -lah /home/oracle/.Xauthority
-rw-------. 1 oracle oinstall 62 Sep 12 14:58 /home/oracle/.Xauthority
[oracle@o12c ~]$
[oracle@o12c ~]$
[oracle@o12c ~]$ xauth list
o12c.localdomain/unix:10  MIT-MAGIC-COOKIE-1  0a9f61e7313ebd9919b03dabe07f9697
[oracle@o12c ~]$
[oracle@o12c ~]$ xclock
SSH gateway: X11 authentication failed. Error: Can't open display: localhost:11.0
[oracle@o12c ~]$ DISPLAY=localhost:10.0
[oracle@o12c ~]$ xclock
Warning: Missing charsets in String to FontSet conversion
[oracle@o12c ~]$

So moral of story. match the number after : under xauth list with the DISPLAY variable plus run xauth add with values from the session where it is working for :10.0 and it should work.

ORA-01630: max # extents (4096) reached in temp segment in tablespace HARVEY

ERROR at line 1:
ORA-01630: max # extents (4096) reached in temp segment in tablespace HARVEY

The solution is

Get the maximum extents and either increase the value to something bigger or make it unlimited.

select tablespace_name, min_extents, max_extents from DBA_TABLESPACES order by 1;

SQL> alter tablespace HARVEY default storage (maxextents unlimited);

ORA-32000: write to SPFILE requested but SPFILE is not modifiable

While trying to alter a parameter getting the error below

SQL> alter system set db_recovery_file_dest_size=7G scope=both sid='*';
alter system set db_recovery_file_dest_size=7G scope=both sid='*'
*
ERROR at line 1:
ORA-32000: write to SPFILE requested but SPFILE is not modifiable


SQL>


if you look at alert you will see an error like this

Using parameter settings in server-side spfile +DATA/spfilecdbrac1.ora
WARNING: spfile modifications are disabled because multiple spfiles were used
System parameters with non-default values:

or

Using parameter settings in server-side pfile /u01/app/oracle/product/12.1.0.2/db_1/dbs/initcdbrac1.ora
WARNING: spfile modifications are disabled because multiple spfiles were used
System parameters with non-default values:

or

Using parameter settings in server-side spfile +DATA/cdbrac/spfilecdbracharv.ora
WARNING: spfile modifications are disabled because multiple spfiles were used
System parameters with non-default values:

depending upon your environment.

The first thing you should do is

[oracle@rac1 admin]$ srvctl config database -d cdbracp
Database unique name: cdbracp
Database name: cdbrac
Oracle home: /u01/app/oracle/product/12.1.0.2/db_1
Oracle user: oracle
Spfile: +DATA/cdbrac/parameterfile/spfilecdbracharv.ora
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA,FRA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: dba
Database instances: cdbrac1,cdbrac2
Configured nodes: rac1,rac2
Database is administrator managed
[oracle@rac1 admin]$

Now startup nomount the database to get the spfile contents. In mine case there were two different files so what I did was

SQL> create pfile='/u01/app/oracle/product/12.1.0.2/db_1/dbs/my.ora' from spfile='+DATA/spfilecdbrac1.ora';

File created.

SQL>

[oracle@rac1 dbs]$ cat my.ora
*.SPFILE='+DATA/cdbrac/spfilecdbracharv.ora'
[oracle@rac1 dbs]$

SQL> create pfile='/u01/app/oracle/product/12.1.0.2/db_1/dbs/my2.ora' from spfile='+DATA/cdbrac/spfilecdbracharv.ora';

File created.

SQL>

[oracle@rac1 dbs]$ cat my2.ora
*.SPFILE='+DATA/cdbrac/spfilecdbrac.ora'
[oracle@rac1 dbs]$


SQL> create pfile='/u01/app/oracle/product/12.1.0.2/db_1/dbs/my3.ora' from spfile='+DATA/cdbrac/spfilecdbrac.ora';

File created.

SQL>

[oracle@rac1 dbs]$ cat my3.ora
cdbrac2.__data_transfer_cache_size=0
cdbrac1.__data_transfer_cache_size=0
cdbrac2.__db_cache_size=956301312
cdbrac1.__db_ca
.
.
.


You might have to do this till you see the file which holds the parameters.

Now we know that the parameter file is at +DATA/cdbrac/spfilecdbrac.ora which has all parameters.

Next thing you make sure that there is no other parameter file in use and configure this parameter file as main


[oracle@rac1 admin]$ srvctl modify database -d cdbracp -p +DATA/cdbrac/spfilecdbrac.ora
[oracle@rac1 admin]$ srvctl config database -d cdbracp
Database unique name: cdbracp
Database name: cdbrac
Oracle home: /u01/app/oracle/product/12.1.0.2/db_1
Oracle user: oracle
Spfile: +DATA/cdbrac/spfilecdbrac.ora
Password file:
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools:
Disk Groups: DATA,FRA
Mount point paths:
Services:
Type: RAC
Start concurrency:
Stop concurrency:
OSDBA group: dba
OSOPER group: dba
Database instances: cdbrac1,cdbrac2
Configured nodes: rac1,rac2
Database is administrator managed
[oracle@rac1 admin]$

Now you can start the database without any issues and change parameter

[oracle@rac1 admin]$ srvctl start database -d cdbracp
[oracle@rac1 admin]$

SQL> alter system set db_recovery_file_dest_size=7G scope=both sid='*';

System altered.

SQL>

Setting up git on window machine

I am trying to setup git on my windows machine and trying to connect to bitbucket.org to access repository.

When I am testing the connection I am getting the error


harvey@HARVEY-L MINGW64 /d/HarveyGitTest (master)
$ ssh -T git@bitbucket.org
Warning: Permanently added the RSA host key for IP address '104.192.143.2' to the list of known hosts.
Permission denied (publickey).

harvey@HARVEY-L MINGW64 /d/HarveyGitTest (master)

What is going on here ?

I will give you a background before I move ahead. I have installed 64bit git client for windows and using its command line.

The command line opens using Git Bash utility.

The problem is that I need to add my local public key on to bitbucket.

1. Generating ssh key pair

Open another session of Git Bash and

cd .ssh

see if you can find id_rsa and id_rsa.pub file.

id_rsa -->> This file if exists contains my private key
id_rsa.pub -->> This file if exists contains my public key

In case the files do not exist then generate them using command

ssh-keygen -t rsa

and then follow the prompts. Now you would be able to see both of these files id_rsa and id_rsa.pub.

Now get the contents of id_rsa.pub using

cat id_rsa.pub

copy the contents and then go to bitbucket.org

On right side click on your icon >> View Profile.

you will see a button "Bitbucket Settings" under there on the page, click it.

One the left side click on SSH Keys >> Add Key

Enter Label and paste the key and click Add Key to save it.

Now go back to Git Bash where you were getting the error and run the command below:

 ssh -T git@bitbucket.org

harvey@HARVEY-L MINGW64 /d/HarveyGitTest (master)
$ ssh -T git@bitbucket.org
logged in as harv.

You can use git or hg to connect to Bitbucket. Shell access is disabled.

harvey@HARVEY-L MINGW64 /d/HarveyGitTest (master)

Enjoy!!!

Simplest way to setup WiFi on Raspberry PI using command line

Most of the time we access Raspberry PI from command line and in this case if we have to connect PI to a network then it might be bit tricky. The easiest way is to modify the file and add bits from network.

In our case the SSID is TestWiF, the password for the wifi is P455WOr6 and key_management is WPA-PSK

pi@raspberrypi:~ $ sudo cat /etc/wpa_supplicant/wpa_supplicant.conf 

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev

update_config=1

country=GB

network={

ssid="TestWiFi"

psk="P455WOr6"

key_mgmt=WPA-PSK

}

pi@raspberrypi:~ $ 

Once the modifications are done just restart the PI

pi@raspberrypi:~ $ sudo shutdown -r now

Test SQL Server database connectivity from local machine

Simplest way is create an empty file by right click and new . Name it to something .UDL

Now double click on it and it should open something like below now enter the details and test connection

In sqlite re-organizing the database

SQLite is like a big file and has tables inside that database file. Some times there is a need to delete a large number of records from the table in that case the records are deleting and leaving empty pockets behind in that large file. After the deletion the file size still remains same.

But there is a way to reduce the database file size and that can be achieved using a command VACUUM;

Below is the example on sqlite to reduce the size of database file to what is used and omitting the blank pockets out.

$ ls -l
-rw-rw-r--  1 rag rag 1627136 Jan  1 15:00 templog.db

$ sqlite3 templog.db
sqlite> . timer on
sqlite>
sqlite> vacuum;
CPU Time: user 0.360000 sys 0.250000
sqlite>

$ ls -l templog.db
-rw-rw-r-- 1 rag rag 399360 Jan  1 15:15 templog.db

you can see that the size of the database file is reduced to what is under use at the moment.

ORA-01153: an incompatible media recovery is active

Want to enable flashback on DR but getting the error. 

Reason is because MRP process is running. So we stop MRP process and then try enabling flashback. After that start MRP process.


Stop MRP:

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
Database altered.
SQL>

Enable flashback:

SQL> ALTER DATABASE FLASHBACK ON;
Database altered.
SQL>

Start MRP:

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;
Database altered.
SQL>

All Steps:

SQL> select flashback_on from v$database;
FLASHBACK_ON
------------------
NO
SQL>
SQL> ALTER DATABASE FLASHBACK ON;
ALTER DATABASE FLASHBACK ON
*
ERROR at line 1:
ORA-01153: an incompatible media recovery is active

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
Database altered.
SQL> ALTER DATABASE FLASHBACK ON;
Database altered.
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;
Database altered.
SQL> select flashback_on from v$database;
FLASHBACK_ON
------------------
YES
SQL> 

Done. 

Get what proxy setting the database is using

SQL >

var proxy varchar2(100)
var no_proxy varchar2(100)
exec utl_http.get_proxy(:proxy,:no_proxy) ;
print proxy
print no_proxy

ORA-01565: error in identifying file '+DATA/cdbrac/spfilecdbrac.ora'

[oracle@rac1 ~]$ srvctl start database -d cdbrac
PRCR-1079 : Failed to start resource ora.cdbrac.db
CRS-5017: The resource action "ora.cdbrac.db start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/cdbrac/spfilecdbrac.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/cdbrac/spfilecdbrac.ora
ORA-15056: additional error message
ORA-17503: ksfdopn:2 Failed to open file +DATA/cdbrac/spfilecdbrac.ora
ORA-15173: entry 'spfilecdbrac.ora' does not exist in directory 'cdbrac'
ORA-06512: at line 4
. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/rac1/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.cdbrac.db' on 'rac1' failed
CRS-5017: The resource action "ora.cdbrac.db start" encountered the following error:
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/cdbrac/spfilecdbrac.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/cdbrac/spfilecdbrac.ora
ORA-15056: additional error message
ORA-17503: ksfdopn:2 Failed to open file +DATA/cdbrac/spfilecdbrac.ora
ORA-15173: entry 'spfilecdbrac.ora' does not exist in directory 'cdbrac'
ORA-06512: at line 4
. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/rac2/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.cdbrac.db' on 'rac2' failed
CRS-2632: There are no more servers to try to place resource 'ora.cdbrac.db' on that would satisfy its placement policy
[oracle@rac1 ~]$


Do not worry log on to asm and locate where the spfile is

ASMCMD> cd +DATA/CDBRAC/PARAMETERFILE/
ASMCMD> ls -l
Type           Redund  Striped  Time             Sys  Name
PARAMETERFILE  UNPROT  COARSE   APR 29 13:00:00  Y    spfile.271.909835045
ASMCMD>

Now you want this under +DATA/CDBRAC as per the error so create an alias in ASM

ASMCMD> pwd
+DATA/CDBRAC
ASMCMD>
ASMCMD> mkalias +DATA/cdbrac/PARAMETERFILE/spfile.271.909835045 spfilecdbrac.ora
ASMCMD> ls -l
Type           Redund  Striped  Time             Sys  Name
                                                 Y    310A14D691145E69E0536538A8C0D511/
                                                 Y    310A3223A63F6D46E0536538A8C02A10/
                                                 Y    CONTROLFILE/
                                                 Y    DATAFILE/
                                                 Y    FD9AC20F64D244D7E043B6A9E80A2F2F/
                                                 Y    ONLINELOG/
                                                 Y    PARAMETERFILE/
                                                 Y    PASSWORD/
                                                 Y    TEMPFILE/
PARAMETERFILE  UNPROT  COARSE   APR 29 13:00:00  N    spfilecdbrac.ora => +DATA/CDBRAC/PARAMETERFILE/spfile.271.909835045
ASMCMD>

[oracle@rac1 ~]$ srvctl start database -d cdbrac
[oracle@rac1 ~]$

Worked like a charm !!!

PRVF-0002 : could not retrieve local node name

[oracle@racdr1 grid]$ ./runcluvfy.sh  stage -pre crsinst -n racdr1 -verbose

WARNING:
Could not access or create trace file path "/tmp/bootstrap/cv/log". Trace information could not be collected

PRVF-0002 : could not retrieve local node name
[oracle@racdr1 grid]$ hostname
racdr1.localdomain
[oracle@racdr1 grid]$

Getting above error while doing the cluster verify for oracle 12c GI.

For diagnosing I have done

[oracle@racdr1 grid]$ hostname --long
hostname: Unknown host
[oracle@racdr1 grid]$
[oracle@racdr1 grid]$
[oracle@racdr1 grid]$ hostname --long --verbose
gethostname()=`racdr1.localdomain'
Resolving `racdr1.localdomain' ...
hostname: Unknown host
[oracle@racdr1 grid]$

It should not say "Unknown host"

Lets check /etc/hosts file

[root@racdr1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
[root@racdr1 ~]#


what it turned out was the my dns setting were wrong basically once you change in /etc/resolv.conf then do the following so that it does not change after reboot.

[root@racdr1 ~]# chattr -i /etc/resolv.conf
update details and then
[root@racdr1 ~]# chattr +i /etc/resolv.conf


[oracle@racdr1 grid]$ hostname --long
racdr1.localdomain
[oracle@racdr1 grid]$
[oracle@racdr1 grid]$ hostname --long --verbose
gethostname()=`racdr1.localdomain'
Resolving `racdr1.localdomain' ...
Result: h_name=`racdr1.localdomain'
Result: h_addr_list=`192.168.56.111'
racdr1.localdomain
[oracle@racdr1 grid]$

Hurray !!!

GUI is not working on Linux RedHat 6.7 and 6.8 after yum update machine on virtual box

I have recently done yum update on my virtual machine running from Virtual box with RedHat Linux 6.7 and 6.8 .

The out come was as below, GUI was blank.

Looks like some issues in X. The solution is simple just move the xorg.conf file as below and restart the virtual machine.

[root@racdr1 ~]# ls -lah /etc/X11/xorg.conf

-rw-r--r--. 1 root root 534 Apr 22 16:07 /etc/X11/xorg.conf

[root@racdr1 ~]#

[root@racdr1 ~]#

[root@racdr1 ~]#

[root@racdr1 ~]# cat /etc/X11/xorg.conf

# VirtualBox generated configuration file

# based on /etc/X11/xorg.conf.

Section "Monitor"

  Identifier   "Monitor[0]"

  ModelName    "VirtualBox Virtual Output"

  VendorName   "Oracle Corporation"

EndSection

Section "Device"

  BoardName    "VirtualBox Graphics"

  Driver       "vboxvideo"

  Identifier   "Device[0]"

  VendorName   "Oracle Corporation"

EndSection

Section "Screen"

  SubSection "Display"

    Depth      24

  EndSubSection

  Device       "Device[0]"

  Identifier   "Screen[0]"

  Monitor      "Monitor[0]"

EndSection

[root@racdr1 ~]#

[root@racdr1 ~]#

[root@racdr1 ~]# mv /etc/X11/xorg.conf /etc/X11/xorg.conf.old

[root@racdr1 ~]# shutdown -r now

Oracle remap_tablespace

There are instances in DBA's life when we have to perform and export and import of some data. While doing the import we want to move the data or new tablespace this can be done by remap_tablespace parameter in impdp.

Example:

Suppose there is a database DB1 which has a table ITEM owned by USER1 but the tablespace for this table is MYUSERS . Which means that location where this table is stored physically is MYUSERS tablespace.

Database: DB1
TableName: ITEM
Owner of table: USER1
Tablespace of table: MYUSERS

Now you want to perform the export of that table and import into another database NEWDB under MYNEWUSERS tablespace.

This can be done as below:

• Perform the export as usual.
• While doing the import include this parameter EMAP_TABLESPACE=MYUSER:MYNEWUSERS

Outcome will be
Table Name: ITEM
Owner : USER1
Tablespace under NEWDB: MYNEWUSERS

Cannot verify the shared state for device /dev/oracleasm/disks/SYSCFG1 due to Universally Unique Identifiers (UUIDs) not being found, or different values being found, for this device across nodes

While doing the installation of oracle 12GI on redhat I was getting a warning in the installation.

Please note that I am using Oracle Virtual Box

The details are: 5.0.18r106667
Oracle: 12.0.1.2
Linux: Redhat 6.3 x86_64

Please note that it was not showing the candidates properly I had to manually add the path

Device Checks for ASM - This is a prerequisite check to verify that the specified devices meet the requirements for ASM.
  Check Failed on Nodes: [rac2,  rac1]
Verification result of failed node: rac2
Expected Value
 : cvuqdisk-1.0.9-1
Actual Value
 : cvuqdisk-1.0.9-1
 Details:
 -
Cannot verify the shared state for device /dev/oracleasm/disks/SYSCFG3 due to Universally Unique Identifiers (UUIDs) not being found, or different values being found, for this device across nodes: [rac1, rac2]  - Cause: Cause Of Problem Not Available  - Action: User Action Not Available
 -
Cannot verify the shared state for device /dev/oracleasm/disks/SYSCFG2 due to Universally Unique Identifiers (UUIDs) not being found, or different values being found, for this device across nodes: [rac1, rac2]  - Cause: Cause Of Problem Not Available  - Action: User Action Not Available
 -
Cannot verify the shared state for device /dev/oracleasm/disks/SYSCFG1 due to Universally Unique Identifiers (UUIDs) not being found, or different values being found, for this device across nodes: [rac1, rac2]  - Cause: Cause Of Problem Not Available  - Action: User Action Not Available
 -
PRVF-9802 : Attempt to get 'udev' information from node "rac2" failed No UDEV rule found for device(s) specified  - Cause:  An attempt to read the ''udev'' permission or rule file failed, or the permission or rule file did not contain any rules for the specified device or devices.  - Action:  Make sure that the ''udev'' permission or rule directory is created, the ''udev'' permission or rule file is available and accessible by the user running the check and that the permission or rule file contains the correct ''udev'' rule for the device or devices being checked.
Back to Top
Verification result of failed node: rac1
Expected Value
 : cvuqdisk-1.0.9-1
Actual Value
 : cvuqdisk-1.0.9-1
 Details:
 -
Cannot verify the shared state for device /dev/oracleasm/disks/SYSCFG3 due to Universally Unique Identifiers (UUIDs) not being found, or different values being found, for this device across nodes: [rac1, rac2]  - Cause: Cause Of Problem Not Available  - Action: User Action Not Available
 -
Cannot verify the shared state for device /dev/oracleasm/disks/SYSCFG2 due to Universally Unique Identifiers (UUIDs) not being found, or different values being found, for this device across nodes: [rac1, rac2]  - Cause: Cause Of Problem Not Available  - Action: User Action Not Available
 -
Cannot verify the shared state for device /dev/oracleasm/disks/SYSCFG1 due to Universally Unique Identifiers (UUIDs) not being found, or different values being found, for this device across nodes: [rac1, rac2]  - Cause: Cause Of Problem Not Available  - Action: User Action Not Available
 -
PRVF-9802 : Attempt to get 'udev' information from node "rac1" failed No UDEV rule found for device(s) specified  - Cause:  An attempt to read the ''udev'' permission or rule file failed, or the permission or rule file did not contain any rules for the specified device or devices.  - Action:  Make sure that the ''udev'' permission or rule directory is created, the ''udev'' permission or rule file is available and accessible by the user running the check and that the permission or rule file contains the correct ''udev'' rule for the device or devices being checked.

Why it was occurring:

For some reason oracleasm was not able to see the disk consistently across nodes.

Fix: In order to fix it have done below:

yum update -y

and new version of linux is now "Red Hat Enterprise Linux Server release 6.7 (Santiago)" 

Made sure that below is installed across nodes for oracleasm

kmod-oracleasm-2.0.8-6.el6_7.x86_64

oracleasmlib-2.0.4-1.el6.x86_64.rpm from here 

oracleasm-support-2.1.8-1.el6.x86_64.rpm from here

On Virtual box all the asm disks are SATA and shared.

After these steps I could see GI install able to see the candidate disks

below is how the VM is configured in Virtual Box

Simple steps to re-use the disk for asm using oracleasm

log in as root

dd if=/dev/zero of=/dev/sdb1 bs=1024 count=100

oracleasm deletedisk ASM1

oracleasm createdisk ASM1 /dev/sdb1

oracleasm scandisks

oracleasm listdisks

PRVG-10122 : ASMLib configuration value set to configuration parameter "ORACLEASM_UID" on the node "XXXXXX" does not match with cluster nodes

Checking ASMLib configuration.
  Node Name                             Status                
  ------------------------------------  ------------------------
  rac1                                  (failed) ASMLib configuration is incorrect.
  rac2                                  (failed) ASMLib configuration is incorrect.

ERROR:
PRVG-10122 : ASMLib configuration value set to configuration parameter "ORACLEASM_UID" on the node "rac1" does not match with cluster nodes
PRVG-10122 : ASMLib configuration value set to configuration parameter "ORACLEASM_GID" on the node "rac1" does not match with cluster nodes
PRVG-10122 : ASMLib configuration value set to configuration parameter "ORACLEASM_ENABLED" on the node "rac1" does not match with cluster nodes
PRVG-10122 : ASMLib configuration value set to configuration parameter "ORACLEASM_UID" on the node "rac2" does not match with cluster nodes
PRVG-10122 : ASMLib configuration value set to configuration parameter "ORACLEASM_GID" on the node "rac2" does not match with cluster nodes
PRVG-10122 : ASMLib configuration value set to configuration parameter "ORACLEASM_ENABLED" on the node "rac2" does not match with cluster nodes
Result: Check for ASMLib configuration failed.


There are few things we can do. First is run oracleasm configure on both nodes and see what are you getting.

In my case I was getting same results, not much help !

[root@rac1 ~]# oracleasm configure
ORACLEASM_ENABLED=true
ORACLEASM_UID=oracle
ORACLEASM_GID=dba
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER=""
ORACLEASM_SCANEXCLUDE=""
ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"
[root@rac1 ~]#

[root@rac2 ~]# oracleasm configure
ORACLEASM_ENABLED=true
ORACLEASM_UID=oracle
ORACLEASM_GID=dba
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER=""
ORACLEASM_SCANEXCLUDE=""
ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"
[root@rac2 ~]#


Next steps is cat oracleasm

[root@rac1 ~]# cat /etc/sysconfig/oracleasm
#
# This is a configuration file for automatic loading of the Oracle
# Automatic Storage Management library kernel driver.  It is generated
# By running /etc/init.d/oracleasm configure.  Please use that method
# to modify this file
#

# ORACLEASM_ENABLED: 'true' means to load the driver on boot.
ORACLEASM_ENABLED=false

# ORACLEASM_UID: Default UID owning the /dev/oracleasm mount point.
ORACLEASM_UID=

# ORACLEASM_GID: Default GID owning the /dev/oracleasm mount point.
ORACLEASM_GID=

# ORACLEASM_SCANBOOT: 'true' means fix disk perms on boot
ORACLEASM_SCANBOOT=true

# ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block
# size reported by the underlying disk instead of the physical. The
# default is 'false'
ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false
[root@rac1 ~]#

[root@rac2 ~]# cat /etc/sysconfig/oracleasm
#
# This is a configuration file for automatic loading of the Oracle
# Automatic Storage Management library kernel driver.  It is generated
# By running /etc/init.d/oracleasm configure.  Please use that method
# to modify this file
#

# ORACLEASM_ENABLED: 'true' means to load the driver on boot.
ORACLEASM_ENABLED=true

# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.
ORACLEASM_UID=oracle

# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.
ORACLEASM_GID=dba

# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.
ORACLEASM_SCANBOOT=true

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER=""

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE=""

# ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size
# reported by the underlying disk instead of the physical. The default
# is 'false'
ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false

[root@rac2 ~]#



Looks like node rac1 has some issues as it is missing the values.

List oracleasm under /etc/sysconfig

[root@rac1 ~]# cd /etc/sysconfig/
[root@rac1 sysconfig]#
[root@rac1 sysconfig]# ls -lah oracle*
-rw-r--r--. 1 root root 772 Feb  9  2013 oracleasm
-rw-r--r--. 1 root root 973 Apr 20 10:36 oracleasm-_dev_oracleasm
lrwxrwxrwx. 1 root root  24 Apr 20 09:29 oracleasm.rpmsave -> oracleasm-_dev_oracleasm
[root@rac1 sysconfig]#



[root@rac2 ~]#  cd /etc/sysconfig/
[root@rac2 sysconfig]#
[root@rac2 sysconfig]# ls -lah oracle*
lrwxrwxrwx. 1 root root  24 Apr 20 10:40 oracleasm -> oracleasm-_dev_oracleasm
-rw-r--r--. 1 root root 973 Apr 20 10:41 oracleasm-_dev_oracleasm
[root@rac2 sysconfig]#


Perform below:

[root@rac1 sysconfig]# mv oracleasm oracleasm.old
[root@rac1 sysconfig]# ln -s oracleasm-_dev_oracleasm oracleasm

[root@rac1 sysconfig]# ls -lah oracle*
lrwxrwxrwx. 1 root root  24 Apr 20 12:59 oracleasm -> oracleasm-_dev_oracleasm
-rw-r--r--. 1 root root 973 Apr 20 10:36 oracleasm-_dev_oracleasm
-rw-r--r--. 1 root root 772 Feb  9  2013 oracleasm.old
lrwxrwxrwx. 1 root root  24 Apr 20 09:29 oracleasm.rpmsave -> oracleasm-_dev_oracleasm
[root@rac1 sysconfig]#


[root@rac1 ~]# cat /etc/sysconfig/oracleasm
#
# This is a configuration file for automatic loading of the Oracle
# Automatic Storage Management library kernel driver.  It is generated
# By running /etc/init.d/oracleasm configure.  Please use that method
# to modify this file
#

# ORACLEASM_ENABLED: 'true' means to load the driver on boot.
ORACLEASM_ENABLED=true

# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.
ORACLEASM_UID=oracle

# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.
ORACLEASM_GID=dba

# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.
ORACLEASM_SCANBOOT=true

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning
ORACLEASM_SCANORDER=""

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan
ORACLEASM_SCANEXCLUDE=""

# ORACLEASM_USE_LOGICAL_BLOCK_SIZE: 'true' means use the logical block size
# reported by the underlying disk instead of the physical. The default
# is 'false'
ORACLEASM_USE_LOGICAL_BLOCK_SIZE=false

[root@rac1 ~]#


we can also remove oracleasm.old and oracleasm.rpmsave

The issue is fixed now.

PRCD-1084 : Failed to start service tstdb_ha

Received the error while starting the database service.

[oracle@host01 (tstdbp1) ~]$ srvctl start service -d tstdbp -s tstdb_ha
PRCD-1084 : Failed to start service tstdb_ha
PRCR-1079 : Failed to start resource ora.tstdbp.tstdb_ha.svc
CRS-2800: Cannot start resource 'ora.tstdbp.db' as it is already in the INTERMEDIATE state on server 'host02'
CRS-2632: There are no more servers to try to place resource 'ora.tstdbp.tstdb_ha.svc' on that would satisfy its placement policy
CRS-2800: Cannot start resource 'ora.tstdbp.db' as it is already in the INTERMEDIATE state on server 'host01'
[oracle@host01 (tstdbp1) ~]$

When I had a look at clusterware resources I could see what was happening

[grid@host01 ~]$ crsctl stat res -t
ora.tstdbp.db
      1        ONLINE  INTERMEDIATE host01              Stuck Archiver,STABLE
      2        ONLINE  INTERMEDIATE host02              Stuck Archiver,STABLE
ora.tstdbp.tstdb_ha.svc
      1        ONLINE  OFFLINE                               STABLE
      2        ONLINE  OFFLINE                               STABLE

Looks like the archiver is stuck and alert log explains it.

alert log says
ORA-19815: WARNING: db_recovery_file_dest_size of 21474836480 bytes is 100.00% used, and has 0 remaining bytes available.

added a bit of space and then ran a full backup to clear the archive logs.

You can look at show all under RMAN to see what will work for you.

MRP0: Background Media Recovery terminated with error 742

after starting the DR database using srvctl start database, alter log showing that background Media Recovery is terminated with error 742

what I have found is that it is more like a warning rather than an actual error.

This will happen if you do the following when recovering standby database.

At command "ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;" what we are saying is cancel the standby recovery. so when you will start DR database using srvctl you will get that warning means real time apply is not happening.

SQL> startup nomount
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.
Total System Global Area 5344731136 bytes
Fixed Size                  2262656 bytes
Variable Size            4412410240 bytes
Database Buffers          922746880 bytes
Redo Buffers                7311360 bytes
SQL> alter database mount standby database;
Database altered.
SQL> alter database open read only;
Database altered.
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;
Database altered.
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
Database altered.
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>
[oracle@testdb01 (db1tesd) bin]$
[oracle@testdb01 (db1tesd) bin]$
[oracle@testdb01 (db1tesd) bin]$ srvctl start database -d db1tesd
[oracle@testdb01 (db1tesd) bin]$ 

To fix this and start real time apply issue the following command and test using srvctl stop and start on DR.

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;
Database altered.
SQL> 

ORA-10458: standby database requires recovery

For some reason when I was trying to start me DR instance I was getting the error that need recovery for the database. Something like this:

oracle@testdb01 (db1tesd) bin]$ srvctl start database -d db1tesd
PRCR-1079 : Failed to start resource ora.db1tesd.db
CRS-5017: The resource action "ora.db1tesd.db start" encountered the following error:
ORA-10458: standby database requires recovery
ORA-01196: file 1 is inconsistent due to a failed media recovery session
ORA-01110: data file 1: '+DATA/db1tesd/datafile/system.284.881230789'
. For details refer to "(:CLSN00107:)" in "/u01/app/11.2.0/grid/log/testdb01/agent/ohasd/oraagent_grid//oraagent_grid.log".
CRS-2674: Start of 'ora.db1tesd.db' on 'testdb01' failed
[oracle@testdb01 (db1tesd) bin]$ 

In order to fix the issue what I did was started standby in nomount

• mount as standby
• open database in read only
• recovery managed standby database using current logfile 

Actual steps:

SQL> startup nomount
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.
Total System Global Area 5344731136 bytes
Fixed Size                  2262656 bytes
Variable Size            4412410240 bytes
Database Buffers          922746880 bytes
Redo Buffers                7311360 bytes
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT
*
ERROR at line 1:
ORA-01507: database not mounted

SQL> alter database mount standby database;
Database altered.
SQL> alter database open read only;
Database altered.
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;
Database altered.
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
Database altered.
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>
[oracle@testdb01 (db1tesd) bin]$
[oracle@testdb01 (db1tesd) bin]$
[oracle@testdb01 (db1tesd) bin]$ srvctl start database -d db1tesd
[oracle@testdb01 (db1tesd) bin]$ 

Or make sure that log shipping process is enabled on primary in DGMGRL by issuing below on primary

alter system set log_archive_dest_state_2=enable scope=both sid='*';

below on DR:

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;