Sunday, March 3, 2013

My RAC have lots of room for the improvement and learning


Below is the example on how the raccheck utility will run and what kind of output we get out of it. This is amazing.

[oracle@node01 Desktop]$ ./raccheck -a

CRS stack is running and CRS_HOME is not set. Do you want to set CRS_HOME to /u01/app/11.2.0/grid?[y/n][y]y

Checking ssh user equivalency settings on all nodes in cluster

Node node02 is configured for ssh user equivalency for oracle user


Searching for running databases . . . . .

.
List of running databases registered in OCR
1. RAC
2. None of above

Select databases from list for checking best practices. For multiple databases, select 1 for All or comma separated number like 1,2 etc [1-2][1].1
. .


Checking Status of Oracle Software Stack - Clusterware, ASM, RDBMS

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
-------------------------------------------------------------------------------------------------------
                                                 Oracle Stack Status                          
-------------------------------------------------------------------------------------------------------
Host Name  CRS Installed  ASM HOME       RDBMS Installed  CRS UP    ASM UP    RDBMS UP  DB Instance Name
-------------------------------------------------------------------------------------------------------
node01      Yes             Yes             Yes             Yes        Yes      Yes      RAC1    
node02      Yes             Yes             Yes             Yes        Yes      Yes      RAC2    
-------------------------------------------------------------------------------------------------------

91 of the included audit checks require root privileged data collection . If sudo is not configured or the root password is not available, audit checks which  require root privileged data collection can be skipped.


1. Enter 1 if you will enter root password for each  host when prompted

2. Enter 2 if you have sudo configured for oracle user to execute root_raccheck.sh script

3. Enter 3 to skip the root privileged collections

4. Enter 4 to exit and work with the SA to configure sudo  or to arrange for root access and run the tool later.

Please indicate your selection from one of the above options[1-4][1]:- 1

*** Checking Best Practice Recommendations (PASS/WARNING/FAIL) ***



Log file for collections and audit checks are at
/home/oracle/Desktop/raccheck_030313_004245/raccheck.log

=============================================================
                    Node name - node01                              
=============================================================
Collecting - ASM DIsk I/O stats
Collecting - ASM Disk Groups
Collecting - ASM Diskgroup Attributes
Collecting - ASM disk partnership imbalance
Collecting - ASM diskgroup attributes
Collecting - ASM initialization parameters
Collecting - Active sessions load balance for RAC database
Collecting - Archived Destination Status for RAC database
Collecting - Cluster Interconnect Config for RAC database
Collecting - Database Archive Destinations for RAC database
Collecting - Database Files for RAC database
Collecting - Database Instance Settings for RAC database
Collecting - Database Parameters for RAC database
Collecting - Database Properties for RAC database
Collecting - Database Registry for RAC database
Collecting - Database Sequences for RAC database
Collecting - Database Undocumented Parameters for RAC database
Collecting - Database Workload Services for RAC database
Collecting - Dataguard Status for RAC database
Collecting - Files not opened by ASM
Collecting - Log Sequence Numbers for RAC database
Collecting - Percentage of asm disk  Imbalance
Collecting - Process for shipping Redo to standby for RAC database
Collecting - Redo Log information for RAC database
Collecting - Standby redo log creation status before switchover for RAC database
Collecting - CPU Information
Collecting - CRS active version
Collecting - CRS oifcfg
Collecting - CRS software version
Collecting - CSS Reboot time
Collecting - CSS disktimout
Collecting - Cluster interconnect (clusterware)
Collecting - Clusterware OCR healthcheck
Collecting - Clusterware Resource Status
Collecting - Huge pages configuration
Collecting - Kernel parameters
Collecting - Linux module config.
Collecting - Maximum number of semaphore sets on system
Collecting - Maximum number of semaphores on system
Collecting - Maximum number of semaphores per semaphore set
Collecting - Memory Information
Collecting - OS Packages
Collecting - Operating system release information and kernel version
Collecting - Oracle Executable Attributes
Collecting - Patches for Grid Infrastructure
Collecting - Patches for RDBMS Home
Collecting - Shared memory segments
Collecting - Table of file system defaults
Collecting - Voting disks (clusterware)
Collecting - number of semaphore operations per semop system call
Preparing to run root privileged commands  node01.  Please enter root password when prompted.
root@node01's password:
Collecting - CRS user limits configuration
Collecting - CRS user time zone check
Collecting - Custom rc init scripts (rc.local)
Collecting - Interconnect interface config
Collecting - Network interface stats
Collecting - OCFS2 disks
/tmp/root_raccheck.sh: line 34: /sbin/mounted.ocfs2: No such file or directory
Collecting - Root Open File Limit
Collecting - ocsf status
/tmp/root_raccheck.sh: line 44: /etc/init.d/o2cb: No such file or directory
Collecting - root time zone check


Data collections completed. Checking best practices on node01.
--------------------------------------------------------------------------------------


 WARNING => OCR is NOT being backed up daily
 INFO =>    $CRS_HOME/log/hostname/client directory has too many older log files.
 INFO =>    user_dump_dest has trace files older than 30 days for RAC
 INFO =>    background_dump_dest has files older than 30 days for RAC
 INFO =>    At some times checkpoints are not being completed for RAC
 WARNING => Open files limit (ulimit -n) for current user is NOT set to recommended value >= 65536 or unlimited
 INFO =>    audit_file_dest has audit files older than 30 days for RAC
 WARNING => Controlfile is NOT multiplexed for RAC
 WARNING => One or more redo log groups are NOT multiplexed for RAC
 WARNING => /tmp is NOT on a dedicated filesystem
 WARNING => kernel.shmmax parameter is NOT configured according to recommendation
 WARNING => Open file limit for root user (ulimit -n) is NOT >= 65536 or unlimited
 WARNING => pam_limits NOT configured properly for shell limits
 WARNING => NIC bonding is NOT configured for public network (VIP)
 WARNING => OSWatcher is not running as is recommended.
 INFO =>    Jumbo frames (MTU ~9000) are not configured for interconnect
 FAIL =>    Database parameter DB_BLOCK_CHECKSUM is NOT set to recommended value on RAC1 instance
 FAIL =>    Database parameter DB_LOST_WRITE_PROTECT is NOT set to recommended value on RAC1 instance
 INFO =>    OS parameter vm.swappiness is NOT set to the recommended value
 WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for RAC
 FAIL =>    Flashback on PRIMARY is not configured for RAC
 FAIL =>    No one high redundancy diskgroup configured for RAC
 INFO =>    Operational Best Practices
 INFO =>    Consolidation Database Practices
 INFO =>    Network failure prevention best practices
 INFO =>    Computer failure prevention best practices
 INFO =>    Data corruption prevention best practices
 INFO =>    Logical corruption prevention best practices
 INFO =>    Storage failures prevention best practices
 INFO =>    Database/Cluster/Site failure prevention best practices
 INFO =>    Client failover operational best practices
 WARNING => fast_start_mttr_target has NOT been changed from default on RAC1 instance

 INFO =>    IMPORTANT: Oracle Database Patch 13466801, 13343438, 13348650, 13696251 13919095 or 14275572 PSU is NOT applied to RDBMS Home /u01/app/oracle/product/11.2.0/db_1
 WARNING => kernel parameter rp_filter is set to 1.
 INFO =>    Information about hanganalyze and systemstate dump
 FAIL =>    Your Configuration is NOT in Compliance with Oracle Security Alert for CVE-2012-1675 /u01/app/11.2.0/grid
 FAIL =>    Your Configuration is NOT in Compliance with Oracle Security Alert for CVE-2012-1675 /u01/app/oracle/product/11.2.0/db_1
 INFO =>    Software maintenance best practices
 WARNING => Database Parameter memory_target is not set to the recommended value on RAC1 instance
 INFO =>    Database failure prevention best practices
 WARNING => Database Archivelog Mode should be set to ARCHIVELOG for RAC
 FAIL =>    Primary database is NOT protected with Data Guard (standby database) for real-time data protection and availability for RAC
 WARNING => avahi-daemon process is running


Best Practice checking completed.Checking recommended patches on node01.
---------------------------------------------------------------------------------


Collecting patch inventory on  CRS HOME /u01/app/11.2.0/grid
Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/11.2.0/db_1
---------------------------------------------------------------------------------
1 Recommended CRS patches for 112030 from /u01/app/11.2.0/grid on node01
---------------------------------------------------------------------------------
Patch#   CRS  ASM    RDBMS RDBMS_HOME                              Patch-Description                          
---------------------------------------------------------------------------------
14275572  no          no  /u01/app/oracle/product/11.2.0/db_1GRID INFRASTRUCTURE PATCH SET UPDATE 11    
---------------------------------------------------------------------------------


---------------------------------------------------------------------------------
1 Recommended RDBMS patches for 112030 from /u01/app/oracle/product/11.2.0/db_1 on node01
---------------------------------------------------------------------------------
Patch#   RDBMS    ASM     type                Patch-Description                      
---------------------------------------------------------------------------------
14275572  no             merge               GRID INFRASTRUCTURE PATCH SET UPDATE 11.
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------


---------------------------------------------------------------------------------
              Clusterware patches summary report
---------------------------------------------------------------------------------
Total patches  Applied on CRS Applied on RDBMS Applied on ASM
---------------------------------------------------------------------------------
1              0              0                0            
---------------------------------------------------------------------------------


---------------------------------------------------------------------------------
              RDBMS homes patches summary report
---------------------------------------------------------------------------------
Total patches  Applied on RDBMS Applied on ASM ORACLE_HOME  
---------------------------------------------------------------------------------
 1              0              0                /u01/app/oracle/product/11.2.0/db_1
---------------------------------------------------------------------------------


=============================================================
                    Node name - node02                              
=============================================================
Collecting - CPU Information
Collecting - CRS active version
Collecting - CRS oifcfg
Collecting - CRS software version
Collecting - Cluster interconnect (clusterware)
Collecting - Huge pages configuration
Collecting - Kernel parameters
Collecting - Linux module config.
Collecting - Maximum number of semaphore sets on system
Collecting - Maximum number of semaphores on system
Collecting - Maximum number of semaphores per semaphore set
Collecting - Memory Information
Collecting - OS Packages
Collecting - Operating system release information and kernel version
Collecting - Oracle Executable Attributes
Collecting - Patches for Grid Infrastructure
Collecting - Patches for RDBMS Home
Collecting - Shared memory segments
Collecting - Table of file system defaults
Collecting - number of semaphore operations per semop system call
Preparing to run root privileged commands  node02.  Please enter root password when prompted.
root@node02's password:
Collecting - CRS user limits configuration
Collecting - CRS user time zone check
Collecting - Interconnect interface config
Collecting - Network interface stats
Collecting - OCFS2 disks
/tmp/root_raccheck.sh: line 29: /sbin/mounted.ocfs2: No such file or directory
Collecting - Root Open File Limit
Collecting - ocsf status
/tmp/root_raccheck.sh: line 39: /etc/init.d/o2cb: No such file or directory
Collecting - root time zone check


Data collections completed. Checking best practices on node02.
--------------------------------------------------------------------------------------


 INFO =>    $CRS_HOME/log/hostname/client directory has too many older log files.
 INFO =>    user_dump_dest has trace files older than 30 days for RAC
 INFO =>    background_dump_dest has files older than 30 days for RAC
 WARNING => Open files limit (ulimit -n) for current user is NOT set to recommended value >= 65536 or unlimited
 INFO =>    audit_file_dest has audit files older than 30 days for RAC
 WARNING => /tmp is NOT on a dedicated filesystem
 WARNING => kernel.shmmax parameter is NOT configured according to recommendation
 WARNING => Open file limit for root user (ulimit -n) is NOT >= 65536 or unlimited
 WARNING => pam_limits NOT configured properly for shell limits
 WARNING => NIC bonding is NOT configured for public network (VIP)
 WARNING => OSWatcher is not running as is recommended.
 INFO =>    Jumbo frames (MTU ~9000) are not configured for interconnect
 FAIL =>    Database parameter DB_BLOCK_CHECKSUM is NOT set to recommended value on RAC2 instance
 FAIL =>    Database parameter DB_LOST_WRITE_PROTECT is NOT set to recommended value on RAC2 instance
 INFO =>    OS parameter vm.swappiness is NOT set to the recommended value
 WARNING => Database parameter DB_BLOCK_CHECKING on PRIMARY is NOT set to the recommended value. for RAC
 WARNING => fast_start_mttr_target has NOT been changed from default on RAC2 instance

 INFO =>    IMPORTANT: Oracle Database Patch 13466801, 13343438, 13348650, 13696251 13919095 or 14275572 PSU is NOT applied to RDBMS Home /u01/app/oracle/product/11.2.0/db_1
 WARNING => kernel parameter rp_filter is set to 1.
 FAIL =>    Your Configuration is NOT in Compliance with Oracle Security Alert for CVE-2012-1675 /u01/app/11.2.0/grid
 FAIL =>    Your Configuration is NOT in Compliance with Oracle Security Alert for CVE-2012-1675 /u01/app/oracle/product/11.2.0/db_1
 WARNING => Database Parameter memory_target is not set to the recommended value on RAC2 instance
 WARNING => avahi-daemon process is running


Best Practice checking completed.Checking recommended patches on node02.
---------------------------------------------------------------------------------


Collecting patch inventory on  CRS HOME /u01/app/11.2.0/grid
Collecting patch inventory on ORACLE_HOME /u01/app/oracle/product/11.2.0/db_1
---------------------------------------------------------------------------------
1 Recommended CRS patches for 112030 from /u01/app/11.2.0/grid on node02
---------------------------------------------------------------------------------
Patch#   CRS  ASM    RDBMS RDBMS_HOME                              Patch-Description                          
---------------------------------------------------------------------------------
14275572  no          no  /u01/app/oracle/product/11.2.0/db_1GRID INFRASTRUCTURE PATCH SET UPDATE 11    
---------------------------------------------------------------------------------


---------------------------------------------------------------------------------
1 Recommended RDBMS patches for 112030 from /u01/app/oracle/product/11.2.0/db_1 on node02
---------------------------------------------------------------------------------
Patch#   RDBMS    ASM     type                Patch-Description                      
---------------------------------------------------------------------------------
14275572  no             merge               GRID INFRASTRUCTURE PATCH SET UPDATE 11.
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------


---------------------------------------------------------------------------------
              Clusterware patches summary report
---------------------------------------------------------------------------------
Total patches  Applied on CRS Applied on RDBMS Applied on ASM
---------------------------------------------------------------------------------
1              0              0                0            
---------------------------------------------------------------------------------


---------------------------------------------------------------------------------
              RDBMS homes patches summary report
---------------------------------------------------------------------------------
Total patches  Applied on RDBMS Applied on ASM ORACLE_HOME  
---------------------------------------------------------------------------------
 1              0              0                /u01/app/oracle/product/11.2.0/db_1
---------------------------------------------------------------------------------




---------------------------------------------------------------------------------
                      CLUSTERWIDE CHECKS
---------------------------------------------------------------------------------
---------------------------------------------------------------------------------

Detailed report (html) - /home/oracle/Desktop/raccheck_RAC_030313_004245/raccheck_RAC_030313_004245.html


UPLOAD(if required) - /home/oracle/Desktop/raccheck_RAC_030313_004245.zip




[oracle@node01 Desktop]$