Backend WLS or EM application seems to be down

When trying to access the web interface of Oracle 11g Release 1 Grid Control after restart of the server I am getting the following message:

Backend WLS or EM application seems to be down 

In the first try I receive the following error:


$  ./Oracle/Middleware/oms11g/bin/emctl start oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Starting WebTier...
WebTier Successfully Started
Starting Oracle Management Server...
Oracle Management Server Could Not Be Started
Oracle Management Server is not functioning because of the following reason:
Connection to the repository failed. Verify that the repository connection information provided is correct.
Please check /home/oracle/Oracle/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log for error details

This is very clear message and rectified by starting the listener.


To resolve the main issue I did the following:

$ pwd
/home/oracle
$  ./Oracle/Middleware/oms11g/bin/emctl start oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Starting WebTier...
WebTier Successfully Started
Starting Oracle Management Server...
Oracle Management Server Could Not Be Started
Oracle Management Server is Up
$ 

And it worked, now I can access the Oracle 11g Release 1 Grid Control's web interface.

Few other things which you can do is:

Check if the listener is running and you can connect to database.

Another is check status of oms with details option

$ ./Oracle/Middleware/oms11g/bin/emctl status oms -details
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host : *******************
HTTP Console Port   : ****
HTTPS Console Port  : ****
HTTP Upload Port    : ****
HTTPS Upload Port   : ****
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1

If you are still getting the same error then make sure that the listener is running.


lsnrctl status listener

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 11-JUL-2012

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=orcl)))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   Linux Error: 111: Connection refused
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=gcdb)))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   Linux Error: 111: Connection refused
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=XXXXXX)(Port=XXXX)))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   Linux Error: 111: Connection refused


stop and restart the listener and try stopping and starting the listener again


Oracle Grid Control 12c

[oracle@oem01 bin]$ ./emctl stop oms -all
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down
[oracle@oem01 bin]$
[oracle@oem01 bin]$
[oracle@oem01 bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
Starting WebTier...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
[oracle@oem01 bin]$
[oracle@oem01 bin]$
[oracle@oem01 bin]$
[oracle@oem01 bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
[oracle@oem01 bin]$
[oracle@oem01 bin]$
[oracle@oem01 bin]$
[oracle@oem01 bin]$ ./emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : oem01.localdomain
HTTP Console Port          : 7789
HTTPS Console Port         : 7801
HTTP Upload Port           : 4890
HTTPS Upload Port          : 4901
EM Instance Home           : /u01/app/oracle/Middleware2/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /u01/app/oracle/Middleware2/gc_inst/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://oem01.localdomain:7801/em
Upload URL: https://oem01.localdomain:4901/empbs/upload

WLS Domain Information
Domain Name      : GCDomain
Admin Server Host: oem01.localdomain

Managed Server Information
Managed Server Instance Name: EMGC_OMS1
Managed Server Instance Host: oem01.localdomain
WebTier is Up
Oracle Management Server is Up
[oracle@oem01 bin]$

Oracle Grid Control installation steps

There are very few places which will tell you on how to install Oracle Grid Control. Of course you will be able to find something in Oracle Documentation. I tried following many things and did the installation of Oracle Grid Control Installation on Linux.

For the complete installation of Oracle Grid Control we need three things:

1. One Oracle database which will be used by Grid Control
2. Oracle Middleware weblogic server
3. Oracle Grid Control

In this installation example I have used:

Oracle database 11gR1

Oracle Middleware weblogic server version 10.3.2.0.

Oracle Enterprise Manager 11g Grid Control Release 1 (11.1.0.1.0)

Oracle Grid Control can be downloaded from here 

If you want to see Oracle Official documentation for this release then click here

As described above first steps is to have the oracle database for grid control and I have called this gcdb, below are some of the screen shots which I feel are important to show:

  I have used only 300M as SGA, please note that this is only a test system.

Once the installation of Oracle database is done then there is a need for the installation of Oracle Middleware.

After the completion of Oracle Middleware I did the installation of Oracle Grid Controll. In this installation I did not check the pre-request document correctly and got few issues and those issues are discussed in this blog and I have demonstrated on how I have resolved them.

 

Above issue was resolve using:

$ pwd
/u01/app/oracle/product/11.2.0/dbhome_1
$ cd bin
$ ./emca -deconfig dbcontrol db -repos drop -SYS_PWD ****** -SYSMAN_PWD ******
STARTED EMCA at Apr 29, 2011 10:00:53 AM
EM Configuration Assistant, Version 11.2.0.0.2 Production
Copyright (c) 2003, 2005, Oracle.  All rights reserved.
Enter the following information:
Database SID: gcdb
Listener port number: 1521
Do you wish to continue? [yes(Y)/no(N)]: Y
Apr 29, 2011 10:01:17 AM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /u01/app/oracle/cfgtoollogs/emca/gcdb/emca_2011_04_29_10_00_53.log.
Apr 29, 2011 10:01:17 AM oracle.sysman.emcp.EMDBPreConfig performDeconfiguration
WARNING: EM is not configured for this database. No EM-specific actions can be performed.
Apr 29, 2011 10:01:17 AM oracle.sysman.emcp.ParamsManager checkListenerStatusForDBControl
WARNING: Error initializing SQL connection. SQL operations cannot be performed
Apr 29, 2011 10:01:17 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Dropping the EM repository (this may take a while) ...
Apr 29, 2011 10:02:54 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully dropped
Enterprise Manager configuration completed successfully
FINISHED EMCA at Apr 29, 2011 10:02:54 AM
$

Next Error I received was because initial parameters were not set properly

These were altered in initial parameter and then database was restarted. But the error were not over as I did not read the pre-requirement document nicely :) 

 This was because the filesize of the undo tablespace was less than 200MB this was a simple fix:

alter database datafile '/u01/app/oracle/oradata/gcdb/undotbs01.dbf'  resize 300M;

# cd /home/oracle/Oracle/Middleware/oms11g/
# ./allroot.sh
Starting to execute allroot.sh .........
Starting to execute /home/oracle/Oracle/Middleware/oms11g/root.sh ......
Running Oracle 11g root.sh script...
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /home/oracle/Oracle/Middleware/oms11g
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Adding entry to /etc/oratab file...
Finished execution of  /home/oracle/Oracle/Middleware/oms11g/root.sh ......

Starting to execute /home/oracle/Oracle/Middleware/agent11g/root.sh ......
Running Oracle 11g root.sh script...
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /home/oracle/Oracle/Middleware/agent11g
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
Adding entry to /etc/oratab file...
Finished execution of  /home/oracle/Oracle/Middleware/agent11g/root.sh ......
# 

cd $ORACLE_HOME/dbhome_1/sysman/admin/emdrep/bin
./RepManager hostname 1521 gcdb  -sys_password ******* -repos_user sysman -repos_password ******* -action drop
Getting temporary tablespace from database...
Found temporary tablespace: TEMP
Checking SYS Credentials ... rem error switch
OK.
rem error switch
Dropping the repository..
Checking for Repos User ... Exists.
Repos User exists..
Clearing EM Contexts ... Failed.
Dropping EM users ...
Done.
Dropping Repos User ... Done.
Dropping Roles/Synonymns/Tablespaces ... Done.
ORA-06550: line 1, column 7:
ORA-06550: line 1, column 7:
ORA-06550: line 22, column 44:
PL/SQL: ORA-00942: table or view does not exist
ORA-06550: line 18, column 11:
ORA-04031: unable to allocate 148 bytes of shared memory ("shared
ORA-06512: at line 155
Dropping Repository failed. Please check file /u01/app/oracle/product/11.2.0/dbhome_1/sysman/log/emrepmgr.log.23942 for detailed errors.

This was because the SGA was only 300M so I changed the SGA to 800M and restarted the database and then ran the script once again. 

./RepManager hostname 1521 gcdb  -sys_password ******* -repos_user sysman -repos_password ******* -action drop
Getting temporary tablespace from database...
Found temporary tablespace: TEMP
Checking SYS Credentials ... rem error switch
OK.
rem error switch
Dropping the repository..
Checking for Repos User ... Exists.
Repos User exists..
Clearing EM Contexts ... Failed.
Dropping EM users ...
Done.
Dropping Repos User ... Done.
Dropping Roles/Synonymns/Tablespaces ... Done.
ORA-06550: line 1, column 7:
ORA-06550: line 1, column 7:
ORA-06550: line 22, column 44:
PL/SQL: ORA-00942: table or view does not exist
ORA-06550: line 18, column 11:
Dropping Repository failed. Please check file /u01/app/oracle/product/11.2.0/dbhome_1/sysman/log/emrepmgr.log.24404 for detailed errors

That's it then I clicked on retry. 

The installation of the Oracle Grid Control is done. Now you have to open ports in firewall so that other machines can access the Grid Control interface.

I hope this will help and if there are questions then feel free to send message in comments.