Sunday, September 13, 2009

Grid Agent Failed to start with HTTP LISTENER failed

The agent fails to start with: Starting agent ....... failed. Failed to start HTTP listener. Consult the log files in: /u01/app/oracle/product/agent10g/sysman/log

2007-07-09 09:06:18 Thread-1 Starting Agent 10.2.0.3.0 from /u01/app/oracle/product/agent10g2007-07-09 09:06:18 Thread-1 : Startup of HTTP LISTENER failure (00716)

1. Agent crashes. As a result it is still listening on its port, rendering it unavailable. This port is defined in the /sysman/config/emd.properties file in the EMD_URL parameter.2. The agent is part of a High Availability cluster with multiple VIP's defined, and one VIP was just moved to another node, where there was an agent already running on another VIP.
SolutionCASE 1:
1. Get the Agent port number by looking at the EMD_URL parameter as defined in the /sysman/config/emd.properties file (if it's a cluster agent, look under //sysman/config)
2. Check to see if the port is still in use (usually it's 1830 or 3872):
netstat -an grep
If it is in use you will see something like the following:
tcp 0 0 0.0.0.0:3872 0.0.0.0:* LISTEN
3. If this is the case then kill off the rogue emagent processes
ps -ef grep emagent awk ' {print $2}' xargs kill -9
4. Re-start the agent

CASE 2:
This can be solved by using "ListenOnAllNICs=false" in emd.properties. This is explained in step F.2 inNote 406014.1 How to Configure Grid Control Agents to Monitor Virtual Hostname in HA environments