Details
-
Bug
-
Resolution: Cannot Reproduce
-
Medium
-
None
-
2.2
-
None
Description
Hi,
I run 2 instances based on the same image (amiId: ami-08dc3b61) and ebs snapshot (snap-13f6037a). I have already did this a few times before, but this time the startup of elastic agent failed with BindException.
2009-03-13 06:38:09,564 INFO [main] [ElasticAgentBootstrap] Using tunnnelling. Registering 'httpt' and 'httpst' protocols. 1 [main] INFO com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap - Using tunnnelling. Registering 'httpt' and 'httpst' protocols. 2009-03-13 06:38:11,729 INFO [com.sun.sungrid.service.tunnel.server.TunnelServer] [TunnelServer] Waiting for tunnel connection. 2166 [com.sun.sungrid.service.tunnel.server.TunnelServer] INFO com.sun.sungrid.service.tunnel.server.TunnelServer - Waiting for tunnel connection. 2009-03-13 06:38:12,136 INFO [com.sun.sungrid.service.tunnel.server.TunnelServer] [TunnelServer] Accepted tunnel connection. 2573 [com.sun.sungrid.service.tunnel.server.TunnelServer] INFO com.sun.sungrid.service.tunnel.server.TunnelServer - Accepted tunnel connection. 2009-03-13 06:38:12,139 INFO [com.sun.sungrid.service.tunnel.server.TunnelServer] [TunnelServer] Waiting for tunnel connection. 2576 [com.sun.sungrid.service.tunnel.server.TunnelServer] INFO com.sun.sungrid.service.tunnel.server.TunnelServer - Waiting for tunnel connection. 2009-03-13 06:38:12,214 INFO [com.sun.sungrid.service.tunnel.server.TunnelServer] [TunnelServer] Accepted tunnel connection. 2651 [com.sun.sungrid.service.tunnel.server.TunnelServer] INFO com.sun.sungrid.service.tunnel.server.TunnelServer - Accepted tunnel connection. 2009-03-13 06:38:12,215 INFO [com.sun.sungrid.service.tunnel.server.TunnelServer] [TunnelServer] Waiting for tunnel connection. 2652 [com.sun.sungrid.service.tunnel.server.TunnelServer] INFO com.sun.sungrid.service.tunnel.server.TunnelServer - Waiting for tunnel connection. 2009-03-13 06:38:12,571 INFO [pool-1-thread-3] [LocalAcceptor] Waiting for local connection on port 4527. 3008 [pool-1-thread-3] INFO com.sun.sungrid.service.tunnel.server.LocalAcceptor - Waiting for local connection on port 4527. 2009-03-13 06:38:12,677 ERROR [pool-1-thread-2] [TunnelAcceptor] Error while accepting tunnel connections. java.net.BindException: Address already in use at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359) at java.net.ServerSocket.bind(ServerSocket.java:319) at java.net.ServerSocket.<init>(ServerSocket.java:185) at java.net.ServerSocket.<init>(ServerSocket.java:97) at javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:156) at com.sun.sungrid.service.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:93) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) at java.lang.Thread.run(Thread.java:595) 3114 [pool-1-thread-2] ERROR com.sun.sungrid.service.tunnel.server.TunnelAcceptor - Error while accepting tunnel connections. java.net.BindException: Address already in use at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359) at java.net.ServerSocket.bind(ServerSocket.java:319) at java.net.ServerSocket.<init>(ServerSocket.java:185) at java.net.ServerSocket.<init>(ServerSocket.java:97) at javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:156) at com.sun.sungrid.service.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:93) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) at java.lang.Thread.run(Thread.java:595)
Here is the output of netstat
# netstat -lnp Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:1666 0.0.0.0:* LISTEN 1771/p4d tcp 0 0 :::3690 :::* LISTEN 8909/svnserve tcp 0 0 :::4527 :::* LISTEN 1434/java tcp 0 0 :::26224 :::* LISTEN 1434/java tcp 0 0 :::22 :::* LISTEN 883/sshd udp 0 0 0.0.0.0:68 0.0.0.0:* 788/dhclient Active UNIX domain sockets (only servers) Proto RefCnt Flags Type State I-Node PID/Program name Path unix 2 [ ACC ] STREAM LISTENING 2713 866/dbus-daemon /var/run/dbus/system_bus_socket # ps -ocmd -p 1434 CMD java -server -Xms32m -Xmx512m -XX:MaxPermSize=256m -cp /opt/bamboo-elastic-agent/bin/../lib:/opt/bamboo-elastic-agent/bin/../lib/spring-jdbc-2.0.7.jar:/opt/bamboo-elastic-agent/bin/../lib/webwork-2.2.5.jar:/opt
Bamboo elastic agent process has 2 sockets open for listening - 4527 and 26224. When I compare it with the second instance where the startup succeeded I can see that there should be a third socket open - 46593. Probably opening of that socket was the problem. But I cannot see nothing that occupies that port. So maybe it was a hazard between 2 threads of the same app?
Cheers,
Lucas
