Details
-
Bug
-
Resolution: Fixed
-
Low
-
5.7.1
Description
When using Bamboo behind a proxy/load balancer - remote agents could fail to connect to the agent server
This occurs because the tunnel does not send a HTTP Host header as part of its request to the remote server, even when the Client Broker URL is specified as a hostname.
In this scenario, the agent exits and leave Bamboo waiting for an agent connection until the schedule cleans it up.
As a workaround a user must ensure that the load balancer will direct any requests without a Host header to Bamboo. This may require allocating a dedicated IP for Bamboo depending on the situation.
It is unclear from my end whether it is the agent or the tunnel that needs to inject the header (suspect the latter).
Agent log output for this situation is included below, in this case the HTTP proxy (HAProxy) is returning HTTP 503 because it is not configured with a default virtual host:
2014-11-26 06:43:56,389 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-e1/5.7.1/80406de314efffaefb1fa9e589ef8ca953a46a5f/ to /opt/bamboo-elastic-agent 2014-11-26 06:43:57,142 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-us-e1/5.7.1/80406de314efffaefb1fa9e589ef8ca953a46a5f/ to /opt/bamboo-elastic-agent 2014-11-26 06:43:57,142 INFO [main] [S3Utils] Fetching the list of remote objects... 2014-11-26 06:44:00,293 INFO [main] [S3Utils] Found 594 files in s3://bamboo-agent-release-us-e1/5.7.1/80406de314efffaefb1fa9e589ef8ca953a46a5f/ 2014-11-26 06:44:00,731 INFO [main] [S3Utils] Found 1592 files in /opt/bamboo-elastic-agent 2014-11-26 06:44:00,731 INFO [main] [S3Utils] Generating the list of files to fetch from S3... 2014-11-26 06:44:00,757 INFO [main] [S3Utils] Generating the list of files to remove... 2014-11-26 06:44:00,763 INFO [main] [S3Utils] Removing 1303 files from /opt/bamboo-elastic-agent 2014-11-26 06:44:02,263 INFO [main] [S3Utils] Fetching 341 files to /opt/bamboo-elastic-agent 2014-11-26 06:44:35,179 INFO [main] [S3Utils] Fetched 290 MB from S3 2014-11-26 06:44:35,182 INFO [main] [ElasticAgentInstaller] Starting [java, -server, -Xms32m, -Xmx256m, -XX:MaxPermSize=128m, -XX:+HeapDumpOnOutOfMemoryError, -Dimagefiles.version=3.3, -Dbamboo.agent.installDir=/opt/bamboo-elastic-agent, -cp, /opt/bamboo-elastic-agent/boot/annotations-13.0.jar:/opt/bamboo-elastic-agent/boot/log4j-1.2.17.jar:/opt/bamboo-elastic-agent/boot/gson-2.2.2-atlassian-1.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-api-agent-bootstrap-5.7.1.jar:/opt/bamboo-elastic-agent/boot/commons-io-2.4.jar:/opt/bamboo-elastic-agent/boot/fugue-2.1.0.jar:/opt/bamboo-elastic-agent/boot/jackson-core-2.1.1.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-elastic-shared-5.7.1.jar:/opt/bamboo-elastic-agent/boot/stax-api-1.0-2.jar:/opt/bamboo-elastic-agent/boot/jcl-over-slf4j-1.7.7.jar:/opt/bamboo-elastic-agent/boot/guava-bridge-11.0.2-atlassian-01.jar:/opt/bamboo-elastic-agent/boot/atlassian-aws-1.0.79.1.jar:/opt/bamboo-elastic-agent/boot/commons-codec-1.8.jar:/opt/bamboo-elastic-agent/boot/joda-time-2.3.jar:/opt/bamboo-elastic-agent/boot/guava-11.0.2-atlassian-01.jar:/opt/bamboo-elastic-agent/boot/commons-lang-2.6.jar:/opt/bamboo-elastic-agent/boot/slf4j-log4j12-1.7.7.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-bootstrap-5.7.1.jar:/opt/bamboo-elastic-agent/boot/jackson-databind-2.1.1.jar:/opt/bamboo-elastic-agent/boot/httpclient-4.2.5.jar:/opt/bamboo-elastic-agent/boot/jackson-annotations-2.1.1.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-elastic-5.7.1.jar:/opt/bamboo-elastic-agent/boot/atlassian-annotations-0.9.jar:/opt/bamboo-elastic-agent/boot/bcprov-jdk15on-1.48.jar:/opt/bamboo-elastic-agent/boot/aws-java-sdk-1.8.10.1.jar:/opt/bamboo-elastic-agent/boot/slf4j-api-1.7.7.jar:/opt/bamboo-elastic-agent/boot/aws-java-sdk-core-1.8.10.1.jar:/opt/bamboo-elastic-agent/boot/bcpkix-jdk15on-1.48.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-core-agent-bootstrap-5.7.1.jar:/opt/bamboo-elastic-agent/boot/atlassian-tunnel-1.0.jar:/opt/bamboo-elastic-agent/boot/jsr305-1.3.9.jar:/opt/bamboo-elastic-agent/boot/httpcore-4.2.5.jar:, com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap] 2014-11-26 06:44:35,526 INFO [main] [ElasticAgentBootstrap] Starting Agent Bootstrap using Java 1.6.0_45 from Sun Microsystems Inc. 2014-11-26 06:44:36,180 INFO [main] [ElasticAgentBootstrap] Using tunnelling. Setting virtual host name to http://bamboo.synrgy.mobi/agentServer/ 2014-11-26 06:44:36,180 INFO [main] [ElasticAgentBootstrap] Using tunnelling for HTTP(S). Registering 'httpt' and 'httpst' protocols. 2014-11-26 06:44:36,184 INFO [main] [ElasticAgentBootstrap] HTTP(S) tunnel: enabled 2014-11-26 06:44:36,185 INFO [main] [ElasticAgentBootstrap] JMS tunnel: enabled 2014-11-26 06:44:36,213 INFO [main] [ElasticAgentBootstrap] Starting tunnel server, waiting for 2 connections. 2014-11-26 06:44:42,140 INFO [main] [AgentBootstrap] Agent bootstrap using baseUrl: httpt://127.0.0.1:46593/agentServer/ 2014-11-26 06:44:42,378 INFO [main] [RemoteAgentHomeLocatorForBootstrap] Creating agent home located at [/home/bamboo/bamboo-agent-home] 2014-11-26 06:44:42,379 INFO [main] [RemoteAgentHomeLocatorForBootstrap] Using agent home located at [/home/bamboo/bamboo-agent-home] 2014-11-26 06:44:42,380 INFO [main] [AgentUuidInitializer] Generating new UUID for this agent: d6b6dcd0-565c-40ce-a4c3-095061684a50 2014-11-26 06:44:42,393 INFO [main] [AgentContext] Requesting fingerprint, url: httpt://127.0.0.1:46593/agentServer/GetFingerprint.action?hostName=ip-172-31-54-88.ec2.internal&agentType=elastic&instanceId=i-31d9a2d0&version=3&agentUuid=d6b6dcd0-565c-40ce-a4c3-095061684a50 2014-11-26 06:44:42,532 INFO [tunnellogger-thread] [LocalAcceptor] [tunnelserver:26224-1-thread-1] Accepted a local connection on port 46593, tunnel age: 0s 2014-11-26 06:44:42,532 INFO [tunnellogger-thread] [LocklessLogger] [tunnellogger-thread] Logging system is operating in a lockless mode Exiting due to fatal exception. com.atlassian.bamboo.agent.bootstrap.RemoteAgentHttpException: HTTP status code 503 received in response to fingerprint request. at com.atlassian.bamboo.agent.bootstrap.AgentContext.initFingerprint(AgentContext.java:137) at com.atlassian.bamboo.agent.bootstrap.AgentContext.initServerSession(AgentContext.java:100) at com.atlassian.bamboo.agent.bootstrap.AgentContext.run(AgentContext.java:91) at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.run(AgentBootstrap.java:94) at com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap.run(ElasticAgentBootstrap.java:78) at com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap.main(ElasticAgentBootstrap.java:58) 2014-11-26 06:44:43,299 FATAL [main] [AgentBootstrap] Exiting due to fatal exception. com.atlassian.bamboo.agent.bootstrap.RemoteAgentHttpException: HTTP status code 503 received in response to fingerprint request. at com.atlassian.bamboo.agent.bootstrap.AgentContext.initFingerprint(AgentContext.java:137) at com.atlassian.bamboo.agent.bootstrap.AgentContext.initServerSession(AgentContext.java:100) at com.atlassian.bamboo.agent.bootstrap.AgentContext.run(AgentContext.java:91) at com.atlassian.bamboo.agent.bootstrap.AgentBootstrap.run(AgentBootstrap.java:94) at com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap.run(ElasticAgentBootstrap.java:78) at com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap.main(ElasticAgentBootstrap.java:58)
Suggested remediation:
- Ensure agent/tunnel specifies a Host header on name based requests