Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-14494

Ec2 agent unable to download shared artefact due to HTTP 502 response

    XMLWordPrintable

Details

    Description

      The agent was preparing the build and tried to download a shared artefact:

      16-Apr-2014 04:29:03 	Build A1. JIRA Master Tier 1 - CI Functional Tests - Func 06 #6713 (MASTERONE-CIFUNCHAL-FUNC06-6713) started building on agent Elastic Agent on i-83d0c1d2
16-Apr-2014 04:29:03 	Build working directory is /opt/bamboo-agent/bamboo-agent-home/xml-data/build-dir/MASTERONE-CIFUNCHAL-FUNC06
16-Apr-2014 04:29:03 	Executing build A1. JIRA Master Tier 1 - CI Functional Tests - Func 06 #6713 (MASTERONE-CIFUNCHAL-FUNC06-6713)
16-Apr-2014 04:29:03 	Preparing artifact for use: hallelujah-dist
16-Apr-2014 04:29:29 	Failure in artifact preparation phase during processing of: Subscription for Shared artifact: [hallelujah-dist], pattern: [jira-*-dist.zip] anchored at: [jira-distribution/jira-func-tests-runner/target], destination: []
16-Apr-2014 04:29:29 	1 error(s) found when performing pre-build actions.

      Looking at the agent's log, it failed to communicate with the server:

      2014-04-16 04:29:03,888 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [DefaultBuildAgent] Running build phase: com.atlassian.bamboo.v2.build.task.InitializeBuild
2014-04-16 04:29:03,889 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [RemoteExecutionPhaseServiceImpl] A1. JIRA Master Tier 1 - CI Functional Tests - Func 06 #6713 (MASTERONE-CIFUNCHAL-FUNC06-6713) VCS sync started
2014-04-16 04:29:03,987 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [CheckoutDirectoriesSnapshotHelper] Build working directory is /opt/bamboo-agent/bamboo-agent-home/xml-data/build-dir/MASTERONE-CIFUNCHAL-FUNC06
2014-04-16 04:29:03,988 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [DefaultBuildAgent] Running build phase: com.atlassian.bamboo.build.pipeline.tasks.PrepareBuildTask
2014-04-16 04:29:03,989 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [PrepareBuildTask] Executing build A1. JIRA Master Tier 1 - CI Functional Tests - Func 06 #6713 (MASTERONE-CIFUNCHAL-FUNC06-6713)
2014-04-16 04:29:03,990 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [PrepareBuildTask] Preparing artifact for use: hallelujah-dist
2014-04-16 04:29:29,976 ERROR [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [HttpMessageSender] Failed to send message.
java.io.IOException: Unexpected HTTP status code: 502 Bad Gateway
        at com.atlassian.bamboo.v2.build.agent.remote.sender.HttpMessageSender.send(HttpMessageSender.java:105)
        at com.atlassian.bamboo.build.artifact.BambooRemoteArtifactHandler.internalRetrieve(BambooRemoteArtifactHandler.java:74)
        at com.atlassian.bamboo.build.artifact.BambooRemoteArtifactHandler.retrieve(BambooRemoteArtifactHandler.java:53)
        at com.atlassian.bamboo.build.artifact.AbstractArtifactManager$3.call(AbstractArtifactManager.java:341)
        at com.atlassian.bamboo.build.artifact.AbstractArtifactManager$3.call(AbstractArtifactManager.java:335)
        at com.atlassian.bamboo.plugin.BambooPluginUtils.callUnsafeCode(BambooPluginUtils.java:86)
        at com.atlassian.bamboo.build.artifact.AbstractArtifactManager.retrieveArtifact(AbstractArtifactManager.java:334)
        at com.atlassian.bamboo.build.artifact.AbstractArtifactManager.retrieve(AbstractArtifactManager.java:294)
        at com.atlassian.bamboo.build.pipeline.tasks.PrepareBuildTask.prepareArtifacts(PrepareBuildTask.java:150)
        at com.atlassian.bamboo.build.pipeline.tasks.PrepareBuildTask.call(PrepareBuildTask.java:98)
        at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.build(DefaultBuildAgent.java:181)
        at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild(BuildAgentControllerImpl.java:101)
        at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent$1.run(DefaultBuildAgent.java:108)
        at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:49)
        at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:31)
        at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:20)
        at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:52)
        at java.lang.Thread.run(Thread.java:662)
2014-04-16 04:29:29,977 ERROR [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [BambooPluginUtils] Unable to retrieve artifact hallelujah-dist for MASTERONE-CIFUNCHAL-FUNC06-6713 via com.atlassian.bamboo.build.artifact.BambooRem
oteArtifactHandler@56dbd887
java.lang.RuntimeException: java.io.IOException: Unexpected HTTP status code: 502 Bad Gateway
        at com.atlassian.bamboo.v2.build.agent.remote.sender.HttpMessageSender.send(HttpMessageSender.java:111)
        at com.atlassian.bamboo.build.artifact.BambooRemoteArtifactHandler.internalRetrieve(BambooRemoteArtifactHandler.java:74)
        at com.atlassian.bamboo.build.artifact.BambooRemoteArtifactHandler.retrieve(BambooRemoteArtifactHandler.java:53)
        at com.atlassian.bamboo.build.artifact.AbstractArtifactManager$3.call(AbstractArtifactManager.java:341)
        at com.atlassian.bamboo.build.artifact.AbstractArtifactManager$3.call(AbstractArtifactManager.java:335)
        at com.atlassian.bamboo.plugin.BambooPluginUtils.callUnsafeCode(BambooPluginUtils.java:86)
        at com.atlassian.bamboo.build.artifact.AbstractArtifactManager.retrieveArtifact(AbstractArtifactManager.java:334)
        at com.atlassian.bamboo.build.artifact.AbstractArtifactManager.retrieve(AbstractArtifactManager.java:294)
        at com.atlassian.bamboo.build.pipeline.tasks.PrepareBuildTask.prepareArtifacts(PrepareBuildTask.java:150)
        at com.atlassian.bamboo.build.pipeline.tasks.PrepareBuildTask.call(PrepareBuildTask.java:98)
        at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.build(DefaultBuildAgent.java:181)
        at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild(BuildAgentControllerImpl.java:101)
        at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent$1.run(DefaultBuildAgent.java:108)
        at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:49)
        at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:31)
        at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:20)
        at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:52)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Unexpected HTTP status code: 502 Bad Gateway
        at com.atlassian.bamboo.v2.build.agent.remote.sender.HttpMessageSender.send(HttpMessageSender.java:105)
        ... 17 more
2014-04-16 04:29:29,978 WARN [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [PrepareBuildTask] Failure in artifact preparation phase during processing of: Subscription for Shared artifact: [hallelujah-dist], pattern: [jira-*-dist.zip] anchored at: [jira-distribution/jira-func-tests-runner/target], destination: []
2014-04-16 04:29:29,978 WARN [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [PrepareBuildTask] 1 error(s) found when performing pre-build actions.

      However, immediately after that, it was able to talk to the server again:

      2014-04-16 04:29:29,992 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [RemoteExecutionPhaseServiceImpl] A1. JIRA Master Tier 1 - CI Functional Tests - Func 06 #6713 (MASTERONE-CIFUNCHAL-FUNC06-6713) execution finished
2014-04-16 04:29:30,004 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [DefaultBuildAgent] Detected '1' errors in build. Aborting build process.
2014-04-16 04:29:30,004 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [RemoteResultProcessor] Build A1. JIRA Master Tier 1 - CI Functional Tests - Func 06 #6713 (MASTERONE-CIFUNCHAL-FUNC06-6713) completed on Bamboo agent. Sending results to server.
2014-04-16 04:29:30,101 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [DefaultBuildAgent] Setting context to null on Elastic Agent on i-83d0c1d2/138832944
2014-04-16 04:29:30,101 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [BuildAgentControllerImpl] Agent 'Elastic Agent on i-83d0c1d2' ready to take build from queue...
2014-04-16 04:29:30,211 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [BuildAgentControllerImpl] A1. JIRA Master Tier 1 - CI Functional Tests - Func 14 #6713 (MASTERONE-CIFUNCHAL-FUNC14-6713) taken from queue.
2014-04-16 04:29:30,212 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [RemoteExecutionPhaseServiceImpl] A1. JIRA Master Tier 1 - CI Functional Tests - Func 14 #6713 (MASTERONE-CIFUNCHAL-FUNC14-6713) assigned to agent 615844060
2014-04-16 04:29:30,225 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [DefaultBuildAgent] Setting new context on Elastic Agent on i-83d0c1d2/138832944
2014-04-16 04:29:30,272 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [BuildAgentControllerImpl] Build A1. JIRA Master Tier 1 - CI Functional Tests - Func 14 #6713 (MASTERONE-CIFUNCHAL-FUNC14-6713) started building on agent Elastic Agent on i-83d0c1d2
2014-04-16 04:29:30,272 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [DefaultBuildAgent] Running build phase: com.atlassian.bamboo.v2.build.task.InitializeBuild
2014-04-16 04:29:30,273 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [RemoteExecutionPhaseServiceImpl] A1. JIRA Master Tier 1 - CI Functional Tests - Func 14 #6713 (MASTERONE-CIFUNCHAL-FUNC14-6713) VCS sync started
2014-04-16 04:29:30,342 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [CheckoutDirectoriesSnapshotHelper] Build working directory is /opt/bamboo-agent/bamboo-agent-home/xml-data/build-dir/MASTERONE-CIFUNCHAL-FUNC14
2014-04-16 04:29:30,342 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [DefaultBuildAgent] Running build phase: com.atlassian.bamboo.build.pipeline.tasks.PrepareBuildTask
2014-04-16 04:29:30,343 INFO [1-BAM::Elastic Agent on i-83d0c1d2::Agent:pool-4-thread-1] [PrepareBuildTask] Executing build A1. JIRA Master Tier 1 - CI Functional Tests - Func 14 #6713 (MASTERONE-CIFUNCHAL-FUNC14-6713)

      The following logs in the server log look relavent:

      2014-04-16 04:29:29,969 INFO [bambooAgentMessageListenerConnector-1] [DefaultErrorHandler] Recording an error: Failure in artifact preparation phase during processing of: Subscription for Shared artifact: [hallelujah-dist], pattern: [jira-
2014-04-16 04:29:29,982 WARN [bambooAgentMessageListenerConnector-1] [CurrentlyBuildingImpl] Timer being stopped, but timer isn't started
2014-04-16 04:29:30,203 INFO [bambooAgentMessageListenerConnector-1] [CurrentlyBuildingContainer] startBuildingOnAgent called for [MASTERONE-CIFUNCHAL-FUNC14-6713], agent 615844060
2014-04-16 04:29:30,203 INFO [bambooAgentMessageListenerConnector-1] [CurrentlyBuildingContainer] Agent 615844060 finished building MASTERONE-CIFUNCHAL-FUNC06-6713, but the build is still processed by the server. Detaching it from the agent

      It doesn't happen very often, but grepping the logs for Failure in artifact preparation phase during processing revealed at least five cases.

      Impact: the build goes red, a new build has to be kicked off.
      Workaround: none (beside restarting the build which is still a manual thing)

      (server: Atlassian Bamboo version 5.5-m1 build 4300)

      Attachments

        Activity

          People

            pbruski Przemek Bruski
            mmeinhold Martin Meinhold
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: