Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-18517

Docker does not start successfully on Elastic Agents using EBS-only instances

      Summary

      The docker daemon doesn't start successfully on Elastic Agents using a EBS-only instance type (such as T2, M4) with the Ubuntu Stock Image. This causes the first build of any build using a docker task to fail. The first docker command will hang for about a minute before timing out with an error. The docker daemon appears to listen after the first docker command fails and works thereafter.

      Steps to Reproduce

      1. Start an Elastic Agent with the Ubuntu Stock Image and configured to use a T2 instance type.
      2. Start a build using a docker task on that agent. E.g. Docker pull to fetch an image from a private repository.

      Expected Results

      Build succeeds

      Actual Results

      Build hangs (HungBuildKiller may step in) and eventually fails:

      build	14-Jun-2017 22:11:16	Not logged in to https://index.docker.io/v1/
      error	14-Jun-2017 22:11:16	Error occurred while running Task 'Pull Image(1)' of type com.atlassian.bamboo.plugins.bamboo-docker-plugin:task.docker.cli.
      error	14-Jun-2017 22:11:16	com.atlassian.bamboo.task.TaskException: Failed to execute task
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.plugins.docker.service.PullService.execute(PullService.java:41)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.plugins.docker.tasks.cli.DockerCliTask.execute(DockerCliTask.java:60)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.task.TaskExecutorImpl.lambda$executeTasks$3(TaskExecutorImpl.java:317)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.task.TaskExecutorImpl.executeTaskWithPrePostActions(TaskExecutorImpl.java:246)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.task.TaskExecutorImpl.executeTasks(TaskExecutorImpl.java:317)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.task.TaskExecutorImpl.execute(TaskExecutorImpl.java:118)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.build.pipeline.tasks.ExecuteBuildTask.call(ExecuteBuildTask.java:74)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.build(DefaultBuildAgent.java:216)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl$1.call(BuildAgentControllerImpl.java:139)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl$1.call(BuildAgentControllerImpl.java:130)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.variable.CustomVariableContextImpl.withVariableSubstitutor(CustomVariableContextImpl.java:221)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild(BuildAgentControllerImpl.java:129)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent$1.run(DefaultBuildAgent.java:138)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:51)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:31)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:20)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:52)
      error	14-Jun-2017 22:11:16		at java.lang.Thread.run(Thread.java:745)
      error	14-Jun-2017 22:11:16	Caused by: com.atlassian.bamboo.plugins.docker.client.DockerException: Error running Docker pull command. See messages above for details.
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.plugins.docker.client.DockerCmd.pull(DockerCmd.java:282)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.plugins.docker.service.PullService.execute(PullService.java:37)
      error	14-Jun-2017 22:11:16		... 17 more
      error	14-Jun-2017 22:11:16	Caused by: com.atlassian.utils.process.ProcessException: Error executing '/usr/bin/docker login -u bamboo -p ******** -e bamboo@bamboo.com', exit code: -1
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.plugins.docker.process.DefaultDockerProcessService.execute(DefaultDockerProcessService.java:53)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.plugins.docker.process.DefaultDockerProcessService.executeSilently(DefaultDockerProcessService.java:63)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.plugins.docker.process.DockerTaskProcessService.executeSilently(DockerTaskProcessService.java:75)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.plugins.docker.client.DockerCmd$AuthorisedCommand.login(DockerCmd.java:346)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.plugins.docker.client.DockerCmd$AuthorisedCommand.execute(DockerCmd.java:311)
      error	14-Jun-2017 22:11:16		at com.atlassian.bamboo.plugins.docker.client.DockerCmd.pull(DockerCmd.java:267)
      

      Workaround

      Add the below command to your Instance startup script at Bamboo Administration > Overview > Image Configurations > Edit elastic image configuration:

      service docker restart
      

            [BAM-18517] Docker does not start successfully on Elastic Agents using EBS-only instances

            The same error effect is also visible for a M5 instance without EBS and the login work-around works for this M5 instance too.

            Gerhard Forster (Inactive) added a comment - The same error effect is also visible for a M5 instance without EBS and the login work-around works for this M5 instance too.

            Same problem but slightly different configuration. We use a M4 General Purpose Large instance based on the Ubuntu Stock Image. The build still hangs on the first Docker task and will not return. After some time, Bamboo will detect that the build is hung and you have to stop it manually.

            Restarting the docker service during image creation does solve the problem though.

            Sander Brienen [Avisi] added a comment - Same problem but slightly different configuration. We use a M4 General Purpose Large instance based on the Ubuntu Stock Image. The build still hangs on the first Docker task and will not return. After some time, Bamboo will detect that the build is hung and you have to stop it manually. Restarting the docker service during image creation does solve the problem though.

              Unassigned Unassigned
              jowen@atlassian.com Jeremy Owen
              Affected customers:
              2 This affects my team
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: