Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-13588

Deployment projects don't start EC2 instances automatically

      I was able to reproduce the behavior when regular plans started EC2 instances and worked fine, but the deployment project stayed queued after passing the configured 1 minute threshold. See the attached screenshots.

      The deployment ran successfully after manually starting an elastic agent.

            [BAM-13588] Deployment projects don't start EC2 instances automatically

            At 'Mon Jul 28 23:35:40 EST 2014', actually. So they meant last Monday

            Robert Watkins added a comment - At 'Mon Jul 28 23:35:40 EST 2014', actually. So they meant last Monday

            Well, it hit my OD environment sometime overnight.

            Robert Watkins added a comment - Well, it hit my OD environment sometime overnight.

            Ops Team added a comment -

            Atlassian confirmed on Twitter that OnDemand is scheduled to receive 5.6 this coming Monday.

            https://twitter.com/atlassian/status/494222979325517824

            Ops Team added a comment - Atlassian confirmed on Twitter that OnDemand is scheduled to receive 5.6 this coming Monday. https://twitter.com/atlassian/status/494222979325517824

            Well, late July. But I can confirm that it does work.

            Robert Watkins added a comment - Well, late July. But I can confirm that it does work.

            This bug will be released with Bamboo 5.6 in early July.

            James Dumay added a comment - This bug will be released with Bamboo 5.6 in early July.

            @Tomasz Create a timed instance inside Bamboo instance image configuration and set it for every 15 minutes.
            This will check and ensure that you have a Deployment agent running every 15 minutes. If it shuts down due to timed termination of some sort, the check you create in bamboo will start another.

            Michael Dalzell added a comment - @Tomasz Create a timed instance inside Bamboo instance image configuration and set it for every 15 minutes. This will check and ensure that you have a Deployment agent running every 15 minutes. If it shuts down due to timed termination of some sort, the check you create in bamboo will start another.

            That is unfortunately a major bummer. Is there a know workaround? It there a way to use own servers for build and deployment purposes instead of EC2 instances? Anyone?

            Tomasz Rdzak added a comment - That is unfortunately a major bummer. Is there a know workaround? It there a way to use own servers for build and deployment purposes instead of EC2 instances? Anyone?

            The fix to this issue has been scheduled for 5.6

            Marcin Gardias added a comment - The fix to this issue has been scheduled for 5.6

            Deployment projects are a fundamental feature of Bamboo. I'm considering going back to just deploying software from build projects... why on earth was the entire concept of deployment projects put into production when it doesn't even work correctly?

            In 23 days this issue will be ONE YEAR old, with no updates except "oops, we closed the ticket". Complete fail on all fronts.

            jason.mitcheson added a comment - Deployment projects are a fundamental feature of Bamboo. I'm considering going back to just deploying software from build projects... why on earth was the entire concept of deployment projects put into production when it doesn't even work correctly? In 23 days this issue will be ONE YEAR old, with no updates except "oops, we closed the ticket". Complete fail on all fronts.

            Ops Team added a comment -

            Sad to see this is still an issue. Had to pull the plug on our implementation of AoD Bamboo because of it and the lack of deployment plan cloning.

            Ops Team added a comment - Sad to see this is still an issue. Had to pull the plug on our implementation of AoD Bamboo because of it and the lack of deployment plan cloning.

            We have also had instances shut down during deployment. Its frustrating that I need to contact Atlassian every time this happens to have our on demand instance restarted. This can leave us hanging for an our and having to revert back to older deployment methods.

            I have implemented a solution to check and ensure that a deployment agent is running every 15 minutes. The down side to this, if an instance shuts down there is a possable 15 minute wait to have it re-spawned then normally a 5-10 for the instance to become available.

            Very frustrating.

            Michael Dalzell added a comment - We have also had instances shut down during deployment. Its frustrating that I need to contact Atlassian every time this happens to have our on demand instance restarted. This can leave us hanging for an our and having to revert back to older deployment methods. I have implemented a solution to check and ensure that a deployment agent is running every 15 minutes. The down side to this, if an instance shuts down there is a possable 15 minute wait to have it re-spawned then normally a 5-10 for the instance to become available. Very frustrating.

            We've had multiple instance shut down during a deployment, Bamboo doesn't report this as a fail and we need to reach out to Atlassian to restart the Bamboo service for us when it happens.

            Deployments having no visibility or control over instances is costing us money as we're wasting time requesting support and re-running deployments that should have already taken place. This issue is an important one for us.

            Luke Huxley added a comment - We've had multiple instance shut down during a deployment, Bamboo doesn't report this as a fail and we need to reach out to Atlassian to restart the Bamboo service for us when it happens. Deployments having no visibility or control over instances is costing us money as we're wasting time requesting support and re-running deployments that should have already taken place. This issue is an important one for us.

            This week I observed that we had one elastic instance running and it shut down only a few minutes after completing a deployment. Yes our inactivity timeout is 120 minutes.

            Apart from the fact that instances are not spawned in order to process a queued deployment, it would seem a deployment does not extend the time before an instance is shut down due to inactivity.

            I have never seen an instance shut down during deployment, and think this is unlikely. Yet, if we had a problem with the release and needed to redeploy an earlier version (roll back), I would hope the instance is still running!

            Lars Janssen added a comment - This week I observed that we had one elastic instance running and it shut down only a few minutes after completing a deployment. Yes our inactivity timeout is 120 minutes. Apart from the fact that instances are not spawned in order to process a queued deployment, it would seem a deployment does not extend the time before an instance is shut down due to inactivity. I have never seen an instance shut down during deployment, and think this is unlikely. Yet, if we had a problem with the release and needed to redeploy an earlier version (roll back), I would hope the instance is still running!

            +1

            It's been over a month - any progress?

            J. Matthew Pryor added a comment - +1 It's been over a month - any progress?

            We're having to request the Bamboo service get restarted quite regularly as elastic instances dedicated to deployments are getting terminated before the deployments are complete, this would not be happening if deployment projects were spawning the elastic instances.

            What's the progress on this bug?

            Luke Huxley added a comment - We're having to request the Bamboo service get restarted quite regularly as elastic instances dedicated to deployments are getting terminated before the deployments are complete, this would not be happening if deployment projects were spawning the elastic instances. What's the progress on this bug?

            Ditto - thanks for looking into this Nathan.

            Scott Carpenter added a comment - Ditto - thanks for looking into this Nathan.

            Thanks

            Jason Mitcheson added a comment - Thanks

            NathanA added a comment -

            This issues is marked as resolved - I assume you are still encountering this problem? I'll re-open the issue so that Support can have a look at it.

            NathanA added a comment - This issues is marked as resolved - I assume you are still encountering this problem? I'll re-open the issue so that Support can have a look at it.

            Atlassian?! Hello?!

            Jason Mitcheson added a comment - Atlassian?! Hello?!

            We have run into this issue as well - and again for us it effectively breaks Bamboo as an automated deployment tool.

            Scott Carpenter added a comment - We have run into this issue as well - and again for us it effectively breaks Bamboo as an automated deployment tool.

            ... and here.
            I still have to run this work around to check for running instances.

            Mikey

            Michael Dalzell added a comment - ... and here. I still have to run this work around to check for running instances. Mikey

            laufzeit added a comment -

            Same here.

            laufzeit added a comment - Same here.

            I am still seeing this in production, and I do not see a fix version set on the ticket yet. What gives?

            Jason Mitcheson added a comment - I am still seeing this in production, and I do not see a fix version set on the ticket yet. What gives?

            Hi there,

            It seems like this has been erroneously resolved before the work in progress has been signed off.

            I will update the ticket with the correct resolution and fix version when its ready to go.

            Apologies,
            James Dumay

            James Dumay added a comment - Hi there, It seems like this has been erroneously resolved before the work in progress has been signed off. I will update the ticket with the correct resolution and fix version when its ready to go. Apologies, James Dumay

            Good to see this one has gotten some attention. Do we have an idea of when this is going to hit AoD (or has it already)?

            Matthew Hillier added a comment - Good to see this one has gotten some attention. Do we have an idea of when this is going to hit AoD (or has it already)?

            For anyone who's interested, we've decided to wire all of our deployments projects and environments to a dedicated Instance Configuration, which is using the standard EBS bamboo image on a micro instance, and then configured an Instance Schedule to check that at least 2 of these are running every 15 minutes. It's not fool-proof, however it means that we can now deploy almost on-demand (maximum of 15mins delay if the instance happens to have stopped), for a small price (~$15 per month, per instance).

            Matthew Hillier added a comment - For anyone who's interested, we've decided to wire all of our deployments projects and environments to a dedicated Instance Configuration, which is using the standard EBS bamboo image on a micro instance, and then configured an Instance Schedule to check that at least 2 of these are running every 15 minutes. It's not fool-proof, however it means that we can now deploy almost on-demand (maximum of 15mins delay if the instance happens to have stopped), for a small price (~$15 per month, per instance).

            +1
            This has rendered the deployment plans for us a little useless.

            Michael Dalzell added a comment - +1 This has rendered the deployment plans for us a little useless.

            Mihai added a comment -

            @Matthew you may ask the users who don't have access to start instances manually to watch this issue. I was told that if it gets enough votes Atlassian may consider to fix it in a future version though that would mean in a year or so. Until then good luck !

            Mihai added a comment - @Matthew you may ask the users who don't have access to start instances manually to watch this issue. I was told that if it gets enough votes Atlassian may consider to fix it in a future version though that would mean in a year or so. Until then good luck !

            This is going to cause us some serious issues as we have users who need to be able to deploy, but won't necessarily have access to start instances manually.

            Also, in it's current configuration, if I setup a dedicated Instance Configuration to ensure that Deployments aren't dependant on Build instances, then Bamboo will never automatically create an instance of the Deployer.

            Matthew Hillier added a comment - This is going to cause us some serious issues as we have users who need to be able to deploy, but won't necessarily have access to start instances manually. Also, in it's current configuration, if I setup a dedicated Instance Configuration to ensure that Deployments aren't dependant on Build instances, then Bamboo will never automatically create an instance of the Deployer.

            Bart Smyth added a comment -

            Broken for me too. Annoyed to see this hasn't worked for over a month.

            Bart Smyth added a comment - Broken for me too. Annoyed to see this hasn't worked for over a month.

            Mihai added a comment -

            So the elastic agent is not elastic. I'm wondering what kind of QA is in place there. It's a basic function of the CI .

            Mihai added a comment - So the elastic agent is not elastic. I'm wondering what kind of QA is in place there. It's a basic function of the CI .

            Doh! I don't really want to have to use http://aws.amazon.com/mturk/ alongside http://aws.amazon.com/ec2/

            +1 (natch)

            Tim O'Donoghue added a comment - Doh! I don't really want to have to use http://aws.amazon.com/mturk/ alongside http://aws.amazon.com/ec2/ +1 (natch)

            Our testers are currently stuck waiting for a build to start an instance before being able to perform a deployment.

            Luke Huxley added a comment - Our testers are currently stuck waiting for a build to start an instance before being able to perform a deployment.

            +1, what is the point of having instances fire up automatically for builds, but not deploys?

            We might make a build, then wait for certain stakeholders to build and sign off, then click the 'deploy' button. The EC2 instance might not be still running at that point.

            Lars Janssen added a comment - +1, what is the point of having instances fire up automatically for builds, but not deploys? We might make a build, then wait for certain stakeholders to build and sign off, then click the 'deploy' button. The EC2 instance might not be still running at that point.

            Any update on this? Without this, scheduled deployments are completely useless.

            Robert Watkins added a comment - Any update on this? Without this, scheduled deployments are completely useless.

              mgardias Marcin Gardias
              akhachatryan ArmenA
              Affected customers:
              37 This affects my team
              Watchers:
              44 Start watching this issue

                Created:
                Updated:
                Resolved: