Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-3650

Shutting down my Elastic Image doesn't delete my ebs volume

      After working with elastic bamboo for a while on BEAC, I make changes to the volume, generate my new snapshot then shutdown all my agents and start them up with a new volume. If i do that enough, the volume list in the AWS console will get full and I wont be able to create any more volumes, stopping me from starting any working agents.

      Bamboo should delete the volume when it shuts down an agent, it doesn't seam to be doing that.

            [BAM-3650] Shutting down my Elastic Image doesn't delete my ebs volume

            I wanted to ask you whether it possible to download "early access" version of Bamboo containing this issue fixed?

            I was trying to find Bamboo 2.3 M2 ("Release Date: 03/Apr/09" according to: http://jira.atlassian.com/browse/BAM/fixforversion/14520 JIRA page),
            but there's noting here: http://www.atlassian.com/software/bamboo/BambooEAPDownloads.jspa saying "There are currently no EAP releases, please visit the download centre to download the latest versions."
            (compared to Confluence's page - http://www.atlassian.com/software/confluence/DevReleaseDownloads.jspa - which has some "early access" stuff kindly published by you )

            So, should I better wait till 2.2.2 or try building from source according to http://confluence.atlassian.com/display/BAMBOO/Building+a+Bamboo+War+Distribution+From+Source ?

            Kind regards,
            Krystian Nowak

            Krystian Nowak added a comment - I wanted to ask you whether it possible to download "early access" version of Bamboo containing this issue fixed? I was trying to find Bamboo 2.3 M2 ("Release Date: 03/Apr/09" according to: http://jira.atlassian.com/browse/BAM/fixforversion/14520 JIRA page), but there's noting here: http://www.atlassian.com/software/bamboo/BambooEAPDownloads.jspa saying "There are currently no EAP releases, please visit the download centre to download the latest versions." (compared to Confluence's page - http://www.atlassian.com/software/confluence/DevReleaseDownloads.jspa - which has some "early access" stuff kindly published by you ) So, should I better wait till 2.2.2 or try building from source according to http://confluence.atlassian.com/display/BAMBOO/Building+a+Bamboo+War+Distribution+From+Source ? Kind regards, Krystian Nowak

            We had to stop the instance because it took up a volume and we couldn't start any new ones!

            m@ (Inactive) added a comment - We had to stop the instance because it took up a volume and we couldn't start any new ones!

            I removed reasons of hanging orphan volumes.
            However if such error will happen again, please don't stop the instance - just give an instance details to check the reason why volume was not attached - maybe logs will contain some useful information.

            Marek Went (Inactive) added a comment - I removed reasons of hanging orphan volumes. However if such error will happen again, please don't stop the instance - just give an instance details to check the reason why volume was not attached - maybe logs will contain some useful information.

            Volume has to be removed AFTER new snapshot is created.

            Marek Went (Inactive) added a comment - Volume has to be removed AFTER new snapshot is created.

            1. You change the volume and create new snapshot. Your instance is running.
            2. You have to change agent settings - specify new snapshot ID to be used when agent starts - Bamboo UI.
            3. You stop instance? Or just agent?
            4. You start instance again?

            If I need to make an update to my volume I'll:

            1. Shutdown all but one of my elastic instances, and on the final instance disable the agent.
            2. SSH to the final instance and kill the agent process.
            3. Make the changes to my volume and test them.
            4. Create a snapshot from the command line
            5. Update the bamboo config to use the new snapshot and start a new instance
            6. Test the new instance has the updated volume and works properly.
            7. Shut down the original instance and start up some more agents to match the previous number.

            It's usually not until I start up an instance and notice that the builds aren't working that I realise I have run out of volumes. If anything, some feedback in bamboo during the boot process about being unable to attach the volume would be a start.

            Maybe:

            • By leaving an instance running with the original image this one isn't getting detached?
            • I installed some other software, postgres etc (details here) which may mean the volume is in use and could not be unmounted during the shutdown process

            m@ (Inactive) added a comment - 1. You change the volume and create new snapshot. Your instance is running. 2. You have to change agent settings - specify new snapshot ID to be used when agent starts - Bamboo UI. 3. You stop instance? Or just agent? 4. You start instance again? If I need to make an update to my volume I'll: Shutdown all but one of my elastic instances, and on the final instance disable the agent. SSH to the final instance and kill the agent process. Make the changes to my volume and test them. Create a snapshot from the command line Update the bamboo config to use the new snapshot and start a new instance Test the new instance has the updated volume and works properly. Shut down the original instance and start up some more agents to match the previous number. It's usually not until I start up an instance and notice that the builds aren't working that I realise I have run out of volumes. If anything, some feedback in bamboo during the boot process about being unable to attach the volume would be a start. Maybe: By leaving an instance running with the original image this one isn't getting detached? I installed some other software, postgres etc (details here ) which may mean the volume is in use and could not be unmounted during the shutdown process

            Marek Went (Inactive) added a comment - - edited

            Bamboo does detach and delete volume on agent exit (sometimes even more than once).
            So, this is not just not deleting volume, problem has to be somehow related to real environment contraints (maybe to mch volumes at some point of time).
            Deleted volume exists some time in the could after delete (in deleting state) - maybe this is related to this fact.

            Just to clarify:
            1. You change the volume and create new snapshot. Your instance is running.
            2. You have to change agent settings - specify new snapshot ID to be used when agent starts - Bamboo UI.
            3. You stop instance? Or just agent?
            4. You start instance again?

            Maybe such sequence doesn't remove "old" snapshot volume - so we end up with "zombie" volumes?

            I'll investigate it, but if you have any suggestions - would be nice.

            Marek Went (Inactive) added a comment - - edited Bamboo does detach and delete volume on agent exit (sometimes even more than once). So, this is not just not deleting volume, problem has to be somehow related to real environment contraints (maybe to mch volumes at some point of time). Deleted volume exists some time in the could after delete (in deleting state) - maybe this is related to this fact. Just to clarify: 1. You change the volume and create new snapshot. Your instance is running. 2. You have to change agent settings - specify new snapshot ID to be used when agent starts - Bamboo UI. 3. You stop instance? Or just agent? 4. You start instance again? Maybe such sequence doesn't remove "old" snapshot volume - so we end up with "zombie" volumes? I'll investigate it, but if you have any suggestions - would be nice.

            m@ (Inactive) added a comment - This happened again: https://bamboo.extranet.atlassian.com/browse/CONFFUNC-UNIT-95/

            The main problem is when an instance is started its not clear that it could not allocate an instance so the agent starts and tries to build stuff. It ends up failing because nothing has been configured.

            CONFFUNC-UNIT-44 is an example of what happens.

            m@ (Inactive) added a comment - The main problem is when an instance is started its not clear that it could not allocate an instance so the agent starts and tries to build stuff. It ends up failing because nothing has been configured. CONFFUNC-UNIT-44 is an example of what happens.

              mwent Marek Went (Inactive)
              mjensen m@ (Inactive)
              Affected customers:
              0 This affects my team
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: