EBS volume handling needs a revamp

Motivation:

https://support.atlassian.com/browse/JST-5357

Populate Volumes from S3 and keep the volumes active when elastic instance shuts down.

My library EBS is not normally loaded from a snapshot. In fact I use a snapshot only to build a new library EBS when Bamboo has deleted one. I have two library EBS volumes, one for us-east-1a and one for us-east-1b, and my script picks the appropriate one for the availability zone. What happens as part of the script is that the MD5 hash of the zipped library is obtained from S3 and compared against one stored on the EBS. If they differ then the zipped library is downloaded and an unzip update is performed, the new MD5 replaces the old one.

I'll attach the scripts for clarity.

So each volume will be brought up to date the first time it gets mounted after the libraries have changed.

This all seems to work fine, except for those occasions on which my library EBS discs are mysteriously deleted when an instance ends.

LATER:

Setting the shut down cron entry to "set instance to 0" instead of "shutdown all instances" doesn't make any difference, both ways the volume is deleted. Closing down an instance manually from the admin console is fine.

It seems there is another way of dealing with attached volumes using the SOAP api and do less scripting.

I've done some further investigation. I had got away with specifically closing down the instance from the bamboo console a couple of times but on other occasions the deletions still occurred. The running instance, at scripting level, doesn't seem to get the opportunity to detach the library EBS because terminate simply shuts it down. For fairly obvious reasons EC-2 doesn't wait around for rc0.d scripts to finish before closing the instance. The work-around I'm going to try for now is to create an timed "at" job to detach my volume just before the time at which the scheduler normally shuts it down, obviously clumsy and requires messing with if we change our schedules or do ad-hock timings. Also any builds between the time the at job fires and shut down will fail.

What I don't understand is why you don't use the "delete on terminate" attachment setting, which should eliminate the need for all this explicit volume handling by Bamboo. You can specify EBS mounts from snapshot on the runInstances call, and specify, at that point that the volumes are to be deleted on instance termination. see http://docs.amazonwebservices.com/AWSEC2/latest/CommandLineReference/index.html?ApiReference-cmd-StartInstances.html or you can set deleteOnTerminate with the ec2-modify-instance-property (though the command line version is a bit dodgy.)

I've checked and the deleteOnTerminate flag is false on both attached volumes. (Incidentally the ec2-describe-instance-property and the associated modify is missing from the ec2 api on the instance, are you working with an old API version?)

What I'd like to get a look at is the instance termination code in the server, then I wouldn't be working so much in the dark. What I suppose is happening is something like:

1) SOAP call to request the agent to close
2) Attached volumes retrieve using describeVolumes
2) terminateInstance called.
4) volumes from 2) deleted.

with the checkElasticAgent.sh just there as a backstop in case the agent goes down but the terminateInstance fails or isn't called.

I think the current volume management system needs a revamp, though whether your would regard this as a bug report or a request for enhancement is another matter.