it's highly ironic that the bamboo remote agents lack this very very very very (repeat 100 times) basic feature while Bamboo licensing strategy is based on how many remote agents you have...

Ran into this issue with automatic branch management branches leaving working directories on the agents. Are there any updates on this issue? Thank you

I found that Bill Neff's script has a couple important bugs in it, related to the regexes and wild cards.  It will delete lots of directories that it doesn't mean to.

Here's a fixed version:

ls $BAMBOO_AGENT_HOME/xml-data/build-dir/ | perl -ne 'print if s/^([A-Z]+-[A-Z]+\d+)-.*$/$1/' | sort | uniq | perl -nle '$status = system(qw(curl --fail -I -s), "$ENV{BAMBOO_URL}/rest/api/latest/result/$_", qw(-o /dev/null)) >> 8; if ($status == 22) {print "$_ is expired.  Delete."} elsif ($status == 0) {print "$_ is still active"} else {print "Unknown status for $_"}'

I didn't include actually deleting the items - left as an exercise.

Thank you very much for your interest and support for this improvement request. With great pleasure I'd like to announce that we've decided to qualify this issue onto our short term roadmap.
Please watch this issue for future updates.

Krystian Brazulewicz
Atlassian Bamboo Development Team

Just chiming in, we're starting to ramp up our remote agents and licences (and of course, paying the associated cost for them now) and this is becoming absolutely critical for us.  We've got scripts and hacks all over the place to keep 10 odd remote agents in check which is becoming harder to manage and account for.

@bneff that script is a huge step for us in the right direction, thank you.  It's a bit unreliable though as we're not really checking if the fail is project not found in bamboo, or if the fail was API issue/network/auth/something.  But we'll look to continue with that line of thinking though. So thank you again

Cassius, so, we do essentially the same, except that we just delete everything older than "7 days", or to bring the disk down to 70% full, which hits first instead of deleting only the expired/deleted plans.

We then have a plan with enough jobs that theoretically every agent, if they were doing nothing, could run them in parallel, and that every agent is needed. In practice, of course, some will be doing something, but that's fine. This then checks the directory, and does the deletion. This is then run every hour. Sometimes agents are missed, but in general, every agent is hit every day, and in this way you can use Bamboo to clean itself up. Which is really something that we want Atlassian to implement!  Especially since this solution only probabilistically cleans every agent, instead of guaranteeing that every agent is hit x times per day.

Thanks Bill.

This actually sounds like it would be a reasonable way to go about it.  Unfortunately our agents are containerized and doing cron jobs in containers is a no-no   Perhaps we can run cron jobs on the agent/container hosts that simply instantiate new containers that attach to the build-dir persistent volumes and run your script and exit.  Then we may be cooking.

We run that script on every remote bamboo agent.

Bill: Thanks! Does this assume all agents are on the local host and not remote agents?

Paul: Just to be clear, my comment about hack scripts wasn't referring to your python scripts.  More a general comment that it shouldn't be necessary for anyone to have to do it.

This time as an actual code block...

ls $BAMBOO_AGENT_HOME/xml-data/build-dir/ | grep '.*-.*-.*' | perl -pe 's/^(.*-.*)-.*$/$1/' | sort | uniq | while read PROJECT ; do if ! curl --fail -I -s "$BAMBOO_URL/rest/api/latest/result/$PROJECT" > /dev/null; then echo "$PROJECT is expired.  Delete it"; rm -rf $BAMBOO_AGENT_HOME/xml-data/build-dir/$PROJECT*; else echo "$PROJECT is still active" ; fi; done

While we also wait for this to be fixed, we use a command line script that we cron on a weekly basis that has results similar to the attached python script, intended to only clean up the working directory of expired builds.  Just set or replace the $BAMBOO_AGENT_HOME and $BAMBOO_URL variables appropriately.

```

ls $BAMBOO_AGENT_HOME/xml-data/build-dir/ | grep '...' | perl pe 's/^(..)-.$/$1/' | sort | uniq | while read PROJECT ; do if ! curl --fail -I -s "$BAMBOO_URL/rest/api/latest/result/$PROJECT" > /dev/null; then echo "$PROJECT is expired. Delete it"; rm -rf $BAMBOO_AGENT_HOME/xml-data/build-dir/$PROJECT*; else echo "$PROJECT is still active" ; fi; done

```

I agree that this is not a minor problem. That every customer has to jury rig their own solution for this is extraordinarily surprising in my eyes.

For certain Atlassian will have rigged up a solution for this using scripts or some such, or using a solution similar to the "monit" that was mentioned on one forum. Sadly that "monit" is no longer available.

There should, at the very least, be an example script that we can deploy to clean these areas up automatically. ideally this should be available out of the box. We have been playing around with the different options assuming that we were just not configuring it correctly, when it turns out that what we are trying to do is simply not possible.

This really should be fixed and an out of the box option given to clean up disk space for deleted plans.

+1 this is NOT minor.

How is it possible that this ticket has been open for 4 years and such a vital function is being ignored?

Does Atlassian not use Bamboo for doing their own builds? Because if they do then they're certainly experiencing this problem too.  Fix it to help yourselves, Atlassian!  Having your customers write hack scripts to cleanup after Bamboo isn't a solution.

+1 for sooner than later, very annoying and error prone on live servers...

I need to manually cleanup build directories every week now in 5 remote agents, gets really annoying, I dont want to do that. I have the build expiry on to expire everything, but obviously it does not include build dirs. People have full node_modules, maven repos etc in those dirs so disk spac gets eaten really fast. 

This is ridiculous! Hw hasn't it been fixed yet, disk space is not infinite!

Kicking this issue. It's been Open sinds 2013 :O
It has a Low priority, which is really not justified as this is a serious issue (with probably an easy fix as the cleanup does happen on the default agent).
It's labeled as an improvement, but I actually think this is a bug as it does work correctly on the default agent (server).

It happened multiple times that builds failed due to lack of disk space because of this.
For modern automation software, it's weird that this is bugged (for 4+ years).

As well, experiencing that when plan branches are enabled, we rapidly run out of disk space because we use the feature branch model for development.

The only workaround solution that I found was by purging the xml-data/build-dir/ when it is full.

Any updates on how to better handle this? Is the Bamboo Predator Plugin the way to go?

Nice find!  Will try that out immediately.

Greetings everyone,

I found Bamboo Predator Plugin from Atlassian Labs. It can help.

New version available.  I don't know how to delete the old version so just check the dates.

This one uses the REST interface so it's much more reliable than the old method.

Thanks Paul

I realize I haven't updated the clean script I attached to this JIRA; the one here no longer works due to changes in the way the Bamboo server responds to requests for non-existent plan branch pages.  I'll post a fixed version.

Can we get an update on this feature request? 

If knowing when to delete the working directories is an issue, perhaps a simpler approach would be something Atlassian could implement in the near term. Say, delete any working directories more than N-days old? Or keep the last N working directories for a given plan? Where N is configurable by the user.

Whats the point of voting, if requests dont get bumped up to the top of the roadmap ??

Any update on this ?

This is now using 116Gb of our disk space - and we are expected to manually clean up the garbage left behind by Bamboo - seriously?

This should be a bug in my opinion. I was under the impression that the "expiry" feature also handles the remote agents.

Do you have na update on this matter for us please.
We are well away in the next year now.

Thanks,

I.

Hi,

I just wanted to provide an update on this request. At this stage it's unlikely that we'll address it before the end of the year.

Thanks everyone for your feedback, please know that we read it and take it into account when prioritising our roadmap but due to other factors we are unable to put this development on top of the backlog.

Thanks,

Sten Pittet
Bamboo Product Manager

+1 sooner than later.

Very surprised (or not) that this is still "minor" priority, unassigned, and has received no comments or input from any Atlassian reps. For a company that so strongly promotes git flow and branching workflows, this is an embarrassing oversight that should be fixed ASAP. It's a major pain.

+1 addressing this sooner than later. This issue is annoying. Additional to the disk running full we have problems with Inodes being used up. With NPM it's trivial to create tens of thousands of files with every branch. This uses all available Inodes fast and additional builds fail with an "No space left on device" message.

The script is attached to this issue. You need to change the "bamboo_url" variable to point to your Bamboo server URL, obviously. By default, this script doesn't do anything it just shows you what it would do. It make it do something, use the --run flag. Also, it doesn't delete things immediately, normally: instead it moves them into a different directory (.oldbuilds) just in case. There's a --clean option which deletes the previous contents of .oldbuilds before cleaning new stuff. And, you need to pass a the directory to be checked as the argument: by default it checks BUILDDIR in the user's home directory, which I make a symlink to point to the workspace directory (not on Windows of course ).

Basically I run the script in a cron job with --run --clean; this will clean out builds from the previous run, then clean new builds, once a day. This gives me a day to recover a build if, somehow, it got removed by accident (note, this has never happened to us, not once, but...)

One other safety feature: because we decide a build is no good if we can't access its page on the bamboo server, if the script discovers that ALL the builds are marked for deletion it will not do anything, because it figures that something is wrong with the bamboo server or your access to it.

@psmith1 That cleanup script sounds really useful - can you share?

+1 for seeing this on the short-term roadmap - Bamboo is a disk space hog and we don't all have S3 at our disposal.

I agree that issues like this continue to show how disappointing Atlassian support is, particularly for Bamboo and Crucible/Fisheye.

I don't have a perfect solution to this but what I did locally is implement a Python script that runs on each agent and queries the web interface for each workspace that it finds on the local filesystem, and deletes the workspaces for plans that don't exist on the Bamboo server. I wrote it in Python so it would work the same across all of our build platforms (GNU/Linux, Mac OSX, Windows).

I then arrange for this script to be run daily on all our build agents (via cron or whatever).

It's not ideal (in particular it doesn't necessarily solve the issue Ron Chan described above, with reuse of workspaces, unless it just so happens that the cleanup script runs before the workspace is reused... in our environment reusing workspaces is actually not such a problem as we have very complete dependency information so whatever is out of date gets rebuilt anyway) but it does work and keeps us from having to constantly worry about disk space on our build agent systems.

I can publish it if people are interested.

Amen, brother. Definitely not "minor."

With feature based development, this is killing us.

How can this issue be minor?
In any feature-based development environment this will lead to build errors due to expired disk space on a very regular basis.

So again I need to implement my own workarounds & fixes for a major usability problem of an Atlassian product which could and should be easily fixed by them. For now I'll probably have to do this. But this time after I have finalized our migration strategy to TeamCity, which goes far beyond and this, has Automatixc Checkout cleaning, ensures that there is enough free space left before executing a plan, offers a disk usage report and even has a Disk Space Watcher

Teamcity vs. Atlassian: 4:0

Is there any chance of this getting done soon?

I agree totally with Geoff and Kelly and the other 41 voters. Native branch support was a primary reason we switched to Bamboo. If it's not cleaning itself up (like on our feature branches), it is also killing us and is not as great a feature as I expect. +1 for seeing this on the short-term roadmap.

+1 for seeing this on the short-term roadmap

With feature based development, this is killing us. We are having to allocate greater and greater disk space to our build nodes and having to touch the servers more often to try and stay ahead of "disk full" errors. Being able to specify how long to keep the workspace folder, even if it's controlled per agent and not per job, is a key piece of functionality missing from Bamboo for us.

-Kelly G. Schoenhofen

We've come across an issue on how this "reuse of branches" causes problems with Deployment projects.

• Create a release ("release-PROJ-1") and link to build PROJ build #1 (e.g. Bamboo url http://localhost:8085/browse/PROJ0-1)
• Deploy this
• Then at some point when Bamboo expires this particular branch
• Creating a new branch, and say it happens to use the same url
• When you try to create a release linking to this new branch, Bamboo says:
  "No release with the artifacts of this build exists."

Seems the Bamboo release will still reference the url of the old branch even though a new one has been created (which just happens to reuse it).

This has got to be a bug. Has anyone seen this before?

The only short-term workaround I can think of is to have Bamboo NOT expire any of our branches, so that each branch url is distinct. However, since we have over 200 build plans and use branches heavily, that also prevents us from easily cleaning up unreleased branches.

ah, never mind, it's in the URL.

One other thing you might consider. At least display the "real" working directory name on the build page somewhere so that if you have to do it by hand you are at least certain that the plan's directory is the correct one. We branch frequently (with each release) and so have many plans. You currently have to dig in the logs a bit to figure out which working directory goes with which branch.

Thanks for the info Filip. The actual deleting of the working directory isn't the difficult part, it's knowing when to delete it: when it's no longer needed by Bamboo. And also doing so in an automated way rather than by hand.

However, can you describe why you're deleting only parts of the working directory? Why not simply delete the entire working directory?

In the mean time we are using a ugly batch script to fix this:

ECHO Build directory: %BUILD_PATH%
ECHO Working directory: %WORKING_PATH%
ECHO Artifact directory: %ARTIFACT_PATH%
ECHO Repository directory: %REPOSITORY_PATH%

ECHO Removing all subfolders of working directory except folder %ARTIFACT_DIR%
FOR /D %%d IN ("%WORKING_PATH%*") DO IF /I "%%d" NEQ "%ARTIFACT_PATH%" RD "%%d" /S /Q

ECHO Removing all files directly below working directory.
FOR %%d IN ("%WORKING_PATH%*") DO del "%%d" /Q

ECHO Removing old working directories except %REPOSITORY_PATH%
FOR /D %%d IN ("%BUILD_PATH%*") DO IF /I "%%d" NEQ "%WORKING_PATH%" IF /I "%%d" NEQ "%REPOSITORY_PATH%" RD "%%d" /S /Q

EXIT 0