Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-13589

Agents should remove workspaces for plans that no longer exist

    • 19
    • 26
    • Our product teams collect and evaluate feedback from a number of different sources. To learn more about how we use customer feedback in the planning process, check out our new feature policy.

      1. clean-bamboo
        3 kB
      2. clean-bamboo
        3 kB

          [BAM-13589] Agents should remove workspaces for plans that no longer exist

          sylvain.t added a comment - - edited

          it's highly ironic that the bamboo remote agents lack this very very very very (repeat 100 times) basic feature while Bamboo licensing strategy is based on how many remote agents you have...

          sylvain.t added a comment - - edited it's highly ironic that the bamboo remote agents lack this very very very very (repeat 100 times) basic feature while Bamboo licensing strategy is based on how many remote agents you have...

          Any plans around that?

          Gonchik Tsymzhitov added a comment - Any plans around that?

          nakuls added a comment -

          Ran into this issue with automatic branch management branches leaving working directories on the agents. Are there any updates on this issue? Thank you

          nakuls added a comment - Ran into this issue with automatic branch management branches leaving working directories on the agents. Are there any updates on this issue? Thank you

          Atlassian Update – 7 August 2018

          Hi everyone,

          Thanks for your interest in this issue.
          This request is considered a potential addition to our longer-term roadmap.

          We'll typically review this request in about 6 months time, at which point we’ll consider whether we need to alter its status.

          For the nearest future we've decided to prioritise other areas of the Bamboo roadmap, including:

          1. Performance and stability improvements
          2. Providing building blocks for High Availability and Disaster Recovery solutions
          3. Improving permission system
          4. Allowing per-project allocation of resources
          5. Improving Bitbucket Server and Jira integrations

          You can learn more about our approach to highly voted server suggestions here.

          To learn more on how your suggestions are reviewed, see our updated workflow for server feature suggestions.

          Kind regards,
          Bamboo Team

          Krystian Brazulewicz added a comment - Atlassian Update – 7 August 2018 Hi everyone, Thanks for your interest in this issue. This request is considered a potential addition to our longer-term roadmap. We'll typically review this request in about 6 months time, at which point we’ll consider whether we need to alter its status. For the nearest future we've decided to prioritise other areas of the Bamboo roadmap, including: Performance and stability improvements Providing building blocks for High Availability and Disaster Recovery solutions Improving permission system Allowing per-project allocation of resources Improving Bitbucket Server and Jira integrations You can learn more about our approach to highly voted server suggestions here . To learn more on how your suggestions are reviewed, see our updated workflow for server feature suggestions . Kind regards, Bamboo Team

          I found that Bill Neff's script has a couple important bugs in it, related to the regexes and wild cards.  It will delete lots of directories that it doesn't mean to.

          Here's a fixed version:

          ls $BAMBOO_AGENT_HOME/xml-data/build-dir/ | perl -ne 'print if s/^([A-Z]+-[A-Z]+\d+)-.*$/$1/' | sort | uniq | perl -nle '$status = system(qw(curl --fail -I -s), "$ENV{BAMBOO_URL}/rest/api/latest/result/$_", qw(-o /dev/null)) >> 8; if ($status == 22) {print "$_ is expired.  Delete."} elsif ($status == 0) {print "$_ is still active"} else {print "Unknown status for $_"}'

          I didn't include actually deleting the items - left as an exercise.

          Ken Williams added a comment - I found that Bill Neff's script has a couple important bugs in it, related to the regexes and wild cards.  It will delete lots of directories that it doesn't mean to. Here's a fixed version: ls $BAMBOO_AGENT_HOME/xml-data/build-dir/ | perl -ne 'print if s/^([A-Z]+-[A-Z]+\d+)-.*$/$1/' | sort | uniq | perl -nle '$status = system(qw(curl --fail -I -s), "$ENV{BAMBOO_URL}/ rest /api/latest/result/$_" , qw(-o /dev/ null )) >> 8; if ($status == 22) {print "$_ is expired.  Delete." } elsif ($status == 0) {print "$_ is still active" } else {print "Unknown status for $_" }' I didn't include actually deleting the items - left as an exercise.

          Thank you very much for your interest and support for this improvement request. With great pleasure I'd like to announce that we've decided to qualify this issue onto our short term roadmap.
          Please watch this issue for future updates.

          Krystian Brazulewicz
          Atlassian Bamboo Development Team

          Krystian Brazulewicz added a comment - Thank you very much for your interest and support for this improvement request. With great pleasure I'd like to announce that we've decided to qualify this issue onto our short term roadmap. Please watch this issue for future updates. Krystian Brazulewicz Atlassian Bamboo Development Team

          Just chiming in, we're starting to ramp up our remote agents and licences (and of course, paying the associated cost for them now) and this is becoming absolutely critical for us.  We've got scripts and hacks all over the place to keep 10 odd remote agents in check which is becoming harder to manage and account for.

           

          @bneff that script is a huge step for us in the right direction, thank you.  It's a bit unreliable though as we're not really checking if the fail is project not found in bamboo, or if the fail was API issue/network/auth/something.  But we'll look to continue with that line of thinking though. So thank you again

          Darren Nolan added a comment - Just chiming in, we're starting to ramp up our remote agents and licences (and of course, paying the associated cost for them now) and this is becoming absolutely critical for us.  We've got scripts and hacks all over the place to keep 10 odd remote agents in check which is becoming harder to manage and account for.   @bneff that script is a huge step for us in the right direction, thank you.  It's a bit unreliable though as we're not really checking if the fail is project not found in bamboo, or if the fail was API issue/network/auth/something.  But we'll look to continue with that line of thinking though. So thank you again

          Cassius, so, we do essentially the same, except that we just delete everything older than "7 days", or to bring the disk down to 70% full, which hits first instead of deleting only the expired/deleted plans.

          We then have a plan with enough jobs that theoretically every agent, if they were doing nothing, could run them in parallel, and that every agent is needed. In practice, of course, some will be doing something, but that's fine. This then checks the directory, and does the deletion. This is then run every hour. Sometimes agents are missed, but in general, every agent is hit every day, and in this way you can use Bamboo to clean itself up. Which is really something that we want Atlassian to implement!  Especially since this solution only probabilistically cleans every agent, instead of guaranteeing that every agent is hit x times per day.

          development.licensing@redwood.com added a comment - Cassius, so, we do essentially the same, except that we just delete everything older than "7 days", or to bring the disk down to 70% full, which hits first instead of deleting only the expired/deleted plans. We then have a plan with enough jobs that theoretically every agent, if they were doing nothing, could run them in parallel, and that every agent is needed. In practice, of course, some will be doing something, but that's fine. This then checks the directory, and does the deletion. This is then run every hour. Sometimes agents are missed, but in general, every agent is hit every day, and in this way you can use Bamboo to clean itself up. Which is really something that we want Atlassian to implement!  Especially since this solution only probabilistically cleans every agent, instead of guaranteeing that every agent is hit  x times per day.

          Thanks Bill.

          This actually sounds like it would be a reasonable way to go about it.  Unfortunately our agents are containerized and doing cron jobs in containers is a no-no   Perhaps we can run cron jobs on the agent/container hosts that simply instantiate new containers that attach to the build-dir persistent volumes and run your script and exit.  Then we may be cooking.

          Cassius JOHN-ADAMS added a comment - Thanks Bill. This actually sounds like it would be a reasonable way to go about it.  Unfortunately our agents are containerized and doing cron jobs in containers is a no-no   Perhaps we can run cron jobs on the agent/container hosts that simply instantiate new containers that attach to the build-dir persistent volumes and run your script and exit.  Then we may be cooking.

          Bill Neff added a comment -

          We run that script on every remote bamboo agent.

          Bill Neff added a comment - We run that script on every remote bamboo agent.

            mgardias Marcin Gardias
            pbruski Przemek Bruski
            Votes:
            281 Vote for this issue
            Watchers:
            197 Start watching this issue

              Created:
              Updated:
              Resolved: