Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-15792

Prevent Bamboo source code checkout if Bitbucket Server is in maintenance mode

    • 15
    • 22
    • Our product teams collect and evaluate feedback from a number of different sources. To learn more about how we use customer feedback in the planning process, check out our new feature policy.

      On Using the Stash Backup Client documentation, in the How it works we have:

      2. Checks that all Git and database operations have completed.

      Based on the information above, if running a build in Bamboo that has "Source Code Checkout" task using a Stash repository, this build should have no problems on completing the cloning task before Stash going on Maintenance mode.

      However, when Stash is in Maintenance mode by Using Stash DIY Backup documentation, the same Bamboo build does not happens and the following error gets thrown:

      ... Unable to detect changes. This error may have occurred due to an authentication issue with Stash. Save the repository configuration to refresh the authentication details.
      (25 Mar 2015, 9:10:11 AM)
      

      It would be interesting on having a feature in Bamboo that when a Repository host "Stash" is selected, prior running a "Source Code Checkout" task, Bamboo would call a REST API requesting Stash status and, in case, the same returns PAUSED, MAINTENANCE the build in Bamboo would stop and a notification message will explain appropriated the reason.

          Form Name

            [BAM-15792] Prevent Bamboo source code checkout if Bitbucket Server is in maintenance mode

            Amanda Culver added a comment - - edited

            This is now 5 years old and you're STILL considering it ?

            Amanda Culver added a comment - - edited This is now 5 years old and you're STILL considering it ?

            Are there any news regarding this?

            Alfonso Murolo added a comment - Are there any news regarding this?

            "Expect to hear an update on our progress within the next 6 months." - July 2018

            Which month measurement are you using at Atlassian?

            Deleted Account (Inactive) added a comment - - edited "Expect to hear an update on our progress within the next 6 months." - July 2018 Which month measurement are you using at Atlassian?

            @kbrazulewicz
            What is the ETC on delivering a fix for this Bug?
            The last update from Atlassian was in 2018.

            James Brown added a comment - @kbrazulewicz What is the ETC on delivering a fix for this Bug? The last update from Atlassian was in 2018.

            Hi Mihai

            As far as I can tell the issue is still present. I've found two cases where PAUSE can cause problems. 

            On build startup the scenario/flow is this:

            1. Build is queued when everything is running
            2. Bitbucket goes into backup maintenance mode
              1. Prior to starting backup where Bitbucket is offline, we set Bamboo state to PAUSED to prevent it checking for new commits, with errors in log as a consequence.
            3. Bamboo is now paused (but build is still in queue)
            4. Build is dequeued, and starting to build 
            5. On Checkout (if not the workaround is implemented), it fails to checkout sourcecode as Bitbucket/Git is offline. And build fails

            In this scenario, it should either not start new builds even though they are queued or at least be able to wait for Bitbucket to become online again so it can checkout sourcecode and perform the build. 

            Not fail build as we have clearly signaled to Bamboo that it should PAUSE it's actions. 

             

            On build finalize (artifact upload) the scenario/flow is this

            1. Build is running (can be running or in queue before PAUSE state is set)
            2. Bamboo goes into pause state (no matter the reason - but on our installation again due to Bitbucket backup)
            3. When build is finished with success, Bamboo starts to upload artifacts (while in paused state). 
              1. If Bamboo is paused, this fails, but the build just hangs there. So something fails, but it never "dies". Consequence is that a build agent is busy until the job is manually stopped.

            If the same work around is added as a final task, we now wait for Bamboo paused state to end, and then Bamboo gets to upload artifact. 

            In this scenario I would expect the product to know that if PAUSED then it needs to wait uploading artifacts until Bamboo is no longer paused, and it is ready to accept incoming artifact requests from build agents. 

             

            Let me know if the above still isn't clear enough. 

            Best regards

            /Anders

            Anders Kjærgaard Hansen added a comment - Hi Mihai As far as I can tell the issue is still present. I've found two cases where PAUSE can cause problems.  On build startup the scenario/flow is this: Build is queued when everything is running Bitbucket goes into backup maintenance mode Prior to starting backup where Bitbucket is offline, we set Bamboo state to PAUSED to prevent it checking for new commits, with errors in log as a consequence. Bamboo is now paused (but build is still in queue) Build is dequeued, and starting to build  On Checkout (if not the workaround is implemented), it fails to checkout sourcecode as Bitbucket/Git is offline. And build fails In this scenario, it should either not start new builds even though they are queued or at least be able to wait for Bitbucket to become online again so it can checkout sourcecode and perform the build.  Not fail build as we have clearly signaled to Bamboo that it should PAUSE it's actions.    On build finalize (artifact upload) the scenario/flow is this Build is running (can be running or in queue before PAUSE state is set) Bamboo goes into pause state (no matter the reason - but on our installation again due to Bitbucket backup) When build is finished with success, Bamboo starts to upload artifacts (while in paused state).  If Bamboo is paused, this fails, but the build just hangs there. So something fails, but it never "dies". Consequence is that a build agent is busy until the job is manually stopped. If the same work around is added as a final task, we now wait for Bamboo paused state to end, and then Bamboo gets to upload artifact.  In this scenario I would expect the product to know that if PAUSED then it needs to wait uploading artifacts until Bamboo is no longer paused, and it is ready to accept incoming artifact requests from build agents.    Let me know if the above still isn't clear enough.  Best regards /Anders

            So i am still a bit in doubt, will Bamboo still issue errors if it's in paused mode ? If that's not the case then the workaround given by some of you is usable until they come up with a better solution.

            Mihai Marinescu added a comment - So i am still a bit in doubt, will Bamboo still issue errors if it's in paused mode ? If that's not the case then the workaround given by some of you is usable until they come up with a better solution.

            Atlassian Update – 24 July 2018

            Hi everyone,

            Thank you for your votes and thoughts on this issue.

            We fully understand that many of you are dependent on this functionality.

            After careful consideration, we've decided to prioritise this feature on Bamboo roadmap. We hope to start development after our current projects are completed.

            Expect to hear an update on our progress within the next 6 months.

            To learn more on how your suggestions are reviewed, see our updated workflow for server feature suggestions.

            Kind regards,

            Bamboo Team

            Krystian Brazulewicz added a comment - Atlassian Update – 24 July 2018 Hi everyone, Thank you for your votes and thoughts on this issue. We fully understand that many of you are dependent on this functionality. After careful consideration, we've decided to prioritise this feature on Bamboo roadmap. We hope to start development after our current projects are completed. Expect to hear an update on our progress within the next 6 months. To learn more on how your suggestions are reviewed, see our updated workflow for server feature suggestions . Kind regards, Bamboo Team

            For those who come here in despair..  

            I think we've managed to pull of a workaround to this. 

            It's a two step procedure.

            First is to make sure BitBucket backup scripts pauses Bamboo before backup, and resumes it again once done.

            1. Send a POST request to /server/pause (https://docs.atlassian.com/bamboo/REST/5.5.0/#d2e591)
            2. Run bitbucket backup stuff
            3. Send a POST request to /server/resume

            When we do the above, a GET to /server will return a state not equal to RUNNING while running backups. Using this information, I have inserted a new task in our plan as the first step - before checkout - that just loops and waits until Bamboo is running again, meaning Bitbucket is done with its backup.

            I have still only tested it manually, but the next weeks will show if it is working as supposed in production.

            Only downside is that once you pause Bamboo, the queue will never be emptied - which I guess it the original idea of pausing it.

            But in this case, having Bamboo not failing our builds is more important than the other - at least as it seems right now.

            The task I've added is just a PowerShell script doing the following - remember to set it before checkout task . Any scripting language that can call a REST api will do. But remember it has to be in Bamboo config, not in your repos - as it should run before checkout.  

            function Get-BambooState
            {
                #Call Bamboo rest API 
                $response = Invoke-RestMethod -Uri http://myBambooServer/rest/api/latest/server -ContentType application/json
            
                #Look for status running
                $state = $response.restServerStatusInfo.state
            
                return $state
            }
            
            $state = Get-BambooState
            
            while($state -ne "RUNNING")
            {
                Write-Host "Bamboo is paused - waiting..."
                Start-Sleep -Seconds 60
                $state = Get-BambooState
            }
            
            Write-Host "Bamboo is running - moving on to checkout from BitBucket"
            

            Hope this might help someone else out there.

            For Atlassian folks - I consider this a quite ugly hack and misuse of other existing features - so this doesn't mean I don't expect a proper solution from you guys.

            Anders Kjærgaard Hansen added a comment - For those who come here in despair..   I think we've managed to pull of a workaround to this.  It's a two step procedure. First is to make sure BitBucket backup scripts pauses Bamboo before backup, and resumes it again once done. Send a POST request to /server/pause ( https://docs.atlassian.com/bamboo/REST/5.5.0/#d2e591 ) Run bitbucket backup stuff Send a POST request to /server/resume When we do the above, a GET to /server will return a state not equal to RUNNING while running backups. Using this information, I have inserted a new task in our plan as the first step - before checkout - that just loops and waits until Bamboo is running again, meaning Bitbucket is done with its backup. I have still only tested it manually, but the next weeks will show if it is working as supposed in production. Only downside is that once you pause Bamboo, the queue will never be emptied - which I guess it the original idea of pausing it. But in this case, having Bamboo not failing our builds is more important than the other - at least as it seems right now. The task I've added is just a PowerShell script doing the following - remember to set it before checkout task . Any scripting language that can call a REST api will do. But remember it has to be in Bamboo config, not in your repos - as it should run before checkout.   function Get-BambooState { #Call Bamboo rest API $response = Invoke-RestMethod -Uri http: //myBambooServer/ rest /api/latest/server -ContentType application/json #Look for status running $state = $response.restServerStatusInfo.state return $state } $state = Get-BambooState while ($state -ne "RUNNING" ) { Write-Host "Bamboo is paused - waiting..." Start-Sleep -Seconds 60 $state = Get-BambooState } Write-Host "Bamboo is running - moving on to checkout from BitBucket" Hope this might help someone else out there. For Atlassian folks - I consider this a quite ugly hack and misuse of other existing features - so this doesn't mean I don't expect a proper solution from you guys.

            I consider bamboo as abandoned and on life support. Milk the product as long as they can and then drop it of the edge of the earth.

            Its shameful that issues like this are considered minor and are not even on a roadmap.

            I will happy to admit to be wrong, but the silence of the Bamboo Team is deafening

            Erick van Rijk added a comment - I consider bamboo as abandoned and on life support. Milk the product as long as they can and then drop it of the edge of the earth. Its shameful that issues like this are considered minor and are not even on a roadmap. I will happy to admit to be wrong, but the silence of the Bamboo Team is deafening

            If they will they will suggest you use the Zero Downtime backup: https://confluence.atlassian.com/bitbucketserver/using-bitbucket-zero-downtime-backup-829920023.html

            As if backups are the only reason for Bitbucket to be in maintenance.

            A configurable retry mechanism to avoid immediately failing the build might be the way to go.

            Joris van Eijden added a comment - If they will they will suggest you use the Zero Downtime backup: https://confluence.atlassian.com/bitbucketserver/using-bitbucket-zero-downtime-backup-829920023.html As if backups are the only reason for Bitbucket to be in maintenance. A configurable retry mechanism to avoid immediately failing the build might be the way to go.

            Ping... Just came into another morning where 8 builds where dropped due to this. 

            8 builds - that should've used 5 hours each, that is 40 hours delay in our pipeline. 

            Are there any official Bamboo response to this issue - whether you investigate, consider or ignore it? 

            This is one of those small instability issues where developers get a "FAILED" mail/notification from Bamboo - which is false.

            They developer has done nothing wrong, code is perfectly fine - and issues like these tend to drop credibility when Bamboo reports errors,

            In my book, this should be a high priority issue to get fixed - and something that (seen from the outside - agreed) could be solved by a simple if-statement around the dequeue function. 

             

            if (!Bamboo.Paused)
                 dequeue();
            else
                waitSomeMore();
            

             And of course another around the check repos for changes step. 

            Can/will anyone Bamboo releated please reply to this? 

             

            Best regards

            /Anders

            Anders Kjærgaard Hansen added a comment - Ping... Just came into another morning where 8 builds where dropped due to this.  8 builds - that should've used 5 hours each, that is 40 hours delay in our pipeline.  Are there any official Bamboo response to this issue - whether you investigate, consider or ignore it?  This is one of those small instability issues where developers get a "FAILED" mail/notification from Bamboo - which is false. They developer has done nothing wrong, code is perfectly fine - and issues like these tend to drop credibility when Bamboo reports errors, In my book, this should be a high priority issue to get fixed - and something that (seen from the outside - agreed) could be solved by a simple if-statement around the dequeue function.    if (!Bamboo.Paused)      dequeue(); else     waitSomeMore();  And of course another around the check repos for changes step.  Can/will anyone Bamboo releated please reply to this?    Best regards /Anders

            Adam Pryce added a comment -

            We receive hundreds of errors a day resulting in noise that drowns true errors from maintenance periods. Nothing has actually failed, it's just that Bamboo couldn't reach Bitbucket server for a few minutes. A quiet period or a way for Bamboo to ask Bitbucket for status would be great!

            Our use case would work just fine with the proposed solution.

            Adam Pryce added a comment - We receive hundreds of errors a day resulting in noise that drowns true errors from maintenance periods. Nothing has actually failed, it's just that Bamboo couldn't reach Bitbucket server for a few minutes. A quiet period or a way for Bamboo to ask Bitbucket for status would be great! Our use case would work just fine with the proposed solution.

            For our concern this is a much needed feature.

            We have the following setup with Bitbucket backup routine.

            1. Call bamboo API - set to PAUSED
            2. Wait a few minutes
            3. Make Bitbucket backup
            4. Wait a few minutes
            5. Call Bamboo API - set to RUNNING

            I would have expected that while Bamboo is paused - no builds would be pulled from the queue, and no checking for changes would occur.

            In our world - all plans starts with a checkout task. So when Bitbucket is offline, it makes no sense to start a build.

            We have quite long running builds, due to a massive chain of integration test - replicating a real life scenario with an office installation and two remote installations - and data exchange betweens these enviroments.
            On top of that, we nightly restores two replicas of customer environments, restoring TB databases and exercising with newest Master code.

            This means that in reality, the build queue is almost never empty in Bamboo - and even more rarely all build agents are vacant.

            So I cannot wait until Bamboo is "done" - and then take backup - because Bamboo nevers gets done.

            So in summary - what would make me happy
            PAUSED means:

            • no repository checking (to avoid check errors in log)
            • no dequeuing (to avoid failure checking source code out)

            Anders Kjærgaard Hansen added a comment - For our concern this is a much needed feature. We have the following setup with Bitbucket backup routine. Call bamboo API - set to PAUSED Wait a few minutes Make Bitbucket backup Wait a few minutes Call Bamboo API - set to RUNNING I would have expected that while Bamboo is paused - no builds would be pulled from the queue, and no checking for changes would occur. In our world - all plans starts with a checkout task. So when Bitbucket is offline, it makes no sense to start a build. We have quite long running builds, due to a massive chain of integration test - replicating a real life scenario with an office installation and two remote installations - and data exchange betweens these enviroments. On top of that, we nightly restores two replicas of customer environments, restoring TB databases and exercising with newest Master code. This means that in reality, the build queue is almost never empty in Bamboo - and even more rarely all build agents are vacant. So I cannot wait until Bamboo is "done" - and then take backup - because Bamboo nevers gets done. So in summary - what would make me happy PAUSED means: no repository checking (to avoid check errors in log) no dequeuing (to avoid failure checking source code out)

            It is quite disappointing that a small yet extremely important issue like this hasn't been resolved yet. How can you have an "enterprise" scm/CI/CD tool that can't offer the ability to do a backup without breaking everything?

            Red Ventures added a comment - It is quite disappointing that a small yet extremely important issue like this hasn't been resolved yet. How can you have an "enterprise" scm/CI/CD tool that can't offer the ability to do a backup without breaking everything?

            pending and up worked, will restart mule/jboss, apps already deployed

            Rudi Lemmer added a comment - pending and up worked, will restart mule/jboss, apps already deployed

            Suggestion:
            If polling stash for changes fails, bamboo checks for maintenance mode and logs an INFO in stead of an ERROR when stash is in maintenance mode.

            Joris van Eijden added a comment - Suggestion: If polling stash for changes fails, bamboo checks for maintenance mode and logs an INFO in stead of an ERROR when stash is in maintenance mode.

            It's more than this.

            Bamboo polls Stash for changes, Bamboo (or explicit steps in the user's build plan) checks out Stash repos, and Bamboo updates Stash with build results. Even if all of these were preceded by a check of Stash's /status REST endpoint and waited if it reported it was in MAINTENANCE mode, there are still race conditions. Preventing Bamboo source code checkout if Stash is in maintenance mode dances around the real problem — Stash doesn't support zero downtime backups (STASH-4255).

            Deleted Account (Inactive) added a comment - It's more than this. Bamboo polls Stash for changes, Bamboo (or explicit steps in the user's build plan) checks out Stash repos, and Bamboo updates Stash with build results. Even if all of these were preceded by a check of Stash's /status REST endpoint and waited if it reported it was in MAINTENANCE mode, there are still race conditions. Preventing Bamboo source code checkout if Stash is in maintenance mode dances around the real problem — Stash doesn't support zero downtime backups ( STASH-4255 ).

            Horribly inconvenient to get all these errors every night
            Can Atlassian recommend a workaround?

            Joris van Eijden added a comment - Horribly inconvenient to get all these errors every night Can Atlassian recommend a workaround?

            bwstefn added a comment -

            +1 for major

            bwstefn added a comment - +1 for major

            B Ohana added a comment -

            +1 for moving from minor -> major. This is polluting our bamboo notification logs making it difficult to spot real issues.

            Is there any workaround for this problem? Is it possible to selectively purge messages around the backup window from the db?

            B Ohana added a comment - +1 for moving from minor -> major. This is polluting our bamboo notification logs making it difficult to spot real issues. Is there any workaround for this problem? Is it possible to selectively purge messages around the backup window from the db?

            i keep wondering why Atlassian doesn't run into these problem themselves? is it because they use Jenkin for the CI server??

            • please upgrade this to major +1
            • my dev lead also asked to migrate to a different CI tools +1

            Eclipse Trading added a comment - i keep wondering why Atlassian doesn't run into these problem themselves? is it because they use Jenkin for the CI server?? please upgrade this to major +1 my dev lead also asked to migrate to a different CI tools +1

            What makes it more difficult that the errors are visible for all the users. I'm getting also comments to migrate to a different CI tool.

            Erick van Rijk added a comment - What makes it more difficult that the errors are visible for all the users. I'm getting also comments to migrate to a different CI tool.

            Can this be increased from "minor prio" to "major" or higher please? We're getting lots of false positives in the logs, reducing log efficiency and some of our users are starting to complain we should use a different build tool if Bamboo is so broken.

            Deleted Account (Inactive) added a comment - Can this be increased from "minor prio" to "major" or higher please? We're getting lots of false positives in the logs, reducing log efficiency and some of our users are starting to complain we should use a different build tool if Bamboo is so broken.

            I have the same problem and would appreciate a fix, as it makes bamboo spam warnings every night.

            Christian Simon added a comment - I have the same problem and would appreciate a fix, as it makes bamboo spam warnings every night.

            Really annoying, makes the error log not very useful

            Colin Cornaby added a comment - Really annoying, makes the error log not very useful

            I have the same behavior. +1 vote to fix this

            Erick van Rijk added a comment - I have the same behavior. +1 vote to fix this

              851f15845f55 Mateusz Szmal
              rsperafico Rafael Sperafico (Inactive)
              Votes:
              131 Vote for this issue
              Watchers:
              90 Start watching this issue

                Created:
                Updated:
                Resolved: