Build activity page hangs where there are huge no of builds in queue status in Bamboo

XMLWordPrintable

    • 4
    • Severity 3 - Minor
    • 6

      Issue Summary

      The build activity page hangs, doesn't respond and keeps on rendering data under the Queue section when there are huge no of builds in queue status in Bamboo and after a few mins, the page pop-up to kill or wait for the page to respond.

      This is reproducible on Data Center:

      Steps to Reproduce

      1. Start a large no of builds and keep them in the queue waiting for an agent to pick them up. (you can use a schedule trigger to replicate this)
      2. Once there are 4k-6k builds waiting in the queue, you can check the queued builds count from <Bamboo-Shared-Home>/serverstate/queue folder (ls|wc -l) or using the below DB query: 
        select count(*) from buildresultsummary b where life_cycle_state='Queued' and build_type='BUILD'; 
      3. Now access the build activity page from Build > Build activity tab.

      Expected Results

      The build activity page shows all the queued builds listed under the queue section.

      Actual Results

      Build activity page hangs, it shows like the below for the queue section and doesn't respond:

      No exceptions or error messages in the Bamboo server logs.

      Workaround

      The only way to fix this issue as of now is by clearing the queued builds from the Bamboo db.
      You can use the below workaround if you don't want those builds to be in a queue state and can clean up the queued builds by following the below steps: 

      1. Stop your Bamboo instance.
      2. Take a backup for Bamboo db so that you can revert to that in case of any issues while performing the below operations in DB.
      3. Verify the build queue count by running the below query:
         select count(*) from buildresultsummary b where life_cycle_state='Queued' and build_type='BUILD'; 
      4. Perform the below SQL update query now to update the status of the builds which are in queue status to Cancelled status:
         update buildresultsummary set life_cycle_state='NotBuilt' where life_cycle_state='Queued'; 

        The update count will be double the count returned in the above step which is completely fine as we are updating the job as well as build plan status.

      5. Verify the build queue count again by running the below query:
         select count(*) from buildresultsummary b where life_cycle_state='Queued' and build_type='BUILD'; 

        It should return 0

      6. Start the Bamboo now.
      7. Try to access the build activity page now and it should fix the issue.

            Assignee:
            Sylwester Jeruzal (Inactive)
            Reporter:
            Jyothi Charupalli
            Votes:
            2 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: