Stale build results recovery process is slow

XMLWordPrintable

    • 2

      For different reasons, the Bamboo queue can get quite busy with thousands of jobs to run. This can decrease the performance significantly in a way that the only option is to restart the appplication.

      For those jobs that were queued this will cause some inconsistencies in the database. We call them "stale build results" and Bamboo checks that during the startup and runs some queries to fix it. But the application will only start after this process finishes. In the logs, we can see a message like the one below:

      INFO [localhost-startStop-1] [PendingBuildResultsStateRestorationImpl] Cleaning up xxx stale build results

      This process can take a long time, depending on your system and the number of stale build results. We have seen cases where it can take more than 7 hours. Investigating this process further, we can see Bamboo runs multiple queries, but some of them are redundant as we can see below in the SQL logs:

      2021-08-10 18:24:05.902 -03 [42] [bamboo_specs] LOG:  execute <unnamed>: update BUILDRESULTSUMMARY set LIFE_CYCLE_STATE=$1 where BUILDRESULTSUMMARY_ID=$2
2021-08-10 18:24:05.902 -03 [42] [bamboo_specs] DETAIL:  parameters: $1 = 'NotBuilt', $2 = '11239427'
2021-08-10 18:24:05.931 -03 [42] [bamboo_specs] LOG:  execute <unnamed>: update BUILDRESULTSUMMARY set BUILD_COMPLETED_DATE=$1, DELTA_STATE=$2 where BUILDRESULTSUMMARY_ID=$3
2021-08-10 18:24:05.931 -03 [42] [bamboo_specs] DETAIL:  parameters: $1 = '2021-08-10 18:24:05.917', $2 = 'NONE', $3 = '11239427'
2021-08-10 18:24:06.294 -03 [40] [bamboo_specs] LOG:  execute <unnamed>: update BUILDRESULTSUMMARY set BUILD_COMPLETED_DATE=$1 where BUILDRESULTSUMMARY_ID=$2
2021-08-10 18:24:06.294 -03 [40] [bamboo_specs] DETAIL:  parameters: $1 = '2021-08-10 18:24:06.29', $2 = '11239427'
2021-08-10 18:24:06.431 -03 [46] [bamboo_specs] LOG:  execute <unnamed>: update BUILDRESULTSUMMARY set BUILD_COMPLETED_DATE=$1 where BUILDRESULTSUMMARY_ID=$2
2021-08-10 18:24:06.431 -03 [46] [bamboo_specs] DETAIL:  parameters: $1 = '2021-08-10 18:24:06.402', $2 = '11239427'
2021-08-10 18:24:06.524 -03 [43] [bamboo_specs] LOG:  execute <unnamed>: update BUILDRESULTSUMMARY set BUILD_COMPLETED_DATE=$1 where BUILDRESULTSUMMARY_ID=$2
2021-08-10 18:24:06.524 -03 [43] [bamboo_specs] DETAIL:  parameters: $1 = '2021-08-10 18:24:06.517', $2 = '11239427'

      We have 5 update queries changing the same database row, and 4 of them are just changing the BUILD_COMPLETED_DATE repeatedly.

      Also, this process does not restart from where it was left off. For example: if Bamboo starts the recovery process with 10000 results, run half of them, and then the application is restarted. It will run the process from the beginning with 10000 results instead of 5000.

      Suggestion: Improve the recovery process in a way it doesn't run unnecessary queries and if possible a mechanism to start where it was left off.

            Assignee:
            Mateusz Szmal
            Reporter:
            Guilherme Luz (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: