Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-18305

Investigate possible race condition when starting scheduled builds after scheduled backup

XMLWordPrintable

      Summary

      It looks like some scheduled builds will randomly be droped with the message:

      2017-03-22 01:02:02,668 INFO [10-BAM::PlanExec:pool-17-thread-3] [ExecutionLimitsServiceImpl] : PLAN-KEY cannot be run because the system is PAUSING

      Steps to Reproduce

      1. Configure multiple scheduled builds to run during the time a scheduled backup is running
      2. Wait for the builds to happen after the backup is finished

      Expected Results

      Bamboo will run the scheduled builds after the backup is finished

      Actual Results

      Bamboo will sometimes try running buids when the server is still PAUSED and it will just avoid running those

      Some log examples of this problem:

      2017-03-21 01:06:39,027 INFO [scheduler_Worker-10] [XmlMigrator] Export completed. 01:06:39.021
2017-03-21 01:06:39,041 INFO [scheduler_Worker-6] [QuartzTriggerJob] Running quartz strategy job for CHAIN:EDC-FTTHC
2017-03-21 01:06:39,041 INFO [scheduler_Worker-8] [QuartzTriggerJob] Running quartz strategy job for CHAIN:TLR-FAT
2017-03-21 01:06:39,041 INFO [scheduler_Worker-10] [ServerLifecycleManagerImpl] Server state changed to 'RUNNING' from 'PAUSING'
2017-03-21 01:06:39,042 INFO [scheduler_Worker-3] [QuartzTriggerJob] Running quartz strategy job for CHAIN:EDC-UFTD
2017-03-21 01:06:39,042 INFO [scheduler_Worker-2] [QuartzTriggerJob] Running quartz strategy job for CHAIN:EDC-PIT
2017-03-21 01:06:39,044 INFO [10-BAM::PlanExec:pool-17-thread-3] [ScheduledBuildDetectionAction] Detecting changes for TLR-FAT...
2017-03-21 01:06:39,045 INFO [10-BAM::PlanExec:pool-17-thread-1] [ScheduledBuildDetectionAction] Detecting changes for EDC-UFTD...
2017-03-21 01:06:39,046 INFO [10-BAM::PlanExec:pool-17-thread-4] [ScheduledBuildDetectionAction] Detecting changes for EDC-FTTHC...
2017-03-21 01:06:39,046 INFO [10-BAM::PlanExec:pool-17-thread-2] [ScheduledBuildDetectionAction] Detecting changes for EDC-PIT...
2017-03-21 01:06:39,051 INFO [10-BAM::PlanExec:pool-17-thread-2] [ScheduledBuildDetectionAction] : Schedule build EDC-PIT-1690 triggered
2017-03-21 01:06:39,052 INFO [10-BAM::PlanExec:pool-17-thread-1] [ScheduledBuildDetectionAction] : Schedule build EDC-UFTD-475 triggered
2017-03-21 01:06:39,053 INFO [10-BAM::PlanExec:pool-17-thread-3] [ScheduledBuildDetectionAction] : Schedule build TLR-FAT-402 triggered
2017-03-21 01:06:39,066 INFO [10-BAM::PlanExec:pool-17-thread-4] [ScheduledBuildDetectionAction] : Schedule build EDC-FTTHC-413 triggered
--
--
2017-03-22 01:02:02,650 INFO [scheduler_Worker-7] [XmlMigrator] Export completed. 01:02:02.644
2017-03-22 01:02:02,668 INFO [scheduler_Worker-1] [QuartzTriggerJob] Running quartz strategy job for CHAIN:TLR-FAT
2017-03-22 01:02:02,668 INFO [scheduler_Worker-9] [QuartzTriggerJob] Running quartz strategy job for CHAIN:EDC-PIT
2017-03-22 01:02:02,668 INFO [scheduler_Worker-5] [QuartzTriggerJob] Running quartz strategy job for CHAIN:EDC-UFTD
2017-03-22 01:02:02,668 INFO [scheduler_Worker-3] [QuartzTriggerJob] Running quartz strategy job for CHAIN:EDC-FTTHC
2017-03-22 01:02:02,668 INFO [10-BAM::PlanExec:pool-17-thread-3] [ExecutionLimitsServiceImpl] : TLR-FAT cannot be run because the system is PAUSING
2017-03-22 01:02:02,669 INFO [10-BAM::PlanExec:pool-17-thread-3] [ExecutionLimitsServiceImpl] : EDC-PIT cannot be run because the system is PAUSING
2017-03-22 01:02:02,669 INFO [10-BAM::PlanExec:pool-17-thread-3] [ExecutionLimitsServiceImpl] : EDC-UFTD cannot be run because the system is PAUSING
2017-03-22 01:02:02,669 INFO [10-BAM::PlanExec:pool-17-thread-3] [ExecutionLimitsServiceImpl] : EDC-FTTHC cannot be run because the system is PAUSING
2017-03-22 01:02:02,669 INFO [10-BAM::PlanExec:pool-17-thread-4] [ExecutionLimitsServiceImpl] : ZZZ-BD48 cannot be run because the system is PAUSING
2017-03-22 01:02:02,670 INFO [10-BAM::PlanExec:pool-17-thread-2] [ExecutionLimitsServiceImpl] : TLR-TCA1 cannot be run because the system is PAUSING
2017-03-22 01:02:02,671 INFO [scheduler_Worker-7] [ServerLifecycleManagerImpl] Server state changed to 'RUNNING' from 'PAUSING'
--
--
2017-03-23 01:02:10,415 INFO [scheduler_Worker-6] [XmlMigrator] Export completed. 01:02:10.411
2017-03-23 01:02:10,427 INFO [scheduler_Worker-5] [QuartzTriggerJob] Running quartz strategy job for CHAIN:EDC-FTTHC
2017-03-23 01:02:10,428 INFO [scheduler_Worker-4] [QuartzTriggerJob] Running quartz strategy job for CHAIN:EDC-UFTD
2017-03-23 01:02:10,428 INFO [scheduler_Worker-8] [QuartzTriggerJob] Running quartz strategy job for CHAIN:EDC-PIT
2017-03-23 01:02:10,428 INFO [scheduler_Worker-10] [QuartzTriggerJob] Running quartz strategy job for CHAIN:EDC-UPT
2017-03-23 01:02:10,428 INFO [10-BAM::PlanExec:pool-17-thread-4] [ExecutionLimitsServiceImpl] : ZZZ-BD48 cannot be run because the system is PAUSING
2017-03-23 01:02:10,428 INFO [scheduler_Worker-9] [QuartzTriggerJob] Running quartz strategy job for CHAIN:TLR-FAT
2017-03-23 01:02:10,430 INFO [scheduler_Worker-6] [ServerLifecycleManagerImpl] Server state changed to 'RUNNING' from 'PAUSING'
2017-03-23 01:02:10,432 INFO [10-BAM::PlanExec:pool-17-thread-4] [ScheduledBuildDetectionAction] Detecting changes for EDC-FTTHC...
2017-03-23 01:02:10,434 INFO [10-BAM::PlanExec:pool-17-thread-1] [ScheduledBuildDetectionAction] Detecting changes for EDC-PIT...
2017-03-23 01:02:10,434 INFO [10-BAM::PlanExec:pool-17-thread-3] [ScheduledBuildDetectionAction] Detecting changes for EDC-UPT...
2017-03-23 01:02:10,437 INFO [10-BAM::PlanExec:pool-17-thread-2] [ScheduledBuildDetectionAction] Detecting changes for EDC-UFTD...
2017-03-23 01:02:10,438 INFO [10-BAM::PlanExec:pool-17-thread-4] [ScheduledBuildDetectionAction] : Schedule build EDC-FTTHC-434 triggered
2017-03-23 01:02:10,439 INFO [10-BAM::PlanExec:pool-17-thread-3] [ScheduledBuildDetectionAction] : Schedule build EDC-UPT-2988 triggered
2017-03-23 01:02:10,442 INFO [10-BAM::PlanExec:pool-17-thread-1] [ScheduledBuildDetectionAction] : Schedule build EDC-PIT-1693 triggered
2017-03-23 01:02:10,445 INFO [10-BAM::PlanExec:pool-17-thread-2] [ScheduledBuildDetectionAction] : Schedule build EDC-UFTD-476 triggered

      In this case, the plan TLR-FAT was configured to run at 00:05 AM. Backup starts at 00:00 AM so usually the plan is built after 1 AM when the export is finished, but sometimes it is dropped and not run.

      Workaround

      Option 1 - Configure an external backup tool and avoid using the scheduled backup
      Option 2 - Do not configure builds to run when the server is running the scheduled backup

            [BAM-18305] Investigate possible race condition when starting scheduled builds after scheduled backup

            SET Analytics Bot made changes -
            UIS Original: 1 New: 2
            Zaro made changes -
            Status Original: Needs Triage [ 10030 ] New: Gathering Impact [ 12072 ]
            Zaro made changes -
            Priority Original: Highest [ 1 ] New: Low [ 4 ]
            SET Analytics Bot made changes -
            UIS Original: 2 New: 1
            SET Analytics Bot made changes -
            UIS Original: 23 New: 2
            Bugfix Automation Bot made changes -
            Status Original: Gathering Impact [ 12072 ] New: Needs Triage [ 10030 ]
            Bugfix Automation Bot made changes -
            Priority Original: Low [ 4 ] New: Highest [ 1 ]
            SET Analytics Bot made changes -
            UIS Original: 2 New: 23
            SET Analytics Bot made changes -
            Support reference count Original: 2 New: 3
            SET Analytics Bot made changes -
            UIS Original: 21 New: 2

              Unassigned Unassigned
              dsantos Daniel Santos (Inactive)
              Affected customers:
              6 This affects my team
              Watchers:
              8 Start watching this issue

                Created:
                Updated: