HungBuildKiller terminating job prematurely on Bamboo Server

XMLWordPrintable

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Low
    • None
    • Affects Version/s: 12.1.9
    • Component/s: Builds
    • None
    • Severity 3 - Minor

      Issue Summary

      • Long-running builds or initial builds on new branch plans are unexpectedly force-stopped by Bamboo’s Build Hanging Monitor. 
      • Build logs downloaded from the agent show continuous, normal activity right up to the moment of termination.
      • Bamboo server logs report that the build hung due to log inactivity and exceeded build time:
      Bamboo has detected that <PLAN-JOB-KEY> may have hung. Logs - last write was Xms ago (max Yms). BuildTime - currently building for Xms (expected max 0.0ms = 0 * 2.5)
       

      This issue occurs due to a combination of two factors in Bamboo's build hanging detection mechanism:

      Cause 1 : Initial builds default to an expected duration of 0

      • Bamboo calculates expected build time by multiplying the average duration of prior successful builds by a multiplier (default 2.5x).
      • For new branch plans or builds without a prior successful run, the average duration is 0 ms.
      • Consequently, the build duration check (elapsed >= expected max) evaluates to true immediately from the start of the build, disabling the duration safeguard and relying entirely on the "log quiet" threshold.
      • Because aborted/killed builds do not record a duration, reruns remain stuck with an expected duration of 0 ms until a build successfully completes.

      Cause 2 : When Live Logs are disabled, the server stops receiving log timestamps for unwatched builds

      • When live log transmission is disabled globally or not active, remote agents only stream log lines to the server while a user is actively viewing the live log tab in the UI.
      • If the build is unwatched (or ~30 seconds after navigating away), the agent suspends streaming logs over the network and writes exclusively to its local disk.
      • Because the server receives no new log packets, its internal "last log received" timer freezes.
      • Once the configured log-quiet threshold (e.g., 20 minutes) is reached, the server determines the build is inactive and terminates it, even though the agent is actively executing and logging locally.

      Steps to Reproduce

      • Create a new plan branch which should run for a long time, keep sleep 1000 in one of the script task. 
      • In the hung build killer settings for the Job, keep log quiet time as 10 mins
      • Run the plan branch

      Expected Results

      The plan fails as both the hung build killer condition is met

      1. Expected Build Time - calculated as 0 as the build never completed sucesfully
      2. Log quiet time - Is set as 10 mins

      Actual Results

      Hung build killer kills the build prematurely. 

      Workaround

      • Adjust Plan Build Monitoring: Increase the log quiet timeout or configure custom build monitoring thresholds under Plan Configuration > Miscellaneous / Build Monitoring.
      • Disable the Hung Build killer settings for the plan which is affected. 

              Assignee:
              Unassigned
              Reporter:
              Shashank Kumar
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: