Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-20069

Force kill build feature is enforced even when it is disabled

XMLWordPrintable

      Summary

      Whenever we stop an ongoing build running in an agent, Force kill build feature is enforced despite it being disabled in both plan level and globally. This is causing unexpected child processes (spawned by the agent) to get killed as a result of stopping the build.

      For instance, when we execute a plan that starts a JBoss process and once it is finished, when we run a new plan and stop after this is started, the force kill build feature kills the actively running jboss pid on the agent (which was started by another plan and is expected to be running still). This is reproducible most of the times, but not always.

      Steps to Reproduce

      1. Disable Force kill build feature both globally and in plan level.
      2. Run a build that has only one script task with the below line in a remote agent. 
        sleep 60
      3. Stop the ongoing build.

      Expected Results

      The build is stopped without killing the processes and child processes by Force kill build feature.

      Actual Results

      The below exception is thrown in the agent log file and the child processes are force killed too:

      2018-08-30 14:44:06,129 INFO [agentMessageListenerConnector-1] [DefaultBuildAgent] Cancel build request for 'ABC-GOOG-JOB1-4' received on agent 'jey-dell'.
2018-08-30 14:44:06,129 INFO [agentMessageListenerConnector-1] [DefaultBuildAgent] Interrupting task com.atlassian.bamboo.build.pipeline.tasks.ExecuteBuildTask@3044368c.
2018-08-30 14:44:06,129 INFO [agentMessageListenerConnector-1] [RemoteExecutionPhaseServiceImpl] ABC-GOOG-JOB1-4 execution cancelled
2018-08-30 14:44:06,129 WARN [0-BAM::jey-dell::Agent:pool-3-thread-1] [LatchedRunnable] Interrupted waiting for ExternalProcess pump to complete
...
2018-08-30 14:44:06,205 INFO [0-BAM::jey-dell::Agent:pool-3-thread-1] [RemoteResultProcessor] Build abc - google - Default Job #4 (ABC-GOOG-JOB1-4) completed on Bamboo agent. Sending results to server.
2018-08-30 14:44:06,205 INFO [agentMessageListenerConnector-1] [ManageHungBuildMessage] Captured runtimeMxBean: 24895@jey-dell
2018-08-30 14:44:06,205 INFO [agentMessageListenerConnector-1] [ManageHungBuildMessage] Captured pid is: 24895
2018-08-30 14:44:06,205 INFO [agentMessageListenerConnector-1] [ProcessManagement] Force Stop build feature is enabled for current plan. Either Bamboo has detected the build has hung or it has been manually stopped.
2018-08-30 14:44:06,206 INFO [agentMessageListenerConnector-1] [ProcessManagement] Attempting to generate stack trace and terminate spawned sub-processes of process id: 24895
2018-08-30 14:44:06,221 INFO [0-BAM::jey-dell::Agent:pool-3-thread-1] [DefaultBuildAgent] Changing context: ABC-GOOG-JOB1-4 -> null on jey-dell/6c89eea7
2018-08-30 14:44:06,221 INFO [0-BAM::jey-dell::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Agent 'jey-dell' ready to take build from queue...
2018-08-30 14:44:06,221 INFO [0-BAM::jey-dell::Agent:pool-3-thread-1] [BuildAgentControllerImpl] Agent 4259841 checking build queue for executables...
2018-08-30 14:44:06,224 INFO [agentMessageListenerConnector-1] [ProcessManagement] getRelatedProcesses PID: 24895
2018-08-30 14:44:06,225 INFO [agentMessageListenerConnector-1] [ProcessManagement] getting children pids for pid: 24895
2018-08-30 14:44:06,225 INFO [agentMessageListenerConnector-1] [ProcessManagement] returning: 0 children from pid: 24895
2018-08-30 14:44:06,225 INFO [agentMessageListenerConnector-1] [ProcessManagement] related size: 0
2018-08-30 14:44:06,226 INFO [agentMessageListenerConnector-1] [ProcessManagement] returning: 47 children from pid: 1
2018-08-30 14:44:06,226 INFO [agentMessageListenerConnector-1] [ProcessManagement] Looking for orphans processes of PID: 24895 PGID: 24892
2018-08-30 14:44:06,226 INFO [agentMessageListenerConnector-1] [ProcessManagement] Adding orphan with pid:25273 since the pgid: 24892 is the same.
2018-08-30 14:44:06,226 INFO [agentMessageListenerConnector-1] [ProcessManagement] 24895 is a descendant of 24893! Not adding it in the descendants list
2018-08-30 14:44:06,226 INFO [agentMessageListenerConnector-1] [ProcessManagement] related size: 1
2018-08-30 14:44:06,226 INFO [agentMessageListenerConnector-1] [ProcessManagement] Found related process: pid: 25273 ppid: 1 pgid: 24892 %cpu: 0.0 %mem: 0.0 cmd: sleep
2018-08-30 14:44:06,226 INFO [agentMessageListenerConnector-1] [ProcessManagement] getStackTraceAndKillRelatedProcesses for 1 processes
2018-08-30 14:44:06,227 INFO [agentMessageListenerConnector-1] [ProcessManagement] getting stack trace
2018-08-30 14:44:06,227 INFO [agentMessageListenerConnector-1] [ProcessManagement] Executing kill -3 25273
2018-08-30 14:44:11,230 INFO [agentMessageListenerConnector-1] [ProcessManagement] gently killing pids
2018-08-30 14:44:11,230 INFO [agentMessageListenerConnector-1] [ProcessManagement] Killing: 25273
2018-08-30 14:44:11,230 INFO [agentMessageListenerConnector-1] [ProcessManagement] Executing kill 25273
2018-08-30 14:44:16,248 INFO [agentMessageListenerConnector-1] [ProcessManagement] rudely killing pids (only if necessary)
2018-08-30 14:44:16,248 INFO [agentMessageListenerConnector-1] [ProcessManagement] Force Stop build feature is enabled for current plan. Either Bamboo has detected the build has hung or it has been manually stopped.
2018-08-30 14:44:16,248 INFO [agentMessageListenerConnector-1] [ProcessManagement] Has finished generating stack trace and terminating spawned sub-processes of process id: 24895
2018-08-30 14:44:16,248 INFO [agentMessageListenerConnector-1] [ManageHungBuildMessage] stopping the build (just in case it doesn't finish...)
2018-08-30 14:44:16,248 INFO [agentMessageListenerConnector-1] [DefaultBuildAgent] Cancel build request for 'ABC-GOOG-JOB1-4' received on agent 'jey-dell'.

      Workaround

      There is no workaround other than using an old version of Bamboo (before 6.4) that doesn't have this feature yet.

            [BAM-20069] Force kill build feature is enforced even when it is disabled

            Eduardo Alvarenga (Inactive) made changes -
            Remote Link Original: This issue links to "BDEV-14913 (Jira)" [ 390676 ] New: This issue links to "BDEV-14913 (Hello Jira)" [ 390676 ]
            Monique Khairuliana (Inactive) made changes -
            Workflow Original: Bamboo Workflow 2016 v1 - Restricted [ 2805808 ] New: JAC Bug Workflow v3 [ 3382965 ]
            Status Original: Resolved [ 5 ] New: Closed [ 6 ]
            Marcin Oles made changes -
            Assignee New: Marcin Oles [ moles ]
            Marcin Oles made changes -
            Fix Version/s New: 6.8.0 [ 81602 ]
            Resolution New: Fixed [ 1 ]
            Status Original: Open [ 1 ] New: Resolved [ 5 ]
            Owen made changes -
            Symptom Severity Original: Major [ 14431 ] New: Severity 2 - Major [ 15831 ]
            SET Analytics Bot made changes -
            UIS New: 1
            Pawel Skierczynski made changes -
            Remote Link New: This issue links to "BDEV-14913 (Jira)" [ 390676 ]
            Krystian Brazulewicz made changes -
            Labels Original: da-warranty hung-build New: bamboo-bugfix-kanban-candidate da-warranty hung-build
            Krystian Brazulewicz made changes -
            Occurrence Factor New: 25% [ 12831 ]
            Symptom Severity Original: Minor [ 14432 ] New: Major [ 14431 ]
            Status Original: Needs Triage [ 10030 ] New: Open [ 1 ]
            Oleksandr Tkachenko made changes -
            Labels Original: hung-build New: da-warranty hung-build

              moles Marcin Oles
              jinbasekaran Jeyanthan I (Inactive)
              Affected customers:
              2 This affects my team
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: