is there an API to support this? Where I can script this in and hit the API to kill the hung-up builds?

If this is not going to be built into Bamboo, would it be possible to have an officially supported version/variant of the HungBuildKiller plugin? We upgraded to 6.0 recently and the HungBuildKiller isn't available again.

Guys, FWIW

take a look at this plugin: https://marketplace.atlassian.com/plugins/com.atlassian.buildeng.hung-build-killer/server/overview

I don't think this one is on Atlassian's priority list at all. I made my previous comment on this issue around 3 years ago and obvious they have no intension for a fix.

We really need to see this feature implemented, it's critical as we have hundreds of build plans.

This ticket has been around for quite some time, it would be nice to have an update on the status/intent 😐

This is a big problem for us that makes Bamboo CI not a very good fit despite all the great integration with other Atlassian tools.

Guys, please implement this feature as it would be a life saver.

Imagine build setups with hundreds of build plans - it's impossible to handle hung builds manually.

Any news on when we could expect an automatic failure and killing of an Hung job ?
It is a feature that would really be useful for us, as one of our build agent is block and can't build anything else when a Job is hung.

This feature would save us a lot of delays. We often get hung jobs in our overnight builds. Those build agents cannot take on anymore builds in the queue. People are unhappy the next morning. 2 of our overnight builds take a little more that 5-7 hours in duration. In our most recent release branch 2 of our overnight builds are now taking 4-5 hours in duration (we're improving our build duration, but are still far from quick turnaround). But still, loosing a night has a major negative effect on our development teams.

Are we able to have the current HungBuildKiller plugin source code made visible in order to fix bugs in it on our own?

Hi Adrian,

Sorry about the confusion. The build queue improvements are already available on the latest version of Bamboo. At this stage we do not had a deadline yet for providing a build timeout.

Thanks,

Sten

hi Sten, thanks for the update. Is there any chance we can get access to the last version of the source code?

Hi everyone,

Due to the current priorities on the roadmap we haven't had a chance to plan this improvement. At this stage we are unlikely to address it in the next 6 months and unfortunately we will not be able to update the plugin as well. We've done some improvements in the past to improve our queuing system but we will have to wait more before we can provide builds timeouts.

Thank you for your patience.

Sten Pittet
Bamboo Product Manager

hi Claudio, please don't ask me. I don't work in Atlassian anymore. As a workaround, I guess you can pay a bit more and spin your own Bamboo server on EC2.

Hey Adrian Deccico, the plugin would be absolutely great, if it was installed in Bamboo Cloud. For Cloud users, it's impossible to solve this issue without either built-in support or making the plugin preinstalled in some future.

completely agree completely

Junit tests hang if a test setup throws an exception. While this is obviously not a Bamboo issue, it would be great if Bamboo could fail the build for taking to long.

This would be great, some builds get hung up and run forever until we can manually cancel them. Seems to happen a lot on OSX as well.

This timeout should be configurable for each build plan, since certain builds may have a longer timeouts than others.

Well they apparently removed that from Maven so we're stuck with nothing. 60+ votes and an extremely useful and obvious feature... why is this still not in the product in 2014??

hi Sebastien and J. Matthew, there seems to be a recent release of the HBK plugin that seems to be working against the last Bamboo release. Please check: https://answers.atlassian.com/questions/288726/when-will-hung-build-killer-pluggin-be-supported-in-bamboo-5-4-x

I agree - there is already a hung build event, and there is already a stop build action; the next logical step would be an option we can set to automatically execute the latter upon firing of the former?

This is unassigned with 62 votes. I pay for the product is that enough to get this ticket moving forward?

When it hang I'm wasting my time, so I'm looking for a continuous that is more automatic and is designed with the fact that things can go wrong.

@Adrian Thanks for the info! It really helps.

Adding to the previous comment:

The plugin tries sending the kill command anyway, regardless of the type of agent but due to architecture differences it will not work for local agents as it does for remote ones.

Hi kooper@ncsa.illinois.edu and iamliurui , the plugin only works for remote agents, not local ones. It is mentioned in the description: https://marketplace.atlassian.com/plugins/com.atlassian.buildeng.hung-build-killer

I have the same issue with Rob.

The HBK detects the hung event, but the build is not killed. Only a comment is added.

I am using HungBuildKiller plugin but it does not seem to work with my SBT builds. The build is marked as hung and the plugin kicks in but does not kill the build. All I see in the logs for bamboo is the following:

2013-07-02 10:10:14,219 WARN [QuartzScheduler_Worker-2] [BuildHangingMonitor] Bamboo has detected that DFDL-MASTER1-JOB1-17 may have hung.
2013-07-02 10:10:14,219 WARN [QuartzScheduler_Worker-2] [BuildHangingMonitor] Time Since Last Log: 2371630, Max log time: 1200000
2013-07-02 10:10:14,219 WARN [QuartzScheduler_Worker-2] [BuildHangingMonitor] Expected Build Time: 2589457.5, Current Build Time: 2613299
2013-07-02 10:10:14,225 INFO [5-BranchDetectionBackgroundThread:pool-9-thread-3] [BranchDetectionServiceImpl] Detecting branches for plan DFDL-MASTER
2013-07-02 10:10:14,248 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKiller] Handling event com.atlassian.bamboo.event.BuildHungEvent[source=com.atlassian.bamboo.build.monitoring.BuildHangingMonitor@4b4d2962] job=DFDL-MASTER1-JOB1
2013-07-02 10:10:14,248 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKiller] getting the build definition from key: DFDL-MASTER1-JOB1
2013-07-02 10:10:14,309 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKiller] Custom configuration doesn't contain:custom.com.atlassian.bamboo.plugin.hungbuildkiller.hung.enabled
2013-07-02 10:10:14,319 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKillerUtil] captured value: true
2013-07-02 10:10:14,319 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKillerUtil] captured value: true
2013-07-02 10:10:14,319 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKiller] Returning default: true
2013-07-02 10:10:14,319 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKillerUtil] getting bandana key: custom.com.atlassian.bamboo.plugin.hungbuildkiller.hung.enabled.global
2013-07-02 10:10:14,321 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKillerUtil] bandana value: null
2013-07-02 10:10:14,321 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKillerUtil] captured value: null
2013-07-02 10:10:14,321 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKiller] hung build detected
2013-07-02 10:10:14,321 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKiller] Custom configuration doesn't contain:custom.com.atlassian.bamboo.plugin.hungbuildkiller.add_comment_when_hung.enabled
2013-07-02 10:10:14,321 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKillerUtil] captured value: true
2013-07-02 10:10:14,321 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKillerUtil] captured value: true
2013-07-02 10:10:14,321 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKiller] Returning default: true
2013-07-02 10:10:14,321 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKillerUtil] getting bandana key: custom.com.atlassian.bamboo.plugin.hungbuildkiller.add_comment_when_hung.enabled
2013-07-02 10:10:14,323 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKillerUtil] bandana value: null
2013-07-02 10:10:14,323 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKillerUtil] captured value: null
2013-07-02 10:10:14,332 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKiller] adding hung build comment
2013-07-02 10:10:14,533 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [DashboardCacheListener] event: com.atlassian.bamboo.event.BuildCommentedEvent[source=com.atlassian.buildeng.HungBuildKiller@64ba977a] job=DFDL-MASTER1-JOB1
2013-07-02 10:10:14,549 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKiller] remote agent id is: 131073
2013-07-02 10:10:14,549 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKiller] agentCommandSender: com.atlassian.bamboo.v2.build.agent.AgentCommandSenderImpl@2d005e48
2013-07-02 10:10:14,551 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-3] [HungBuildKiller] sending command to local agent

I just ran into this issue. Being on Bamboo OnDemand means I cannot install the mentioned plugin.

The build, executed on an EC2 Large instance, ran for 1 week! (I was unable to check earlier)

That's money down the drain.

Please fix this, seriously.

hi br.bill et al. Just to announce that new version of HBK plugin now supports Windows It can be download here: https://plugins.atlassian.com/plugin/details/848939

hi br.bill, Hung builds are a serious problem for us as well but as I mentioned, we managed to fix it with this plugin.

I am adding a document that better explains how and when the plugin works. Please note that so far the plugin only support Linux/Unix/MAC. Windows support won't be difficult to add though.

cheers

Adrian, thanks for your reply. If I can get your plugin accepted here I will try a pilot of it.

I don't know if it will help enough – it depends on if, when it "kills the build", it will work when the build has already died but the Bamboo web server doesn't know/report that the build has stopped.

I add my vote to add this capability. The inability to kill a hung build in general is a huge problem.

Hi br.bill please take a look at my plugin. It's working perfectly for us in three large Bamboo instances.

The second related huge problem is seeing that the build is not running but Bamboo still thinks it is, even if the agent is dead on the build machine.

I saw this issue in the past but I believe it's already fixed. Anyway, it's unrelated with this issue.

I add my vote to add this capability. The inability to kill a hung build in general is a huge problem. The second related huge problem is seeing that the build is not running but Bamboo still thinks it is, even if the agent is dead on the build machine.

"Popular" feature requests are important, but failure to work at all is more important.

hi amazkovoi, we just published a plugin that listens for the build hung event and the canceled event. After its execution you will get a stack trace and the related processes will be killed not leaving any orphan that could fail the next build. For more information please go here: https://plugins.atlassian.com/plugin/details/848939?versionId=848940

Hi Dom,

At the moment this is not on our short term road map, as we are implementing features that seem to be more popular.

It is possible to develop this as a plugin. One would need to write a listener and listen for the BuildHungEvent:
http://confluence.atlassian.com/display/BAMBOO/Bamboo+Event+Listener+Module
Then stop the build.

Cheers,
Anton

Like everyone else that commented on this bug, we're performing nightly builds and having a "build stops on hung" would be extremely usefull.

Any updates on that basic feature?

Thanks,

Dom

This would tremendously useful to us as well. A hung nightly build sends out notifications, but they aren't seen til the next day at the earliest. A setting to kill hung builds would be a valuable tool, on top of the current detect-and-notify system.

Also would like this feature implemented. We run a series of tests every night at midnight. A notification about one of those builds hanging in the middle of the night is not practical for us. No-one would see it until the next morning, at which time there'll be 4/5 major test builds in the queue since the previous night. It'd be far more useful if Bamboo could kill the build after a pre-defined period of inactivity.

We are using the notification feature but kill and fail would be far better for our team.

We also have multi threaded tests that lock up at least monthly and depending on email filters or volume of other notifications, we might not find out about a timeout / locked build for 20h. If it failed after 6h (or 3x average time), we could fix the problem faster.

Now that the detection of builds which take too long have been done in Bamboo 2.2. (see BAM-509), the next step is to be able to automatically kill off the issue when the timeout has been detected.