• Icon: Suggestion Suggestion
    • Resolution: Fixed
    • 2.2
    • Builds
    • None
    • Our product teams collect and evaluate feedback from a number of different sources. To learn more about how we use customer feedback in the planning process, check out our new feature policy.

      i'd like to be able to define a threshhold on build time that makes the build count as a failure. Probably defined as a percentage of average.

      I currently have a build that locks up horribly from from time to time.

      It would be great if it could be automatically killed and counted as a failure whenver the build time gets to twice the average, or 30 minutes past the average, or even 100 minutes (regardless of the average).

            [BAM-509] Timeout builds when they take too long

            edwin added a comment -

            Hi guys,

            You'd be glad to hear that we have released Bamboo 2.2 today. In this release, we have added the capability to automatically detect builds which have taken a long time to complete. You can set up a notification on this event, and be notified by email or IM about it to take action. For more details, please check out our release notes here: http://confluence.atlassian.com/display/BAMBOO/Bamboo+2.2+Release+Notes.

            I will be closing off this issue for now. For those of you wishing for Bamboo to also automatically stop and kill the build when it detects a hang, please watch BAM-931 or vote on it.

            Cheers,
            Edwin
            Atlassian Bamboo

            edwin added a comment - Hi guys, You'd be glad to hear that we have released Bamboo 2.2 today. In this release, we have added the capability to automatically detect builds which have taken a long time to complete. You can set up a notification on this event, and be notified by email or IM about it to take action. For more details, please check out our release notes here: http://confluence.atlassian.com/display/BAMBOO/Bamboo+2.2+Release+Notes . I will be closing off this issue for now. For those of you wishing for Bamboo to also automatically stop and kill the build when it detects a hang, please watch BAM-931 or vote on it. Cheers, Edwin Atlassian Bamboo

            I just want the notification.

            Let me manually kill the build so I can debug, do thread dumps, see what the box is doing....

            Nick Menere [Atlassian] (Inactive) added a comment - I just want the notification. Let me manually kill the build so I can debug, do thread dumps, see what the box is doing....

            Marcel May added a comment -

            It would be nice to have a notification, too.

            Eg a mail warning of abnormal build time increase.

            For killing a build, what about the feature of adding a kill script run upon timeout?
            Eg for builds started by a script. It would be extremely customizable

            Marcel May added a comment - It would be nice to have a notification, too. Eg a mail warning of abnormal build time increase. For killing a build, what about the feature of adding a kill script run upon timeout? Eg for builds started by a script. It would be extremely customizable

            Mu suggestion would be for instance a default timeout of ten times the usual build time for all builds where Bamboo has sufficient data on the build time, as a fail safe.

            @Andrew – using signals would introduce a platform difference, wouldn't it? Windows doesn't have signals, and I don't think signal handlers are very standard or available in different JVMs either.

            For stack traces I guess you could create a plugin which connects to the forked java process via JMX to extract this information, just like jstack.

            Asgeir Storesund Nilsen added a comment - Mu suggestion would be for instance a default timeout of ten times the usual build time for all builds where Bamboo has sufficient data on the build time, as a fail safe. @Andrew – using signals would introduce a platform difference, wouldn't it? Windows doesn't have signals, and I don't think signal handlers are very standard or available in different JVMs either. For stack traces I guess you could create a plugin which connects to the forked java process via JMX to extract this information, just like jstack.

            If one assumes that bamboo will kill a timed out build via signals, it would be nice to be able to specify the signal(s) that is uses. This would allow one to send the -QUIT signal to java processes, which will cause them to print out backtraces for threads, before the process dies.

            We've rolled our own solution for this where our java test runner has a separate java thread that watches for timeout. When we hit the build timeout, we add an error to our junit report, with the stack backtrace of the main thread. So, we will see a unit test report for all the tests that got run up to the point where it hung, and which individual test caused it to hang.

            Because of the variety of things that could be built by bamboo, doing all these things in a way independent of what your doing is probably not realistic. But one thing that bamboo could do that we can't is to mark the build somehow as failed due to timeout instead of the normal failure.

            Andrew Bachmann added a comment - If one assumes that bamboo will kill a timed out build via signals, it would be nice to be able to specify the signal(s) that is uses. This would allow one to send the -QUIT signal to java processes, which will cause them to print out backtraces for threads, before the process dies. We've rolled our own solution for this where our java test runner has a separate java thread that watches for timeout. When we hit the build timeout, we add an error to our junit report, with the stack backtrace of the main thread. So, we will see a unit test report for all the tests that got run up to the point where it hung, and which individual test caused it to hang. Because of the variety of things that could be built by bamboo, doing all these things in a way independent of what your doing is probably not realistic. But one thing that bamboo could do that we can't is to mark the build somehow as failed due to timeout instead of the normal failure.

            Geoffrey added a comment -

            We're having this problem as well; all of the above sounds good to me.

            Geoffrey added a comment - We're having this problem as well; all of the above sounds good to me.

            And an option to retry on timeout

            Anders Sveen added a comment - And an option to retry on timeout

            +1 from me.

            Automatically killing a build might be problematic. I'd appreciate at least a notification when a build has thrashed for longer than the threshold so I can at least take action myself.

            Perhaps any build falling outside of 2 standard deviations from the mean build time should result in a warning notification.

            Christopher Owen [Atlassian] added a comment - +1 from me. Automatically killing a build might be problematic. I'd appreciate at least a notification when a build has thrashed for longer than the threshold so I can at least take action myself. Perhaps any build falling outside of 2 standard deviations from the mean build time should result in a warning notification.

            Would be great to have this feature – it sucks to have builds queue up when a stuck build keeps running.

            Nick Sieger added a comment - Would be great to have this feature – it sucks to have builds queue up when a stuck build keeps running.

            Percentage of average sounds great to me.

            Evan Leonard added a comment - Percentage of average sounds great to me.

              Unassigned Unassigned
              don.willis@atlassian.com Don Willis
              Votes:
              33 Vote for this issue
              Watchers:
              16 Start watching this issue

                Created:
                Updated:
                Resolved: