Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-3905

Build Success does not clear the overall Build Failure status

    • Icon: Bug Bug
    • Resolution: Answered
    • Icon: Low Low
    • None
    • 2.6.2
    • None
    • None

      When viewing the Build Status for a given PR overall, as well as for individual commits, a single Build Failure will result in the overall Build Status to be shown as Failure even when subsequent builds are successful.

      Note how the overall PR build status shows failure (far right), but the popup build history shows the most recent build was successful:
      (see BUG-OverallBuild.png)

      Here again, note how the given commit (the top one) shows build failure (far right), but the popup build history for it shows the most recent build was successful:
      (see BUG-CommitBuild.png)

      I am seeing this behavior on Stash v2.6.2 while using the Jenkins webhook v1.1 (https://github.com/Nerdwin15/stash-jenkins-postreceive-webhook). I filed an Issue there first (https://github.com/Nerdwin15/stash-jenkins-postreceive-webhook/issues/29), but I can't see any spot in the plugin code where this icon choice would be controlled. Thus, I'm guessing it's an overall behavior in Stash itself.

        1. BUG-CommitBuild.png
          10 kB
          Chuck Burgess
        2. BUG-OverallBuild.png
          10 kB
          Chuck Burgess

            [BSERV-3905] Build Success does not clear the overall Build Failure status

            Owen made changes -
            Workflow Original: Stash Workflow - Restricted [ 1447728 ] New: JAC Bug Workflow v3 [ 3136375 ]
            Owen made changes -
            Workflow Original: Stash Workflow [ 559270 ] New: Stash Workflow - Restricted [ 1447728 ]
            ThiagoBomfim (Inactive) made changes -
            Security Original: Reporters and Developers [ 10250 ]

            Thanks for that info, Charles... it did help me see that the key was being passed correctly from stashnotifier.

            The mistake I made in my testing may have been not running repeated builds consecutively while the toggle was OFF. I must have toggled it, ran ONE, toggled it again, ran ONE, and expected the two results to affect each other. I now realize why that's not the case.

            With the toggle OFF, I ran several builds, and I can now confirm that the one build status visible in Stash is being overwritten correctly.

            Hopefully this record of my experience will help someone later on who gets himself into this mess by virtue of his own misunderstanding of what Stash is displaying.

            Thanks again everyone.

            Chuck Burgess added a comment - Thanks for that info, Charles... it did help me see that the key was being passed correctly from stashnotifier. The mistake I made in my testing may have been not running repeated builds consecutively while the toggle was OFF. I must have toggled it, ran ONE, toggled it again, ran ONE, and expected the two results to affect each other. I now realize why that's not the case. With the toggle OFF, I ran several builds, and I can now confirm that the one build status visible in Stash is being overwritten correctly. Hopefully this record of my experience will help someone later on who gets himself into this mess by virtue of his own misunderstanding of what Stash is displaying. Thanks again everyone.

            CharlesA added a comment -

            Hi Chuck,

            The JSON looks right - as long as it uses the same key for subsequence builds it should work. It's obviously not sending the right JSON. It would be great to see exactly what it was posting in your environment. If you looked at the AO_CFE8FA_BUILD_STATUS DB table you would know for sure, but I guarantee it's still appending the build number.

            None of that necessarily helps. I'm looking at the code, and as long as "Keep repeated builds" is false the key should be correct.

            Charles

            CharlesA added a comment - Hi Chuck, The JSON looks right - as long as it uses the same key for subsequence builds it should work. It's obviously not sending the right JSON. It would be great to see exactly what it was posting in your environment. If you looked at the AO_CFE8FA_BUILD_STATUS DB table you would know for sure, but I guarantee it's still appending the build number. None of that necessarily helps. I'm looking at the code, and as long as "Keep repeated builds" is false the key should be correct. Charles

            Hey Roger,

            Georg Grütter on the stashnotifier plugin project had this to say – https://github.com/jenkinsci/stashnotifier-plugin/issues/16#issuecomment-26087081

            Could you verify that the JSON examples he shows in his comment are indeed the format expected by Stash for the two use cases we've been discussing here? Using stashnotifier 1.5 against Stash 2.6.2, I see zero difference in what the "Builds" popup shows for a given commit regardless of whether or not stashnotifier's "Keep repeated builds in Stash" option is enabled/disabled. I've rerun the same build against the same commit multiple times, with the flag on and off, and each build still adds another build item to the list of builds in that "Builds" popup.

            With that option toggled off, I should be seeing the most recent build in the "Builds" popup just being updated with a new timestamp, shouldn't I? Instead, I see a new build record, and the previous one (that should have been overwritten) still appears below it.

            Chuck Burgess added a comment - Hey Roger, Georg Grütter on the stashnotifier plugin project had this to say – https://github.com/jenkinsci/stashnotifier-plugin/issues/16#issuecomment-26087081 Could you verify that the JSON examples he shows in his comment are indeed the format expected by Stash for the two use cases we've been discussing here? Using stashnotifier 1.5 against Stash 2.6.2, I see zero difference in what the "Builds" popup shows for a given commit regardless of whether or not stashnotifier's "Keep repeated builds in Stash" option is enabled/disabled. I've rerun the same build against the same commit multiple times, with the flag on and off, and each build still adds another build item to the list of builds in that "Builds" popup. With that option toggled off, I should be seeing the most recent build in the "Builds" popup just being updated with a new timestamp, shouldn't I? Instead, I see a new build record, and the previous one (that should have been overwritten) still appears below it.

            Ah, so it's not so much a "build history" as it is a "collection of peer builds".

            So, given an example of one changesetid resulting in standalone builds from three separate build servers (dev, test, staging), and one of the three builds fails, then Stash's behavior is to show an overall build status of "failed"? As in, the only way for an overall build status of green is to have zero individual red builds?

            I guess that's the last lingering ambiguity, to me – in going through that "updating-build-status-for-commits" doc, after the "Oh no! A red build" section, let's say a fourth build occurs that is green. I don't mean the "bad build #3 just gets rerun and overwrites its red record with a green record"... I mean a fourth separate build, say an "acceptance" build. What does Stash show as the overall build status? Is it red, because the rules are "any Red equals Overall Red" and "All Green is required for Overall Green"? I guess if that doc stated those rules, I would have understood it all initially. It's just that the presentation of the detailed list of builds looks like I'd expect a "build history" to be shown. I concede that the "list of peer builds" concept fits that layout too, of course.

            if any Stash-side change comes from this, I hope it's just a clarification in that doc page, even just something like – NOTE: as mentioned earlier, this is a list of peer builds (e.g. "These could be a unit test build, an integration test build and a database matrix build"... this is not intended to show a "history of builds" as in from one build server. That could head off any "rule of least surprise" reports like mine.

            I'll continue to pursue things on the stashnotifier side. Thanks again for all the followup and explanations, Roger.

            Chuck Burgess added a comment - Ah, so it's not so much a "build history" as it is a "collection of peer builds". So, given an example of one changesetid resulting in standalone builds from three separate build servers (dev, test, staging), and one of the three builds fails, then Stash's behavior is to show an overall build status of "failed"? As in, the only way for an overall build status of green is to have zero individual red builds? I guess that's the last lingering ambiguity, to me – in going through that "updating-build-status-for-commits" doc, after the "Oh no! A red build" section, let's say a fourth build occurs that is green. I don't mean the "bad build #3 just gets rerun and overwrites its red record with a green record"... I mean a fourth separate build, say an "acceptance" build. What does Stash show as the overall build status? Is it red, because the rules are "any Red equals Overall Red" and "All Green is required for Overall Green"? I guess if that doc stated those rules, I would have understood it all initially. It's just that the presentation of the detailed list of builds looks like I'd expect a "build history" to be shown. I concede that the "list of peer builds" concept fits that layout too, of course. if any Stash-side change comes from this, I hope it's just a clarification in that doc page, even just something like – NOTE: as mentioned earlier, this is a list of peer builds (e.g. "These could be a unit test build, an integration test build and a database matrix build"... this is not intended to show a "history of builds" as in from one build server. That could head off any "rule of least surprise" reports like mine. I'll continue to pursue things on the stashnotifier side. Thanks again for all the followup and explanations, Roger.

            Hi Chuck,

            Apologies if my explanation wasn't quite clear. I'll have another go below, but this page explains it more concisely: https://developer.atlassian.com/stash/docs/latest/how-tos/updating-build-status-for-commits.html

            The intention with having a list of builds per commit in the build history popup is to capture multiple different test/build/deploy runs, rather than successive attempts at the same build. For example, you might see that your separate unit test job and functional test job is passing, but your job to deploy to a staging server is not, and this would show as 3 different build statuses no matter how many attempts were made for a given commit.

            Stash is designed this way because for a given commit, a build result should (in theory) be the same every time for a given state of the code base. Of course, things can go wrong in a build environment that aren't related to the code itself, so Stash will happily accept an updated status notifications for the same build again, showing the most recent result. With Atlassian's build system (Bamboo), we include the build number in the status description to provide a visual indication, but not in the build key. This design is also influenced by the behaviour of pull requests where restrictions can be implemented on build status.

            In terms of "build history", Stash will of course capture build results for each different commit, after all it is the changes in the code that represent the changing state of the system. The build status column shown when viewing the commit list gives an indication of that history (albeit rolled up for all build jobs).

            In conclusion, I think this means that the "Keep repeated builds in Stash" parameter is the culprit here, insofar as it isn't entirely compatible with the intention for build status handling in Stash. That said, if you think those interim build statuses (for a given commit) are particularly important in a Stash context, I'd be happy to consider an argument for surfacing them.

            Cheers,
            Roger

            Roger Barnes (Inactive) added a comment - Hi Chuck, Apologies if my explanation wasn't quite clear. I'll have another go below, but this page explains it more concisely: https://developer.atlassian.com/stash/docs/latest/how-tos/updating-build-status-for-commits.html The intention with having a list of builds per commit in the build history popup is to capture multiple different test/build/deploy runs, rather than successive attempts at the same build. For example, you might see that your separate unit test job and functional test job is passing, but your job to deploy to a staging server is not, and this would show as 3 different build statuses no matter how many attempts were made for a given commit. Stash is designed this way because for a given commit, a build result should (in theory) be the same every time for a given state of the code base. Of course, things can go wrong in a build environment that aren't related to the code itself, so Stash will happily accept an updated status notifications for the same build again, showing the most recent result. With Atlassian's build system (Bamboo), we include the build number in the status description to provide a visual indication, but not in the build key. This design is also influenced by the behaviour of pull requests where restrictions can be implemented on build status. In terms of "build history", Stash will of course capture build results for each different commit, after all it is the changes in the code that represent the changing state of the system. The build status column shown when viewing the commit list gives an indication of that history (albeit rolled up for all build jobs). In conclusion, I think this means that the "Keep repeated builds in Stash" parameter is the culprit here, insofar as it isn't entirely compatible with the intention for build status handling in Stash. That said, if you think those interim build statuses (for a given commit) are particularly important in a Stash context, I'd be happy to consider an argument for surfacing them. Cheers, Roger

            Hey Roger,

            Thanks for looking in on this. So is it Stash's position that if a given commit has multiple builds recorded against it, only the first build's result is what gets displayed forever on the commit? In other words, Stash's expectation is that there should never be a "build history", just one build, and that any in-fact subsequent builds should overwrite the "first build's" record each and every time? If that was the case, I would not expect that Stash even had the capability of displaying a list of builds on a given commit. It is this observation that leans me away from that includeBuildNumberInKey toggle and more towards which icon is chosen by Stash (based on how the db query chooses the build record?).

            Here's what I mean. If stashnotifier did not send a build number each time, then Stash would record each new build as an overwrite of the previous build. As such, Stash would not have a "build history" to show, because it would only be holding one record – the most recent build record, because it overwrote its predecessor, who itself had overwrote its predecessor, who itself ... . That appears to me to be the use case behavior driven by not including a build number.

            However, when sending a build number, Stash records each and every build record separately, thus allowing Stash to display that build history popup that you get when you click on the build icon at the right end of the commit line. Because Stash allows itself to hold multiple builds, I would expect that the "overall icon" chosen to be shown on the commit list would be based only on its last build, not its first build. Again, it is this observation that has me leaning more towards Stash on this bug, since Stash could (should) choose its icon based on the last build record rather than the first.

            On the note of trying the plugin with that setting disabled, I've actually filed a bug with them because set or not, the result is the same. It looks like stashnotifier is sending the build number whether or not that setting is set. If that toggle was working properly, I would expect that each subsequent build sent to Stash would result in Stash overwriting its current build record, and thus Stash would only ever have one build record to consider when it chose the icon to display. So, if the toggle was working, then it could act as a workaround for the Stash (mis)behavior, at the cost of Stash not keeping full build history.

            Am I misunderstanding Stash's intent here with regard to "build history"?

            Chuck Burgess added a comment - Hey Roger, Thanks for looking in on this. So is it Stash's position that if a given commit has multiple builds recorded against it, only the first build's result is what gets displayed forever on the commit? In other words, Stash's expectation is that there should never be a "build history", just one build, and that any in-fact subsequent builds should overwrite the "first build's" record each and every time? If that was the case, I would not expect that Stash even had the capability of displaying a list of builds on a given commit. It is this observation that leans me away from that includeBuildNumberInKey toggle and more towards which icon is chosen by Stash (based on how the db query chooses the build record?). Here's what I mean. If stashnotifier did not send a build number each time, then Stash would record each new build as an overwrite of the previous build. As such, Stash would not have a "build history" to show, because it would only be holding one record – the most recent build record, because it overwrote its predecessor, who itself had overwrote its predecessor, who itself ... . That appears to me to be the use case behavior driven by not including a build number. However, when sending a build number, Stash records each and every build record separately, thus allowing Stash to display that build history popup that you get when you click on the build icon at the right end of the commit line. Because Stash allows itself to hold multiple builds, I would expect that the "overall icon" chosen to be shown on the commit list would be based only on its last build, not its first build. Again, it is this observation that has me leaning more towards Stash on this bug, since Stash could (should) choose its icon based on the last build record rather than the first. On the note of trying the plugin with that setting disabled, I've actually filed a bug with them because set or not, the result is the same. It looks like stashnotifier is sending the build number whether or not that setting is set. If that toggle was working properly, I would expect that each subsequent build sent to Stash would result in Stash overwriting its current build record, and thus Stash would only ever have one build record to consider when it chose the icon to display. So, if the toggle was working, then it could act as a workaround for the Stash (mis)behavior, at the cost of Stash not keeping full build history. Am I misunderstanding Stash's intent here with regard to "build history"?
            Roger Barnes (Inactive) made changes -
            Resolution New: Answered [ 9 ]
            Status Original: Needs Triage [ 10030 ] New: Closed [ 6 ]

              Unassigned Unassigned
              d8f3b5f508a6 Chuck Burgess
              Affected customers:
              0 This affects my team
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: