Voted! 

Some automated UI end-to-end tests are just too unpredictable for red/green results.

Selenium's IE webdriver failed to create the "@" symbol and instead wrote 2... failing the test early despite it being a driver issue. 

@Joseph You can vote for this feature if you haven't done that already.

@Kumar, navigated to this issue for the same reason. Noticing browserstack failures (with screenshots showing the webdriver failed for unpredictable reasons).. would like to have the option to mark passed. 

My Usecase: 

We are running UI tests in release pipeline. Now, UI tests can be fragile and can have failures sometimes because of several reasons. We are working on making them more and more stable however we also want to have option to make failed build pass (red -> green) so that we can test the failed cases manually and make the failed build in bamboo pass if we verify that the failed cases are false positives.

I think this feature will be very useful for UI tests. Is there any plan for implementation?

From zigarn on BAM-14725:

With plan branches, we have this problem :

• trunk has no tests
• branch has tests

If we configure Maven with 'The build will produce test results.' or JUnit Parser task, the trunk will fail, but if we disable tests parsing, branch will not display tests results.

It would be better to have 2 levels : 'The build MUST produce test results' (current behavior) and 'The build MAY produce test results' (will not fail if there is no tests but will parse them if they exist)

Quarantine only covers some of the use cases specified here. Let's keep this open, or spin off a splinter ticket for the remaining functionality.

It seems this issue is still open so, I'll comment here first. It would be nice if the threshold percentage were capable of taking fractions of a percent. 99.8 for example. Currently it only accepts integers like 99.

For test plans with thousands of cases, this doesn't offer enough granularity if you only want to accept as passing a build with 2 or 3 failed test cases.

Thanks
David

Here's another scenario, where failing test shouldn't necessarily break a build.

We're using test driven development (TDD), and therefore our task workflow is:

• Write a test case, validating the implementation the new feature or bugfix. This test should fail, until the functionality specified in the issue is implemented. The test will be part of a test first/in progress suite.
• Implement the functionality according to the test. When the feature has been implemented correctly and the associated tests will pass, the test will be upgraded to a regression test suite.

In Bamboo I then have 2 builds,

• A regression test build, which should fail if one or more of the tests in the regression test categorys.
• A complete build which is should never fail because of test faulures. This build is used as a indicator om the total test and functionality status, including tasks in progress.

At the moment I'm unable to follow the progress of incomplete functionality/bug fixes in Bamboo, because this would cause breaking builds .

Best Regards
Mikis

In addition to this, I'd like to add our use case:

We are running unit tests which include some test cases with incomplete and skipped tests ... those are being marked incomplete or skipped.

Bamboo unfortunately marks such builds as failed!

The reason for marking a test incomplete or skipped is to tell we don't have a successful case, but it doesn't have any error.

We would need Bamboo to not behave like we would have a failing build.

Could you please support / implement the usage of incomplete / skipped tests? This feature is very urgent for our software development process.

Bet regards,
Max Horváth

From forum, http://forums.atlassian.com/thread.jspa?threadID=18232&tstart=0

I have this same problem as well. My use case is that I have 2 projects to build, one dependent on the other. If the parent project compiles but unit tests fail, I still want to be able to compile the child project and run its unit tests. Currently, if unit tests in the parent project fail, the dependent build will not be started. This is a major problem.

The build failure notifications are also confusing in that its hard to determine the source of the build failure. If you have unit tests failing as well as a compile failure, the email notification only indicates that the build failed. There should be three possible build outcomes for each plan: PASS, FAIL, PASS-WITH-FAILED-UNIT-TESTS. The FAIL state indicates that there was some problem besides a unit test failure. The PASS-WITH-FAILED-UNIT-TESTS state indicates that the build succeeded, but unit tests failed. Differentiating between the different types of build failures will make it much easier to triage.
-Tom Keeney

A flip side to this would be, the child builds would fail because of the faulty parent build, as suggested in http://forums.atlassian.com/thread.jspa?messageID=257250178#257250178 we could perhaps provide the option to pass builds with failed tests on per plan basis?

Our Use Case this is the "site-deploy" maven goal which we wish to be considered successful as long as maven reports success despite the fact that tests may be failing.