-
Bug
-
Resolution: Unresolved
-
Low
-
None
-
5.10.3, 6.9.0, 6.9.1
-
6
-
Severity 3 - Minor
-
3
-
Summary
According to the NUnit Parser documentation, Bamboo uses XML reports produced by NUnit to report and track the history of the tests. According to the XML file, we have a total of 838 tests:
name="" total="838" errors="8" failures="15" not-run="16" inconclusive="0" ignored="16" skipped="0" invalid="0"
When looking at the build logs, we can also see the total count reporting 838 tests:
build 23-May-2016 19:49:26 Test Run Summary build 23-May-2016 19:49:26 Overall result: Failed build 23-May-2016 19:49:26 Test Count: 838, Passed: 799, Failed: 23, Inconclusive: 0, Skipped: 16 build 23-May-2016 19:49:26 Failed Tests - Failures: 15, Errors: 8, Invalid: 0 build 23-May-2016 19:49:26 Skipped Tests - Ignored: 16, Explicit: 0, Other: 0 build 23-May-2016 19:49:26 Start time: 2016-05-24 01:49:21Z build 23-May-2016 19:49:26 End time: 2016-05-24 01:49:26Z build 23-May-2016 19:49:26 Duration: 5.320 seconds build 23-May-2016 19:49:26 build 23-May-2016 19:49:26 Results (nunit2) saved as TestResults.xml
However, once the build finishes, Bamboo displays the following information in the Build result summary page:
total="752" failures="22" not-run="16"
The problem is that the NUnit parser counts the number of tests based on the test case name from the XML report. If there are duplicated test case names, as seen in the example below, Bamboo will not take them into account.
<test-suite type="ParameterizedTest" name="IsEncounterInpatient_Tests" executed="True" result="Success" success="True" time="0.088" asserts="0">
<results>
<test-case name="HRP.EDS.DVE.Lib.EditsV1.Tests.Unit.EditHandling.DynamicEdits.DataValidationHelperTests.Professional, P" executed="True" result="Success" success="True" time="0.066" asserts="1" />
<test-case name="HRP.EDS.DVE.Lib.EditsV1.Tests.Unit.EditHandling.DynamicEdits.DataValidationHelperTests.Institutional, I" executed="True" result="Success" success="True" time="0.000" asserts="1" />
<test-case name="HRP.EDS.DVE.Lib.EditsV1.Tests.Unit.EditHandling.DynamicEdits.DataValidationHelperTests.Professional, I" executed="True" result="Success" success="True" time="0.000" asserts="1" />
<test-case name="HRP.EDS.DVE.Lib.EditsV1.Tests.Unit.EditHandling.DynamicEdits.DataValidationHelperTests.Institutional, P" executed="True" result="Success" success="True" time="0.000" asserts="1" />
</results>
</test-suite>
<test-suite type="ParameterizedTest" name="IsEncounterOutpatient_Tests" executed="True" result="Success" success="True" time="0.004" asserts="0">
<results>
<test-case name="HRP.EDS.DVE.Lib.EditsV1.Tests.Unit.EditHandling.DynamicEdits.DataValidationHelperTests.Professional, P" executed="True" result="Success" success="True" time="0.000" asserts="1" />
<test-case name="HRP.EDS.DVE.Lib.EditsV1.Tests.Unit.EditHandling.DynamicEdits.DataValidationHelperTests.Professional, I" executed="True" result="Success" success="True" time="0.000" asserts="1" />
<test-case name="HRP.EDS.DVE.Lib.EditsV1.Tests.Unit.EditHandling.DynamicEdits.DataValidationHelperTests.Institutional, P" executed="True" result="Success" success="True" time="0.000" asserts="1" />
<test-case name="HRP.EDS.DVE.Lib.EditsV1.Tests.Unit.EditHandling.DynamicEdits.DataValidationHelperTests.Institutional, I" executed="True" result="Success" success="True" time="0.000" asserts="1" />
</results>
</test-suite>
Expected Results
Bamboo should be able to parse and display the correct results even if there are duplicated test case names.
Actual Results
Bamboo does not parse duplicated test case names. The results displayed in the Build result summary page differs from the XML report.
![[Extranet] Page](/images/icons/generic_link_16.png "[Extranet] Page"){kind=icon}
[BAM-17738] NUnit parser doesn't display correct results due to duplicated test case names
Jeff Whaling
added a comment - The company I work for (careLearning) is seeing the same behavior in the NUnit and XUnit version of testing.
Jeff Whaling
added a comment - The company I work for (careLearning) is seeing the same behavior in the NUnit and XUnit version of testing.
This bug has a massive impact on the potential quality of your customers software, the fact that this defect has been around since 07/Jul/2016 is really disappointing to see. To work around this bug my team will need to update thousands of test's - using stupid naming conventions like " login test 1 " , " login test 2 " , " login test 3 " etc, just to get around this defect. I would urge Atlassian to move on this, as the work around is too time consuming and encourages bad practice AND it leaves their customers in a bad predicament. This defect being outstanding for 3 years is quite frankly appalling.