-
Bug
-
Resolution: Unresolved
-
Low
-
None
-
5.10.3, 6.9.0, 6.9.1
-
6
-
Severity 3 - Minor
-
3
-
Summary
According to the NUnit Parser documentation, Bamboo uses XML reports produced by NUnit to report and track the history of the tests. According to the XML file, we have a total of 838 tests:
name="" total="838" errors="8" failures="15" not-run="16" inconclusive="0" ignored="16" skipped="0" invalid="0"
When looking at the build logs, we can also see the total count reporting 838 tests:
build 23-May-2016 19:49:26 Test Run Summary build 23-May-2016 19:49:26 Overall result: Failed build 23-May-2016 19:49:26 Test Count: 838, Passed: 799, Failed: 23, Inconclusive: 0, Skipped: 16 build 23-May-2016 19:49:26 Failed Tests - Failures: 15, Errors: 8, Invalid: 0 build 23-May-2016 19:49:26 Skipped Tests - Ignored: 16, Explicit: 0, Other: 0 build 23-May-2016 19:49:26 Start time: 2016-05-24 01:49:21Z build 23-May-2016 19:49:26 End time: 2016-05-24 01:49:26Z build 23-May-2016 19:49:26 Duration: 5.320 seconds build 23-May-2016 19:49:26 build 23-May-2016 19:49:26 Results (nunit2) saved as TestResults.xml
However, once the build finishes, Bamboo displays the following information in the Build result summary page:
total="752" failures="22" not-run="16"
The problem is that the NUnit parser counts the number of tests based on the test case name from the XML report. If there are duplicated test case names, as seen in the example below, Bamboo will not take them into account.
<test-suite type="ParameterizedTest" name="IsEncounterInpatient_Tests" executed="True" result="Success" success="True" time="0.088" asserts="0">
<results>
<test-case name="HRP.EDS.DVE.Lib.EditsV1.Tests.Unit.EditHandling.DynamicEdits.DataValidationHelperTests.Professional, P" executed="True" result="Success" success="True" time="0.066" asserts="1" />
<test-case name="HRP.EDS.DVE.Lib.EditsV1.Tests.Unit.EditHandling.DynamicEdits.DataValidationHelperTests.Institutional, I" executed="True" result="Success" success="True" time="0.000" asserts="1" />
<test-case name="HRP.EDS.DVE.Lib.EditsV1.Tests.Unit.EditHandling.DynamicEdits.DataValidationHelperTests.Professional, I" executed="True" result="Success" success="True" time="0.000" asserts="1" />
<test-case name="HRP.EDS.DVE.Lib.EditsV1.Tests.Unit.EditHandling.DynamicEdits.DataValidationHelperTests.Institutional, P" executed="True" result="Success" success="True" time="0.000" asserts="1" />
</results>
</test-suite>
<test-suite type="ParameterizedTest" name="IsEncounterOutpatient_Tests" executed="True" result="Success" success="True" time="0.004" asserts="0">
<results>
<test-case name="HRP.EDS.DVE.Lib.EditsV1.Tests.Unit.EditHandling.DynamicEdits.DataValidationHelperTests.Professional, P" executed="True" result="Success" success="True" time="0.000" asserts="1" />
<test-case name="HRP.EDS.DVE.Lib.EditsV1.Tests.Unit.EditHandling.DynamicEdits.DataValidationHelperTests.Professional, I" executed="True" result="Success" success="True" time="0.000" asserts="1" />
<test-case name="HRP.EDS.DVE.Lib.EditsV1.Tests.Unit.EditHandling.DynamicEdits.DataValidationHelperTests.Institutional, P" executed="True" result="Success" success="True" time="0.000" asserts="1" />
<test-case name="HRP.EDS.DVE.Lib.EditsV1.Tests.Unit.EditHandling.DynamicEdits.DataValidationHelperTests.Institutional, I" executed="True" result="Success" success="True" time="0.000" asserts="1" />
</results>
</test-suite>
Expected Results
Bamboo should be able to parse and display the correct results even if there are duplicated test case names.
Actual Results
Bamboo does not parse duplicated test case names. The results displayed in the Build result summary page differs from the XML report.