[#BAM-1020] Inconsistent Test Results between builds

Bamboo

Inconsistent Test Results between builds

Created: 13/Mar/07 01:51 PM Updated: 25/Apr/07 08:24 PM

Component/s:

Tests

Affects Version/s:

1.0

Fix Version/s:

1.1

Time Tracking:

Not Specified

File Attachments:	1. BusinessServicesTest.xml.bugfix (27 kB)
Environment:	Standalone

Participants:	Edwin Wong [Atlassian], Mark Chaimungkalanont [Atlassian] and Rich Dredge
Since last comment:	1 year, 24 weeks, 3 days ago
Number of comments:	3
Internal Work Bucket:	Confirmed
Labels:

Description

« Hide

For some reason, Bamboo does not recognize all the test case results from build to build. We have a total of 176 tests cases, and sometimes it recognizes all 176, and sometimes it recognizes only a subset. I cannot determine a pattern or relationship to the ones that are included in the subset, or excluded from the total results. We use TestNG, and the output result xml file should be junit compliant.

Is anyone else experiencing this? I can collect some more details if you need any.

All

Comments

Work Log

Change History

Sort Order:

[ Permlink | « Hide ]

Mark Chaimungkalanont [Atlassian] added a comment - 13/Mar/07 02:33 PM

Rich,

Can you please attach your atlassian-bamboo.log file this issue? Perhaps that will reveal more information.

If you experience a build that does not find all the test results again, would it be possible to zip up all of your TestNG XML files and attach it to this issue (I can make the issue private if the data is confidential) so we can better see what is going on?

Cheers,

Mark C

[ Permlink | « Hide ]

Rich Dredge added a comment - 30/Mar/07 09:33 AM

Mark,

I attached a version that Bamboo is reporting: 3 tests total, 1 test failed.

It reported the only failure is:
com.ihg.dec.apps.testCases.businessServices.communication.CommunicationTestCase invoking the assignDependencies method.

[ Permlink | « Hide ]

Edwin Wong [Atlassian] added a comment - 25/Apr/07 07:49 PM

Rich,

I think we have gotten to the bottom of this one.

Looking at your XML results file, we see some <skipped/> tags which were not supported by Bamboo. This also caused the parsing to stop after the first <skipped/> tag was reached.

I have updated our code so that it no longer fails when <skipped/> tags are encountered. The fix should be available in the next release.

Cheers
Edwin