New and Improved 3.13 Beta. Highlights: Shareable filters and dashboards and lots of other goodies. Any feedback can be raised as JIRA issues in the JIRA project.
Issue Details (XML | Word | Printable)

Key: BAM-974
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Critical Critical
Assignee: Brydie McCoy [Atlassian]
Reporter: Johannes Stamminger
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
Bamboo

Bamboo penetrated perforce server with repeated requests on plan creation

Created: 06/Mar/07 08:25 AM   Updated: 09/Apr/07 11:16 PM
Component/s: Administration (Projects), Repository (Other)
Affects Version/s: 1.0
Fix Version/s: 1.0.4

Time Tracking:
Not Specified

Environment:
Java:
java version "1.5.0_10"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_10-b03)
Java HotSpot(TM) Client VM (build 1.5.0_10-b03, mixed mode, sharing)

OS:
Linux <host> 2.6.13-15.11-default #1 Mon Jul 17 09:43:01 UTC 2006 i686 i686 i386 GNU/Linux
(is some old SuSE (9 or 10?) installation)


Internal Priority: 2. High
Participants: Brydie McCoy [Atlassian], Edwin Wong [Atlassian], Johannes Stamminger and Mark Chaimungkalanont [Atlassian]
Since last comment: 1 year, 21 weeks, 2 days ago
Internal Work Bucket: Confirmed
Number of comments: 7
Labels:


 Description  « Hide
While one plan was executing I tried creation of a new one. After having pressed the "Continue" button on the perforce repository details page, the ui was blocked, no interaction any longer possible with the bamboo.

In the logfiles I detected entries like

INFO | jvm 1 | 2007/03/06 13:37:17 | /usr/local/bin/p4 -u <perforce user> -p <perforce server>:<perforce port> -c <perforce clientspec name> client -o

occuring repeatedly multiple times per second without any further error message.

Maybe it was not the best idea but I renamed the p4-executable (the morning after a stop-start I had loss of data so I tried to avoid this - but this is a different story). After some time (not immediately!) this perforce server penetration stopped with

INFO | jvm 1 | 2007/03/06 14:20:37 | com.atlassian.bamboo.repository.RepositoryException: Failed to get changes: Cannot find executable at /usr/local/bin/p4
INFO | jvm 1 | 2007/03/06 14:20:37 | Cannot find executable at /usr/local/bin/p4
INFO | jvm 1 | 2007/03/06 14:20:37 | Cannot find executable at /usr/local/bin/p4
INFO | jvm 1 | 2007/03/06 14:20:37 | Cannot find executable at /usr/local/bin/p4
INFO | jvm 1 | 2007/03/06 14:20:37 | Cannot find executable at /usr/local/bin/p4
INFO | jvm 1 | 2007/03/06 14:20:37 | Cannot find executable at /usr/local/bin/p4
INFO | jvm 1 | 2007/03/06 14:20:37 | Cannot find executable at /usr/local/bin/p4
INFO | jvm 1 | 2007/03/06 14:20:37 | Cannot find executable at /usr/local/bin/p4
INFO | jvm 1 | 2007/03/06 14:20:37 |
INFO | jvm 1 | 2007/03/06 14:20:37 | at com.atlassian.bamboo.repository.perforce.PerforceRepository.getChangesSinceLastBuild(PerforceRepository.java:290)
INFO | jvm 1 | 2007/03/06 14:20:37 | at com.atlassian.bamboo.build.strategy.PollingBuildStrategy.requiresBuild(PollingBuildStrategy.java:126)
INFO | jvm 1 | 2007/03/06 14:20:37 | at com.atlassian.bamboo.build.DefaultBuildChangeDetector.checkIfBuildNeedsBuilding(DefaultBuildChangeDetector.java:166)
INFO | jvm 1 | 2007/03/06 14:20:37 | at com.atlassian.bamboo.build.DefaultBuildChangeDetector.doTask(DefaultBuildChangeDetector.java:114)
INFO | jvm 1 | 2007/03/06 14:20:37 | at com.atlassian.bamboo.build.DefaultBuildChangeDetector.run(DefaultBuildChangeDetector.java:77)
INFO | jvm 1 | 2007/03/06 14:20:37 | at edu.emory.mathcs.backport.java.util.concurrent.helpers.ThreadHelpers$1.run(ThreadHelpers.java:34)
INFO | jvm 1 | 2007/03/06 14:20:37 | at java.lang.Thread.run(Thread.java:595)

Though bamboo itself still claims itself as running.

After stop & start all seems to be fine again (seems no data lost).

Note 1: there was currently executing another plan
Note 2: after re-doing same steps again this did not re-occur. But I noticed by way of a failed build that to my fault the perforce depot referring to in the plan config was not yet included in the perforce client spec.

(may be those aspects are related to this issue)



 All   Comments   Work Log   Change History      Sort Order: Ascending order - Click to sort in descending order
Johannes Stamminger added a comment - 07/Mar/07 03:41 AM
It just happened again (on creation of another plan).

Now without having another plan running in parallel.

But again with wrong depot specification (this time a typo).

This time I did stop bamboo and started again. The perforce server penetration re-started immediately again.
I had to rename the p4 to something else, then start the server (leading to a.m. exception and - most important - to disabled build queues. Another stop, renaming p4 to the original name again, and start later the bamboo server was operable again and I could fix the config error (and after that enable the queues again).
Another point: I had to restart my browser as otherwise I got the main page but clicking the login link did not change anything.


Johannes Stamminger added a comment - 07/Mar/07 03:43 AM
Tailing the logfile without the server being broken shows that the perforce server is penetrated then, too. Therefore something else must have happened most probably due to the wrong perforce depot spec ... ?

So IMHO there are two issues here: server breaks and perforce server is asked much too frequently (for what purpose is it asked repeatedly the same?)


Johannes Stamminger added a comment - 08/Mar/07 03:19 AM
Once again I observed now the hanging: but
  • it happened now one step later (on Builder Configuration)
  • now there was no false perforce depot config
  • the hanging was related to the browser "only", restarting the browser was necessary to reach the server again. I did not tried this approach the last hang times, but it is most probable that this would have reached then, too ... ?

Mark Chaimungkalanont [Atlassian] added a comment - 08/Mar/07 02:20 PM
Johannes,

Could you please attach your full atlassian-bamboo.log file to this issue so we can have a better look at this?

My guess was that the p4 executable was incorrectly named when they were running the checkout the first time. Currently no validation happens with the Perforce server settings itself, so if the P4 stuff isn't setup properly, it just keeps on trying to check out each time. throws and error and eventually the plan gets disabled.

I'm not sure what could be causing the hanging to be honest, but hopefully the full logs should give us something to work off

CHeers

Mark C


Brydie McCoy [Atlassian] added a comment - 30/Mar/07 01:57 AM
Hi Johannes,

Just letting you know, there have been two developments with this issue,

Firstly we have set Bamboo to cache the client information. This will remove the continuous polling of the repository (changes will be in the next release). However this will mean that if you change the client root in perforce, you will be required to restart Bamboo (you can track the progress of this issue her: http://jira.atlassian.com/browse/BAM-1082)

Secondly I have added some extra validation when you configure your plan to ensure that the depot exists before it tries to connect (change released with 1.0.3).

As for the server hanging, I can't seem to find an immediate cause for this. Is it still occuring? If it is, could you please attach the atlassian-bamboo.log.

Cheers
Brydie


Johannes Stamminger added a comment - 02/Apr/07 03:48 AM
Hi,

sorry for my late responding to the last comments, but I was on holidays.

a) Perforce client root changes do not happen that often, I can live with having to restart the bamboo then .

b) The validation is fine to me, too.

c) I am still recovering from my holidays , but as I stated already above and as far as I remember: the hanging might not be on server side or at least on server side only after adding a plan with invalid repository configuration (I am not sure about that).
But the last times I got a "hanging" this seemed to be related to the client UI (the browser) "only". Restarting the browser fixed it. And it seemed to happen very often while adding a plan.

I stopped the bamboo before my holidays (it is a test installation in parallel to our existing cruisecontrol based continuous build server only). Maybe I will start it again some days in the future and try to reproduce a "hanging" again. Until then I cannot provide those logfiles you requested, sorry.

Regards,
Johannes


Brydie McCoy [Atlassian] added a comment - 03/Apr/07 01:09 AM
Hi Johannes,

In that case we will close this issue for now. If the problems re-surface let us know.

Cheers
Brydie.