Lock-up on BEAC

BEAC was completely locked this morning; thread-dump (kill -3) and SIGTERM had no effect. The last entries in the log were this:

@4000000046adb29228fd62a4 2007-07-30 04:42:32,685 INFO [BAM:Trunk:BuildExe] [SaveBuildTask] Indexing build results...
@4000000046adb292342962cc 2007-07-30 04:42:32,874 INFO [BAM::PipelineSelector] [PipelineSelector] Not found queue for build results BuildResults: CONFSTABFUNC-T5MY-0
@4000000046adb29331185624 2007-07-30 04:42:33,821 INFO [BAM::BuildChangeDetector] [DefaultBuildChangeDetector] Detected that build "Confluence Stable - Tomcat, MySQL" needs building.
@4000000046adb293311861dc 2007-07-30 04:42:33,822 INFO [BAM::PipelineSelector] [PipelineSelector] Placing build results BuildResults: CONFSTABFUNC-T5MY-0
@4000000046adb2940455eaac 2007-07-30 04:42:34,072 FATAL [BAM:Trunk:BuildExe] [SaveBuildTask] Could not save the build results BuildResults: CONFFUNC-CWD-87
@4000000046adb2940455f27c java.io.IOException: Lock obtain timed out: Lock@/opt/j2ee/domains/bamboo.atlassian.com/confluence/tomcat/temp/lucene-bcef63d283c10275c19b5c900805c7fc-write.lock
@4000000046adb2940455fa4c at org.apache.lucene.store.Lock.obtain(Lock.java:58)
@4000000046adb2940455fa4c at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:255)
@4000000046adb2940456486c at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:225)
@4000000046adb29404564c54 at com.atlassian.bamboo.index.IndexConnectionManager.getWriter(IndexConnectionManager.java:114)
@4000000046adb2940456503c at com.atlassian.bamboo.index.IndexConnectionManager.index(IndexConnectionManager.java:85)
@4000000046adb2940456503c at com.atlassian.bamboo.index.DefaultBuildResultsIndexer.indexBuildResultKeepConnection(DefaultBuildResultsIndexer.java:167)
@4000000046adb29404565bf4 at com.atlassian.bamboo.index.DefaultBuildResultsIndexer.indexBuildResult(DefaultBuildResultsIndexer.java:111)
@4000000046adb29404565fdc at com.atlassian.bamboo.build.pipeline.tasks.SaveBuildTask.call(SaveBuildTask.java:74)
@4000000046adb29404565fdc at edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176)
@4000000046adb29404567b34 at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:987)
@4000000046adb29404567b34 at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:528)
@4000000046adb29404567f1c at com.atlassian.bamboo.build.pipeline.concurrent.NamedThreadFactory$1.run(NamedThreadFactory.java:31)
@4000000046adb29404568ad4 at edu.emory.mathcs.backport.java.util.concurrent.helpers.ThreadHelpers$1.run(ThreadHelpers.java:34)
@4000000046adb29404568ad4 at java.lang.Thread.run(Thread.java:619)
@4000000046adb29404b3a2dc 2007-07-30 04:42:34,078 INFO [BAM:Trunk:BuildExe] [DefaultErrorHandler] Recording error: Could not save the build results : Confluence Trunk - Crowd Build
@4000000046adb29404b3aaac java.io.IOException: Lock obtain timed out: Lock@/opt/j2ee/domains/bamboo.atlassian.com/confluence/tomcat/temp/lucene-bcef63d283c10275c19b5c900805c7fc-write.lock
@4000000046adb29404b3ae94 at org.apache.lucene.store.Lock.obtain(Lock.java:58)
@4000000046adb29404b3b27c at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:255)
@4000000046adb29404b3d1bc at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:225)
@4000000046adb29404b3d1bc at com.atlassian.bamboo.index.IndexConnectionManager.getWriter(IndexConnectionManager.java:114)
@4000000046adb29404b3d5a4 at com.atlassian.bamboo.index.IndexConnectionManager.index(IndexConnectionManager.java:85)
@4000000046adb29404b3d98c at com.atlassian.bamboo.index.DefaultBuildResultsIndexer.indexBuildResultKeepConnection(DefaultBuildResultsIndexer.java:167)
@4000000046adb29404b3e15c at com.atlassian.bamboo.index.DefaultBuildResultsIndexer.indexBuildResult(DefaultBuildResultsIndexer.java:111)
@4000000046adb29404b3e544 at com.atlassian.bamboo.build.pipeline.tasks.SaveBuildTask.call(SaveBuildTask.java:74)
@4000000046adb29404b3fcb4 at edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:176)
@4000000046adb29404b3fcb4 at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:987)
@4000000046adb29404b4009c at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:528)
@4000000046adb29404b40484 at com.atlassian.bamboo.build.pipeline.concurrent.NamedThreadFactory$1.run(NamedThreadFactory.java:31)
@4000000046adb29404b446ec at edu.emory.mathcs.backport.java.util.concurrent.helpers.ThreadHelpers$1.run(ThreadHelpers.java:34)
@4000000046adb29404b44ad4 at java.lang.Thread.run(Thread.java:619)
@4000000046adb29404b44ad4 2007-07-30 04:42:34,078 INFO [BAM:Trunk:BuildExe] [SaveBuildTask] Finished building CONFFUNC-CWD-87.
@4000000046adb29405fe81fc 2007-07-30 04:42:34,100 INFO [BAM::PipelineSelector] [PipelineSelector] Not found queue for build results BuildResults: CONFSTABFUNC-T5MY-0
@4000000046adb2be2f4fafcc 2007-07-30 04:43:16,793 INFO [BAM:Clusters:BuildExe] [AbstractBuilder] Completed building build "Confluence Trunk - Cluster Tomcat, Postgres, non-random" return code = 0
@4000000046adb2c50fc358d4 2007-07-30 04:43:23,264 INFO [BAM:Clusters:BuildExe] [ExecuteBuildTask] Running plugin 'Artifact Copier'
@4000000046adb2c51561a7dc 2007-07-30 04:43:23,358 INFO [BAM:Clusters:BuildExe] [ExecuteBuildTask] Running plugin 'Clover Results Collector'
@4000000046adb2c51564e014 2007-07-30 04:43:23,358 INFO [BAM:Clusters:BuildExe] [ExecuteBuildTask] Build log size: 78454
@4000000046adb2c5156c2f2c 2007-07-30 04:43:23,359 INFO [BAM:Clusters:BuildExe] [ExecuteBuildTask] Setting build artifacts