Bamboo build are getting cancelled with exception "Build xxx had to be cancelled: it was marked as queued but was not present in the queue for (at least) the past 720 seconds."

Issue Summary

This is reproducible on Data Center: (yes) 

Sometimes build are getting cancelled in Bamboo with exception "Build xxx had to be cancelled\: it was marked as queued but was not present in the queue for (at least) the past 720 seconds."

The build logs will print the message that the builds are completed, but later the exception related to build cancellation would be printed 

Steps to Reproduce

1. Build a plan in Bamboo which produces big build results and has good number of test cases to process.
2. Run the plan.

Expected Results

• The plan should complete without any errors.

Actual Results

• Below error is seen on Bamboo

  Build had to be cancelled: it was marked as queued but was not present in the queue for (at least) the past 720 seconds.
  

• Below messages are seen in the logs
  Build logs

  10:22:38 Build abc-def-xyz-5 completed.
  10:34:25 Build abc-def-xyz-5  had to be cancelled: it was marked as queued but was not present in the queue for (at least) the past 720 seconds.
  

  Agent logs

  abc-def-xyz-5 completed on Bamboo agent. Sending results to server.
  abc-def-xyz-5 -> null on Elastic Agent
  

  Processing is complete on the agent and it sends the logs to the server.

  Bamboo server logs

  10:22:38 abc-def-xyz-5 is now available on the server
  

  10:27:16 Agent 1969553452 finished building abc-def-xyz-5, but the build is still processed by the server. Detaching it from the agent
  

  Bamboo is still processing the result after ~ 5 mins it received on the server and it detaches this result from the agent

  10:34:25 [OrphanedBuildMonitorJob] abc-def-xyz-5 marked as Queued but not present in queue since Thu Nov 24 10:21:25 UTC 2022
  10:34:25 Build abc-def-xyz-5 had to be cancelled: it was marked as queued but was not present in the queue for (at least) the past 720 seconds.
  

  The orphan build monitor kills this Job while we assume this is still being proceed on the server and the error is seen

Workaround

The below workaround may not work in all the scenarios

• The first workaround is to rerun the build.

• Try to increase number of threads for processing the results with the below

  -Dbamboo.max.concurrent.expensive.messages=2
  

  This property can be added to JVM_SUPPORT_RECOMMENDED_ARGS property, please refer configuring your system properties to understand how to do these changes.