Issue Summary

Late build result arrival for a job that was marked incomplete due to the agent going offline and is now queued from a rebuild will cause the Build Activity dashboard to be unable to render.

Steps to Reproduce

1. Create a plan with two jobs
   • Job 1: Script Task: sleep30;
   • Job 2: Script Task: pwd;
2. Dedicate Job 1 to a remote agent 1
3. Dedicate Job 2 to another agent, doesn't matter what. Local is OK. It just needs to succeed.
4. Start the plan
5. Send SIGSTOP (kill -19) to remote agent 1 (running Job 1) wrapper and JVM processes to pause them (do this before sleep finishes)
6. Wait for the agent to be registered offline and Job 1 to go gray, will occur within 10 minutes (but depends when the last heartbeat was sent)
7. Rerun the same build
8. Send SIGCONT (kill -18) to remote agent 1 (running Job 1) JVM processes and wrapper process to resume the process and allow it to finish processing the build
9. Visit the Builds > Build Activity UI

Expected Results

Build Activity dashboard renders

Actual Results

Build activity dashboard Building and Queued sections are blank

The below exception is thrown in the atlassian-bamboo.log file:

2020-03-30 14:05:48,320 INFO [http-nio-8085-exec-12] [AccessLogFilter] bamboo GET http://localhost:8085/build/admin/ajax/getDashboardSummary.action?_=1585537548170 180357kb
....
2020-03-30 14:05:48,329 WARN [http-nio-8085-exec-12] [QueuedBuildJsonDecorator] Could not get CurrentlyBuilding for [SLEEP-SLEEP-2]
2020-03-30 14:05:48,330 ERROR [http-nio-8085-exec-12] [JsonStreamResult]
java.lang.NullPointerException: Entry removal semantics not supported (on messageType).
	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:895)
	at com.atlassian.bamboo.struts.JsonStream.put(JsonStream.java:77)
	at com.atlassian.bamboo.ww2.actions.build.AbstractBuildJsonDecorator.getBaseJson(AbstractBuildJsonDecorator.java:104)
	at com.atlassian.bamboo.ww2.actions.build.QueuedBuildJsonDecorator.getJson(QueuedBuildJsonDecorator.java:108)
	at com.atlassian.bamboo.ww2.actions.build.BuildJsonProviderAction.streamQueuedBuilds(BuildJsonProviderAction.java:180)
	at com.atlassian.bamboo.ww2.actions.build.BuildJsonProviderAction.streamJson(BuildJsonProviderAction.java:154)
	at com.atlassian.bamboo.ww2.actions.build.DashboardSummaryAction.streamJson(DashboardSummaryAction.java:12)
	at com.atlassian.bamboo.struts.JsonStreamResult.execute(JsonStreamResult.java:35)
	at com.opensymphony.xwork2.DefaultActionInvocation.executeResult(DefaultActionInvocation.java:375)
	at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:279)
	at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:181)
	at com.atlassian.bamboo.ww2.interceptors.BambooWorkflowInterceptor.validateWorkflow(BambooWorkflowInterceptor.java:36)
	at com.atlassian.bamboo.ww2.interceptors.BambooWorkflowInterceptor.doIntercept(BambooWorkflowInterceptor.java:20)
	at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
	at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)

Workaround

Option 1

Restart Bamboo and the build activity dashboard will work again.

Option 2

1. Query the build queue API to find the phantom job causing the issue: $BAMBOO_BASE_URL/rest/api/latest/queue?expand=queuedBuilds

   <queue expand="queuedBuilds">
   <link href="$BAMBOO_BASE_URL/rest/api/latest/queue" rel="self"/>
   <queuedBuilds expand="queuedBuild" start-index="0" max-result="1" size="1">
   <queuedBuild planKey="SLEEP-SLEEP-JOB1" buildNumber="2" buildResultKey="SLEEP-SLEEP-JOB1-2">
   <triggerReason>Rerun build</triggerReason>
   <link href="$BAMBOO_BASE_URL/rest/api/latest/result/SLEEP-SLEEP-JOB1-2" rel="self"/>
   </queuedBuild>
   </queuedBuilds>
   </queue>
   

2. On a busy instance with lots of builds in the queue it might be hard to identify which JOB is the one causing the issue, so combine it with the error in the logs when accessing the dashboard:

   2020-02-27 13:15:45,336 WARN [http-nio-8085-exec-846] [QueuedBuildJsonDecorator] Could not get CurrentlyBuilding for [SLEEP-SLEEP-2]
   2020-02-27 13:15:45,336 ERROR [http-nio-8085-exec-846] [JsonStreamResult] 
   java.lang.NullPointerException: Entry removal semantics not supported (on messageType).
   	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:226)
   

3. Once the job is identified, you can fix the dashboard by deleting the queued job via REST using the build key from the Queue API. Example below:

   BAMBOO_BASE_URL=http://localhost:8085
   BAMBOO_ADMIN_USER=bamboo
   BUILD_RESULT_KEY=SLEEP-SLEEP-JOB1-2
   curl --user $BAMBOO_ADMIN_USER -X DELETE "$BAMBOO_BASE_URL/rest/api/latest/queue/$BUILD_RESULT_KEY"