Details
-
Bug
-
Resolution: Fixed
-
Low
-
None
-
Severity 3 - Minor
-
Description
Summary
As soon as the deletion service encounters an error removing one plan, the entire service stops. The next time scheduled deletion occurs, it will often encounter the exact same error. If the problem goes undetected, this can result in a large backlog of undeleted plans which will never be removed until the root cause is resolved.
Steps to Reproduce
Simulating the problem with manual database updates:
- Create a plan with a plan branch (PLAN1)
- Mark the top level plan for deletion by updating the BUILD table, MARKED_FOR_DELETION = TRUE (manually through the database, root cause of this symptom is not yet identified)
- Don't mark the plan branches for deletion.
- Delete two other plans manually through the UI (PLAN2, PLAN3)
Expected Results
Deletion service identifies a problem performing background deletion of PLAN1 (healthcheck notification?) however continues to remove PLAN2 and PLAN3.
Actual Results
Deletion service will fail to delete PLAN1 and will stop. PLAN2 and PLAN3 will also not be deleted due to the issues with PLAN1.
One example of an exception in the atlassian-bamboo.log that has caused issues with the deletion service:
2017-08-09 06:17:29,903 INFO [scheduler_Worker-9] [DeletionServiceImpl] Deleting 20 TopLevelPlan(s) and/or ChainBranch(es) marked for deletion 2017-08-09 06:17:29,903 INFO [scheduler_Worker-9] [DeletionServiceImpl] Deleting PLAN1 2017-08-09 06:17:29,960 ERROR [scheduler_Worker-9] [SqlExceptionHelper] Batch entry 0 delete from BUILD where BUILD_ID=16779486 was aborted. Call getNextException to see the cause. 2017-08-09 06:17:29,960 ERROR [scheduler_Worker-9] [SqlExceptionHelper] ERROR: update or delete on table "build" violates foreign key constraint "fk_jdvuiqpnjmsvf3llia8jvvg03" on table "build" Detail: Key (build_id)=(16779486) is still referenced from table "build". 2017-08-09 06:17:29,960 ERROR [scheduler_Worker-9] [DeletionServiceImpl] Unable to complete delayed deletion: org.springframework.dao.DataIntegrityViolationException: could not execute batch; SQL [delete from BUILD where BUILD_ID=?]; constraint [fk_jdvuiqpnjmsvf3llia8jvvg03]; nested exception is org.hibernate.exception.ConstraintViolationException: could not execute batch
Workaround
The workaround is to correct the root cause of the first plan causing a failure in the deletion service. Raise a ticket with Atlassian Support and we'll help identify what the cause is and provide a solution