Details
-
Bug
-
Resolution: Fixed
-
Medium
-
5.4.2
-
22
-
Severity 2 - Major
-
2
-
Description
Please note that this bug doesn't have any major impact on your instance (i.e. it doesn't make your instance freeze or become unusable).
Summary
When you delete a plan, Bamboo actually marks it and its plan branches for deletion as opposed to performing the operation right away. A daily task is run to perform the removal of plans in that state later in the day.
This bug is about an inconsistency that happens in the database while marking "Plans" and its "Plan Branches" for deletion. The bug is that sometimes the "Plan" is marked for deletion but it children "Plan Branches" are not.
When the daily tasks is run, Bamboo fails to remove that plan (or perhaps this could block deletion of other plans as well as the one with the inconsistency).
Steps to Reproduce
The problem happens intermittently and we don't have the steps to reproduce it.
Expected Results
It is expected that there is no inconsistency while marking plans for deletion.
Actual Results
The below exception is thrown in the atlassian-bamboo.log file:
2014-01-15 08:50:47,707 WARN [QuartzScheduler_Worker-9] [JDBCExceptionReporter] SQL Error: 0, SQLState: 23503 2014-01-15 08:50:47,708 ERROR [QuartzScheduler_Worker-9] [JDBCExceptionReporter] Batch entry 0 delete from BUILD where BUILD_ID='512561036' was aborted. Call getNextException to see the cause. 2014-01-15 08:50:47,708 WARN [QuartzScheduler_Worker-9] [JDBCExceptionReporter] SQL Error: 0, SQLState: 23503 2014-01-15 08:50:47,708 ERROR [QuartzScheduler_Worker-9] [JDBCExceptionReporter] ERROR: update or delete on table "build" violates foreign key constraint "fk3c9ce4eb2b11c18" on table "build" Detail: Key (build_id)=(512561036) is still referenced from table "build". 2014-01-15 08:50:47,708 WARN [QuartzScheduler_Worker-9] [JDBCExceptionReporter] SQL Error: 0, SQLState: 23503 2014-01-15 08:50:47,708 ERROR [QuartzScheduler_Worker-9] [JDBCExceptionReporter] Batch entry 0 delete from BUILD where BUILD_ID='512561036' was aborted. Call getNextException to see the cause. 2014-01-15 08:50:47,708 WARN [QuartzScheduler_Worker-9] [JDBCExceptionReporter] SQL Error: 0, SQLState: 23503 2014-01-15 08:50:47,708 ERROR [QuartzScheduler_Worker-9] [JDBCExceptionReporter] ERROR: update or delete on table "build" violates foreign key constraint "fk3c9ce4eb2b11c18" on table "build" Detail: Key (build_id)=(512561036) is still referenced from table "build". 2014-01-15 08:50:47,708 ERROR [QuartzScheduler_Worker-9] [SessionImpl] Could not synchronize database state with session 2014-01-15 08:50:47,708 ERROR [QuartzScheduler_Worker-9] [DeletionServiceImpl] Unable to complete delayed deletion: org.springframework.dao.DataIntegrityViolationException: Hibernate operation: Could not execute JDBC batch update; SQL []; Batch entry 0 delete from BUILD where BUILD_ID='512561036' was aborted. Call getNextException to see the cause.; nested exception is java.sql.BatchUpdateException: Batch entry 0 delete from BUILD where BUILD_ID='512561036' was aborted. Call getNextException to see the cause. Caused by: java.sql.BatchUpdateException: Batch entry 0 delete from BUILD where BUILD_ID='512561036' was aborted. Call getNextException to see the cause. at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2569) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1796) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:407) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2708) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeBatch(NewProxyPreparedStatement.java:1723) at net.sf.hibernate.impl.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:54) at net.sf.hibernate.impl.BatcherImpl.executeBatch(BatcherImpl.java:128) at net.sf.hibernate.impl.BatcherImpl.prepareStatement(BatcherImpl.java:61) at net.sf.hibernate.impl.BatcherImpl.prepareStatement(BatcherImpl.java:58) at net.sf.hibernate.impl.BatcherImpl.prepareBatchStatement(BatcherImpl.java:111) at net.sf.hibernate.persister.EntityPersister.delete(EntityPersister.java:564) at net.sf.hibernate.impl.ScheduledDeletion.execute(ScheduledDeletion.java:29) at net.sf.hibernate.impl.SessionImpl.execute(SessionImpl.java:2451) at net.sf.hibernate.impl.SessionImpl.executeAll(SessionImpl.java:2437) at net.sf.hibernate.impl.SessionImpl.execute(SessionImpl.java:2399) at net.sf.hibernate.impl.SessionImpl.flush(SessionImpl.java:2263) at net.sf.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:61) at org.springframework.orm.hibernate.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:510) at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:662) at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:632) at sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:304) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198) at com.sun.proxy.$Proxy20.commit(Unknown Source) at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:314) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy45.deletePlan(Unknown Source) at com.atlassian.bamboo.deletion.DeletionServiceImpl.doDeletePlan(DeletionServiceImpl.java:626) at com.atlassian.bamboo.deletion.DeletionServiceImpl.doDeletePlans(DeletionServiceImpl.java:612) at com.atlassian.bamboo.deletion.DeletionServiceImpl.executeDelayedDeletion(DeletionServiceImpl.java:454) at sun.reflect.GeneratedMethodAccessor1840.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:304) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149) at com.atlassian.bamboo.security.acegi.intercept.aopalliance.AuthorityOverrideMethodSecurityInterceptor.invoke(AuthorityOverrideMethodSecurityInterceptor.java:29) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy109.executeDelayedDeletion(Unknown Source) at com.atlassian.bamboo.deletion.DeletionServiceJob.execute(DeletionServiceJob.java:26) at org.quartz.core.JobRunShell.run(JobRunShell.java:202) at com.atlassian.bamboo.quartz.SystemAuthorizedThreadPool$1.run(SystemAuthorizedThreadPool.java:38) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)
Atlassian internal note: Seen on CBAC (5.4-m1 build 4201 - 24 Dec 13)
Workaround
Before going through the workaround below, please note that you must perform a valid backup of your Bamboo instance. If you are unsure about how to do so, please refer to our documentation: Exporting data for backup
After backing up your instance, there are 2 approaches to solving this issue:
Approach 1
You can undelete the conflicted Plans by running a SQL query. These plans will then show up in the UI. After that you can proceed to deleting them again using the Bamboo UI. To show these plans again in the UI, run the query below:
update build set marked_for_deletion = 'false' where build_id in (select master.build_id from build master, build child where master.marked_for_deletion = 'true' and child.marked_for_deletion = 'false' and child.master_id = master.build_id)
Approach 2
Alternatively of course you can mark the conflicting plan branches as deletable with the following SQL query. We don't really prefer this option of marking things deletable through SQL statements:
update build set marked_for_deletion = 'true' where build_id in (select child.build_id from build master, build child where master.marked_for_deletion = 'true' and child.marked_for_deletion = 'false' and child.master_id = master.build_id)
Side note: different DB servers might require different queries
Note that the above queries are for PostgreSQL and need to be adapted to MySQL. For example, this is the latest query that should be run in a MySQL DB:
update build set marked_for_deletion=true where build_id in (select child.build_id from (select * from build) as master, (select * from build) as child where master.marked_for_deletion=true and child.marked_for_deletion=false and child.master_id=master.build_id);
Attachments
Issue Links
- is duplicated by
-
BAM-20292 Race condition when a branch is created while a plan is being deleted creates database inconsistencies that break the deletion service
- Closed
- mentioned in
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
- relates to
-
BUILDENG-5146 Loading...