Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-14215

Branches not marked for deletion when master plan/job is - causes deletion to fail

XMLWordPrintable

      Please note that this bug doesn't have any major impact on your instance (i.e. it doesn't make your instance freeze or become unusable).

      Summary

      When you delete a plan, Bamboo actually marks it and its plan branches for deletion as opposed to performing the operation right away. A daily task is run to perform the removal of plans in that state later in the day.

      This bug is about an inconsistency that happens in the database while marking "Plans" and its "Plan Branches" for deletion. The bug is that sometimes the "Plan" is marked for deletion but it children "Plan Branches" are not.

      When the daily tasks is run, Bamboo fails to remove that plan (or perhaps this could block deletion of other plans as well as the one with the inconsistency).

      Steps to Reproduce

      The problem happens intermittently and we don't have the steps to reproduce it.

      Expected Results

      It is expected that there is no inconsistency while marking plans for deletion.

      Actual Results

      The below exception is thrown in the atlassian-bamboo.log file:

      2014-01-15 08:50:47,707 WARN [QuartzScheduler_Worker-9] [JDBCExceptionReporter] SQL Error: 0, SQLState: 23503
2014-01-15 08:50:47,708 ERROR [QuartzScheduler_Worker-9] [JDBCExceptionReporter] Batch entry 0 delete from BUILD where BUILD_ID='512561036' was aborted.  Call getNextException to see the cause.
2014-01-15 08:50:47,708 WARN [QuartzScheduler_Worker-9] [JDBCExceptionReporter] SQL Error: 0, SQLState: 23503
2014-01-15 08:50:47,708 ERROR [QuartzScheduler_Worker-9] [JDBCExceptionReporter] ERROR: update or delete on table "build" violates foreign key constraint "fk3c9ce4eb2b11c18" on table "build"
  Detail: Key (build_id)=(512561036) is still referenced from table "build".
2014-01-15 08:50:47,708 WARN [QuartzScheduler_Worker-9] [JDBCExceptionReporter] SQL Error: 0, SQLState: 23503
2014-01-15 08:50:47,708 ERROR [QuartzScheduler_Worker-9] [JDBCExceptionReporter] Batch entry 0 delete from BUILD where BUILD_ID='512561036' was aborted.  Call getNextException to see the cause.
2014-01-15 08:50:47,708 WARN [QuartzScheduler_Worker-9] [JDBCExceptionReporter] SQL Error: 0, SQLState: 23503
2014-01-15 08:50:47,708 ERROR [QuartzScheduler_Worker-9] [JDBCExceptionReporter] ERROR: update or delete on table "build" violates foreign key constraint "fk3c9ce4eb2b11c18" on table "build"
  Detail: Key (build_id)=(512561036) is still referenced from table "build".
2014-01-15 08:50:47,708 ERROR [QuartzScheduler_Worker-9] [SessionImpl] Could not synchronize database state with session
2014-01-15 08:50:47,708 ERROR [QuartzScheduler_Worker-9] [DeletionServiceImpl] Unable to complete delayed deletion:
org.springframework.dao.DataIntegrityViolationException: Hibernate operation: Could not execute JDBC batch update; SQL []; Batch entry 0 delete from BUILD where BUILD_ID='512561036' was aborted.  Call getNextException to see the cause.; nested exception is java.sql.BatchUpdateException: Batch entry 0 delete from BUILD where BUILD_ID='512561036' was aborted.  Call getNextException to see the cause.
Caused by: java.sql.BatchUpdateException: Batch entry 0 delete from BUILD where BUILD_ID='512561036' was aborted.  Call getNextException to see the cause.
        at org.postgresql.jdbc2.AbstractJdbc2Statement$BatchResultHandler.handleError(AbstractJdbc2Statement.java:2569)
        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1796)
        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:407)
        at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2708)
        at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeBatch(NewProxyPreparedStatement.java:1723)
        at net.sf.hibernate.impl.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:54)
        at net.sf.hibernate.impl.BatcherImpl.executeBatch(BatcherImpl.java:128)
        at net.sf.hibernate.impl.BatcherImpl.prepareStatement(BatcherImpl.java:61)
        at net.sf.hibernate.impl.BatcherImpl.prepareStatement(BatcherImpl.java:58)
        at net.sf.hibernate.impl.BatcherImpl.prepareBatchStatement(BatcherImpl.java:111)
        at net.sf.hibernate.persister.EntityPersister.delete(EntityPersister.java:564)
        at net.sf.hibernate.impl.ScheduledDeletion.execute(ScheduledDeletion.java:29)
        at net.sf.hibernate.impl.SessionImpl.execute(SessionImpl.java:2451)
        at net.sf.hibernate.impl.SessionImpl.executeAll(SessionImpl.java:2437)
        at net.sf.hibernate.impl.SessionImpl.execute(SessionImpl.java:2399)
        at net.sf.hibernate.impl.SessionImpl.flush(SessionImpl.java:2263)
        at net.sf.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:61)
        at org.springframework.orm.hibernate.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:510)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:662)
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:632)
        at sun.reflect.GeneratedMethodAccessor99.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:304)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
        at com.sun.proxy.$Proxy20.commit(Unknown Source)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:314)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at com.sun.proxy.$Proxy45.deletePlan(Unknown Source)
        at com.atlassian.bamboo.deletion.DeletionServiceImpl.doDeletePlan(DeletionServiceImpl.java:626)
        at com.atlassian.bamboo.deletion.DeletionServiceImpl.doDeletePlans(DeletionServiceImpl.java:612)
        at com.atlassian.bamboo.deletion.DeletionServiceImpl.executeDelayedDeletion(DeletionServiceImpl.java:454)
        at sun.reflect.GeneratedMethodAccessor1840.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:304)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
        at com.atlassian.bamboo.security.acegi.intercept.aopalliance.AuthorityOverrideMethodSecurityInterceptor.invoke(AuthorityOverrideMethodSecurityInterceptor.java:29)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at com.sun.proxy.$Proxy109.executeDelayedDeletion(Unknown Source)
        at com.atlassian.bamboo.deletion.DeletionServiceJob.execute(DeletionServiceJob.java:26)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at com.atlassian.bamboo.quartz.SystemAuthorizedThreadPool$1.run(SystemAuthorizedThreadPool.java:38)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525)

      Atlassian internal note: Seen on CBAC (5.4-m1 build 4201 - 24 Dec 13)

      Workaround

      Before going through the workaround below, please note that you must perform a valid backup of your Bamboo instance. If you are unsure about how to do so, please refer to our documentation: Exporting data for backup

      After backing up your instance, there are 2 approaches to solving this issue:

      Approach 1
      You can undelete the conflicted Plans by running a SQL query. These plans will then show up in the UI. After that you can proceed to deleting them again using the Bamboo UI. To show these plans again in the UI, run the query below:

      update build set marked_for_deletion = 'false' where build_id in (select master.build_id from build master, build child where master.marked_for_deletion = 'true' and child.marked_for_deletion = 'false' and child.master_id = master.build_id)

      Approach 2
      Alternatively of course you can mark the conflicting plan branches as deletable with the following SQL query. We don't really prefer this option of marking things deletable through SQL statements:

      update build set marked_for_deletion = 'true' where build_id in (select child.build_id from build master, build child where master.marked_for_deletion = 'true' and child.marked_for_deletion = 'false' and child.master_id = master.build_id)

      Side note: different DB servers might require different queries
      Note that the above queries are for PostgreSQL and need to be adapted to MySQL. For example, this is the latest query that should be run in a MySQL DB:

      update build 
set marked_for_deletion=true 
where build_id in (select child.build_id 
from (select * from build) as master, (select * from build) as child 
where master.marked_for_deletion=true and child.marked_for_deletion=false and child.master_id=master.build_id);

            [BAM-14215] Branches not marked for deletion when master plan/job is - causes deletion to fail

            Marcin Gardias added a comment -

            Note to self: verify that it works. Lightweight branches are going to fix this for Jobs, check that chains work as well.

            Marcin Gardias added a comment - Note to self: verify that it works. Lightweight branches are going to fix this for Jobs, check that chains work as well.

            Don Willis added a comment -

            Having a quick look at the code it seems like this could block deletion of other plans as well as the one with the inconsistency.

            Don Willis added a comment - Having a quick look at the code it seems like this could block deletion of other plans as well as the one with the inconsistency.

            Deric Lee (Inactive) added a comment -

            The above given SQL only works for postgres and does not fit into MySQL syntax. This is the equivalent query in MySQL to set branch plan deletion to true if the master plan is marked for deletion:

            update build 
set marked_for_deletion=true 
where build_id in (select child.build_id 
from (select * from build) as master, (select * from build) as child 
where master.marked_for_deletion=true and child.marked_for_deletion=false and child.master_id=master.build_id);

            Deric Lee (Inactive) added a comment - The above given SQL only works for postgres and does not fit into MySQL syntax. This is the equivalent query in MySQL to set branch plan deletion to true if the master plan is marked for deletion: update build set marked_for_deletion= true where build_id in ( select child.build_id from ( select * from build) as master, ( select * from build) as child where master.marked_for_deletion= true and child.marked_for_deletion= false and child.master_id=master.build_id);

            brad.hanks added a comment -

            Atlassian, this issue was identified as the root problem for an issues being discussed in this thread https://support.atlassian.com/servicedesk/customer/portal/10/BSP-17206 and we are out of time on this problem and need a resolution ASAP. We need a resolution on getting the database corruption problem fixed - we're betting our business on your tooling and this problem is now directly impacting on our ability to deliver to our customers. When can we expect a resolution for this issue and what are the immediate next steps?

            A secondary question: How can i raise this as a critical issue at Atlassian support?
            If you would like to correspond with me separately, email me at brad.hanks@messagesystems.com

            brad.hanks added a comment - Atlassian, this issue was identified as the root problem for an issues being discussed in this thread https://support.atlassian.com/servicedesk/customer/portal/10/BSP-17206 and we are out of time on this problem and need a resolution ASAP. We need a resolution on getting the database corruption problem fixed - we're betting our business on your tooling and this problem is now directly impacting on our ability to deliver to our customers. When can we expect a resolution for this issue and what are the immediate next steps? A secondary question: How can i raise this as a critical issue at Atlassian support? If you would like to correspond with me separately, email me at brad.hanks@messagesystems.com

            Don Willis added a comment -

            A customer also encountered this bug.

            I asked them to run the following to undelete the conflicted ones, and then to redelete them.

            update build set marked_for_deletion = 'false' where build_id in (select master.build_id from build master, build child where master.marked_for_deletion = 'true' and child.marked_for_deletion = 'false' and child.master_id = master.build_id)

            Alternatively of course you can mark the conflicting plan branches as deletable with the following. I just don't really like the idea of marking things deletable through ad-hoc sql statements.

            update build set marked_for_deletion = 'true' where build_id in (select child.build_id from build master, build child where master.marked_for_deletion = 'true' and child.marked_for_deletion = 'false' and child.master_id = master.build_id)

            There was some confusion with this particular support case that this bug might be preventing start up. It does not.

            Don Willis added a comment - A customer also encountered this bug. I asked them to run the following to undelete the conflicted ones, and then to redelete them. update build set marked_for_deletion = ' false ' where build_id in (select master.build_id from build master, build child where master.marked_for_deletion = ' true ' and child.marked_for_deletion = ' false ' and child.master_id = master.build_id) Alternatively of course you can mark the conflicting plan branches as deletable with the following. I just don't really like the idea of marking things deletable through ad-hoc sql statements. update build set marked_for_deletion = ' true ' where build_id in (select child.build_id from build master, build child where master.marked_for_deletion = ' true ' and child.marked_for_deletion = ' false ' and child.master_id = master.build_id) There was some confusion with this particular support case that this bug might be preventing start up. It does not.

            bmccoy added a comment -

            Curiously the plans affected all seemed to be related... So maybe it was something specific that did happen when those plans got deleted.

            bmccoy added a comment - Curiously the plans affected all seemed to be related... So maybe it was something specific that did happen when those plans got deleted.

            bmccoy added a comment - - edited

            I believe this was caused by branches not being marked as deleted when their parent job was. We have resolved the issue in the db and CBAC successfully completed its deletion over the weekend.

            The question is how the db got into this state.

            • I don't believe it has anything to do with the server crashes
            • There could be some concurrency issues with branch creation and being marked for deletion
              • i.e. a short period of time where branch detection can still run, after we have marked the others for deletion
            • Perhaps there was something external that caused the branch detection to continue to run
              • Branch creation relies on the plan being in the cache. Things get removed from the cache via events
              • Maybe the event queue was particularly busy one day and it took a long time for the cache event to hit.

            There is also the question as to whether we should add some redundancy into the code to cope with these kind of situations.

            bmccoy added a comment - - edited I believe this was caused by branches not being marked as deleted when their parent job was. We have resolved the issue in the db and CBAC successfully completed its deletion over the weekend. The question is how the db got into this state. I don't believe it has anything to do with the server crashes There could be some concurrency issues with branch creation and being marked for deletion i.e. a short period of time where branch detection can still run, after we have marked the others for deletion Perhaps there was something external that caused the branch detection to continue to run Branch creation relies on the plan being in the cache. Things get removed from the cache via events Maybe the event queue was particularly busy one day and it took a long time for the cache event to hit. There is also the question as to whether we should add some redundancy into the code to cope with these kind of situations.

            Martin Meinhold added a comment -

            Bumping up to critical because if I understand the code correctly, prevents the deletion of stages, top level plans and projects.

            Martin Meinhold added a comment - Bumping up to critical because if I understand the code correctly, prevents the deletion of stages, top level plans and projects.

              mgardias Marcin Gardias
              mmeinhold Martin Meinhold
              Affected customers:
              4 This affects my team
              Watchers:
              17 Start watching this issue

                Created:
                Updated:
                Resolved: