Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-10773

Deadlock during update of UPDATED_ISSUES table

    XMLWordPrintable

Details

    Description

      From the PostgreSQL logs (note this almost certainly is not restricted to just PostgreSQL):

      2017-10-26 16:11:09 UTC LOG:  process 2923 detected deadlock while waiting for ShareLock on transaction 997420284 after 1000.073 ms
      2017-10-26 16:11:09 UTC STATEMENT:  UPDATE "AO_777666_UPDATED_ISSUES" SET "UPDATE_TIME" = $1 WHERE "ISSUE" = $2
      2017-10-26 16:11:09 UTC ERROR:  deadlock detected
      2017-10-26 16:11:09 UTC DETAIL:  Process 2923 waits for ShareLock on transaction 997420284; blocked by process 2762.
              Process 2762 waits for ShareLock on transaction 997420283; blocked by process 2923.
              Process 2923: UPDATE "AO_777666_UPDATED_ISSUES" SET "UPDATE_TIME" = $1 WHERE "ISSUE" = $2
              Process 2762: UPDATE "AO_777666_UPDATED_ISSUES" SET "UPDATE_TIME" = $1 WHERE "ISSUE" = $2
      2017-10-26 16:11:09 UTC HINT:  See server log for query details.
      2017-10-26 16:11:09 UTC STATEMENT:  UPDATE "AO_777666_UPDATED_ISSUES" SET "UPDATE_TIME" = $1 WHERE "ISSUE" = $2
      

      And the atlassian-bitbucket.log:

      2018-04-13 07:37:51,505 ERROR [AtlassianEvent::thread-17]  c.a.s.i.e.AsyncBatchingInvokersTransformer There was an exception thrown trying to dispatch event 'com.atlassian.bitbucket.event.pull.PullRequestRescopedEvent[source=com.atlassian.stash.internal.pull.DefaultPullRequestService@6b5b09f3]' for the invoker 'com.atlassian.diagnostics.internal.platform.event.MonitoredListenerInvoker@6f444e5e'
      java.lang.RuntimeException: Listener: com.atlassian.stash.internal.jira.index.event.IndexingEventListener event: com.atlassian.bitbucket.event.pull.PullRequestRescopedEvent
              at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:50)
              at com.atlassian.diagnostics.internal.platform.event.EventSystemMonitor.invokeMonitored(EventSystemMonitor.java:109)
              at com.atlassian.diagnostics.internal.platform.event.MonitoredListenerInvoker.invoke(MonitoredListenerInvoker.java:32)
              at com.atlassian.stash.internal.event.AsyncBatchingInvokersTransformer$AsyncInvokerBatch.invoke(AsyncBatchingInvokersTransformer.java:109)
              at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:38)
              at com.atlassian.sal.core.executor.ThreadLocalDelegateRunnable.run(ThreadLocalDelegateRunnable.java:34)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at com.atlassian.stash.internal.event.EventThreadFactory.lambda$createThread$0(EventThreadFactory.java:24)
              at java.lang.Thread.run(Thread.java:748)
              ... 1 frame trimmed
      Caused by: java.lang.reflect.UndeclaredThrowableException: null
              at com.sun.proxy.$Proxy2028.save(Unknown Source)
              at com.atlassian.stash.internal.jira.index.ao.AoUpdatedIssuesDao.lambda$update$1(AoUpdatedIssuesDao.java:72)
              at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
              at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:352)
              at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
              at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
              at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
              at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
              at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
              at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
              at com.atlassian.stash.internal.jira.index.ao.AoUpdatedIssuesDao.update(AoUpdatedIssuesDao.java:66)
              at com.atlassian.stash.internal.jira.index.DefaultJiraIndexService.lambda$updateIssueActivity$17(DefaultJiraIndexService.java:362)
              at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:21)
              at com.atlassian.stash.internal.sal.spi.HostContextAccessorImpl.doInTransaction(HostContextAccessorImpl.java:54)
              at com.atlassian.sal.core.transaction.HostContextTransactionTemplate.execute(HostContextTransactionTemplate.java:18)
              at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
              at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
              at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
              at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
              at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
              at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
              at com.atlassian.stash.internal.jira.index.DefaultJiraIndexService.updateIssueActivity(DefaultJiraIndexService.java:362)
              at com.atlassian.stash.internal.jira.index.DefaultJiraIndexService.lambda$reindex$11(DefaultJiraIndexService.java:238)
              at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:21)
              at com.atlassian.stash.internal.sal.spi.HostContextAccessorImpl.doInTransaction(HostContextAccessorImpl.java:54)
              at com.atlassian.sal.core.transaction.HostContextTransactionTemplate.execute(HostContextTransactionTemplate.java:18)
              at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
              at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
              at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
              at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
              at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
              at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
              at com.atlassian.stash.internal.jira.index.DefaultJiraIndexService.reindex(DefaultJiraIndexService.java:201)
              at com.atlassian.stash.internal.jira.index.event.IndexingEventListener.reindexPullRequest(IndexingEventListener.java:164)
              at com.atlassian.stash.internal.jira.index.event.IndexingEventListener.onPullRequestRescoped(IndexingEventListener.java:89)
              at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:40)
              ... 9 common frames omitted
      Caused by: org.postgresql.util.PSQLException: ERROR: deadlock detected
        Detail: Process 26709 waits for ShareLock on transaction 158618340; blocked by process 25123.
      Process 25123 waits for ShareLock on transaction 158618341; blocked by process 26709.
        Hint: See server log for query details.
        Where: while updating tuple (854,162) in relation "AO_777666_UPDATED_ISSUES"
              at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2477)
              at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2190)
              at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:300)
              at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428)
              at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354)
              at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:169)
              at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:136)
              at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
              at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
              at net.java.ao.EntityProxy.save(EntityProxy.java:579)
              at net.java.ao.EntityProxy.invoke(EntityProxy.java:115)
              ... 44 common frames omitted
      

      This only impacts Bitbucket Server 5.4.0 and later; earlier versions did not include JIRA issue activity journaling.

      Attachments

        Issue Links

          Activity

            People

              spetrucev Stefan
              behumphreys Ben Humphreys
              Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: