Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-9743

Pull request rescoping deadlocks on Bitbucket Server 5.0.0

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Highest Highest
    • 5.1.0, 5.0.1
    • 5.0.0
    • None
    • None

      Notice: Although this bug report was raised against an environment involving Microsoft SQL Server, the problem itself is not specific to this environment. It is known to impact MySQL 5.7 and likely impacts other databases. Testing on PostgreSQL suggests it does not experience a deadlock, however all customers running version 5.0.0 should upgrade to 5.0.1 or later.

      Summary

      After upgrading to 5.0.0 on Windows connected to SQL Server pull request operations (decline, merge, add comment, etc) never complete.

      Looking at the thread dumps, we see several threads that are stuck trying to flush requests to the DB for example:

      "http-nio-8085-exec-10" daemon prio=5 tid=0x000000000000001c nid=0 runnable 
         java.lang.Thread.State: RUNNABLE
      	at java.net.SocketInputStream.socketRead0(Native Method)
      	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      	at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1877)
      	at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6642)
      	- locked <0x0000000048d652ab> (a com.microsoft.sqlserver.jdbc.TDSReader)
      	at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:7959)
      	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:438)
      	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:385)
      	at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505)
      	at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
      	- locked <0x0000000015cc4a77> (a java.lang.Object)
      	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:191)
      	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:166)
      	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(SQLServerPreparedStatement.java:328)
      	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
      	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
      	at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:205)
      	at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3009)
      	at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3503)
      	at org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89)
      	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:589)
      	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:463)
      	at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
      	at org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:50)
      	at org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1396)
      	at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1889)
      	at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:366)
      	at org.hibernate.internal.CriteriaImpl.uniqueResult(CriteriaImpl.java:388)
      	at com.atlassian.stash.internal.pull.HibernatePullRequestParticipantDao.findByPullRequestAndUser(HibernatePullRequestParticipantDao.java:55)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
      	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
      ...

      and

      "pull-request-rescoping:thread-1" daemon prio=5 tid=0x00000000000000e7 nid=0 runnable 
         java.lang.Thread.State: RUNNABLE
      	at java.net.SocketInputStream.socketRead0(Native Method)
      	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      	at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1877)
      	at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6642)
      	- locked <0x000000000dec746f> (a com.microsoft.sqlserver.jdbc.TDSReader)
      	at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:7959)
      	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatementBatch(SQLServerPreparedStatement.java:2072)
      	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtBatchExecCmd.doExecute(SQLServerPreparedStatement.java:1967)
      	at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505)
      	at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
      	- locked <0x0000000060b639fd> (a java.lang.Object)
      	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:191)
      	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:166)
      	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeBatch(SQLServerPreparedStatement.java:1858)
      	at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:125)
      	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
      	at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:111)
      	at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.doExecuteBatch(BatchingBatch.java:97)
      	at org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl.execute(AbstractBatchImpl.java:147)
      	at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.executeBatch(JdbcCoordinatorImpl.java:206)
      	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:618)
      	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:463)
      	at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
      	at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
      	at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1435)
      	at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:491)
      	at org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3201)
      	at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2411)
      	at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:467)
      	at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.beforeCompletionCallback(JdbcResourceLocalTransactionCoordinatorImpl.java:146)
      	at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.access$100(JdbcResourceLocalTransactionCoordinatorImpl.java:38)
      	at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:220)
      	at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:68)
      	at org.springframework.orm.hibernate5.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:582)
      	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761)
      	at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730)
      	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:150)
      	at com.atlassian.stash.internal.pull.comment.drift.DriftCommentUpdateProcessor.process(DriftCommentUpdateProcessor.java:192)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
      	at com.sun.proxy.$Proxy280.process(Unknown Source)
      	at com.atlassian.stash.internal.pull.DefaultPullRequestService.updateComments(DefaultPullRequestService.java:1851)
      	at com.atlassian.stash.internal.pull.DefaultPullRequestService.rescope(DefaultPullRequestService.java:746)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      ...

      Environment

      • Windows 2012 R2
      • SQL Server
      • Upgraded from 4.14.x

      Workaround

      None. The only fix for this issue is to rollback the 5.0.0 upgrade (it is not possible to downgrade) or upgrade to 5.0.1.

      This issue only appears to affect SQL Server. Testing on PostgreSQL, for example, has shown no issues.

            [BSERV-9743] Pull request rescoping deadlocks on Bitbucket Server 5.0.0

            Bitbucket Server 5.0.1 has now been released and is available for download: https://www.atlassian.com/software/bitbucket/download

            Ben Humphreys added a comment - Bitbucket Server 5.0.1 has now been released and is available for download: https://www.atlassian.com/software/bitbucket/download

            For anyone affected by this, we're in the process of releasing 5.0.1. Please watch this issue to be notified when the release is complete and has been published for downloading.

            Bryan Turner (Inactive) added a comment - For anyone affected by this, we're in the process of releasing 5.0.1. Please watch this issue to be notified when the release is complete and has been published for downloading.

              jpalacios Juan Palacios (Inactive)
              jethomas Jeff Thomas
              Affected customers:
              1 This affects my team
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: