Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-9743

Pull request rescoping deadlocks on Bitbucket Server 5.0.0

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Highest
    • 5.1.0, 5.0.1
    • 5.0.0
    • None
    • None

    Description

      Notice: Although this bug report was raised against an environment involving Microsoft SQL Server, the problem itself is not specific to this environment. It is known to impact MySQL 5.7 and likely impacts other databases. Testing on PostgreSQL suggests it does not experience a deadlock, however all customers running version 5.0.0 should upgrade to 5.0.1 or later.

      Summary

      After upgrading to 5.0.0 on Windows connected to SQL Server pull request operations (decline, merge, add comment, etc) never complete.

      Looking at the thread dumps, we see several threads that are stuck trying to flush requests to the DB for example:

      "http-nio-8085-exec-10" daemon prio=5 tid=0x000000000000001c nid=0 runnable 
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1877)
	at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6642)
	- locked <0x0000000048d652ab> (a com.microsoft.sqlserver.jdbc.TDSReader)
	at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:7959)
	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:438)
	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:385)
	at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
	- locked <0x0000000015cc4a77> (a java.lang.Object)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:191)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:166)
	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(SQLServerPreparedStatement.java:328)
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
	at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:205)
	at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3009)
	at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:3503)
	at org.hibernate.action.internal.EntityInsertAction.execute(EntityInsertAction.java:89)
	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:589)
	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:463)
	at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
	at org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:50)
	at org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1396)
	at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1889)
	at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:366)
	at org.hibernate.internal.CriteriaImpl.uniqueResult(CriteriaImpl.java:388)
	at com.atlassian.stash.internal.pull.HibernatePullRequestParticipantDao.findByPullRequestAndUser(HibernatePullRequestParticipantDao.java:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
...

      and 

      "pull-request-rescoping:thread-1" daemon prio=5 tid=0x00000000000000e7 nid=0 runnable 
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1877)
	at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6642)
	- locked <0x000000000dec746f> (a com.microsoft.sqlserver.jdbc.TDSReader)
	at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:7959)
	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatementBatch(SQLServerPreparedStatement.java:2072)
	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtBatchExecCmd.doExecute(SQLServerPreparedStatement.java:1967)
	at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7505)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2444)
	- locked <0x0000000060b639fd> (a java.lang.Object)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:191)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:166)
	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeBatch(SQLServerPreparedStatement.java:1858)
	at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:125)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)
	at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.performExecution(BatchingBatch.java:111)
	at org.hibernate.engine.jdbc.batch.internal.BatchingBatch.doExecuteBatch(BatchingBatch.java:97)
	at org.hibernate.engine.jdbc.batch.internal.AbstractBatchImpl.execute(AbstractBatchImpl.java:147)
	at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.executeBatch(JdbcCoordinatorImpl.java:206)
	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:618)
	at org.hibernate.engine.spi.ActionQueue.executeActions(ActionQueue.java:463)
	at org.hibernate.event.internal.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:337)
	at org.hibernate.event.internal.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:39)
	at org.hibernate.internal.SessionImpl.doFlush(SessionImpl.java:1435)
	at org.hibernate.internal.SessionImpl.managedFlush(SessionImpl.java:491)
	at org.hibernate.internal.SessionImpl.flushBeforeTransactionCompletion(SessionImpl.java:3201)
	at org.hibernate.internal.SessionImpl.beforeTransactionCompletion(SessionImpl.java:2411)
	at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.beforeTransactionCompletion(JdbcCoordinatorImpl.java:467)
	at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.beforeCompletionCallback(JdbcResourceLocalTransactionCoordinatorImpl.java:146)
	at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.access$100(JdbcResourceLocalTransactionCoordinatorImpl.java:38)
	at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:220)
	at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:68)
	at org.springframework.orm.hibernate5.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:582)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730)
	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:150)
	at com.atlassian.stash.internal.pull.comment.drift.DriftCommentUpdateProcessor.process(DriftCommentUpdateProcessor.java:192)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
	at com.sun.proxy.$Proxy280.process(Unknown Source)
	at com.atlassian.stash.internal.pull.DefaultPullRequestService.updateComments(DefaultPullRequestService.java:1851)
	at com.atlassian.stash.internal.pull.DefaultPullRequestService.rescope(DefaultPullRequestService.java:746)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
...

      Environment

      • Windows 2012 R2
      • SQL Server
      • Upgraded from 4.14.x

      Workaround

      None. The only fix for this issue is to rollback the 5.0.0 upgrade (it is not possible to downgrade) or upgrade to 5.0.1.

      This issue only appears to affect SQL Server. Testing on PostgreSQL, for example, has shown no issues.

      Attachments

        Issue Links

          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Activity

            People

              jpalacios Juan Palacios (Inactive)
              jethomas Jeff Thomas
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: