Deadlock during user creation

XMLWordPrintable

    • Type: Bug
    • Resolution: Fixed
    • Priority: Medium
    • 2.2.8, 2.3
    • Affects Version/s: 2.3
    • Component/s: None

      When two threads are creating a user the following sequence can occur:

      T1: get lock on user cache, create user, taking lock on user table, release lock on user cache
      T2: get lock on user cache, look up user, block due to lock belonging to T1
      T1: try to take lock on user cache before changing password, blocks due to lock belonging to T2

      See the following thread dump for details:
      "ExecuteThread: '14' for queue: 'default'" daemon prio=5 tid=0x0141eba8 nid=0x24 runnable [701fc000..702019c0]
      at java.net.SocketInputStream.socketRead0(Native Method)
      at java.net.SocketInputStream.read(SocketInputStream.java:129)
      at oracle.net.ns.Packet.receive(Unknown Source)
      at oracle.net.ns.DataPacket.receive(Unknown Source)
      at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
      at oracle.net.ns.NetInputStream.read(Unknown Source)
      at oracle.net.ns.NetInputStream.read(Unknown Source)
      at oracle.net.ns.NetInputStream.read(Unknown Source)
      at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:979)
      at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:951)
      at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:435)
      at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:182)
      at oracle.jdbc.driver.T4CPreparedStatement.execute_for_rows(T4CPreparedStatement.java:630)
      at oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:8975)

      • locked <0x7e10a0a8> (a oracle.jdbc.driver.T4CPreparedStatement)
      • locked <0x7e109e08> (a oracle.jdbc.driver.T4CConnection)
        at weblogic.jdbc.wrapper.PreparedStatement.executeBatch(PreparedStatement.java:169)
        ...
        at com.atlassian.confluence.rpc.soap.services.UsersSoapService.addUser(Unknown Source)
        at com.atlassian.confluence.rpc.soap.ConfluenceSoapServiceDelegator.addUser(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:324)
        at org.springframework.aop.framework.AopProxyUtils.invokeJoinpointUsingReflection(AopProxyUtils.java:61)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:149)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:116)
        at com.atlassian.confluence.util.profiling.SpringProfilingInterceptor.invoke(SpringProfilingInterceptor.java:18)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:138)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:56)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:138)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:152)
        at $Proxy48.addUser(Unknown Source)
        at com.atlassian.confluence.rpc.soap.ConfluenceSoapServiceImpl.addUser(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:324)
        at com.atlassian.confluence.rpc.auth.TokenAuthenticationInvocationHandler.handleAuthenticatedMethod(TokenAuthenticationInvocationHandler.java:100)
        at com.atlassian.confluence.rpc.auth.TokenAuthenticationInvocationHandler.invoke(TokenAuthenticationInvocationHandler.java:59)
        at $Proxy49.addUser(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:324)
        at org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:388)
        at org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:283)
        at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
        at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
        at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
        at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
        at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:453)
        at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
        at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:699)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
        at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
        at com.atlassian.confluence.rpc.soap.AxisSoapServer.service(AxisSoapServer.java:68)
        at sun.reflect.GeneratedMethodAccessor526.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:324)
        at org.springframework.aop.framework.AopProxyUtils.invokeJoinpointUsingReflection(AopProxyUtils.java:61)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:149)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:116)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:56)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:138)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:152)
        at $Proxy44.service(Unknown Source)
        at com.atlassian.confluence.servlet.SpringManagedServlet.service(SpringManagedServlet.java:66)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
        at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1072)
        at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465)
        at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:28)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
        at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:182)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
        at com.atlassian.seraph.filter.LoginFilter.doFilter(LoginFilter.java:177)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
        at org.springframework.orm.hibernate.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:170)
        at bucket.custom.FlushingSpringSessionInViewFilter.doFilterInternal(FlushingSpringSessionInViewFilter.java:31)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:73)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
        at com.atlassian.core.filters.gzip.GzipFilter.doFilter(GzipFilter.java:61)
        at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:27)
        at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:6987)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121)
        at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3892)
        at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2766)
        at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224)
        at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)

      "ExecuteThread: '13' for queue: 'default'" daemon prio=5 tid=0x0096bff0 nid=0x23 waiting for monitor entry [702ff000..703019c0]
      at bucket.user.persistence.dao.hibernate.BucketUserDAO.findUserByUsername(BucketUserDAO.java:63)

      • waiting to lock <0xa0e58c38> (a net.sf.ehcache.Cache)
        ...
        at com.atlassian.confluence.rpc.auth.TokenAuthenticationManager.login(TokenAuthenticationManager.java:36)

      "ExecuteThread: '12' for queue: 'default'" daemon prio=5 tid=0x0096b400 nid=0x22 waiting for monitor entry [703ff000..704019c0]
      at bucket.user.persistence.dao.hibernate.BucketUserDAO.findUserByUsername(BucketUserDAO.java:63)

      • waiting to lock <0xa0e58c38> (a net.sf.ehcache.Cache)
        ...
        at com.citi.cibtech.eae.apps.confluence.auth.CustomUserAccessor.getUser(CustomUserAccessor.java:43)
        at com.citi.cibtech.eae.apps.confluence.auth.SMAuthenticator.getUser(SMAuthenticator.java:155)

      "ExecuteThread: '11' for queue: 'default'" daemon prio=5 tid=0x0096a810 nid=0x21 waiting for monitor entry [704ff000..705019c0]
      at bucket.user.persistence.dao.hibernate.BucketUserDAO.findUserByUsername(BucketUserDAO.java:63)

      • waiting to lock <0xa0e58c38> (a net.sf.ehcache.Cache)
        ...
        at com.citi.cibtech.eae.apps.confluence.auth.SMAuthenticator.getUser(SMAuthenticator.java:155)

      "ExecuteThread: '10' for queue: 'default'" daemon prio=5 tid=0x0143b730 nid=0x20 waiting for monitor entry [705fe000..706019c0]
      at bucket.user.persistence.dao.hibernate.BucketUserDAO.findUserByUsername(BucketUserDAO.java:63)

      • waiting to lock <0xa0e58c38> (a net.sf.ehcache.Cache)
        ...
        at com.atlassian.user.impl.osuser.OSUGroupManager.getGroups(OSUGroupManager.java:239)
        ...
        at com.atlassian.confluence.security.DefaultSpacePermissionManager.hasPermissionViaGroups(DefaultSpacePermissionManager.java:204)

      "ExecuteThread: '9' for queue: 'default'" daemon prio=5 tid=0x013f5c68 nid=0x1f waiting for monitor entry [706ff000..707019c0]
      at bucket.user.persistence.dao.hibernate.BucketUserDAO.findUserByUsername(BucketUserDAO.java:63)

      • waiting to lock <0xa0e58c38> (a net.sf.ehcache.Cache)
        ...
        at com.citi.cibtech.eae.apps.confluence.auth.SMAuthenticator.getUser(SMAuthenticator.java:155)

      "ExecuteThread: '8' for queue: 'default'" daemon prio=5 tid=0x010b3410 nid=0x1e waiting for monitor entry [707ff000..708019c0]
      at bucket.user.persistence.dao.hibernate.BucketUserDAO.findUserByUsername(BucketUserDAO.java:63)

      • waiting to lock <0xa0e58c38> (a net.sf.ehcache.Cache)
        ...
        at com.atlassian.confluence.rpc.auth.TokenAuthenticationManager.login(TokenAuthenticationManager.java:36)

      "ExecuteThread: '7' for queue: 'default'" daemon prio=5 tid=0x010b3270 nid=0x1d waiting for monitor entry [708ff000..709019c0]
      at bucket.user.persistence.dao.hibernate.BucketUserDAO.findUserByUsername(BucketUserDAO.java:63)

      • waiting to lock <0xa0e58c38> (a net.sf.ehcache.Cache)
        ...
        at com.citi.cibtech.eae.apps.confluence.auth.SMAuthenticator.getUser(SMAuthenticator.java:155)

      "ExecuteThread: '6' for queue: 'default'" daemon prio=5 tid=0x00fd8bd0 nid=0x1c waiting for monitor entry [709ff000..70a019c0]
      at bucket.user.persistence.dao.hibernate.BucketUserDAO.findUserByUsername(BucketUserDAO.java:63)

      • waiting to lock <0xa0e58c38> (a net.sf.ehcache.Cache)
        ...
        at com.citi.cibtech.eae.apps.confluence.auth.SMAuthenticator.getUser(SMAuthenticator.java:155)

      "ExecuteThread: '5' for queue: 'default'" daemon prio=5 tid=0x00fd8a30 nid=0x1b waiting for monitor entry [70afc000..70b019c0]
      at bucket.user.persistence.dao.hibernate.BucketUserDAO.findUserByUsername(BucketUserDAO.java:63)

      • waiting to lock <0xa0e58c38> (a net.sf.ehcache.Cache)
        ...
        at com.atlassian.user.impl.osuser.OSUUserManager.getOpensymphonyUser(OSUUserManager.java:75)
        at com.atlassian.user.impl.osuser.OSUUserManager.saveUser(OSUUserManager.java:159)
        at com.atlassian.user.impl.osuser.OSUUserManager.alterPassword(OSUUserManager.java:104)
        ...
        at com.atlassian.confluence.rpc.soap.services.UsersSoapService.addUser(Unknown Source)

            Assignee:
            Unassigned
            Reporter:
            Tom Davies
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: