Confluence
  1. Confluence
  2. CONF-12201

Race condition issue with DefaultHibernateUser

    Details

    • Last commented by user?:
      true
    • Internal Complexity:
      4
    • Internal Value:
      4

      Description

      While debugging an issue in our custom authenticator that creates and updates user accounts during logins, I found something that appears to be a bug in how DefaultHibernateUser objects are being handled (and cached?) in confluence.

      It looks like the DefaultHibernateUser objects are being cached with its (groups) collections attached with the hibernate session that performed an operation on the object recently (in a different thread). When another thread with a new hibernate session attempts to update such a user instance the exception below is thrown. We are not passing the objects between threads via http session or in any other way. Each thread obtains an instance of the user class via userAccessor.

      Steps to reproduce:

      You need two threads that get the user instance via an instance of UserAccessor, and call UserAccessor#saveUser(confUser).

      1. thread 1 opens a hibernate session
      2. thread 2 opens a hibernate session
      3. thread 1 fetches a user
      4. thread 1 saves the user
      5. thread 2 fetches the same user as thread #1
      6. thread 2 saves the user
      7. the exception is thrown in thread 2
      8. thread 1 closes the hibernate session - this is important, it seems that if the session was closed before thread 2 called saveUser, everything would be ok
      9. thread 2 closes the hibernate session

      The exception:

      org.springframework.orm.hibernate.HibernateSystemException: Illegal attempt to associate a collection with two open sessions; 
      nested exception is net.sf.hibernate.HibernateException: Illegal attempt to associate a collection with two open sessions
      Caused by: net.sf.hibernate.HibernateException: Illegal attempt to associate a collection with two open sessions
              at net.sf.hibernate.collection.PersistentCollection.setCurrentSession(PersistentCollection.java:257)
              at net.sf.hibernate.impl.OnUpdateVisitor.processCollection(OnUpdateVisitor.java:38)
              at net.sf.hibernate.impl.AbstractVisitor.processValue(AbstractVisitor.java:69)
              at net.sf.hibernate.impl.AbstractVisitor.processValues(AbstractVisitor.java:36)
              at net.sf.hibernate.impl.AbstractVisitor.process(AbstractVisitor.java:91)
              at net.sf.hibernate.impl.SessionImpl.doUpdateMutable(SessionImpl.java:1466)
              at net.sf.hibernate.impl.SessionImpl.doUpdate(SessionImpl.java:1480)
              at net.sf.hibernate.impl.SessionImpl.saveOrUpdate(SessionImpl.java:1403)
              at org.springframework.orm.hibernate.HibernateTemplate$12.doInHibernate(HibernateTemplate.java:604)
              at org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:370)
              at org.springframework.orm.hibernate.HibernateTemplate.saveOrUpdate(HibernateTemplate.java:601)
              at com.atlassian.user.impl.hibernate.HibernateUserManager.saveUser(HibernateUserManager.java:171)
              at com.atlassian.user.impl.cache.CachingUserManager.saveUser(CachingUserManager.java:143)
              at com.atlassian.user.impl.delegation.DelegatingUserManager.saveUser(DelegatingUserManager.java:144)
              at bucket.user.DefaultUserAccessor.saveUser(DefaultUserAccessor.java:224)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:296)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:177)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
              at com.atlassian.spring.interceptors.SpringProfilingInterceptor.invoke(SpringProfilingInterceptor.java:20)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
              at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:107)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
              at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
              at $Proxy22.saveUser(Unknown Source)
              at com.sun.dse.wikis.auth.ConfluenceUserManager$1.doInHibernate(ConfluenceUserManager.java:96)
              at org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:370)
              at org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:337)
              at com.sun.dse.wikis.auth.ConfluenceUserManager.updateUser(ConfluenceUserManager.java:79)
              at com.sun.dse.wikis.auth.SsoFilter.doFilter(SsoFilter.java:124)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:217)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185)
              at com.atlassian.confluence.util.ClusterHeaderFilter.doFilter(ClusterHeaderFilter.java:35)
              at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:217)
              at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185)
              at org.springframework.orm.hibernate.support.OpenSessionInV
      

        Activity

        Hide
        Igor Minar added a comment - - edited

        Disconnecting the user object from the current session once I'm done using it seems to help.

        user = userAccessor.getUser(name);
        // do stuff with user
        userAccessor.saveUser(user);
        //commit transaction
        session.evict(user)
        

        But there is one more problem. If two threads (with different hibernate sessions) obtain two instances of the same user (when the cache is empty) and try to save them an exception is thrown:

        org.springframework.orm.hibernate.HibernateSystemException: a different object with the same identifier value was already associated with the session: 13177238, 
        of class: com.atlassian.user.impl.hibernate.DefaultHibernateUser; 
        nested exception is net.sf.hibernate.NonUniqueObjectException: a different object with the same identifier value was already associated with the session: 13177238, 
        of class: com.atlassian.user.impl.hibernate.DefaultHibernateUser
        Caused by: net.sf.hibernate.NonUniqueObjectException: a different object with the same identifier value was already associated with the session: 13177238, 
        of class: com.atlassian.user.impl.hibernate.DefaultHibernateUser
                at net.sf.hibernate.impl.SessionImpl.checkUniqueness(SessionImpl.java:1687)
                at net.sf.hibernate.impl.SessionImpl.doUpdateMutable(SessionImpl.java:1453)
                at net.sf.hibernate.impl.SessionImpl.doUpdate(SessionImpl.java:1480)
                at net.sf.hibernate.impl.SessionImpl.saveOrUpdate(SessionImpl.java:1403)
                at org.springframework.orm.hibernate.HibernateTemplate$12.doInHibernate(HibernateTemplate.java:604)
                at org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:370)
                at org.springframework.orm.hibernate.HibernateTemplate.saveOrUpdate(HibernateTemplate.java:601)
                at com.atlassian.user.impl.hibernate.HibernateUserManager.saveUser(HibernateUserManager.java:171)
                at com.atlassian.user.impl.cache.CachingUserManager.saveUser(CachingUserManager.java:143)
                at com.atlassian.user.impl.delegation.DelegatingUserManager.saveUser(DelegatingUserManager.java:144)
                at bucket.user.DefaultUserAccessor.saveUser(DefaultUserAccessor.java:224)
                at sun.reflect.GeneratedMethodAccessor886.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:296)
                at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:177)
                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144)
                at com.atlassian.spring.interceptors.SpringProfilingInterceptor.invoke(SpringProfilingInterceptor.java:20)
                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
                at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:107)
                at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
                at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
                at $Proxy22.saveUser(Unknown Source)
                at com.sun.dse.wikis.auth.ConfluenceUserManager$1.doInHibernate(ConfluenceUserManager.java:96)
                at org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:370)
                at org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:337)
                at com.sun.dse.wikis.auth.ConfluenceUserManager.updateUser(ConfluenceUserManager.java:79)
                at com.sun.dse.wikis.auth.SsoFilter.doFilter(SsoFilter.java:124)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:217)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185)
                at com.atlassian.confluence.util.ClusterHeaderFilter.doFilter(ClusterHeaderFilter.java:35)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:217)
                at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185)
                at com.atlassian.johnson.filters.AbstractJohnsonFilter.doFilter(AbstractJohnsonFilter.java:72)
                at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:217)
                at org.apache.catalina.core.Appli
        

        To me it looks like UserAccessor#getUser(String name) should be synchronized on the name object in order to avoid this.

        Show
        Igor Minar added a comment - - edited Disconnecting the user object from the current session once I'm done using it seems to help. user = userAccessor.getUser(name); // do stuff with user userAccessor.saveUser(user); //commit transaction session.evict(user) But there is one more problem. If two threads (with different hibernate sessions) obtain two instances of the same user (when the cache is empty) and try to save them an exception is thrown: org.springframework.orm.hibernate.HibernateSystemException: a different object with the same identifier value was already associated with the session: 13177238, of class: com.atlassian.user.impl.hibernate.DefaultHibernateUser; nested exception is net.sf.hibernate.NonUniqueObjectException: a different object with the same identifier value was already associated with the session: 13177238, of class: com.atlassian.user.impl.hibernate.DefaultHibernateUser Caused by: net.sf.hibernate.NonUniqueObjectException: a different object with the same identifier value was already associated with the session: 13177238, of class: com.atlassian.user.impl.hibernate.DefaultHibernateUser at net.sf.hibernate.impl.SessionImpl.checkUniqueness(SessionImpl.java:1687) at net.sf.hibernate.impl.SessionImpl.doUpdateMutable(SessionImpl.java:1453) at net.sf.hibernate.impl.SessionImpl.doUpdate(SessionImpl.java:1480) at net.sf.hibernate.impl.SessionImpl.saveOrUpdate(SessionImpl.java:1403) at org.springframework.orm.hibernate.HibernateTemplate$12.doInHibernate(HibernateTemplate.java:604) at org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:370) at org.springframework.orm.hibernate.HibernateTemplate.saveOrUpdate(HibernateTemplate.java:601) at com.atlassian.user.impl.hibernate.HibernateUserManager.saveUser(HibernateUserManager.java:171) at com.atlassian.user.impl.cache.CachingUserManager.saveUser(CachingUserManager.java:143) at com.atlassian.user.impl.delegation.DelegatingUserManager.saveUser(DelegatingUserManager.java:144) at bucket.user.DefaultUserAccessor.saveUser(DefaultUserAccessor.java:224) at sun.reflect.GeneratedMethodAccessor886.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:296) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:177) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:144) at com.atlassian.spring.interceptors.SpringProfilingInterceptor.invoke(SpringProfilingInterceptor.java:20) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:107) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at $Proxy22.saveUser(Unknown Source) at com.sun.dse.wikis.auth.ConfluenceUserManager$1.doInHibernate(ConfluenceUserManager.java:96) at org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:370) at org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:337) at com.sun.dse.wikis.auth.ConfluenceUserManager.updateUser(ConfluenceUserManager.java:79) at com.sun.dse.wikis.auth.SsoFilter.doFilter(SsoFilter.java:124) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:217) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185) at com.atlassian.confluence.util.ClusterHeaderFilter.doFilter(ClusterHeaderFilter.java:35) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:217) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:185) at com.atlassian.johnson.filters.AbstractJohnsonFilter.doFilter(AbstractJohnsonFilter.java:72) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:217) at org.apache.catalina.core.Appli To me it looks like UserAccessor#getUser(String name) should be synchronized on the name object in order to avoid this.
        Hide
        Anatoli Kazatchkov [Atlassian] added a comment -

        Hi Igor,

        This issues seems to look very much like your second problem. Can you please check what version of atlassian-user library you are using as the bug was fixed in 1.28. If you use the updated version of the library and still have the exception thrown can you please give more details on what you are trying to do when the exception is thrown (code snippet? )

        If two threads (with different hibernate sessions) obtain two instances of the same user (when the cache is empty) and try to save them an exception is thrown:

        I don't think this is the reason for the exception, after all it reports the same identifier withing one session.

        Caused by: net.sf.hibernate.NonUniqueObjectException: a different object with the same identifier value was already associated with the session: 13177238, 
        

        The first reported problem is a bug. It happens because we use the collection from the cache that is already associated with an open hibernate session and try to attach it to a new session.
        http://jira.atlassian.com/browse/USER-228

        Show
        Anatoli Kazatchkov [Atlassian] added a comment - Hi Igor, This issues seems to look very much like your second problem. Can you please check what version of atlassian-user library you are using as the bug was fixed in 1.28. If you use the updated version of the library and still have the exception thrown can you please give more details on what you are trying to do when the exception is thrown (code snippet? ) If two threads (with different hibernate sessions) obtain two instances of the same user (when the cache is empty) and try to save them an exception is thrown: I don't think this is the reason for the exception, after all it reports the same identifier withing one session. Caused by: net.sf.hibernate.NonUniqueObjectException: a different object with the same identifier value was already associated with the session: 13177238, The first reported problem is a bug. It happens because we use the collection from the cache that is already associated with an open hibernate session and try to attach it to a new session. http://jira.atlassian.com/browse/USER-228
        Hide
        Igor Minar added a comment -

        Hi Anatoli,

        Yes, the second issue looks like USER-239. We are using Confluence 2.8, which shipped with atlassian user 1.27. I'll upgrade to the latest version and check if we still see the issue.

        The first problem is very likely caused by USER-228. Caching persistent objects is really a bad idea, especially if they are still attached to a session that retrieved them from the database. We have hundreds of exceptions in our logs because of this issue. It would be great if it got fixed soon. For now the workaround that I described earlier (using session.evict()) works, but that's pretty ugly.

        Thanks for looking into this!

        /i

        Show
        Igor Minar added a comment - Hi Anatoli, Yes, the second issue looks like USER-239. We are using Confluence 2.8, which shipped with atlassian user 1.27. I'll upgrade to the latest version and check if we still see the issue. The first problem is very likely caused by USER-228. Caching persistent objects is really a bad idea, especially if they are still attached to a session that retrieved them from the database. We have hundreds of exceptions in our logs because of this issue. It would be great if it got fixed soon. For now the workaround that I described earlier (using session.evict()) works, but that's pretty ugly. Thanks for looking into this! /i
        Hide
        Igor Minar added a comment -

        I'm seeing the same issue with addMembership method :-/

        http://kenai.com/jira/browse/SUNWIKIS-7

        Can we get it fixed please?

        Show
        Igor Minar added a comment - I'm seeing the same issue with addMembership method :-/ http://kenai.com/jira/browse/SUNWIKIS-7 Can we get it fixed please?
        Hide
        Anatoli Kazatchkov [Atlassian] added a comment -

        The same problem indeed. We should probably stop playing a "whack a mole" game and address [USER-244] issue instead.

        Show
        Anatoli Kazatchkov [Atlassian] added a comment - The same problem indeed. We should probably stop playing a "whack a mole" game and address [USER-244] issue instead.
        Hide
        Matt Ryall [Atlassian] added a comment -

        We've rewritten user management in Confluence completely for version 3.5. It should no longer suffer from problems like this.

        Show
        Matt Ryall [Atlassian] added a comment - We've rewritten user management in Confluence completely for version 3.5. It should no longer suffer from problems like this.

          People

          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:
              Last commented:
              3 years, 6 weeks ago