Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-12319

Adding users to a large group is slow with default (Hibernate) user management

XMLWordPrintable

      While debugging an outage at wikis.sun.com I noticed that the code in HibernateUserManager#addMembership generates some extremely inefficient queries that were giving our db and network hard time:

      membership = dGroup.getLocalMembers();

if (membership == null)
{
       membership = new HashSet();
       dGroup.setLocalMembers(membership);
}

membership.add(user);

      The last line of the code translates to:

      DEBUG 2008-07-02 11:35:43,801 [service-j2ee-3] BatcherImpl:log - select localmembe0_.groupid as groupid__, localmembe0_.userid as userid__, defaulthib1_.id as id0_, defaulthib1_.name as name0_, defaulthib1_.password as password0_, defaulthib1_.email as email0_, defaulthib1_.created as created0_, defaulthib1_.fullname as fullname0_ from local_members localmembe0_ inner join users defaulthib1_ on localmembe0_.userid=defaulthib1_.id where localmembe0_.groupid=?

DEBUG 2008-07-02 11:35:43,806 [service-j2ee-3] BatcherImpl:log - insert into local_members (groupid, userid) values (?, ?)

      Which means retrieve*all* members of a given group and then insert the user to the db.

      If you run this query on our db with 25k users in a group, you run into some really big problems. By that I mean that the query can easily run several minutes and affect the overall db performance.

      The code should be rewritten so that the uniqueness constraint is checked by a SELECT and if no dupe is found the INSERT can follow, otherwise this code will never scale.

        1. search-3.37.zip
          73 kB

            [CONFSERVER-12319] Adding users to a large group is slow with default (Hibernate) user management

            Anatoli added a comment -

            Hi Igor,

            the patch is almost certainly will NOT make it into 3.1. I would say it is more likely to make it in 3.1.1 but I don't want to disappoint you in case it does not.

            Anatoli.

            Anatoli added a comment - Hi Igor, the patch is almost certainly will NOT make it into 3.1. I would say it is more likely to make it in 3.1.1 but I don't want to disappoint you in case it does not. Anatoli.

            Igor Minar added a comment -

            Thanks Anatoli, I did some testing and the patched atlassian-user resolved all the known issues.

            As far as I can tell at the moment, everything seems to work as expected.

            Will the patch make it to 3.1?

            cheers,
            Igor

            Igor Minar added a comment - Thanks Anatoli, I did some testing and the patched atlassian-user resolved all the known issues. As far as I can tell at the moment, everything seems to work as expected. Will the patch make it to 3.1? cheers, Igor

            Anatoli added a comment -

            Hi Igor,

            We run all our test against the build with the flag and found some problems that are [logged here|USER-267]. A patch jar is attached to that case.

            Anatoli.

            Anatoli added a comment - Hi Igor, We run all our test against the build with the flag and found some problems that are [logged here|USER-267] . A patch jar is attached to that case. Anatoli.

            Igor Minar added a comment -

            sadly, even 3.0.2 still can't remove users via the admin interface. It fails with:

            net.sf.hibernate.LazyInitializationException: Failed to lazily initialize a collection - no session or session was closed
            at net.sf.hibernate.collection.PersistentCollection.initialize(PersistentCollection.java:209)
            at net.sf.hibernate.collection.PersistentCollection.write(PersistentCollection.java:84)
            at net.sf.hibernate.collection.Set.remove(Set.java:162)
            at com.atlassian.user.impl.hibernate.HibernateGroupManager.removeMembership(HibernateGroupManager.java:405)
            at com.atlassian.user.impl.cache.CachingGroupManager.removeMembership(CachingGroupManager.java:178)
            at com.atlassian.user.impl.delegation.DelegatingGroupManager.removeMembership(DelegatingGroupManager.java:234)
            at bucket.user.DefaultUserAccessor.removeMembership(DefaultUserAccessor.java:527)
            at com.atlassian.confluence.user.DefaultUserAccessor.removeMembership(DefaultUserAccessor.java:97)
            at com.atlassian.confluence.user.DefaultUserAccessor.removeUser(DefaultUserAccessor.java:226)

            Igor Minar added a comment - sadly, even 3.0.2 still can't remove users via the admin interface. It fails with: net.sf.hibernate.LazyInitializationException: Failed to lazily initialize a collection - no session or session was closed at net.sf.hibernate.collection.PersistentCollection.initialize(PersistentCollection.java:209) at net.sf.hibernate.collection.PersistentCollection.write(PersistentCollection.java:84) at net.sf.hibernate.collection.Set.remove(Set.java:162) at com.atlassian.user.impl.hibernate.HibernateGroupManager.removeMembership(HibernateGroupManager.java:405) at com.atlassian.user.impl.cache.CachingGroupManager.removeMembership(CachingGroupManager.java:178) at com.atlassian.user.impl.delegation.DelegatingGroupManager.removeMembership(DelegatingGroupManager.java:234) at bucket.user.DefaultUserAccessor.removeMembership(DefaultUserAccessor.java:527) at com.atlassian.confluence.user.DefaultUserAccessor.removeMembership(DefaultUserAccessor.java:97) at com.atlassian.confluence.user.DefaultUserAccessor.removeUser(DefaultUserAccessor.java:226)

            Igor Minar added a comment -

            Hi Anatoli,

            sorry, due to how this issue is linked with the other issue, I didn't notice that it was fixed in 3.0.2.

            I'm going to test it now and will let you know if the fix really works.

            Have you by chance ran your test suite against a build with the flag on? That would be a good indicator if there are any remaining issues.

            thanks,
            Igor

            Igor Minar added a comment - Hi Anatoli, sorry, due to how this issue is linked with the other issue, I didn't notice that it was fixed in 3.0.2. I'm going to test it now and will let you know if the fix really works. Have you by chance ran your test suite against a build with the flag on? That would be a good indicator if there are any remaining issues. thanks, Igor

            Anatoli added a comment -

            Hi Igor,

            how can a fix that breaks other confluence functionality be called "proper fix"?

            What functionality are you referring to? If you are talking about the problem you have mentioned before:

            We found a problem. When the experimental mapping is turned on, we can't remove users from groups via the admin UI. Is there an alternative or enhanced solution?

            then it has been fixed in 3.0.2

            If you meant something else, please let us know so that we can investigate it further.

            Anatoli.

            Anatoli added a comment - Hi Igor, how can a fix that breaks other confluence functionality be called "proper fix"? What functionality are you referring to? If you are talking about the problem you have mentioned before: We found a problem. When the experimental mapping is turned on, we can't remove users from groups via the admin UI. Is there an alternative or enhanced solution? then it has been fixed in 3.0.2 If you meant something else, please let us know so that we can investigate it further. Anatoli.

            Igor Minar added a comment -

            Partha, how can a fix that breaks other confluence functionality be called "proper fix"? Am I missing something?

            Igor Minar added a comment - Partha, how can a fix that breaks other confluence functionality be called "proper fix"? Am I missing something?

            Partha added a comment -

            Dear Watchers,

            Currently, the fix which requires you to use the flag -Dcom.atlassian.user.experimentalMapping=true is included in both 3.0.2 and the upcoming 3.1 release.

            The naming has not been changed (i.e. it still says experimentalMapping), however it is the proper fix for situations where you have a large number of local users AND these are mapped to local groups.

            This issue has not been closed however, since the fix does not cover the situation where you have external users (from LDAP for example) that are mapped to local groups. Development would like to include this fix as well, before closing this issue.
            Also they need to polish this off (i.e. change the naming of the parameter or remove it altogether).

            Kind Regards,
            Partha Kamal

            Partha added a comment - Dear Watchers, Currently, the fix which requires you to use the flag -Dcom.atlassian.user.experimentalMapping=true is included in both 3.0.2 and the upcoming 3.1 release. The naming has not been changed (i.e. it still says experimentalMapping), however it is the proper fix for situations where you have a large number of local users AND these are mapped to local groups. This issue has not been closed however, since the fix does not cover the situation where you have external users (from LDAP for example) that are mapped to local groups. Development would like to include this fix as well, before closing this issue. Also they need to polish this off (i.e. change the naming of the parameter or remove it altogether). Kind Regards, Partha Kamal

            Andreas Richter added a comment -

            Hi Igor,

            Thanks for your detailed answer. It saves time. I have to think about what I'll do with this information.

            Best regards,
            Andreas

            Andreas Richter added a comment - Hi Igor, Thanks for your detailed answer. It saves time. I have to think about what I'll do with this information. Best regards, Andreas

            Igor Minar added a comment -

            Andreas,

            This is not a Tangosol issue, but rather a problem caused by incorrect use of hibernate.

            Right now when you add a single user to your confluence-users group,
            hibernate is instructed to pull all 200k of your users from the db,
            create a java collection for these users, add the new user to the
            collection and then persist the collection into the db
            (which probably results in a single insert sql which could have been
            used without creating the collection with users in the first place).

            So even ehcache is affected by this issue, but it's not causing major
            issues with hundreds of thousands of users. Things are just slow.
            When you set up a cluster, the problem is that tangosol is instructed
            to replicate the huge collection of users on all of your nodes, and
            this takes forever + makes the app too busy which results in cluster
            panics.

            • the mapping also breaks adding and removing users to a group via the Custom Space User Management plugin. Is this still an issue?

            yes, still a problem

            • Has somebody tried to add an user to a group via UI (administrators section). Is this working?

            haven't tried this one

            /i

            Igor Minar added a comment - Andreas, This is not a Tangosol issue, but rather a problem caused by incorrect use of hibernate. Right now when you add a single user to your confluence-users group, hibernate is instructed to pull all 200k of your users from the db, create a java collection for these users, add the new user to the collection and then persist the collection into the db (which probably results in a single insert sql which could have been used without creating the collection with users in the first place). So even ehcache is affected by this issue, but it's not causing major issues with hundreds of thousands of users. Things are just slow. When you set up a cluster, the problem is that tangosol is instructed to replicate the huge collection of users on all of your nodes, and this takes forever + makes the app too busy which results in cluster panics. the mapping also breaks adding and removing users to a group via the Custom Space User Management plugin. Is this still an issue? yes, still a problem Has somebody tried to add an user to a group via UI (administrators section). Is this working? haven't tried this one /i

              alynch Andrew Lynch (Inactive)
              15d9a6950818 Igor Minar
              Affected customers:
              11 This affects my team
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: