Severe performance degradation of "Bulk Edit" as "Assignable Users" increases

We recently switched our project that handles "Support Requests" from restricting assignable users to project members (< 100) to assignable users including all employees that might use JIRA (3984).

Before this change, "Bulk Edit" -> "Edit Issues" was previously a comfortable speed of a few seconds or less. Now, if I select "Edit Issues" and "Next", for a selection of just 38 issues - it took 34 seconds to switch to the next screen just now. This is the best case as I am running this operation at 11:12pm at night. Earlier today during peak time in the afternoon, the same operation with 51 issues was not completing in several minutes (we didn't time it other than to note that several minutes had passed at which point we gave up waiting).

As the operation was taking a long time to complete, and as I knew it was happening, I began to investigate.

I noted that the JIRA process was consuming 100% of one CPU core. As I know this typically means that one Java thread is running at full speed, I used "jstack" to view the thread stacks. I noted one thread in particular here:

"http-8080-7" daemon prio=10 tid=0x00007fef800e6000 nid=0x4df0 runnable [0x00007fef9145b000]
   java.lang.Thread.State: RUNNABLE
        at java.lang.String.toLowerCase(String.java:2390)
        at com.atlassian.crowd.embedded.impl.IdentifierUtils.toLowerCase(IdentifierUtils.java:42)
        at com.atlassian.crowd.embedded.impl.IdentifierUtils.compareToInLowerCase(IdentifierUtils.java:56)
        at com.atlassian.crowd.embedded.impl.IdentifierUtils.equalsInLowerCase(IdentifierUtils.java:70)
        at com.atlassian.crowd.embedded.api.UserComparator.equal(UserComparator.java:74)
        at com.atlassian.crowd.embedded.ofbiz.OfBizUser.equals(OfBizUser.java:101)
        at java.util.ArrayList.indexOf(ArrayList.java:216)
        at java.util.ArrayList.contains(ArrayList.java:199)
        at java.util.AbstractCollection.retainAll(AbstractCollection.java:369)
        at com.atlassian.jira.issue.fields.Assignees.bulkOptionsForFrotherControl(Assignees.java:115)
        at com.atlassian.jira.issue.fields.AssigneeSystemField.getBulkEditHtml(AssigneeSystemField.java:202)

Once I had time to look more closely - I noticed that the above has a dreaded call to AbstractCollection.retainAll(). I checked the implementation of Assignees.bulkOptionsForFrotherControl() and found:

        ...
        for (Issue issue : issues)
        {
            List<User> issueAssignableUsers = assigneeService.getAssignableUsers(issue, actionDescriptor);
            if (bulkAssignableUsers == null)
            {
                bulkAssignableUsers = issueAssignableUsers;
            }
            else
            {
                // Keep filtering the list to only include users assignable for ALL issues.
                bulkAssignableUsers.retainAll(issueAssignableUsers);
            }
            ...
         }
         ...

The AbstractCollection.retainAll() implementation in Java 6 is:

    public boolean retainAll(Collection<?> c) {
        boolean modified = false;
        Iterator<E> e = iterator();
        while (e.hasNext()) {
            if (!c.contains(e.next())) {
                e.remove();
                modified = true;
            }
        }
        return modified;
    }

To compare two unordered lists using the AbstractCollection.retainAll() method means it must compare each element to each other element. For 3984 assignable users - to compare 3984 to 3984 users takes 3984 x 3984 = 15,872,256 operations to complete.

To make matters worse - it's not a simple comparison. As you can see from the stack trace, it is calling IdentifierUtils.toLowerCase for the left side, and presumedly the right side, before performing the comparison. So the 16 million operations are actually 16 million complex operations.

Finally - the above happens for every issue except the first. Presuming that each of the issues has the same set of assignable users - 3984 users in my case - then a bulk edit of 51 issues would run the 16 million operations x 50 issues = 800 million operations.

Wow.

I believe this issue should be directed towards the new "Enterprise" team to fix.

Thanks!