Crowd throws OutOfMemoryError while syncing users to Hipchat Data Center

Summary

When embedded Crowd sync's user data from the Crowd db to HipChat db, the process spawns thousands of threads and throws an OutOfMemoryError exception. This is only triggered when an excessive amount of users are brought into Crowd per the directory integration specifically sync'ing users from an external Crowd/Jira server instance or LDAP/AD.

Environment

• Hipchat Data Center 3.1.2
• Directory integration linked to an external Crowd/Jira server instance or LDAP/AD

Steps to Reproduce

1. Setup the directory integration to the external directory with ~20k users
2. Run the initial sync
3. Run the following command for the number of threads spun up by Crowd:

   while true; do date; ps -o nlwp `ps aux | grep [c]rowd | awk '{print $2}'`; sleep 0.5; done > /tmp/crowd_thread_count.txt
   

Expected Results

The sync runs and all users are brought over (or denied due to license constraints)

Actual Results

The sync fails and throws the below exception to /var/log/hipchat/atlassian-crowd.log:

java.lang.OutOfMemoryError: unable to create new native thread

Workaround

• Break down your directory connector into two or more based on filters.
• Use SAML for SSO