LexoRank Balance takes a very long time on larger instances

Summary

When a LexoRank balance job is scheduled on a large instance (200k+ issues) it can take a very long period of time to complete and performs slowly. On an instance with two LexoRank fields it is taking more than 12 hours for each field.

Environment

• Intel® Xeon® Processor E5462 x 2 (8 threads)
• 12 gb RAM. 6gb heap allocated.
• SSD for database and JIRA.
• MySQL database, mounted on the same server.
• Linux 3.13.0-39-generic #66-Ubuntu SMP Tue Oct 28 13:30:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Instance size:

___ Database Statistics ____________________

     Issues                                        : 965632
     Projects                                      : 910
     Custom Fields                                 : 460
     Workflows                                     : 278
     Users                                         : 7798
     Groups                                        : 314
     Attachments                                   : 462454
     Comments                                      : 155671

Two LexoRank custom fields:

+----------+----------+
| count(*) | FIELD_ID |
+----------+----------+
|   827972 |    15180 |
|   827972 |    15182 |

Steps to Reproduce

Schedule a balance (either POST to the endpoint or through the LexoRank management tools in 6.7.0+).

Expected Results

The balance completes within a suitable time frame.

Actual Results

The balance takes a significant amount of time to complete.

Notes

This can be sped up by increasing the available resources on the server, reducing the load and also verifying an appropriate read / write speed is present on the Lucene directory. It can be tested as per our Test the Disk Speed. Also note we do not support NFS mounts for Lucene as per JIRA Supported Platforms.

Data

There are 6 thread dumps, with associated CPU % information, attached to this issue contained within balancing_operations.tar.gz. It appears that this is a single-threaded operation as only lexorank-executor-thread-0 is present.