-
Bug
-
Resolution: Fixed
-
Medium (View bug fix roadmap)
-
6.6.0, 6.6.11, 6.6.12
-
None
-
6.06
-
Summary
When a LexoRank balance job is scheduled on a large instance (200k+ issues) it can take a very long period of time to complete and performs slowly. On an instance with two LexoRank fields it is taking more than 12 hours for each field.
Environment
- Intel® Xeon® Processor E5462 x 2 (8 threads)
- 12 gb RAM. 6gb heap allocated.
- SSD for database and JIRA.
- MySQL database, mounted on the same server.
- Linux 3.13.0-39-generic #66-Ubuntu SMP Tue Oct 28 13:30:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Instance size:
___ Database Statistics ____________________
Issues : 965632
Projects : 910
Custom Fields : 460
Workflows : 278
Users : 7798
Groups : 314
Attachments : 462454
Comments : 155671
Two LexoRank custom fields:
+----------+----------+ | count(*) | FIELD_ID | +----------+----------+ | 827972 | 15180 | | 827972 | 15182 |
Steps to Reproduce
Schedule a balance (either POST to the endpoint or through the LexoRank management tools in 6.7.0+).
Expected Results
The balance completes within a suitable time frame.
Actual Results
The balance takes a significant amount of time to complete.
Notes
This can be sped up by increasing the available resources on the server, reducing the load and also verifying an appropriate read / write speed is present on the Lucene directory. It can be tested as per our Test the Disk Speed. Also note we do not support NFS mounts for Lucene as per JIRA Supported Platforms.
Data
There are 6 thread dumps, with associated CPU % information, attached to this issue contained within balancing_operations.tar.gz
. It appears that this is a single-threaded operation as only lexorank-executor-thread-0 is present.
- is related to
-
-
- Closed
-
-
-
- Closed
-
-
JDEV-33155 Failed to load
-
JSEV-121 You do not have permission to view this issue
-
- incorporates
-
JDEV-32430 Loading...
-
-
- is a clone of
-
- mentioned in
-
-
-
-
-
-
-
- relates to
-
- was cloned as
-
-
[JSWSERVER-11491] LexoRank Balance takes a very long time on larger instances
It's been 20 hrs and we're only 43% complete:
{
"lexoRankBalancingServiceStatus": {
"balancingDisabled": false,
"balanceHandlerRunning": true
},
"lexoRankBalancerStatus": {
"balancerLocked": true,
"perFieldStatus": [
{
"fieldName": "Rank",
"fieldId": 12970,
"numRankedIssues": 1173415,
"percentComplete": 43.12,
"distribution": [
667450,
505965,
0
]
}
]
},
"totalIssueCount": 1151865
}
I'm partly glad it's single threaded; at least it's not exhausting the system for regular users.
Hope a faster way can be found without affecting user experience.
There are also very little clues when lexorank reblance kicks off. Nothing registered in the logs--just a sudden bump in CPU cycles, and GC collection activity.
It took us almost 10 days to balance 2 Rank fields for 430.000 issues, using JIRA Agile 6.7.11.
This was tested in a test-environment with same setup as production.
We feel we cannot upgrade JIRA and JIRA Agile in production until this is fixed somehow.
Still not sure about what impact this will have for the users, while balancing in the background.