Lexorank operation can fail for 10 minutes and cause communication breakdown when creating multiple subtasks on transition with 3rd party plugins

Summary

Lexorank operation can fail for 10 minutes and cause communication breakdown when creating multiple subtasks on transition with 3rd party plugins

Environment

• Instance is setup to use the BobSwift create on transition plugin (I believe scriptrunner could also do this), as well as the JIRA Workflow Toolbox extension in order to be able to create upwards of 20 subtasks when a particular issue is transitioned, based upon the values of custom fields in that issue
• instance has this problem on 7.2.6, 7.2.8, and 7.3.4
• This problem is related to the previous bug of JSWSERVER-13756. The major difference being that JIRA does not deadlock in this case when this happens now. However it can still generate an error and appear to temporarily have a problem in regards to the indexes of this parent issue/child issues for a few minutes.

Steps to Reproduce

1. Create a workflow that will generate 20 different subtasks upon transition that are all children of the same parent issue.
2. Create two different parent issues in this project
3. Execute this same transition on both issues at nearly the same time

Expected Results

JIRA will create all the subtasks and continue on it's way happily

Actual Results

1. JIRA displays a spinning wheel for a several seconds before it throws a communication breakdown error
2. Also it appears that the problem is in regards to ranking operations that are failing:

   2017-04-18 18:52:09,175 http-nio-4933-exec-24 uri:/secure/WorkflowUIDispatcher.jspa username:administrator DEBUG administrator 1131x1376x1 1xldvnm 0:0:0:0:0:0:0:1 /secure/WorkflowUIDispatcher.jspa [c.a.g.service.lexorank.LexoRankOperation] RANK_INITIAL end   [fieldId=10300, issueToRankIssueId=31332, issueToRankAroundIssueId=null
   2017-04-18 18:52:09,175 http-nio-4933-exec-24 uri:/secure/WorkflowUIDispatcher.jspa username:administrator WARN administrator 1131x1376x1 1xldvnm 0:0:0:0:0:0:0:1 /secure/WorkflowUIDispatcher.jspa [c.a.g.customfield.lexorank.LexoRankCFType] Unable to retrieve rank for field [10300] and issue [31332]
   2017-04-18 18:52:09,175 http-nio-4933-exec-24 uri:/secure/WorkflowUIDispatcher.jspa username:administrator WARN administrator 1131x1376x1 1xldvnm 0:0:0:0:0:0:0:1 /secure/WorkflowUIDispatcher.jspa [c.a.g.customfield.lexorank.LexoRankCFType] gh.lexorank.service.error.retrytimeout 
   

3. In this case it took the system almost exactly 10 minutes in order to finally finish the initial ranking option for issue 31332:

   2017-04-18 19:02:18,667 http-nio-4933-exec-24 uri:/secure/WorkflowUIDispatcher.jspa username:administrator DEBUG administrator 1131x1376x1 1xldvnm 0:0:0:0:0:0:0:1 /secure/WorkflowUIDispatcher.jspa [c.a.g.service.lexorank.LexoRankOperation] Ranked issue[id=31332] initially with rank[1|i007wn:] for rank field[id=10300]  
   

Notes

• JIRA does appear to recover from this problem eventually, unlike the previous JSWSERVER-13756, but it does appear to take several minutes before the system is able to rank all the issues. In some cases a foreground index might also be required to update the indexes.
• If only transitioning one of these issues, this problem does not appear to happen. However having more than one JIRA user in the system is expected to happen, so the occurrence of these nearly simultaneous transitions is likely.
• Feels like maybe we are hitting up against a limitation of multiple threads all trying to obtain a lock on the rank field of an issue

Workaround

• Disable Bobswift plugin (which prevents the creation of the subtasks)
• Manually create subtasks