-
Bug
-
Resolution: Fixed
-
High (View bug fix roadmap)
-
6.2.5, 6.3.4, 6.3.14, 6.4.6
-
PostgreSQL (likely other DBs that support isolation)
-
6.02
-
Problem description:
On an active JIRA, concurrent read access to links (such as when an issue is being displayed) and write access to links (such as when a new link or sub-task is created or deleted) may result in errors:
- when a link/subtask is created, and this problem happens, the link/subtask will not be displayed at the parent issue (even after creation succeeds)
- when a subtask (or linked issue) is deleted, the parent task “breaks”, the issue page displays an error and there’s a NullPointerException in the server log.
This is a race condition, so exact reproduction sequence is hard to define. However, I managed to reproduce the problem with 100% certainty using debugger and running JIRA on Postres.
Investigation:
DefaultIssueLinkManager has a global per-instance cache of the inward/outward links. It is not transactional, and shared between threads that run transactional changes on the database, and threads that do reading. As a result, the following may happen:
- Thread 1: change transaction starts
- Thread 1: DefaultIssueLinkManager performs a write operation on a link in the DB
- Thread 1: cleans the cache for affected issues
- Thread 2: requests getInwardLinks or getOutwardLinks from DefaultIssueLinkManager for the affected issue
- Thread 2: because Thread 1’s transaction is not committed yet, old state is read and saved in the cache
- Thread 1: commits the transaction
- Thread 1 or any other thread: calls getInwardLinks or getOutwardLinks - but gets corrupt cached value, which relates to the state before the transaction has committed.
How to reproduce using debugger:
- Start JIRA instance with Postgres - I couldn’t reproduce on HSQL
- Create issue X and sub-task S
- Prepare a class with a piece of code that would a) start a new thread, b) in the thread, request getOutwardLinks() for a specified issue ID, c) wait for the thread to finish.
- Place a breakpoint in DefaultIssueDeleteHelper.deleteIssue(), on the line removeIssueLinks(user, issue);
- In JIRA, delete sub-task S — catch the breakpoint in debugger.
- Step over “removeIssueLinks”. Inside, the links will be removed, and cache will be cleared several times.
- Before continuing, use debugger’s Evaluate feature to call the prepared method that retrieves outward links in another thread - call it on the parent issue X.
- Then you can verify that IssueLinkManager’s cache will contain an invalid entry for issue X with the link that has just been deleted.
- In JIRA web interface, if you open the parent issue’s page, there will be errors, and there will be an NPE in the log.
Note that this is only an example - the same problem can happen in most any place where links cache is being invalidated.
The NPE:
[INFO] [talledLocalContainer] 2014-11-19 03:50:24,925 http-bio-2990-exec-19 ERROR admin 230x15313x1 890nfl fe80:0:0:0:cae0:ebff:fe18:7e59%4 /secure/AjaxIssueAction!default.jspa [atlassian.plugin.web.DefaultWebInterfaceManager] Could not evaluate condition 'com.atlassian.plugin.web.conditions.AndCompositeCondition@477a8ce9' for descriptor: com.atlassian.jira.jira-view-issue-plugin:view-subtasks (null) [INFO] [talledLocalContainer] java.lang.NullPointerException [INFO] [talledLocalContainer] at com.atlassian.jira.security.DefaultPermissionManager.doIssuePermissionCheck(DefaultPermissionManager.java:283) [INFO] [talledLocalContainer] at com.atlassian.jira.security.DefaultPermissionManager.hasPermission(DefaultPermissionManager.java:195) [INFO] [talledLocalContainer] at com.atlassian.jira.security.WorkflowBasedPermissionManager.hasPermission(WorkflowBasedPermissionManager.java:97) [INFO] [talledLocalContainer] at com.atlassian.jira.security.DefaultPermissionManager.hasPermission(DefaultPermissionManager.java:190) [INFO] [talledLocalContainer] at com.atlassian.jira.security.WorkflowBasedPermissionManager.hasPermission(WorkflowBasedPermissionManager.java:90) [INFO] [talledLocalContainer] at sun.reflect.GeneratedMethodAccessor542.invoke(Unknown Source) [INFO] [talledLocalContainer] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [INFO] [talledLocalContainer] at java.lang.reflect.Method.invoke(Method.java:606) [INFO] [talledLocalContainer] at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:83) [INFO] [talledLocalContainer] at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28) [INFO] [talledLocalContainer] at com.sun.proxy.$Proxy9.hasPermission(Unknown Source) [INFO] [talledLocalContainer] at com.atlassian.jira.config.DefaultSubTaskManager.getSubTaskBean(DefaultSubTaskManager.java:391) [INFO] [talledLocalContainer] at sun.reflect.GeneratedMethodAccessor689.invoke(Unknown Source) [INFO] [talledLocalContainer] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [INFO] [talledLocalContainer] at java.lang.reflect.Method.invoke(Method.java:606) [INFO] [talledLocalContainer] at com.atlassian.plugin.osgi.hostcomponents.impl.DefaultComponentRegistrar$ContextClassLoaderSettingInvocationHandler.invoke(DefaultComponentRegistrar.java:129) [INFO] [talledLocalContainer] at com.sun.proxy.$Proxy63.getSubTaskBean(Unknown Source) [INFO] [talledLocalContainer] at sun.reflect.GeneratedMethodAccessor689.invoke(Unknown Source) [INFO] [talledLocalContainer] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [INFO] [talledLocalContainer] at java.lang.reflect.Method.invoke(Method.java:606) [INFO] [talledLocalContainer] at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:154) [INFO] [talledLocalContainer] at com.sun.proxy.$Proxy63.getSubTaskBean(Unknown Source) [INFO] [talledLocalContainer] at com.atlassian.jira.plugin.viewissue.HasSubTaskCondition.getSubTaskBean(HasSubTaskCondition.java:57) [INFO] [talledLocalContainer] at com.atlassian.jira.plugin.viewissue.HasSubTaskCondition.shouldDisplay(HasSubTaskCondition.java:41) [INFO] [talledLocalContainer] at com.atlassian.plugin.web.conditions.OrCompositeCondition.shouldDisplay(OrCompositeCondition.java:14) [INFO] [talledLocalContainer] at com.atlassian.plugin.web.conditions.AndCompositeCondition.shouldDisplay(AndCompositeCondition.java:14) [INFO] [talledLocalContainer] at com.atlassian.plugin.web.conditions.AndCompositeCondition.shouldDisplay(AndCompositeCondition.java:14) [INFO] [talledLocalContainer] at com.atlassian.plugin.web.DefaultWebInterfaceManager.filterFragmentsByCondition(DefaultWebInterfaceManager.java:153) [INFO] [talledLocalContainer] at com.atlassian.plugin.web.DefaultWebInterfaceManager.getDisplayableWebPanelDescriptors(DefaultWebInterfaceManager.java:116)
Workaround:
If you are seeing this error on issues, please refer to https://confluence.atlassian.com/x/mhDtKQ.
- causes
-
JSEV-670 You do not have permission to view this issue
- is duplicated by
-
JDEV-33892 Failed to load
- mentioned in
-
Page
Failed to load
- was cloned as
-
JDEV-32815 Failed to load
[JRASERVER-40942] When issue link is being created or deleted in a transaction, concurrent access to IssueLinkManager may lead to corrupt data in the links cache
Ingo Kampe
added a comment - Would be interested in a backport for 6.4.x. Any chance? Hi avegdani,
The fix to this issue has been finalised and it will be shipped with JIRA 7.0.1
Regards,
Oswaldo Hernández.
JIRA Bugmaster.
[Atlassian].
Amir Vegdani
added a comment - Hi Oswaldo,
There is no update on this issue since June 2015. How much time your "good progress" would take to resolve this issue.
This is a lot more serious issue. All user are manually working on stuff, which were automated before. This is not good customer experience.
Amir Vegdani
added a comment - Hi Oswaldo,
There is no update on this issue since June 2015. How much time your "good progress" would take to resolve this issue.
This is a lot more serious issue. All user are manually working on stuff, which were automated before. This is not good customer experience. 
Kristian Peacocke
added a comment - I too see this issue - it only impacts subtasks being created "automatically" (so via script and or different plugins as part of the parent workflow transition). The link appears to be broken on both sides.
Interestingly enough, if you manually create another subtask then every subtask appears under the issue, and the subtasks issues also begin to render the parent issue information.
Kristian Peacocke
added a comment - I too see this issue - it only impacts subtasks being created "automatically" (so via script and or different plugins as part of the parent workflow transition). The link appears to be broken on both sides.
Interestingly enough, if you manually create another subtask then every subtask appears under the issue, and the subtasks issues also begin to render the parent issue information. I am seeing a problem like this in Jira 6.3.15 with one small difference. Everyone has mentioned that the sub-task links have been showing up on the Parent side, and it has been the sub-task side that has been broken. What we have seen is that the sub-tasks are created (between 6 and 8 of them), but the links are not showing on the Parent. From the sub-task side, everything appears to be correct. After an amount of time (we have unconfirmed anywhere between 1 and 4 hours) the links appear and everything is fine.
Same problem here and it is much more serious since I have updated from JIRA 6.2.2 to 6.4.8 and ported my Script Runner scripts from Python to Groovy.
Adding issueLinkManager.clearCache() to my post functions does not really help. Is there any better workaround for the problem?
Hi chuckv,
The issue is currently being worked on by one of the members of my team and we are making good progress on it so far. We hope to be able to give you an update soon.
Regards,
Oswaldo Hernández.
JIRA Bugmaster.
[Atlassian].
Hi ingo.kampe,
Unfortunately this is not possible. All bugfix work is currently done in JIRA 7.0.x (the current maintenance line for JIRA) and the code changes to address this issue were extensive.
We encourage all customers to upgrade to JIRA 7.0.x as soon as they are able to so that they can receive the latest bugfixes we are working on.
Regards,
Oswaldo Hernández.
JIRA Bugmaster.
[Atlassian].