Details
-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
8.20.12
-
8.2
-
2
-
Severity 2 - Major
-
1
-
Description
Issue Summary
This is reproducible on Data Center: (yes) / (no)
Concurrent calls to delete issues with Large number of Issuelinks causes DB contention and blocked/blocking SQLs are seen at Database level, This leads to DB performance and thus all the HTTP threads gets Stuck at Database causing outage.
Steps to Reproduce
- Create large number of issues, more than 2000~ PA-1 to PA-2000
- link above 2k issues to any other Issue PA-2001 (to be deleted) in same or different project.
- Attempt to delete the issue PA-2001. The UI will show that the operation did not complete, but the operation continues in the backend.
Additional information
During the Delete operation with linked issue, 2 threads are active:
- Reading information from the jiraissue table:
java.net.SocketInputStream.socketRead0(java.base@11.0.15/Native Method) java.net.SocketInputStream.socketRead(java.base@11.0.15/SocketInputStream.java:115) java.net.SocketInputStream.read(java.base@11.0.15/SocketInputStream.java:168) ... com.atlassian.jira.ofbiz.DefaultOfBizDelegator.findByPrimaryKey(DefaultOfBizDelegator.java:335) com.atlassian.jira.ofbiz.DefaultOfBizDelegator.findById(DefaultOfBizDelegator.java:327) com.atlassian.jira.ofbiz.WrappingOfBizDelegator.findById(WrappingOfBizDelegator.java:192) com.atlassian.jira.issue.managers.DefaultIssueManager.getIssue(DefaultIssueManager.java:143) com.atlassian.jira.issue.managers.DefaultIssueManager.getIssueObject(DefaultIssueManager.java:323)
- Deleting issuelinks:
java.net.SocketInputStream.socketRead0(java.base@11.0.15/Native Method) java.net.SocketInputStream.socketRead(java.base@11.0.15/SocketInputStream.java:115) java.net.SocketInputStream.read(java.base@11.0.15/SocketInputStream.java:168) ... com.atlassian.jira.diagnostic.connection.DiagnosticPreparedStatement.executeUpdate(DiagnosticPreparedStatement.java:69) org.ofbiz.core.entity.jdbc.SQLProcessor.executeUpdate(SQLProcessor.java:562) org.ofbiz.core.entity.GenericDAO.deleteByAnd(GenericDAO.java:1285) org.ofbiz.core.entity.GenericDAO.deleteByAnd(GenericDAO.java:1252) org.ofbiz.core.entity.GenericHelperDAO.removeByAnd(GenericHelperDAO.java:236) org.ofbiz.core.entity.GenericDelegator.removeByAnd(GenericDelegator.java:1338) org.ofbiz.core.entity.GenericDelegator.removeByAnd(GenericDelegator.java:1317) com.atlassian.jira.ofbiz.DefaultOfBizDelegator.removeByAnd(DefaultOfBizDelegator.java:236) com.atlassian.jira.ofbiz.WrappingOfBizDelegator.removeByAnd(WrappingOfBizDelegator.java:131) com.atlassian.jira.issue.link.DefaultIssueLinkManager.removeIssueLinkInternal(DefaultIssueLinkManager.java:182) com.atlassian.jira.issue.link.DefaultIssueLinkManager.deleteIssueLinksFromIssue(DefaultIssueLinkManager.java:267)
From SQL logs, we can see that between every single issuelink delete, Jira runs a select pointing to a specific issue ID (which I believe is every linked issue):
2022-10-18 09:19:49,507-0700 https-jsse-nio-8443-exec-4 url: /secure/DeleteIssue.jspa; user: ad_XXXXX ad_XXXXX 503x19687478x4 97cy0g /secure/DeleteIssue.jspa 0ms "SELECT pkey, issuenum, PROJECT, REPORTER, ASSIGNEE, CREATOR, issuetype, SUMMARY, DESCRIPTION, ENVIRONMENT, PRIORITY, RESOLUTION, issuestatus, CREATED, UPDATED, DUEDATE, RESOLUTIONDATE, VOTES, WATCHES, TIMEORIGINALESTIMATE, TIMEESTIMATE, TIMESPENT, WORKFLOW_ID, SECURITY, FIXFOR, COMPONENT, ARCHIVED, ARCHIVEDBY, ARCHIVEDDATE FROM dbo.jiraissue WHERE ID='3166192'" 2022-10-18 09:19:49,509-0700 https-jsse-nio-8443-exec-4 url: /secure/DeleteIssue.jspa; user: ad_XXXXX ad_XXXXX 503x19687478x4 97cy0g /secure/DeleteIssue.jspa 2ms "SELECT pkey, issuenum, PROJECT, REPORTER, ASSIGNEE, CREATOR, issuetype, SUMMARY, DESCRIPTION, ENVIRONMENT, PRIORITY, RESOLUTION, issuestatus, CREATED, UPDATED, DUEDATE, RESOLUTIONDATE, VOTES, WATCHES, TIMEORIGINALESTIMATE, TIMEESTIMATE, TIMESPENT, WORKFLOW_ID, SECURITY, FIXFOR, COMPONENT, ARCHIVED, ARCHIVEDBY, ARCHIVEDDATE FROM dbo.jiraissue WHERE ID='3166191'" 2022-10-18 09:19:49,511-0700 https-jsse-nio-8443-exec-4 url: /secure/DeleteIssue.jspa; user: ad_XXXXX ad_XXXXX 503x19687478x4 97cy0g /secure/DeleteIssue.jspa 2ms "SELECT pkey, issuenum, PROJECT, REPORTER, ASSIGNEE, CREATOR, issuetype, SUMMARY, DESCRIPTION, ENVIRONMENT, PRIORITY, RESOLUTION, issuestatus, CREATED, UPDATED, DUEDATE, RESOLUTIONDATE, VOTES, WATCHES, TIMEORIGINALESTIMATE, TIMEESTIMATE, TIMESPENT, WORKFLOW_ID, SECURITY, FIXFOR, COMPONENT, ARCHIVED, ARCHIVEDBY, ARCHIVEDDATE FROM dbo.jiraissue WHERE ID='3166190'" 2022-10-18 09:19:49,511-0700 https-jsse-nio-8443-exec-4 url: /secure/DeleteIssue.jspa; user: ad_XXXXX ad_XXXXX 503x19687478x4 97cy0g /secure/DeleteIssue.jspa 0ms "SELECT pkey, issuenum, PROJECT, REPORTER, ASSIGNEE, CREATOR, issuetype, SUMMARY, DESCRIPTION, ENVIRONMENT, PRIORITY, RESOLUTION, issuestatus, CREATED, UPDATED, DUEDATE, RESOLUTIONDATE, VOTES, WATCHES, TIMEORIGINALESTIMATE, TIMEESTIMATE, TIMESPENT, WORKFLOW_ID, SECURITY, FIXFOR, COMPONENT, ARCHIVED, ARCHIVEDBY, ARCHIVEDDATE FROM dbo.jiraissue WHERE ID='3166189'" 2022-10-18 09:19:49,513-0700 https-jsse-nio-8443-exec-4 url: /secure/DeleteIssue.jspa; user: ad_XXXXX ad_XXXXX 503x19687478x4 97cy0g /secure/DeleteIssue.jspa 1ms "SELECT pkey, issuenum, PROJECT, REPORTER, ASSIGNEE, CREATOR, issuetype, SUMMARY, DESCRIPTION, ENVIRONMENT, PRIORITY, RESOLUTION, issuestatus, CREATED, UPDATED, DUEDATE, RESOLUTIONDATE, VOTES, WATCHES, TIMEORIGINALESTIMATE, TIMEESTIMATE, TIMESPENT, WORKFLOW_ID, SECURITY, FIXFOR, COMPONENT, ARCHIVED, ARCHIVEDBY, ARCHIVEDDATE FROM dbo.jiraissue WHERE ID='3166188'" 2022-10-18 09:19:49,514-0700 https-jsse-nio-8443-exec-4 url: /secure/DeleteIssue.jspa; user: ad_XXXXX ad_XXXXX 503x19687478x4 97cy0g /secure/DeleteIssue.jspa 1ms "SELECT pkey, issuenum, PROJECT, REPORTER, ASSIGNEE, CREATOR, issuetype, SUMMARY, DESCRIPTION, ENVIRONMENT, PRIORITY, RESOLUTION, issuestatus, CREATED, UPDATED, DUEDATE, RESOLUTIONDATE, VOTES, WATCHES, TIMEORIGINALESTIMATE, TIMEESTIMATE, TIMESPENT, WORKFLOW_ID, SECURITY, FIXFOR, COMPONENT, ARCHIVED, ARCHIVEDBY, ARCHIVEDDATE FROM dbo.jiraissue WHERE ID='3166187'" 2022-10-18 09:19:49,515-0700 https-jsse-nio-8443-exec-4 url: /secure/DeleteIssue.jspa; user: ad_XXXXX ad_XXXXX 503x19687478x4 97cy0g /secure/DeleteIssue.jspa 1ms "SELECT pkey, issuenum, PROJECT, REPORTER, ASSIGNEE, CREATOR, issuetype, SUMMARY, DESCRIPTION, ENVIRONMENT, PRIORITY, RESOLUTION, issuestatus, CREATED, UPDATED, DUEDATE, RESOLUTIONDATE, VOTES, WATCHES, TIMEORIGINALESTIMATE, TIMEESTIMATE, TIMESPENT, WORKFLOW_ID, SECURITY, FIXFOR, COMPONENT, ARCHIVED, ARCHIVEDBY, ARCHIVEDDATE FROM dbo.jiraissue WHERE ID='3166186'"
All queries run extremely fast, but the sheer amount of queries required makes it so a linked issue record is removed every 5 seconds:
2022-10-18 09:19:50,503-0700 https-jsse-nio-8443-exec-4 url: /secure/DeleteIssue.jspa; user: ad_XXXXX ad_XXXXX 503x19687478x4 97cy0g /secure/DeleteIssue.jspa 1ms "DELETE FROM dbo.issuelink WHERE ID='3272625'" ... 2022-10-18 09:19:55,672-0700 https-jsse-nio-8443-exec-4 url: /secure/DeleteIssue.jspa; user: ad_XXXXX ad_XXXXX 503x19687478x4 97cy0g /secure/DeleteIssue.jspa 1ms "DELETE FROM dbo.issuelink WHERE ID='3272626'"
Since this is repeated for every issue linked, the operations ends up taking several hours. If multiple deletes happen at the same time, we'll see thread exhaustion due to Contention at DB side.
Expected Results
- Deletion of Issue with Large number of issue link should not be expensive and not cause DB contention.
Actual Results
- Deletion of issue is expensive as it iterates all the issue linked issue and performs cleanup and re-indexing operations.
Workaround
None at this time of writing
Attachments
Issue Links
- is related to
-
-
- Gathering Impact
-