Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-70522

Updating custom field value during issue deletion can cause stuck threads

XMLWordPrintable

      Issue Summary

      Any update actions on custom fields value during the deletion process of the same issue can cause that thread to be stuck on the database. This could cause the delete transaction and thread to run indefinitely or delete operation will be rolledback, this can also cause deadlocks any type of databases.
      The issue is therefore never deleted and eventually, you will get a timeout message in the GUI.

      Steps to Reproduce

      1. Install CPrime Power Script Plugin - the script is necessary to trigger custom field update action,
      2. Create new "Number Field" custom field,
      3. Create SIL scripts which edit new custom field:
      4. Create synchronous SIL listener on Issue Link Delete Event
      5. Create issue A and B,
      6. Set on both the custom field value,
      7. Link issue A to B - A blocks B
      8. Remove issue A.

      SIL Script example

      customfield_10108=1;

       This can occur when using any scripted function that attempts to update the parent issue that is deleted, Script Runner etc. 

      Expected Results

      Issue A is deleted.

      Actual Results

      Issue A persist

      Before database connections are terminated:
      Two stuck connections on a database, originating from the same transaction

      Thread dump with similar stack, if you have thready installed you may see the /url:/secure/DeleteIssue.jspa endpoint for each of the following stacks as three separate threads: 

      java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
java.net.SocketInputStream.read(SocketInputStream.java:171)
java.net.SocketInputStream.read(SocketInputStream.java:141)
org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:140)
org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:109)
org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:67)
org.postgresql.core.PGStream.receiveChar(PGStream.java:280)
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1916)
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288)
org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430)
org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356)
org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:168)
org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:135)
org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98)
org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98)
com.querydsl.sql.dml.SQLDeleteClause.execute(SQLDeleteClause.java:208)
com.atlassian.jira.issue.managers.CachingCustomFieldManager.lambda$removeCustomFieldValues$10(CachingCustomFieldManager.java:459)
com.atlassian.jira.issue.managers.CachingCustomFieldManager$$Lambda$1809/794030538.run(Unknown Source)
com.atlassian.jira.database.DefaultQueryDslAccessor$1.lambda$execute$1(DefaultQueryDslAccessor.java:91)
com.atlassian.jira.database.DefaultQueryDslAccessor$1$$Lambda$1810/716640284.run(Unknown Source)
com.atlassian.jira.database.DatabaseAccessorImpl.executeQuery(DatabaseAccessorImpl.java:68)
com.atlassian.jira.database.DefaultQueryDslAccessor$1.execute(DefaultQueryDslAccessor.java:90)
com.atlassian.jira.issue.managers.CachingCustomFieldManager.removeCustomFieldValues(CachingCustomFieldManager.java:456)
com.atlassian.jira.issue.managers.DefaultIssueDeleteHelper.deleteIssue(DefaultIssueDeleteHelper.java:141)
com.atlassian.jira.issue.managers.DefaultIssueManager.deleteIssue(DefaultIssueManager.java:723)
com.atlassian.jira.issue.managers.RequestCachingIssueManager.deleteIssue(RequestCachingIssueManager.java:232)
com.atlassian.jira.bc.issue.DefaultIssueService.delete(DefaultIssueService.java:402)
com.atlassian.jira.bc.issue.DefaultIssueService.delete(DefaultIssueService.java:378)
com.atlassian.jira.web.action.issue.DeleteIssue.doExecute(DeleteIssue.java:23)

       java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:140) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:109) at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:67) at org.postgresql.core.PGStream.receiveChar(PGStream.java:280) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1916) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288) - locked <0x00007fe0c1c113b8> (a org.postgresql.core.v3.QueryExecutorImpl) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:168) at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:135) at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98) at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98) at org.ofbiz.core.entity.jdbc.SQLProcessor.executeUpdate(SQLProcessor.java:562) at org.ofbiz.core.entity.GenericDAO.deleteByAnd(GenericDAO.java:1281) at org.ofbiz.core.entity.GenericDAO.deleteByAnd(GenericDAO.java:1248) at org.ofbiz.core.entity.GenericHelperDAO.removeByAnd(GenericHelperDAO.java:236) at org.ofbiz.core.entity.GenericDelegator.removeByAnd(GenericDelegator.java:1334) at org.ofbiz.core.entity.GenericDelegator.removeByAnd(GenericDelegator.java:1313) at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.removeByAnd(DefaultOfBizDelegator.java:236) at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.removeByAnd(WrappingOfBizDelegator.java:131) at com.atlassian.jira.issue.link.DefaultIssueLinkManager.removeIssueLinkInternal(DefaultIssueLinkManager.java:176) at com.atlassian.jira.issue.link.DefaultIssueLinkManager.removeIssueLink(DefaultIssueLinkManager.java:160) at com.atlassian.jira.bc.issue.link.DefaultIssueLinkService.delete(DefaultIssueLinkService.java:200) at com.atlassian.jira.web.action.issue.DeleteLink.doExecute(DeleteLink.java:35) 
      java.lang.Thread.State: RUNNABLEjava.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:140) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:109) at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:67) at org.postgresql.core.PGStream.receiveChar(PGStream.java:280) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1916) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:288) - locked <0x00007fe0bec08078> (a org.postgresql.core.v3.QueryExecutorImpl) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:430) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:356) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:168) at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:135) at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98) at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98) at com.querydsl.sql.dml.SQLDeleteClause.execute(SQLDeleteClause.java:208) at com.atlassian.jira.issue.managers.CachingCustomFieldManager.lambda$removeCustomFieldValues$10(CachingCustomFieldManager.java:459) at com.atlassian.jira.issue.managers.CachingCustomFieldManager$$Lambda$4537/764526941.run(Unknown Source) at com.atlassian.jira.database.DefaultQueryDslAccessor$1.lambda$execute$1(DefaultQueryDslAccessor.java:91) at com.atlassian.jira.database.DefaultQueryDslAccessor$1$$Lambda$4538/156444641.run(Unknown Source) at com.atlassian.jira.database.DatabaseAccessorImpl.executeQuery(DatabaseAccessorImpl.java:68)

      After database connections are being terminated:
      Exception in the logs:

      2019-12-16 16:36:43,853 http-nio-8090-exec-20 ERROR      [o.a.c.c.C.[.[localhost].[/jira].[action]] Servlet.service() for servlet [action] in context with path [/jira] threw exception [com.querydsl.core.QueryException: Caught PSQLException for delete from public.customfieldvalue
    where customfieldvalue.issue = ?] with root cause
org.postgresql.util.PSQLException: FATAL: terminating connection due to administrator command
  Where: while deleting tuple (8,13) in relation "customfieldvalue"

      You'll also see locks on the following tables in the DB: 

      userassociation

      customfieldvalue

      issuelink

      Workaround

      Until this bug is resolved, it may be a good idea to not listen on issue link deleted events 

        1. 6.png
          6.png
          49 kB

              szarazinski Sławomir Zaraziński
              ddudziak Stasiu
              Votes:
              26 Vote for this issue
              Watchers:
              40 Start watching this issue

                Created:
                Updated:
                Resolved: