Uploaded image for project: 'JIRA Service Desk Server'
  1. JIRA Service Desk Server
  2. JSDSERVER-5436

Retrieving Issue SLAs can Causes Contention with Cluster Lock

    XMLWordPrintable

    Details

      Description

      Problem

      Due to a fix in JSDSERVER-5299 to stop race conditions, SLA locks were added. There has been some unexpected behaviours such as:

      • Locks may become congested due to high load causing in IO failures. The result is that locks may not be released, thus causing related tickets to become unresponsive.
      • Re-indexing results in update locks for each SLA per ticket causing high contention and drastically slowing down indexing speeds.
      Diagnosis

      Issues may be unresponsive and

      • The cluster health check may show messages like this:
        "Node 'i-0fb543049bcd09a2e' has been holding cluster lock, 'sla_issue_update_232939', for 20,550 seconds." appears
        
      • Thread Dumps will show:
        http-nio-8080-exec-42" #1390 daemon prio=5 os_prio=0 tid=0x00007fa92c888000 nid=0x1b9a waiting on condition [0x00007fa77613b000]
        java.lang.Thread.State: TIMED_WAITING (sleeping)
        	at java.lang.Thread.sleep(Native Method)
        	at com.atlassian.beehive.db.DatabaseClusterLock.sleep(DatabaseClusterLock.java:530)
        	at com.atlassian.beehive.db.DatabaseClusterLock.uninterruptibleWait(DatabaseClusterLock.java:102)
        	at com.atlassian.beehive.db.DatabaseClusterLock.lock(DatabaseClusterLock.java:82)
        	at com.atlassian.servicedesk.internal.sla.customfield.SlaFieldUpdateLockManagerImpl.lockSlaUpdate(SlaFieldUpdateLockManagerImpl.java:21)
        	at com.atlassian.servicedesk.internal.sla.customfield.SLACFType.getValueFromIssue(SLACFType.java:434)
        	at com.atlassian.servicedesk.internal.sla.customfield.SLACFType.getValueFromIssue(SLACFType.java:88)
        	at com.atlassian.jira.issue.customfields.impl.AbstractSingleFieldType.getValueFromIssue(AbstractSingleFieldType.java:71)
        	at com.atlassian.servicedesk.internal.sla.customfield.SLAFieldManagerImpl.getOptionalFieldValue(SLAFieldManagerImpl.java:191)
        	at com.atlassian.servicedesk.internal.sla.searcher.SLACustomFieldIndexer.addDocumentFields(SLACustomFieldIndexer.java:160)
        	at com.atlassian.servicedesk.internal.sla.searcher.SLACustomFieldIndexer.lambda$addDocumentFieldsSearchable$0(SLACustomFieldIndexer.java:150)
        	at com.atlassian.servicedesk.internal.sla.searcher.SLACustomFieldIndexer$$Lambda$1101/585067346.run(Unknown Source)
        	at com.atlassian.servicedesk.internal.util.SafeRunner.lambda$run$0(SafeRunner.java:40)
        	at com.atlassian.servicedesk.internal.util.SafeRunner$$Lambda$1102/840344782.call(Unknown Source)
        	at com.atlassian.servicedesk.bootstrap.lifecycle.LifecycleLock.lambda$withPluginLifecycleSafety$0(LifecycleLock.java:51)
        	at com.atlassian.servicedesk.bootstrap.lifecycle.LifecycleLock$$Lambda$1103/2107628499.call(Unknown Source)
        	at com.atlassian.ozymandias.SafePluginPointAccess.call(SafePluginPointAccess.java:263)
        	at com.atlassian.servicedesk.internal.util.SafeRunner.run(SafeRunner.java:44)
        	at com.atlassian.servicedesk.internal.sla.searcher.SLACustomFieldIndexer.addDocumentFieldsSearchable(SLACustomFieldIndexer.java:150)
        	at com.atlassian.jira.issue.index.indexers.impl.AbstractCustomFieldIndexer.addIndex(AbstractCustomFieldIndexer.java:40)
        	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory$Builder.add(DefaultIssueDocumentFactory.java:84)
        	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory$Builder.addAll(DefaultIssueDocumentFactory.java:75)
        	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory.apply(DefaultIssueDocumentFactory.java:50)
        	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory.apply(DefaultIssueDocumentFactory.java:30)
        	at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy.get(DefaultIssueIndexer.java:556)
        	at com.atlassian.jira.issue.index.DefaultIssueIndexer.lambda$reindexIssues$1(DefaultIssueIndexer.java:166)
        	at com.atlassian.jira.issue.index.DefaultIssueIndexer$$Lambda$1082/646135629.perform(Unknown Source)
        	at com.atlassian.jira.issue.index.DefaultIssueIndexer.lambda$null$2(DefaultIssueIndexer.java:308)
        	at com.atlassian.jira.issue.index.DefaultIssueIndexer$$Lambda$1084/999436899.get(Unknown Source)
        	at com.atlassian.jira.index.SimpleIndexingStrategy.get(SimpleIndexingStrategy.java:7)
        	at com.atlassian.jira.index.SimpleIndexingStrategy.get(SimpleIndexingStrategy.java:5)
        	at com.atlassian.jira.issue.index.DefaultIssueIndexer.lambda$perform$3(DefaultIssueIndexer.java:306)
        	at com.atlassian.jira.issue.index.DefaultIssueIndexer$$Lambda$1083/620300249.consume(Unknown Source)
        	at com.atlassian.jira.util.collect.CollectionUtil.foreach(CollectionUtil.java:39)
        	at com.atlassian.jira.util.collect.CollectionUtil.foreach(CollectionUtil.java:52)
        	at com.atlassian.jira.issue.util.IssueObjectIssuesIterable.foreach(IssueObjectIssuesIterable.java:24)
        	at com.atlassian.jira.issue.index.DefaultIssueIndexer.perform(DefaultIssueIndexer.java:282)
        	at com.atlassian.jira.issue.index.DefaultIssueIndexer.reindexIssues(DefaultIssueIndexer.java:162)
        	at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:571)
        	at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:547)
        	at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:530)
        	at com.atlassian.jira.issue.index.DefaultIndexManager.release(DefaultIndexManager.java:519)
        	at sun.reflect.GeneratedMethodAccessor2224.invoke(Unknown Source)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        	at java.lang.reflect.Method.invoke(Method.java:498)
        	at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:22)
        	at com.sun.proxy.$Proxy34.release(Unknown Source)
        	at com.atlassian.jira.workflow.OSWorkflowManager.enableIndexingForThisThread(OSWorkflowManager.java:914)
        	at com.atlassian.jira.workflow.OSWorkflowManager.doWorkflowAction(OSWorkflowManager.java:790)
        	at com.atlassian.jira.bc.issue.DefaultIssueService.transition(DefaultIssueService.java:492)
        	at com.atlassian.jira.web.action.workflow.SimpleWorkflowAction.doExecute(SimpleWorkflowAction.java:28)
        	at webwork.action.ActionSupport.execute(ActionSupport.java:165)
        

      Performing a re-index will have noticeably slower speeds and thread dumps will show that indexing threads are constantly performing cluster locks

      "IssueIndexer:thread-7" #353 prio=5 os_prio=0 tid=0x00007fcb200f6000 nid=0x1362a runnable [0x00007fcc643f9000]
         java.lang.Thread.State: RUNNABLE
      	at java.net.SocketInputStream.socketRead0(Native Method)
      	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      	at java.net.SocketInputStream.read(SocketInputStream.java:171)
      	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
      	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
      	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
      	- locked <0x00000004078afec0> (a java.lang.Object)
      	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
      	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
      	- locked <0x00000004078b2778> (a sun.security.ssl.AppInputStream)
      	at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3008)
      	...
      	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98)
      	at org.apache.commons.dbcp2.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:98)
      	at org.ofbiz.core.entity.jdbc.SQLProcessor.executeUpdate(SQLProcessor.java:562)
      	at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.bulkUpdateByAnd(DefaultOfBizDelegator.java:565)
      	...
      	at com.atlassian.jira.cluster.lock.JiraClusterLockDao.tryUpdateAcquireLock(JiraClusterLockDao.java:49)
      	at com.atlassian.beehive.db.DatabaseClusterLock.tryLockUsingDatabase(DatabaseClusterLock.java:281)
      	at com.atlassian.beehive.db.DatabaseClusterLock.tryLock(DatabaseClusterLock.java:171)
      	at com.atlassian.beehive.db.DatabaseClusterLock.lock(DatabaseClusterLock.java:77)
      	at com.atlassian.servicedesk.internal.sla.customfield.SlaFieldUpdateLockManagerImpl.lockSlaUpdate(SlaFieldUpdateLockManagerImpl.java:21)
      	at com.atlassian.servicedesk.internal.sla.customfield.SLACFType.getValueFromIssue(SLACFType.java:434)
      	at com.atlassian.servicedesk.internal.sla.customfield.SLACFType.getValueFromIssue(SLACFType.java:88)
      	at com.atlassian.jira.issue.customfields.impl.AbstractSingleFieldType.getValueFromIssue(AbstractSingleFieldType.java:71)
      	at com.atlassian.servicedesk.internal.sla.customfield.SLAFieldManagerImpl.getOptionalFieldValue(SLAFieldManagerImpl.java:191)
      	...
      	at com.atlassian.jira.issue.index.DefaultIssueIndexer$$Lambda$1207/1229406744.get(Unknown Source)
      	at com.atlassian.jira.index.SimpleIndexingStrategy.get(SimpleIndexingStrategy.java:7)
      	at com.atlassian.jira.index.SimpleIndexingStrategy.get(SimpleIndexingStrategy.java:5)
      	at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call(MultiThreadedIndexingStrategy.java:33)
      	at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call(MultiThreadedIndexingStrategy.java:31)
      	...
      
      Workarounds

      None

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              szaporozhtsev Sergey Zaporozhtsev (Inactive)
              Reporter:
              dchan David Chan
              Votes:
              1 Vote for this issue
              Watchers:
              12 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: