Uploaded image for project: 'Jira Software Data Center'
  1. Jira Software Data Center
  2. JSWSERVER-20933

DVCS syncing process blocked waiting on cluster lock

XMLWordPrintable

      Issue Summary

      We notice that out of 140 sync requests from GitHub through webhook, only 6 at the max is runnable. The rest of them have contention on a cluster lock.

      "http-nio-8081-exec-102 url:/rest/bitbucket...ository/22436/sync" #1465 daemon prio=5 os_prio=0 tid=0x00007ef11c0ad000 nid=0x172e waiting on condition [0x00007ee92d031000]
 java.lang.Thread.State: TIMED_WAITING (sleeping)
 at java.lang.Thread.sleep(Native Method)
 at com.atlassian.beehive.db.DatabaseClusterLock.sleep(DatabaseClusterLock.java:621)
 at com.atlassian.beehive.db.DatabaseClusterLock.uninterruptibleWait(DatabaseClusterLock.java:127)
 at com.atlassian.beehive.db.DatabaseClusterLock.lock(DatabaseClusterLock.java:107)
 ...
 at com.atlassian.jira.plugins.dvcs.service.MessageExecutor.tryToProcessNextMessage(MessageExecutor.java:185)
 at com.atlassian.jira.plugins.dvcs.service.MessageExecutor$$Lambda$6105/2048706363.accept(Unknown Source)
...
  

      Workaround

      None yet

      Next steps

      Investigate replacing / removing cluster lock.

        1. 2021-08-26_14-30-31.png
          2021-08-26_14-30-31.png
          132 kB
        2. 2021-08-26_14-37-53.png
          2021-08-26_14-37-53.png
          2.37 MB
        3. 2021-08-30_09-07-08.png
          2021-08-30_09-07-08.png
          443 kB
        4. 2021-08-30_09-29-26.png
          2021-08-30_09-29-26.png
          2.13 MB

          Form Name

            [JSWSERVER-20933] DVCS syncing process blocked waiting on cluster lock

            Edgar Garcia added a comment -

            Is this issue happening to Jira Server edition? 

            Edgar Garcia added a comment - Is this issue happening to Jira Server edition? 

            Murakami added a comment -

            Backport done to 8.13.14

            Murakami added a comment - Backport done to 8.13.14

            Alex Gallien added a comment -

            This bug seems to have totally taken down one of our Jira DC nodes today, was totally frozen with over 250 threads stuck waiting for a lock.

            Alex Gallien added a comment - This bug seems to have totally taken down one of our Jira DC nodes today, was totally frozen with over 250 threads stuck waiting for a lock.

            Aaron Matthys added a comment -

            This bug makes our DVCS integration to Gitlab.com completely useless. I feel the severity should be major instead of minor. 

            Aaron Matthys added a comment - This bug makes our DVCS integration to Gitlab.com completely useless. I feel the severity should be major instead of minor. 

              szarazinski Sławomir Zaraziński
              rarmstrong@atlassian.com Rory Armstrong (Inactive)
              Affected customers:
              8 This affects my team
              Watchers:
              27 Start watching this issue

                Created:
                Updated:
                Resolved: