Issue Summary

      We notice that out of 140 sync requests from GitHub through webhook, only 6 at the max is runnable. The rest of them have contention on a cluster lock.

      "http-nio-8081-exec-102 url:/rest/bitbucket...ository/22436/sync" #1465 daemon prio=5 os_prio=0 tid=0x00007ef11c0ad000 nid=0x172e waiting on condition [0x00007ee92d031000]
       java.lang.Thread.State: TIMED_WAITING (sleeping)
       at java.lang.Thread.sleep(Native Method)
       at com.atlassian.beehive.db.DatabaseClusterLock.sleep(DatabaseClusterLock.java:621)
       at com.atlassian.beehive.db.DatabaseClusterLock.uninterruptibleWait(DatabaseClusterLock.java:127)
       at com.atlassian.beehive.db.DatabaseClusterLock.lock(DatabaseClusterLock.java:107)
       ...
       at com.atlassian.jira.plugins.dvcs.service.MessageExecutor.tryToProcessNextMessage(MessageExecutor.java:185)
       at com.atlassian.jira.plugins.dvcs.service.MessageExecutor$$Lambda$6105/2048706363.accept(Unknown Source)
      ...
        

      Workaround

      None yet

      Next steps

      Investigate replacing / removing cluster lock.

        1. 2021-08-30_09-29-26.png
          2021-08-30_09-29-26.png
          2.13 MB
        2. 2021-08-30_09-07-08.png
          2021-08-30_09-07-08.png
          443 kB
        3. 2021-08-26_14-37-53.png
          2021-08-26_14-37-53.png
          2.37 MB
        4. 2021-08-26_14-30-31.png
          2021-08-26_14-30-31.png
          132 kB

            [JSWSERVER-20933] DVCS syncing process blocked waiting on cluster lock

            Ignat (Inactive) made changes -
            Remote Link New: This issue links to "Page (Confluence)" [ 647575 ]
            Cathy S made changes -
            Remote Link New: This issue links to "Page (Confluence)" [ 615504 ]
            Ignat (Inactive) made changes -
            Remote Link New: This issue links to "Page (Confluence)" [ 602909 ]
            Andriy Yakovlev [Atlassian] made changes -
            Fix Version/s New: 8.13.14 [ 97813 ]
            Murakami made changes -
            Remote Link New: This issue links to "Page (Confluence)" [ 599873 ]
            Ignat (Inactive) made changes -
            Resolution New: Fixed [ 1 ]
            Status Original: Waiting for Release [ 12075 ] New: Closed [ 6 ]
            Bugfix Automation Bot made changes -
            Support reference count Original: 15 New: 14
            SET Analytics Bot made changes -
            UIS Original: 59 New: 62
            Bugfix Automation Bot made changes -
            Support reference count Original: 14 New: 15
            Jeferson Martinelli (Inactive) made changes -
            Affects Version/s New: 8.13.13 [ 97814 ]

              szarazinski Sławomir Zaraziński
              rarmstrong@atlassian.com Rory Armstrong (Inactive)
              Affected customers:
              8 This affects my team
              Watchers:
              27 Start watching this issue

                Created:
                Updated:
                Resolved: