Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-66557

ClusterMessageHandlerServiceThread can stop checking messages if Throwable is encountered

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Low Low
    • None
    • 7.2.12, 7.6.2
    • Data Center - Other

      Summary

      ClusterMessageHandlerServiceThread is capable of entering a state where it is no longer checking messages. This can cause inconsistency between JIRA Data Center Nodes.

      Environment

      • JIRA Data Center

      Expected Results

      • ClusterMessageHandlerServiceThread always checks messages every three seconds.

      Expected stacktrace while waiting:

      "ClusterMessageHandlerServiceThread:thread-1" #109 prio=5 os_prio=0 tid=0x00007ff544016800 nid=0x7ee7 waiting on condition [0x00007ff508581000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000d18db240> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

      Actual Results

      • ClusterMessageHandlerServiceThread enters a state where it no longer checks messages every three seconds.
        • Thread is no longer in TIMED_WAITING and is stuck in WAITING

      Unexpected stacktrace. No longer checking messages:

      • Affected threads would show this stacktrace and be stuck in this state without changing.
      "ClusterMessageHandlerServiceThread:thread-1" prio=5 tid=0x000000000000006c nid=0 waiting on condition 
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for <0x0000000020f2b558> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

      Notes

      • Developer review "found that the ClusterMessageHandlerServiceThread can go into WAITING state if any of the ClusterMessage consumers throw a Throwable exception."

      Workaround

      • No workaround is available.
      • ClusterMessageHandlerServiceThread that is no longer checking messages requires node to be restarted.

              Unassigned Unassigned
              tevans Tim Evans (Inactive)
              Votes:
              3 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated: