Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-73781

Jira deadlocks in PicoСontainer while starting-up and replaying events from RMI TCP Connection

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Low
    • None
    • 8.20.7
    • Data Center - Other
    • None

    Description

      Issue Summary

      This is a similar occurrence to JRASERVER-70816 where the localhost-startStop-1 and RMI TCP Connection threads are deadlocking during DC node startup.

      Any simultaneous calls to <T> T getComponent(final Class<T> componentType) and to <T> List<T> getComponents(final Class<T> componentType) in DefaultPicoContainer from parallel threads can lead to deadlock.
      Due to changes in how RMI works, it is possible that you may see multiple RMI threads in thread dumps.

      This is reproducible on Data Center: Yes (Only in multi-node instances).

      Steps to Reproduce

      Cannot be reproduced at will.

      Expected Results

      Jira starts up and is able to replicate its cache normally.

      Actual Results

      The below exception is thrown in the Tomcat logs file:

      2022-04-21 15:48:25,409-0400 DeadlockDetection:thread-1 ERROR      [c.a.jira.startup.LauncherContextListener] A deadlock has been detected on JIRA startup for the following threads: ["localhost-startStop-1" daemon prio=x Id=xx WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@217168b9 owned by "RMI TCP Connection(1)-xx.xxx.xx.xxx" Id=xx, "RMI TCP Connection(1)-xx.xxx.xx.xxx" daemon prio=x Id=xx BLOCKED on org.picocontainer.DefaultPicoContainer@2fb692e7 owned by "localhost-startStop-1" Id=xx]
      2022-04-21 15:48:25,413-0400 DeadlockDetection:thread-1 ERROR      [c.a.jira.startup.LauncherContextListener] "localhost-startStop-1" daemon prio=x Id=xx WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@217168b9 owned by "RMI TCP Connection(1)-xx.xxx.xx.xxx" Id=xx
          	at java.base@11.0.12/jdk.internal.misc.Unsafe.park(Native Method)
          	-  waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@217168b9
          	at java.base@11.0.12/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
          	at java.base@11.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
          	at java.base@11.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917)
          	at java.base@11.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240)
          	at java.base@11.0.12/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:267)
          	at org.picocontainer.behaviors.Locked.getComponentInstance(Locked.java:38)
          	at org.picocontainer.DefaultPicoContainer.getInstance(DefaultPicoContainer.java:698)
          	...
          
      2022-04-21 15:48:25,415-0400 DeadlockDetection:thread-1 ERROR      [c.a.jira.startup.LauncherContextListener] "RMI TCP Connection(1)-xx.xxx.xx.xxx" daemon prio=x Id=xx BLOCKED on org.picocontainer.DefaultPicoContainer@2fb692e7 owned by "localhost-startStop-1" Id=xx
          	at org.picocontainer.DefaultPicoContainer.getConverters(DefaultPicoContainer.java:1106)
          	-  blocked on org.picocontainer.DefaultPicoContainer@2fb692e7
          	at org.picocontainer.parameters.BasicComponentParameter.getConverters(BasicComponentParameter.java:129)
          	at org.picocontainer.parameters.BasicComponentParameter.access$000(BasicComponentParameter.java:51)
          	at org.picocontainer.parameters.BasicComponentParameter$1.resolveInstance(BasicComponentParameter.java:118)
          	at org.picocontainer.parameters.ComponentParameter$1.resolveInstance(ComponentParameter.java:136)
          	at org.picocontainer.injectors.SingleMemberInjector.getParameter(SingleMemberInjector.java:78)
          	at org.picocontainer.injectors.ConstructorInjector$CtorAndAdapters.getParameterArguments(ConstructorInjector.java:309)
          	at org.picocontainer.injectors.ConstructorInjector$1.run(ConstructorInjector.java:335)
      

      Workaround

      The below steps only need to be applied once, when affected by this problem.

      1. Shut down the affected node.
      2. Delete the contents from the '<Jira_home>/localq/' directory.
      3. Start the Jira node once again.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              abrancalhao@atlassian.com Armando Neto
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: