Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-75889

User cache replication sometimes fails due to the class loader being overridden

    XMLWordPrintable

Details

    Description

      Issue Summary

      Regression of JRASERVER-62071: Installed plugins initializing Data Center cache with wrong class loader (fixed in Jira Server and Data Center 7.3.4, 7.2.9)

      This means that when a user is created on Node 1 and has to be replicated on Node 2 or more, it may fail. Due to the wrong class loader being initialised.

      This is reproducible on Data Center: (yes)

      Steps to Reproduce

      1. Create multi-node DC instance
      2. Find / build a plug-in that starts a thread in an event/module event listener. (Example user lookup)
      3. Create a user on Node1
      4. Observe logs on Node2 for the cache replication event. It may sometimes fail due to the above plugin's class loader being loaded at startup.

      Expected Results

      Cache replication should always load the Jira Core class loader stack

      Actual Results

      Inconsistent class loader being initialized.

      The below exception is thrown in the atlassian-jira.log file:

      2023-03-22 05:18:13,240+0100 localq-reader-73 ERROR      [c.a.j.c.distribution.localq.LocalQCacheOpReader] [LOCALQ] [VIA-COPY] Abandoning sending: LocalQCacheOp{cacheName='com.atlassian.jira.crowd.embedded.ofbiz.EagerOfBizUserCache.userCache', action=PUT, key={10000,bug181krw}, value == null ? false, replicatePutsViaCopy=true, creationTimeInMillis=1679458693200} from cache replication queue: [queueId=queue_node4_3_9e22b2ee283109ab44b3ddeb56f9ed7a_put, queuePath=/var/atlassian/application-data/jira/localq/queue_node4_3_9e22b2ee283109ab44b3ddeb56f9ed7a_put], failuresCount: 1/1. Removing from queue. Error: java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: 
              java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: 
              java.lang.ClassNotFoundException: net.sf.ehcache.Element not found by com.javahollic.jira.jemh-ui [234] (no security manager: RMI class loader disabled)
      com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$UnrecoverableFailure: java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: 
              java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: 
              java.lang.ClassNotFoundException: net.sf.ehcache.Element not found by com.javahollic.jira.jemh-ui [234] (no security manager: RMI class loader disabled)
              at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:90)
              at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:96)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      Caused by: java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: 
              java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: 
              java.lang.ClassNotFoundException: net.sf.ehcache.Element not found by com.javahollic.jira.jemh-ui [234] (no security manager: RMI class loader disabled)
      

      Workaround

      • Disable the plugin flagged in the logs.
      • Restart the node without the plugin.
      • Re-enabled the plugin to restore the class loader.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              4bbe53f6c66f Louis (Inactive)
              Votes:
              7 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated: