Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-75891

regression - Installed plugins are initializing Data Center EagerOfBizUserCache with wrong class loader

    XMLWordPrintable

Details

    Description

      Problem

      In a JIRA Data Center Environment, ClassNotFoundException can be seen in the logs due to an installed plugin thread initializing JIRA Data Center cache with its own classloader. The error was fixed previously in JRASERVER-62071 - Installed plugins initializing Data Center cache with wrong class loader but resurfaced in recent 9.X releases
       

      2023-03-23 05:50:45,791+0100 localq-reader-74 WARN      [c.a.j.c.distribution.localq.LocalQCacheOpReader] [LOCALQ] [VIA-COPY] Runtime exception: UnrecoverableFailure occurred when processing: LocalQCacheOp{cacheName='com.atlassian.jira.crowd.embedded.ofbiz.EagerOfBizUse
      rCache.userCache', action=PUT, key={10000,YY_XXX}, value == null ? false, replicatePutsViaCopy=true, creationTimeInMillis=1679547045779} from cache replication queue: [queueId=queue_node4_4_9e22b2ee283109ab44b3ddeb56f9ed7a_put, queuePath=/var/atlassian/application-data/
      jira/localq/queue_node4_4_9e22b2ee283109ab44b3ddeb56f9ed7a_put], failuresCount: 1/1, error: java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: 
              java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: 
              java.lang.ClassNotFoundException: net.sf.ehcache.Element not found by com.package.XXXX [234] (no security manager: RMI class loader disabled)
      2023-03-23 05:50:45,791+0100 localq-reader-74 ERROR      [c.a.j.c.distribution.localq.LocalQCacheOpReader] [LOCALQ] [VIA-COPY] Abandoning sending: LocalQCacheOp{cacheName='com.atlassian.jira.crowd.embedded.ofbiz.EagerOfBizUserCache.userCache', action=PUT, key={10000,u_m
      oze}, value == null ? false, replicatePutsViaCopy=true, creationTimeInMillis=1679547045779} from cache replication queue: [queueId=queue_node4_4_9e22b2ee283109ab44b3ddeb56f9ed7a_put, queuePath=/var/atlassian/application-data/jira/localq/queue_node4_4_9e22b2ee283109ab44b
      .
      .
      .
      Caused by: java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: 
              java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: 
              java.lang.ClassNotFoundException: net.sf.ehcache.Element not found by com.package.XXX [234] (no security manager: RMI class loader disabled)
              at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:389)
              at sun.rmi.transport.Transport$1.run(Transport.java:200)
              at sun.rmi.transport.Transport$1.run(Transport.java:197)
              at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
              at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
              at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
              at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
              at java.security.AccessController.doPrivileged(Native Method)
              at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
              at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303)
              at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279)
              at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
              at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227)
              at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179)
              at com.sun.proxy.$Proxy33.put(Unknown Source)
              at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.lambda$send$2(LocalQCacheOpRMISender.java:69)
              at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.withCachePeer(CachingRMICachePeerManager.java:94)
              at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:65)
              ... 6 more
       

       

      Environment

      Jira Data Center 9.X
      .

      Steps to Reproduce

      • Create a user from a third-party plugin code that runs for example in a Caesium thread , Jira will run getClass().getClassloader()and use it in the RMI call , which will be failing for not finding that class

      Expected Results

      Jira uses its own classloader and cache replication finishes correctly

      Actual Results

      Jira fails in a race condition and uses the wrong classloader , causing the RMI invocation to fail at the destination node

      Workaround

      A rolling restart of all Jira's nodes

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              f0ea184c9b1c Mohamed Kouki
              Votes:
              3 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated: