regression - Installed plugins are initializing Data Center EagerOfBizUserCache with wrong class loader

XMLWordPrintable

    • 9.04
    • 4
    • Severity 2 - Major
    • 2

      Problem

      In a JIRA Data Center Environment, ClassNotFoundException can be seen in the logs due to an installed plugin thread initializing JIRA Data Center cache with its own classloader. The error was fixed previously in JRASERVER-62071 - Installed plugins initializing Data Center cache with wrong class loader but resurfaced in recent 9.X releases
       

      2023-03-23 05:50:45,791+0100 localq-reader-74 WARN      [c.a.j.c.distribution.localq.LocalQCacheOpReader] [LOCALQ] [VIA-COPY] Runtime exception: UnrecoverableFailure occurred when processing: LocalQCacheOp{cacheName='com.atlassian.jira.crowd.embedded.ofbiz.EagerOfBizUse
rCache.userCache', action=PUT, key={10000,YY_XXX}, value == null ? false, replicatePutsViaCopy=true, creationTimeInMillis=1679547045779} from cache replication queue: [queueId=queue_node4_4_9e22b2ee283109ab44b3ddeb56f9ed7a_put, queuePath=/var/atlassian/application-data/
jira/localq/queue_node4_4_9e22b2ee283109ab44b3ddeb56f9ed7a_put], failuresCount: 1/1, error: java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: 
        java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: 
        java.lang.ClassNotFoundException: net.sf.ehcache.Element not found by com.package.XXXX [234] (no security manager: RMI class loader disabled)
2023-03-23 05:50:45,791+0100 localq-reader-74 ERROR      [c.a.j.c.distribution.localq.LocalQCacheOpReader] [LOCALQ] [VIA-COPY] Abandoning sending: LocalQCacheOp{cacheName='com.atlassian.jira.crowd.embedded.ofbiz.EagerOfBizUserCache.userCache', action=PUT, key={10000,u_m
oze}, value == null ? false, replicatePutsViaCopy=true, creationTimeInMillis=1679547045779} from cache replication queue: [queueId=queue_node4_4_9e22b2ee283109ab44b3ddeb56f9ed7a_put, queuePath=/var/atlassian/application-data/jira/localq/queue_node4_4_9e22b2ee283109ab44b
.
.
.
Caused by: java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: 
        java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: 
        java.lang.ClassNotFoundException: net.sf.ehcache.Element not found by com.package.XXX [234] (no security manager: RMI class loader disabled)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:389)
        at sun.rmi.transport.Transport$1.run(Transport.java:200)
        at sun.rmi.transport.Transport$1.run(Transport.java:197)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303)
        at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279)
        at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
        at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227)
        at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179)
        at com.sun.proxy.$Proxy33.put(Unknown Source)
        at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.lambda$send$2(LocalQCacheOpRMISender.java:69)
        at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.withCachePeer(CachingRMICachePeerManager.java:94)
        at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:65)
        ... 6 more

       

      Environment

      Jira Data Center 9.X
      .

      Steps to Reproduce

      • Create a user from a third-party plugin code that runs for example in a Caesium thread , Jira will run getClass().getClassloader()and use it in the RMI call , which will be failing for not finding that class

      Expected Results

      Jira uses its own classloader and cache replication finishes correctly

      Actual Results

      Jira fails in a race condition and uses the wrong classloader , causing the RMI invocation to fail at the destination node

      Workaround

      A rolling restart of all Jira's nodes

            Assignee:
            Unassigned
            Reporter:
            Mohamed Kouki (Inactive)
            Votes:
            4 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated: