Installed plugins initializing Data Center cache with wrong class loader

Summary

In a JIRA Data Center Environment, ClassNotFoundException can be seen in the logs due to an installed plugin thread initializing JIRA Data Center cache with its own classloader.

Environment

• JIRA Data Center
• Reproduced with JIRA Data Center 7.1.x with Structure Plugin 3.1

Steps to Reproduce

1. Setup a JIRA Data Center with at least 2 nodes.
2. Find / build a plug-in which starts a thread in a constructor / plugin event / module event listener.
3. That thread inside of the 3rd party plug-in / custom made plugin should be the first who initialises the cache. For example the thread can call com.atlassian.jira.user.ApplicationUsers.byKey to cause cache initialisation.
4.  NOT FOR PRODUCTION USE. You can use the attached jira-cache-accessor-plugin.jar as a demo how to init the cache with the wrong classloader.
5. Observe the logs during JIRA startup. Alternative - see the logs on cache synchronization.

Expected Results

JIRA webapp classloader is used to initialize and access the cache, and there are no errors with replicating the cache to other nodes

Actual Results

1. Cache is initialized with the plugin's class loader, rather than the correct class loader for the code. 
    In the sample plug-in we can see that  see now which class loader used to init the EagerOfBizUserCache. I.e. initialized with class loader: BundleDelegatingClassLoader. To see that I've modified EagerOfBizUserCache to log which class loader is used to initialize the cache.
2. Errors similar to the following are thrown in the application logs during cache replication to other nodes. Note that this errors will be show at "healthy node" and actual problem is at other node(s).

   2016-05-26 12:03:12,218 http-nio-8080-exec-8 ERROR anonymous 723x169744x1 1fhqh5i 10.223.41.99,10.233.128.11 /rest/gadget/1.0/login [n.s.ehcache.distribution.RMISynchronousCacheReplicator] Exception on replication of removeNotification. RemoteException occurred in server thread; nested exception is: 
       java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: 
       java.lang.ClassNotFoundException: com.atlassian.jira.crowd.embedded.ofbiz.DirectoryEntityKey not found by com.almworks.jira.structure [18] (no security manager: RMI class loader disabled). Continuing...
   java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: 
   java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: 
   java.lang.ClassNotFoundException: com.atlassian.jira.crowd.embedded.ofbiz.DirectoryEntityKey not found by com.almworks.jira.structure [18] (no security manager: RMI class loader disabled)
   at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:354)
   

   or

   2016-08-01 16:44:50,516 Caesium-1-4 ERROR      [n.s.ehcache.distribution.RMISynchronousCacheReplicator] Exception on replication of putNotification. RemoteException occurred in server thread; nested exception is: 
       	java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: 
       	java.lang.ClassNotFoundException: net.sf.ehcache.Element not found by com.almworks.jira.structure [17] (no security manager: RMI class loader disabled). Continuing...
   java.rmi.ServerException: RemoteException occurred in server thread; nested exception is: 
   	java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is: 
   Caused by: java.lang.ClassNotFoundException: net.sf.ehcache.Element not found by com.almworks.jira.structure [17] (no security manager: RMI class loader disabled)
   	at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:396)
   

Workaround (for plugin developers)

• Plugin developers may consider wrapping such calls with a class loader switching code, to ensure that the right class loader is used. For example, in a case of calling ApplicationUser's static methods, the class loader switch may be as easy as:

  ClassLoader previous = Thread.currentThread().getContextClassLoader();
  try {
     Thread.currentThread().setContextClassLoader(ApplicationUsers.class.getClassLoader());
     ApplicationUsers.getKey() 
  } finally {
     Thread.currentThread().setContextClassLoader(previous);
  }
  

Workaround (for users)

1. Disable the plugin that triggered the cache problem: Structure or other
2. Restart node(s) not having mentioned error
3. Enable plugin again

Alternative Workaround (for users)

If the workaround above does not work on your case, try the following:

1. Disable the plugin triggering the cache problem (Structure, ScriptRunner or other).
2. Remove node1 from the load balancer and restart JIRA on that node.
3. When node1 is back up, re-add it to the load balancer and restart node2.
4. Repeat steps 2-3 for the other nodes.
5. After all nodes have been restarted, enable the plugin causing the issue again.

Structure

Almworks has released new version 3.3.0.jira7 which has the fix.