Hi Atlassian team.

As I let you know in a support ticket (https://support.atlassian.com/servicedesk/agent/JSP/issue/JSP-249425), I've been struggling with a strange problem on the plugin I was developing for Jira.
Sometimes, not always, and not for all plugins, when I uninstall and reinstall a plugin (or reload it via FastDev), plugin usage raises the following exception :

Caused by: org.springframework.osgi.service.importer.ServiceProxyDestroyedException: service proxy has been destroyed
[INFO] [talledLocalContainer] 	at org.springframework.osgi.service.importer.support.internal.aop.ServiceDynamicInterceptor$ServiceLookUpCallback.doWithRetry(ServiceDynamicInterceptor.java:105)
[INFO] [talledLocalContainer] 	at org.springframework.osgi.service.importer.support.internal.support.RetryTemplate.execute(RetryTemplate.java:83)
[INFO] [talledLocalContainer] 	at org.springframework.osgi.service.importer.support.internal.aop.ServiceDynamicInterceptor.lookupService(ServiceDynamicInterceptor.java:430)
[INFO] [talledLocalContainer] 	at org.springframework.osgi.service.importer.support.internal.aop.ServiceDynamicInterceptor.getTarget(ServiceDynamicInterceptor.java:415)
[INFO] [talledLocalContainer] 	at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:62)
[INFO] [talledLocalContainer] 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
[INFO] [talledLocalContainer] 	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
[INFO] [talledLocalContainer] 	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
[INFO] [talledLocalContainer] 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
[INFO] [talledLocalContainer] 	at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:56)
[INFO] [talledLocalContainer] 	at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:39)
[INFO] [talledLocalContainer] 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
[INFO] [talledLocalContainer] 	at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
[INFO] [talledLocalContainer] 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
[INFO] [talledLocalContainer] 	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
[INFO] [talledLocalContainer] 	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
[INFO] [talledLocalContainer] 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
[INFO] [talledLocalContainer] 	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
[INFO] [talledLocalContainer] 	at com.sun.proxy.$Proxy3232.find(Unknown Source)
[INFO] [talledLocalContainer] 	at com.ovyka.jira.testplugin.dao.impl.ContextDAOImpl.getContextsForProjectAndIssueType(ContextDAOImpl.java:87)

At first I thought it was coming from the fact I was using Spring autowiring of beans inside my plugin, but declaring them as component (not shared) didn't change anything.

I have investigated more and finally found the cause (and solution) of the problem.

I ran a very deep debug session, going down into Spring Dynamic Modules, Spring Framework and Jira code. I found something really weird about the bundle context (context class of OSGi bundles, which is what Jira plugin really are) : the bundle being called when the ServiceProxyDestroyed exception was raised was NOT the currently deployed plugin version....

To illustrate this, I put some debug points in Felix (OSGi framework implementation used by Jira) BundleContextImpl class, in the invalidate() method (called when a plugin is uninstalled), and in the constructor (called when the plugin is installed). I also place a breakpoint in Spring Dynamic Modules ServiceDynamicInterceptor class, which is were the ServiceProxyDestroyed exception is raised. At each of these breakpoints, I took a "snapshot" of the bundle context for the bundle being used, and here is what came of it :

During uninstallation of a plugin, in BundleContextImpl.invalidate() method, the bundle context is as follows :

{org.apache.felix.framework.BundleContextImpl@41676} 
 m_logger = {com.atlassian.plugin.osgi.container.felix.FelixLoggerBridge@38134} 
 m_felix = {org.apache.felix.framework.Felix@38135} "org.apache.felix.framework [0]"
 m_bundle = {org.apache.felix.framework.BundleImpl@41677} "com.ovyka.jira.tesplugin [180]"
 m_valid = false

Note the OSGi id of the bundle com.ovyka.jira.tesplugin : 180

When installing the new version of the bundle, here is what I got in the BundleContextImpl constructor :
 {org.apache.felix.framework.BundleContextImpl@41793} 
 m_logger = {com.atlassian.plugin.osgi.container.felix.FelixLoggerBridge@38134} 
 m_felix = {org.apache.felix.framework.Felix@38135} "org.apache.felix.framework [0]"
 m_bundle = {org.apache.felix.framework.BundleImpl@41794} "com.ovyka.jira.tesplugin [181]"
 m_valid = true

Note the OSGi id of the testplugin plugin is different : 181.

Now if i try to use a feature of my plugin, the ServiceProxyDestroyed exception raising breakpoint is hit, and the associated bundle context is :

 {org.apache.felix.framework.BundleContextImpl@41970} 
 m_logger = {com.atlassian.plugin.osgi.container.felix.FelixLoggerBridge@38134} 
 m_felix = {org.apache.felix.framework.Felix@38135} "org.apache.felix.framework [0]"
 m_bundle = {org.apache.felix.framework.BundleImpl@41985} "com.ovyka.jira.tesplugin [173]"
 m_valid = false

Note that the OSGi id is 173 (so not the uninstalled one, nore the newly installed one), and the bundle context is marked as invalid (m_valid = false).

What I concluded from this is that Jira was keeping a reference to every version of the plugin, and was calling one of them, wether or not it was the latest one.
Actually, after a little more digging, I found out it was not totally exact.
In fact, Jira DOES keep a reference to all the version of the plugin, but does not call one of them : it calls ALL of them.

When, after reaching the first ServiceProxyDestroyed raising breakpoint I let the application continue its work, the breakpoint was hit once again, with a different bundle context (180 in the example above).

After that, it is not called anymore, but the plugin feature work is done. Adding a breakpoint on the called plugin method that Jira was trying to execute, I found out it was actually called, with a different bundle context (181 in the example above), which is why no exception is raised, and the work is done.

So here is the updated conclusion :
Jira keeps a reference to all versions of a bundle (after multiple install/reinstall), and when the plugin is called, Jira calls all of the bundles, one by one.

• For all the inactive versions, a ServiceProxyDestroyed exception is raised (sometimes also ClassNotFound or NoClassDefFound, for classes that ARE in Jira provided classpath (Jira services references by plugins)).
• For the active version, the plugin does its work, without any exception.
  So this exception is not blocking, as it does not prevent the work of the plugin to be done, but may be very dangerous, because if the plugin does not require any OSGi reference, no exception will be raised, and the work may done several times...

Now what was left to do was to know why all bundles were kept by Jira (increasing the amount of memory used, by the way). I climbed back the call stack from the ServiceProxyDestroyed raising point, and found out the source of the call was an IssueEventListener. After a small test, I found out the reason of the problem, and of course its solution.

The reason why Jira keeps a reference to all bundle versions is that, when a plugin adds an EventListener to jira, it has to register it into the EventPublisher : eventPublisher.register(this);
This is not a process linked to OSGi. It is done "programatically", and none of the OSGi bundle uninstalling processes being aware of this link, the listener is NEVER unregistered from the EventPlublisher, if the developer does not unregister it : it stays in the list of listeners registered in Jira.

That explains why my plugin was called multiple times when I was doing an action triggering an IssueEvent. The EventPublisher was calling all of the registered IssueEventListeners, including the ones registered by removed plugins. And this explains also the ServiceProxyDestroyed exception, because when a bundle version is uninstalled, the ServiceProxy provided by Spring Dynamic Modules are actually destroyed, so all the services injected in the uninstalled plugin now are obsolete and linked to nothing.

The solution to this problem (actually one of the solutions) is very simple, and is given in your tutorial page (https://developer.atlassian.com/jiradev/jira-platform/guides/other/tutorial-writing-jira-event-listeners-with-the-atlassian-event-library) : Make the listener implement DisposableBean and implement the destroy() method with the line : eventPublisher.unregister(this);

I think you should emphasize the fact that this is very important, because given the number of pages about this ServiceProxyDestroyed exception around the net, I am not the only one to have missed this point

Now another solution to this problem would be to avoid asking developers to register their listener into the EventPublisher.

A very efficient way to do this would be to implement the OSGi White Board pattern. When developing a listener, the developer would just have to declare it in the atlassian-plugin.xml (for example, declare it using a <listener class=""/> tag, as it is done for rest endpoints), and Jira would just have to publish it as an OSGi service, with restricted access to Jira core only. Now the EventPublisher, instead of maintaining a list of registered listeners, which as we saw does not work well with the dynamic nature of OSGi bundles, would just ask for all services published under a specific marker interface, for example PluginJiraListener (just an example).
OSGi framework would then give it all services implementing this interface, which are available at the time the call is made by the EventPublisher.

This would require a modification of jira core or Jira plugin management plugin, but would avoid this problem and would be a much nicer way to deal with dynamic plugins.
And isn't it how this is done in Confluence ?

Would you consider doing this modification in Jira system ?