-
Bug
-
Resolution: Obsolete
-
Medium
-
2.3, 2.4, 2.5, 2.7, 2.8, 2.9, 3.0, 5.1.5
When a node of a cluster starts it first fires up a cluster service and then initializes plugin subsystem. It means that for a period of time between the start up of the cluster service and initialization of plugin sysbsystem ( ConfluencePluginManager ) cluster does not have access to plugin classes. If during this interval a distributed cache was updated with class from a plugin on a different node and update was received we get ClassNotFoundException
2008-07-18 14:20:52,358 ERROR [Logger@9247854 3.3.1/389] [Coherence] log 2008-07-18 14:20:52.332 Oracle Coherence GE 3.3.1/389 <Error> (thread=ReplicatedCache, member=3): java.io.IOException: readObject failed: java.lang.ClassNotFoundException: com.atlassian.confluence.extra.jira.CacheKey at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1362) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1208) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:242) at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:585) at com.tangosol.io.ResolvingObjectInputStream.resolveClass(ResolvingObjectInputStream.java:68) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1544) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1466) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1699) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1305) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348) at com.tangosol.util.ExternalizableHelper.readSerializable(ExternalizableHelper.java:2084) at com.tangosol.util.ExternalizableHelper.readObject(ExternalizableHelper.java:2202) at com.tangosol.coherence.component.net.Message.readObject(Message.CDB:3) at com.tangosol.coherence.component.util.CacheHandler.populateCache(CacheHandler.CDB:23) at com.tangosol.coherence.component.util.daemon.queueProcessor.service.ReplicatedCache$CacheUpdate.onReceived(ReplicatedCache.CDB:5) at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onMessage(Service.CDB:9) at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onNotify(Service.CDB:123) at com.tangosol.coherence.component.util.daemon.queueProcessor.service.ReplicatedCache.onNotify(ReplicatedCache.CDB:3) at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:35) at java.lang.Thread.run(Thread.java:613) ClassLoader: com.atlassian.plugin.classloader.DelegationClassLoader@9ed2e4 at com.tangosol.util.ExternalizableHelper.readSerializable(ExternalizableHelper.java:2092) at com.tangosol.util.ExternalizableHelper.readObject(ExternalizableHelper.java:2202) at com.tangosol.coherence.component.net.Message.readObject(Message.CDB:3) at com.tangosol.coherence.component.util.CacheHandler.populateCache(CacheHandler.CDB:23) at com.tangosol.coherence.component.util.daemon.queueProcessor.service.ReplicatedCache$CacheUpdate.onReceived(ReplicatedCache.CDB:5) at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onMessage(Service.CDB:9) at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onNotify(Service.CDB:123) at com.tangosol.coherence.component.util.daemon.queueProcessor.service.ReplicatedCache.onNotify(ReplicatedCache.CDB:3) at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:35) at java.lang.Thread.run(Thread.java:613)
Steps to reproduce:
1. start node1
2. access a page on node1 with a jira issues macro (wait until issues are loaded)
3. start node2 and observe the exception in the log file.
[CONFSERVER-12486] ClassNotFoundException logged on cluster node startup
Still occuring in confluence 3.0.0_01
==> /var/log/java/extranet.atlassian.com/sysout.log <==
2009-06-15 21:10:03,798 INFO [main] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 3.0.0_01 (build #1627)
2009-06-15 21:10:07,947 INFO [main] [confluence.cluster.tangosol.TangosolClusterManager] startCluster Bringing up cluster service
2009-06-15 21:10:35,837 ERROR [Logger@9267279 3.3.1/389] [Coherence] log 2009-06-15 21:10:35.835 Oracle Coherence GE 3.3.1/389 <Error> (thread=ReplicatedCache, member=2): Failed to deserialize a key for cache com.atlassian.confluence.extra.jira.JiraIssuesMacro
2009-06-15 21:10:35,845 ERROR [Logger@9267279 3.3.1/389] [Coherence] log 2009-06-15 21:10:35.836 Oracle Coherence GE 3.3.1/389 <Error> (thread=ReplicatedCache, member=2): An exception (java.io.IOException) occurred reading Message LeaseUpdate Type=9 for Service=ReplicatedCache{Name=ReplicatedCache, State=(SERVICE_STARTED), Id=2, Version=3.0, OldestMemberId=1}
2009-06-15 21:10:35,845 ERROR [Logger@9267279 3.3.1/389] [Coherence] log 2009-06-15 21:10:35.836 Oracle Coherence GE 3.3.1/389 <Error> (thread=ReplicatedCache, member=2): Terminating ReplicatedCache due to unhandled exception: java.io.IOException
2009-06-15 21:10:35,867 ERROR [Logger@9267279 3.3.1/389] [Coherence] log 2009-06-15 21:10:35.836 Oracle Coherence GE 3.3.1/389 <Error> (thread=ReplicatedCache, member=2):
java.io.IOException: readObject failed: java.lang.ClassNotFoundException: com.atlassian.confluence.extra.jira.cache.CacheKey
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1387)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1233)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:604)
at com.tangosol.io.ResolvingObjectInputStream.resolveClass(ResolvingObjectInputStream.java:68)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at com.tangosol.util.ExternalizableHelper.readSerializable(ExternalizableHelper.java:2084)
at com.tangosol.util.ExternalizableHelper.readObject(ExternalizableHelper.java:2202)
at com.tangosol.coherence.component.net.Message.readObject(Message.CDB:3)
at com.tangosol.coherence.component.net.message.LeaseMessage.read(LeaseMessage.CDB:11)
at com.tangosol.coherence.component.net.message.leaseMessage.ResourceMessage.read(ResourceMessage.CDB:5)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onNotify(Service.CDB:110)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.ReplicatedCache.onNotify(ReplicatedCache.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:35)
at java.lang.Thread.run(Thread.java:619)
ClassLoader: com.atlassian.confluence.util.ConfluenceUberClassLoader@117b450
at com.tangosol.util.ExternalizableHelper.readSerializable(ExternalizableHelper.java:2092)
at com.tangosol.util.ExternalizableHelper.readObject(ExternalizableHelper.java:2202)
at com.tangosol.coherence.component.net.Message.readObject(Message.CDB:3)
at com.tangosol.coherence.component.net.message.LeaseMessage.read(LeaseMessage.CDB:11)
at com.tangosol.coherence.component.net.message.leaseMessage.ResourceMessage.read(ResourceMessage.CDB:5)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onNotify(Service.CDB:110)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.ReplicatedCache.onNotify(ReplicatedCache.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:35)
at java.lang.Thread.run(Thread.java:619)
2009-06-15 21:10:40,780 ERROR [Logger@9267279 3.3.1/389] [Coherence] log 2009-06-15 21:10:40.780 Oracle Coherence GE 3.3.1/389 <Error> (thread=ReplicatedCache, member=2): An exception (java.io.IOException: readObject failed: java.lang.ClassNotFoundException: com.atlassian.confluence.extra.jira.cache.CacheKey
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1387)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1233)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:604)
at com.tangosol.io.ResolvingObjectInputStream.resolveClass(ResolvingObjectInputStream.java:68)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at com.tangosol.util.ExternalizableHelper.readSerializable(ExternalizableHelper.java:2084)
at com.tangosol.util.ExternalizableHelper.readObject(ExternalizableHelper.java:2202)
at com.tangosol.coherence.component.net.Message.readObject(Message.CDB:3)
at com.tangosol.coherence.component.util.CacheHandler.populateCache(CacheHandler.CDB:23)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.ReplicatedCache$CacheUpdate.onReceived(ReplicatedCache.CDB:5)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onMessage(Service.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onNotify(Service.CDB:123)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.ReplicatedCache.onNotify(ReplicatedCache.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:35)
at java.lang.Thread.run(Thread.java:619)
ClassLoader: com.atlassian.confluence.util.ConfluenceUberClassLoader@117b450) occurred while populating cache: CacheHandler{Name=com.atlassian.confluence.extra.jira.JiraIssuesMacro, Index=69, ServiceName=ReplicatedCache, ClassLoader=null}
2009-06-15 21:10:40,785 ERROR [Logger@9267279 3.3.1/389] [Coherence] log 2009-06-15 21:10:40.780 Oracle Coherence GE 3.3.1/389 <Error> (thread=ReplicatedCache, member=2):
java.io.IOException: readObject failed: java.lang.ClassNotFoundException: com.atlassian.confluence.extra.jira.cache.CacheKey
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1387)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1233)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:604)
at com.tangosol.io.ResolvingObjectInputStream.resolveClass(ResolvingObjectInputStream.java:68)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351)
at com.tangosol.util.ExternalizableHelper.readSerializable(ExternalizableHelper.java:2084)
at com.tangosol.util.ExternalizableHelper.readObject(ExternalizableHelper.java:2202)
at com.tangosol.coherence.component.net.Message.readObject(Message.CDB:3)
at com.tangosol.coherence.component.util.CacheHandler.populateCache(CacheHandler.CDB:23)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.ReplicatedCache$CacheUpdate.onReceived(ReplicatedCache.CDB:5)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onMessage(Service.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onNotify(Service.CDB:123)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.ReplicatedCache.onNotify(ReplicatedCache.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:35)
at java.lang.Thread.run(Thread.java:619)
ClassLoader: com.atlassian.confluence.util.ConfluenceUberClassLoader@117b450
at com.tangosol.util.ExternalizableHelper.readSerializable(ExternalizableHelper.java:2092)
at com.tangosol.util.ExternalizableHelper.readObject(ExternalizableHelper.java:2202)
at com.tangosol.coherence.component.net.Message.readObject(Message.CDB:3)
at com.tangosol.coherence.component.util.CacheHandler.populateCache(CacheHandler.CDB:23)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.ReplicatedCache$CacheUpdate.onReceived(ReplicatedCache.CDB:5)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onMessage(Service.CDB:9)
at com.tangosol.coherence.component.util.daemon.queueProcessor.Service.onNotify(Service.CDB:123)
at com.tangosol.coherence.component.util.daemon.queueProcessor.service.ReplicatedCache.onNotify(ReplicatedCache.CDB:3)
at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:35)
at java.lang.Thread.run(Thread.java:619)
Michael Breu
added a comment - Hi Andrew,
Indeed it turned out that is not a bug related to clustering, but rather a bug in the invite-Plugin. We have adapted it, to be cluster-friendly, by changing the Mail-API, it used.
The invite plugin has problems to read its configuration while the old configuration is still loaded by another class loader. This problem is solved now. We are in contact with Jens Schumacher to bring our changes back into the main development path.
Cheers
Michael
Michael Breu
added a comment - Hi Andrew,
Indeed it turned out that is not a bug related to clustering, but rather a bug in the invite-Plugin. We have adapted it, to be cluster-friendly, by changing the Mail-API, it used.
The invite plugin has problems to read its configuration while the old configuration is still loaded by another class loader. This problem is solved now. We are in contact with Jens Schumacher to bring our changes back into the main development path.
Cheers
Michael Hi Michael,
What exactly do you mean by 'forgets its configuration status'? Do you mean configuration values are reset to the defaults, and did this happen on the first node (that was working correctly) when the second node joins the cluster?
Regards,
Andrew Lynch
Michael Breu
added a comment - Hello,
I have a similar problem with an extension of the invite-Plugin to make it clusterable.
However it seems not merely a cosmetic problem. Rather it looks like that the invite plugin forgets its configuration status after the second node joins the cluster.
Michael Breu
added a comment - Hello,
I have a similar problem with an extension of the invite-Plugin to make it clusterable.
However it seems not merely a cosmetic problem. Rather it looks like that the invite plugin forgets its configuration status after the second node joins the cluster.
Jeremy,
I had a look at the log files in the support case you have pointed to. There are some differences between this exception and the exception customer is having.
This excepting happens when tangosol tries to de-serialize data it receives from the wire, i.e. the data another node sends. However in customer's case de-serialization happens when the data is put in cache :
2008-10-01 01:36:15,073 INFO [STDOUT] 2008-10-01 01:36:15,073 ERROR [Logger@19423616 3.3.1/389] [Coherence] log 2008-10-01 01:36:15.073 Oracle Coherence GE 3.3.1/389 <Error> (thread=ajp-0.0.0.0-6101-2, member=n/a): (Wrapped) java.io.IOException: readObject failed: java.lang.ClassNotFoundException: com.atlassian.confluence.extra.jira.CacheKey at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1358) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1204) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:604) at com.tangosol.io.ResolvingObjectInputStream.resolveClass(ResolvingObjectInputStream.java:68) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1575) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1496) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1329) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:351) at com.tangosol.util.ExternalizableHelper.readSerializable(ExternalizableHelper.java:2084) at com.tangosol.util.ExternalizableHelper.readObject(ExternalizableHelper.java:2202) at com.tangosol.util.ExternalizableHelper.fromBinary(ExternalizableHelper.java:205) at com.tangosol.util.ExternalizableHelper.fromBinary(ExternalizableHelper.java:188) at com.atlassian.confluence.cache.tangosol.ListenerTangosolCacheFactory$2.entryInserted(ListenerTangosolCacheFactory.java:49) at com.tangosol.util.MapEvent.dispatch(MapEvent.java:191) at com.tangosol.util.MapEvent.dispatch(MapEvent.java:164) at com.tangosol.util.MapListenerSupport.fireEvent(MapListenerSupport.java:556) at com.tangosol.coherence.component.util.SafeNamedCache.translateMapEvent(SafeNamedCache.CDB:7) at com.tangosol.coherence.component.util.SafeNamedCache.entryInserted(SafeNamedCache.CDB:1) at com.tangosol.util.MapEvent.dispatch(MapEvent.java:191) at com.tangosol.util.MapEvent.dispatch(MapEvent.java:164) at com.tangosol.util.MapListenerSupport.fireEvent(MapListenerSupport.java:556) at com.tangosol.coherence.component.util.LocalCache$CacheHandler.translateMapEvent(LocalCache.CDB:5) at com.tangosol.coherence.component.util.LocalCache$CacheHandler.entryInserted(LocalCache.CDB:1) at com.tangosol.util.MapEvent.dispatch(MapEvent.java:191) at com.tangosol.util.MapEvent.dispatch(MapEvent.java:164) at com.tangosol.util.MapListenerSupport.fireEvent(MapListenerSupport.java:556) at com.tangosol.net.cache.OldCache.dispatchEvent(OldCache.java:1716) at com.tangosol.net.cache.OldCache$Entry.onAdd(OldCache.java:1784) at com.tangosol.util.SafeHashMap.put(SafeHashMap.java:247) at com.tangosol.net.cache.OldCache.put(OldCache.java:253) at com.tangosol.net.cache.OldCache.put(OldCache.java:221)
It is very strange that tangosol needs to do serialization/de-serialization inside the same jvm.
Although in both cases the reason for the exception is a wrong classLoader that is used by Tangosol when de-serializing data, the cause that led to the exception is different.
Christopher Owen [Atlassian]
added a comment - It's not a 2.9 blocker. It's a bug that's been exposed since more plugins have moved out of web-inf and into the bundled plugins system. While it is ugly in the logs, it doesn't impact functionality.
Christopher Owen [Atlassian]
added a comment - It's not a 2.9 blocker. It's a bug that's been exposed since more plugins have moved out of web-inf and into the bundled plugins system. While it is ugly in the logs, it doesn't impact functionality. 
Per Fragemann [Atlassian]
added a comment - Hi, what is the status on this issue?
In answer to the question, would switching a Hazelcast cache suffer from the same bug:
Answer
When an object is put into a Hazelcast cache, it will be serialized to bytes, which are transferred to the master node and the backup node (assuming one configured). On these nodes, the object will be deserialized from the bytes.
Any request to get the object back will involve a request to the master node, which will serialize the master copy to bytes, which are transferred back to the requesting node, where the object will be deserialized from the bytes and returned to the caller.
Upshot is that Hazelcast will suffer from the same problem.