-
Bug
-
Resolution: Fixed
-
Highest
-
2.10.2
-
None
-
jdk6u12 (32bit and 64bit), mysql5, war, cluster - but there is only one node in the cluster
In the last several weeks we've been seeing a lot of confluence instabilities at wikis.sun.com - all of them were related to running out of heap space. Several iterations of increasing Xmx didn't help (we started at 3GB and now we are at 5GB and 64bit JVM).
I took several memory dumps during outages and analyzed them with Eclipse Memory Analyzer, which repeatedly found two issues:
- SAXParser memory leaks
- hundreds of instances of net.sf.hibernate.impl.SessionImpl retaining 750MB+ heap memory
At the same time I see that the dump contains total of 163 threads. Given that Confluence uses OpenSessionInView pattern, I'd expect to see less than 163 live session instances (since not all the threads are j2ee service threads).
I'm attaching some annotated screenshots from Eclipse Memory Analyzer.
- relates to
-
-
- Closed
-
[CONFSERVER-14989] Possible net.sf.hibernate.impl.SessionImpl Memory Leaks
Igor Minar
added a comment - Andrew, is the patch being included in 3.1? Igor Minar
added a comment - Thanks Andrew. We'd really appreciate to have this resolved soon and stabilize our site.
Igor Minar
added a comment - Thanks Andrew. We'd really appreciate to have this resolved soon and stabilize our site. Hi Igor,
I will be investigating the feasibility of a fix of this issue for 3.1.
I don't believe CONF-12319 is really a related issue, although it may exacerbate this issue.
Regards,
Andrew Lynch
Igor Minar
added a comment - can you please add CONF-12319 as a related issue? thnx Igor Minar
added a comment - Some more info from the heap dump:
Each user object (com.atlassian.user.impl.hibernate.DefaultHibernateUser) was present in the dump five times. Thanks to CONF-12319 and our 120+k users, that results in 600+k user objects on the heap
All these user objects were retained on the heap due to references from hibernate collections.
We have the user cache set to 130k, so due to CONF-12319, I'd expect all of the user objects to be on the heap, but only once. It seems that there are some big inefficiencies going on here.
Igor Minar
added a comment - Some more info from the heap dump:
Each user object (com.atlassian.user.impl.hibernate.DefaultHibernateUser) was present in the dump five times. Thanks to CONF-12319 and our 120+k users, that results in 600+k user objects on the heap
All these user objects were retained on the heap due to references from hibernate collections.
We have the user cache set to 130k, so due to CONF-12319 , I'd expect all of the user objects to be on the heap, but only once. It seems that there are some big inefficiencies going on here. Hi guys,
Any plans to get this fixed soon? We ran out of memory twice in the last week and we already use 5.5GB heap and Conf 3.0.1. I'm just examining the heap dumps and was able to confirm that the problems were caused by this particular bug.
thanks
Igor Minar
added a comment - - edited Hi guys,
Any plans to get this fixed soon? We ran out of memory twice in the last week and we already use 5.5GB heap and Conf 3.0.1. I'm just examining the heap dumps and was able to confirm that the problems were caused by this particular bug.
thanks I'm in favour of both option #1 and option #3. That is, we should patch Hibernate to clean up its session when it is closed and we should not store persistent objects in the app server session.
To implement the latter, Seraph should allow the application to provide the object which is to be stored in the session. Confluence can provide a Principal which is a DefaultUser or something else not attached to the database session.
Igor Minar
added a comment - 1) or 3) sound good.
With 3) there is still a risk that the detached object will hold on to a large object tree though.
Igor Minar
added a comment - 1) or 3) sound good.
With 3) there is still a risk that the detached object will hold on to a large object tree though. 
thanks