Igor,

You mentioned that you only started noticing the problem in the last several weeks. Has anything been changed on the instance that might have triggered the problem? Any confluence upgrades, plugin upgrades, new plugin installed? The additional info might help us to narrow down the search for the bug.

Anatoli.

Hi Anatoli,

It's hard to tell why this started happening only recently. That's because our traffic grows quite rapidly (min 10% each month, a lot more during spikes).

There were two recent changes:

• upgrade to confluence 2.10.2
• expansion of the coherence caches

I know, I know, the cache must be culprit! That's what I thought for a few weeks and kept on increasing the heap size. Then when I started analyzing the heap dumps I found out that the cache consumes less then 1GB of heap and a lot more space is consumed by the SAXParser and SessionImpl objects.

The first thing that I started to suspect when looking at the dumps was that a reference to the hibernate session instance was stored in the http session. That would explain why there were more of these instances in the heap than the actual number of concurrent connections that were being processed.

Our session expiration policy is set to 8 hours, which means that it can take a while for abandoned sessions to be gc-ed. This is not a problem if the session doesn't contain a lot of data, but could lead to problems similar to what we've been experiencing if there are bulky objects stored in the http session.

So I think I'd start by checking what's being stored in the http session.

Hi Igor,

As far as I can tell from your heap dumps, the reason that the SessionImpls cannot be collected is because you have references to User objects (DefaultHibernateUser) stored within your HttpSession.
Seraph is placing these DefaultHibernateUser objects within the HttpSession, and unfortunately they contain references to SessionImpls (which can be grow quite large if a significant amount of work is performed on them, and unfortunately this is not cleared entirely when the Session is closed).

Regards,
Andrew Lynch

The code in question (in DefaultAuthenticator) :

final Principal user = getUser(username);
...
request.getSession().setAttribute(LOGGED_IN_KEY, user);

We could fix this by :
1) Properly nulling out the maps in the SessionImpl when it is closed (this would also fix the same symptons in CONF-10575 and USER-228 ).
2) Not store this in the HttpSession (not sure if this is a real a option)
3) Only storing detached objects in the HttpSession.

1) or 3) sound good.

With 3) there is still a risk that the detached object will hold on to a large object tree though.

I'm in favour of both option #1 and option #3. That is, we should patch Hibernate to clean up its session when it is closed and we should not store persistent objects in the app server session.

To implement the latter, Seraph should allow the application to provide the object which is to be stored in the session. Confluence can provide a Principal which is a DefaultUser or something else not attached to the database session.

Hi guys,

Any plans to get this fixed soon? We ran out of memory twice in the last week and we already use 5.5GB heap and Conf 3.0.1. I'm just examining the heap dumps and was able to confirm that the problems were caused by this particular bug.

thanks

Some more info from the heap dump:

Each user object (com.atlassian.user.impl.hibernate.DefaultHibernateUser) was present in the dump five times. Thanks to CONF-12319 and our 120+k users, that results in 600+k user objects on the heap

All these user objects were retained on the heap due to references from hibernate collections.

We have the user cache set to 130k, so due to CONF-12319, I'd expect all of the user objects to be on the heap, but only once. It seems that there are some big inefficiencies going on here.

can you please add CONF-12319 as a related issue? thnx

Hi Igor,

I will be investigating the feasibility of a fix of this issue for 3.1.
I don't believe CONF-12319 is really a related issue, although it may exacerbate this issue.

Regards,
Andrew Lynch

Thanks Andrew. We'd really appreciate to have this resolved soon and stabilize our site.

Andrew, is the patch being included in 3.1?

Hi Igor,

It will be in 3.1.1.

Regards,
Andrew Lynch

thanks