We consistently get OutOfMemoryErrors after running Confluence for several days. Java heapdump (ibm jdk feature) shows 149 PreRenderedMailNotificationQueueItems that sit in ErrorQueuedTaskQueue holding 100M+ of memory from garbage colection (heapdump file can be provided for your amusement). I've spent some time looking into this problem and here is what I found
First, how the PreRenderedMailNotificationQueueItems get to the error queue. Most of our users did not bother choosing between text and html email notifications, so their confluence.prefs.email.mimetype comes from preferences-default.xml and is equal to "html". Due to apparent typo in MailNotificationQueueItem#getMimeTypeForUser (MailNotificationQueueItem.java around line 159), string "html" is not considered to represent html content and is used as is by SMTPMailServerImpl. Since "html" is not know mime type, send message fails with javax.mail.ParseException. By the way, ErrorQueuedTaskQueue#handleException looses nested exceptions and stack trace and all I saw in the logs was almost useless "send message failed".
Second, why PreRenderedMailNotificationQueueItems hold so much RAM. Although I am not 100% sure but looking at the heapdump and the source code it appears that PreRenderedMailNotificationQueueItems.VelocityRenderedQueueItem#context holds whole bunch of different stuff, including http request, response and much more. Since PreRenderedMailNotificationQueueItem is already pre-rendered you may as well clean VelocityRenderedQueueItem#context and probably other members before queueing it for delivery. Another suggestion is to make ErrorQueuedTaskQueue weak – I would rather loose few failed mail notifications than my entire wiki server.
Hope this helps. I've made some changes to our local Confluence installation to help me debug this problem better and I will update this bug report if I find something new.