Issue Summary

This is reproducible on Data Center: yes

When mail notifications cannot be delivered, they are accumulated in the mail ErrorQueue causing eventually an OutOfMemory situation. 

Steps to Reproduce

1. Create 4-5 users in Confluence, providing a valid mail address
2. Login as each one of the user created, browse to User Profile > Settings > Emails and enable "Subscribe to daily updates"
3. Login as a Confluence administrator and browse to General Configuration > Mail server.
   • Add a new SMTP Mail Server
   • Validate that you can send a test mail to each of the users mails created earlier, and the mail is received in their inbox. 
4. Create ~5000 new empty pages in Confluence, this will generate a mail about "daily updates" of approximately ~10MBs each
5. Browse to General Configuration > Scheduled Jobs, search Email Daily Reports and click on Edit to set a new cron schedule.
   • For example "0 * * * * ?" will run this job every single minute.  
6. Confirm that the users receive a mail on their inbox with a subject "Confluence changes in the last 24 hours" 
7. Now, shutdown the mail server so the mails can't be delivered and they are moved to the Mail ErrorQueue. 

Expected Results

Confluence keeps running as usual, without impact. The mails on the ErrorQueue are not kept permanently loaded in the memory.

Actual Results

Eventually, if enough mails get accumulated in the ErrorQueue, due to their size or the amount of mails (or a combination of both), an OutOfMemoryError is thrown in the atlassian-confluence.log file:

2023-06-07 16:00:07,837 ERROR [Caesium-1-1] [scheduler.caesium.impl.SchedulerQueueWorker] executeJob Unhandled exception thrown by job QueuedJob[jobId=DailyReportJob,deadline=1686153580000]
java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.Arrays.copyOfRange(Arrays.java:4030)
        at java.base/java.lang.StringUTF16.newString(StringUTF16.java:1025)
        at java.base/java.lang.StringBuilder.toString(StringBuilder.java:453)
        at org.jsoup.internal.StringUtil.releaseBuilder(StringUtil.java:367)
        at org.jsoup.nodes.Element.html(Element.java:1721)
        at org.jsoup.nodes.Document.outerHtml(Document.java:277)
        at com.atlassian.botocss.Botocss.inject(Botocss.java:145)
        at com.atlassian.confluence.mail.BotocssMailContentProcessor.process(BotocssMailContentProcessor.java:54)
        at com.atlassian.confluence.mail.DefaultMailContentProcessor.process(DefaultMailContentProcessor.java:21)
        at com.atlassian.confluence.mail.template.AbstractMailNotificationQueueItem.transformForEmail(AbstractMailNotificationQueueItem.java:256)
        at com.atlassian.confluence.mail.template.AbstractMailNotificationQueueItem.preRenderBody(AbstractMailNotificationQueueItem.java:200)
        at com.atlassian.confluence.mail.template.PreRenderedMailNotificationQueueItem$Builder.render(PreRenderedMailNotificationQueueItem.java:393)
        at com.atlassian.confluence.mail.jobs.DefaultDailyReportManager.queueDailyReportEmail(DefaultDailyReportManager.java:146)
        at com.atlassian.confluence.mail.jobs.DefaultDailyReportManager.generateDailyReports(DefaultDailyReportManager.java:110)
...
...

If collected a HeapDump, you will observe there is one instance of com.atlassian.core.task.ErrorQueuedTaskQueue occupying most part of the memory.

If you analyse further, you will see there are dozens/hundreds of objects of type
com.atlassian.confluence.mail.template.PreRenderedMailNotificationQueueItem that are holding all the data of the non-delivered mails.

Workaround

Workaround 1)

Fix the issue related in the SMTP Mail Server that is causing the mail notifications not being delivered successfully.

As an example, this might be so simple as the mail server is temporarily down, or the mail size is higher that the max message size configured in the Mail server side. 
Workaround 2)

Monitor regularly the ErrorQueue in the Confluence Mail Queue and delete manually every time that there are mails accumulated.
Workaround 3)

Restart Confluence