Details
-
Bug
-
Resolution: Fixed
-
High
-
5.4.2, 5.7.1, 5.7.4, 5.8.4, 5.8.5, 5.8.17, 5.9.1-beta4, 5.9.12, 5.10.2
-
294
-
Severity 1 - Critical
-
Description
This issue consists of 2 underlying symptoms relating to the PDF Conversion Service where the Confluence system can become unresponsive or crash:
- Sustained high CPU usage
- Occasional OutOfMemoryError
Confluence Improvements:
The PDF Conversion Service is built on top of a 3rd party library (Aspose), requiring an ongoing Enterprise support case between Atlassian and the Aspose team. Outside of this dependency, we have independently investigated and implemented a number of changes in the way we utilise the Aspose library in order to address this issue. Thus far, we believe that we have resolved the sustained high CPU usage with the following improvements:
- The conversion service is executed by a group of threads, managed by a thread pool. The thread pool is now dynamically managed to be inline with available system resources including the current CPU usage, memory usage as well as the current system load. In other words, the system will not launch more conversions if the system is busy and additional resources are not available.
- The latest Aspose library has been implemented, including many improvements and bug fixes.
Remaining Issue:
The Aspose library continues to require a high amount of memory for the PDF conversion service, which means the system might still experience an OutOfMemoryError. This is more prevalent when converting the extra-large PDF files (e.g. file size is above 100MB), which could exceed the available heap memory. Once the OutOfMemoryError occurs, the system is likely to be slow to respond to user requests, and a Confluence restart is recommended.
Workaround for OutOfMemoryError:
- Reduce the system level maximum attachment size to <100MB
- Increase the heap memory for the Confluence instance if possible
Further Development
The Confluence development team is looking into additional strategies to tackle the potential memory issue.
Please refer to https://jira.atlassian.com/browse/CONF-43363 for any ongoing issues relating to the OutOfMemoryError issue
If you have any ongoing issues with respect to CPU spikes following upgrading to version 5.10.4+, please feel free to comment here or raise a support request.
Best regards,
Minh Tran
Confluence BugMaster
Atlassian
Summary
Symptoms:
- When adding large PDF or Office files to a Confluence page the attachment does not render as thumbnail on the page
- There is also possibility that the CPU usage will be spiking to 100% during the time and if it does
- The threads are not released until Confluence instance is restarted
Following appears in the thread dumps:
"conversion-thread-0-internal" #1530 daemon prio=1 os_prio=0 tid=0x0000000001184000 nid=0x8eb runnable [0x00007f67075f3000] java.lang.Thread.State: RUNNABLE at com.aspose.pdf.internal.p560.z13.m7(Unknown Source) at com.aspose.pdf.internal.p560.z13.m2(Unknown Source) at com.aspose.pdf.internal.p560.z13.m6(Unknown Source) at com.aspose.pdf.internal.p560.z13.m1(Unknown Source) at com.aspose.pdf.internal.p559.z1.m1(Unknown Source) at com.aspose.pdf.internal.p543.z4.m2(Unknown Source) - locked <0x00000000fcf6cc78> (a java.lang.Object) at com.aspose.pdf.internal.p502.z4.<init>(Unknown Source) at com.aspose.pdf.internal.p502.z3.<init>(Unknown Source) at com.aspose.pdf.internal.p502.z4.m1(Unknown Source) at com.aspose.pdf.internal.p525.z1.m2(Unknown Source) at com.aspose.pdf.internal.p525.z1.m1(Unknown Source) at com.aspose.pdf.internal.p525.z1.m1(Unknown Source) at com.aspose.pdf.internal.p525.z1.m1(Unknown Source) at com.aspose.pdf.internal.p525.z1.m1(Unknown Source) at com.aspose.pdf.internal.p512.z20.m2(Unknown Source) at com.aspose.pdf.internal.p512.z20.m4(Unknown Source) at com.aspose.pdf.internal.p512.z7.m2(Unknown Source) at com.aspose.pdf.internal.p512.z7.m1(Unknown Source) at com.aspose.pdf.devices.z1.m1(Unknown Source) - locked <0x00000000d45001e0> (a com.aspose.pdf.Document) at com.aspose.pdf.devices.z1.m1(Unknown Source) at com.aspose.pdf.devices.ImageDevice.m1(Unknown Source) at com.aspose.pdf.devices.JpegDevice.processInternal(Unknown Source) at com.aspose.pdf.devices.JpegDevice.process(Unknown Source) at com.atlassian.plugins.conversion.convert.image.ImagingConverter.thumbnailForPdf(Unknown Source) at com.atlassian.plugins.conversion.convert.image.ImagingConverter.generateThumbnailDirect(Unknown Source) at com.atlassian.confluence.plugins.conversion.impl.runnable.JVMConversionRunnable.doWork(JVMConversionRunnable.java:67) at com.atlassian.confluence.plugins.conversion.impl.runnable.ConversionRunnable.run(ConversionRunnable.java:39) at com.atlassian.confluence.plugins.conversion.impl.runnable.JVMConversionRunnable.run(JVMConversionRunnable.java:18) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Steps to Reproduce
- Edit a page
- Select Files and Images from the Insert dropdown. Can be reproduced using samplepdf.pdf
- Upload a PDF with number in its filename
Expected Results
The PDF thumbnail should render as it does for smaller PDF files
Actual Results
The PDF does not render properly, also see the symptoms above.
Refer the below Screenshot :
Workaround
Add the JVM Parameter -Dconfluence.document.conversion.threads=3 and restart Confluence for the changes to take effect. While this is not a fix for the problem, it should reduce the number of threads spawned for document conversion; and subsequently reduce overall CPU usage. This parameter is only valid for the affected versions listed (i.e, Confluence 5.7 and above).
Alternatively, it is possible to disable the file conversion service altogether: https://confluence.atlassian.com/display/CONFKB/File+conversion+service+causes+out+of+memory+errors+in+Confluence+5.7+or+above
Vendor ticket
Aspose is the third-party library Confluence uses for processing PDF files. The reference ticket for this bug:
http://www.aspose.com/community/forums/thread/672293/locking-all-threads-on-the-server.aspx
Attachments
Issue Links
- has a regression in
-
-
- Closed
-
- is duplicated by
-
-
- Closed
-
- is related to
-
-
- Closed
-
-
-
- Closed
-
- was split into
-
-
- Closed
-
- is blocked by
-
PSR-52 Loading...
- mentioned in
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-