-
Bug
-
Resolution: Fixed
-
High
-
5.4.2, 5.7.1, 5.7.4, 5.8.4, 5.8.5, 5.8.17, 5.9.1-beta4, 5.9.12, 5.10.2
-
294
-
Severity 1 - Critical
-
This issue consists of 2 underlying symptoms relating to the PDF Conversion Service where the Confluence system can become unresponsive or crash:
- Sustained high CPU usage
- Occasional OutOfMemoryError
Confluence Improvements:
The PDF Conversion Service is built on top of a 3rd party library (Aspose), requiring an ongoing Enterprise support case between Atlassian and the Aspose team. Outside of this dependency, we have independently investigated and implemented a number of changes in the way we utilise the Aspose library in order to address this issue. Thus far, we believe that we have resolved the sustained high CPU usage with the following improvements:
- The conversion service is executed by a group of threads, managed by a thread pool. The thread pool is now dynamically managed to be inline with available system resources including the current CPU usage, memory usage as well as the current system load. In other words, the system will not launch more conversions if the system is busy and additional resources are not available.
- The latest Aspose library has been implemented, including many improvements and bug fixes.
Remaining Issue:
The Aspose library continues to require a high amount of memory for the PDF conversion service, which means the system might still experience an OutOfMemoryError. This is more prevalent when converting the extra-large PDF files (e.g. file size is above 100MB), which could exceed the available heap memory. Once the OutOfMemoryError occurs, the system is likely to be slow to respond to user requests, and a Confluence restart is recommended.
Workaround for OutOfMemoryError:
- Reduce the system level maximum attachment size to <100MB
- Increase the heap memory for the Confluence instance if possible
Further Development
The Confluence development team is looking into additional strategies to tackle the potential memory issue.
Please refer to https://jira.atlassian.com/browse/CONF-43363 for any ongoing issues relating to the OutOfMemoryError issue
If you have any ongoing issues with respect to CPU spikes following upgrading to version 5.10.4+, please feel free to comment here or raise a support request.
Best regards,
Minh Tran
Confluence BugMaster
Atlassian
Summary
Symptoms:
- When adding large PDF or Office files to a Confluence page the attachment does not render as thumbnail on the page
- There is also possibility that the CPU usage will be spiking to 100% during the time and if it does
- The threads are not released until Confluence instance is restarted
Following appears in the thread dumps:
"conversion-thread-0-internal" #1530 daemon prio=1 os_prio=0 tid=0x0000000001184000 nid=0x8eb runnable [0x00007f67075f3000] java.lang.Thread.State: RUNNABLE at com.aspose.pdf.internal.p560.z13.m7(Unknown Source) at com.aspose.pdf.internal.p560.z13.m2(Unknown Source) at com.aspose.pdf.internal.p560.z13.m6(Unknown Source) at com.aspose.pdf.internal.p560.z13.m1(Unknown Source) at com.aspose.pdf.internal.p559.z1.m1(Unknown Source) at com.aspose.pdf.internal.p543.z4.m2(Unknown Source) - locked <0x00000000fcf6cc78> (a java.lang.Object) at com.aspose.pdf.internal.p502.z4.<init>(Unknown Source) at com.aspose.pdf.internal.p502.z3.<init>(Unknown Source) at com.aspose.pdf.internal.p502.z4.m1(Unknown Source) at com.aspose.pdf.internal.p525.z1.m2(Unknown Source) at com.aspose.pdf.internal.p525.z1.m1(Unknown Source) at com.aspose.pdf.internal.p525.z1.m1(Unknown Source) at com.aspose.pdf.internal.p525.z1.m1(Unknown Source) at com.aspose.pdf.internal.p525.z1.m1(Unknown Source) at com.aspose.pdf.internal.p512.z20.m2(Unknown Source) at com.aspose.pdf.internal.p512.z20.m4(Unknown Source) at com.aspose.pdf.internal.p512.z7.m2(Unknown Source) at com.aspose.pdf.internal.p512.z7.m1(Unknown Source) at com.aspose.pdf.devices.z1.m1(Unknown Source) - locked <0x00000000d45001e0> (a com.aspose.pdf.Document) at com.aspose.pdf.devices.z1.m1(Unknown Source) at com.aspose.pdf.devices.ImageDevice.m1(Unknown Source) at com.aspose.pdf.devices.JpegDevice.processInternal(Unknown Source) at com.aspose.pdf.devices.JpegDevice.process(Unknown Source) at com.atlassian.plugins.conversion.convert.image.ImagingConverter.thumbnailForPdf(Unknown Source) at com.atlassian.plugins.conversion.convert.image.ImagingConverter.generateThumbnailDirect(Unknown Source) at com.atlassian.confluence.plugins.conversion.impl.runnable.JVMConversionRunnable.doWork(JVMConversionRunnable.java:67) at com.atlassian.confluence.plugins.conversion.impl.runnable.ConversionRunnable.run(ConversionRunnable.java:39) at com.atlassian.confluence.plugins.conversion.impl.runnable.JVMConversionRunnable.run(JVMConversionRunnable.java:18) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Steps to Reproduce
- Edit a page
- Select Files and Images from the Insert dropdown. Can be reproduced using samplepdf.pdf
- Upload a PDF with number in its filename
Expected Results
The PDF thumbnail should render as it does for smaller PDF files
Actual Results
The PDF does not render properly, also see the symptoms above.
Refer the below Screenshot :
Workaround
Add the JVM Parameter -Dconfluence.document.conversion.threads=3 and restart Confluence for the changes to take effect. While this is not a fix for the problem, it should reduce the number of threads spawned for document conversion; and subsequently reduce overall CPU usage. This parameter is only valid for the affected versions listed (i.e, Confluence 5.7 and above).
Alternatively, it is possible to disable the file conversion service altogether: https://confluence.atlassian.com/display/CONFKB/File+conversion+service+causes+out+of+memory+errors+in+Confluence+5.7+or+above
Vendor ticket
Aspose is the third-party library Confluence uses for processing PDF files. The reference ticket for this bug:
http://www.aspose.com/community/forums/thread/672293/locking-all-threads-on-the-server.aspx
- has a regression in
-
-
- Closed
-
- is duplicated by
-
-
- Closed
-
- is related to
-
-
- Closed
-
-
-
- Closed
-
- was split into
-
-
- Closed
-
- is blocked by
-
PSR-52 Loading...
- mentioned in
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
[CONFSERVER-38233] File Conversion Service causes Sustained CPU Spike
We've faced with this issue in 5.10.8. The same symptoms and exceptions under daily user load.
This might be true for PDF File Conversion but not to PPTX Powerpoint File Convesion up to 6.0.2. Check CONF-45209.
This issue has been resolved in Confluence 5.10.4 which is now available for download.
Dear All,
This issue consists of 2 underlying symptoms relating to the PDF Conversion Service where the Confluence system can become unresponsive or crash:
The PDF Conversion Service is built on top of a 3rd party library (Aspose), requiring an ongoing Enterprise support case between Atlassian and the Aspose team. Outside of this dependency, we have independently investigated and implemented a number of changes in the way we utilise the Aspose library in order to address this issue. Thus far, we believe that we have resolved the sustained high CPU usage with the following improvements:
- The conversion service is executed by a group of threads, managed by a thread pool. The thread pool is now dynamically managed to be inline with available system resources including the current CPU usage, memory usage as well as the current system load. In other words, the system will not launch more conversions if the system is busy and additional resources are not available.
- The latest Aspose library has been implemented, including many improvements and bug fixes.
Remaining Issue:
The Aspose library continues to require a high amount of memory for the PDF conversion service, which means the system might still experience an OutOfMemoryError. This is more prevalent when converting the extra-large PDF files (e.g. file size is above 100MB), which could exceed the available heap memory. Once the OutOfMemoryError occurs, the system is likely to be slow to respond to user requests, and a Confluence restart is recommended.
Workaround for OutOfMemoryError:
- Reduce the system level maximum attachment size to <100MB
- Increase the heap memory for the Confluence instance if possible
Further Development
The Confluence development team is looking into additional strategies to tackle the potential memory issue.
Please refer to https://jira.atlassian.com/browse/CONF-43363 for any ongoing issues relating to the OutOfMemoryError issue
If you have any ongoing issues with respect to CPU spikes following upgrading to version 5.10.x, please feel free to comment here or raise a support request.
Best regards,
Minh Tran
Confluence BugMaster
Atlassian
Is there an update? We are experiencing similar set of problems in 5.8.13.
Dear All,
The new library from Aspose has not fully addressed the issues we are experiencing and we are still able to reproduce CPU spikes with the document conversion feature. We have raised this with Aspose and are waiting for an update on the fix.
In the mean time one of our developers is currently working on an alternate fix that reviews how we manage the Aspose conversion threads in Confluence. Our hope is that this will reduce the overall CPU consumption and work around the issues we are experiencing with the Aspose library.
We will keep you posted when having further information
Best regards,
Minh Tran
Confluence BugMaster
Atlassian
Dear leon.borlinghaus2131104346,
Thanks for raising this issue. I believe this is another issue that our support team will be able to help you with.
Could I ask that you create a support ticket on https://support.atlassian.com?
Once you've done this one of our support engineers will be in touch to work with you in resolving this problem.
Regards,
Minh Tran
Confluence Bugmaster
Atlassian
Hi,
as I understand Aspose has fixed the library causing this problem and yet I do not see this fixed in Confluence 5.10.0. Is there any info when Confluence will have the fixed libraries?
Is it possible to update the Aspose PDF libraries manually?
Cheers,
Jesse Lahtinen
Polar Shift Ltd.
I'm not sure if this is related to this issue, but I'm not able to upload any file to a confluence page anymore. Most of the time it stuck at 100%. Sometimes i get the following error.
Leon Bornlinghaus
added a comment - - edited I'm not sure if this is related to this issue, but I'm not able to upload any file to a confluence page anymore. Most of the time it stuck at 100%. Sometimes i get the following error.
We still have issues with file converstion at 6.4.3 .. sometimes the threads get stuck and take about 20% cpu permanently...
but way better then before (we had crashes with confluence 5.10)