Uploaded image for project: 'Confluence Server'
  1. Confluence Server
  2. CONFSERVER-38233

File Conversion Service causes Sustained CPU Spike

    XMLWordPrintable

    Details

      Description

      Atlassian Resolution as of 11 August 2016

      This issue consists of 2 underlying symptoms relating to the PDF Conversion Service where the Confluence system can become unresponsive or crash:

      • Sustained high CPU usage
      • Occasional OutOfMemoryError

      Confluence Improvements:

      The PDF Conversion Service is built on top of a 3rd party library (Aspose), requiring an ongoing Enterprise support case between Atlassian and the Aspose team. Outside of this dependency, we have independently investigated and implemented a number of changes in the way we utilise the Aspose library in order to address this issue. Thus far, we believe that we have resolved the sustained high CPU usage with the following improvements:

      1. The conversion service is executed by a group of threads, managed by a thread pool. The thread pool is now dynamically managed to be inline with available system resources including the current CPU usage, memory usage as well as the current system load. In other words, the system will not launch more conversions if the system is busy and additional resources are not available.
      2. The latest Aspose library has been implemented, including many improvements and bug fixes.

      Remaining Issue:

      The Aspose library continues to require a high amount of memory for the PDF conversion service, which means the system might still experience an OutOfMemoryError. This is more prevalent when converting the extra-large PDF files (e.g. file size is above 100MB), which could exceed the available heap memory. Once the OutOfMemoryError occurs, the system is likely to be slow to respond to user requests, and a Confluence restart is recommended.
      Workaround for OutOfMemoryError:

      1. Reduce the system level maximum attachment size to <100MB
      2. Increase the heap memory for the Confluence instance if possible

      Further Development

      The Confluence development team is looking into additional strategies to tackle the potential memory issue.
      Please refer to https://jira.atlassian.com/browse/CONF-43363 for any ongoing issues relating to the OutOfMemoryError issue
      If you have any ongoing issues with respect to CPU spikes following upgrading to version 5.10.4+, please feel free to comment here or raise a support request.

      Best regards,
      Minh Tran
      Confluence BugMaster
      Atlassian

      Summary

      Symptoms:

      • When adding large PDF or Office files to a Confluence page the attachment does not render as thumbnail on the page
      • There is also possibility that the CPU usage will be spiking to 100% during the time and if it does
      • The threads are not released until Confluence instance is restarted

      Following appears in the thread dumps:

      "conversion-thread-0-internal" #1530 daemon prio=1 os_prio=0 tid=0x0000000001184000 nid=0x8eb runnable [0x00007f67075f3000]
         java.lang.Thread.State: RUNNABLE
      	at com.aspose.pdf.internal.p560.z13.m7(Unknown Source)
      	at com.aspose.pdf.internal.p560.z13.m2(Unknown Source)
      	at com.aspose.pdf.internal.p560.z13.m6(Unknown Source)
      	at com.aspose.pdf.internal.p560.z13.m1(Unknown Source)
      	at com.aspose.pdf.internal.p559.z1.m1(Unknown Source)
      	at com.aspose.pdf.internal.p543.z4.m2(Unknown Source)
      	- locked <0x00000000fcf6cc78> (a java.lang.Object)
      	at com.aspose.pdf.internal.p502.z4.<init>(Unknown Source)
      	at com.aspose.pdf.internal.p502.z3.<init>(Unknown Source)
      	at com.aspose.pdf.internal.p502.z4.m1(Unknown Source)
      	at com.aspose.pdf.internal.p525.z1.m2(Unknown Source)
      	at com.aspose.pdf.internal.p525.z1.m1(Unknown Source)
      	at com.aspose.pdf.internal.p525.z1.m1(Unknown Source)
      	at com.aspose.pdf.internal.p525.z1.m1(Unknown Source)
      	at com.aspose.pdf.internal.p525.z1.m1(Unknown Source)
      	at com.aspose.pdf.internal.p512.z20.m2(Unknown Source)
      	at com.aspose.pdf.internal.p512.z20.m4(Unknown Source)
      	at com.aspose.pdf.internal.p512.z7.m2(Unknown Source)
      	at com.aspose.pdf.internal.p512.z7.m1(Unknown Source)
      	at com.aspose.pdf.devices.z1.m1(Unknown Source)
      	- locked <0x00000000d45001e0> (a com.aspose.pdf.Document)
      	at com.aspose.pdf.devices.z1.m1(Unknown Source)
      	at com.aspose.pdf.devices.ImageDevice.m1(Unknown Source)
      	at com.aspose.pdf.devices.JpegDevice.processInternal(Unknown Source)
      	at com.aspose.pdf.devices.JpegDevice.process(Unknown Source)
      	at com.atlassian.plugins.conversion.convert.image.ImagingConverter.thumbnailForPdf(Unknown Source)
      	at com.atlassian.plugins.conversion.convert.image.ImagingConverter.generateThumbnailDirect(Unknown Source)
      	at com.atlassian.confluence.plugins.conversion.impl.runnable.JVMConversionRunnable.doWork(JVMConversionRunnable.java:67)
      	at com.atlassian.confluence.plugins.conversion.impl.runnable.ConversionRunnable.run(ConversionRunnable.java:39)
      	at com.atlassian.confluence.plugins.conversion.impl.runnable.JVMConversionRunnable.run(JVMConversionRunnable.java:18)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      
      
      
      
      

      Steps to Reproduce

      1. Edit a page
      2. Select Files and Images from the Insert dropdown. Can be reproduced using samplepdf.pdf
      3. Upload a PDF with number in its filename

        Expected Results

      The PDF thumbnail should render as it does for smaller PDF files

      Actual Results

      The PDF does not render properly, also see the symptoms above.

      Refer the below Screenshot :

      Workaround

      Add the JVM Parameter -Dconfluence.document.conversion.threads=3 and restart Confluence for the changes to take effect. While this is not a fix for the problem, it should reduce the number of threads spawned for document conversion; and subsequently reduce overall CPU usage. This parameter is only valid for the affected versions listed (i.e, Confluence 5.7 and above).

      Alternatively, it is possible to disable the file conversion service altogether: https://confluence.atlassian.com/display/CONFKB/File+conversion+service+causes+out+of+memory+errors+in+Confluence+5.7+or+above

      Vendor ticket

      Aspose is the third-party library Confluence uses for processing PDF files. The reference ticket for this bug:
      http://www.aspose.com/community/forums/thread/672293/locking-all-threads-on-the-server.aspx

        Attachments

        1. 20151110-jstack-1-conversion-thread.txt
          112 kB
        2. 20151113-jstack-1-conversion-thread-60-hrs-later.txt
          116 kB
        3. 2015-11-24_16h07_13.png
          2015-11-24_16h07_13.png
          104 kB
        4. cannot-render.png
          cannot-render.png
          101 kB
        5. java_threads.txt
          112 kB
        6. jstack-20150825-100pct-cpu1.txt
          138 kB
        7. pdf_thumbnail.jpg
          pdf_thumbnail.jpg
          41 kB
        8. samplepdf.pdf
          425 kB
        9. samplepdf0199.pdf
          425 kB
        10. screenshot-16-pdfs-in-editor-5.8.15.png
          screenshot-16-pdfs-in-editor-5.8.15.png
          211 kB
        11. screenshot-16-pdfs-in-view-5.8.15.png
          screenshot-16-pdfs-in-view-5.8.15.png
          155 kB
        12. screenshot-16-pdfs-in-view-after-refresh-5.8.15.png
          screenshot-16-pdfs-in-view-after-refresh-5.8.15.png
          354 kB
        13. screenshot-17.png
          screenshot-17.png
          15 kB
        14. screenshot-editor-in-confluence-5.7.5.png
          screenshot-editor-in-confluence-5.7.5.png
          97 kB
        15. screenshot-editor-in-confluence-5.8.15.png
          screenshot-editor-in-confluence-5.8.15.png
          75 kB
        16. screenshot-in-confluence-5.7.5.png
          screenshot-in-confluence-5.7.5.png
          107 kB
        17. screenshot-in-confluence-5.8.15.png
          screenshot-in-confluence-5.8.15.png
          104 kB
        18. screenshot-physiology-editor-in-confluence-5.8.15.png
          screenshot-physiology-editor-in-confluence-5.8.15.png
          123 kB
        19. screenshot-physiology-in-confluence-5.8.15.png
          screenshot-physiology-in-confluence-5.8.15.png
          143 kB
        20. screenshot-physiology-preview-in-confluence-5.8.15.png
          screenshot-physiology-preview-in-confluence-5.8.15.png
          853 kB

          Issue Links

            Activity

              People

              Assignee:
              fxu Feng Xu
              Reporter:
              msayed Mizan Sayed [Atlassian]
              Votes:
              146 Vote for this issue
              Watchers:
              175 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: