Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-41164

High resolution TIFF images causing Out Of Memory during generation of preview

      Summary

      When adding TIFF images with high resolution (10K - 16K) the attachment does not render a thumbnail on the page
      In the case of the 16K image, the conversion processes consume all available heap memory and turning to Out Of Memory error.
      In case of 2GB of heap memory the heap usage on Out of Memory event will be the following:

      Steps to Reproduce

      1. Get TIFF with big pixel size (i.e. Hubble’s Largest Images) or scan from one of the production scanners like Océ TC4
      2. Adapt the uploading limit setting OR convert the file into grayscale to reduce the file size but keep the pixel size.
      3. Add it to the page
      4. Push Save button

      Expected Results

      The preview generated or notification occurred that's not possible to generate a preview due to bug pixel size

      Actual Results

      The system became irresponsive and in the log a lot of methods through an error: java.lang.OutOfMemoryError: Java heap space

      Workaround

      Disable the file conversions service

      You may choose to disable the file conversion service to prevent any further out of memory errors.
      This means that only images (excluding tif) and PDF files will appear in the preview. Users won't be able to preview or comment on other files.

      To disable the file conversion service:

      1. Go to Cog icon > Add-ons.
      2. Locate the Confluence Document Conversion Library system add-on.
      3. Disable the File Conversions (file-conversions) module.

      Once disabled, Office and other files won't appear in the preview (even if they were previously uploaded), and thumbnails for existing Office and PDF files will revert back to being placeholder thumbnails (with an icon, rather than a preview of the document). You also won't be able to view any existing inline comments on the file (as the preview is disabled).

            [CONFSERVER-41164] High resolution TIFF images causing Out Of Memory during generation of preview

            Quan Pham added a comment -

            A fix for this issue is available to Server and Data Center customers in Confluence 7.0.1
            Upgrade now or check out the Release Notes to see what other issues are resolved.

            Quan Pham added a comment - A fix for this issue is available to Server and Data Center customers in Confluence 7.0.1 Upgrade now or check out the Release Notes to see what other issues are resolved.

            Hi csadmin1773753211

            I was referring to the ~100kb tifs that could blow up in memory.

            As for backporting, we'll need to analyse the feasibility internally and backport if it seems okay/safe to do so. We're in the process of doing that as the changes are very recent, so stay tuned.

            Bradley Hyde added a comment - Hi csadmin1773753211 ,  I was referring to the ~100kb tifs that could blow up in memory. As for backporting, we'll need to analyse the feasibility internally and backport if it seems okay/safe to do so. We're in the process of doing that as the changes are very recent, so stay tuned.

            EDAG Group added a comment -

            Hello Brad,

            what is a "small" TIF file 1k, 10k, 50m?

            As pointed out we have this problems on large instances with 4k+ users running
            the recommended 6.13.x Enterprise Release and I prefer using Enterprise Releases furthermore.

            Are there any plans to backport the fix to 6.13?

            Thomas

            EDAG Group added a comment - Hello Brad, what is a "small" TIF file 1k, 10k, 50m? As pointed out we have this problems on large instances with 4k+ users running the recommended 6.13.x Enterprise Release and I prefer using Enterprise Releases furthermore. Are there any plans to backport the fix to 6.13? Thomas

            Bradley Hyde added a comment - - edited

            Those small TIF files (eg, the 100kb files linked above) should no longer blow up and bring down instances in Confluence 7.0, and long-running image conversions will now be interrupted automatically at a timeout defined via system property (default 30 seconds).

            However, there are still concerns about the general memory usage of these conversions, especially for PSD files. Due to this, thumbnail/preview generation for TIF and PSD files has been disabled by default in our latest 7.0 release, with the option of enabling them again via system properties.

            They will be enabled again by default in future once the underlying memory issues are resolved. For now, the issue of re-enabling them by default can be tracked here: https://jira.atlassian.com/browse/CONFSERVER-58798

            You can see the relevant system properties in Confluence 7.0 here: https://confluence.atlassian.com/confeap/upload-files-480346744.html

            Bradley Hyde added a comment - - edited Those small TIF files (eg, the 100kb files linked above) should no longer blow up and bring down instances in Confluence 7.0, and long-running image conversions will now be interrupted automatically at a timeout defined via system property (default 30 seconds). However, there are still concerns about the general memory usage of these conversions, especially for PSD files. Due to this, thumbnail/preview generation for TIF and PSD files has been disabled by default in our latest 7.0 release, with the option of enabling them again via system properties. They will be enabled again by default in future once the underlying memory issues are resolved. For now, the issue of re-enabling them by default can be tracked here: https://jira.atlassian.com/browse/CONFSERVER-58798 You can see the relevant system properties in Confluence 7.0 here:  https://confluence.atlassian.com/confeap/upload-files-480346744.html

            Please take into account, this not only happens uploading TIFF files. F.e. PDF files containing large images also causing a crash.

            Daniel Hindenlang added a comment - Please take into account, this not only happens uploading TIFF files. F.e. PDF files containing large images also causing a crash.

            EDAG Group added a comment -

            Hello Atlassian Guys,

            I can't believe that. This Bug CRASH every Confluence System with a generic > 20M TIFF file and
            has not been fixed since 3 years!

            I've created a ticket 2 months ago and there is no progress!

            Your workarounds above are a bad joke and completely  unacceptable for large systems.

            We are now forced to pay 2000$ for an attachment filter plugin and have to delete the uploaded 3000 tifs manually!

            Thanks

             

            EDAG Group added a comment - Hello Atlassian Guys, I can't believe that. This Bug CRASH every Confluence System with a generic > 20M TIFF file and has not been fixed since 3 years! I've created a ticket 2 months ago and there is no progress! Your workarounds above are a bad joke and completely  unacceptable for large systems. We are now forced to pay 2000$ for an attachment filter plugin and have to delete the uploaded 3000 tifs manually! Thanks  

            This bug hit us hard, 6 crashes in the last 3 days. 4k users handicapped.

            Would be nice to fix that as soon as possible.

            Meanwhile,  I'll try to block Tiff within our nginx proxy, grr

             

             

            EDAG Group added a comment - This bug hit us hard, 6 crashes in the last 3 days. 4k users handicapped. Would be nice to fix that as soon as possible. Meanwhile,  I'll try to block Tiff within our nginx proxy, grr    

            Ran Lavi added a comment -

            This bug causes in the recent weeks repeated crushes (several times a week Confluence stops responding due to Out of Memory) in production instance of Confluence in a large Server instance (about 1,000 users). the Customer is very frustrated! 

            For server instances there is no good workaround. You can use an add-on to limit the size of uploaded Tiff attachments, or simply block any Tiff upload altogether. Also you need to locate all currently uploaded Tiff files and move them to some other external storage.

            For Data center the workaround is to activate the External Process Pool feature, which makes the file conversion to run in the a separate process and thus preventing the crash of Confluence.

            Ran Lavi added a comment - This bug causes in the recent weeks repeated crushes (several times a week Confluence stops responding due to Out of Memory) in production instance of Confluence in a large Server instance (about 1,000 users). the Customer is very frustrated!  For server instances there is no good workaround. You can use an add-on to limit the size of uploaded Tiff attachments, or simply block any Tiff upload altogether. Also you need to locate all currently uploaded Tiff files and move them to some other external storage. For Data center the workaround is to activate the External Process Pool feature, which makes the file conversion to run in the a separate process and thus preventing the crash of Confluence.

            Myro added a comment - - edited

            Or maybe simply being able to disable tif file preview / upload would help as a workaround, as long as the bug remains unresolved...

            Myro added a comment - - edited Or maybe simply being able to disable tif file preview / upload would help as a workaround, as long as the bug remains unresolved...

            I just spent 8 hours of my business day tracking down a large TIFF that exploded the Wiki when thumbnailed. Complete system down * 3. Frustrating.

            You could be so much smarter about this:

            1. Don't create thumbnails as part of the wiki jvm. Offload that work somewhere else.
            2. Look at metadata and don't expand large files into memory.
            3. Selectively disable filetypes from preview. (Excell files often break).

            But primarily just fix the preview system in Confluence Server.

             

            johan_ferner added a comment - I just spent 8 hours of my business day tracking down a large TIFF that exploded the Wiki when thumbnailed. Complete system down * 3. Frustrating. You could be so much smarter about this: Don't create thumbnails as part of the wiki jvm. Offload that work somewhere else. Look at metadata and don't expand large files into memory. Selectively disable filetypes from preview. (Excell files often break). But primarily just fix the preview system in Confluence Server.  

              bhyde Bradley Hyde
              ashaleev Anton Shaleev
              Affected customers:
              30 This affects my team
              Watchers:
              29 Start watching this issue

                Created:
                Updated:
                Resolved: