Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-100761

The Attachment Text Extractor fails due to incorrect musl library selection in Snappy v1.1.10.8

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Medium Medium
    • None
    • 9.2.7
    • Search - Indexing
    • None

      Problem

      The issue occurs when the Attachment Text Extractor fails due to a missing dependency on libc.musl-x86_64.so.1 which is a bug of Snappy 1.1.10.8 library bundled with Confluence 9.2.7. The is a post in Snappy github regarding this issue.

      Environment

      • Confluence 9.2.7 installed on Linux
      • musl library should be installed on the host

      Steps to Reproduce

      1. Install Confluence 9.2.7 on Linux
      2. Drag and drop a PDF onto any page and wait for text extractor to run
      3. Check that you can search for the text from attachment in Quick Search 
      4. Shut down Confluence and confirm no temp libraries in /opt/atlassian/confluence/temp/
      5. Install musl libraries:
        sudo apt install musl musl-dev musl-tools
      6. Start Confluence, drag and drop any PDF onto a page and wait for text extractor to run
      7. atlassian-confluence.log shows this error:
        2025-08-29 01:15:17,728 ERROR [Caesium-1-3] [atlassian.scheduler.core.JobLauncher] launchAndBuildResponse Scheduled job with ID 'LuceneContentIndexFlusher' failed due to binary incompatibilities
        java.lang.NoClassDefFoundError: Could not initialize class org.xerial.snappy.Snappy
        	at com.atlassian.confluence.internal.util.io.SnappyDataCompressor.uncompress(SnappyDataCompressor.java:47)
        	at com.atlassian.confluence.internal.util.io.SnappyDataCompressor.lambda$uncompress$0(SnappyDataCompressor.java:29)
        	at com.atlassian.confluence.internal.index.attachment.AttachmentExtractedTextHelper.toString(AttachmentExtractedTextHelper.java:24)
        	at java.base/java.util.Optional.flatMap(Unknown Source)
        	at com.atlassian.confluence.impl.search.v2.extractor.AttachmentExtractedTextExtractor.extractText(AttachmentExtractedTextExtractor.java:123)
        	at com.atlassian.confluence.internal.index.v2.CompositeExtractor.extract(CompositeExtractor.java:53)
        	at com.atlassian.confluence.internal.index.v2.Extractor2DocumentBuilder.build(Extractor2DocumentBuilder.java:48)
        	at com.atlassian.confluence.internal.index.v2.AtlassianContentDocumentBuilder.build(AtlassianContentDocumentBuilder.java:67)
        	at com.atlassian.confluence.internal.index.v2.AtlassianContentDocumentBuilder.build(AtlassianContentDocumentBuilder.java:34)
        	at com.atlassian.confluence.internal.index.v2.MergingAtlassianDocumentBuilder.lambda$buildDocuments$0(MergingAtlassianDocumentBuilder.java:38)
        	at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
        	at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
        	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
        	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
        	at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
        	at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
        	at com.atlassian.confluence.internal.index.v2.MergingAtlassianDocumentBuilder.buildDocuments(MergingAtlassianDocumentBuilder.java:38)
        	at com.atlassian.confluence.internal.index.v2.MergingAtlassianDocumentBuilder.build(MergingAtlassianDocumentBuilder.java:31)
        	at com.atlassian.confluence.internal.search.tasks.AddDocumentIndexTask.perform(AddDocumentIndexTask.java:56)
        	at com.atlassian.confluence.internal.search.tasks.UpdateDocumentIndexTask.perform(UpdateDocumentIndexTask.java:90)
        	at com.atlassian.confluence.internal.search.LuceneIncrementalIndexManager.lambda$performFlush$0(LuceneIncrementalIndexManager.java:177)
        ...
        	at com.atlassian.confluence.impl.search.IndexFlushScheduler.lambda$new$1(IndexFlushScheduler.java:118)
        	at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.doRunJob(JobRunnerWrapper.java:121)
        	at com.atlassian.confluence.impl.schedule.caesium.JobRunnerWrapper.runJob(JobRunnerWrapper.java:83)
        	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$JobRunnerWithStats.runJob(CaesiumSchedulerService.java:743)
        	at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
        	at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
        	at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
        	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:545)
        	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeLocalJobWithRetryOnFailure(CaesiumSchedulerService.java:500)
        	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:440)
        	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
        	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
        	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
        	at java.base/java.lang.Thread.run(Unknown Source)
        Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: /opt/atlassian/confluence/temp/snappy-1.1.10-ab15ec3d-5300-4afc-84c0-1d12dcc14e3d-libsnappyjava.so: libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory [in thread "attachment-text-extraction-worker-1"]
        	at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)
        	at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(Unknown Source)
        	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(Unknown Source)
        	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(Unknown Source)
        	at java.base/java.lang.ClassLoader.loadLibrary(Unknown Source)
        	at java.base/java.lang.Runtime.load0(Unknown Source)
        	at java.base/java.lang.System.load(Unknown Source)
        	at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:182)
        	at org.xerial.snappy.SnappyLoader.loadSnappyApi(SnappyLoader.java:157)
        	at org.xerial.snappy.Snappy.init(Snappy.java:70)
        	at org.xerial.snappy.Snappy.<clinit>(Snappy.java:47)
        	at com.atlassian.confluence.internal.util.io.SnappyDataCompressor.compress(SnappyDataCompressor.java:57)
        	at com.atlassian.confluence.internal.util.io.SnappyDataCompressor.lambda$compress$1(SnappyDataCompressor.java:39)
        	at com.atlassian.confluence.internal.index.attachment.DefaultAttachmentExtractedTextManager$CompressedAttachmentDataStream.getInputStream(DefaultAttachmentExtractedTextManager.java:97)
        	at com.atlassian.confluence.impl.pages.attachments.filesystem.ContentDirectoryStructureAttachmentDataFileSystemV004.saveAttachmentData(ContentDirectoryStructureAttachmentDataFileSystemV004.java:98)
        	at com.atlassian.confluence.impl.pages.attachments.filesystem.FileSystemAttachmentDataDao.saveDataForAttachment(FileSystemAttachmentDataDao.java:142)
        	at com.atlassian.confluence.impl.pages.attachments.filesystem.FileSystemAttachmentDataDao.saveDataForAttachment(FileSystemAttachmentDataDao.java:126)
        	at com.atlassian.confluence.internal.index.attachment.DefaultAttachmentExtractedTextManager.saveContent(DefaultAttachmentExtractedTextManager.java:56)
        	at com.atlassian.confluence.internal.index.attachment.AttachmentTextExtractionFunction.apply(AttachmentTextExtractionFunction.java:77)
        	at com.atlassian.confluence.internal.index.attachment.AttachmentTextExtractionFunction.apply(AttachmentTextExtractionFunction.java:24)
        	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
        	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:355)
        	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196)
        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:379)
        	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
        	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184)
        	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:223)
        	at jdk.proxy4/jdk.proxy4.$Proxy303.apply(Unknown Source)
        	at com.atlassian.confluence.internal.index.attachment.LocalAttachmentTextExtractionService.lambda$submit$1(LocalAttachmentTextExtractionService.java:43)
        	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown Source)
        	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        	... 1 more
        

      Expected Results

      The content of the attachment should be reachable on quick search

      Actual Results

      The text extraction fails and quick search won't return any search results for text within the attachment PDF

      Workarounds

      Workaround 1 - create the missing dependency sym link for libc.musl-x86_64.so.1 (***recommended***)

      It appears that the musl org.xerial.snappy_snappy-java-1.1.10.8.jar:org/xerial/snappy/native/Linux/x86_64-musl/libsnappyjava.so which is a copy deployed into /opt/atlassian/confluence/temp/ has a dependency on libc.musl-x86_64.so.1:

      # ls -l /lib/ | grep -i musl
      lrwxrwxrwx  1 root root      25 Aug 21  2021 ld-musl-x86_64.so.1 -> x86_64-linux-musl/libc.so
      drwxr-xr-x  2 root root    4096 Aug 27 01:36 x86_64-linux-musl
      
      • To "fix" the missing dependency, create a new symlink for the missing libc.musl-x86_64.so.1:
        sudo ln -s /usr/lib/x86_64-linux-musl/libc.so /lib/libc.musl-x86_64.so.1
        
        # ls -l /lib/ | grep -i musl
        lrwxrwxrwx  1 root root      25 Aug 21  2021 ld-musl-x86_64.so.1 -> x86_64-linux-musl/libc.so
        lrwxrwxrwx  1 root root      34 Aug 29 01:58 libc.musl-x86_64.so.1 -> /usr/lib/x86_64-linux-musl/libc.so
        drwxr-xr-x  2 root root    4096 Aug 27 01:36 x86_64-linux-musl
        
      • Restart Confluence.

      Workaround 2 - force use the org/xerial/snappy/native/Linux/x86_64/libsnappyjava.so

      You can force override the snappy architecture library to use by pre-configuring System properties org.xerial.snappy.lib.path and org.xerial.snappy.lib.name.

      To do this, we can add this into setenv.sh:

      # force use of 'Linux/x86_64' libsnappyjava.so
      unzip -j -o $CATALINA_HOME/confluence/WEB-INF/lib/org.xerial.snappy*.jar -d $CATALINA_HOME/temp/ org/xerial/snappy/native/Linux/x86_64/libsnappyjava.so
      if [ -f $CATALINA_HOME/temp/libsnappyjava.so ]; then
          CATALINA_OPTS="-Dorg.xerial.snappy.lib.path=$CATALINA_HOME/temp -Dorg.xerial.snappy.lib.name=libsnappyjava.so ${CATALINA_OPTS}"
      fi
      

      Once Confluence is restarted and a PDF was dropped onto a page, you can confirm that the x86_64 snappy library was used.

      # cd /opt/atlassian/confluence/temp
      # ls -l | grep -i snappy
      -rw-r--r-- 1 confluence confluence 281272 Jul 19 21:17 libsnappyjava.so
      
      # ldd libsnappyjava.so
      	linux-vdso.so.1 (0x00007ffd571d6000)
      	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000072b8fb919000)
      	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000072b8fb600000)
      	/lib64/ld-linux-x86-64.so.2 (0x000072b8fbd2a000)
      

              Unassigned Unassigned
              f8be9ce75683 Erhan Baz (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: