Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-68542

Data Center - Reindex snapshot file fails to create if greater than 8GB

    XMLWordPrintable

Details

    • 7.12
    • 3
    • Severity 3 - Minor
    • 20
    • Hide
      Atlassian Update – 21 December 2018

      Dear Jira users,

      We’re glad to announce that this issue will be addressed in our upcoming 8.0 release.

      You can find more details about our 8.0 beta release here — https://community.developer.atlassian.com/t/beta-for-jira-8-0-is-up-for-grabs/25588

      Looking forward to your feedback!

      Kind regards,
      Syed Masood
      Product Manager, Jira Server and Data Center

      Show
      Atlassian Update – 21 December 2018 Dear Jira users, We’re glad to announce that this issue will be addressed in our upcoming 8.0 release. You can find more details about our 8.0 beta release here — https://community.developer.atlassian.com/t/beta-for-jira-8-0-is-up-for-grabs/25588 Looking forward to your feedback! Kind regards, Syed Masood Product Manager, Jira Server and Data Center

    Description

      Summary

      Data Center Full re-index fails to create the snapshot file if the output is greater than 8GB. Following the exception. The issue is a limitation of Class TarArchiveOutputStream from the default library used by Jira.

      2018-12-03 01:24:09,851 IssueIndexer:thread-5 INFO AC66773 1206x3595648x2 jpi0p6 30.239.20.221 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 97% complete. Current index: Issue
      2018-12-03 01:27:07,737 automation-queue-claimer:thread-1 WARN      [c.c.j.p.automation.queue.JiraAutomationQueueExecutor] JQL search indexes are currently not available (Indexing: true, Scheduler: STANDBY). No automation items were claimed from the queue for processing on this node since most automation rules depend on JQL indexes being available.
      2018-12-03 01:32:07,746 automation-queue-claimer:thread-1 WARN      [c.c.j.p.automation.queue.JiraAutomationQueueExecutor] JQL search indexes are currently not available (Indexing: true, Scheduler: STANDBY). No automation items were claimed from the queue for processing on this node since most automation rules depend on JQL indexes being available.
      2018-12-03 01:35:30,773 Caesium-1-3 WARN ServiceRunner     [c.o.scriptrunner.analytics.FocalPoint] Bad action name: runbhv_script
      2018-12-03 01:35:31,391 JiraTaskExectionThread-499 INFO AC66773 1206x3595648x2 jpi0p6 30.239.20.221 /secure/admin/IndexReIndex.jspa [c.a.j.index.ha.DefaultIndexCopyService] Index backup started. Requesting node: va10plpjir001_node1
      2018-12-03 01:41:59,266 JiraTaskExectionThread-499 ERROR AC66773 1206x3595648x2 jpi0p6 30.239.20.221 /secure/admin/IndexReIndex.jspa [c.a.event.internal.AsynchronousAbleEventDispatcher] There was an exception thrown trying to dispatch event [com.atlassian.jira.issue.index.ReindexAllCompletedEvent@4bb8e92a] from the invoker [SingleParameterMethodListenerInvoker{method=public void com.atlassian.jira.index.ha.DefaultReplicatedIndexManager.onReindexAllCompleted(com.atlassian.jira.issue.index.ReindexAllCompletedEvent), listener=com.atlassian.jira.index.ha.DefaultReplicatedIndexManager@30ce42e3}]
      java.lang.RuntimeException: entry size '10704527832' is too big ( > 8589934591 ). Listener: com.atlassian.jira.index.ha.DefaultReplicatedIndexManager event: com.atlassian.jira.issue.index.ReindexAllCompletedEvent
      	at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:55)
      	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:38)
      	at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
      	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:88)
      	at com.atlassian.event.internal.EventPublisherImpl.invokeListeners(EventPublisherImpl.java:204)
      	at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:115)
      	at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:341)
      	at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexAll(DefaultIndexManager.java:261)
      	... 3 filtered
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:38)
      	at com.sun.proxy.$Proxy30.reIndexAll(Unknown Source)
      	at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:66)
      	at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:49)
      	at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.doReindex(ReIndexAsyncIndexerCommand.java:25)
      	at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:63)
      	at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.call(ReIndexAsyncIndexerCommand.java:16)
      	at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:26)
      	at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:458)
      	at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:426)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:216)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.RuntimeException: entry size '10704527832' is too big ( > 8589934591 )
      	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.failForBigNumber(TarArchiveOutputStream.java:620)
      	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.failForBigNumbers(TarArchiveOutputStream.java:605)
      	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.putArchiveEntry(TarArchiveOutputStream.java:285)
      	at com.atlassian.jira.util.compression.Archiver.compressDirectory(Archiver.java:71)
      	at com.atlassian.jira.util.compression.Archiver.compressDirectory(Archiver.java:73)
      	at com.atlassian.jira.index.ha.IndexUtils.takeIndexSnapshot(IndexUtils.java:161)
      	at com.atlassian.jira.index.ha.IndexUtils.takeIndexSnapshot(IndexUtils.java:128)
      	at com.atlassian.jira.index.ha.DefaultIndexCopyService$MessageConsumer.copyIndex(DefaultIndexCopyService.java:146)
      	at com.atlassian.jira.index.ha.DefaultIndexCopyService$MessageConsumer.backupIndex(DefaultIndexCopyService.java:127)
      	at com.atlassian.jira.index.ha.DefaultIndexCopyService.backupIndex(DefaultIndexCopyService.java:63)
      	at com.atlassian.jira.index.ha.DefaultReplicatedIndexManager.onReindexAllCompleted(DefaultReplicatedIndexManager.java:207)
      	... 3 filtered
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:40)
      	... 26 more
      2018-12-03 01:42:00,728 JiraTaskExectionThread-499 INFO AC66773 1206x3595648x2 jpi0p6 30.239.20.221 /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex took: 19713118ms. Indexer: DefaultIndexManager: paths: [/apps/atlassian/data/caches/indexes/comments, /apps/atlassian/data/caches/indexes/issues, /apps/atlassian/data/caches/indexes/changes, /apps/atlassian/data/caches/indexes/worklogs]
      2018-12-03 01:42:00,745 JiraTaskExectionThread-499 INFO AC66773 1206x3595648x2 jpi0p6 30.239.20.221 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 97% complete. Current index: SearchRequest
      2018-12-03 01:42:18,025 SharedEntityIndexer:thread-2 INFO AC66773 1206x3595648x2 jpi0p6 30.239.20.221 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 98% complete. Current index: SearchRequest
      2018-12-03 01:43:33,585 SharedEntityIndexer:thread-3 INFO AC66773 1206x3595648x2 jpi0p6 30.239.20.221 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 99% complete. Current index: SearchRequest
      2018-12-03 01:44:30,648 JiraTaskExectionThread-499 INFO AC66773 1206x3595648x2 jpi0p6 30.239.20.221 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 99% complete. Current index: PortalPage
      2018-12-03 01:44:49,412 SharedEntityIndexer:thread-8 INFO AC66773 1206x3595648x2 jpi0p6 30.239.20.221 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 100% complete. Current index: PortalPage
      2018-12-03 01:44:49,423 JiraTaskExectionThread-499 INFO AC66773 1206x3595648x2 jpi0p6 30.239.20.221 /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex took: 168667ms. Indexer: SharedEntityIndexManager: paths: []
      2018-12-03 01:44:49,434 JiraTaskExectionThread-499 INFO AC66773 1206x3595648x2 jpi0p6 30.239.20.221 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 100% complete. Current index: 
      2018-12-03 01:44:49,434 JiraTaskExectionThread-499 INFO AC66773 1206x3595648x2 jpi0p6 30.239.20.221 /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All complete. Total time: 19881785ms. Reindex run: 182
      2018-12-03 01:44:49,434 JiraTaskExectionThread-499 INFO AC66773 1206x3595648x2 jpi0p6 30.239.20.221 /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing finished
      

      Workaround

      1. Shut down your Jira applications;
      2. Open the jira-config.properties file (located at the root of your Jira application home directory) in a text editor;
        • This file may not exist if you are using a new installation or an upgraded installation where your previous JIRA application versions had never been customized. If this file does not exist, create it using a text editor;
      3. Edit the jira.index.snapshot.copy.archiver property in all nodes as follow (you can add it, if it's not there);
        jira.index.snapshot.copy.archiver = zip
        
      4. Save your modifications to the jira-config.properties file.
      5. Restart your JIRA applications.

      Attachments

        Issue Links

          Activity

            People

              klopacinski Karol Lopacinski
              amacedo@atlassian.com Alessandro Macedo
              Votes:
              4 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: