Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-72125

Index replication service is paused indefinitely after failing to obtain an index snapshot from another node

XMLWordPrintable

    • 7.13
    • 184
    • Severity 2 - Major
    • 1,889
    • Hide

      Hi Team,

      we're happy to announce that this issue is fixed in 9.1.0 release, available → https://www.atlassian.com/software/jira/update. 

      Before Jira 9.1 it was admin's responsibility to ensure that only one node starts at a time and that all nodes in the cluster are healthy, so that each of them can provide a healthy index snapshot. 

      Starting from Jira 9.1 the synchronous node start-up is enforced by the application. The start-up procedure ensures the local index is healthy before Jira can continue start-up. Jira will not start without a healthy index.

      The start-up procedure is performed under a cluster lock, guaranteeing that only one node at a time executes it.

      The index start-up procedure:

      1. Re-index missing data if a local issue index is less than 10% behind the database.
      1. Load a recent index snapshot from shared-home directory if one is available.
      1. Otherwise trigger a full re-index.

      Please refer to this Knowledge Base article for more information.

      With that we continue our Scale & performance roadmap work around index management, stay tuned for more stuff coming later this year.

      Cheers

      Andrzej Kotas
      Product Manager - Jira DC

      Show
      Hi Team, we're happy to announce that this issue is fixed in 9.1.0  release, available → https://www.atlassian.com/software/jira/update .  Before Jira 9.1 it was admin's responsibility to ensure that only one node starts at a time and that all nodes in the cluster are healthy, so that each of them can provide a healthy index snapshot.  Starting from Jira 9.1 the synchronous node start-up is enforced by the application. The start-up procedure ensures the local index is healthy before Jira can continue start-up. Jira will not start without a healthy index. The start-up procedure is performed under a cluster lock, guaranteeing that only one node at a time executes it. The index start-up procedure: Re-index missing data if a local issue index is less than 10% behind the database. Load a recent index snapshot from shared-home directory if one is available. Otherwise trigger a full re-index. Please refer to this  Knowledge Base  article for more information. With that we continue our  Scale & performance   roadmap work  around index management, stay tuned for more stuff coming later this year. Cheers Andrzej Kotas Product Manager - Jira DC

      Issue Summary

      Jira pauses the cluster index replication service when requesting an index snapshot from another node. If the sending node fails to provide an snapshot for any reason, the cluster index replication service will remain paused indefinitely.

      Steps to Reproduce

      1. Deploy a two-node Jira Data Center cluster. Ensure one of the nodes does not have a valid index, and the other will not be able to provide a valid snapshot. For example, as described in JRASERVER-62669.
      2. Start one of the nodes while the other is already up.

      Expected Results

      • The starting node will obtain a valid index snapshot from another node.
      • If that does not happen over a certain period of time, the starting node will either request another index snapshot, or at least unpause the index replication service.

      Actual Results

      • Starting node requests an index snapshot from any other node in the cluster:
        Starting node requests an index snapshot
        2021-02-14 04:20:41,530+0000 localhost-startStop-1 INFO      [c.a.jira.startup.ClusteringLauncher] Checking local index on node start
2021-02-14 04:20:41,534+0000 localhost-startStop-1 INFO      [c.a.jira.cluster.DefaultClusterManager] Current node: 10.0.80.208 index can't be rebuilt. Requesting an index from any other node. Current list of other nodes: [10.0.12.156, 10.0.66.220, 10.0.174.242]
(...)
2021-02-14 04:20:41,540+0000 localhost-startStop-1 INFO      [c.a.jira.cluster.DefaultClusterManager] Sending message: "Backup Index" - request to create index snapshot from node: ANY on current node: 10.0.80.208
      • The starting node then pauses index replication while waiting for an index snapshot to be provided by another node:
        Starting node pauses index replication
        2021-02-14 04:20:41,534+0000 localhost-startStop-1 INFO      [c.a.j.index.ha.DefaultNodeReindexService] [INDEX-REPLAY] Pausing node re-index service
java.lang.Exception
    at com.atlassian.jira.index.ha.DefaultNodeReindexService.pause(DefaultNodeReindexService.java:213)
    at com.atlassian.jira.cluster.DefaultClusterManager.requestCurrentIndexFromNode(DefaultClusterManager.java:138)
    at com.atlassian.jira.cluster.DefaultClusterManager.checkIndex(DefaultClusterManager.java:131)
    at com.atlassian.jira.startup.ClusteringLauncher.start(ClusteringLauncher.java:37)
    at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:168)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:146)
    at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:301)
    at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:196)
    at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:137)
    at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:104)
    at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
    at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:102)
    at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:154)
    at com.atlassian.jira.startup.LauncherContextListener.initSlowStuffInBackground(LauncherContextListener.java:139)
    at com.atlassian.jira.startup.LauncherContextListener.contextInitialized(LauncherContextListener.java:101)
    ... 5 filtered
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
      • The sending node fails to provide an index snapshot for any reason (i.e. due to JRASERVER-62669):
        Sending node fails to provide an index snapshot
        2021-02-14 04:20:42,118+0000 ClusterMessageHandlerServiceThread:thread-1 INFO      [c.a.j.index.ha.DefaultIndexCopyService] Received message: "Backup Index" - request to create index snapshot from node: 10.0.80.208 on current node: 10.0.12.156
2021-02-14 04:20:42,118+0000 ClusterMessageHandlerServiceThread:thread-1 INFO      [c.a.j.index.ha.DefaultIndexCopyService] Index backup started. Requesting node: 10.0.80.208, currentNode: 10.0.12.156
2021-02-14 04:20:42,120+0000 ClusterMessageHandlerServiceThread:thread-1 WARN      [c.a.j.index.ha.DefaultIndexCopyService] Index backup failed - latest index operation not found. Requesting node: 10.0.80.208, currentNode: 10.0.12.156

        This can happen when the requested node is able to provide a good copy too.

      • Starting node will keep waiting for an index snapshot indefinitely, while also keeping the replication service paused. Note the timestamps in this snippet, more than a day later: 
        2021-02-15 10:04:19,691+0000 NodeReindexServiceThread:thread-0 INFO      [c.a.j.index.ha.DefaultNodeReindexService] [INDEX-REPLAY] Node re-index service is not running: currentNode.isClustered=true, notRunningCounter=21396, paused=true, lastPausedStacktrace=java.lang.Throwable
        at com.atlassian.jira.index.ha.DefaultNodeReindexService.pause(DefaultNodeReindexService.java:215)
        at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:62)
        at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:51)
        at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.doReindex(ReIndexAsyncIndexerCommand.java:27)
        at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:63)
        at com.atlassian.jira.web.action.admin.index.ReIndexAsyncIndexerCommand.call(ReIndexAsyncIndexerCommand.java:18)
        at com.atlassian.jira.web.action.admin.index.AbstractAsyncIndexerCommand.call(AbstractAsyncIndexerCommand.java:26)
        at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:533)
        at com.atlassian.jira.task.TaskManagerImpl$TaskCallableDecorator.call(TaskManagerImpl.java:491)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at com.atlassian.jira.task.ForkedThreadExecutor$ForkedRunnableDecorator.run(ForkedThreadExecutor.java:216)
        at java.lang.Thread.run(Thread.java:748)

      Please notice, the "Node re-index service is not running" exceptions don't necessarily indicate that the instance is indeed affected by this bug report, especially when a newly added cluster node could appear to be hanging on startup. To identify that the node is indeed affected by the bug, please verify if other symptoms listed above are evident. If not, and if you're starting a new node up in a Data Center cluster, it's probably just a matter of waiting as the node could simply be recovering the indexes from an existing snapshot and catching up on the updates between the snapshot time and the time the new node started up. Depending on the JIRA's size, it's not unusual to see a cluster node taking one hour to startup

      • Cluster index replication will fall behind in the starting node, eventually leading to a failed Cluster Index Replication health check, and symptoms such as:
        • Missing issues in agile boards.
        • Searches bringing incomplete or inconsistent results.

      Workaround

      • request index from another node via admin panel / Copy the Search Index from another node
      • restore index from index backup
      • restart node
      • LB should not redirect users to node with no index: JRASERVER-66970

      Notes

      • As of 8.19.0 we introduced fetching index snapshot from shared on startup which will prevent this issues to happen. For this feature to work index snapshot must be available in `export/indexsnapshots` directory of shared home. A service creating snapshot of indexes is enabled by default in 24 hours cycle in that version. See details in JRASERVER-66649.
      • In Jira 9.0 we've ensure that Jira instance will create index snapshot and save it to shared home directory only when index on this instance is consistent. More details on how to handle situations where an index is not consistent can be found here: Indexing inconsistency troubleshooting

        1. Screen Shot 2021-08-05 at 12.18.32 PM.png
          Screen Shot 2021-08-05 at 12.18.32 PM.png
          265 kB

            [JRASERVER-72125] Index replication service is paused indefinitely after failing to obtain an index snapshot from another node

            Conny Postma made changes -
            Remote Link Original: This issue links to "Page (Atlassian Documentation)" [ 626031 ]
            Thiago Masutti made changes -
            Remote Link Original: This issue links to "ASCI-8 (Bulldog)" [ 556435 ] New: This issue links to "ASCI-8 (JIRA Server (Bulldog))" [ 556435 ]
            Thiago Masutti made changes -
            Remote Link Original: This issue links to "PSR-707 (Bulldog)" [ 610728 ] New: This issue links to "PSR-707 (JIRA Server (Bulldog))" [ 610728 ]

            Azfar Masut added a comment -

            Remove index folders from your Jira node $JIRA-HOME/caches/indexesV1 helps to remediate the problem as a workaround, we also added -Dcom.atlassian.jira.status.index.check=false to startup parameter

            Azfar Masut added a comment - Remove index folders from your Jira node $JIRA-HOME/caches/indexesV1 helps to remediate the problem as a workaround, we also added -Dcom.atlassian.jira.status.index.check=false to startup parameter

            Maciej Swinarski (Inactive) added a comment -

            Hi a0ef4d4784b7, this is expected. 

            When a node performa a full reindex it is:

            • not accepting user traffic (out of LB)
            • not updating the index (outside of the full-reindex)

            During this time such node does not replicated any internal index changes to other nodes and does not consume any index changes from other nodes. During this time you may see such entries in the log:

            2023-06-05 14:33:19,745+0000 NodeReindexServiceThread:thread-0 INFO      [c.a.j.index.ha.DefaultNodeReindexService] [INDEX-REPLAY] Node re-index service is not running: currentNode.isClustered=true, notRunningCounter=768, paused=true, lastPausedStacktrace=com.atlassian.jira.index.ha.DefaultNodeReindexService$StackCollector

            After the node finished the full re-index, created a snapshot and send the snapshot to shared the index replication should start working and there should be no such log entires. If this is the case please create a support request with the support.zip(s) so this can be investigated.

            Best regards,

            mac

            Maciej Swinarski (Inactive) added a comment - Hi a0ef4d4784b7 , this is expected.  When a node performa a full reindex it is: not accepting user traffic (out of LB) not updating the index (outside of the full-reindex) During this time such node does not replicated any internal index changes to other nodes and does not consume any index changes from other nodes. During this time you may see such entries in the log: 2023-06-05 14:33:19,745+0000 NodeReindexServiceThread:thread-0 INFO      [c.a.j.index.ha.DefaultNodeReindexService] [INDEX-REPLAY] Node re-index service is not running: currentNode.isClustered= true , notRunningCounter=768, paused= true , lastPausedStacktrace=com.atlassian.jira.index.ha.DefaultNodeReindexService$StackCollector After the node finished the full re-index, created a snapshot and send the snapshot to shared the index replication should start working and there should be no such log entires. If this is the case please create a support request with the support.zip(s) so this can be investigated. Best regards, mac

            Saurabh Gupta added a comment - - edited

            Found this issue in Jira 9.4.6 while re-indexing the testing instance created via the data center performance script.

            2023-06-05 14:33:19,745+0000 NodeReindexServiceThread:thread-0 INFO      [c.a.j.index.ha.DefaultNodeReindexService] [INDEX-REPLAY] Node re-index service is not running: currentNode.isClustered=true, notRunningCounter=768, paused=true, lastPausedStacktrace=com.atlassian.jira.index.ha.DefaultNodeReindexService$StackCollector

            Although the re-indexing completed successfully. 

            Saurabh Gupta added a comment - - edited Found this issue in Jira 9.4.6 while re-indexing the testing instance created via the data center performance script. 2023-06-05 14:33:19,745+0000 NodeReindexServiceThread:thread-0 INFO      [c.a.j.index.ha.DefaultNodeReindexService] [INDEX-REPLAY] Node re-index service is not running: currentNode.isClustered= true , notRunningCounter=768, paused= true , lastPausedStacktrace=com.atlassian.jira.index.ha.DefaultNodeReindexService$StackCollector Although the re-indexing completed successfully. 
            Neel made changes -
            Remote Link New: This issue links to "Page (Confluence)" [ 771959 ]
            MichaƂ Gozdera made changes -
            Remote Link New: This issue links to "Page (Confluence)" [ 760645 ]
            Rinish made changes -
            Remote Link New: This issue links to "Page (Confluence)" [ 755034 ]
            Maciej Swinarski (Inactive) made changes -
            Assignee New: Maciej Swinarski [ mswinarski ]

              mswinarski Maciej Swinarski (Inactive)
              vfontes Vinicius Fontes
              Affected customers:
              83 This affects my team
              Watchers:
              121 Start watching this issue

                Created:
                Updated:
                Resolved: