Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-66864

As an JIRA Datacenter Administrator I want pause NodeReindexServiceThread for replication

XMLWordPrintable

    • 2
    • 4
    • We collect Jira feedback from various sources, and we evaluate what we've collected when planning our product roadmap. To understand how this piece of feedback will be reviewed, see our Implementation of New Features Policy.

      Problem Definition

      In case of Jira Datacenter node was down for quite some time, but not long enough to receive full index, it tries to catchup the replication queue from another nodes.
      This is done in NodeReindexServiceThread, stack example:

      NodeReindexServiceThread:thread-1 
    [java.net.SocketInputStream.socketRead0(Native Method),
    java.net.SocketInputStream.socketRead(SocketInputStream.java:116),
 ...
     com.atlassian.jira.database.DatabaseAccessorImpl.executeQuery(DatabaseAccessorImpl.java:80),
     com.atlassian.jira.database.DatabaseAccessorImpl.runInTransaction(DatabaseAccessorImpl.java:93),
     com.atlassian.jira.database.DefaultQueryDslAccessor.executeQuery(DefaultQueryDslAccessor.java:65),
     com.atlassian.jira.issue.worklog.DatabaseWorklogStore.getByIssue(DatabaseWorklogStore.java:188),
     com.atlassian.jira.issue.worklog.DefaultWorklogManager.getByIssue(DefaultWorklogManager.java:103),
 ....
     com.atlassian.jira.issue.index.DefaultIssueDocumentFactory.apply(DefaultIssueDocumentFactory.java:50),
     com.atlassian.jira.issue.index.DefaultIssueDocumentFactory.apply(DefaultIssueDocumentFactory.java:30),
     com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy.get(DefaultIssueIndexer.java:556),
 ...
     com.atlassian.jira.issue.util.IssueObjectIssuesIterable.foreach(IssueObjectIssuesIterable.java:24),
     com.atlassian.jira.issue.index.DefaultIssueIndexer.perform(DefaultIssueIndexer.java:282),
     com.atlassian.jira.issue.index.DefaultIssueIndexer.reindexIssues(DefaultIssueIndexer.java:162),
     com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:571),
     com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssueObjects(DefaultIndexManager.java:468),
     sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method),
     sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62),
     sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43),
     java.lang.reflect.Method.invoke(Method.java:498),
     com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:22),
     com.sun.proxy.$Proxy17.reIndexIssueObjects(Unknown Source),
     com.atlassian.jira.index.ha.DefaultNodeReindexService.updateIssueIndex(DefaultNodeReindexService.java:404),
     com.atlassian.jira.index.ha.DefaultNodeReindexService.updateAffectedIndexes(DefaultNodeReindexService.java:298),
     com.atlassian.jira.index.ha.DefaultNodeReindexService.reIndex(DefaultNodeReindexService.java:252)

      You may not need to wait for replication to finish, for example: you want to run Full-reindex at this node.
      If replication is not paused, Full-reindex process (DefaultIndexManager) will need to compete with NodeReindexServiceThread for the lock and this might fail. 

      2018-02-28 12:24:18,466 JiraTaskExectionThread-2 ERROR admin 743x1093x1 ykx2f7 10.1.1.1 /secure/admin/jira/IndexReIndex.jspa [c.a.j.issue.index.DefaultIndexManager] Wait attempt timed out - waited 30000 milliseconds
com.atlassian.jira.issue.index.IndexException: Wait attempt timed out - waited 30000 milliseconds
...

      Suggested Solution

      Add option to pause NodeReindexServiceThread for replication.

      Workaround

      • Wait until replication is done (you can use REST API to check queue length) and then do Reindex

              Unassigned Unassigned
              ayakovlev@atlassian.com Andriy Yakovlev [Atlassian]
              Votes:
              6 Vote for this issue
              Watchers:
              13 Start watching this issue

                Created:
                Updated: