Details
-
Bug
-
Resolution: Fixed
-
Low
-
8.10.0, 8.13.0, 8.14.0, 8.13.2, 8.14.1, 8.15.0
-
8.1
-
2
-
Severity 2 - Major
-
25
-
Description
Issue Summary
When starting Jira 8.13 in Disaster Recovery mode, the index restoration, from a snapshot, fails.
Steps to Reproduce
- Create an index snapshot as instructed here
- Note that you can't create a snapshot on demand but rather wait for the Cron to execute in the specified time
- After the index snapshot has been generated, stop Jira
If you don't have a DR environment set up, you can replicate in your test environment as follows:- Remove the folder /<sharedhome>/indexarchive/
- Enable DR by adding disaster.recovery=true to jira-config.properties
- Remove index folders from your Jira node $JIRA-HOME/caches/indexesV1
- Start Jira in DR mode
Expected Results
Index restoration is completed successfully from the snapshot in the first start-up and Jira start correctly.
Actual Results
Starting as of Jira 8.10, 8.11, and 8.12, there are changes in how Jira perform reindex operations (and also introduced index versioning) and looks like the index snapshot recovery is being called from a thread that doesn't provide the Jira context, hence failing as we can see below:
2021-02-11 15:04:07,546+0000 localhost-startStop-1 INFO [c.a.j.index.ha.DefaultIndexRecoveryManager] [INDEX-FIXER] Re-indexing issues updated in the last {1 days, 0 hours, 30 minutes, and 23 seconds}. (Note: it's an intentionally wider range)
2021-02-11 15:04:07,546+0000 localhost-startStop-1 INFO [c.a.j.index.ha.DefaultIndexRecoveryManager] [INDEX-FIXER] Recovering search indexes - 60% complete... [INDEX-FIXER] Re-indexing issues modified in the last {1 days, 0 hours, 30 minutes, and 23 seconds}. $
2021-02-11 15:04:07,577+0000 localhost-startStop-1 ERROR [c.a.jira.startup.LauncherContextListener] Unable to start JIRA.
java.lang.IllegalStateException: Incorrect usage of JIRA/lucene search API. You can only create/use: ManagedIndexSearcher inside a context (request or Jira-Thread-Local). Check: JiraThreadLocalUtils for details.
at com.atlassian.jira.index.ManagedIndexSearcherFactory.createFrom(ManagedIndexSearcherFactory.java:15)
at com.atlassian.jira.issue.index.ThreadLocalSearcherCache$Cache.retrieveEntitySearcher(ThreadLocalSearcherCache.java:142)
at com.atlassian.jira.issue.index.ThreadLocalSearcherCache.getSearcher(ThreadLocalSearcherCache.java:40)
at com.atlassian.jira.issue.index.DefaultIndexManager.getEntitySearcher(DefaultIndexManager.java:930)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
... 2 filtered
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:38)
at com.sun.proxy.$Proxy37.getEntitySearcher(Unknown Source)
at com.atlassian.jira.versioning.EntityVersioningManagerImpl.getSearcher(EntityVersioningManagerImpl.java:322)
at com.atlassian.jira.versioning.EntityVersioningManagerImpl.getLocalVersionsInOneQuery(EntityVersioningManagerImpl.java:169)
at com.atlassian.jira.versioning.EntityVersioningManagerImpl.getLocalVersions(EntityVersioningManagerImpl.java:160)
at com.atlassian.jira.versioning.EntityVersioningManagerWithStats.getLocalVersions(EntityVersioningManagerWithStats.java:252)
at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager.getCorrespondingEntityIndexVersions(DefaultIndexRecoveryManager.java:346)
at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager.filterOutAlreadyIndexedEntities(DefaultIndexRecoveryManager.java:335)
at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager.reindexOutdatedEntities(DefaultIndexRecoveryManager.java:290)
at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager.reindexWithVersionCheckEntitiesUpdatedInTheLast(DefaultIndexRecoveryManager.java:239)
at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager.recoverIndexFromBackup(DefaultIndexRecoveryManager.java:173)
at com.atlassian.jira.index.ha.DisasterRecoveryLauncher.restoreIndex(DisasterRecoveryLauncher.java:108)
at com.atlassian.jira.index.ha.DisasterRecoveryLauncher.start(DisasterRecoveryLauncher.java:68)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDBActivated(DefaultJiraLauncher.java:174)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$postDbLaunch$2(DefaultJiraLauncher.java:146)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrEnqueue(DatabaseConfigurationManagerImpl.java:301)
at com.atlassian.jira.config.database.DatabaseConfigurationManagerImpl.doNowOrWhenDatabaseActivated(DatabaseConfigurationManagerImpl.java:196)
at com.atlassian.jira.startup.DefaultJiraLauncher.postDbLaunch(DefaultJiraLauncher.java:137)
at com.atlassian.jira.startup.DefaultJiraLauncher.lambda$start$0(DefaultJiraLauncher.java:104)
at com.atlassian.jira.util.devspeed.JiraDevSpeedTimer.run(JiraDevSpeedTimer.java:31)
at com.atlassian.jira.startup.DefaultJiraLauncher.start(DefaultJiraLauncher.java:102)
at com.atlassian.jira.startup.LauncherContextListener.initSlowStuff(LauncherContextListener.java:154)
at com.atlassian.jira.startup.LauncherContextListener.initSlowStuffInBackground(LauncherContextListener.java:139)
at com.atlassian.jira.startup.LauncherContextListener.contextInitialized(LauncherContextListener.java:101)
... 5 filtered
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Workaround
- Start Jira without Disaster Recovery mode
- Go to Cog > System > Index
- Restore the index manually using the snapshot name, which should be located in /<shared-home/import/indexsnapshot
