Problem Definition

During Jira shutdown, depending on the configuration, Insight can store index to disk during shutdown (documentation article).

During the next startup, if the instance has many objects and object-ticket connections, restoring the index from the disk may take longer time since the problem is single-threaded. This effectively prolongs Insight startup.

Example:

• Restoring object-tickets connection only takes 6 minutes:

  [root@ip-10-9-31-40 log]# grep 'ObjectTicketConnection' atlassian-jira*
  atlassian-jira.log:2022-07-12 11:10:03,774+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Insight cache will be restored from file for ObjectTicketConnection
  atlassian-jira.log:2022-07-12 11:10:09,964+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 50000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:10:16,562+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 100000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:10:27,021+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 150000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:10:34,395+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 200000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:10:44,494+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 250000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:10:51,964+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 300000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:11:02,504+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 350000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:11:11,389+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 400000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:11:22,902+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 450000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:11:32,322+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 500000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:11:44,067+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 550000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:11:57,417+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 600000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:12:08,706+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 650000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:12:22,751+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 700000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:12:36,812+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 750000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:12:48,560+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 800000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:13:03,056+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 850000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:13:17,887+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 900000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:13:33,063+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 950000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:13:45,724+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 1000000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:14:01,447+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 1050000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:14:17,867+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 1100000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:14:36,174+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 1150000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:14:56,181+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 1200000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:15:16,256+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 1250000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:15:36,545+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 1300000 ObjectTicketConnection so far
  atlassian-jira.log:2022-07-12 11:15:56,820+0000 InsightLauncherThread WARN anonymous     [i.r.i.index.model.InsightPersistedIndexManager] Restored 1350000 ObjectTicketConnection so far
  

• Thread dumps show a single thread occupying a whole CPU core while restoring the index:
  
• An example thread stack trace during this action:

  "InsightLauncherThread" #411 prio=5 os_prio=0 cpu=354184,11ms elapsed=365,44s tid=0x00007f6408b5f800 nid=0x4f49 runnable  [0x00007f6355be5000]
     java.lang.Thread.State: RUNNABLE
      at com.google.common.collect.ImmutableSet.copyOf(ImmutableSet.java:241)
      at com.atlassian.jira.plugin.profile.DarkFeatures.<init>(DarkFeatures.java:21)
      at com.atlassian.jira.config.feature.DefaultFeatureManager.fetchDarkFeatures(DefaultFeatureManager.java:194)
      at com.atlassian.jira.config.feature.DefaultFeatureManager.lambda$getDarkFeatures$1(DefaultFeatureManager.java:183)
      at com.atlassian.jira.config.feature.DefaultFeatureManager$$Lambda$620/0x0000000800902440.get(Unknown Source)
      at com.atlassian.jira.cache.request.RequestCacheImpl.get(RequestCacheImpl.java:96)
      at com.atlassian.jira.config.feature.DefaultFeatureManager.getDarkFeatures(DefaultFeatureManager.java:183)
      at com.atlassian.jira.config.feature.DefaultFeatureManager.isEnabled(DefaultFeatureManager.java:110)
      at com.atlassian.jira.config.feature.DefaultFeatureManager.isEnabled(DefaultFeatureManager.java:123)
      at com.atlassian.jira.config.feature.DefaultFeatureManager.isEnabled(DefaultFeatureManager.java:134)
      at jdk.internal.reflect.GeneratedMethodAccessor579.invoke(Unknown Source)
      at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.13/Unknown Source)
      at java.lang.reflect.Method.invoke(java.base@11.0.13/Unknown Source)
      at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
      at com.sun.proxy.$Proxy249.isEnabled(Unknown Source)
      at jdk.internal.reflect.GeneratedMethodAccessor579.invoke(Unknown Source)
      at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.13/Unknown Source)
      at java.lang.reflect.Method.invoke(java.base@11.0.13/Unknown Source)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
      at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
      at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
      at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
      at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
      at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
      at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
      at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
      at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
      at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
      at com.sun.proxy.$Proxy4159.isEnabled(Unknown Source)
      at io.riada.insight.index.model.ObjectTicketConnectionIndexImpl.isMemoryImprovementFeatureFlagEnabled(ObjectTicketConnectionIndexImpl.java:385)
      at io.riada.insight.index.model.ObjectTicketConnectionIndexImpl.addToInternalCacheMaps(ObjectTicketConnectionIndexImpl.java:278)
      - locked <0x00000003a0a9e5b0> (a io.riada.insight.index.model.ObjectTicketConnectionIndexImpl)
      at io.riada.insight.index.model.ObjectTicketConnectionIndexImpl.restoreObjectFromFile(ObjectTicketConnectionIndexImpl.java:172)
      at io.riada.insight.index.model.ObjectTicketConnectionIndexImpl.restoreObjectFromFile(ObjectTicketConnectionIndexImpl.java:36)
      at io.riada.insight.index.model.InsightIndexBase$$Lambda$1815/0x0000000802baa840.accept(Unknown Source)
      at io.riada.insight.index.model.InsightPersistedIndexManager.restoreCacheFromDisk(InsightPersistedIndexManager.java:123)
      at io.riada.insight.index.model.InsightIndexBase.restoreCacheFromDisk(InsightIndexBase.java:161)
      at io.riada.insight.index.model.ObjectTicketConnectionIndexImpl.restoreCacheFromDisk(ObjectTicketConnectionIndexImpl.java:152)
      at com.riadalabs.jira.plugins.insight.services.core.InsightIndexServiceImpl.restoreCacheFromDisk(InsightIndexServiceImpl.java:536)
      at com.riadalabs.jira.plugins.insight.services.launcher.InsightLauncher.restoreInsightCache(InsightLauncher.java:479)
      at com.riadalabs.jira.plugins.insight.services.launcher.InsightLauncher.initialInsightIndex(InsightLauncher.java:319)
      at com.riadalabs.jira.plugins.insight.services.launcher.InsightLauncher.lambda$onLifecycleEvent$0(InsightLauncher.java:298)
      at com.riadalabs.jira.plugins.insight.services.launcher.InsightLauncher$$Lambda$4733/0x0000000804970c40.run(Unknown Source)
      at com.atlassian.sal.core.executor.ThreadLocalDelegateRunnable.run(ThreadLocalDelegateRunnable.java:34)
  

Suggested Resolution

Implement index restoration procedure to be a multi-threaded operation. This will significantly shorten the Insight startup sequence.

Workaround (Insight 8.7.7 and higher)

A workaround could be to disable storing Insight index on shutdown, which will force Insight reindex during the next startup.

This is because Insight indexing is mutli-threaded already and it happens really fast when having Insight on version 8.7.7 and higher (even when having high object count).

However, on later JSM versions, this can trigger long reindex due to this bug: https://jira.atlassian.com/browse/JSDSERVER-13886