Issue Summary

Import triggered during startup before index consistency has been checked, creating duplicate objects or new objects with no attribute values.

Insight index consistency check and potential reindex should occur before the import event happens. However, if the node is restarted and there is an import task scheduled while Jira is starting up, the import operation will run prior to index check consistency and potential reindex. This means if the index is corrupt or not located on the disk, import will:

• Either create duplicate objects, or
• Create new objects, without any attribute values.

Steps to Reproduce

Not all circumstances of successful reproduction are known, but the general steps to reproduce this are:

1. Stop Jira node where Insight is installed, making sure the Insight cache is not stored on shutdown (or the Insight cache has been deleted before the startup).
2. Start Jira
3. During the startup, a scheduled import task starts to run before re-indexing task has started or index consistency has been checked

Expected Results

Re-indexing / index consistency check is done first, then the imports starts executing afterwards.

Actual Results

Import starts before the indexing. Since Import relies on the data from the index, missing or corrupted index will cause the import task to create duplicated objects or new objects with no attribute values.

Log excerpt:

• During startup, import job kicks in, leaving many warning while creating / updating objects:

  2022-01-18 07:00:07,794+0000 insight-InsightImportThreadGroup-worker-thread-7 WARN admin     [c.r.j.p.i.services.core.ObjectServiceImpl] Failed to create object Object TEST (null), ObjectTypeId: 1538, Attributes: (AttributeTypeId: 12008, value(s):[null(Object TEST)],AttributeTypeId: 12065, value(s):[null(110)],AttributeTypeId: 12062, value(s):[null(Object TEST)],AttributeTypeId: 12063, value(s):[null(test)],AttributeTypeId: 12064, value(s):[null(26470)]), avatar: false with actions: [CREATE, HAVING_OBJECT_REFERENCE]
  java.lang.NullPointerException
  	at com.riadalabs.jira.plugins.insight.services.core.ObjectServiceImpl.createAttributeAddAction(ObjectServiceImpl.java:976)
  	at com.riadalabs.jira.plugins.insight.services.core.ObjectServiceImpl.createStartingAttributeActions(ObjectServiceImpl.java:953)
  	at com.riadalabs.jira.plugins.insight.services.core.ObjectServiceImpl.createNewObject(ObjectServiceImpl.java:597)
  	at com.riadalabs.jira.plugins.insight.services.core.ObjectServiceImpl.storeObjectInternal(ObjectServiceImpl.java:444)
  	at com.riadalabs.jira.plugins.insight.services.core.ObjectServiceImpl.storeObject(ObjectServiceImpl.java:383)
  	at com.riadalabs.jira.plugins.insight.services.core.ObjectServiceImpl.storeObject(ObjectServiceImpl.java:264)
  	at com.riadalabs.jira.plugins.insight.services.imports.common.importjobprovider.ImportJobProvider.storeObjectImportBean(ImportJobProvider.java:161)
  	at com.riadalabs.jira.plugins.insight.services.imports.common.importjobprovider.ImportObjectJobProvider$ImportObjectsJob.createObject(ImportObjectJobProvider.java:283)
  	at com.riadalabs.jira.plugins.insight.services.imports.common.importjobprovider.ImportObjectJobProvider$ImportObjectsJob.performActionOnPreparedObject(ImportObjectJobProvider.java:197)
  	at com.riadalabs.jira.plugins.insight.services.imports.common.importjobprovider.ImportObjectJobProvider$ImportObjectsJob.executeTask(ImportObjectJobProvider.java:113)
  	at com.riadalabs.jira.plugins.insight.services.imports.common.importjobprovider.ImportObjectJobProvider$ImportObjectsJob.executeTask(ImportObjectJobProvider.java:74)
  	at com.riadalabs.jira.plugins.insight.services.core.multithreadservice.InsightServiceJob.call(InsightServiceJob.java:42)
  	at com.atlassian.sal.core.executor.ThreadLocalDelegateCallable.call(ThreadLocalDelegateCallable.java:38)
  	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  	at java.base/java.lang.Thread.run(Thread.java:829)
  ...
  2022-01-18 07:00:07,806+0000 insight-InsightImportThreadGroup-worker-thread-7 ERROR admin    [c.r.j.p.i.services.core.ObjectServiceImpl] RuntimeException:java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
  com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
  	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050)
  	at com.google.common.cache.LocalCache.get(LocalCache.java:3952)
  	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
  	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
  	at io.riada.insight.index.model.InsightIndexBase.load(InsightIndexBase.java:69)
  	at io.riada.insight.index.model.ObjectIndexImpl.loadObject(ObjectIndexImpl.java:128)
  	at com.riadalabs.jira.plugins.insight.services.core.ObjectServiceImpl.deleteObject(ObjectServiceImpl.java:1335)
  	at com.riadalabs.jira.plugins.insight.services.core.ObjectServiceImpl.deleteObject(ObjectServiceImpl.java:1325)
  

• Index consistency check / reindexing starting after the import job has been finished (effectively prolonging Insight startup):

  2022-01-18 07:10:28,483+0000 InsightLauncherThread WARN anonymous     [c.r.j.p.i.services.core.InsightIndexServiceImpl] Integrity check failed, number of objects mismatched, database: 592901, index: 1611
  2022-01-18 07:10:28,639+0000 InsightLauncherThread WARN anonymous     [c.r.j.p.i.services.core.InsightIndexServiceImpl] Integrity check failed, number of objects jira issue values mismatched, database: 250398, index: 93
  2022-01-18 07:10:28,640+0000 InsightLauncherThread WARN anonymous     [c.r.j.p.i.services.core.InsightIndexServiceImpl] Integrity check failed, number of objects schemas mismatched, database: 88, index: 6
  2022-01-18 07:10:28,641+0000 InsightLauncherThread WARN anonymous     [c.r.j.p.i.services.core.InsightIndexServiceImpl] Integrity check failed, number of objects types mismatched, database: 1080, index: 80
  2022-01-18 07:10:28,642+0000 InsightLauncherThread WARN anonymous     [c.r.j.p.i.services.core.InsightIndexServiceImpl] Integrity check failed, number of objects type attribute mismatched, database: 7809, index: 69
  ...
  2022-01-18 07:10:29,614+0000 insight-InsightThreadGroup-worker-thread-13 WARN Anonymous user     [c.r.j.p.i.s.core.impl.ClusterAwareReindexServiceImpl] Node node1 is locking to perform reindex. Insight is locked
  

Mitigation

One of the following should help you to mitigate the problem:

• Restart Insight node when there is no import job scheduled
• Disable import tasks before restarting the node
• Extend Tomcat shutdown timeout, so that the index has enough time to be stored onto the disk without being interrupted. This will make sure the index does not ends up inconsistent during node startup.
• Persist index to disk, and before starting the scheduled node, paste the persisted index over the existing one (this one should be consistent with the database during startup).