Jira full reindex process does not terminate when it encounters a relevant failure

Issue Summary

When the full reindex encounters a failure, it will continue the reindex process untill 100% completeness.
Then it will mark the reindex as failed only at the end of the process.

On large instances, on which a full reindex can take several hours, if the failure occurred within the first minutes, the process would still continue for hours only to report back as failed.

The full reindex process should stop on the first failure and report that to the administrator.

Steps to Reproduce

1. Install a vanilla instance of Jira.
2. Have a reasonable amount of data so the full reindex takes more than a couple of minutes.
3. Introduce an inconsistency that would cause the full reindex to fail.
   • For example the problem reported on Reindexing JIRA causes a Nullpointer Error.
4. Run a full foreground reindex.

Expected Results

The full reindex fails as soon as the first error occurs.

Actual Results

The full reindex continues untill 100% and then will be reported as completed with errors.
On large instances, an administrator may wait for hours only to know the reindex failed because of an issue that occurred on the initial minutes.

Here's an example of a production instance:

• Full reindex started on 2024-03-25 at 16:53.

  2024-03-25 16:53:43,087+0000 JiraTaskExecutionThread-1 INFO user001     [c.a.j.r.v2.index.ReindexResource] Re-indexing started
  2024-03-25 16:53:43,088+0000 JiraTaskExecutionThread-1 INFO user001     [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All starting...
  

• A relevant error occurred on 2024-03-25 at 17:33:11 – ~40 minutes after the reindex started.

  2024-03-25 17:33:11,990+0000 IssueIndexer:thread-14 WARN      [c.a.jira.index.AccumulatingResultBuilder] java.lang.NullPointerException
  java.lang.RuntimeException: java.lang.NullPointerException
  	at com.atlassian.jira.index.DefaultIndex$Failure.<init>(DefaultIndex.java:100)
  	at com.atlassian.jira.issue.index.DefaultIssueIndexer$EntityOperation.lambda$perform$6(DefaultIssueIndexer.java:887)
  	at java.base/java.util.HashMap.forEach(HashMap.java:1337)
  	at com.atlassian.jira.issue.index.DefaultIssueIndexer$EntityOperation.perform(DefaultIssueIndexer.java:887)
  	at com.atlassian.jira.issue.index.DefaultIssueIndexer.lambda$processInnerBatch$6(DefaultIssueIndexer.java:338)
  	at com.atlassian.jira.index.SimpleIndexingStrategy.apply(SimpleIndexingStrategy.java:7)
  ...
  	at java.base/java.lang.Thread.run(Thread.java:829)
  Caused by: java.lang.NullPointerException
  	at com.atlassian.greenhopper.customfield.epiclink.EpicLinkCFType$EpicLinkCustomFieldProvider.lambda$splitIssuesOnProjectType$2(EpicLinkCFType.java:386)
  	at java.base/java.util.stream.Collectors.lambda$groupingBy$53(Collectors.java:1134)
  	at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
  	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
  	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
  	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
  	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
  	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
  	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
  	at com.atlassian.greenhopper.customfield.epiclink.EpicLinkCFType$EpicLinkCustomFieldProvider.splitIssuesOnProjectType(EpicLinkCFType.java:386)
  	at com.atlassian.greenhopper.customfield.epiclink.EpicLinkCFType$EpicLinkCustomFieldProvider.getCustomFieldInfo(EpicLinkCFType.java:362)
  	at com.atlassian.jira.issue.index.managers.NonNullCustomFieldProviderWithStats.getCustomFieldInfo(NonNullCustomFieldProviderWithStats.java:54)
  	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory.lambda$getPersistedFieldsWithData$5(DefaultIssueDocumentFactory.java:176)
  	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
  	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory.getPersistedFieldsWithData(DefaultIssueDocumentFactory.java:175)
  	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory.createDocuments(DefaultIssueDocumentFactory.java:109)
  	at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy.get(DefaultIssueIndexer.java:1424)
  	at com.atlassian.jira.issue.index.DefaultIssueIndexer$IssuesOperation.createDocuments(DefaultIssueIndexer.java:1019)
  	at com.atlassian.jira.issue.index.DefaultIssueIndexer$EntityOperation.perform(DefaultIssueIndexer.java:876)
  	... 10 more
  2024-03-25 17:33:11,990+0000 IssueIndexer:thread-14 WARN      [c.a.jira.index.AccumulatingResultBuilder] 
  

• The full reindex continued besides the error.

  2024-03-25 17:50:17,964+0000 IssueIndexer:thread-26 INFO      [c.a.j.r.v2.index.ReindexResource] Re-indexing is 10% complete. Current index: Issue
  

• The full reindex is reported as completed with errors on 2024-03-26 at 02:54

  2024-03-26 02:54:18,752+0000 JiraTaskExecutionThread-1 INFO user001     [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex took: 128230ms. Indexer: SharedEntityIndexManager: paths: []
  2024-03-26 02:54:18,752+0000 JiraTaskExecutionThread-1 INFO user001     [c.a.j.index.ha.DefaultNodeReindexService] [INDEX-REPLAY] Starting node re-index service, paused=false, running period=5sec, delay=10sec
  2024-03-26 02:54:18,753+0000 JiraTaskExecutionThread-1 INFO user001     [c.a.j.r.v2.index.ReindexResource] Re-indexing finished
  2024-03-26 02:54:18,753+0000 JiraTaskExecutionThread-1 ERROR user001     [c.a.jira.task.TaskManagerImpl] Task 'Jira Indexing' failed.
  com.atlassian.jira.index.IndexingFailureException: Indexing completed with 1 errors
  	at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:88)
  	at com.atlassian.jira.util.index.CompositeIndexLifecycleManager.reIndexAll(CompositeIndexLifecycleManager.java:51)
  

In the above example we would save the administrators several hours if the reindex process stopped on the initial failure.

Workaround

Follow the application logs for entries logged by the c.a.jira.index.AccumulatingResultBuilder package.
These are the errors that would impact the reindex process and will