Rebuilding Content Indexes From Scratch on Confluence Data Center does not Reindex some of the Apps (Plugins) Data

Issue Summary

Following the content index rebuild from scratch in Confluence Data Center, Confluence will then rebuild its index from the database automatically on startup and all of contents in your site should be re-indexed within this process.
The auto-reindex process is only applicable in Confluence Data Center instances as this is a part of Index Recovery action, in which index rebuild from scratch enforces.

However, it's been noticed that some of the apps (plugins) data does not get reindexed successfully after performing content index rebuild from scratch. This then resulted in missing or unsearchable apps (plugins) data in Confluence.

The issue occurs due to the asynchronous nature of the two particular processes within Confluence startup:

• Plugins load, which includes 2 phases.
  • early plugins startup: all plugins that do not require DB connections are loaded.
  • late plugins startup: plugins that require DB connections are loaded.
• Auto re-indexing process.

As these processes are asynchronous, the auto re-indexing process will then be initiated shortly after the late plugins startup commenced. Since the late startup phase plugins are still loading/ongoing, it is then possible that the auto-reindexing process then could not get all of the content extractors to be re-indexed and will then only re-index some of the apps (plugins) data (which requires DB connections to be loaded).
This resulting in missing or unsearchable apps (plugins) data in Confluence Data Center instance post performing index rebuild from scratch.

Please refer to the sample log snippet below:

2019-07-31 10:34:23,302 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 6.13.4 [build 7901 based on commit hash 0b59262db6a7cd3dbeee6d1b
1416b77c01706b36] - synchrony version 2.1.0-master-9d112c9d
...
2019-07-31 10:34:32,443 INFO [localhost-startStop-1] [confluence.cluster.hazelcast.HazelcastClusterManager] startCluster Confluence cluster node identifier is [3d4474c0]
...
2019-07-31 10:34:53,655 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] earlyStartup Plugin system earlyStartup begun
...
2019-07-31 10:35:20,404 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] logTime Plugin system earlyStartup ended
...
2019-07-31 10:35:20,450 INFO [localhost-startStop-1] [atlassian.plugin.manager.DefaultPluginManager] lateStartup Plugin system lateStartup begun
2019-07-31 10:35:20,453 WARN [localhost-startStop-1] [confluence.impl.index.DefaultIndexRecoveryService] triggerIndexRecovererModuleDescriptors Index recovery is required for main index, starting now
2019-07-31 10:35:20,498 WARN [localhost-startStop-1] [confluence.impl.index.DefaultIndexRecoveryService] triggerIndexRecovererModuleDescriptors Could not recover main index, the system will attempt to do a full re-index
...

Environment

This issue affects Confluence Data Center instances.

Steps to Reproduce

In our test environment, we're able to replicate the issue for Confluence Questions app.

Test 1 - Confluence DC auto-reindex

1. Set up a Confluence Data Center instance and install Confluence Questions add-on in this instance.
2. Create a question "testquestion" with a topic "testtopic".
3. Stop Confluence.
4. Now, please then rebuild Confluence contents indexes from scratch.
5. After auto-index finishes and starts up, you should find that <ConfluenceBaseURL>/questions/topics shows no topics.

Test 2 - Manual reindex

1. Now, go to Confluence Admin >> Content Indexing page (<ConfluenceBaseURL>/admin/search-indexes.action)
2. Click "Rebuild Index" 
3. After manual-index finishes, you should find that <ConfluenceBaseURL>/questions/topics now shows topics as expected.

Expected Results

Rebuilding content indexes from scratch should rebuild all of Confluence's contents on startup.

Actual Results

Rebuilding content indexes from scratch does not rebuild all of Confluence's contents, resulting in some of the apps' data (which required DB connections to be loaded, e.g. Confluence Questions app) to be missing or not searchable.

Workaround

This issue can be workaround by triggering the content indexing manually from the Content Indexing page (<ConfluenceBaseURL>/admin/search-indexes.action) after Confluence have been started up.