When running a full reindex, the first 15% goes very quickly. On our medium sized instance (index is 20gb), the first 15% takes around 10 minutes. After that, indexing slows down to around 1% every 4 or 5 minutes.

This means that while a reindex seems like it will take around 1 hour at the start, it ends up taking around 8 hours. As this means a node is down, this is not ideal. If we do not have a working index for whatever reason, this means 8 hours of serious degradation.

To illustrate the issue, this is a visual from Splunk of the progress logging (eg "Re-index progress: 2443260 / 5695851. 42% complete"):

You can see that there is lots of "Re-index progress" logging happening at the start at then it slows down and never recovers.

Not sure how many versions this affects but we have seen it at least as far back as 6.3, and likely earlier.

Possible workaround

This has gone faster than using the UI in some cases so may minimise your downtime.

1. Disable traffic to the instance at the load balancer
2. Shut down Confluence
3. Delete <home>/index and <home>/journal
4. Follow the instructions here to enable index debugging permanently (you can reverse this later but it's required at restart)
5. Restart Confluence
6. Follow the index progress in the logs (look for "Re-index progress*" in the logs)
7. Once it is complete, re-enable traffic to the instance