Issue Summary

Create Space request can cause performance issues if there are large amount of indexing queue lag.

This is reproducible on Data Center: Yes

Steps to Reproduce

1. Setup an environment and try to push a lot of content(create new pages, comments etc) using REST API script.
2. Ensure content indexing queue is lagging by some good amount(20k or more).
3. Initiate a create space request.
4. It will take a long time till entire indexing queue is processed.
5. In the mean time, try to push some more create space request and you would observe all the threads are stuck as initial create space request is holding the index lock.
6. Apart from above, if we are using Questions for Confluence plugin, we will see lots of question plugins request also waiting for the lock too. This is similar to issue as raised under CONFSERVER-66182 but instead of reindex thread, the lock is now being held by the create space request under this scenario.

Expected Results

If there is an issue with one space create request, then that should not cause other space create request to halt. In addition to it, this request should not effect the requests from other plugin modules.

Actual Results

One create space request can cause other space create requests or questions plugin request to halt processing if it holds the reindex lock for a long time.

Logging details

We can see below log snippet from atlassian-confluence.log that further confirms that the create space request does not release the index lock for almost 51 hours and it is still further trying to process other indexing items:

2023-07-14 14:22:57,978 WARN [https-jsse-nio2-8443-exec-92] [search.lucene.queue.JournalIndexTaskQueue] flushQueue Flushed 169000 tasks out of 279893  (186642926 ms over lock timeout)
 -- url: /confluence/rest/create-dialog/1.0/space-blueprint/create-space | referer: https://XXXXXX/confluence/display/SPACEKEY/PAGE-NAME-XX | traceId: c6a5461caf5907d6 | userName: XXXXXXXX

Workaround

Currently there is no known workaround for this behavior. A workaround will be added here when available