Improve activity plugin performance

The activity plugin does not perform well on sites with large amounts of traffic, especially when a large volume of activity data has been accumulated.

The activity plugin uses a simple QueueExecutor which is queue of work handled by a single thread. I think it was implemented this way to guarantee single threaded writes to the index (and to offload the work from the thread generating the indexable event i.e. http workers)

The usage plugin index manager optimizes the index after every 15th write and on c.a.c. we have reached some tipping point where the optimization is taking longer then the amount of time in which the work queue is being filled. Bear in mind that the optimization is being conducted by the executor thread which is managing the work queue. Once the queue is full, any other thread trying to add work to the queue is blocked, and for a significant amount of time, stalling requests to the web app.

The main problems at the moment are:

1) We optimize on every 15th write in the usage plugin. Confluence 2.3 and beyond uses a compound index format which in turn allows us to optimize at a greatly reduced frequency.

2) Optimization of the usage plugin index on c.a.c is taking a fairly long time. This index is ~2GB (!!) and growing. Contrast this with the main Confluence content index which is only ~200MB

3) Optimization in Confluence locks out use of the index until the optimization is finished.

4) The activity plugin index on c.a.c is not in compound index format. I think this may be because it started out life this way and it would require a fresh start to be built as a compound index. Problem: we can't just start again without losing all of the current usage statistics.

We could alleviate 1 by scheduling a job in the plugin to do the optimization but that relies on a solution to 4. Problem 2 is something we discussed at some point during the activity plugin integration. We really need an event horizon for usage events to be rolled up into more coarse grained aggregates.

Problem 3 is the killer though and is something that impacts Confluence as a whole. We really need to look at getting a solution like JIRAs ConcurrentLuceneConnection into Confluence ASAP.

The quickest thing we can do to at the moment to get the activity plugin back into production would be to change the queue policy to discard any new work when the queue is full. We might also want to reduce the optimization frequency. Having client threads do the work themselves when the queue is full is not an option as they will just block on the index manager anyway when optimization is occuring.

We should also look to reduce the size of the activity index, it grows far too rapidly at the moment. We store events with accuracy to the second, this should be changed to the hour.