Asynchronous global permission cache loading

Loading cache in the background

In the default setup, the global permission cache is created synchronously. i.e. all web requests are blocked until cache is rebuilt. As this operation might take minutes, we came up with a mechanism that keeps using the old cache while the new one is being loaded in the background and replaces the old one only once the new one is ready.

First loading of the cache will still be blocking, because there's no old value to use.

Disadvantages

Because the old cache is still used after an invalidation event has come, it means that changes to permission settings (FishEye/Crucible access, "can add repository" permission) are delayed. The greatest disadvantage is that users whose access has been revoked can still access FishEye until the cache is rebuilt.

When to use

The asynchronous global permission cache loading should only be used when the application often becomes unresponsive due to long running global permission cache reloading.

Enabling global permission cache reloading

Set JVM system property to enable asynchronous global permission cache loading:

fisheye.use.async.global.permission.cache=true

Original ticket description

Summary

When the instance has huge user base, ie. 100,000 users, FishEye is unresponsive due to Global Permissions cache recreation.
This may take couple minutes to finish, while any http request is not able to load (As they have to wait for Global Permissions cache to be recreated).

Steps to Reproduce

1. Connect FishEye to one or more directory with a significant number of users
   1. The synchronization of each directory will happen every hour
   2. After the synchronization of each directory, if there is at least one change, the Global Permissions cache will be recreated

Expected Results

The recreation of the Global Permissions cache does not impact the FishEye performances and the user interface is always responsive.

Actual Results

When checking the time required for the creation of the global permission cache, high values are reported.
In the following example, the recreation of the cache took 202192 milliseconds (~ 3 minutes):

DEBUG [qtp787122337-77080 ] fisheye Timer-output - -> Loading global permission cache mem_usage = 2655336032 heapsize = 3533176832 freemem = 877840800
DEBUG [qtp787122337-77080 ] fisheye Timer-output - <- Loading global permission cache  time 202192 millis mem_usage = 2427321272 heapsize = 3300917248 freemem = 873595976

During this time all http requests will be be queued.

Notes

The number of users can be checked via the repository by running the following query on FishEye 4.0+:

select directory_id, count(directory_id)
from cwd_user
group by directory_id

Workaround

At the moment FishEye/Crucible will struggle to cope with such large user bases assuming all of them need access to FishEye/Crucible.  But if you don't expect all those users to use FishEye/Crucible and can narrow them down, consider following workaround:

1. create in remote directory group of set of groups that would together contain all the users expected to have access to FishEye/Crucible, if such group(s) don't exist yet
2. if remote directory can be configured to return only subset of users (e.g. Atlassian Crowd), configure it so FishEye/Crucible will synchronise only those groups
3. or configure appropriate filters in User Directories administration of FishEye/Crucible to synchronise only those groups