JWT auth is inefficient when many Application Links exist

Issue Summary

This is reproducible on Data Center: yes

JWT based authentication is severely impacted by the existence of multiple application links. This can be most easily noticed when many LFS clients are making lots of requests for large numbers relatively small LFS objects. Database CPU usage may increase substantially, and when under high request load (typically due to CI systems) the all requests may be impacted causing performance issues apparent to interactive (web UI) users.

Bitbucket profiler logs also highlight this problem, for example:

2023-05-16 07:57:55,756 | http-nio-7990-exec-121 | *1EXEI8Kx477x6330820x225 | myuser | 78pnl7
[1394.1ms] - "GET /rest/git-lfs/storage/MYPROJECT/myrepo/880f1572f0a34c8fbb5b203866ce48a998d29503584a56969742da6e52646797 HTTP/1.1"
 [313.7ms] [count: 302, avg: 1.0ms] - String com.atlassian.stash.internal.plugin.PluginSettingDao.get(String,String)
 [32.4ms] - Authentication org.springframework.security.authentication.AuthenticationProvider.authenticate(Authentication)
  [24.9ms] - attemptAuthentication - com.atlassian.bitbucket.server.bitbucket-git-lfs:git-lfs-jwt-auth-handler
   [2.9ms] - ApplicationUser com.atlassian.bitbucket.user.UserService.getUserById(int)
  [7.5ms] - PluginAuthenticationProvider - checking whether user is LICENSED_USER
 [1.4ms] [count: 2, avg: 0.7ms] - String com.atlassian.stash.internal.plugin.PluginSettingDao.get(String,String)

The symptom above can be seen in the 302 requests to PluginSettingDao.get() and the fact it took 313ms total to complete. This is just for authentication and should be MUCH faster, and require no more then a few PluginSettingDao.get() calls.

The problem comes about because in atlassian-jwt in PluginJwtRegistry.getIssuer() it iterates over all JwtIssuerRegistry instances, of which Bitbucket has three by default:

• AppLinksJwtIssuerRegistry
• ConnectJwtIssuerRegistry
• LfsJwtRegistry

The order here matters because the slowness is caused by AppLinksJwtIssuerRegistry (more on that in a minute). If LfsJwtRegistry is "queried" before AppLinksJwtIssuerRegistry to see if it the issuer of a given JWT then authentication is fast. The order however is not deterministic as the JwtIssuerRegistry are stored in a hashmap and the key is the Java object id, so from one application restart to the next the order is changed. So this will mean different nodes in a cluster may have different performance and will change from restart to restart.

The performance problem with AppLinksJwtIssuerRegistry is that calls JwtApplinkFinderImpl.find() which iterates over all AppLinks calls getProperty() looking for: `applinks.application.<applink id>.plugin-key`. This property isn't present, and since 2015 Bitbucket's plugin settings doesn't cache negative lookup results.

To make matters worse, it doesn't just get the issuer once, but it does so four times, so the number of database requests is 4 x <# of applinks>

Steps to Reproduce

This is most easily reproduced by configuring a Bitbucket instance with many (say 10+) application links, then building a repository with thousands of small LFS objects. Then run 10 clones concurrently, this should put enough load on the database for the problem to be evident.

Workaround

Currently there is no known workaround for this behavior. A workaround will be added here when available