Issue Summary

This is reproducible on Data Center: Yes

This is related to the Rate Limiting feature in Confluence. As per the  https://confluence.atlassian.com/doc/improving-instance-stability-with-rate-limiting-992679004.html, Rate limiting targets only external REST API requests, which means that requests made within Confluence aren’t limited in any way.

However, the following one of the REST calls which is part of the Confluence Health check and triggered by the Atlassian Audit Plugin is treated as an External call.

 /rest/auditing/latest/statistics/database/usage

Confluence applies a Rate limiting in that and blocked the same with 429 HTTP response code, even printing the log message in the application log file.

<IP AppServer1>	-	-	[11/Jul/2022:00:24:57	+0200]	GET /rest/auditing/latest/statistics/database/usage HTTP/1.1	429	708	-	Apache-HttpClient/4.5.13 (Java/11.0.12)	11742
<IP AppServer2>	-	-	[11/Jul/2022:00:29:18	+0200]	GET /rest/auditing/latest/statistics/database/usage HTTP/1.1	429	708	-	Apache-HttpClient/4.5.13 (Java/11.0.12)	13427
<IP AppServer1>	-	-	[11/Jul/2022:01:24:57	+0200]	GET /rest/auditing/latest/statistics/database/usage HTTP/1.1	429	708	-	Apache-HttpClient/4.5.13 (Java/11.0.12)	15849
<IP AppServer2>	-	-	[11/Jul/2022:01:29:18	+0200]	GET /rest/auditing/latest/statistics/database/usage HTTP/1.1	429	709	-	Apache-HttpClient/4.5.13 (Java/11.0.12)	8764

2022-07-10 10:24:57,224 WARN [http-nio-XXXX-exec-43] [ratelimiting.internal.filter.RateLimitFilter] lambda$userHasBeenRateLimited$0 User [rate_limiting_anonymous_user-67d5362d-3b2f-4531-9039-5f041bdd402a] has been rate limited
 -- url: /rest/auditing/latest/statistics/database/usage | traceId: 7e44ae2e6ad83648 | userName: anonymous

2022-07-10 13:29:18,470 WARN [http-nio-XXXX-exec-28] [ratelimiting.internal.filter.RateLimitFilter] lambda$userHasBeenRateLimited$0 User [rate_limiting_anonymous_user-67d5362d-3b2f-4531-9039-5f041bdd402a] has been rate limited
 -- url: /rest/auditing/latest/statistics/database/usage | traceId: 9c58bf09e9dfd150 | userName: anonymous

Steps to Reproduce

1. Apply the rate-limiting for anonymous users and block them from the rate-limiting page.
2. Now, navigate the Troubleshooting and support tools page and let confluence complete the Health check for all modules. 
3. If you check the Access logs ( Proxy or confluence ), also application logs you will see the above logs. 

From the rate-limiting plugin developer page, the Ratelimiting logic just checks the following *.applinks, some gagdgets, Mywork, and REST Auth calls and consider them as an Internal call. All other calls, even those triggered by the application treated as external calls. 

The mechanism to determine if it’s an internal call is pretty specific. We check if it is: * an Applinks request

• one of the many whitelist urls: https://bitbucket.org/atlassian/atlassian-ratelimiting/src/e604f6db27f29238dfe1c61c[…]g/internal/confluence/ConfluenceRateLimitingConfiguration.java

• of which /rest/auditing/latest/statistics/database/usage doesn’t seem to be part of:

"/*/rest/applinks/*",
"/**/rest/capabilities",
"/**/rest/capabilities/navigation",
"/*/rest/gadgets/1.0/g/*",
"/**/rest/mywork/latest/status/notification/count",
"/**/rest/jira-metadata/1.0/metadata/cache",
"/**/rest/api/content",
"/**/rest/quickreload/latest/0",
"/**/rest/mywork/1/client",
"/*/rest/token-auth/api/*"
**

 

Expected Results

Confluence can identify and differentiate the External and Internal REST calls and apply the Rate Limiting on the actual external calls which come from the client not from the application itself. 

Actual Results

• The mechanism to determine if it’s an internal call is pretty specific. Rate-limiting logic just checks the following *.applinks, some gagdgets,Mywork, and REST Auth calls and consider them as an Internal call. 

                  "/**/rest/applinks/**",
                  "/**/rest/capabilities",
                  "/**/rest/capabilities/navigation",
                  "/**/rest/gadgets/1.0/g/**",
                  "/**/rest/mywork/latest/status/notification/count",
                  "/**/rest/jira-metadata/1.0/metadata/cache",
                  "/**/rest/api/content",
                  "/**/rest/quickreload/latest/0",
                  "/**/rest/mywork/1/client",
                  "/**/rest/token-auth/api/**"
  

• Other  REST calls are considered as the external calls, same as for this one /rest/auditing/latest/statistics/database/usage and others and blocking them as per the rate limiting feature.  

Workaround

We provide a system property to add more whitelist URLs 

-Dcom.atlassian.ratelimiting.whitelisted-url-patterns="/**/rest/auditing/latest/statistics/database/usage" 

using * as a wildcard that matches anything