Issue Summary

Confluence experiences sudden thread pool exhaustion after running stably for a period of time.

• Occurrence & Timing: The issue may occur when a multiple of 24 hours has elapsed since Confluence was started. There is currently no evidence linking this to scheduled jobs. In customer' case, this problem occurs at 4:37 when customer had a restart time at 4:30. However, they customer delayed the restart time to  4:40, this problem  occurs at 4:47.

• Impact: When the issue occurs, the number of stuck threads increases rapidly until the HTTP thread pool is completely exhausted, rendering Confluence inaccessible to end users.
• Recovery & Recurrence: Confluence cannot self-recover from this state; a restart is required as a temporary workaround. However, the issue tends to recur periodically (approximately once a month) following a restart.
• Other information:   In thread dump , looks all stuck thread are waiting for a lock object, but owner of the lock object is not output. Sometimes , dead locks show in the thread dump. 
• Cause: When Jira integration features call Jira through the Application Links proxy and Jira responds with a redirect, Confluence follows it with no limit and appends the root-relative Location to the Application Link URL, doubling the context path. If the doubled URL is redirected again, the request loops until the thread fails with a StackOverflowError. Occasionally this leaves a lock in Confluence's cache-settings cache held, and the outage starts when the cache entries loaded at startup expire, at start time + N × 24 hours.

• Affected configurations: The following configuration can produce the duplicated-context-path redirect scenario:
  • Confluence has an Application Link to Jira using a URL with a context path, such as https://jira.example.com/jira.
  • Jira, an SSO filter, or a reverse proxy returns an origin-relative redirect containing that context path, such as Location: /jira/login.jsp
  • The requests to doubled path (/jira/jira/...) is also redirected, allowing the chain to continue rather than terminating.

Steps to Reproduce

1. Create an Application Link from Confluence to Jira using a URL with a context path, e.g. https://jira.example.com/jira.
2. Configure Jira, or a proxy in front of it, to redirect unauthenticated requests (including unknown paths) to /jira/login.jsp.
3. Sign in to Confluence as a user with permission to access the application. For the anonymous-fallback scenario, use a user who has not authorized the Application Link.
4. Request the following endpoint, relative to the Confluence base URL: /plugins/servlet/applinks/proxy?appId=<applicationLinkID>&path=/secure/ViewProfile.jspa
5. Inspect the outbound requests, response status codes, and Location headers to verify that the intended redirect loop occurs.
6. Check the Confluence application log for a StackOverflowError with repeated proxy response-handler and retryRequest frames.

Expected Results

Confluence should maintain stable, normal operation continuously over time without requiring periodic restarts. A redirect from Jira should be resolved against the requested URL, and a redirect loop should fail only that request.

Actual Results

Confluence requests /jira/jira/login.jsp and keeps following redirects until the request fails StackOverflowError.
If the error leaves a cache lock held, Confluence exhausts its thread pool at the next start time + N × 24 hours. Thread dumps taken with jstack -l then show HTTP threads waiting in LocalCache$Segment.lockedGetOrLoad, called from CachingCacheSettingsManager.getDefaults.

Workaround

Where the Application Link URL points (reverse proxy, load balancer or Jira), return an immediate HTTP 403 or 404 for the identified duplicated-context-path requests. Do not redirect it, as that keeps the loop going. 
 
Alternatively, as a temporary mitigation, restart Confluence as soon as possible after a StackOverflowError involving AbstractProxyServlet.retryRequest appears in atlassian-confluence.log