Medium Issue Summary
This is reproducible on Data Center: yes
When trying to authenticate a new user (oauth) from, for example, the Jira dev panel so the user can navigate from Jira to Bitbucket, the authentication times out, as the call takes more than 10s which exceeds Jira's 10s HTTP timeout.
This can also happen for other applinked integrations, for example Bamboo to Bitbucket.
Steps to Reproduce
This issue can manifest in big, busy instances with thousands of tokens in the plugin_settings table.
- From the Jira development panel, click on commits/branches/pull requests
- Click on Authenticate to see related [commits/branches...]
- After a few seconds we get the following message: "Unable to retrieve commit information"
Expected Results
The OAuth dance completes in a timely manner (< 10s), and the user is directed to the relevant commit/branch/PR in Bitbucket.
Actual Results
In Jira, the below exception is thrown in the atlassian-jira.log file:
2024-07-24 02:39:04,608-0700 https-jsse-nio-8443-exec-8 ERROR xxxx 158x9324247x2 60sow1 xxxxxxxxxxx /plugins/servlet/applinks/oauth/login-dance/authorize [c.a.a.oauth.auth.OAuthApplinksServlet] An error occurred when performing the oauth 'dance' for application link ' Bitbucket (<applinkID>) <BITBUCKET_URL> stash' com.atlassian.sal.api.net.ResponseException: java.net.SocketTimeoutException: Read timed out at com.atlassian.sal.core.net.HttpClientRequest.executeAndReturn(HttpClientRequest.java:122) at com.atlassian.plugins.rest.module.jersey.JerseyRequest.executeAndReturn(JerseyRequest.java:131) at com.atlassian.plugins.rest.module.jersey.JerseyRequest.execute(JerseyRequest.java:113) at jdk.internal.reflect.GeneratedMethodAccessor14941.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566)
In addition, one sees that the POST consistently takes > 10 s in the atlassian-bitbucket-access-<date>.log on the Bitbucket side:
grep -r -h "POST /plugins/servlet/oauth/request-token HTTP/1.1" | grep "o\*" xxxx | https | o*17RF874x165x7917258x6 | - | 2024-07-24 02:46:15,812 | "POST /plugins/servlet/oauth/request-token HTTP/1.1" | "" "Apache-HttpClient/4.5.14 (Java/11.0.23)" | 200 | 0 | 126 | - | 16843 | - | xxxx | https | o*17RF874x127x7874173x4 | - | 2024-07-24 02:07:20,533 | "POST /plugins/servlet/oauth/request-token HTTP/1.1" | "" "Apache-HttpClient/4.5.14 (Java/11.0.23)" | 200 | 0 | 126 | - | 18973 | - | xxxx | https | o*17RF874x156x7905843x5 | - | 2024-07-24 02:36:30,878 | "POST /plugins/servlet/oauth/request-token HTTP/1.1" | "" "Apache-HttpClient/4.5.14 (Java/11.0.23)" | 200 | 0 | 126 | - | 16873 | - | xxxxx | https | o*ADDHEFx125x8282318x0 | - | 2024-07-24 02:05:30,744 | "POST /plugins/servlet/oauth/request-token HTTP/1.1" | "" "Apache-HttpClient/4.5.14 (Java/11.0.23)" | 200 | 0 | 126 | - | 16277 | - | xxxxx | https | o*ADDHEFx127x8283814x0 | - | 2024-07-24 02:07:41,705 | "POST /plugins/servlet/oauth/request-token HTTP/1.1" | "" "Apache-HttpClient/4.5.14 (Java/11.0.23)" | 200 | 0 | 126 | - | 15417 | - | xxxxx | https | o*ADDHEFx158x8300527x2 | - | 2024-07-24 02:39:08,184 | "POST /plugins/servlet/oauth/request-token HTTP/1.1" | "" "Apache-HttpClient/4.5.14 (Java/11.0.23)" | 200 | 0 | 126 | - | 13585 | - | xxxxx | https | o*1SOG8G6x124x8095014x0 | - | 2024-07-24 02:04:52,876 | "POST /plugins/servlet/oauth/request-token HTTP/1.1" | "" "Apache-HttpClient/4.5.14 (Java/11.0.23)" | 200 | 0 | 126 | - | 16706 | - | xxxxx | https | o*1SOG8G6x127x8097364x2 | - | 2024-07-24 02:07:51,878 | "POST /plugins/servlet/oauth/request-token HTTP/1.1" | "" "Apache-HttpClient/4.5.14 (Java/11.0.23)" | 200 | 0 | 126 | - | 17563 | - | xxxxx | https | o*1SOG8G6x131x8100833x3 | - | 2024-07-24 02:11:50,166 | "POST /plugins/servlet/oauth/request-token HTTP/1.1" | "" "Apache-HttpClient/4.5.14 (Java/11.0.23)" | 200 | 0 | 126 | - | 17669 | - |
In atlassian-profiler-<date>.log we can see entries such as:
xxxxxxx| https-jsse-nio-8443-exec-30 | *ADDHEFx158x8300527x2 | - | - [13584.6ms] - "POST /plugins/servlet/oauth/request-token HTTP/1.1" [7694.0ms] [count: 5917, avg: 1.3ms] - String com.atlassian.stash.internal.plugin.PluginSettingDao.get(String,String)
xxxxx | http-nio-7990-exec-85 | *B4FBWTx251x8709430x51 | - | - [17150.9ms] - "POST /plugins/servlet/oauth/request-token HTTP/1.1" [8757.7ms] [count: 9784, avg: 0.9ms] - String com.atlassian.stash.internal.plugin.PluginSettingDao.get(String,String)
Workaround
On each Jira node:
- From <jira-install>/bin, open setenv.sh.
- Find the section JVM_SUPPORT_RECOMMENDED_ARGS=
- Add the following to the supported arguments
-Dhttp.socketTimeout=30000
- restart the jira nodes.
