Due to large request size Jira receives error 400 from Bitbucket and/or Fisheye when refreshing dev panel cache

Issue Summary

Due to large request size Jira receives error 400 (bad request) from Bitbucket and/or Fisheye when refreshing dev panel cache.

Steps to Reproduce

1. Integrate Bitbucket and / or Fisheye and Jira using application links
2. Execute GET request against <bitbucket_url>/rest/remote-link-aggregation/latest/aggregation endpoint that is longer than 8k in size:

   /rest/remote-link-aggregation/latest/aggregation?globalId=TEST-6224&globalId=TEST3-6224&globalId=TEST4-216&globalId=TEST2-18464&globalId=TEST-6543&globalId=TEST3-6543&globalId=TEST5-931&globalId=TEST1-674&globalId=TEST1-675&globalId=TEST2-17213&globalId=TEST2-14338&globalId=TEST10-1126&globalId=TEST11-1126&globalId=TEST2-18466&globalId=TEST5-1625&globalId=TEST5-2285&globalId=TEST5-2286&globalId=TEST5-2287&globalId=TEST5-2727&globalId==TEST13-215&globalId==TEST13-216&globalId=TEST1-620&globalId=TEST1-621&globalId=TEST5-1633&globalId=TEST5-1634&globalId=TEST6-1006&globalId=TEST5-1635&globalId=TEST2-17898&globalId=TEST14-536&globalId=TEST2-15631&globalId=TEST5-661&globalId=TEST2-17222&globalId=TEST4-341&globalId==TEST12-772&globalId=TEST-6246&globalId=TEST3-6246&globalId=TEST-5570&globalId=TEST3-5570&globalId=TEST1-452&globalId=TEST2-14347&globalId=TEST1-622&globalId=TEST2-17907&globalId=TEST-4506&globalId=TEST3-4506&globalId=TEST14-609&globalId=TEST5-1251&globalId=TEST2-18837&globalId=TEST5-1253&globalId=TEST4-266&globalId=TEST2-14351&globalId=TEST2-17913&globalId=TEST2-17914&globalId=TEST5-2736&globalId=TEST-6254&globalId=TEST3-6254&globalId=TEST1-719&globalId=TEST2-17235&globalId=TEST-4878&globalId=TEST3-4878&globalId=TEST2-18477&globalId=TEST-7177&globalId=TEST3-7177&globalId=TEST2-18478&globalId=TEST1-739&globalId=TEST5-3128&globalId=TEST-2058&globalId=TEST3-2058&globalId=TEST1-720&globalId=TEST2-16660&globalId=TEST2-18485&globalId=TEST-6560&globalId=TEST3-6560&globalId=TEST5-592&globalId=TEST-5577&globalId=TEST3-5577&globalId=TEST-6561&globalId=TEST3-6561&globalId=TEST5-593&globalId=TEST2-14368&globalId=TEST4-179&globalId=TEST2-17925&globalId=TEST7-163&globalId=TEST6-854&globalId=TEST-1606&globalId=TEST3-1606&globalId=TEST5-726&globalId=TEST1-740&globalId=TEST2-12037&globalId=TEST2-17249&globalId=TEST-7282&globalId=TEST3-7282&globalId=TEST16-541&globalId=TEST17-541&globalId=TEST18-541&globalId=TEST2-14387&globalId=TEST-7183&globalId=TEST3-7183&globalId=TEST2-17253&globalId=TEST2-17254&globalId=TEST2-18849&globalId=TEST-7184&globalId=TEST3-7184&globalId=TEST4-325&globalId=TEST2-18497&globalId=TEST2-16673&globalId=TEST2-16674&globalId=TEST2-17941&globalId=TEST2-16675&globalId=TEST2-16676&globalId=TEST1-742&globalId=TEST2-16678&globalId=TEST2-16679&globalId=TEST2-18851&globalId=TEST2-16680&globalId=TEST2-17944&globalId=TEST2-16681&globalId=TEST7-564&globalId=TEST2-17261&globalId=TEST2-18852&globalId=TEST2-16691&globalId=TEST-7058&globalId=TEST3-7058&globalId==TEST13-217&globalId=TEST14-507&globalId=TEST5-1680&globalId=TEST-5582&globalId=TEST3-5582&globalId=TEST5-1683&globalId=TEST1-455&globalId=TEST2-17273&globalId=TEST-6832&globalId=TEST3-6832&globalId=TEST1-571&globalId=TEST5-3185&globalId=TEST7-490&globalId=TEST2-17959&globalId=TEST2-17297&globalId=TEST5-2340&globalId=TEST6-1016&globalId==TEST13-218&globalId=TEST2-17299&globalId=TEST2-17300&globalId=TEST5-1690&globalId=TEST5-3191&globalId=TEST14-494&globalId=TEST-6576&globalId=TEST3-6576&globalId=TEST5-1044&globalId=TEST2-17303&globalId=TEST6-1237&globalId=TEST6-1238&globalId=TEST6-1239&globalId==TEST13-193&globalId==TEST13-194&globalId=TEST2-17963&globalId=TEST2-17311&globalId=TEST-851&globalId=TEST3-851&globalId=TEST7-491&globalId=TEST-6579&globalId=TEST3-6579&globalId=TEST14-538&globalId=TEST14-539&globalId=TEST7-221&globalId=TEST2-18872&globalId=TEST14-541&globalId=TEST7-224&globalId=TEST5-1707&globalId=TEST5-3199&globalId=TEST7-226&globalId=TEST10-1130&globalId=TEST11-1130&globalId=TEST5-1709&globalId=TEST2-17319&globalId=TEST5-2349&globalId=TEST2-17320&globalId=TEST5-1710&globalId=TEST5-2350&globalId=TEST5-1712&globalId=TEST5-1300&globalId=TEST2-17322&globalId=TEST5-2357&globalId=TEST-879&globalId=TEST3-879&globalId=TEST1-624&globalId=TEST6-1252&globalId=TEST2-17326&globalId=TEST-886&globalId=TEST3-886&globalId=TEST7-252&globalId=TEST2-14483&globalId=TEST2-15702&globalId=TEST2-17327&globalId=TEST5-3208&globalId=TEST7-452&globalId=TEST8-971&globalId=TEST-4200&globalId=TEST3-4200&globalId=TEST-902&globalId=TEST3-902&globalId=TEST5-536&globalId=TEST19-84&globalId=TEST2-17340&globalId=TEST-6318&globalId=TEST3-6318&globalId=TEST5-1721&globalId=TEST1-721&globalId=TEST2-13258&globalId=TEST5-1722&globalId=TEST5-1723&globalId=TEST2-17345&globalId=TEST2-17346&globalId=TEST5-1724&globalId=TEST15-19&globalId=TEST6-1257&globalId=TEST5-1725&globalId=TEST2-17349&globalId=TEST2-17996&globalId=TEST1-692&globalId=TEST19-80&globalId=TEST2-18506&globalId=TEST2-18507&globalId=TEST-6939&globalId=TEST3-6939&globalId=TEST5-1731&globalId=TEST2-14513&globalId=TEST5-1726&globalId=TEST6-953&globalId=TEST2-18508&globalId=TEST2-17352&globalId=TEST-7294&globalId=TEST3-7294&globalId=TEST2-17355&globalId=TEST2-18510&globalId=TEST4-255&globalId=TEST4-327&globalId=TEST5-1727&globalId=TEST5-1728&globalId=TEST2-13276&globalId=TEST6-787&globalId=TEST5-2780&globalId=TEST2-18515&globalId=TEST2-13281&globalId=TEST2-17368&globalId==TEST12-570&globalId=TEST2-18519&globalId=TEST2-17371&globalId=TEST-7062&globalId=TEST3-7062&globalId=TEST-987&globalId=TEST3-987&globalId=TEST5-680&globalId=TEST8-527&globalId=TEST7-546&globalId=TEST2-15753&globalId=TEST-3011&globalId=TEST3-3011&globalId=TEST7-549&globalId=TEST2-17374&globalId=TEST19-77&globalId=TEST2-17375&globalId=TEST2-17376&globalId=TEST5-3299&globalId=TEST-6606&globalId=TEST3-6606&globalId=TEST5-2363&globalId=TEST5-2364&globalId=TEST-1002&globalId=TEST3-1002&globalId=TEST5-677&globalId=TEST5-2365&globalId=TEST5-1755&globalId=TEST2-18918&globalId=TEST7-454&globalId=TEST2-17382&globalId=TEST2-14588&globalId=TEST14-610&globalId=TEST-3015&globalId=TEST3-3015&globalId=TEST-3459&globalId=TEST3-3459&globalId=TEST2-17386&globalId=TEST1-633&globalId=TEST5-1776&globalId=TEST5-1777&globalId=TEST14-581&globalId=TEST1-464&globalId=TEST14-582&globalId=TEST8-771&globalId=TEST2-17389&globalId=TEST5-1779&globalId=TEST-6942&globalId=TEST3-6942&globalId=TEST-4576&globalId=TEST3-4576&globalId=TEST2-15792&globalId=TEST6-1023&globalId=TEST2-17392&globalId=TEST8-850&globalId=TEST4-110&globalId=TEST2-18537&globalId=TEST5-1784&globalId=TEST1-723&globalId=TEST-7217&globalId=TEST3-7217&globalId=TEST7-497&globalId=TEST-7219&globalId=TEST3-7219&globalId=TEST-7221&globalId=TEST3-7221&globalId=TEST16-560&globalId=TEST17-560&globalId=TEST18-560&globalId=TEST2-14638&globalId=TEST2-13347&globalId=TEST2-18539&globalId=TEST5-2801&globalId=TEST2-15809&globalId=TEST5-1787&globalId=TEST2-18543&globalId=TEST2-17403&globalId=TEST-4939&globalId=TEST3-4939&globalId=TEST8-853&globalId=TEST5-1789&globalId=TEST5-1790&globalId=TEST-5624&globalId=TEST3-5624&globalId=TEST14-553&globalId=TEST-6333&globalId=TEST3-6333&globalId=TEST2-17408&globalId=TEST5-2376&globalId=TEST2-18043&globalId=TEST5-2378&globalId=TEST2-17409&globalId=TEST6-1093&globalId=TEST2-17413&globalId=TEST-7314&globalId=TEST3-7314&globalId=TEST2-13391&globalId=TEST-6626&globalId=TEST3-6626&globalId=TEST5-1322&globalId=TEST-6341&globalId=TEST3-6341&globalId=TEST7-552&globalId=TEST5-2814&globalId=TEST1-466&globalId=TEST-6344&globalId=TEST3-6344&globalId=TEST2-16768&globalId=TEST10-1323&globalId=TEST11-1323&globalId=TEST5-1796&globalId=TEST5-1797&globalId=TEST4-328&globalId=TEST-7070&globalId=TEST3-7070&globalId=TEST2-18049&globalId=TEST2-14716&globalId=TEST5-2826&globalId=TEST2-15876&globalId=TEST2-15883&globalId=TEST5-620&globalId=TEST5-2836&globalId=TEST5-2837&globalId=TEST5-1812&globalId=TEST1-724&globalId=TEST4-223&globalId==TEST12-394&globalId==TEST12-395&globalId==TEST12-396&globalId==TEST12-401&globalId==TEST12-404&globalId=TEST2-15897&globalId=TEST5-1818&globalId=TEST5-1820&globalId=TEST2-13450&globalId=TEST5-2397&globalId=TEST2-16792&globalId=TEST5-2398&globalId=TEST5-2399&globalId=TEST5-2400&globalId=TEST-241&globalId=TEST3-241&globalId=TEST5-2401&globalId=TEST-4626&globalId=TEST3-4626&globalId=TEST4-256&globalId=TEST2-18053&globalId=TEST1-500&globalId=TEST7-269&globalId=TEST7-500&globalId=TEST8-792&globalId=TEST2-18055&globalId=TEST5-1842&globalId=TEST5-1843&globalId=TEST14-611&globalId=TEST2-18057&globalId=TEST5-1848&globalId=TEST5-2404&globalId=TEST-7087&globalId=TEST3-7087&globalId=TEST4-257&globalId=TEST-4232&globalId=TEST3-4232&globalId=TEST1-665&globalId=TEST2-18062&globalId=TEST6-1031&globalId=TEST2-18063&globalId=TEST6-1105&globalId=TEST2-18065&globalId=TEST5-1855&globalId=TEST6-1108&globalId=TEST5-1856&globalId=TEST5-1857&globalId=TEST2-18069&globalId=TEST-7088&globalId=TEST3-7088&globalId=TEST-1108&globalId=TEST3-1108&globalId=TEST-7089&globalId=TEST3-7089&globalId=TEST5-2868&globalId=TEST8-550&globalId=TEST7-379&globalId==TEST12-793&globalId=TEST5-1863&globalId=TEST14-431&globalId=TEST5-1864&globalId=TEST7-506&globalId=TEST7-507&globalId=TEST5-2870&globalId=TEST5-1865&globalId=TEST7-270&globalId=TEST5-2872&globalId=TEST5-2873&globalId=TEST2-16834&globalId=TEST2-18080&globalId=TEST5-2408&"

• • or wait until Jira requests an update automatically.

Expected Results

• In case when manual cURL request is executed, Bitbucket and / or Fisheye returns the results properly.
• In case when Jira requests an update, the dev panel is refreshed and no error is logged in the logs.

Actual Results

• After executing the cURL request, Bitbucket and / or Fisheye the following response is received by Jira:

  </head><body><h1>HTTP Status 400 – Bad Request</h1><hr class="line" /><p><b>Type</b> Exception Report</p><p><b>Message</b> Request header is too large</p><p><b>Description</b> The server cannot or will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing).</p><p><b>Exception</b></p><pre>java.lang.IllegalArgumentException: Request header is too large
  

• When Jira requests an update, one of the following errors get logged in Jira side:
  • Example 1:

    2019-06-06 21:16:34,102 Caesium-1-3 ERROR ServiceRunner     [c.a.j.p.devstatus.provider.DefaultDevSummaryPollService] Refresh failure
    com.atlassian.jira.plugin.devstatus.provider.DataProviderRefreshFailure: Data Provider refresh failed with error code -1 and message - com.atlassian.sal.api.net.ResponseException: org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected]
    	at com.atlassian.jira.plugin.devstatus.provider.DefaultCachingProviderHelper.refreshProvider(DefaultCachingProviderHelper.java:79)
    	at com.atlassian.jira.plugin.devstatus.provider.DefaultDevSummaryPollService.handlePollingSuccess(DefaultDevSummaryPollService.java:69)
    	at com.atlassian.jira.plugin.devstatus.provider.DefaultDevSummaryPollService.lambda$performPull$1(DefaultDevSummaryPollService.java:51)
    	at com.atlassian.jira.plugin.devstatus.provider.source.applink.PollResult$PollResultSuccess.fold(PollResult.java:51)
    	at com.atlassian.jira.plugin.devstatus.provider.DefaultDevSummaryPollService.performPull(DefaultDevSummaryPollService.java:47)
    	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
    

• • Example 2:

    2020-05-04 17:06:47,242+0200 Caesium-1-2 ERROR anonymous     [c.a.j.p.devstatus.provider.DefaultDevSummaryPollService] Refresh failure
    com.atlassian.jira.plugin.devstatus.provider.DataProviderRefreshFailure: Data Provider [Id: aec41341-413c-35bd-b05e-4ae260594c8c, Type: stash] refresh failed with error code -1 and message - com.atlassian.sal.api.net.ResponseException: javax.net.ssl.SSLProtocolException: Connection reset]
    	at com.atlassian.jira.plugin.devstatus.provider.DefaultCachingProviderHelper.refreshProvider(DefaultCachingProviderHelper.java:78)
    	at com.atlassian.jira.plugin.devstatus.provider.DefaultDevSummaryPollService.handlePollingSuccess(DefaultDevSummaryPollService.java:69)
    	at com.atlassian.jira.plugin.devstatus.provider.DefaultDevSummaryPollService.lambda$performPull$1(DefaultDevSummaryPollService.java:51)
    	at com.atlassian.jira.plugin.devstatus.provider.source.applink.PollResult$PollResultSuccess.fold(PollResult.java:51)
    

• • Example 3:

    2020-04-21 23:29:46,249+0200 Caesium-1-2 ERROR ServiceRunner [c.a.j.p.devstatus.provider.DefaultDevSummaryPollService] Refresh failure
    com.atlassian.jira.plugin.devstatus.provider.DataProviderRefreshFailure: Data Provider [Id: 3817a292-a5ca-3d31-8e6c-78ce507f0da4, Type: stash] refresh failed with error code 414 and message - HTTP status 414 Request-URI Too Large]
    at com.atlassian.jira.plugin.devstatus.provider.DefaultCachingProviderHelper.refreshProvider(DefaultCachingProviderHelper.java:78)
    

Note

The Dev status plugin has a guaranteed delivery feature, so in case Jira does not receive an update from Bitbucket and / or Fisheye for some time, Jira reaches out to Bitbucket and / or Fisheye and requests an update, to which Bitbucket and / or Fisheye replies with a list of issues that should be modified. Then, Jira reaches out again to seek all of the updates (which is what is failing in this case).

Workarounds

Option 1 (preferred)

Decrease the fetch size on Jira side for guaranteed delivery, so even if some fetches induced by webhooks fail, guaranteed to deliver will deliver them later. For doing that:

1. On Bitbucket side add the following into bitbucket.properties file:

   plugin.jira-development-integration.issues.updated.max=200
   

1. Adjust this number up and down to find the maximum fetch size that doesn't fail for your particular case, remembering that issue keys vary in length so there may be more or less output added to the URL.
2. Restart Bitbucket

Option 2 - change headers size

If there is a reverse-proxy in front of Bitbucket and / or Fisheye, follow both Step 1 and Step 2. Otherwise, follow only Step 1.

Step 1

1. On Bitbucket server, increase Tomcat header size by editing bitbucket.properties and adding:

   server.max-http-header-size=32768
   

   On Fisheye side, increase Jetty header size by setting the jetty.http.requestHeaderSize system property with the value 32768.

1. Restart Bitbucket and / or Fisheye
2. This will change Tomcat's / Jetty's max HTTP header size to 32kb. If you still experience the error, you can set higher header size.

References:

• Bitbucket Server config properties - Server Connectors
• Fisheye JVM System Properties

Step 2

If you are using a reverse-proxy, you also need to make sure to set request line limit to the same size (32768 or larger):

• Apache: http://httpd.apache.org/docs/2.2/mod/core.html#limitrequestline
• Nginx: http://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers