Uploaded image for project: 'Jira Software Server and Data Center'
  1. Jira Software Server and Data Center
  2. JSWSERVER-20190

Due to large request size Jira receives error 400 from Bitbucket when refreshing dev panel cache

    XMLWordPrintable

    Details

      Description

      Issue Summary

      Due to large request size Jira receives error 400 (bad request) from Bitbucket when refreshing dev panel cache.

      Steps to Reproduce

      1. Integrate Bitbucket and Jira using application links
      2. Execute GET request against <bitbucket_url>/rest/remote-link-aggregation/latest/aggregation endpoint that is longer than 8k in size:
        /rest/remote-link-aggregation/latest/aggregation?globalId=TEST-6224&globalId=TEST3-6224&globalId=TEST4-216&globalId=TEST2-18464&globalId=TEST-6543&globalId=TEST3-6543&globalId=TEST5-931&globalId=TEST1-674&globalId=TEST1-675&globalId=TEST2-17213&globalId=TEST2-14338&globalId=TEST10-1126&globalId=TEST11-1126&globalId=TEST2-18466&globalId=TEST5-1625&globalId=TEST5-2285&globalId=TEST5-2286&globalId=TEST5-2287&globalId=TEST5-2727&globalId==TEST13-215&globalId==TEST13-216&globalId=TEST1-620&globalId=TEST1-621&globalId=TEST5-1633&globalId=TEST5-1634&globalId=TEST6-1006&globalId=TEST5-1635&globalId=TEST2-17898&globalId=TEST14-536&globalId=TEST2-15631&globalId=TEST5-661&globalId=TEST2-17222&globalId=TEST4-341&globalId==TEST12-772&globalId=TEST-6246&globalId=TEST3-6246&globalId=TEST-5570&globalId=TEST3-5570&globalId=TEST1-452&globalId=TEST2-14347&globalId=TEST1-622&globalId=TEST2-17907&globalId=TEST-4506&globalId=TEST3-4506&globalId=TEST14-609&globalId=TEST5-1251&globalId=TEST2-18837&globalId=TEST5-1253&globalId=TEST4-266&globalId=TEST2-14351&globalId=TEST2-17913&globalId=TEST2-17914&globalId=TEST5-2736&globalId=TEST-6254&globalId=TEST3-6254&globalId=TEST1-719&globalId=TEST2-17235&globalId=TEST-4878&globalId=TEST3-4878&globalId=TEST2-18477&globalId=TEST-7177&globalId=TEST3-7177&globalId=TEST2-18478&globalId=TEST1-739&globalId=TEST5-3128&globalId=TEST-2058&globalId=TEST3-2058&globalId=TEST1-720&globalId=TEST2-16660&globalId=TEST2-18485&globalId=TEST-6560&globalId=TEST3-6560&globalId=TEST5-592&globalId=TEST-5577&globalId=TEST3-5577&globalId=TEST-6561&globalId=TEST3-6561&globalId=TEST5-593&globalId=TEST2-14368&globalId=TEST4-179&globalId=TEST2-17925&globalId=TEST7-163&globalId=TEST6-854&globalId=TEST-1606&globalId=TEST3-1606&globalId=TEST5-726&globalId=TEST1-740&globalId=TEST2-12037&globalId=TEST2-17249&globalId=TEST-7282&globalId=TEST3-7282&globalId=TEST16-541&globalId=TEST17-541&globalId=TEST18-541&globalId=TEST2-14387&globalId=TEST-7183&globalId=TEST3-7183&globalId=TEST2-17253&globalId=TEST2-17254&globalId=TEST2-18849&globalId=TEST-7184&globalId=TEST3-7184&globalId=TEST4-325&globalId=TEST2-18497&globalId=TEST2-16673&globalId=TEST2-16674&globalId=TEST2-17941&globalId=TEST2-16675&globalId=TEST2-16676&globalId=TEST1-742&globalId=TEST2-16678&globalId=TEST2-16679&globalId=TEST2-18851&globalId=TEST2-16680&globalId=TEST2-17944&globalId=TEST2-16681&globalId=TEST7-564&globalId=TEST2-17261&globalId=TEST2-18852&globalId=TEST2-16691&globalId=TEST-7058&globalId=TEST3-7058&globalId==TEST13-217&globalId=TEST14-507&globalId=TEST5-1680&globalId=TEST-5582&globalId=TEST3-5582&globalId=TEST5-1683&globalId=TEST1-455&globalId=TEST2-17273&globalId=TEST-6832&globalId=TEST3-6832&globalId=TEST1-571&globalId=TEST5-3185&globalId=TEST7-490&globalId=TEST2-17959&globalId=TEST2-17297&globalId=TEST5-2340&globalId=TEST6-1016&globalId==TEST13-218&globalId=TEST2-17299&globalId=TEST2-17300&globalId=TEST5-1690&globalId=TEST5-3191&globalId=TEST14-494&globalId=TEST-6576&globalId=TEST3-6576&globalId=TEST5-1044&globalId=TEST2-17303&globalId=TEST6-1237&globalId=TEST6-1238&globalId=TEST6-1239&globalId==TEST13-193&globalId==TEST13-194&globalId=TEST2-17963&globalId=TEST2-17311&globalId=TEST-851&globalId=TEST3-851&globalId=TEST7-491&globalId=TEST-6579&globalId=TEST3-6579&globalId=TEST14-538&globalId=TEST14-539&globalId=TEST7-221&globalId=TEST2-18872&globalId=TEST14-541&globalId=TEST7-224&globalId=TEST5-1707&globalId=TEST5-3199&globalId=TEST7-226&globalId=TEST10-1130&globalId=TEST11-1130&globalId=TEST5-1709&globalId=TEST2-17319&globalId=TEST5-2349&globalId=TEST2-17320&globalId=TEST5-1710&globalId=TEST5-2350&globalId=TEST5-1712&globalId=TEST5-1300&globalId=TEST2-17322&globalId=TEST5-2357&globalId=TEST-879&globalId=TEST3-879&globalId=TEST1-624&globalId=TEST6-1252&globalId=TEST2-17326&globalId=TEST-886&globalId=TEST3-886&globalId=TEST7-252&globalId=TEST2-14483&globalId=TEST2-15702&globalId=TEST2-17327&globalId=TEST5-3208&globalId=TEST7-452&globalId=TEST8-971&globalId=TEST-4200&globalId=TEST3-4200&globalId=TEST-902&globalId=TEST3-902&globalId=TEST5-536&globalId=TEST19-84&globalId=TEST2-17340&globalId=TEST-6318&globalId=TEST3-6318&globalId=TEST5-1721&globalId=TEST1-721&globalId=TEST2-13258&globalId=TEST5-1722&globalId=TEST5-1723&globalId=TEST2-17345&globalId=TEST2-17346&globalId=TEST5-1724&globalId=TEST15-19&globalId=TEST6-1257&globalId=TEST5-1725&globalId=TEST2-17349&globalId=TEST2-17996&globalId=TEST1-692&globalId=TEST19-80&globalId=TEST2-18506&globalId=TEST2-18507&globalId=TEST-6939&globalId=TEST3-6939&globalId=TEST5-1731&globalId=TEST2-14513&globalId=TEST5-1726&globalId=TEST6-953&globalId=TEST2-18508&globalId=TEST2-17352&globalId=TEST-7294&globalId=TEST3-7294&globalId=TEST2-17355&globalId=TEST2-18510&globalId=TEST4-255&globalId=TEST4-327&globalId=TEST5-1727&globalId=TEST5-1728&globalId=TEST2-13276&globalId=TEST6-787&globalId=TEST5-2780&globalId=TEST2-18515&globalId=TEST2-13281&globalId=TEST2-17368&globalId==TEST12-570&globalId=TEST2-18519&globalId=TEST2-17371&globalId=TEST-7062&globalId=TEST3-7062&globalId=TEST-987&globalId=TEST3-987&globalId=TEST5-680&globalId=TEST8-527&globalId=TEST7-546&globalId=TEST2-15753&globalId=TEST-3011&globalId=TEST3-3011&globalId=TEST7-549&globalId=TEST2-17374&globalId=TEST19-77&globalId=TEST2-17375&globalId=TEST2-17376&globalId=TEST5-3299&globalId=TEST-6606&globalId=TEST3-6606&globalId=TEST5-2363&globalId=TEST5-2364&globalId=TEST-1002&globalId=TEST3-1002&globalId=TEST5-677&globalId=TEST5-2365&globalId=TEST5-1755&globalId=TEST2-18918&globalId=TEST7-454&globalId=TEST2-17382&globalId=TEST2-14588&globalId=TEST14-610&globalId=TEST-3015&globalId=TEST3-3015&globalId=TEST-3459&globalId=TEST3-3459&globalId=TEST2-17386&globalId=TEST1-633&globalId=TEST5-1776&globalId=TEST5-1777&globalId=TEST14-581&globalId=TEST1-464&globalId=TEST14-582&globalId=TEST8-771&globalId=TEST2-17389&globalId=TEST5-1779&globalId=TEST-6942&globalId=TEST3-6942&globalId=TEST-4576&globalId=TEST3-4576&globalId=TEST2-15792&globalId=TEST6-1023&globalId=TEST2-17392&globalId=TEST8-850&globalId=TEST4-110&globalId=TEST2-18537&globalId=TEST5-1784&globalId=TEST1-723&globalId=TEST-7217&globalId=TEST3-7217&globalId=TEST7-497&globalId=TEST-7219&globalId=TEST3-7219&globalId=TEST-7221&globalId=TEST3-7221&globalId=TEST16-560&globalId=TEST17-560&globalId=TEST18-560&globalId=TEST2-14638&globalId=TEST2-13347&globalId=TEST2-18539&globalId=TEST5-2801&globalId=TEST2-15809&globalId=TEST5-1787&globalId=TEST2-18543&globalId=TEST2-17403&globalId=TEST-4939&globalId=TEST3-4939&globalId=TEST8-853&globalId=TEST5-1789&globalId=TEST5-1790&globalId=TEST-5624&globalId=TEST3-5624&globalId=TEST14-553&globalId=TEST-6333&globalId=TEST3-6333&globalId=TEST2-17408&globalId=TEST5-2376&globalId=TEST2-18043&globalId=TEST5-2378&globalId=TEST2-17409&globalId=TEST6-1093&globalId=TEST2-17413&globalId=TEST-7314&globalId=TEST3-7314&globalId=TEST2-13391&globalId=TEST-6626&globalId=TEST3-6626&globalId=TEST5-1322&globalId=TEST-6341&globalId=TEST3-6341&globalId=TEST7-552&globalId=TEST5-2814&globalId=TEST1-466&globalId=TEST-6344&globalId=TEST3-6344&globalId=TEST2-16768&globalId=TEST10-1323&globalId=TEST11-1323&globalId=TEST5-1796&globalId=TEST5-1797&globalId=TEST4-328&globalId=TEST-7070&globalId=TEST3-7070&globalId=TEST2-18049&globalId=TEST2-14716&globalId=TEST5-2826&globalId=TEST2-15876&globalId=TEST2-15883&globalId=TEST5-620&globalId=TEST5-2836&globalId=TEST5-2837&globalId=TEST5-1812&globalId=TEST1-724&globalId=TEST4-223&globalId==TEST12-394&globalId==TEST12-395&globalId==TEST12-396&globalId==TEST12-401&globalId==TEST12-404&globalId=TEST2-15897&globalId=TEST5-1818&globalId=TEST5-1820&globalId=TEST2-13450&globalId=TEST5-2397&globalId=TEST2-16792&globalId=TEST5-2398&globalId=TEST5-2399&globalId=TEST5-2400&globalId=TEST-241&globalId=TEST3-241&globalId=TEST5-2401&globalId=TEST-4626&globalId=TEST3-4626&globalId=TEST4-256&globalId=TEST2-18053&globalId=TEST1-500&globalId=TEST7-269&globalId=TEST7-500&globalId=TEST8-792&globalId=TEST2-18055&globalId=TEST5-1842&globalId=TEST5-1843&globalId=TEST14-611&globalId=TEST2-18057&globalId=TEST5-1848&globalId=TEST5-2404&globalId=TEST-7087&globalId=TEST3-7087&globalId=TEST4-257&globalId=TEST-4232&globalId=TEST3-4232&globalId=TEST1-665&globalId=TEST2-18062&globalId=TEST6-1031&globalId=TEST2-18063&globalId=TEST6-1105&globalId=TEST2-18065&globalId=TEST5-1855&globalId=TEST6-1108&globalId=TEST5-1856&globalId=TEST5-1857&globalId=TEST2-18069&globalId=TEST-7088&globalId=TEST3-7088&globalId=TEST-1108&globalId=TEST3-1108&globalId=TEST-7089&globalId=TEST3-7089&globalId=TEST5-2868&globalId=TEST8-550&globalId=TEST7-379&globalId==TEST12-793&globalId=TEST5-1863&globalId=TEST14-431&globalId=TEST5-1864&globalId=TEST7-506&globalId=TEST7-507&globalId=TEST5-2870&globalId=TEST5-1865&globalId=TEST7-270&globalId=TEST5-2872&globalId=TEST5-2873&globalId=TEST2-16834&globalId=TEST2-18080&globalId=TEST5-2408&"
        • or wait until Jira requests an update automatically.

      Expected Results

      • In case when manual Curl request is executed, Bitbucket returns the results properly.
      • In case when Jira requests an update, the dev panel is refreshed and no error is logged in the logs.

      Actual Results

      • After executing Curl response, Bitbucket returns the following response:
        </head><body><h1>HTTP Status 400 – Bad Request</h1><hr class="line" /><p><b>Type</b> Exception Report</p><p><b>Message</b> Request header is too large</p><p><b>Description</b> The server cannot or will not process the request due to something that is perceived to be a client error (e.g., malformed request syntax, invalid request message framing, or deceptive request routing).</p><p><b>Exception</b></p><pre>java.lang.IllegalArgumentException: Request header is too large
        
      • In case when Jira requests an update, 1 of the the following error gets logged in Jira logs:
        • Example 1:
          2019-06-06 21:16:34,102 Caesium-1-3 ERROR ServiceRunner     [c.a.j.p.devstatus.provider.DefaultDevSummaryPollService] Refresh failure
          com.atlassian.jira.plugin.devstatus.provider.DataProviderRefreshFailure: Data Provider refresh failed with error code -1 and message - com.atlassian.sal.api.net.ResponseException: org.apache.http.ConnectionClosedException: Premature end of chunk coded message body: closing chunk expected]
          	at com.atlassian.jira.plugin.devstatus.provider.DefaultCachingProviderHelper.refreshProvider(DefaultCachingProviderHelper.java:79)
          	at com.atlassian.jira.plugin.devstatus.provider.DefaultDevSummaryPollService.handlePollingSuccess(DefaultDevSummaryPollService.java:69)
          	at com.atlassian.jira.plugin.devstatus.provider.DefaultDevSummaryPollService.lambda$performPull$1(DefaultDevSummaryPollService.java:51)
          	at com.atlassian.jira.plugin.devstatus.provider.source.applink.PollResult$PollResultSuccess.fold(PollResult.java:51)
          	at com.atlassian.jira.plugin.devstatus.provider.DefaultDevSummaryPollService.performPull(DefaultDevSummaryPollService.java:47)
          	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
          
        • Example 2:
          2020-05-04 17:06:47,242+0200 Caesium-1-2 ERROR anonymous     [c.a.j.p.devstatus.provider.DefaultDevSummaryPollService] Refresh failure
          com.atlassian.jira.plugin.devstatus.provider.DataProviderRefreshFailure: Data Provider [Id: aec41341-413c-35bd-b05e-4ae260594c8c, Type: stash] refresh failed with error code -1 and message - com.atlassian.sal.api.net.ResponseException: javax.net.ssl.SSLProtocolException: Connection reset]
          	at com.atlassian.jira.plugin.devstatus.provider.DefaultCachingProviderHelper.refreshProvider(DefaultCachingProviderHelper.java:78)
          	at com.atlassian.jira.plugin.devstatus.provider.DefaultDevSummaryPollService.handlePollingSuccess(DefaultDevSummaryPollService.java:69)
          	at com.atlassian.jira.plugin.devstatus.provider.DefaultDevSummaryPollService.lambda$performPull$1(DefaultDevSummaryPollService.java:51)
          	at com.atlassian.jira.plugin.devstatus.provider.source.applink.PollResult$PollResultSuccess.fold(PollResult.java:51)
          
        • Example 3:
          2020-04-21 23:29:46,249+0200 Caesium-1-2 ERROR ServiceRunner [c.a.j.p.devstatus.provider.DefaultDevSummaryPollService] Refresh failure
          com.atlassian.jira.plugin.devstatus.provider.DataProviderRefreshFailure: Data Provider [Id: 3817a292-a5ca-3d31-8e6c-78ce507f0da4, Type: stash] refresh failed with error code 414 and message - HTTP status 414 Request-URI Too Large]
          at com.atlassian.jira.plugin.devstatus.provider.DefaultCachingProviderHelper.refreshProvider(DefaultCachingProviderHelper.java:78)
          

      Note

      The Dev status plugin has a guaranteed delivery feature, so in case Jira does not receive an update from Bitbucket for some time, Jira reaches out to Bitbucket and requests an update, to which Bitbucket replies with a list of issues that should be modified. Then, Jira reaches out again to seek all of the updates (which is what is failing in this case).

      Workaround

      Option 1 (prefered)

      • Decrease fetch size on Jira side for guaranteed delivery, so even if some fetches induced by webhooks fail, guaranteed deliver will deliver them later.
        On Bitbucket side add the following into bitbucket.properties file:
        plugin.jira-development-integration.issues.updated.max=200
        
        • Adjust this number up and down to find the maximum fetch size that doesn't fail for your particular case. Why is it necessary - because issue keys vary in length so there may be more or less issues fitting into URL.
      • Restart Bitbucket

      Option 2 - change headers size

      If there is a reverse-proxy in front of Bitbucket, follow both Step 1 and Step 2. Otherwise, follow only Step 1.

      Step 1

      On Bitbucket server, increase Tomcat header size following the steps below:

      1. Edit bitbucket.properties and add:
        server.max-http-header-size=32768
        
      1. * This will change tomcat's max HTTP header size to 32kb. If you still experience the error, you can set higher header size.
      2. * Reference: Bitbucket Server config properties - Server Connectors
      3. Restart Bitbucket

       When implementing the workaround, if you have server.tomcat.max-http-header-size defined in your bitbucket.properties, you should remove it and replace with server.max-http-header-size. The values are functionally the same, so to avoid conflicts, only one should be present. As server.tomcat.max-http-header-size will be removed in a future version of Bitbucket, we recommend you go with server.max-http-header-size

      Step 2

      If you are using a reverse-proxy, you also need to make sure to set request line limit to the same size (32768 or larger):

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              aermolenko Anton Ermolenko
              Reporter:
              mfilipan Marko Filipan
              Votes:
              40 Vote for this issue
              Watchers:
              72 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: