Uploaded image for project: 'Confluence Server'
  1. Confluence Server
  2. CONFSERVER-45599

Collaborative Editing fails to start with 502 Bad Gateway (or 404) errors for sockjs.min.js, synchrony.min.js when Confluence is running in a zh_CN.UTF-8 (or other non en_GB) locale

    Details

    • Symptom Severity:
      Major
    • Support reference count:
      16
    • Sprint:
      Enterprise is nifty, Enterprise is optimal, Enterprise is pleasing
    • Testing Notes:
      Hide

      Although the fix is for a Windows server, we should test it against Confluence running on Linux or Mac OS to avoid any regressions. Once it's set, collaborative editing should work as expected.

      • To avoid regression, we need to test Confluence against a Windows docker instance.
      • Check with special characters like Chinese

      Fix for locale:

      • will use set locale (US) for http date formats
      • also set synchrony JVM properties user.country and user.language to default values US and en respectively (can be overriden in case of unexpected side effects)

      Potential concerns:

      • Does locale affect how Synchrony interacts with the editor content? Answer should be no, but might need to do some testing to verify. Potentially test Confluence against a docker instance where the locale/display language is Chinese? or configure Bamboo environment to use Chinese locale

      Make sure to carry out a manual test end to end of the collaborative editing feature when the fix is in. i.e. starting Synchrony, Confluence and editing a page.

      Show
      Although the fix is for a Windows server, we should test it against Confluence running on Linux or Mac OS to avoid any regressions. Once it's set, collaborative editing should work as expected. To avoid regression, we need to test Confluence against a Windows docker instance. Check with special characters like Chinese Fix for locale: will use set locale (US) for http date formats also set synchrony JVM properties user.country and user.language to default values US and en respectively (can be overriden in case of unexpected side effects) Potential concerns: Does locale affect how Synchrony interacts with the editor content? Answer should be no, but might need to do some testing to verify. Potentially test Confluence against a docker instance where the locale/display language is Chinese? or configure Bamboo environment to use Chinese locale Make sure to carry out a manual test end to end of the collaborative editing feature when the fix is in. i.e. starting Synchrony, Confluence and editing a page.
    • Occurrence Factor:
      10%
    • QA Demo Status:
      Not Done
    • QA Kickoff Status:
      Done

      Description

      Summary

      When using Confluence 6.0 with Collaborative editing enabled on a Server with non-Western locale we see

      Loading the editor's taking longer than usual. Give it a few moments, then refresh your page if it still doesn't load. Speak to your Confluence admin if that doesn't fix it.

      Environment

      • Language zh_CN.UTF-8 on the server (also seen in Japanese and Korean)
      • No Forward or Reverse Proxy - server.xml is correct with no proxy information (reverse proxy tends to throw 404 errors but is the same cause and resolution)
      • Firewall and anti-virus is turned off/not installed
      • Running on VM
      • no add-ons (i.e. fresh install)
      • Tested access on different networks
      • Accessing Confluence though the default port 8090 (Base URL as http://confluence.mycompany.com:8090/
      • using the Synchrony proxy though /synchrony_proxy

      Root Cause

      Synchrony-proxy uses Jetty Proxy to handle HTTP requests/responses, and it only recognizes a certain range of characters as legal. Synchrony's date formatter uses the user's default locale, so some non-western (in support cases, mostly Chinese, one Korean case), characters are treated as 'illegal', which results in an HTTP protocol violation, i.e. 502 Bad Gateway error in the browser, 400 internally.

      Steps to Reproduce

      Linux
      1. Change the Confluence environment to zh_CN.UTF-8. On Linux, you can do this with
        sudo locale-gen zh_CN.UTF-8
        LANG="zh_CN.UTF-8"
        
      2. When you run date you will see
        017年 01月 10日 星期二 16:48:32 AEDT
        
      3. run start-confluence.sh in this environment
      4. run
        curl -i -N -I http://localhost:8091/synchrony/resources/js/vendor/sockjs.min.js
        

        and check the headers, the date format will be in Chinese, or garbled Chinese

        Date: �, 10 2017 05:56:51 GMT
        
      5. Create a new page in Confluence, error will appear in browser and developer tools / network tab
      Windows 10
      1. Go to "Start Menu" > "Settings" > "Time & language"
      2. Select "Region & language"
      3. Select Chinese (If Chinese is not an option, select 'Add a language' to add it to the list) > "Set as default" (if this option is greyed out, select "Options" instead – you may have to download the language pack) and restart the machine
      4. Start Confluence as normal, and the result should be the red loading banner for collaborative editing and the 502 Gateway errors in the Developer Tools console (if you had Confluence working before, you may have to clear the cache, otherwise the browser will used the cached js files)
      Mac / Windows (and possibly other platforms)

      If using Confluence 6.0.2 or greater, create a new file in the Confluence home directory called synchrony-args.properties and add the following:

      user.country=CN
      user.language=zh
      

      Restart Confluence. You should now get the red loading banner for collaborative editing and see 502 Gateway errors in the console

      Expected Results

      Collaborative editing works as expected.

      Actual Results

      In the HAR files collected we see

              "response": {
                "status": 502,
                "statusText": "Bad Gateway",
                "httpVersion": "HTTP/1.1",
                "headers": [
                  {
                    "name": "Server",
                    "value": "Apache-Coyote/1.1"
                  },
                  {
                    "name": "Connection",
                    "value": "close"
                  },
                  {
                    "name": "Content-Length",
                    "value": "0"
                  },
                  {
                    "name": "Date",
                    "value": "Thu, 15 Dec 2016 05:01:36 GMT"
                  }
                ],
                "cookies": [],
                "content": {
                  "mimeType": "application/x-javascript",
                  "size": 0,
                  "text": ""
                },
                "redirectURL": "",
                "headersSize": 130,
                "bodySize": -1
              },
      
      
      

      atlassian-synchrony-proxy.log has

      2016-12-29 16:10:03 DEBUG synchronyProxyFilter:618 - 1675778603 proxying failed
      org.eclipse.jetty.client.HttpResponseException: HTTP protocol violation: bad response on HttpConnectionOverHTTP@4618ad61(l:/127.0.0.1:52209 <-> r:localhost/127.0.0.1:8091,closed=false)[HttpChannelOverHTTP@5d87150a(exchange=HttpExchange@4dfc1fb5 req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@6c87be00(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@6f776554{s=START}],recv=HttpReceiverOverHTTP@68c92268(rsp=HEADER,failure=null)[HttpParser{s=CLOSE,0 of 17775}]]]
      

      Notes

      • Synchrony starts up and heartbeat is received
        2016-12-09 14:18:00,248 DEBUG [http-nio-8090-exec-8] [plugins.synchrony.bootstrap.DefaultSynchronyMonitor] isSynchronyUp Checking Synchrony heartbeat on: http://localhost:8091/synchrony/heartbeat
        2016-12-09 14:18:00,263 DEBUG [http-nio-8090-exec-8] [plugins.synchrony.bootstrap.DefaultSynchronyMonitor] isSynchronyUp Response received from Synchrony heartbeat: status HTTP/1.1 200 OK
        
      • Requests for the .js files are received
        {"synchrony":{"message":"synchrony.middleware.memoize-response [info] request for static resource [\"0.0.0.0\" \"/synchrony/resources/js/vendor/sockjs.min.js\" nil]","ns":"synchrony.middleware.memoize-response","level":"info"},"message":"synchrony.middleware.memoize-response [info] request for static resource [\"0.0.0.0\" \"/synchrony/resources/js/vendor/sockjs.min.js\" nil]"}
        {"synchrony":{"message":"synchrony.middleware.memoize-response [info] request for static resource [\"0.0.0.0\" \"/synchrony/resources/js/synchrony.min.js\" nil]","ns":"synchrony.middleware.memoize-response","level":"info"},"message":"synchrony.middleware.memoize-response [info] request for static resource [\"0.0.0.0\" \"/synchrony/resources/js/synchrony.min.js\" nil]"}
        

      You can test that Synchrony can be polled with

      curl -i -N -I http://localhost:8091/synchrony/resources/js/vendor/sockjs.min.js
      

      and returns (note the date)

      HTTP/1.1 200 OK
      Access-Control-Allow-Headers: content-type, accept, x-token, x-atlassian-mau-ignore
      Access-Control-Max-Age: 86400
      Content-Encoding: gzip
      Access-Control-Allow-Origin: *
      Content-Type: application/javascript;charset=utf-8
      Content-Length: 17775
      Etag: "pWzRSOOq34MQbGLrouECnppGJ10"
      Date: �, 10 � 2017 04:45:07 GMT
      Vary: Accept-Encoding
      Cache-Control: max-age=300
      Access-Control-Allow-Methods: GET, PUT, POST, OPTIONS
      

      Workaround

      (Upgrade to 6.0.2+ if not already) Override the default locale for Synchrony

      1. Create new file synchrony-args.properties in <home-directory> and add the following lines:
        user.language=en
        user.country=US
        
      2. Restart Confluence for changes to take effect

        Attachments

        1. 502BadGateway.png
          502BadGateway.png
          104 kB
        2. Region&language.png
          Region&language.png
          810 kB
        3. SetAsDefault.png
          SetAsDefault.png
          815 kB
        4. Settings.png
          Settings.png
          2.61 MB
        5. Time&language.png
          Time&language.png
          792 kB

          Issue Links

            Activity

              People

              • Votes:
                4 Vote for this issue
                Watchers:
                20 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Last commented:
                  19 weeks, 4 days ago