Collaborative Editing fails to start with 502 Bad Gateway (or 404) errors for sockjs.min.js, synchrony.min.js when Confluence is running in a zh_CN.UTF-8 (or other non en_GB) locale

Summary

When using Confluence 6.0 with Collaborative editing enabled on a Server with non-Western locale we see

Environment

• Language zh_CN.UTF-8 on the server (also seen in Japanese and Korean)
• No Forward or Reverse Proxy - server.xml is correct with no proxy information (reverse proxy tends to throw 404 errors but is the same cause and resolution)
• Firewall and anti-virus is turned off/not installed
• Running on VM
• no add-ons (i.e. fresh install)
• Tested access on different networks
• Accessing Confluence though the default port 8090 (Base URL as http://confluence.mycompany.com:8090/
• using the Synchrony proxy though /synchrony_proxy

Root Cause

Synchrony-proxy uses Jetty Proxy to handle HTTP requests/responses, and it only recognizes a certain range of characters as legal. Synchrony's date formatter uses the user's default locale, so some non-western (in support cases, mostly Chinese, one Korean case), characters are treated as 'illegal', which results in an HTTP protocol violation, i.e. 502 Bad Gateway error in the browser, 400 internally.

Steps to Reproduce

Linux

1. Change the Confluence environment to zh_CN.UTF-8. On Linux, you can do this with

   sudo locale-gen zh_CN.UTF-8
   LANG="zh_CN.UTF-8"
   

2. When you run date you will see

   017年 01月 10日 星期二 16:48:32 AEDT
   

3. run start-confluence.sh in this environment
4. run

   curl -i -N -I http://localhost:8091/synchrony/resources/js/vendor/sockjs.min.js
   

   and check the headers, the date format will be in Chinese, or garbled Chinese

   Date: �, 10 2017 05:56:51 GMT
   

5. Create a new page in Confluence, error will appear in browser and developer tools / network tab

Windows 10

1. Go to "Start Menu" > "Settings" > "Time & language"
   
2. Select "Region & language"
   
3. Select Chinese (If Chinese is not an option, select 'Add a language' to add it to the list) > "Set as default" (if this option is greyed out, select "Options" instead – you may have to download the language pack) and restart the machine
   
4. Start Confluence as normal, and the result should be the red loading banner for collaborative editing and the 502 Gateway errors in the Developer Tools console (if you had Confluence working before, you may have to clear the cache, otherwise the browser will used the cached js files)

Mac / Windows (and possibly other platforms)

If using Confluence 6.0.2 or greater, create a new file in the Confluence home directory called synchrony-args.properties and add the following:

user.country=CN
user.language=zh

Restart Confluence. You should now get the red loading banner for collaborative editing and see 502 Gateway errors in the console

Expected Results

Collaborative editing works as expected.

Actual Results

In the HAR files collected we see

        "response": {
          "status": 502,
          "statusText": "Bad Gateway",
          "httpVersion": "HTTP/1.1",
          "headers": [
            {
              "name": "Server",
              "value": "Apache-Coyote/1.1"
            },
            {
              "name": "Connection",
              "value": "close"
            },
            {
              "name": "Content-Length",
              "value": "0"
            },
            {
              "name": "Date",
              "value": "Thu, 15 Dec 2016 05:01:36 GMT"
            }
          ],
          "cookies": [],
          "content": {
            "mimeType": "application/x-javascript",
            "size": 0,
            "text": ""
          },
          "redirectURL": "",
          "headersSize": 130,
          "bodySize": -1
        },

atlassian-synchrony-proxy.log has

2016-12-29 16:10:03 DEBUG synchronyProxyFilter:618 - 1675778603 proxying failed
org.eclipse.jetty.client.HttpResponseException: HTTP protocol violation: bad response on HttpConnectionOverHTTP@4618ad61(l:/127.0.0.1:52209 <-> r:localhost/127.0.0.1:8091,closed=false)[HttpChannelOverHTTP@5d87150a(exchange=HttpExchange@4dfc1fb5 req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@6c87be00(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@6f776554{s=START}],recv=HttpReceiverOverHTTP@68c92268(rsp=HEADER,failure=null)[HttpParser{s=CLOSE,0 of 17775}]]]

Notes

• Synchrony starts up and heartbeat is received

  2016-12-09 14:18:00,248 DEBUG [http-nio-8090-exec-8] [plugins.synchrony.bootstrap.DefaultSynchronyMonitor] isSynchronyUp Checking Synchrony heartbeat on: http://localhost:8091/synchrony/heartbeat
  2016-12-09 14:18:00,263 DEBUG [http-nio-8090-exec-8] [plugins.synchrony.bootstrap.DefaultSynchronyMonitor] isSynchronyUp Response received from Synchrony heartbeat: status HTTP/1.1 200 OK
  

• Requests for the .js files are received

  {"synchrony":{"message":"synchrony.middleware.memoize-response [info] request for static resource [\"0.0.0.0\" \"/synchrony/resources/js/vendor/sockjs.min.js\" nil]","ns":"synchrony.middleware.memoize-response","level":"info"},"message":"synchrony.middleware.memoize-response [info] request for static resource [\"0.0.0.0\" \"/synchrony/resources/js/vendor/sockjs.min.js\" nil]"}
  {"synchrony":{"message":"synchrony.middleware.memoize-response [info] request for static resource [\"0.0.0.0\" \"/synchrony/resources/js/synchrony.min.js\" nil]","ns":"synchrony.middleware.memoize-response","level":"info"},"message":"synchrony.middleware.memoize-response [info] request for static resource [\"0.0.0.0\" \"/synchrony/resources/js/synchrony.min.js\" nil]"}
  

You can test that Synchrony can be polled with

curl -i -N -I http://localhost:8091/synchrony/resources/js/vendor/sockjs.min.js

and returns (note the date)

HTTP/1.1 200 OK
Access-Control-Allow-Headers: content-type, accept, x-token, x-atlassian-mau-ignore
Access-Control-Max-Age: 86400
Content-Encoding: gzip
Access-Control-Allow-Origin: *
Content-Type: application/javascript;charset=utf-8
Content-Length: 17775
Etag: "pWzRSOOq34MQbGLrouECnppGJ10"
Date: �, 10 � 2017 04:45:07 GMT
Vary: Accept-Encoding
Cache-Control: max-age=300
Access-Control-Allow-Methods: GET, PUT, POST, OPTIONS

Workaround

(Upgrade to 6.0.2+ if not already) Override the default locale for Synchrony

1. Create new file synchrony-args.properties in <home-directory> and add the following lines:

   user.language=en
   user.country=US
   

2. Restart Confluence for changes to take effect