[CONFSERVER-45599] Collaborative Editing fails to start with 502 Bad Gateway (or 404) errors for sockjs.min.js, synchrony.min.js when Confluence is running in a zh_CN.UTF-8 (or other non en_GB) locale Created: 19/Dec/2016  Updated: 09/Jun/2017  Resolved: 07/Feb/2017

Status: Resolved
Project: Confluence Server
Component/s: Editor - Collaborative editing
Affects Version/s: 6.0.1, 6.0.2, 6.0.3, 6.0.4
Fix Version/s: 6.0.5

Type: Bug Priority: High
Reporter: James Richards Assignee: Vivian Hu
Resolution: Fixed Votes: 4
Labels: affects-server, cloud-ignore, enterprise, warranty
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File 502BadGateway.png     PNG File Region&language.png     PNG File SetAsDefault.png     PNG File Settings.png     PNG File Time&language.png    
Issue Links:
Reference
is related to CONFSERVER-45567 Confluence cannot connect to Synchron... Resolved
Symptom Severity: Major
Support reference count: 16
Participants:
Last Touched By: Katherine Yabut
Last commented: 41 weeks, 1 day ago
Sprint: Enterprise is nifty, Enterprise is optimal, Enterprise is pleasing
Testing Notes:

Although the fix is for a Windows server, we should test it against Confluence running on Linux or Mac OS to avoid any regressions. Once it's set, collaborative editing should work as expected.

  • To avoid regression, we need to test Confluence against a Windows docker instance.
  • Check with special characters like Chinese

Fix for locale:

  • will use set locale (US) for http date formats
  • also set synchrony JVM properties user.country and user.language to default values US and en respectively (can be overriden in case of unexpected side effects)

Potential concerns:

  • Does locale affect how Synchrony interacts with the editor content? Answer should be no, but might need to do some testing to verify. Potentially test Confluence against a docker instance where the locale/display language is Chinese? or configure Bamboo environment to use Chinese locale

Make sure to carry out a manual test end to end of the collaborative editing feature when the fix is in. i.e. starting Synchrony, Confluence and editing a page.

Occurrence Factor: 10%
QA Demo Status: Not Done
QA Kickoff Status: Done

 Description   

Summary

When using Confluence 6.0 with Collaborative editing enabled on a Server with non-Western locale we see

Loading the editor's taking longer than usual. Give it a few moments, then refresh your page if it still doesn't load. Speak to your Confluence admin if that doesn't fix it.

Environment

  • Language zh_CN.UTF-8 on the server (also seen in Japanese and Korean)
  • No Forward or Reverse Proxy - server.xml is correct with no proxy information (reverse proxy tends to throw 404 errors but is the same cause and resolution)
  • Firewall and anti-virus is turned off/not installed
  • Running on VM
  • no add-ons (i.e. fresh install)
  • Tested access on different networks
  • Accessing Confluence though the default port 8090 (Base URL as http://confluence.mycompany.com:8090/
  • using the Synchrony proxy though /synchrony_proxy

Root Cause

Synchrony-proxy uses Jetty Proxy to handle HTTP requests/responses, and it only recognizes a certain range of characters as legal. Synchrony's date formatter uses the user's default locale, so some non-western (in support cases, mostly Chinese, one Korean case), characters are treated as 'illegal', which results in an HTTP protocol violation, i.e. 502 Bad Gateway error in the browser, 400 internally.

Steps to Reproduce

Linux
  1. Change the Confluence environment to zh_CN.UTF-8. On Linux, you can do this with
    sudo locale-gen zh_CN.UTF-8
    LANG="zh_CN.UTF-8"
    
  2. When you run date you will see
    017年 01月 10日 星期二 16:48:32 AEDT
    
  3. run start-confluence.sh in this environment
  4. run
    curl -i -N -I http://localhost:8091/synchrony/resources/js/vendor/sockjs.min.js
    

    and check the headers, the date format will be in Chinese, or garbled Chinese

    Date: �, 10 2017 05:56:51 GMT
    
  5. Create a new page in Confluence, error will appear in browser and developer tools / network tab
Windows 10
  1. Go to "Start Menu" > "Settings" > "Time & language"
  2. Select "Region & language"
  3. Select Chinese (If Chinese is not an option, select 'Add a language' to add it to the list) > "Set as default" (if this option is greyed out, select "Options" instead – you may have to download the language pack) and restart the machine
  4. Start Confluence as normal, and the result should be the red loading banner for collaborative editing and the 502 Gateway errors in the Developer Tools console (if you had Confluence working before, you may have to clear the cache, otherwise the browser will used the cached js files)
Mac / Windows (and possibly other platforms)

If using Confluence 6.0.2 or greater, create a new file in the Confluence home directory called synchrony-args.properties and add the following:

user.country=CN
user.language=zh

Restart Confluence. You should now get the red loading banner for collaborative editing and see 502 Gateway errors in the console

Expected Results

Collaborative editing works as expected.

Actual Results

In the HAR files collected we see

        "response": {
          "status": 502,
          "statusText": "Bad Gateway",
          "httpVersion": "HTTP/1.1",
          "headers": [
            {
              "name": "Server",
              "value": "Apache-Coyote/1.1"
            },
            {
              "name": "Connection",
              "value": "close"
            },
            {
              "name": "Content-Length",
              "value": "0"
            },
            {
              "name": "Date",
              "value": "Thu, 15 Dec 2016 05:01:36 GMT"
            }
          ],
          "cookies": [],
          "content": {
            "mimeType": "application/x-javascript",
            "size": 0,
            "text": ""
          },
          "redirectURL": "",
          "headersSize": 130,
          "bodySize": -1
        },


atlassian-synchrony-proxy.log has

2016-12-29 16:10:03 DEBUG synchronyProxyFilter:618 - 1675778603 proxying failed
org.eclipse.jetty.client.HttpResponseException: HTTP protocol violation: bad response on HttpConnectionOverHTTP@4618ad61(l:/127.0.0.1:52209 <-> r:localhost/127.0.0.1:8091,closed=false)[HttpChannelOverHTTP@5d87150a(exchange=HttpExchange@4dfc1fb5 req=TERMINATED/null@null res=PENDING/null@null)[send=HttpSenderOverHTTP@6c87be00(req=QUEUED,snd=COMPLETED,failure=null)[HttpGenerator@6f776554{s=START}],recv=HttpReceiverOverHTTP@68c92268(rsp=HEADER,failure=null)[HttpParser{s=CLOSE,0 of 17775}]]]

Notes

  • Synchrony starts up and heartbeat is received
    2016-12-09 14:18:00,248 DEBUG [http-nio-8090-exec-8] [plugins.synchrony.bootstrap.DefaultSynchronyMonitor] isSynchronyUp Checking Synchrony heartbeat on: http://localhost:8091/synchrony/heartbeat
    2016-12-09 14:18:00,263 DEBUG [http-nio-8090-exec-8] [plugins.synchrony.bootstrap.DefaultSynchronyMonitor] isSynchronyUp Response received from Synchrony heartbeat: status HTTP/1.1 200 OK
    
  • Requests for the .js files are received
    {"synchrony":{"message":"synchrony.middleware.memoize-response [info] request for static resource [\"0.0.0.0\" \"/synchrony/resources/js/vendor/sockjs.min.js\" nil]","ns":"synchrony.middleware.memoize-response","level":"info"},"message":"synchrony.middleware.memoize-response [info] request for static resource [\"0.0.0.0\" \"/synchrony/resources/js/vendor/sockjs.min.js\" nil]"}
    {"synchrony":{"message":"synchrony.middleware.memoize-response [info] request for static resource [\"0.0.0.0\" \"/synchrony/resources/js/synchrony.min.js\" nil]","ns":"synchrony.middleware.memoize-response","level":"info"},"message":"synchrony.middleware.memoize-response [info] request for static resource [\"0.0.0.0\" \"/synchrony/resources/js/synchrony.min.js\" nil]"}
    

You can test that Synchrony can be polled with

curl -i -N -I http://localhost:8091/synchrony/resources/js/vendor/sockjs.min.js

and returns (note the date)

HTTP/1.1 200 OK
Access-Control-Allow-Headers: content-type, accept, x-token, x-atlassian-mau-ignore
Access-Control-Max-Age: 86400
Content-Encoding: gzip
Access-Control-Allow-Origin: *
Content-Type: application/javascript;charset=utf-8
Content-Length: 17775
Etag: "pWzRSOOq34MQbGLrouECnppGJ10"
Date: �, 10 � 2017 04:45:07 GMT
Vary: Accept-Encoding
Cache-Control: max-age=300
Access-Control-Allow-Methods: GET, PUT, POST, OPTIONS

Workaround

(Upgrade to 6.0.2+ if not already) Override the default locale for Synchrony

  1. Create new file synchrony-args.properties in <home-directory> and add the following lines:
    user.language=en
    user.country=US
    
  2. Restart Confluence for changes to take effect


 Comments   
Comment by Yuriy Porokhnyavyy [ 30/Dec/2016 ]

6.0.3 the same

Comment by Rob Biggs [ 03/Jan/2017 ]

Seeing the same behavior after an update to 6.0.3 on Linux. Setting synchrony.proxy.enabled to false did not fix the issue. We are using an apache proxy. Previous versions of Confluence did not have issue.

Comment by James Richards [ 03/Jan/2017 ]

Hello Rob Biggs,

This bug is specifically for Windows without a reverse proxy. Are you seeing the 502 Bad Gateway error in the network tools tab of the browser when using a Linux server?

Either way, can you open a ticket with support at https://support.atlassian.com/ and attach

Comment by James Richards [ 11/Jan/2017 ]

Rob Biggs,

Since you posted your comment we're done a lot more investigation and are now confident that the root cause for this issue is specifically for users with Chinese language environment (Windows/Linux and with or without reverse proxy). The symptoms however can have many causes. So, if you haven't already you can test if you have this issue or submit a support request and we can investigate further.

Thanks,
James.

Comment by Minh Tran [ 07/Feb/2017 ]

A fix for this issue is now available for Confluence Server customers.
Upgrade now or check out the Release Notes to see what other issues are resolved.

Generated at Wed Nov 22 21:54:24 UTC 2017 using JIRA 7.6.0-m0129#76001-sha1:1a6e0e8c245893d5a5e460cd20f41de3eb03152d.