-
Bug
-
Resolution: Fixed
-
Medium
-
6.7.1, 6.10.0, 6.15.2, 6.15.7, 6.15.8, 6.15.9
-
11
-
Severity 2 - Major
-
52
-
Issue Summary
Confluence becomes unresponsive after all http threads get stuck waiting on a response to establish a websockets connection with Synchrony. Synchrony process shows close to max CPU usage and the top output indicates that many of its threads are long running and consuming a lot of CPU.
Environment
- Confluence Server 6.15.2
- Synchrony-proxy running in front of Synchrony
Notes
Synchrony logs show the following error:
{
"synchrony":{
"message":"tilt.bayeux.server [error] websocket error",
"ns":"tilt.bayeux.server",
"throwable":"java.lang.IllegalStateException: No HttpDecoder and no HttpServerCodec in the pipeline
at io.netty.handler.codec.http.websocketx.WebSocketServerHandshaker.handshake(WebSocketServerHandshaker.java:181)
at aleph.http.server$initialize_websocket_handler$fn__15817.invoke(server.clj:560)
at aleph.http.server$initialize_websocket_handler.invokeStatic(server.clj:559)
at aleph.http$websocket_connection.invokeStatic(http.clj:176)
at tilt.bayeux.server$handle_websocket$fn__51728.invoke(server.clj:489)
at tilt.bayeux.server.(take?)(server.clj:489)
at tilt.bayeux.server$handle_websocket$fn__51728.invoke(server.clj:484)
at clojure.lang.AFn.run(AFn.java:22)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)",
"level":"error"
}
}
Synchrony-proxy logs also indicate that something has gone wrong during the handshake:
2019-05-01 04:58:28 ERROR c.a.s.p.w.WebSocketProxy:49 afterConnectionEstablished - Handshake error 2019-05-01 04:58:29 ERROR c.a.s.p.w.WebSocketProxy:49 afterConnectionEstablished - Handshake error 2019-05-01 04:58:30 ERROR c.a.s.p.w.WebSocketProxy:49 afterConnectionEstablished - Handshake error 2019-05-01 04:58:31 ERROR c.a.s.p.w.WebSocketProxy:49 afterConnectionEstablished - Handshake error
catalina.out:
01-May-2019 04:58:32.362 SEVERE [SimpleAsyncTaskExecutor-1] org.springframework.web.socket.client.WebSocketConnectionManager.onFailure Failed to connect javax.websocket.DeploymentException: The HTTP request to initiate the WebSocket connection failed at org.apache.tomcat.websocket.WsWebSocketContainer.connectToServerRecursive(WsWebSocketContainer.java:486) at org.apache.tomcat.websocket.WsWebSocketContainer.connectToServer(WsWebSocketContainer.java:195) at org.springframework.web.socket.client.standard.StandardWebSocketClient$1.call(StandardWebSocketClient.java:152) at org.springframework.web.socket.client.standard.StandardWebSocketClient$1.call(StandardWebSocketClient.java:149) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.TimeoutException at sun.nio.ch.PendingFuture.get(PendingFuture.java:197) at org.apache.tomcat.websocket.WsWebSocketContainer.processResponse(WsWebSocketContainer.java:803) at org.apache.tomcat.websocket.WsWebSocketContainer.connectToServerRecursive(WsWebSocketContainer.java:348) ... 5 more
Thread dumps will show most of the HTTP threads waiting for the Synchrony Proxy to get a Synchrony session, as follows:
at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1695) at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1775) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915) at com.atlassian.synchrony.proxy.websocket.WebSocketUpstreamHandler.getSynchronySession(WebSocketUpstreamHandler.java:64)
Workaround
Option 1
Until the bug is fixed, a temporary workaround is to switch Synchrony into "Limited" mode.
Follow the steps below to perform the workaround (if Confluence is unresponsive, perform a restart before proceeding).
- The Administering Collaborative Editing article describes how to change Synchrony's operating modes
- Go to
> General Configuration > Collaborative editing. - Choose Change mode.
- Select "Limited" and choose Change.
Synchrony can also be changed to "Off" mode, but any shared drafts will become unavailable while in "Off" mode.
- See How Do Drafts Work on Confluence for more information
Option 2
If you want to keep CE enabled, point your Load Balancer or Reverse Proxy directly to Synchrony and avoid using the internal proxy. More details here.
- links to
- relates to
-
PSR-318 Loading...