[CWD-5709] Crowd directory does not timeout when using SSL tunnel via forward proxy unless Connection Timeout is explicitly configured

Type: Bug
Resolution: Fixed
Priority: Low
Fix Version/s: 4.1.10, 4.2.4, 4.3.5, 4.4.0
Affects Version/s: 4.3.0, 4.1.9, 4.2.3
Component/s: Directory - Remote Crowd
Labels:
None

Support reference count:
1
Symptom Severity:
Severity 3 - Minor
Bug Fix Policy:
View Atlassian Server bug fix policy

Issue Summary

After the fix for https://jira.atlassian.com/browse/CWD-5678 a remote Crowd directory still does not timeout by default when using SSL tunnel via forward proxy unless a Connection Timeout is explicitly configured.

Requests that do not timeout (have infinite timeout):

HTTP CONNECT request from Crowd to forward proxy
SSL/TLS handshake after tunnel has been created

Steps to Reproduce

These are steps for HTTP CONNECT request as it's simpler to reproduce:

Run `nc -k -l 8881` to simulate not responding forward proxy on port 8881
Add Remote Crowd directory with the following config:
1. URL: https://localhost - It's crucial that it's HTTP*S* so that Crowd will try to open a SSL tunnel via proxy. Host name doesn't matter as we won't make any request to this host anyways because proxy is being simulated
2. Application name: anything
3. Application password: anything
4. Proxy host: localhost (the host on which we're running netcat)
5. Proxy port: 8881
Click Test connection

Expected Results

Request testing connection will eventually timeout.

Actual Results

Request never times out. Getting thread dump of Crowd java process shows that the thread is stuck at socketRead within createTunnelToTarget or createLayeredSocket and startHandshake:

"http-nio-8095-exec-1" #108 daemon prio=5 os_prio=31 tid=0x00007fc0bfc35000 nid=0xd503 runnable [0x000070000914d000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.createTunnelToTarget(MainClientExec.java:485)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:410)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.client.cache.CachingExec.callBackend(CachingExec.java:592)
        at org.apache.http.impl.client.cache.CachingExec.execute(CachingExec.java:269)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.executeCrowdServiceMethod(RestExecutor.java:498)
        at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:370)
        at com.atlassian.crowd.integration.rest.service.RestCrowdClient.searchUsers(RestCrowdClient.java:514)
        at com.atlassian.crowd.integration.rest.service.RestCrowdClient.testConnection(RestCrowdClient.java:504)
        at com.atlassian.crowd.directory.RemoteCrowdDirectory.testConnection(RemoteCrowdDirectory.java:616)
        at com.atlassian.crowd.embedded.core.CrowdDirectoryServiceImpl.testConnection(CrowdDirectoryServiceImpl.java:88)

This thread will be stuck forever.

Workaround

Edit the Crowd user directory configuration and explicitly configure a Connection Timeout value. The default is 5000 and should be explicitly configured to workaround this issue.

relates to

CWD-5678 Remote Crowd directory does not timeout when using SSL tunnel via forward proxy

Closed

BAM-21195 No read timeout for socket operations such as CONNECT and SSL handshake when communicating with Crowd can cause long running HTTP threads

Closed

mentioned in: Page Failed to load; Page Failed to load; Page Failed to load; Page Loading...; Page Loading...; Page Loading...

(3 mentioned in)

Dawid Owoc (Inactive) made changes - 15/Oct/2021 2:17 PM

Remote Link

New: This issue links to "Page (Confluence)" [ 591651 ]

Cathy S made changes - 27/Aug/2021 4:12 PM

Remote Link

New: This issue links to "Page (Confluence)" [ 575469 ]

Cathy S made changes - 16/Aug/2021 1:03 PM

Remote Link

New: This issue links to "Page (Confluence)" [ 573070 ]

Cathy S made changes - 06/Aug/2021 6:34 PM

Remote Link

New: This issue links to "Page (Confluence)" [ 571062 ]

Cathy S made changes - 21/Jul/2021 11:15 AM

Remote Link

New: This issue links to "Page (Confluence)" [ 568232 ]

Cathy S made changes - 09/Jul/2021 6:37 PM

Remote Link

New: This issue links to "Page (Confluence)" [ 564547 ]

Patryk made changes - 05/Jul/2021 10:38 AM

Resolution		New: Fixed [ 1 ]
Status	Original: Waiting for Release [ 12075 ]	New: Closed [ 6 ]

Patryk made changes - 05/Jul/2021 10:37 AM

Fix Version/s		New: 4.3.5 [ 96890 ]
Fix Version/s	Original: 4.3.1 [ 95294 ]

Bugfix Automation Bot made changes - 12/May/2021 2:01 PM

Support reference count

New: 1

Jeremy Owen made changes - 10/May/2021 11:40 PM

Description

Original: h3. Issue Summary

After the fix for https://jira.atlassian.com/browse/CWD-5678 a remote Crowd directory still does not timeout by default when using SSL tunnel via forward proxy unless a *Connection Timeout* is explicitly configured.

Requests that do not timeout (have infinite timeout):
* HTTP CONNECT request from Crowd to forward proxy
* SSL/TLS handshake after tunnel has been created

h3. Steps to Reproduce

These are steps for HTTP CONNECT request as it's simpler to reproduce:
# Run `nc -k -l 8881` to simulate not responding forward proxy on port 8881
# Add Remote Crowd directory with the following config:
## URL: [https://localhost|https://localhost/] - It's crucial that it's HTTP*S* so that Crowd will try to open a SSL tunnel via proxy. Host name doesn't matter as we won't make any request to this host anyways because proxy is being simulated
## Application name: anything
## Application password: anything
## Proxy host: localhost (the host on which we're running netcat)
## Proxy port: 8881
# Click Test connection

h3. Expected Results

Request testing connection will eventually timeout.
h3. Actual Results

Request never times out. Getting thread dump of Crowd java process shows that the thread is stuck at socketRead:
{code:java}
"http-nio-8095-exec-1" #108 daemon prio=5 os_prio=31 tid=0x00007fc0bfc35000 nid=0xd503 runnable [0x000070000914d000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.createTunnelToTarget(MainClientExec.java:485)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:410)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.client.cache.CachingExec.callBackend(CachingExec.java:592)
        at org.apache.http.impl.client.cache.CachingExec.execute(CachingExec.java:269)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.executeCrowdServiceMethod(RestExecutor.java:498)
        at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:370)
        at com.atlassian.crowd.integration.rest.service.RestCrowdClient.searchUsers(RestCrowdClient.java:514)
        at com.atlassian.crowd.integration.rest.service.RestCrowdClient.testConnection(RestCrowdClient.java:504)
        at com.atlassian.crowd.directory.RemoteCrowdDirectory.testConnection(RemoteCrowdDirectory.java:616)
        at com.atlassian.crowd.embedded.core.CrowdDirectoryServiceImpl.testConnection(CrowdDirectoryServiceImpl.java:88)
{code}
This thread will be stuck forever.
h3. Workaround
Edit the Crowd user directory configuration and explicitly configure a *Connection Timeout* value. The default is 5000 and should be explicitly configured to workaround this issue.

New: h3. Issue Summary

After the fix for https://jira.atlassian.com/browse/CWD-5678 a remote Crowd directory still does not timeout by default when using SSL tunnel via forward proxy unless a *Connection Timeout* is explicitly configured.

Requests that do not timeout (have infinite timeout):
* HTTP CONNECT request from Crowd to forward proxy
* SSL/TLS handshake after tunnel has been created

h3. Steps to Reproduce

These are steps for HTTP CONNECT request as it's simpler to reproduce:
# Run `nc -k -l 8881` to simulate not responding forward proxy on port 8881
# Add Remote Crowd directory with the following config:
## URL: [https://localhost|https://localhost/] - It's crucial that it's HTTP*S* so that Crowd will try to open a SSL tunnel via proxy. Host name doesn't matter as we won't make any request to this host anyways because proxy is being simulated
## Application name: anything
## Application password: anything
## Proxy host: localhost (the host on which we're running netcat)
## Proxy port: 8881
# Click Test connection

h3. Expected Results

Request testing connection will eventually timeout.
h3. Actual Results

Request never times out. Getting thread dump of Crowd java process shows that the thread is stuck at socketRead within {{createTunnelToTarget}} or {{createLayeredSocket}} and {{startHandshake}}:
{code:java}
"http-nio-8095-exec-1" #108 daemon prio=5 os_prio=31 tid=0x00007fc0bfc35000 nid=0xd503 runnable [0x000070000914d000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.createTunnelToTarget(MainClientExec.java:485)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:410)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
        at org.apache.http.impl.client.cache.CachingExec.callBackend(CachingExec.java:592)
        at org.apache.http.impl.client.cache.CachingExec.execute(CachingExec.java:269)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.executeCrowdServiceMethod(RestExecutor.java:498)
        at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:370)
        at com.atlassian.crowd.integration.rest.service.RestCrowdClient.searchUsers(RestCrowdClient.java:514)
        at com.atlassian.crowd.integration.rest.service.RestCrowdClient.testConnection(RestCrowdClient.java:504)
        at com.atlassian.crowd.directory.RemoteCrowdDirectory.testConnection(RemoteCrowdDirectory.java:616)
        at com.atlassian.crowd.embedded.core.CrowdDirectoryServiceImpl.testConnection(CrowdDirectoryServiceImpl.java:88)
{code}
This thread will be stuck forever.
h3. Workaround
Edit the Crowd user directory configuration and explicitly configure a *Connection Timeout* value. The default is 5000 and should be explicitly configured to workaround this issue.

Assignee:: Unassigned

Reporter:: Jeremy Owen

Affected customers:: 0 This affects my team

Watchers:: 4 Start watching this issue

Created:: 10/May/2021 7:19 AM

Updated:: 15/Oct/2021 2:17 PM

Resolved:: 05/Jul/2021 10:38 AM

Details

Description

Issue Summary

Steps to Reproduce

Expected Results

Actual Results

Workaround

Attachments

Issue Links

Forms

Activity

People

Dates