Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-21195

No read timeout for socket operations such as CONNECT and SSL handshake when communicating with Crowd can cause long running HTTP threads

    XMLWordPrintable

Details

    Description

      Issue Summary

      No read timeout for socket operations such as CONNECT and SSL handshake when communicating with Crowd can cause long running HTTP threads if there is an issue with connecting to the remote end while establishing the socket. Even if Crowd/proxy becomes available again, that connection can still stay hung which will cause additional requests for that user (if SSO is enabled) to park waiting for a cache lock that it will never get due to the initial thread still being stuck in the CONNECT or handshake.

      Steps to Reproduce

      1. Configure Bamboo 7.2.2 and Crowd 4.2.2
      2. Setup a squid outbound proxy
      3. Enable SSO in Bamboo
      4. Connect to Crowd via squid outbound proxy
      5. Stop the squid proxy
      6. Start netcat on the same port as proxy: 
        sudo nc -k -l 3128
      7. Login to Crowd to start a SSO session
      8. Open Bamboo to try and resume the SSO session.

      Expected Results

      Connection to Crowd times out in a reasonable timeframe.

      Actual Results

      The HTTP thread remains stuck.

      In CONNECT to proxy:

      "http-nio-8085-exec-21" #269 daemon prio=5 os_prio=0 tid=0x00007f4710525800 nid=0x2c6f runnable [0x00007f4621881000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.createTunnelToTarget(MainClientExec.java:485)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:410)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.client.cache.CachingExec.callBackend(CachingExec.java:592)
	at org.apache.http.impl.client.cache.CachingExec.execute(CachingExec.java:269)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.executeCrowdServiceMethod(RestExecutor.java:498)
	at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:370)
	at com.atlassian.crowd.integration.rest.service.RestCrowdClient.userAuthenticated(RestCrowdClient.java:155)
	at com.atlassian.crowd.directory.RemoteCrowdDirectory.userAuthenticated(RemoteCrowdDirectory.java:833)
	at com.atlassian.crowd.directory.RemoteDirectory.userAuthenticated(RemoteDirectory.java:589)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.userAuthenticated(DbCachingRemoteDirectory.java:296)
	at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.userAuthenticated(DirectoryManagerGeneric.java:309)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.transaction.interceptor.TransactionInterceptor$$Lambda$325/1231124175.proceedWithInvocation(Unknown Source)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
	at com.sun.proxy.$Proxy176.userAuthenticated(Unknown Source)
	at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.userAuthenticated(ApplicationServiceGeneric.java:2309)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.transaction.interceptor.TransactionInterceptor$$Lambda$325/1231124175.proceedWithInvocation(Unknown Source)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
	at com.sun.proxy.$Proxy180.userAuthenticated(Unknown Source)
	at com.atlassian.crowd.embedded.core.CrowdServiceImpl.userAuthenticated(CrowdServiceImpl.java:109)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.transaction.interceptor.TransactionInterceptor$$Lambda$325/1231124175.proceedWithInvocation(Unknown Source)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:295)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
	at com.sun.proxy.$Proxy181.userAuthenticated(Unknown Source)
	at com.atlassian.crowd.integration.seraph.CrowdAuthenticator.lambda$triggerUserAuthenticatedNotification$0(CrowdAuthenticator.java:460)
	at com.atlassian.crowd.integration.seraph.CrowdAuthenticator$$Lambda$1381/1901252110.call(Unknown Source)
	at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4876)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
	- locked <0x00000000ed461490> (a com.google.common.cache.LocalCache$StrongWriteEntry)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3952)
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4871)
	at com.atlassian.crowd.integration.seraph.CrowdAuthenticator.triggerUserAuthenticatedNotification(CrowdAuthenticator.java:457)
	at com.atlassian.crowd.integration.seraph.CrowdAuthenticator.getUser(CrowdAuthenticator.java:419)

      Or in SSL handshake:

      "http-nio-8085-exec-41" daemon prio=5 tid=0x00000000000001f3 nid=0 runnable 
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990)
	- locked <0x000000002e93de34> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1388)
	- locked <0x0000000007d60371> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1416)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1400)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.upgrade(DefaultHttpClientConnectionOperator.java:191)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.upgrade(PoolingHttpClientConnectionManager.java:390)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:428)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.client.cache.CachingExec.callBackend(CachingExec.java:592)
	at org.apache.http.impl.client.cache.CachingExec.execute(CachingExec.java:269)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.executeCrowdServiceMethod(RestExecutor.java:498)

      Duplicating requests for the same user will cause additional threads to build up waiting for a cache lock:

      "http-nio-8085-exec-23" #271 daemon prio=5 os_prio=0 tid=0x00007f4710528000 nid=0x2c71 waiting on condition [0x00007f4621682000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000ed461450> (a com.google.common.util.concurrent.SettableFuture)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:502)
	at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:83)
	at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:196)
	at com.google.common.cache.LocalCache$LoadingValueReference.waitForValue(LocalCache.java:3581)
	at com.google.common.cache.LocalCache$Segment.waitForLoadingValue(LocalCache.java:2174)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2161)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3952)
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4871)
	at com.atlassian.crowd.integration.seraph.CrowdAuthenticator.triggerUserAuthenticatedNotification(CrowdAuthenticator.java:457)
	at com.atlassian.crowd.integration.seraph.CrowdAuthenticator.getUser(CrowdAuthenticator.java:419)

      Workaround

      Bamboo must be restarted to release the threads or the TCP connection between Bamboo/outbound proxy/Crowd must be forcibly ended.

      Attachments

        Issue Links

          is caused by

          Bug - A problem which impairs or prevents the functions of the product. CWD-5678 Remote Crowd directory does not timeout when using SSL tunnel via forward proxy

          • Medium - Medium priority issues
          • Closed

          KRAK-4013 Loading...

          is related to

          Bug - A problem which impairs or prevents the functions of the product. CWD-5709 Crowd directory does not timeout when using SSL tunnel via forward proxy unless Connection Timeout is explicitly configured

          • Low - Low priority issues
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. BAM-20248 Scheduler worker thread pool can completely block with a thread waiting for CONNECT response from outbound proxy

          • Medium - Medium priority issues
          • Closed
          mentioned in

          Page Loading...

          Page Loading...

          Testing discovered

          KRAK-4008 Loading...

          Activity

            People

              mgardias Marcin Gardias
              jowen@atlassian.com Jeremy Owen
              Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: