Details
-
Bug
-
Resolution: Fixed
-
Medium
-
6.1.1
-
None
-
1
-
Severity 3 - Minor
-
Description
Issue Summary
Proxied SSH calls via mirror nodes can block for a long time in SecureRandom.generateSeed() as entropy is being gathered from /dev/random on Linux
The thread stack for the affected ssh-scm-request-handler will look like this:
"ssh-scm-request-handler" #6479 daemon prio=5 os_prio=0 tid=0x00007f1ed0588000 nid=0xb0d runnable [0x00007f1eabffb000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at sun.security.provider.NativePRNG$RandomIO.readFully(NativePRNG.java:410)
at sun.security.provider.NativePRNG$RandomIO.implGenerateSeed(NativePRNG.java:427)
- locked <0x00000000c020cdd0> (a java.lang.Object)
at sun.security.provider.NativePRNG$RandomIO.access$500(NativePRNG.java:329)
at sun.security.provider.NativePRNG.engineGenerateSeed(NativePRNG.java:224)
at java.security.SecureRandom.generateSeed(SecureRandom.java:533)
at org.apache.sshd.common.util.security.bouncycastle.BouncyCastleRandom.<init>(BouncyCastleRandom.java:44)
at org.apache.sshd.common.util.security.bouncycastle.BouncyCastleRandomFactory.create(BouncyCastleRandomFactory.java:43)
at org.apache.sshd.common.util.security.bouncycastle.BouncyCastleRandomFactory.create(BouncyCastleRandomFactory.java:28)
at org.apache.sshd.common.random.SingletonRandomFactory.<init>(SingletonRandomFactory.java:39)
at org.apache.sshd.common.BaseBuilder.fillWithDefaultValues(BaseBuilder.java:172)
at org.apache.sshd.client.ClientBuilder.fillWithDefaultValues(ClientBuilder.java:103)
at org.apache.sshd.client.ClientBuilder.fillWithDefaultValues(ClientBuilder.java:49)
at org.apache.sshd.common.BaseBuilder.build(BaseBuilder.java:274)
at org.apache.sshd.client.ClientBuilder.build(ClientBuilder.java:146)
at org.apache.sshd.client.ClientBuilder.build(ClientBuilder.java:49)
at org.apache.sshd.common.BaseBuilder.build(BaseBuilder.java:297)
at org.apache.sshd.client.SshClient.setUpDefaultClient(SshClient.java:761)
at com.atlassian.bitbucket.internal.mirroring.mirror.ssh.ProxySshCommandFactory$ProxySshCommand.buildSshClient(ProxySshCommandFactory.java:286)
at com.atlassian.bitbucket.internal.mirroring.mirror.ssh.ProxySshCommandFactory$ProxySshCommand.run(ProxySshCommandFactory.java:236)
at com.atlassian.bitbucket.internal.ssh.server.SshCommandAdapter$SshCommandRunnable.performAsRequest(SshCommandAdapter.java:341)
at com.atlassian.bitbucket.internal.ssh.server.SshCommandAdapter$SshCommandRunnable.access$1800(SshCommandAdapter.java:275)
at com.atlassian.bitbucket.internal.ssh.server.SshCommandAdapter$SshCommandRunnable$OperationRequestAdapter.withRequest(SshCommandAdapter.java:410)
at com.atlassian.bitbucket.internal.ssh.server.SshCommandAdapter$SshCommandRunnable$OperationRequestAdapter.withRequest(SshCommandAdapter.java:405)
at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:87)
at sun.reflect.GeneratedMethodAccessor1023.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
at com.sun.proxy.$Proxy434.doAsRequest(Unknown Source)
at sun.reflect.GeneratedMethodAccessor1023.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy917.doAsRequest(Unknown Source)
at com.atlassian.bitbucket.internal.ssh.server.SshCommandAdapter$SshCommandRunnable.run(SshCommandAdapter.java:289)
at com.atlassian.bitbucket.internal.ssh.utils.NamePreservingRunnable.run(NamePreservingRunnable.java:30)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Steps to Reproduce
Run in a loop a command against a mirror that gets proxied to the upstream, for example:
ssh -p 7999 <hostname> whoami
On the mirror host itself read from /dev/random to exhaust the entropy pool, for example:
dd if=/dev/random of=/dev/zero bs=8 count=1024
Note that specifically reading from /dev/random is not necessarily required in the case the "whoami" SSH command is executed rapidly enough, as the load from that alone should be enough to exhaust the entropy pool
Expected Results
The "whoami" command should complete in a second or two.
Actual Results
The "whoami" command will take 10+ seconds to complete.
Workaround
Use rng-tools to increase the quantity of entropy in kernel to make /dev/random faster.