Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-12611

Proxied SSH calls via mirror nodes can block for a long time in SecureRandom.generateSeed()

XMLWordPrintable

      Issue Summary

      Proxied SSH calls via mirror nodes can block for a long time in SecureRandom.generateSeed() as entropy is being gathered from /dev/random on Linux

      The thread stack for the affected ssh-scm-request-handler will look like this:

      "ssh-scm-request-handler" #6479 daemon prio=5 os_prio=0 tid=0x00007f1ed0588000 nid=0xb0d runnable [0x00007f1eabffb000]
   java.lang.Thread.State: RUNNABLE
        at java.io.FileInputStream.readBytes(Native Method)
        at java.io.FileInputStream.read(FileInputStream.java:255)
        at sun.security.provider.NativePRNG$RandomIO.readFully(NativePRNG.java:410)
        at sun.security.provider.NativePRNG$RandomIO.implGenerateSeed(NativePRNG.java:427)
        - locked <0x00000000c020cdd0> (a java.lang.Object)
        at sun.security.provider.NativePRNG$RandomIO.access$500(NativePRNG.java:329)
        at sun.security.provider.NativePRNG.engineGenerateSeed(NativePRNG.java:224)
        at java.security.SecureRandom.generateSeed(SecureRandom.java:533)
        at org.apache.sshd.common.util.security.bouncycastle.BouncyCastleRandom.<init>(BouncyCastleRandom.java:44)
        at org.apache.sshd.common.util.security.bouncycastle.BouncyCastleRandomFactory.create(BouncyCastleRandomFactory.java:43)
        at org.apache.sshd.common.util.security.bouncycastle.BouncyCastleRandomFactory.create(BouncyCastleRandomFactory.java:28)
        at org.apache.sshd.common.random.SingletonRandomFactory.<init>(SingletonRandomFactory.java:39)
        at org.apache.sshd.common.BaseBuilder.fillWithDefaultValues(BaseBuilder.java:172)
        at org.apache.sshd.client.ClientBuilder.fillWithDefaultValues(ClientBuilder.java:103)
        at org.apache.sshd.client.ClientBuilder.fillWithDefaultValues(ClientBuilder.java:49)
        at org.apache.sshd.common.BaseBuilder.build(BaseBuilder.java:274)
        at org.apache.sshd.client.ClientBuilder.build(ClientBuilder.java:146)
        at org.apache.sshd.client.ClientBuilder.build(ClientBuilder.java:49)
        at org.apache.sshd.common.BaseBuilder.build(BaseBuilder.java:297)
        at org.apache.sshd.client.SshClient.setUpDefaultClient(SshClient.java:761)
        at com.atlassian.bitbucket.internal.mirroring.mirror.ssh.ProxySshCommandFactory$ProxySshCommand.buildSshClient(ProxySshCommandFactory.java:286)
        at com.atlassian.bitbucket.internal.mirroring.mirror.ssh.ProxySshCommandFactory$ProxySshCommand.run(ProxySshCommandFactory.java:236)
        at com.atlassian.bitbucket.internal.ssh.server.SshCommandAdapter$SshCommandRunnable.performAsRequest(SshCommandAdapter.java:341)
        at com.atlassian.bitbucket.internal.ssh.server.SshCommandAdapter$SshCommandRunnable.access$1800(SshCommandAdapter.java:275)
        at com.atlassian.bitbucket.internal.ssh.server.SshCommandAdapter$SshCommandRunnable$OperationRequestAdapter.withRequest(SshCommandAdapter.java:410)
        at com.atlassian.bitbucket.internal.ssh.server.SshCommandAdapter$SshCommandRunnable$OperationRequestAdapter.withRequest(SshCommandAdapter.java:405)
        at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:87)
        at sun.reflect.GeneratedMethodAccessor1023.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
        at com.sun.proxy.$Proxy434.doAsRequest(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor1023.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
        at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
        at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
        at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
        at com.sun.proxy.$Proxy917.doAsRequest(Unknown Source)
        at com.atlassian.bitbucket.internal.ssh.server.SshCommandAdapter$SshCommandRunnable.run(SshCommandAdapter.java:289)
        at com.atlassian.bitbucket.internal.ssh.utils.NamePreservingRunnable.run(NamePreservingRunnable.java:30)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

      Steps to Reproduce

      Run in a loop a command against a mirror that gets proxied to the upstream, for example: 

      ssh -p 7999 <hostname> whoami

      On the mirror host itself read from /dev/random to exhaust the entropy pool, for example:

      dd if=/dev/random of=/dev/zero bs=8 count=1024

      Note that specifically reading from /dev/random is not necessarily required in the case the "whoami" SSH command is executed rapidly enough, as the load from that alone should be enough to exhaust the entropy pool

      Expected Results

      The "whoami" command should complete in a second or two.

      Actual Results

      The "whoami" command will take 10+ seconds to complete.

      Workaround

      Use rng-tools to increase the quantity of entropy in kernel to make /dev/random faster.

              behumphreys Ben Humphreys
              behumphreys Ben Humphreys
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: