Uploaded image for project: 'Bitbucket Server'
  1. Bitbucket Server
  2. BSERV-12611

Proxied SSH calls via mirror nodes can block for a long time in SecureRandom.generateSeed()

    XMLWordPrintable

    Details

      Description

      Issue Summary

      Proxied SSH calls via mirror nodes can block for a long time in SecureRandom.generateSeed() as entropy is being gathered from /dev/random on Linux

      The thread stack for the affected ssh-scm-request-handler will look like this:

      "ssh-scm-request-handler" #6479 daemon prio=5 os_prio=0 tid=0x00007f1ed0588000 nid=0xb0d runnable [0x00007f1eabffb000]
         java.lang.Thread.State: RUNNABLE
              at java.io.FileInputStream.readBytes(Native Method)
              at java.io.FileInputStream.read(FileInputStream.java:255)
              at sun.security.provider.NativePRNG$RandomIO.readFully(NativePRNG.java:410)
              at sun.security.provider.NativePRNG$RandomIO.implGenerateSeed(NativePRNG.java:427)
              - locked <0x00000000c020cdd0> (a java.lang.Object)
              at sun.security.provider.NativePRNG$RandomIO.access$500(NativePRNG.java:329)
              at sun.security.provider.NativePRNG.engineGenerateSeed(NativePRNG.java:224)
              at java.security.SecureRandom.generateSeed(SecureRandom.java:533)
              at org.apache.sshd.common.util.security.bouncycastle.BouncyCastleRandom.<init>(BouncyCastleRandom.java:44)
              at org.apache.sshd.common.util.security.bouncycastle.BouncyCastleRandomFactory.create(BouncyCastleRandomFactory.java:43)
              at org.apache.sshd.common.util.security.bouncycastle.BouncyCastleRandomFactory.create(BouncyCastleRandomFactory.java:28)
              at org.apache.sshd.common.random.SingletonRandomFactory.<init>(SingletonRandomFactory.java:39)
              at org.apache.sshd.common.BaseBuilder.fillWithDefaultValues(BaseBuilder.java:172)
              at org.apache.sshd.client.ClientBuilder.fillWithDefaultValues(ClientBuilder.java:103)
              at org.apache.sshd.client.ClientBuilder.fillWithDefaultValues(ClientBuilder.java:49)
              at org.apache.sshd.common.BaseBuilder.build(BaseBuilder.java:274)
              at org.apache.sshd.client.ClientBuilder.build(ClientBuilder.java:146)
              at org.apache.sshd.client.ClientBuilder.build(ClientBuilder.java:49)
              at org.apache.sshd.common.BaseBuilder.build(BaseBuilder.java:297)
              at org.apache.sshd.client.SshClient.setUpDefaultClient(SshClient.java:761)
              at com.atlassian.bitbucket.internal.mirroring.mirror.ssh.ProxySshCommandFactory$ProxySshCommand.buildSshClient(ProxySshCommandFactory.java:286)
              at com.atlassian.bitbucket.internal.mirroring.mirror.ssh.ProxySshCommandFactory$ProxySshCommand.run(ProxySshCommandFactory.java:236)
              at com.atlassian.bitbucket.internal.ssh.server.SshCommandAdapter$SshCommandRunnable.performAsRequest(SshCommandAdapter.java:341)
              at com.atlassian.bitbucket.internal.ssh.server.SshCommandAdapter$SshCommandRunnable.access$1800(SshCommandAdapter.java:275)
              at com.atlassian.bitbucket.internal.ssh.server.SshCommandAdapter$SshCommandRunnable$OperationRequestAdapter.withRequest(SshCommandAdapter.java:410)
              at com.atlassian.bitbucket.internal.ssh.server.SshCommandAdapter$SshCommandRunnable$OperationRequestAdapter.withRequest(SshCommandAdapter.java:405)
              at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:87)
              at sun.reflect.GeneratedMethodAccessor1023.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
              at com.sun.proxy.$Proxy434.doAsRequest(Unknown Source)
              at sun.reflect.GeneratedMethodAccessor1023.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
              at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
              at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
              at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
              at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
              at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
              at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
              at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
              at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
              at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
              at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
              at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
              at com.sun.proxy.$Proxy917.doAsRequest(Unknown Source)
              at com.atlassian.bitbucket.internal.ssh.server.SshCommandAdapter$SshCommandRunnable.run(SshCommandAdapter.java:289)
              at com.atlassian.bitbucket.internal.ssh.utils.NamePreservingRunnable.run(NamePreservingRunnable.java:30)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      

      Steps to Reproduce

      Run in a loop a command against a mirror that gets proxied to the upstream, for example: 

      ssh -p 7999 <hostname> whoami
      

      On the mirror host itself read from /dev/random to exhaust the entropy pool, for example:

      dd if=/dev/random of=/dev/zero bs=8 count=1024
      

      Note that specifically reading from /dev/random is not necessarily required in the case the "whoami" SSH command is executed rapidly enough, as the load from that alone should be enough to exhaust the entropy pool

      Expected Results

      The "whoami" command should complete in a second or two.

      Actual Results

      The "whoami" command will take 10+ seconds to complete.

      Workaround

      Use rng-tools to increase the quantity of entropy in kernel to make /dev/random faster.

        Attachments

          Activity

            People

            Assignee:
            behumphreys Ben Humphreys
            Reporter:
            behumphreys Ben Humphreys
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: