SSHD can hang for a long time in NativePRNG

XMLWordPrintable

    • 0
    • 1

      Issue

      NativePRNG (the default implementation on typical Linux installations among others) uses a global lock to synchronise access to /dev/random and /dev/urandom and random can block waiting for sufficient entropy to accumulate causing builds to hang during Source Code Checkout task.

      Logs

      build log
      28-Mar-2017 16:20:02	Creating local git repository in '/path/to/bamboo-remote-home/xml-data/build-dir/PROJ-PLAN-JOB1/.git'.
28-Mar-2017 16:20:02	/usr/bin/git init
28-Mar-2017 16:20:02	Starting task 'Checkout Default Repository' of type 'com.atlassian.bamboo.plugins.vcs:task.vcs.checkout'
28-Mar-2017 16:20:03	Updating source code to revision: 67e21c41dc3e62eee3047db767363a6e8a6a9f95
28-Mar-2017 16:20:03	/usr/bin/git version
28-Mar-2017 16:20:03	/usr/bin/git log -1 --encoding=UTF-8 --format=%H HEAD
      atlassian-bamboo.log
      INFO   | jvm 1    | 2017/03/28 16:20:02 | 2017/03/28 16:20:02,997 INFO [0-BAM::bamboo.remote.agent::Agent:pool-3-thread-1] [TaskExecutorImpl] Starting task 'Checkout Default Repository' of type 'com.atlassian.bamboo.plugins.vcs:task.vcs.checkout'
INFO   | jvm 1    | 2017/03/28 16:20:03 | 2017/03/28 16:20:03,030 INFO [0-BAM::bamboo.remote.agent::Agent:pool-3-thread-1] [VcsCheckoutTask] Updating source code to revision: 67e21c41dc3e62eee3047db767363a6e8a6a9f95
      thread-dump
      "0-BAM::bamboo.remote.agent (2)::Agent:pool-3-thread-1" #69 daemon prio=1 os_prio=0 tid=0x00007f8c6d645800 nid=0x374d runnable [0x00007f8cf45f4000]
   java.lang.Thread.State: RUNNABLE
      at java.io.FileInputStream.readBytes(Native Method)
      at java.io.FileInputStream.read(FileInputStream.java:255)
      at sun.security.provider.NativePRNG$RandomIO.readFully(NativePRNG.java:410)
      at sun.security.provider.NativePRNG$RandomIO.implGenerateSeed(NativePRNG.java:427)
      - locked <0x00000000e0602e38> (a java.lang.Object)
      at sun.security.provider.NativePRNG$RandomIO.access$500(NativePRNG.java:329)
      at sun.security.provider.NativePRNG.engineGenerateSeed(NativePRNG.java:224)
      at java.security.SecureRandom.generateSeed(SecureRandom.java:533)
      at org.apache.sshd.common.util.SecurityUtils$BouncyCastleRandom.<init>(SecurityUtils.java:460)
      at org.apache.sshd.common.util.SecurityUtils$BouncyCastleRandomFactory.create(SecurityUtils.java:441)
      at org.apache.sshd.common.util.SecurityUtils$BouncyCastleRandomFactory.create(SecurityUtils.java:426)
      at org.apache.sshd.common.random.SingletonRandomFactory.<init>(SingletonRandomFactory.java:38)
      at org.apache.sshd.common.BaseBuilder.fillWithDefaultValues(BaseBuilder.java:154)
      at org.apache.sshd.client.ClientBuilder.fillWithDefaultValues(ClientBuilder.java:111)
      at org.apache.sshd.client.ClientBuilder.fillWithDefaultValues(ClientBuilder.java:50)
      at org.apache.sshd.common.BaseBuilder.build(BaseBuilder.java:242)
      at org.apache.sshd.client.ClientBuilder.build(ClientBuilder.java:154)
      at org.apache.sshd.client.ClientBuilder.build(ClientBuilder.java:50)
      at org.apache.sshd.common.BaseBuilder.build(BaseBuilder.java:263)

      Suggestion

      Test for entropy availability and issue a warning

      Workaround

      Running build against Bamboo server

      1. stop Bamboo server
      2. append -Djava.security.egd=file:/dev/./urandom to JVM_SUPPORT_RECOMMENDED_ARGS
        <bamboo-install>/bin/setenv.sh
        JVM_SUPPORT_RECOMMENDED_ARGS="-Djava.security.egd=file:/dev/./urandom"
      3. start Bamboo

      Running build against Bamboo remote agent

      1. stop Bamboo remote agent
      2. append -Djava.security.egd=file:/dev/./urandom to wrapper.java.additional.X:
        <bamboo-remote-agent>/conf/wrapper.conf
        wrapper.java.additional.1=-Dbamboo.home=/path/to/bamboo-remote-home
wrapper.java.additional.2=-Dbamboo.agent.ignoreServerCertName=false
#wrapper.java.additional.3=-Dlog4j.configuration=
#wrapper.java.additional.3=-agentlib:yjpagent
wrapper.java.additional.3=-Djava.security.egd=file:/dev/./urandom

          In the example above, X was replaced by 3 because it was the following number to be used in sequence. Yours may differ.

      3. start Bamboo remote agent

            Assignee:
            Unassigned
            Reporter:
            Rafael Sperafico (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: