Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-7616

executor.max.threads is too big for instances with too many CPUs

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Medium Medium
    • 3.11.0
    • 3.6.0, 3.10.4
    • None

      Problem statement

      After fixing STASH-6868 which describes:

      The default value of executor.max.threads is 100 which is fine for average sized instances, but probably too small for very large instances. This limit is often the first one to be hit on instances with very high (especially SSH clone) hosting workload. The value should be increased, preferably scaling with instance size.

      We introduced a formula for executor.max.threads=20*cpu+100 as can be seen on Stash config properties - Executor.

      The problem is that this value could be too big for big instances, generating errors like:

      Exception in thread "ssh:thread-5" java.lang.OutOfMemoryError: unable to create new native thread
      	at java.lang.Thread.start0(Native Method)
      	at java.lang.Thread.start(Thread.java:714)
      	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950)
      	at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1018)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      Or

      2015-03-24 18:46:39,850 ERROR [AtlassianEvent::thread-23]  c.a.e.i.AsynchronousAbleEventDispatcher There was an exception thrown trying to dispatch event [com.atlassian.crowd.event.user.UserAttributeStoredEvent@d1e8fd3f] from the invoker [SingleParameterMethodListenerInvoker{method=public void com.atlassian.stash.internal.license.DefaultLicensedUserCountCache.onUserUpdatedEvent(com.atlassian.crowd.event.user.UserUpdatedEvent), listener=com.atlassian.stash.internal.license.DefaultLicensedUserCountCache@1301e2fa}]
      java.lang.RuntimeException: unable to create new native thread
      	at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:54) [atlassian-event-2.3.5.jar:na]
      	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:48) [atlassian-event-2.3.5.jar:na]
      	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253) [guava-11.0.2-atlassian-02.jar:na]
      	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:107) [atlassian-event-2.3.5.jar:na]
      

      We should check the number of CPUs available and have a ceiling for that value to avoid such scenarios.

      Workaround

      Set in your stash-config.properties a fixed value for executor.max.threads which is lower and is reasonable to your system. This will overwrite the formula we ship by default once you restart Stash.

              cszmajda Cristan Szmajda (Inactive)
              tbomfim ThiagoBomfim (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: