Medium Problem statement
After fixing STASH-6868 which describes:
The default value of executor.max.threads is 100 which is fine for average sized instances, but probably too small for very large instances. This limit is often the first one to be hit on instances with very high (especially SSH clone) hosting workload. The value should be increased, preferably scaling with instance size.
We introduced a formula for executor.max.threads=20*cpu+100 as can be seen on Stash config properties - Executor.
The problem is that this value could be too big for big instances, generating errors like:
Exception in thread "ssh:thread-5" java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1018) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Or
2015-03-24 18:46:39,850 ERROR [AtlassianEvent::thread-23] c.a.e.i.AsynchronousAbleEventDispatcher There was an exception thrown trying to dispatch event [com.atlassian.crowd.event.user.UserAttributeStoredEvent@d1e8fd3f] from the invoker [SingleParameterMethodListenerInvoker{method=public void com.atlassian.stash.internal.license.DefaultLicensedUserCountCache.onUserUpdatedEvent(com.atlassian.crowd.event.user.UserUpdatedEvent), listener=com.atlassian.stash.internal.license.DefaultLicensedUserCountCache@1301e2fa}]
java.lang.RuntimeException: unable to create new native thread
at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:54) [atlassian-event-2.3.5.jar:na]
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:48) [atlassian-event-2.3.5.jar:na]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253) [guava-11.0.2-atlassian-02.jar:na]
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:107) [atlassian-event-2.3.5.jar:na]
We should check the number of CPUs available and have a ceiling for that value to avoid such scenarios.
Workaround
Set in your stash-config.properties a fixed value for executor.max.threads which is lower and is reasonable to your system. This will overwrite the formula we ship by default once you restart Stash.
