Medium Problem statement
After fixing STASH-6868 which describes:
The default value of executor.max.threads is 100 which is fine for average sized instances, but probably too small for very large instances. This limit is often the first one to be hit on instances with very high (especially SSH clone) hosting workload. The value should be increased, preferably scaling with instance size.
We introduced a formula for executor.max.threads=20*cpu+100 as can be seen on Stash config properties - Executor.
The problem is that this value could be too big for big instances, generating errors like:
Exception in thread "ssh:thread-5" java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:714) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1018) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
Or
2015-03-24 18:46:39,850 ERROR [AtlassianEvent::thread-23] c.a.e.i.AsynchronousAbleEventDispatcher There was an exception thrown trying to dispatch event [com.atlassian.crowd.event.user.UserAttributeStoredEvent@d1e8fd3f] from the invoker [SingleParameterMethodListenerInvoker{method=public void com.atlassian.stash.internal.license.DefaultLicensedUserCountCache.onUserUpdatedEvent(com.atlassian.crowd.event.user.UserUpdatedEvent), listener=com.atlassian.stash.internal.license.DefaultLicensedUserCountCache@1301e2fa}]
java.lang.RuntimeException: unable to create new native thread
at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:54) [atlassian-event-2.3.5.jar:na]
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:48) [atlassian-event-2.3.5.jar:na]
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253) [guava-11.0.2-atlassian-02.jar:na]
at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:107) [atlassian-event-2.3.5.jar:na]
We should check the number of CPUs available and have a ceiling for that value to avoid such scenarios.
Workaround
Set in your stash-config.properties a fixed value for executor.max.threads which is lower and is reasonable to your system. This will overwrite the formula we ship by default once you restart Stash.
![[Atlassian Documentation] Page](https://confluence.atlassian.com/images/icons/favicon.png "[Atlassian Documentation] Page"){kind=icon}
![[Atlassian Documentation] Page](/images/icons/generic_link_16.png "[Atlassian Documentation] Page"){kind=icon}
| Form Name | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
This has been resolved with a number of changes in 3.11.0:
1. The SCM cache now no longer uses the Stash executor service, but manages its own thread pool (that is limited by the number of available hosting tickets throttle.resource.scm-hosting which is 1.5 x the number of CPU cores by default).
2. The internal Hazelcast system executor thread pool no longer scales with executor.max.threads but has an independent limit.
3. The default size of the Stash executor service thread pool has been greatly reduced, executor.max.threads is now equal to the number of CPU cores by default.
4. The threads in the Stash executor service thread pool are now fixed instead of growing and shrinking. They no longer respawn over time and executor.keepAliveTime is now no longer used.