Details
-
Suggestion
-
Resolution: Timed out
-
None
-
0
-
5
-
Description
Summary
Sometimes Bamboo doesn't shut down correctly and its process keeps hanging around, even though the application itself is not accessible anymore.
Environment
This was observed on a unix-based operating system.
Expected Results
Bamboo shuts down correctly (no process hanging around) and the following can be observed inside the <bamboo-installation-directory>/logs/catalina.out file:
2016-03-16 12:08:46,542 INFO [LastResortTerminationThread] [lifecycle] *******************************
2016-03-16 12:08:46,542 INFO [LastResortTerminationThread] [lifecycle] * Bamboo has been shut down *
2016-03-16 12:08:46,542 INFO [LastResortTerminationThread] [lifecycle] *******************************
2016-03-16 12:08:46,542 WARN [LastResortTerminationThread] [ShutdownEnforcer] Calling System.exit()
Actual Results
The following stack trace appears inside the <bamboo-installation-directory>/logs/catalina.out file:
16-Mar-2016 11:29:54.152 INFO [EventSubscriptionCapabilities:thread-1] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [sun.reflect.MethodAccessorImpl]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access. java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [sun.reflect.MethodAccessorImpl]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access. at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1335) at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1321) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1203) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1164) at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1460) at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75) at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1955) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at sun.misc.Unsafe.defineClass(Native Method) at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:63) at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399) at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394) at java.security.AccessController.doPrivileged(Native Method) at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393) at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70) at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) at com.sun.proxy.$Proxy1040.capableOf(Unknown Source) at com.atlassian.event.remote.impl.cache.CapabilitiesCacheLoader.getKeys(CapabilitiesCacheLoader.java:54) at com.atlassian.failurecache.ExpirationDateBasedCacheImpl.updateCacheKeys(ExpirationDateBasedCacheImpl.java:86) at com.atlassian.failurecache.ExpirationDateBasedCacheImpl.refresh(ExpirationDateBasedCacheImpl.java:78) at com.atlassian.event.remote.impl.cache.EventSubscriptionCapabilities.refresh(EventSubscriptionCapabilities.java:76) at com.atlassian.event.remote.impl.cache.EventSubscriptionCapabilities.run(EventSubscriptionCapabilities.java:84) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
If we try to restart Bamboo again, the application will fail to start with the following errors inside the <bamboo-home-directory>/logs/atlassian-bamboo.log file:
2016-03-16 11:50:27,562 INFO [localhost-startStop-1] [SharedFileLocker] Database /Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/home/jms-store/bamboo/KahaDB/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/home/jms-store/bamboo/KahaDB/lock' could not be locked.
This happens because then you'll have two Bamboo processes running at the same time. This can be observed when running a ps aux | grep command:
bruno$ ps aux | grep bamboo bruno 1191 0.0 5.6 4757160 938888 s000 S 11:19AM 5:51.56 /Library/Java/JavaVirtualMachines/jdk1.8.0_71.jdk/Contents/Home/bin/java -Djava.util.logging.config.file=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m -Xmx384m -Djava.endorsed.dirs=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/endorsed -classpath /Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/bin/bootstrap.jar:/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/bin/tomcat-juli.jar -Dcatalina.base=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install -Dcatalina.home=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install -Djava.io.tmpdir=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/temp org.apache.catalina.startup.Bootstrap start bruno 1633 0.0 0.0 2432772 660 s002 S+ 12:02PM 0:00.00 grep bamboo bruno 1553 0.0 4.1 4537820 694892 s002 S 11:49AM 2:03.93 /Library/Java/JavaVirtualMachines/jdk1.8.0_71.jdk/Contents/Home/bin/java -Djava.util.logging.config.file=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m -Xmx384m -Djava.endorsed.dirs=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/endorsed -classpath /Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/bin/bootstrap.jar:/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/bin/tomcat-juli.jar -Dcatalina.base=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install -Dcatalina.home=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install -Djava.io.tmpdir=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/temp org.apache.catalina.startup.Bootstrap start
Is there any way to improve the shut down process to ensure that it doesn't hang and prevent Bamboo from starting up again without having to issue a kill command?
Workaround
Locate the Bamboo process with a ps aux | grep command and kill it using kill -9 <pid>.
