Bamboo process doesn't shut down correctly (randomly)

XMLWordPrintable

    • 0
    • 5

      Summary

      Sometimes Bamboo doesn't shut down correctly and its process keeps hanging around, even though the application itself is not accessible anymore.

      Environment

      This was observed on a unix-based operating system.

      Expected Results

      Bamboo shuts down correctly (no process hanging around) and the following can be observed inside the <bamboo-installation-directory>/logs/catalina.out file:

      2016-03-16 12:08:46,542 INFO [LastResortTerminationThread] [lifecycle] *******************************
2016-03-16 12:08:46,542 INFO [LastResortTerminationThread] [lifecycle] *  Bamboo has been shut down  *
2016-03-16 12:08:46,542 INFO [LastResortTerminationThread] [lifecycle] *******************************
2016-03-16 12:08:46,542 WARN [LastResortTerminationThread] [ShutdownEnforcer] Calling System.exit()

      Actual Results

      The following stack trace appears inside the <bamboo-installation-directory>/logs/catalina.out file:

      16-Mar-2016 11:29:54.152 INFO [EventSubscriptionCapabilities:thread-1] org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading Illegal access: this web application instance has been stopped already. Could not load [sun.reflect.MethodAccessorImpl]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
 java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [sun.reflect.MethodAccessorImpl]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
	at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1335)
	at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1321)
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1203)
	at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1164)
	at org.apache.felix.framework.BundleWiringImpl.findClassOrResourceByDelegation(BundleWiringImpl.java:1460)
	at org.apache.felix.framework.BundleWiringImpl.access$400(BundleWiringImpl.java:75)
	at org.apache.felix.framework.BundleWiringImpl$BundleClassLoader.loadClass(BundleWiringImpl.java:1955)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at sun.misc.Unsafe.defineClass(Native Method)
	at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:63)
	at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399)
	at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:394)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:393)
	at sun.reflect.MethodAccessorGenerator.generateMethod(MethodAccessorGenerator.java:75)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:53)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
	at com.sun.proxy.$Proxy1040.capableOf(Unknown Source)
	at com.atlassian.event.remote.impl.cache.CapabilitiesCacheLoader.getKeys(CapabilitiesCacheLoader.java:54)
	at com.atlassian.failurecache.ExpirationDateBasedCacheImpl.updateCacheKeys(ExpirationDateBasedCacheImpl.java:86)
	at com.atlassian.failurecache.ExpirationDateBasedCacheImpl.refresh(ExpirationDateBasedCacheImpl.java:78)
	at com.atlassian.event.remote.impl.cache.EventSubscriptionCapabilities.refresh(EventSubscriptionCapabilities.java:76)
	at com.atlassian.event.remote.impl.cache.EventSubscriptionCapabilities.run(EventSubscriptionCapabilities.java:84)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

      If we try to restart Bamboo again, the application will fail to start with the following errors inside the <bamboo-home-directory>/logs/atlassian-bamboo.log file:

      2016-03-16 11:50:27,562 INFO [localhost-startStop-1] [SharedFileLocker] Database /Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/home/jms-store/bamboo/KahaDB/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/home/jms-store/bamboo/KahaDB/lock' could not be locked.

      This happens because then you'll have two Bamboo processes running at the same time. This can be observed when running a ps aux | grep command:

      bruno$ ps aux | grep bamboo
bruno            1191   0.0  5.6  4757160 938888 s000  S    11:19AM   5:51.56 /Library/Java/JavaVirtualMachines/jdk1.8.0_71.jdk/Contents/Home/bin/java -Djava.util.logging.config.file=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m -Xmx384m -Djava.endorsed.dirs=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/endorsed -classpath /Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/bin/bootstrap.jar:/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/bin/tomcat-juli.jar -Dcatalina.base=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install -Dcatalina.home=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install -Djava.io.tmpdir=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/temp org.apache.catalina.startup.Bootstrap start
bruno            1633   0.0  0.0  2432772    660 s002  S+   12:02PM   0:00.00 grep bamboo
bruno            1553   0.0  4.1  4537820 694892 s002  S    11:49AM   2:03.93 /Library/Java/JavaVirtualMachines/jdk1.8.0_71.jdk/Contents/Home/bin/java -Djava.util.logging.config.file=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms256m -Xmx384m -Djava.endorsed.dirs=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/endorsed -classpath /Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/bin/bootstrap.jar:/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/bin/tomcat-juli.jar -Dcatalina.base=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install -Dcatalina.home=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install -Djava.io.tmpdir=/Users/bruno/Documents/Atlassian/bamboo/bamboo-5.10.2/install/temp org.apache.catalina.startup.Bootstrap start

      Is there any way to improve the shut down process to ensure that it doesn't hang and prevent Bamboo from starting up again without having to issue a kill command?

      Workaround

      Locate the Bamboo process with a ps aux | grep command and kill it using kill -9 <pid>.

            Assignee:
            Unassigned
            Reporter:
            Bruno Rosa
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: