Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-6933

"Hazelcast instance is not active!" during shutdown

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Low
    • 3.2.0
    • None
    • Enterprise
    • None

    Description

      On most shutdowns, we get this error:

      Jun 30, 2014 6:00:51 AM org.apache.catalina.core.StandardServer await
      INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
      Jun 30, 2014 6:00:51 AM org.apache.coyote.AbstractProtocol pause
      INFO: Pausing ProtocolHandler ["http-bio-7990"]
      Jun 30, 2014 6:00:51 AM org.apache.catalina.core.StandardService stopInternal
      INFO: Stopping service Catalina
      [Full GC [PSYoungGen: 183808K->11627K(197120K)] [ParOldGen: 524009K->523949K(524288K)] 707817K->535576K(721408K) [PSPermGen: 167753K->166659K(166912K)], 1.0898410 secs] [Times: user=2.85 sys=0.00, real=1.09 secs] 
      [Full GC [PSYoungGen: 183808K->0K(197120K)] [ParOldGen: 523949K->513715K(524288K)] 707757K->513715K(721408K) [PSPermGen: 166670K->164600K(164864K)], 0.7192540 secs] [Times: user=1.84 sys=0.00, real=0.72 secs] 
      Exception in thread "rescope:thread-1" com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
              at com.hazelcast.spi.impl.WaitNotifyServiceImpl.shutdown(WaitNotifyServiceImpl.java:204)
              at com.hazelcast.spi.impl.NodeEngineImpl.shutdown(NodeEngineImpl.java:387)
              at com.hazelcast.instance.Node.shutdown(Node.java:420)
              at com.hazelcast.instance.LifecycleServiceImpl.shutdown(LifecycleServiceImpl.java:87)
              at com.hazelcast.instance.HazelcastInstanceProxy.shutdown(HazelcastInstanceProxy.java:237)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
              at java.lang.reflect.Method.invoke(Unknown Source)
              at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:350)
              at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:273)
              at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:540)
              at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:516)
              at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:827)
              at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:485)
              at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:921)
              at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:895)
              at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:841)
              at org.springframework.web.context.ContextLoader.closeWebApplicationContext(ContextLoader.java:579)
              at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:115)
              at com.atlassian.johnson.spring.web.context.JohnsonContextLoaderListener.contextDestroyed(JohnsonContextLoaderListener.java:53)
              at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:5014)
              at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5659)
              at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:232)
              at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1575)
              at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1564)
              at java.util.concurrent.FutureTask.run(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
              at java.lang.Thread.run(Unknown Source)
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#2474886a_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#2474886a_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#2474886a_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#2474886a_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#2474886a_Worker-5] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#2474886a_Worker-6] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#2474886a_Worker-7] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#2474886a_Worker-8] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#2474886a_Worker-9] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#2474886a_Worker-10] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#5ea7f9d8_Worker-1] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#5ea7f9d8_Worker-2] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#5ea7f9d8_Worker-3] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#5ea7f9d8_Worker-4] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#5ea7f9d8_Worker-5] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#5ea7f9d8_Worker-6] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#5ea7f9d8_Worker-7] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#5ea7f9d8_Worker-8] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#5ea7f9d8_Worker-9] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [org.springframework.scheduling.quartz.SchedulerFactoryBean#5ea7f9d8_Worker-10] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
      SEVERE: The web application [] appears to have started a thread named [commit-graph-pumper] but has failed to stop it. This is very likely to create a memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.InheritableThreadLocal] (value [java.lang.InheritableThreadLocal@7b376a1e]) and a value of type [org.apache.felix.framework.BundleContextImpl] (value [org.apache.felix.framework.BundleContextImpl@731816c4]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [] created a ThreadLocal with key of type [com.sun.xml.bind.v2.ClassFactory$1] (value [com.sun.xml.bind.v2.ClassFactory$1@5bca3513]) and a value of type [java.util.WeakHashMap] (value [{class com.atlassian.crowd.integration.rest.entity.UserEntity=java.lang.ref.WeakReference@3862a179, class com.atlassian.crowd.integration.rest.entity.UserEntityList=java.lang.ref.WeakReference@53572bda, class com.atlassian.crowd.integration.rest.entity.MembershipEntity=java.lang.ref.WeakReference@1ff5, class com.atlassian.crowd.integration.rest.entity.GroupEntity=java.lang.ref.WeakReference@8320dd3, class com.atlassian.crowd.integration.rest.entity.MultiValuedAttributeEntityList=java.lang.ref.WeakReference@e299bd9, class com.atlassian.crowd.integration.rest.entity.MembershipsEntity=java.lang.ref.WeakReference@4db6cad2, class com.atlassian.crowd.integration.rest.entity.GroupEntityList=java.lang.ref.WeakReference@1bbdc8da, class javax.xml.bind.annotation.W3CDomHandler=java.lang.ref.WeakReference@48cd6e70, class com.atlassian.crowd.integration.rest.entity.PasswordEntity=java.lang.ref.WeakReference@270da067}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      Jun 30, 2014 6:00:55 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
      SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@6f9dde38]) and a value of type [org.springframework.security.core.context.SecurityContextImpl] (value [org.springframework.security.core.context.SecurityContextImpl@ffffffff: Null authentication]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            cszmajda Cristan Szmajda (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: