Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-64166

When stopping confluence, a lot of HazelcastInstanceNotActiveException displayed in the log

XMLWordPrintable

    • Icon: Suggestion Suggestion
    • Resolution: Unresolved
    • None
    • Data Center - Core
    • None
    • 1
    • We collect Confluence feedback from various sources, and we evaluate what we've collected when planning our product roadmap. To understand how this piece of feedback will be reviewed, see our Implementation of New Features Policy.

      Issue Summary

      When stopping confluence, a lot of HazelcastInstanceNotActiveException displayed in the log

      Steps to Reproduce

      1. Install Data Center with quick start template to AWS
      2. Set-up Confluence
      3. Stop Confluence with below command: 
        systemctl stop confluence

      Expected Results

      Stop confluence without any ERROR or WARN

      Actual Results

      Due to this, a lot of WARN and ERROR logs are shows like the below:

      2021-04-02 04:27:49,129 WARN [hz.ShutdownThread] [com.hazelcast.instance.Node] log [10.0.58.118]:5801 [ConfluenceStack-G9DFCUQ1GEGM] [3.11.6] Terminating forcefully...
 -- referer: http://tsuzu-loadb-17k62n9big6q1-317243873.us-east-1.elb.amazonaws.com/setup/dosetuplicense.action | url: /setup/setupcluster.action | traceId: 00f8defa4b31b4c1 | userName: anonymous | action: setupcluster
2021-04-02 04:27:49,175 ERROR [Catalina-utility-3] [atlassian.confluence.event.ConfluenceEventDispatcher] run There was an exception thrown trying to dispatch event [com.atlassian.confluence.event.events.security.LogoutEvent[source=com.atlassian.confluence.user.listeners.UserSessionExpiryListener@37b7a473]] from the invoker [SingleParameterMethodListenerInvoker{method=public void com.atlassian.confluence.extra.userlister.UserListener.handleLogoutEvent(com.atlassian.confluence.event.events.security.LogoutEvent), listener=com.atlassian.confluence.extra.userlister.UserListener@48a6d514} (timed)]
java.lang.RuntimeException: Problem retrieving a value from cache com.atlassian.confluence.extra.userlister.DefaultUserListManager. Listener: com.atlassian.confluence.extra.userlister.UserListener event: com.atlassian.confluence.event.events.security.LogoutEvent
        at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:57)
...
Caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
        at com.hazelcast.spi.impl.operationservice.impl.InvocationRegistry.register(InvocationRegistry.java:128)

      Workaround

      Currently there is no known workaround for this behavior. A workaround will be added here when available

            [CONFSERVER-64166] When stopping confluence, a lot of HazelcastInstanceNotActiveException displayed in the log

            Gonchik Tsymzhitov added a comment -

            As a result , starting to make full restart

            Gonchik Tsymzhitov added a comment - As a result , starting to make full restart

            Martin Zeller added a comment -

             Yes, I agree to your explanation, makes a lot more sense

            Martin Zeller added a comment -  Yes, I agree to your explanation, makes a lot more sense

            Kashev Dalmia {Soteri} added a comment - - edited

             e4b2cc1af9d1 I don't think that your explanation fully explains this issue. If I have even a single node with "clustering" turned on, I still see all these "Hazelcast instance is not active!" errors. I think the issue is related to the way that the Spring dependency graph fails to capture Hazelcast as an appropriate dependency for many built-in Confluence classes. when as part of cleanup different Confluence components try to run their cleanup code, they try to open database transactions, but this fails due to Hazelcast having been "shut down" prior to all the destroy code being executed. I think that is the bug; Hazelcast should remain active until the last Spring Bean which depends on it has been destroyed.

            Kashev Dalmia {Soteri} added a comment - - edited   e4b2cc1af9d1 I don't think that your explanation fully explains this issue. If I have even a single node with "clustering" turned on, I still see all these "Hazelcast instance is not active!" errors. I think the issue is related to the way that the Spring dependency graph fails to capture Hazelcast as an appropriate dependency for many built-in Confluence classes. when as part of cleanup different Confluence components try to run their cleanup code, they try to open database transactions, but this fails due to Hazelcast having been "shut down" prior to all the destroy code being executed. I think  that  is the bug; Hazelcast should remain active until the last Spring Bean which depends on it has been destroyed.

            Martin Zeller added a comment - - edited

            If you have a two or more node cluster running this is expected, as the node is shutting down and the other nodes try to connect to it.
            Of course it would be nicer if the node announces its shutdown and the others do not try anymore, but this is the responsibility of Atlassian.

            Martin Zeller added a comment - - edited If you have a two or more node cluster running this is expected, as the node is shutting down and the other nodes try to connect to it. Of course it would be nicer if the node announces its shutdown and the others do not try anymore, but this is the responsibility of Atlassian.

              Unassigned Unassigned
              tsuzuki@atlassian.com Tomoko Suzuki
              Votes:
              6 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated: