[CONFSERVER-36694] Deadlock on Hazelcast

Type: Bug
Resolution: Fixed
Priority: High
Fix Version/s: 5.8
Affects Version/s: 5.6
Component/s: None
Labels:

Bug Fix Policy:
View Atlassian Server bug fix policy

On EAC (Confluence Server) we got this on every node and every thread:

"catalina-exec-7" #259 daemon prio=5 os_prio=0 tid=0x00007fa4d8003000 nid=0x58f8 in Object.wait() [0x00007fa369a3f000]
     java.lang.Thread.State: TIMED_WAITING (on object monitor)
     at java.lang.Object.wait(Native Method)
     at com.hazelcast.spi.impl.BasicInvocationFuture.pollResponse(BasicInvocationFuture.java:247)
     - locked <0x00000007a4dd29d0> (a com.hazelcast.spi.impl.BasicInvocationFuture)
     at com.hazelcast.spi.impl.BasicInvocationFuture.waitForResponse(BasicInvocationFuture.java:201)
     at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:180)
     at com.hazelcast.spi.impl.BasicInvocationFuture.get(BasicInvocationFuture.java:160)
     at com.hazelcast.spi.impl.BasicInvocationFuture.getSafely(BasicInvocationFuture.java:172)
     at com.hazelcast.concurrent.lock.LockProxySupport.lock(LockProxySupport.java:80)
     at com.hazelcast.concurrent.lock.LockProxySupport.lock(LockProxySupport.java:74)
     at com.hazelcast.map.proxy.MapProxyImpl.lock(MapProxyImpl.java:256)
     at com.atlassian.confluence.cluster.hazelcast.HazelcastDualLock.lock(HazelcastDualLock.java:34)
     at com.atlassian.confluence.concurrent.LockOperation.doOperation(LockOperation.java:27)
     at com.atlassian.confluence.concurrent.LockOperation.run(LockOperation.java:41)
     at com.atlassian.confluence.core.DefaultHeartbeatManager.startActivity(DefaultHeartbeatManager.java:115)
     at com.atlassian.confluence.pages.actions.HeartbeatAction.startActivity(HeartbeatAction.java:147)
     at sun.reflect.GeneratedMethodAccessor1849.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke(Method.java:483)
     at com.opensymphony.xwork.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:302)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:170)
     at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:35)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.atlassian.confluence.core.ConfluenceWorkflowInterceptor.intercept(ConfluenceWorkflowInterceptor.java:65)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:35)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.atlassian.xwork.interceptors.XsrfTokenInterceptor.intercept(XsrfTokenInterceptor.java:100)
     at com.atlassian.confluence.xwork.ConfluenceXsrfTokenInterceptor.intercept(ConfluenceXsrfTokenInterceptor.java:26)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.atlassian.confluence.security.interceptors.CaptchaInterceptor.intercept(CaptchaInterceptor.java:50)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:35)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.atlassian.confluence.validation.MessageHolderInterceptor.intercept(MessageHolderInterceptor.java:43)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:35)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.atlassian.confluence.util.LoggingContextInterceptor.intercept(LoggingContextInterceptor.java:52)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.atlassian.confluence.core.CancellingInterceptor.intercept(CancellingInterceptor.java:24)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.atlassian.confluence.xwork.HttpMethodValidationInterceptor.intercept(HttpMethodValidationInterceptor.java:72)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.atlassian.confluence.security.websudo.WebSudoInterceptor.intercept(WebSudoInterceptor.java:64)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.atlassian.confluence.themes.ThemeContextInterceptor.intercept(ThemeContextInterceptor.java:68)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.atlassian.confluence.security.actions.PermissionCheckInterceptor.intercept(PermissionCheckInterceptor.java:66)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.atlassian.confluence.setup.webwork.BootstrapAwareInterceptor.intercept(BootstrapAwareInterceptor.java:27)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:35)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.atlassian.confluence.user.actions.UserAwareInterceptor.intercept(UserAwareInterceptor.java:59)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.atlassian.confluence.pages.actions.CommentAwareInterceptor.intercept(CommentAwareInterceptor.java:51)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.atlassian.confluence.pages.actions.PageAwareInterceptor.intercept(PageAwareInterceptor.java:153)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.atlassian.confluence.spaces.actions.SpaceAwareInterceptor.intercept(SpaceAwareInterceptor.java:78)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
     at com.atlassian.confluence.xwork.FlashScopeInterceptor.intercept(FlashScopeInterceptor.java:25)
     at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
...

Hazelcast/Confluence should not deadlock like this, no matter what the reason is.

This happened after a cluster merge, but I don't think that should matter.

relates to

CONFSERVER-36695 Editor Heartbeat action should not wait forever on lock

Closed

mentioned in: Page Failed to load; Page Failed to load

David Rizzuto added a comment - 27/Feb/2015 12:50 AM

https://jira.atlassian.com/browse/CONF-36695 caused this to happen this time. We don't know the underlying cause (why it couldn't get the lock in the first place), however fixing ~~CONF-36695~~ should make this happen less often.

David Rizzuto added a comment - 27/Feb/2015 12:50 AM https://jira.atlassian.com/browse/CONF-36695 caused this to happen this time. We don't know the underlying cause (why it couldn't get the lock in the first place), however fixing CONF-36695 should make this happen less often.

Confluence Data Center

Details

Description

Attachments

Issue Links

Forms

Activity

Collapse comment: David Rizzuto added a comment - 27/Feb/2015 12:50 AM

Expand comment: David Rizzuto added a comment - 27/Feb/2015 12:50 AM

People

Dates