Hibernate Adapter Map is not Thread Safe

XMLWordPrintable

    • 1
    • Severity 1 - Critical

      Summary

      There's an internal Hibernate Adapter Map used to support Hibernate API Versions 2 and 5. The purpose of the map is to translate exceptions and other tasks from one version to another. This map can get corrupted, which can cause threads to get stuck when executing the WeakHashMap method, which is not thread safe and can lock the entire application.

      Symptoms

      Application tasks that rely on achieving connections from the database, such as the Index, may stop working, since the WeakHashMap locks all threads who doesn't have a connection to the database yet. There may be other operations that can be impacted too.

      The way to identify this scenario is by taking thread dumps, where we see the majority of the threads in either Blocked or Waiting statuses, while there's only one thread in Runnable status, executing a WeakHashMap:

      "http-nio-8091-exec-492" daemon prio=5 tid=0x00000000000022bb nid=0 runnable 
   java.lang.Thread.State: RUNNABLE
	at java.util.WeakHashMap.get(WeakHashMap.java:403)
	at com.atlassian.hibernate.adapter.adapters.HibernateExceptionAdapter.adapt(HibernateExceptionAdapter.java:35)
	at com.atlassian.hibernate.adapter.adapters.session.SessionV2Adapter.connection(SessionV2Adapter.java:161)
	at org.springframework.orm.hibernate.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:422)
	at com.atlassian.confluence.impl.hibernate.ConfluenceHibernateTransactionManager.doBegin(ConfluenceHibernateTransactionManager.java:41)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
	at sun.reflect.GeneratedMethodAccessor54.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
	at com.sun.proxy.$Proxy34.getTransaction(Unknown Source)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:427)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
	at com.sun.proxy.$Proxy36.retrieve(Unknown Source)
	at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.lambda$retrieve$0(ConfluenceCachingBandanaPersister.java:80)
	at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister$$Lambda$359/1698215470.get(Unknown Source)
	at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.lambda$get$10(SynchronousExternalCache.java:236)
	at com.atlassian.confluence.impl.vcache.SynchronousExternalCache$$Lambda$361/2117426375.get(Unknown Source)
	at com.atlassian.vcache.internal.core.metrics.TimedSupplier.get(TimedSupplier.java:32)
	at com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache.lambda$null$5(AbstractStableReadExternalCache.java:116)
	at com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache$$Lambda$375/922126746.get(Unknown Source)
	at java.util.Optional.orElseGet(Optional.java:267)
	at com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache.lambda$get$6(AbstractStableReadExternalCache.java:114)
	at com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache$$Lambda$368/1620586459.call(Unknown Source)
	at com.atlassian.vcache.internal.core.service.AbstractExternalCache.perform(AbstractExternalCache.java:83)
	at com.atlassian.vcache.internal.core.service.AbstractExternalCache.perform(AbstractExternalCache.java:69)
	at com.atlassian.vcache.internal.core.service.AbstractStableReadExternalCache.get(AbstractStableReadExternalCache.java:100)
	at com.atlassian.vcache.internal.core.metrics.TimedExternalCache.get(TimedExternalCache.java:67)
	at com.atlassian.confluence.impl.vcache.SynchronousExternalCache.get(SynchronousExternalCache.java:236)
	at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:78)
	at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:71)
	at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)
	at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:24)
	at com.atlassian.confluence.setup.settings.DefaultSettingsManager.getGlobalSettings(DefaultSettingsManager.java:38)
	at com.atlassian.confluence.status.service.DefaultSystemInformationService.getConfluenceInfo(DefaultSystemInformationService.java:97)
	at org.apache.jsp._500page_jsp._jspService(_500page_jsp.java:306)

      Steps to Replicate

      It is unclear on how the application would get into this state.

      Workaround

      Unfortunately, the only way to clear the stuck thread is to restart the application.

            Assignee:
            Nhan Dang
            Reporter:
            Marcelo Horlle (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: