Race condition on i18n caching, which prevent health check monitor plugin running healcheck

XMLWordPrintable

    • 28
    • Severity 3 - Minor
    • 2

      Issue Summary

      When restoring full export from Cloud to Confluence Server, there will be a lot of failed health check warning. The health check task can finishing running due to i18n cache is locked.

      Steps to Reproduce

      We currently can't reproduce the problem

      Expected Results

      The health check plugin run normally.

      Actual Results

      The below exception is thrown in the xxxxxxx.log file:

      2020-08-26 20:12:27,585 WARN [HealthCheck:thread-8] [internal.core.service.VCacheLock] lockWithTimeout Interrupted whilst waiting for a lock on cache: 
      java.lang.InterruptedException
      	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(Unknown Source)
      	at java.base/java.util.concurrent.locks.ReentrantLock.tryLock(Unknown Source)
      	at com.atlassian.vcache.internal.core.service.VCacheLock.lockWithTimeout(VCacheLock.java:50)
      	at com.atlassian.vcache.internal.core.service.VCacheLock.withLock(VCacheLock.java:31)
      	at com.atlassian.vcache.internal.legacy.LegacyJvmCache.get(LegacyJvmCache.java:42)
      	at com.atlassian.vcache.internal.core.metrics.TimedLocalCacheOperations.get(TimedLocalCacheOperations.java:48)
      	at com.atlassian.confluence.impl.vcache.UnblockingRemovalJvmCache.getIfValid(UnblockingRemovalJvmCache.java:68)
      	at com.atlassian.confluence.impl.vcache.UnblockingRemovalJvmCache.get(UnblockingRemovalJvmCache.java:50)
      	at com.atlassian.confluence.util.i18n.CachingI18NBeanFactory.getI18NBean(CachingI18NBeanFactory.java:75)
      	at jdk.internal.reflect.GeneratedMethodAccessor346.invoke(Unknown Source)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
      	at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
      	at com.sun.proxy.$Proxy517.getI18NBean(Unknown Source)
      	at jdk.internal.reflect.GeneratedMethodAccessor346.invoke(Unknown Source)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
      	at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:131)
      	at com.sun.proxy.$Proxy517.getI18NBean(Unknown Source)
      	at com.atlassian.sal.confluence.message.ConfluenceI18nResolver.getI18nBeanForCurrentUserDefaultLocale(ConfluenceI18nResolver.java:54)
      	at com.atlassian.sal.confluence.message.ConfluenceI18nResolver.resolveText(ConfluenceI18nResolver.java:29)
      	at com.atlassian.sal.core.message.AbstractI18nResolver.getText(AbstractI18nResolver.java:49)
      	at jdk.internal.reflect.GeneratedMethodAccessor498.invoke(Unknown Source)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
      	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
      	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
      	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
      	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
      	at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:136)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
      	at com.sun.proxy.$Proxy1933.getText(Unknown Source)
      	at com.atlassian.troubleshooting.healthcheck.DefaultSupportHealthCheckSupplier.getText(DefaultSupportHealthCheckSupplier.java:136)
      	at com.atlassian.troubleshooting.healthcheck.DefaultSupportHealthCheckSupplier.asPluginSuppliedSupportHealthCheck(DefaultSupportHealthCheckSupplier.java:128)
      	at com.atlassian.troubleshooting.healthcheck.DefaultSupportHealthCheckSupplier.lambda$healthChecksFrom$2(DefaultSupportHealthCheckSupplier.java:120)
      	at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
      	at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
      	at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
      	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
      	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
      	at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
      	at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
      	at com.atlassian.troubleshooting.healthcheck.DefaultSupportHealthCheckSupplier.healthChecksFrom(DefaultSupportHealthCheckSupplier.java:122)
      	at com.atlassian.troubleshooting.healthcheck.DefaultSupportHealthCheckSupplier.byInstance(DefaultSupportHealthCheckSupplier.java:87)
      	at com.atlassian.troubleshooting.healthcheck.SupportHealthStatusBuilder.getHelpPathUrl(SupportHealthStatusBuilder.java:109)
      	at com.atlassian.troubleshooting.healthcheck.SupportHealthStatusBuilder.buildStatus(SupportHealthStatusBuilder.java:134)
      	at com.atlassian.troubleshooting.healthcheck.SupportHealthStatusBuilder.ok(SupportHealthStatusBuilder.java:62)
      	at com.atlassian.troubleshooting.healthcheck.checks.ThreadLimitHealthCheck.check(ThreadLimitHealthCheck.java:45)
      	at com.atlassian.troubleshooting.healthcheck.impl.PluginSuppliedSupportHealthCheck.check(PluginSuppliedSupportHealthCheck.java:49)
      	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      	at java.base/java.lang.Thread.run(Unknown Source)
      

      Workaround

      Running confluence with more param but it could slow due to reload i18n resource: 

      CATALINA_OPTS="-Dconfluence.i18n.reloadbundles=true ${CATALINA_OPTS}" 

            Assignee:
            Unassigned
            Reporter:
            Quan Pham
            Votes:
            6 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated: