Uploaded image for project: 'Crowd Data Center'
  1. Crowd Data Center
  2. CWD-2757

Race condition in DirectoryMonitorManagerImpl.addMonitor

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: High High
    • 2.4.2, 2.5
    • 2.4
    • None
    • None

      We sometimes see the following in our logs after a restart:

      2012-01-26 21:07:31,347 ERROR [http-8080-4] c.a.c.d.l.DbCachingRemoteDirectoryInstanceLoaderImpl Could not add a monitor for the directory with id: 262145
      com.atlassian.crowd.manager.directory.monitor.DirectoryMonitorRegistrationException: org.quartz.ObjectAlreadyExistsException: Unable to store Job with name: '262145' and group: 'DirectoryPoller', because one already exists with this identification.
      at com.atlassian.crowd.manager.directory.monitor.poller.AbstractQuartzDirectoryPollerManager.addPoller(AbstractQuartzDirectoryPollerManager.java:29) ~[crowd-core-2.5.0-m0.jar:na]
      at sun.reflect.GeneratedMethodAccessor115.invoke(Unknown Source) ~[na:na]
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_27]
      at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_27]
      at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.framework.adapter.ThrowsAdviceInterceptor.invoke(ThrowsAdviceInterceptor.java:124) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at $Proxy72.authenticate(Unknown Source) [na:na]
      at com.atlassian..user.service.impl.UserServiceImpl.authenticate(UserServiceImpl.java:77) []
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.6.0_27]
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) ~[na:1.6.0_27]
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_27]
      at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_27]
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at com.atlassian..profiling.ProfilingAspect.profileMethod(ProfilingAspect.java:34) [-core-1.0-SNAPSHOT.jar:na]
      at sun.reflect.GeneratedMethodAccessor115.invoke(Unknown Source) ~[na:na]
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) ~[na:1.6.0_27]
      at java.lang.reflect.Method.invoke(Method.java:597) ~[na:1.6.0_27]
      at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110) [spring-tx-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.framework.adapter.ThrowsAdviceInterceptor.invoke(ThrowsAdviceInterceptor.java:124) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) [spring-aop-3.1.0.RELEASE.jar:3.1.0.RELEASE]
      at $Proxy74.authenticate(Unknown Source) [na:na]

      This happens when multiple users try to login at the same time just after a restart. From looking at the sources, there appears to be a race condition in DirectoryMonitorManagerImpl.addMonitor where the check on whether or not the monitor already exists is done without sufficient locking.

            [CWD-2757] Race condition in DirectoryMonitorManagerImpl.addMonitor

            We're seeing this exception multiple times per day. Maybe one explanation for why we're seeing this and Jira/Confluence are not is that

            • we've got a lot of logins going on because all git (push/pull/clone) calls use basic auth.
            • we're using the quartz ramjobstore, which doesn't persist jobs across restarts
            • we've got a LOT of restarts going on, because we redeploy after each green build.

            Michael Heemskerk (Inactive) added a comment - We're seeing this exception multiple times per day. Maybe one explanation for why we're seeing this and Jira/Confluence are not is that we've got a lot of logins going on because all git (push/pull/clone) calls use basic auth. we're using the quartz ramjobstore, which doesn't persist jobs across restarts we've got a LOT of restarts going on, because we redeploy after each green build.

            Currently concerned that we are not seeing the same thing from either JIRA or Connie.

            Justin Koke added a comment - Currently concerned that we are not seeing the same thing from either JIRA or Connie.

              jwalton joe
              mheemskerk Michael Heemskerk (Inactive)
              Affected customers:
              0 This affects my team
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: