Uploaded image for project: 'Jira Software Data Center'
  1. Jira Software Data Center
  2. JSWSERVER-6849

Upon database disconnection AO Ranking Index can break

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • High
    • 6.1.3
    • None
    • None

    Description

      Support have recently come into cases where the ranking chain in GreenHopper has broken when a database connection has failed.

      ie:

      2012-12-01 22:01:20,170 QuartzWorker-0 ERROR ServiceRunner     [NoModule] There was an error getting a DBCP datasource.
      java.lang.RuntimeException: Unable to obtain a connection from the underlying connection pool
      

      An error such as this will then result in a broken rank chain requiring a manual fix which is time consuming for support to perform on a case by case basis:

      2012-12-01 22:36:18,186 QuartzWorker-1 WARN ServiceRunner    Helpdesk [greenhopper.service.rank.RankIndexServiceImpl] Outstanding rank operation transaction discovered. Trying to recover...
      2012-12-01 22:36:18,276 QuartzWorker-1 ERROR ServiceRunner    Helpdesk [greenhopper.customfield.rank.RankIssueEventListener] Failed to create new entry for issue 38911, nextId null and custom field 10004 because conflicting data already exists
      java.lang.IllegalStateException: Failed to create new entry for issue 38911, nextId null and custom field 10004 because conflicting data already exists
      	at com.atlassian.greenhopper.service.rank.RankDaoImpl.saveCreateIssue(RankDaoImpl.java:228)
      	at com.atlassian.greenhopper.service.rank.RankDaoImpl.doInsert(RankDaoImpl.java:159)
      	at com.atlassian.greenhopper.service.rank.RankDaoImpl.executeOutstandingTransaction(RankDaoImpl.java:122)
      	at sun.reflect.GeneratedMethodAccessor3356.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at com.atlassian.activeobjects.tx.TransactionalProxy.invoke(TransactionalProxy.java:79)
      	at com.atlassian.activeobjects.tx.TransactionalProxy.access$000(TransactionalProxy.java:18)
      	at com.atlassian.activeobjects.tx.TransactionalProxy$1.doInTransaction(TransactionalProxy.java:63)
      	at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:25)
      	at com.atlassian.jira.DefaultHostContextAccessor.doInTransaction(DefaultHostContextAccessor.java:34)  <+2>
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at com.atlassian.multitenant.impl.MultiTenantComponentFactoryImpl$AbstractMultiTenantAwareInvocationHandler.invokeInternal(MultiTenantComponentFactoryImpl.java:181)
      	at com.atlassian.multitenant.impl.MultiTenantComponentFactoryImpl$MultiTenantAwareInvocationHandler.invoke(MultiTenantComponentFactoryImpl.java:211)
      	at $Proxy430.doInTransaction(Unknown Source)  <+2>
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at com.atlassian.plugin.osgi.hostcomponents.impl.DefaultComponentRegistrar$ContextClassLoaderSettingInvocationHandler.invoke(DefaultComponentRegistrar.java:129)
      	at $Proxy430.doInTransaction(Unknown Source)  <+2>
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:154)
      	at $Proxy430.doInTransaction(Unknown Source)
      	at com.atlassian.sal.core.transaction.HostContextTransactionTemplate.execute(HostContextTransactionTemplate.java:21)  <+2>
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
      	at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:58)
      	at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:62)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      	at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:56)
      	at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:39)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      	at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      	at $Proxy717.execute(Unknown Source)
      	at com.atlassian.activeobjects.internal.SalTransactionManager.inTransaction(SalTransactionManager.java:23)
      	at com.atlassian.activeobjects.internal.AbstractLoggingTransactionManager.doInTransaction(AbstractLoggingTransactionManager.java:19)
      	at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.executeInTransaction(EntityManagedActiveObjects.java:244)
      	at com.atlassian.activeobjects.osgi.DelegatingActiveObjects.executeInTransaction(DelegatingActiveObjects.java:121)  <+2>
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
      	at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:58)
      	at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:62)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      	at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:56)
      	at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:39)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      	at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      	at $Proxy1931.executeInTransaction(Unknown Source)
      	at com.atlassian.activeobjects.tx.TransactionalProxy.executeInTransaction(TransactionalProxy.java:57)
      	at com.atlassian.activeobjects.tx.TransactionalProxy.invokeInTransaction(TransactionalProxy.java:47)
      	at com.atlassian.activeobjects.tx.TransactionalProxy.invoke(TransactionalProxy.java:35)
      	at $Proxy1932.executeOutstandingTransaction(Unknown Source)
      	at com.atlassian.greenhopper.service.rank.RankIndexServiceImpl$RankIndexSupplier.get(RankIndexServiceImpl.java:407)
      	at com.atlassian.greenhopper.service.rank.RankIndexServiceImpl$RankIndexSupplier.get(RankIndexServiceImpl.java:377)
      	at com.google.common.base.Suppliers$MemoizingSupplier.get(Suppliers.java:109)
      	at com.atlassian.greenhopper.service.rank.RankIndexServiceImpl.getIndex(RankIndexServiceImpl.java:374)
      	at com.atlassian.greenhopper.service.rank.RankIndexServiceImpl.rankLast(RankIndexServiceImpl.java:115)
      	at com.atlassian.greenhopper.service.rank.RankIndexServiceImpl.rankLastForAllFields(RankIndexServiceImpl.java:125)
      	at com.atlassian.greenhopper.customfield.rank.RankIssueEventListener.doOnIssueEvent(RankIssueEventListener.java:30)
      	at com.atlassian.greenhopper.service.issue.AbstractIssueEventListener.onIssueEvent(AbstractIssueEventListener.java:36)  <+2>
      	at java.lang.reflect.Method.invoke(Unknown Source)
      	at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:36)
      	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$2.run(AsynchronousAbleEventDispatcher.java:60)
      	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1.execute(AsynchronousAbleEventDispatcher.java:28)
      	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:56)
      	at com.atlassian.event.internal.EventPublisherImpl.invokeListeners(EventPublisherImpl.java:151)
      	at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:70)
      	at com.atlassian.multitenant.event.DefaultPeeringEventPublisherManager.publish(DefaultPeeringEventPublisherManager.java:17)
      	at com.atlassian.multitenant.event.PeeringEventPublisher.publish(PeeringEventPublisher.java:23)
      	at com.atlassian.jira.event.issue.IssueEventDispatcher.dispatchEvent(IssueEventDispatcher.java:184)
      	at com.atlassian.jira.event.issue.IssueEventDispatcher.dispatchEvent(IssueEventDispatcher.java:134)
      	at com.atlassian.jira.workflow.function.event.FireIssueEventFunction.execute(FireIssueEventFunction.java:57)
      	at com.opensymphony.workflow.AbstractWorkflow.executeFunction(AbstractWorkflow.java:1050)
      	at com.opensymphony.workflow.AbstractWorkflow.transitionWorkflow(AbstractWorkflow.java:1446)
      	at com.opensymphony.workflow.AbstractWorkflow.initialize(AbstractWorkflow.java:615)
      	at com.atlassian.jira.workflow.OSWorkflowManager.createIssue(OSWorkflowManager.java:786)
      	at com.atlassian.jira.issue.managers.DefaultIssueManager.createIssue(DefaultIssueManager.java:441)
      	at com.atlassian.jira.issue.managers.DefaultIssueManager.createIssueObject(DefaultIssueManager.java:518)
      	at com.atlassian.jira.service.util.handler.DefaultMessageHandlerContext.createIssue(DefaultMessageHandlerContext.java:73)
      	at com.atlassian.jira.service.services.mail.DelegatingMessageHandlerContext.createIssue(DelegatingMessageHandlerContext.java:47)
      	at com.atlassian.jira.plugins.mail.handlers.CreateIssueHandler.handleMessage(CreateIssueHandler.java:242)
      	at com.atlassian.jira.plugins.mail.handlers.CreateOrCommentHandler.handleMessage(CreateOrCommentHandler.java:133)
      	at com.atlassian.jira.service.services.mail.MailFetcherService$1.process(MailFetcherService.java:361)
      	at com.atlassian.jira.service.services.mail.MailFetcherService$MessageProviderImpl.getAndProcessMail(MailFetcherService.java:264)
      	at com.atlassian.jira.service.services.mail.MailFetcherService.runImpl(MailFetcherService.java:349)
      	at com.atlassian.jira.service.services.file.AbstractMessageHandlingService.run(AbstractMessageHandlingService.java:250)
      	at com.atlassian.jira.service.JiraServiceContainerImpl.run(JiraServiceContainerImpl.java:61)
      	at com.atlassian.jira.service.ServiceRunner.execute(ServiceRunner.java:47)
      	at org.quartz.core.JobRunShell.run(JobRunShell.java:195)
      	at com.atlassian.multitenant.quartz.MultiTenantThreadPool$MultiTenantRunnable.run(MultiTenantThreadPool.java:72)
      	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
      
      
      Workaround

      Before Atlassian find the root cause of this error please ensure that the correct Validation Query is set up in your dbconfig.xml file as per the instructions given here: https://confluence.atlassian.com/display/JIRA/Surviving+Connection+Closures

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              mandreacchio Michael Andreacchio
              Votes:
              1 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: