JCMA migration assessment may lead to an outage while performing JMCF assessment

XMLWordPrintable

    • 4
    • Severity 1 - Critical
    • 123

      Issue Summary

      Jira Cloud Migration Assistant (JCMA) has a new feature to run pre-migration assessments.

      This includes checking JMCF fields – Jira Misc Custom Fields App.
      Depending on the size of the instance, enabling this assessment may lead to high heap utilization, which may cause FullGC and OOM errors.

      This only occurs if the administrator enabled the assessment feature.

      Steps to Reproduce

      1. Install a vanilla instance of Jira Software Data Center.
      2. Create a couple of JMCF fields with global context.
        • This was validated with 3~4 fields with calculated values.
      3. Create thousands of issues that will have values from the JMCF fields
        • Tested with 175k issues.
      4. Update JCMA to the latest version.
        • Tested with 1.12.12.
      5. Enable the JCMA assessment feature.
      6. Start JCMA assessment.

      Expected Results

      Assessment runs without causing an incident to the instance.

      Actual Results

      The assessment causes memory pressure in the environment, leading to consecutive garbage collection operations and eventually OutOfMemory errors.

      If thread dumps are collected while the issue is occurring, the following can be found.

      "Caesium-1-1" #1048 daemon prio=5 os_prio=0 cpu=98431.95ms elapsed=78134.31s tid=0x00007f097d5af800 nid=0x3bca runnable  [0x00007f092c1d3000]
   java.lang.Thread.State: RUNNABLE
	at org.apache.lucene.util.PriorityQueue.downHeap(PriorityQueue.java:284)
	at org.apache.lucene.util.PriorityQueue.updateTop(PriorityQueue.java:211)
	at org.apache.lucene.search.TopFieldCollector.updateBottom(TopFieldCollector.java:552)
	at org.apache.lucene.search.TopFieldCollector$SimpleFieldCollector$1.collect(TopFieldCollector.java:167)
	at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:233)
	at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:184)
	at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:660)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462)
	at com.atlassian.jira.index.stats.IndexSearcherWithStats.search(IndexSearcherWithStats.java:88)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:583)
	at com.atlassian.jira.index.stats.IndexSearcherWithStats.search(IndexSearcherWithStats.java:68)
	at org.apache.lucene.search.IndexSearcher.searchAfter(IndexSearcher.java:568)
	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:493)
	at com.atlassian.jira.index.stats.IndexSearcherWithStats.search(IndexSearcherWithStats.java:108)
	at com.atlassian.jira.index.DelegateSearcher.search(DelegateSearcher.java:159)
	at com.atlassian.jira.index.DelegateSearcher.search(DelegateSearcher.java:159)
	at com.atlassian.jira.index.UnmanagedIndexSearcher.search(UnmanagedIndexSearcher.java:62)
	at com.atlassian.jira.index.DelegateSearcher.search(DelegateSearcher.java:159)
	at com.atlassian.jira.index.ManagedIndexSearcher.search(ManagedIndexSearcher.java:15)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.searchDocuments(LuceneSearchProvider.java:456)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.runSearch(LuceneSearchProvider.java:448)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.getHits(LuceneSearchProvider.java:231)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:375)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:138)
	at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:143)
	at com.atlassian.jira.bc.issue.search.DefaultSearchService.searchOverrideSecurity(DefaultSearchService.java:124)
	at jdk.internal.reflect.GeneratedMethodAccessor2041.invoke(Unknown Source)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.18/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@11.0.18/Method.java:566)
	at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
	at com.sun.proxy.$Proxy732.searchOverrideSecurity(Unknown Source)
	at jdk.internal.reflect.GeneratedMethodAccessor2041.invoke(Unknown Source)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.18/DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(java.base@11.0.18/Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241)
	at com.sun.proxy.$Proxy4156.searchOverrideSecurity(Unknown Source)
	at com.atlassian.jira.migration.jiraservice.JiraSearchService$searchOverrideSecurity$1.invoke(JiraSearchService.kt:42)
	at com.atlassian.jira.migration.jiraservice.JiraSearchService$searchOverrideSecurity$1.invoke(JiraSearchService.kt:39)
	at kotlin.sequences.SequencesKt__SequencesKt$generateSequence$1.invoke(Sequences.kt:641)
	at kotlin.sequences.GeneratorSequence$iterator$1.calcNext(Sequences.kt:591)
	at kotlin.sequences.GeneratorSequence$iterator$1.hasNext(Sequences.kt:609)
	at kotlin.sequences.FlatteningSequence$iterator$1.ensureItemIterator(Sequences.kt:311)
	at kotlin.sequences.FlatteningSequence$iterator$1.hasNext(Sequences.kt:303)
	at kotlin.sequences.SequencesKt___SequencesKt.toList(_Sequences.kt:816)
	at com.atlassian.jira.migration.guardrails.queries.apps.jmcf.JmcfEntitiesPerCustomFieldQuery.getIssueCountsPerCustomField(JmcfEntitiesPerCustomFieldQuery.kt:91)
	at com.atlassian.jira.migration.guardrails.queries.apps.jmcf.JmcfEntitiesPerCustomFieldQuery.execute-d1pmJ48(JmcfEntitiesPerCustomFieldQuery.kt:46)
	at com.atlassian.jira.migration.guardrails.queries.SingleResultAssessmentQuery.executeAll-IoAF18A(AssessmentQueryBase.kt:55)
	at com.atlassian.jira.migration.guardrails.InstanceAssessmentService.executeQuery(InstanceAssessmentService.kt:307)
	at com.atlassian.jira.migration.guardrails.InstanceAssessmentService.executeQueryAndSave(InstanceAssessmentService.kt:259)
	at com.atlassian.jira.migration.guardrails.InstanceAssessmentService.runAssessment(InstanceAssessmentService.kt:401)
	at com.atlassian.jira.migration.guardrails.InstanceAssessmentService.runJob(InstanceAssessmentService.kt:375)
	at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
	at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
	at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:500)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:495)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:519)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:415)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$$Lambda$4772/0x0000000104ef5c40.accept(Unknown Source)
	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
	at java.lang.Thread.run(java.base@11.0.18/Thread.java:829)

      Workaround

      <TBD>

            Assignee:
            Daniel Serkowski
            Reporter:
            Armando Neto
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: