Uploaded image for project: 'Jira Service Management Data Center'
  1. Jira Service Management Data Center
  2. JSDSERVER-12279

JSM customer notifications delayed due to existing expired tokens in cwd_user_attributes

    XMLWordPrintable

Details

    Description

      Issue Summary

      The Jira Service Management (JSM) Customer Notifications are delayed because of vast numbers of expired tokens in cwd_user_attributes.

      Steps to Reproduce

      • If we have customers who never log in to the customer portal but create requests through email, most users have never logged in to Jira.
      • In this scenario, Jira will send an invite token for each notification to such users. All those tokens are expired in cwd_user_attributes and are not deleted from the table.

      Expected Results

      We should have the best way of reading and updating the DB table and refrain from regularly deleting expired tokens.

      Actual Results

      In the thread dumps, we can see a long running thread which name has the syntax PsmqAsyncExecutors-job:thread-XXX:

      The stack trace of this long running is similar to this:

      com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:464)
      com.google.common.collect.ImmutableSet$Builder.addAll(ImmutableSet.java:541)
      com.atlassian.jira.crowd.embedded.ofbiz.OfBizAttributesBuilder.addAttribute(OfBizAttributesBuilder.java:46)
      com.atlassian.jira.crowd.embedded.ofbiz.OfBizAttributesBuilder.toAttributes(OfBizAttributesBuilder.java:29)
      com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao$UserAttributesCacheLoader.load(OfBizUserDao.java:1134)
      com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao$UserAttributesCacheLoader.load(OfBizUserDao.java:1124)
      com.atlassian.cache.memory.MemoryCacheManager$2.load(MemoryCacheManager.java:205)
      com.atlassian.cache.memory.DelegatingCache.lambda$get$0(DelegatingCache.java:165)
      com.atlassian.cache.memory.DelegatingCache$$Lambda$680/0x000000080075cc40.get(Unknown Source)
      com.atlassian.cache.memory.DelegatingCache.lambda$get$1(DelegatingCache.java:193)
      com.atlassian.cache.memory.DelegatingCache$$Lambda$681/0x000000080075c040.call(Unknown Source)
      com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4868)
      com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533)
      com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282)
      com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159)
      com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049)
      com.google.common.cache.LocalCache.get(LocalCache.java:3966)
      com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4863)
      com.atlassian.cache.memory.DelegatingCache.get(DelegatingCache.java:180)
      com.atlassian.cache.memory.DelegatingCache.get(DelegatingCache.java:165)
      com.atlassian.cache.impl.metrics.InstrumentedCache.get(InstrumentedCache.java:72)
      com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.getUserWithAttributes(OfBizUserDao.java:953)
      com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.findByNameWithAttributes(OfBizUserDao.java:304)
      com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.findByNameWithAttributes(DelegatingUserDao.java:77)
      com.atlassian.jira.crowd.embedded.ofbiz.IndexedUserDao.findByNameWithAttributes(IndexedUserDao.java:67)
      com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.findByNameWithAttributes(DelegatingUserDao.java:77)
      com.atlassian.jira.crowd.embedded.ofbiz.SwitchingUserDao.findByNameWithAttributes(SwitchingUserDao.java:30)
      
      ...
      
      com.atlassian.servicedesk.plugins.notifications.internal.rendering.NotificationContentRendererImpl.renderContent(NotificationContentRendererImpl.java:48)
      com.atlassian.servicedesk.plugins.notifications.internal.rendering.NotificationEmailBodyRendererImpl.renderHtmlContent(NotificationEmailBodyRendererImpl.java:56)
      com.atlassian.servicedesk.plugins.notifications.internal.dispatch.NotificationBatchEntryCreator.consume(NotificationBatchEntryCreator.java:56)
      com.atlassian.servicedesk.plugins.notifications.internal.dispatch.NotificationMessageProcessor.lambda$processMessage$1(NotificationMessageProcessor.java:69)
      com.atlassian.servicedesk.plugins.notifications.internal.dispatch.NotificationMessageProcessor$$Lambda$8947/0x0000000804e16840.apply(Unknown Source)
      
      ...
      
      
      com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.context.AutomationExecutionContextImpl.runInAutomationContext(AutomationExecutionContextImpl.java:27)
      com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.lambda$processJobs$5(PsmqExecutionJobRunnerImpl.java:168)
      com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl$$Lambda$8772/0x0000000804de0840.apply(Unknown Source)
      com.atlassian.pocketknife.step.OptionalStep2.lambda$null$0(OptionalStep2.java:20)
      com.atlassian.pocketknife.step.OptionalStep2$$Lambda$8774/0x0000000804de1040.apply(Unknown Source)
      java.util.Optional.flatMap(java.base@11.0.13/Unknown Source)
      com.atlassian.pocketknife.step.OptionalStep2.lambda$then$1(OptionalStep2.java:20)
      com.atlassian.pocketknife.step.OptionalStep2$$Lambda$8773/0x0000000804de0c40.apply(Unknown Source)
      java.util.Optional.flatMap(java.base@11.0.13/Unknown Source)
      com.atlassian.pocketknife.step.OptionalStep2.then(OptionalStep2.java:20)
      com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.processJobs(PsmqExecutionJobRunnerImpl.java:167)
      com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.accessQueueAndProcessJobs(PsmqExecutionJobRunnerImpl.java:149)
      com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.lambda$runJobsInQueues$1(PsmqExecutionJobRunnerImpl.java:83)
      com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl$$Lambda$8757/0x0000000804dd4c40.run(Unknown Source)
      java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.13/Unknown Source)
      java.util.concurrent.FutureTask.run(java.base@11.0.13/Unknown Source)
      java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.13/Unknown Source)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.13/Unknown Source)
      java.lang.Thread.run(java.base@11.0.13/Unknown Source)
      

      The complete stack trace can be found in the file linked below:
      ThreadFullStackTrace

      Workaround

      No workaround as of now, steps will be shared once we have a workaround

      Attachments

        1. ThreadDumps.png
          ThreadDumps.png
          320 kB
        2. ThreadFullStackTrace
          28 kB

        Issue Links

          Activity

            People

              bornatowski Bartosz Ornatowski
              1880f46753e9 Tripura
              Votes:
              5 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Backbone Issue Sync