Uploaded image for project: 'Jira Service Management Data Center'
  1. Jira Service Management Data Center
  2. JSDSERVER-12279

JSM customer notifications delayed due to existing expired tokens in cwd_user_attributes

      Issue Summary

      The Jira Service Management (JSM) Customer Notifications are delayed because of vast numbers of expired tokens in cwd_user_attributes.

      Steps to Reproduce

      • If we have customers who never log in to the customer portal but create requests through email, most users have never logged in to Jira.
      • In this scenario, Jira will send an invite token for each notification to such users. All those tokens are expired in cwd_user_attributes and are not deleted from the table.

      Expected Results

      We should have the best way of reading and updating the DB table and refrain from regularly deleting expired tokens.

      Actual Results

      In the thread dumps, we can see a long running thread which name has the syntax PsmqAsyncExecutors-job:thread-XXX:

      The stack trace of this long running is similar to this:

      com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:464)
      com.google.common.collect.ImmutableSet$Builder.addAll(ImmutableSet.java:541)
      com.atlassian.jira.crowd.embedded.ofbiz.OfBizAttributesBuilder.addAttribute(OfBizAttributesBuilder.java:46)
      com.atlassian.jira.crowd.embedded.ofbiz.OfBizAttributesBuilder.toAttributes(OfBizAttributesBuilder.java:29)
      com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao$UserAttributesCacheLoader.load(OfBizUserDao.java:1134)
      com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao$UserAttributesCacheLoader.load(OfBizUserDao.java:1124)
      com.atlassian.cache.memory.MemoryCacheManager$2.load(MemoryCacheManager.java:205)
      com.atlassian.cache.memory.DelegatingCache.lambda$get$0(DelegatingCache.java:165)
      com.atlassian.cache.memory.DelegatingCache$$Lambda$680/0x000000080075cc40.get(Unknown Source)
      com.atlassian.cache.memory.DelegatingCache.lambda$get$1(DelegatingCache.java:193)
      com.atlassian.cache.memory.DelegatingCache$$Lambda$681/0x000000080075c040.call(Unknown Source)
      com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4868)
      com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533)
      com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282)
      com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159)
      com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049)
      com.google.common.cache.LocalCache.get(LocalCache.java:3966)
      com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4863)
      com.atlassian.cache.memory.DelegatingCache.get(DelegatingCache.java:180)
      com.atlassian.cache.memory.DelegatingCache.get(DelegatingCache.java:165)
      com.atlassian.cache.impl.metrics.InstrumentedCache.get(InstrumentedCache.java:72)
      com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.getUserWithAttributes(OfBizUserDao.java:953)
      com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.findByNameWithAttributes(OfBizUserDao.java:304)
      com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.findByNameWithAttributes(DelegatingUserDao.java:77)
      com.atlassian.jira.crowd.embedded.ofbiz.IndexedUserDao.findByNameWithAttributes(IndexedUserDao.java:67)
      com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.findByNameWithAttributes(DelegatingUserDao.java:77)
      com.atlassian.jira.crowd.embedded.ofbiz.SwitchingUserDao.findByNameWithAttributes(SwitchingUserDao.java:30)
      
      ...
      
      com.atlassian.servicedesk.plugins.notifications.internal.rendering.NotificationContentRendererImpl.renderContent(NotificationContentRendererImpl.java:48)
      com.atlassian.servicedesk.plugins.notifications.internal.rendering.NotificationEmailBodyRendererImpl.renderHtmlContent(NotificationEmailBodyRendererImpl.java:56)
      com.atlassian.servicedesk.plugins.notifications.internal.dispatch.NotificationBatchEntryCreator.consume(NotificationBatchEntryCreator.java:56)
      com.atlassian.servicedesk.plugins.notifications.internal.dispatch.NotificationMessageProcessor.lambda$processMessage$1(NotificationMessageProcessor.java:69)
      com.atlassian.servicedesk.plugins.notifications.internal.dispatch.NotificationMessageProcessor$$Lambda$8947/0x0000000804e16840.apply(Unknown Source)
      
      ...
      
      
      com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.context.AutomationExecutionContextImpl.runInAutomationContext(AutomationExecutionContextImpl.java:27)
      com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.lambda$processJobs$5(PsmqExecutionJobRunnerImpl.java:168)
      com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl$$Lambda$8772/0x0000000804de0840.apply(Unknown Source)
      com.atlassian.pocketknife.step.OptionalStep2.lambda$null$0(OptionalStep2.java:20)
      com.atlassian.pocketknife.step.OptionalStep2$$Lambda$8774/0x0000000804de1040.apply(Unknown Source)
      java.util.Optional.flatMap(java.base@11.0.13/Unknown Source)
      com.atlassian.pocketknife.step.OptionalStep2.lambda$then$1(OptionalStep2.java:20)
      com.atlassian.pocketknife.step.OptionalStep2$$Lambda$8773/0x0000000804de0c40.apply(Unknown Source)
      java.util.Optional.flatMap(java.base@11.0.13/Unknown Source)
      com.atlassian.pocketknife.step.OptionalStep2.then(OptionalStep2.java:20)
      com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.processJobs(PsmqExecutionJobRunnerImpl.java:167)
      com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.accessQueueAndProcessJobs(PsmqExecutionJobRunnerImpl.java:149)
      com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.lambda$runJobsInQueues$1(PsmqExecutionJobRunnerImpl.java:83)
      com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl$$Lambda$8757/0x0000000804dd4c40.run(Unknown Source)
      java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.13/Unknown Source)
      java.util.concurrent.FutureTask.run(java.base@11.0.13/Unknown Source)
      java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.13/Unknown Source)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.13/Unknown Source)
      java.lang.Thread.run(java.base@11.0.13/Unknown Source)
      

      The complete stack trace can be found in the file linked below:
      ThreadFullStackTrace

      Workaround

      No workaround as of now, steps will be shared once we have a workaround

        1. ThreadDumps.png
          320 kB
          Julien Rey
        2. ThreadFullStackTrace
          28 kB
          Julien Rey

          Form Name

            [JSDSERVER-12279] JSM customer notifications delayed due to existing expired tokens in cwd_user_attributes

            desarrollo added a comment - - edited

            Hello Team,

            This issue should be "Severity 2 - Major" because no workaround has been offered.

            Severity 3 - Minor
            The application or specific feature isn't working as expected, but there is a workaround available. Users experience is impacted, but their job function is not impaired. 
            
            

             

            Also, in our case, our instance sends more than 10k emails a day and these are being sent up to 7 days late.

             

            Please, increase de Symptom Severity to Severity 2 - Major.

             

            Best regards,

            Francisco Gonzalez

            desarrollo added a comment - - edited Hello Team, This issue should be "Severity 2 - Major" because no workaround has been offered. Severity 3 - Minor The application or specific feature isn't working as expected, but there is a workaround available. Users experience is impacted, but their job function is not impaired.    Also, in our case, our instance sends more than 10k emails a day and these are being sent up to 7 days late.   Please, increase de Symptom Severity to Severity 2 - Major.   Best regards, Francisco Gonzalez

              bornatowski Bartosz Ornatowski
              1880f46753e9 Tripura (Inactive)
              Affected customers:
              5 This affects my team
              Watchers:
              15 Start watching this issue

                Created:
                Updated:
                Resolved: