-
Bug
-
Resolution: Fixed
-
Highest
-
5.3.0, 5.3.1, 5.4.0, 5.4.1
-
2
-
Severity 2 - Major
-
40
-
Issue Summary
The Jira Service Management (JSM) Customer Notifications are delayed because of vast numbers of expired tokens in cwd_user_attributes.
Steps to Reproduce
- If we have customers who never log in to the customer portal but create requests through email, most users have never logged in to Jira.
- In this scenario, Jira will send an invite token for each notification to such users. All those tokens are expired in cwd_user_attributes and are not deleted from the table.
Expected Results
We should have the best way of reading and updating the DB table and refrain from regularly deleting expired tokens.
Actual Results
In the thread dumps, we can see a long running thread which name has the syntax PsmqAsyncExecutors-job:thread-XXX:
The stack trace of this long running is similar to this:
com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:464) com.google.common.collect.ImmutableSet$Builder.addAll(ImmutableSet.java:541) com.atlassian.jira.crowd.embedded.ofbiz.OfBizAttributesBuilder.addAttribute(OfBizAttributesBuilder.java:46) com.atlassian.jira.crowd.embedded.ofbiz.OfBizAttributesBuilder.toAttributes(OfBizAttributesBuilder.java:29) com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao$UserAttributesCacheLoader.load(OfBizUserDao.java:1134) com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao$UserAttributesCacheLoader.load(OfBizUserDao.java:1124) com.atlassian.cache.memory.MemoryCacheManager$2.load(MemoryCacheManager.java:205) com.atlassian.cache.memory.DelegatingCache.lambda$get$0(DelegatingCache.java:165) com.atlassian.cache.memory.DelegatingCache$$Lambda$680/0x000000080075cc40.get(Unknown Source) com.atlassian.cache.memory.DelegatingCache.lambda$get$1(DelegatingCache.java:193) com.atlassian.cache.memory.DelegatingCache$$Lambda$681/0x000000080075c040.call(Unknown Source) com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4868) com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533) com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282) com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159) com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049) com.google.common.cache.LocalCache.get(LocalCache.java:3966) com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4863) com.atlassian.cache.memory.DelegatingCache.get(DelegatingCache.java:180) com.atlassian.cache.memory.DelegatingCache.get(DelegatingCache.java:165) com.atlassian.cache.impl.metrics.InstrumentedCache.get(InstrumentedCache.java:72) com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.getUserWithAttributes(OfBizUserDao.java:953) com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.findByNameWithAttributes(OfBizUserDao.java:304) com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.findByNameWithAttributes(DelegatingUserDao.java:77) com.atlassian.jira.crowd.embedded.ofbiz.IndexedUserDao.findByNameWithAttributes(IndexedUserDao.java:67) com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.findByNameWithAttributes(DelegatingUserDao.java:77) com.atlassian.jira.crowd.embedded.ofbiz.SwitchingUserDao.findByNameWithAttributes(SwitchingUserDao.java:30) ... com.atlassian.servicedesk.plugins.notifications.internal.rendering.NotificationContentRendererImpl.renderContent(NotificationContentRendererImpl.java:48) com.atlassian.servicedesk.plugins.notifications.internal.rendering.NotificationEmailBodyRendererImpl.renderHtmlContent(NotificationEmailBodyRendererImpl.java:56) com.atlassian.servicedesk.plugins.notifications.internal.dispatch.NotificationBatchEntryCreator.consume(NotificationBatchEntryCreator.java:56) com.atlassian.servicedesk.plugins.notifications.internal.dispatch.NotificationMessageProcessor.lambda$processMessage$1(NotificationMessageProcessor.java:69) com.atlassian.servicedesk.plugins.notifications.internal.dispatch.NotificationMessageProcessor$$Lambda$8947/0x0000000804e16840.apply(Unknown Source) ... com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.context.AutomationExecutionContextImpl.runInAutomationContext(AutomationExecutionContextImpl.java:27) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.lambda$processJobs$5(PsmqExecutionJobRunnerImpl.java:168) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl$$Lambda$8772/0x0000000804de0840.apply(Unknown Source) com.atlassian.pocketknife.step.OptionalStep2.lambda$null$0(OptionalStep2.java:20) com.atlassian.pocketknife.step.OptionalStep2$$Lambda$8774/0x0000000804de1040.apply(Unknown Source) java.util.Optional.flatMap(java.base@11.0.13/Unknown Source) com.atlassian.pocketknife.step.OptionalStep2.lambda$then$1(OptionalStep2.java:20) com.atlassian.pocketknife.step.OptionalStep2$$Lambda$8773/0x0000000804de0c40.apply(Unknown Source) java.util.Optional.flatMap(java.base@11.0.13/Unknown Source) com.atlassian.pocketknife.step.OptionalStep2.then(OptionalStep2.java:20) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.processJobs(PsmqExecutionJobRunnerImpl.java:167) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.accessQueueAndProcessJobs(PsmqExecutionJobRunnerImpl.java:149) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.lambda$runJobsInQueues$1(PsmqExecutionJobRunnerImpl.java:83) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl$$Lambda$8757/0x0000000804dd4c40.run(Unknown Source) java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.13/Unknown Source) java.util.concurrent.FutureTask.run(java.base@11.0.13/Unknown Source) java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.13/Unknown Source) java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.13/Unknown Source) java.lang.Thread.run(java.base@11.0.13/Unknown Source)
The complete stack trace can be found in the file linked below:
ThreadFullStackTrace
Workaround
No workaround as of now, steps will be shared once we have a workaround
[JSDSERVER-12279] JSM customer notifications delayed due to existing expired tokens in cwd_user_attributes
Remote Link | New: This issue links to "Page (Confluence)" [ 761210 ] |
Remote Link | New: This issue links to "ACE-2975 (Atlassian Support System)" [ 734165 ] |
Remote Link | New: This issue links to "PSR-831 (Bulldog)" [ 734236 ] |
Remote Link | New: This issue links to "Page (Confluence)" [ 725046 ] |
Description |
Original:
h3. Issue Summary
Outgoing mail notifications are delayed because of vast numbers of expired tokens in cwd_user_attributes. h3. Steps to Reproduce * If we have customers who never log in to the customer portal but create requests through email, most users have never logged in to Jira. * In this scenario, Jira will send an invite token for each notification to such users. All those tokens are expired in cwd_user_attributes and are not deleted from the table. h3. Expected Results We should have the best way of reading and updating the DB table and refrain from regularly deleting expired tokens. h3. Actual Results In the thread dumps, we can see a long running thread which name has the syntax *PsmqAsyncExecutors-job:thread-XXX*: !ThreadDumps.png|thumbnail! The stack trace of this long running is similar to this: {code} com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:464) com.google.common.collect.ImmutableSet$Builder.addAll(ImmutableSet.java:541) com.atlassian.jira.crowd.embedded.ofbiz.OfBizAttributesBuilder.addAttribute(OfBizAttributesBuilder.java:46) com.atlassian.jira.crowd.embedded.ofbiz.OfBizAttributesBuilder.toAttributes(OfBizAttributesBuilder.java:29) com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao$UserAttributesCacheLoader.load(OfBizUserDao.java:1134) com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao$UserAttributesCacheLoader.load(OfBizUserDao.java:1124) com.atlassian.cache.memory.MemoryCacheManager$2.load(MemoryCacheManager.java:205) com.atlassian.cache.memory.DelegatingCache.lambda$get$0(DelegatingCache.java:165) com.atlassian.cache.memory.DelegatingCache$$Lambda$680/0x000000080075cc40.get(Unknown Source) com.atlassian.cache.memory.DelegatingCache.lambda$get$1(DelegatingCache.java:193) com.atlassian.cache.memory.DelegatingCache$$Lambda$681/0x000000080075c040.call(Unknown Source) com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4868) com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533) com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282) com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159) com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049) com.google.common.cache.LocalCache.get(LocalCache.java:3966) com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4863) com.atlassian.cache.memory.DelegatingCache.get(DelegatingCache.java:180) com.atlassian.cache.memory.DelegatingCache.get(DelegatingCache.java:165) com.atlassian.cache.impl.metrics.InstrumentedCache.get(InstrumentedCache.java:72) com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.getUserWithAttributes(OfBizUserDao.java:953) com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.findByNameWithAttributes(OfBizUserDao.java:304) com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.findByNameWithAttributes(DelegatingUserDao.java:77) com.atlassian.jira.crowd.embedded.ofbiz.IndexedUserDao.findByNameWithAttributes(IndexedUserDao.java:67) com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.findByNameWithAttributes(DelegatingUserDao.java:77) com.atlassian.jira.crowd.embedded.ofbiz.SwitchingUserDao.findByNameWithAttributes(SwitchingUserDao.java:30) ... com.atlassian.servicedesk.plugins.notifications.internal.rendering.NotificationContentRendererImpl.renderContent(NotificationContentRendererImpl.java:48) com.atlassian.servicedesk.plugins.notifications.internal.rendering.NotificationEmailBodyRendererImpl.renderHtmlContent(NotificationEmailBodyRendererImpl.java:56) com.atlassian.servicedesk.plugins.notifications.internal.dispatch.NotificationBatchEntryCreator.consume(NotificationBatchEntryCreator.java:56) com.atlassian.servicedesk.plugins.notifications.internal.dispatch.NotificationMessageProcessor.lambda$processMessage$1(NotificationMessageProcessor.java:69) com.atlassian.servicedesk.plugins.notifications.internal.dispatch.NotificationMessageProcessor$$Lambda$8947/0x0000000804e16840.apply(Unknown Source) ... com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.context.AutomationExecutionContextImpl.runInAutomationContext(AutomationExecutionContextImpl.java:27) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.lambda$processJobs$5(PsmqExecutionJobRunnerImpl.java:168) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl$$Lambda$8772/0x0000000804de0840.apply(Unknown Source) com.atlassian.pocketknife.step.OptionalStep2.lambda$null$0(OptionalStep2.java:20) com.atlassian.pocketknife.step.OptionalStep2$$Lambda$8774/0x0000000804de1040.apply(Unknown Source) java.util.Optional.flatMap(java.base@11.0.13/Unknown Source) com.atlassian.pocketknife.step.OptionalStep2.lambda$then$1(OptionalStep2.java:20) com.atlassian.pocketknife.step.OptionalStep2$$Lambda$8773/0x0000000804de0c40.apply(Unknown Source) java.util.Optional.flatMap(java.base@11.0.13/Unknown Source) com.atlassian.pocketknife.step.OptionalStep2.then(OptionalStep2.java:20) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.processJobs(PsmqExecutionJobRunnerImpl.java:167) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.accessQueueAndProcessJobs(PsmqExecutionJobRunnerImpl.java:149) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.lambda$runJobsInQueues$1(PsmqExecutionJobRunnerImpl.java:83) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl$$Lambda$8757/0x0000000804dd4c40.run(Unknown Source) java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.13/Unknown Source) java.util.concurrent.FutureTask.run(java.base@11.0.13/Unknown Source) java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.13/Unknown Source) java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.13/Unknown Source) java.lang.Thread.run(java.base@11.0.13/Unknown Source) {code} The complete stack trace can be found in the file linked below: [^ThreadFullStackTrace] h3. Workaround No workaround as of now, steps will be shared once we have a workaround |
New:
h3. Issue Summary
The Jira Service Management (JSM) Customer Notifications are delayed because of vast numbers of expired tokens in cwd_user_attributes. h3. Steps to Reproduce * If we have customers who never log in to the customer portal but create requests through email, most users have never logged in to Jira. * In this scenario, Jira will send an invite token for each notification to such users. All those tokens are expired in cwd_user_attributes and are not deleted from the table. h3. Expected Results We should have the best way of reading and updating the DB table and refrain from regularly deleting expired tokens. h3. Actual Results In the thread dumps, we can see a long running thread which name has the syntax *PsmqAsyncExecutors-job:thread-XXX*: !ThreadDumps.png|thumbnail! The stack trace of this long running is similar to this: {code} com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:464) com.google.common.collect.ImmutableSet$Builder.addAll(ImmutableSet.java:541) com.atlassian.jira.crowd.embedded.ofbiz.OfBizAttributesBuilder.addAttribute(OfBizAttributesBuilder.java:46) com.atlassian.jira.crowd.embedded.ofbiz.OfBizAttributesBuilder.toAttributes(OfBizAttributesBuilder.java:29) com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao$UserAttributesCacheLoader.load(OfBizUserDao.java:1134) com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao$UserAttributesCacheLoader.load(OfBizUserDao.java:1124) com.atlassian.cache.memory.MemoryCacheManager$2.load(MemoryCacheManager.java:205) com.atlassian.cache.memory.DelegatingCache.lambda$get$0(DelegatingCache.java:165) com.atlassian.cache.memory.DelegatingCache$$Lambda$680/0x000000080075cc40.get(Unknown Source) com.atlassian.cache.memory.DelegatingCache.lambda$get$1(DelegatingCache.java:193) com.atlassian.cache.memory.DelegatingCache$$Lambda$681/0x000000080075c040.call(Unknown Source) com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4868) com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3533) com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2282) com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2159) com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2049) com.google.common.cache.LocalCache.get(LocalCache.java:3966) com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4863) com.atlassian.cache.memory.DelegatingCache.get(DelegatingCache.java:180) com.atlassian.cache.memory.DelegatingCache.get(DelegatingCache.java:165) com.atlassian.cache.impl.metrics.InstrumentedCache.get(InstrumentedCache.java:72) com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.getUserWithAttributes(OfBizUserDao.java:953) com.atlassian.jira.crowd.embedded.ofbiz.OfBizUserDao.findByNameWithAttributes(OfBizUserDao.java:304) com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.findByNameWithAttributes(DelegatingUserDao.java:77) com.atlassian.jira.crowd.embedded.ofbiz.IndexedUserDao.findByNameWithAttributes(IndexedUserDao.java:67) com.atlassian.jira.crowd.embedded.ofbiz.DelegatingUserDao.findByNameWithAttributes(DelegatingUserDao.java:77) com.atlassian.jira.crowd.embedded.ofbiz.SwitchingUserDao.findByNameWithAttributes(SwitchingUserDao.java:30) ... com.atlassian.servicedesk.plugins.notifications.internal.rendering.NotificationContentRendererImpl.renderContent(NotificationContentRendererImpl.java:48) com.atlassian.servicedesk.plugins.notifications.internal.rendering.NotificationEmailBodyRendererImpl.renderHtmlContent(NotificationEmailBodyRendererImpl.java:56) com.atlassian.servicedesk.plugins.notifications.internal.dispatch.NotificationBatchEntryCreator.consume(NotificationBatchEntryCreator.java:56) com.atlassian.servicedesk.plugins.notifications.internal.dispatch.NotificationMessageProcessor.lambda$processMessage$1(NotificationMessageProcessor.java:69) com.atlassian.servicedesk.plugins.notifications.internal.dispatch.NotificationMessageProcessor$$Lambda$8947/0x0000000804e16840.apply(Unknown Source) ... com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.context.AutomationExecutionContextImpl.runInAutomationContext(AutomationExecutionContextImpl.java:27) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.lambda$processJobs$5(PsmqExecutionJobRunnerImpl.java:168) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl$$Lambda$8772/0x0000000804de0840.apply(Unknown Source) com.atlassian.pocketknife.step.OptionalStep2.lambda$null$0(OptionalStep2.java:20) com.atlassian.pocketknife.step.OptionalStep2$$Lambda$8774/0x0000000804de1040.apply(Unknown Source) java.util.Optional.flatMap(java.base@11.0.13/Unknown Source) com.atlassian.pocketknife.step.OptionalStep2.lambda$then$1(OptionalStep2.java:20) com.atlassian.pocketknife.step.OptionalStep2$$Lambda$8773/0x0000000804de0c40.apply(Unknown Source) java.util.Optional.flatMap(java.base@11.0.13/Unknown Source) com.atlassian.pocketknife.step.OptionalStep2.then(OptionalStep2.java:20) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.processJobs(PsmqExecutionJobRunnerImpl.java:167) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.accessQueueAndProcessJobs(PsmqExecutionJobRunnerImpl.java:149) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl.lambda$runJobsInQueues$1(PsmqExecutionJobRunnerImpl.java:83) com.atlassian.servicedesk.plugins.automation.internal.execution.engine.runners.async.psmq.PsmqExecutionJobRunnerImpl$$Lambda$8757/0x0000000804dd4c40.run(Unknown Source) java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.13/Unknown Source) java.util.concurrent.FutureTask.run(java.base@11.0.13/Unknown Source) java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.13/Unknown Source) java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.13/Unknown Source) java.lang.Thread.run(java.base@11.0.13/Unknown Source) {code} The complete stack trace can be found in the file linked below: [^ThreadFullStackTrace] h3. Workaround No workaround as of now, steps will be shared once we have a workaround |
Summary | Original: Outgoing notifications delayed due to existing expired tokens in cwd_user_attributes | New: JSM customer notifications delayed due to existing expired tokens in cwd_user_attributes |
Resolution | New: Fixed [ 1 ] | |
Status | Original: Waiting for Release [ 12075 ] | New: Closed [ 6 ] |
Remote Link | New: This issue links to "Page (Confluence)" [ 722509 ] |
Fix Version/s | New: 5.3.3 [ 104303 ] | |
Fix Version/s | New: 5.6.0 [ 104395 ] | |
Fix Version/s | New: 5.4.2 [ 104393 ] | |
Fix Version/s | New: 5.5.1 [ 104298 ] |
Status | Original: In Progress [ 3 ] | New: Waiting for Release [ 12075 ] |