Issue Summary
Asynchronous webhooks use Jira’s request cache outside of the appropriate context. This causes the webhook executor to generate the JSON payload without access to some caches.
As a consequence, environments with heavy webhook utilization may observe some combination of:
- Delays in webhook delivery
- Excessive database pressure
- Dropped webhooks due to queue/service limits
Async webhooks are opt-in in Jira 9 via a feature flag and the only option in Jira 10.
Note on Invalid use of RequestCache errorsSee JRASERVER-72963#comment-3059444 for more details.
Note: There is a whole family of "Invalid use of RequestCache by thread..." issues, one per thread. Find the one you need or file a new one.
Steps to Reproduce
- Configure one or more [issue event] webhooks, including the JSON payload.
- Induce a high-load (100+ updates/seconds) of matching events.
Expected Results
The webhook executor can process each webhook and build its payload body in a reasonable amount of time. Database pressure is reasonable.
Actual Results
Call stacks that should otherwise quickly return with cached values are delayed. This can observed in thread dumps:
Frames with stacks that pass through a cached resource
Frames with stacks that perform a database query
Request cache warning
A warning can be observed in the atlassian-jira.log after the first webhook is fired on each thread after node startup:
Jira 9 (async webhooks enabled via feature flag)
2025-04-22 14:30:00,000+0000 Web-Hook-Events-Processor-1 ERROR [c.a.j.cache.request.RequestCacheRecorderImpl] Invalid use of RequestCache by thread: Web-Hook-Events-Processor.Incorrect usage of JIRA API. You can only create/use: RequestCacheImpl inside a context (request or Jira-Thread-Local). Check: JiraThreadLocalUtil for details. This message will be logged once per thread. This ERROR should be fixed on the caller side (check the stack-trace or thread-name) as it is causing inconsistent data. java.lang.Throwable at com.atlassian.jira.cache.request.RequestCacheRecorderImpl.errorLogOOCThreadUsage(RequestCacheRecorderImpl.java:32) at com.atlassian.jira.cache.request.RequestCacheRecorderImpl.recordRequestCacheOOCUsage(RequestCacheRecorderImpl.java:25) at com.atlassian.jira.cache.request.RequestCacheImpl.get(RequestCacheImpl.java:47) [...] at com.atlassian.jira.plugins.webhooks.serializer.IssueEventSerializer.putFields(IssueEventSerializer.java:59) at com.atlassian.jira.plugins.webhooks.serializer.IssueEventSerializer.putFields(IssueEventSerializer.java:28) at com.atlassian.jira.plugins.webhooks.serializer.AbstractJiraEventSerializer.serialize(AbstractJiraEventSerializer.java:28) at com.atlassian.webhooks.plugin.module.WebHookEventSerializer.serialize(WebHookEventSerializer.java:32) at com.atlassian.webhooks.plugin.PublishTaskFactoryImpl.getEventBody(PublishTaskFactoryImpl.java:67) at com.atlassian.webhooks.plugin.PublishTaskFactoryImpl.getPublishTask(PublishTaskFactoryImpl.java:61) at com.atlassian.webhooks.plugin.WebHookPublisherImpl.publish(WebHookPublisherImpl.java:116) at com.atlassian.webhooks.plugin.WebHookPublisherImpl.publish(WebHookPublisherImpl.java:83) at com.atlassian.webhooks.plugin.WebHookEventsProcessor.lambda$null$2(WebHookEventsProcessor.java:115) at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840)
Jira 10
2025-04-22 14:30:00,000+0000 webhook-dispatcher:thread-1 ERROR [c.a.j.cache.request.RequestCacheRecorderImpl] Invalid use of RequestCache by thread: webhook-dispatcher:thread.Incorrect usage of JIRA API. You can only create/use: RequestCacheImpl inside a context (request or Jira-Thread-Local). Check: JiraThreadLocalUtil for details. This message will be logged once per thread. This ERROR should be fixed on the caller side (check the stack-trace or thread-name) as it is causing inconsistent data. java.lang.Throwable at com.atlassian.jira.cache.request.RequestCacheRecorderImpl.errorLogOOCThreadUsage(RequestCacheRecorderImpl.java:32) at com.atlassian.jira.cache.request.RequestCacheRecorderImpl.recordRequestCacheOOCUsage(RequestCacheRecorderImpl.java:25) at com.atlassian.jira.cache.request.RequestCacheImpl.get(RequestCacheImpl.java:46) [...] at com.atlassian.jira.plugins.webhooks.serializer.WebhookJsonEventFactory.lambda$build$19(WebhookJsonEventFactory.java:200) at java.base/java.util.Optional.filter(Optional.java:218) at com.atlassian.jira.plugins.webhooks.serializer.WebhookJsonEventFactory.build(WebhookJsonEventFactory.java:200) at com.atlassian.jira.plugins.webhooks.serializer.WebhookJsonEventFactory.build(WebhookJsonEventFactory.java:124) at com.atlassian.jira.plugins.webhooks.spi.JiraWebhookEventPublisher.lambda$onIssueEvent$0(JiraWebhookEventPublisher.java:122) at java.base/java.util.Optional.map(Optional.java:260) at com.atlassian.webhooks.WebhookPublishRequest.getPayload(WebhookPublishRequest.java:154) at com.atlassian.webhooks.internal.publish.DefaultWebhookInvocation.<init>(DefaultWebhookInvocation.java:67) at com.atlassian.webhooks.internal.publish.DefaultWebhookInvocation.<init>(DefaultWebhookInvocation.java:44) at com.atlassian.webhooks.internal.DefaultWebhookService.createSingleInvocationFor(DefaultWebhookService.java:445) at com.atlassian.webhooks.internal.DefaultWebhookService.lambda$createInvocationsFor$15(DefaultWebhookService.java:438) [...] at com.atlassian.webhooks.internal.DefaultWebhookService.createInvocationsFor(DefaultWebhookService.java:439) at com.atlassian.webhooks.internal.DefaultWebhookService.lambda$publish$10(DefaultWebhookService.java:271) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840)
Workaround
Jira 9
- Increase webhook executor thread pool and queue sizes, as described on
JRASERVER-74492. - Switch back to synchronous webhooks.
Jira 10
Increase webhook executor thread pool and queue sizes, as described on Tuning webhook performance parameters in Jira Data Center.
- was cloned as
-
JRASERVER-79100 Some async webhooks do not use request cache in Jira Data Center
-
- Closed
-
- was split from
-
-
- Closed
-
- details
-
BORG-1180 Loading...