Issue Summary

Large spaces with a large set of watchers can trigger a lot of batch notification traffic that leads to significant delays in notification delivery.

• This is the flow :
  • every batch email notification task targets a set of recipients
    • in the case of large space with significant number of watchers,  ALL the space watchers
  •  every 10 minutes the batch notification scheduled job kicks in
    • it generates a set of notification tasks for each content updated
      • in the case of large space, this is every page/blog contributed to in the span of 10 minutes
  •  every 10 minutes the thread pool executor schedules (enqueues) a new set of batch notification tasks
    • it comes with 3 worker threads by default
    • the number of recipients per task impacts the time needed for a worker thread to free up
    • the time for a worker thread to free up impacts the waiting time for every task sitting in the queue
    • the order at which tasks were queued impacts when they get executed

• So to sum up
  • for a popular space with a lot of recipients and without a limit to number of recipients per notification task
    • the worker threads get "greedy" tasks to process, and while that is going on, every 10 minutes a scheduled job adds more items to the task queue

Steps to Reproduce

1. Generate some traffic on a large data set
2. Monitor the notification queue, and notice the incremental wait for email tasks in the queue.

Expected Results

Notifications are sent within a reasonable timeframe that ensures the content sent is still of relevance

Actual Results

Notification tasks can sit in the queue for more than a week

Workaround

system properties (requires a restart)

• we can increase the notifications.sender.thread.count from its default value of 3, this will provide batch notifications with more worker threads

scheduled job

• we can configure Send Batched Notifications to run at a different frequency, it is configured to run every 10 minutes by default