Uploaded image for project: 'Jira Service Management Data Center'
  1. Jira Service Management Data Center
  2. JSDSERVER-4897

Loading issue count is slow when there are many queues in single project

    • Icon: Suggestion Suggestion
    • Resolution: Fixed
    • 3.4.2
    • None
    • None
    • We collect Jira Service Desk feedback from various sources, and we evaluate what we've collected when planning our product roadmap. To understand how this piece of feedback will be reviewed, see our Implementation of New Features Policy.

      NOTE: This suggestion is for JIRA Service Desk Server. Using JIRA Service Desk Cloud? See the corresponding suggestion.

      Problem Definition

      A customer saw queues taking up to 20s to load (with 35 queues in a single projects). This was because the count issues method uses jql search using LuceneSearchProvider in JIRA core to count issues for any queue. Issues for all queues are counted at page load, queue update, add issue and the polling on the queues page, to show the issue count lozenge on the left hand panel.

      Suggested Solution

      1. store the project specific setting using the GlobalPropertyDao
      2. currently issue count is done by JQL search in JIRA core
      3. QueueServiceImpl accepts a bool to shouldCount for queues
      4. CustomQueuesItemProvider:getqueues provides a true value for shouldCount
      5. QueueAssemblerImpl:getQueues provides a shouldCount value that is passed through API call and that is false by default.

      New experimental settings

      As of 3.4.2 there will be a new experimental admin REST api available to set whether the queue count should be included on the Queues page.

      A global setting that can be set by JIRA admin is available:

      PUT <contextpath>/rest/servicedeskapi/admin/queues/include-count - true/false body - 200 response (default true)
      

      There is also a project level setting that can be set by either a JIRA or Project Admin.

      PUT <contextpath>/rest/servicedeskapi/admin/queues/<projectkey>/include-count - true/false body - 200 response (default true)
      

      NB: If the global setting is off, then queue count will not be enabled for any project, as it will take precedence.

      This is an experimental api and therefore as it may be moved, or replaced in the future, until it becomes public, will require opt-in with the Experimental-API header

      Workaround

      1. Reduce the amount of queues per project
      2. Simplify the JQL search in each queues (avoid using SLA check when possible)

            [JSDSERVER-4897] Loading issue count is slow when there are many queues in single project

            We've published a KB to help diagnose when to use this feature and include some curl examples: How to resolve slow queues view with high CPU in Jira Service Management

            Allan Gandelman added a comment - We've published a KB to help diagnose when to use this feature and include some curl examples: How to resolve slow queues view with high CPU in Jira Service Management

            Has this issue been fixed in 3.4.2 or should we still use the experimental API workaround?
            The workaround has worked in recent version of JSD and helped on our performance issue, JIRA was running out of memory due this issue)

            Italo Qualisoni [e-Core] added a comment - - edited Has this issue been fixed in 3.4.2 or should we still use the experimental API workaround? The workaround has worked in recent version of JSD and helped on our performance issue, JIRA was running out of memory due this issue)

            Hi Matt,

            Is there a way to reduce refresh time of cached queue count? 2 mins is too long.

            Regards

            Hailin

            Hailin Zhang added a comment - Hi Matt, Is there a way to reduce refresh time of cached queue count? 2 mins is too long. Regards Hailin

            Hi maarten.cautreels

            The queue count will be cached for a minimum of 2 minutes.

            The way it works, is roughly:

            • One queue count request, it will retrieve the existing value in cache (whatever that is, even 0)
            • Add a task to execute on another thread that will perform the count query. This uses a cache key that makes the result unique for the calling user, for each queue.

            The frequency that the queue count is requested, will likely have to do with the frequency of changes occurring on the project, otherwise when the queue decides to force a refresh.

            • The Executor that runs the tasks has a thread pool size of 5, so if there are many requests occurring simultaneously, then they will be queued up, and therefore may seem to not execute as quickly as expect due to the load. This may be visible if the queues contain rather complex or time consuming JQL queries to process.

            At the moment the pool size is not configurable. However, this could be made possible if desired, or otherwise the current size could be increased.

            If feel that either of these would be beneficial to your use case, would greatly recommend replying here with your preference, or raising a support request to get a new ticket created.

            Hope this helps explain the intent of the queue caching, as if the problem is long running JQL queries, it can be expected that queue sizes may be out of date for a period of time, but should eventually catch up.

            The intention is that this behaviour is an alternative to turning off the queue count entirely, for projects that otherwise take too long to load if needs to wait for all this data to be processed before displaying the page.

            Regards
            Matt
            JIRA Service Desk Server developer

            Matthew McMahon (Inactive) added a comment - Hi maarten.cautreels The queue count will be cached for a minimum of 2 minutes. The way it works, is roughly: One queue count request, it will retrieve the existing value in cache (whatever that is, even 0) Add a task to execute on another thread that will perform the count query. This uses a cache key that makes the result unique for the calling user, for each queue. The frequency that the queue count is requested, will likely have to do with the frequency of changes occurring on the project, otherwise when the queue decides to force a refresh. The Executor that runs the tasks has a thread pool size of 5, so if there are many requests occurring simultaneously, then they will be queued up, and therefore may seem to not execute as quickly as expect due to the load. This may be visible if the queues contain rather complex or time consuming JQL queries to process. At the moment the pool size is not configurable. However, this could be made possible if desired, or otherwise the current size could be increased. If feel that either of these would be beneficial to your use case, would greatly recommend replying here with your preference, or raising a support request to get a new ticket created. Hope this helps explain the intent of the queue caching, as if the problem is long running JQL queries, it can be expected that queue sizes may be out of date for a period of time, but should eventually catch up. The intention is that this behaviour is an alternative to turning off the queue count entirely, for projects that otherwise take too long to load if needs to wait for all this data to be processed before displaying the page. Regards Matt JIRA Service Desk Server developer

            We've enabled this on our instance this weekend but it looks like it caches the queue count for a very long time (10 minutes or more?)

            Is that something we can configure? Or how does it exactly work? 

            Maarten Cautreels added a comment - We've enabled this on our instance this weekend but it looks like it caches the queue count for a very long time (10 minutes or more?) Is that something we can configure? Or how does it exactly work? 

            David Yu added a comment - - edited

            Is the experimental API only in cloud? I have 3.2.7 Server installed and getting 404 on "GET <contextpath>/rest/servicedeskapi/admin/queues/<projectkey>" using the experimental header.

            Nm, seems my JSD didn't upgrade automatically with my JIRA upgrade!

            David Yu added a comment - - edited Is the experimental API only in cloud? I have 3.2.7 Server installed and getting 404 on "GET <contextpath>/rest/servicedeskapi/admin/queues/<projectkey>" using the experimental header. Nm, seems my JSD didn't upgrade automatically with my JIRA upgrade!

            David Yu added a comment -

            That's great Matthew, can't wait to try it.

            David Yu added a comment - That's great Matthew, can't wait to try it.

            Hi dyu1856991738

            We have actually just finished implementing an experimental feature that allows the queue count to be requested off thread periodically, with the latest cached value returned immediately on each call to update the queues.

            This will be enabled in a similar way, to the include count, using a REST API.

            There is a global setting.

            PUT <contextpath>/rest/servicedeskapi/admin/queues/cache-count - true/false body - 200 response (default false)

            And project level, that combine to determine the state. This also will only function if Queue Counts are included at all.

            PUT <contextpath>/rest/servicedeskapi/admin/queues/<projectkey>/cache-count - true/false body - 200 response (default false)
            

            Finally, to assist with understanding the current state, and whether the queue count is enabled on project, and whether it is cached.

            GET <contextpath>/rest/servicedeskapi/admin/queues/<projectkey> - 200 response
            

            Matthew McMahon (Inactive) added a comment - Hi dyu1856991738 We have actually just finished implementing an experimental feature that allows the queue count to be requested off thread periodically, with the latest cached value returned immediately on each call to update the queues. This will be enabled in a similar way, to the include count, using a REST API. There is a global setting. PUT <contextpath>/ rest /servicedeskapi/admin/queues/cache-count - true / false body - 200 response ( default false ) And project level, that combine to determine the state. This also will only function if Queue Counts are included at all. PUT <contextpath>/ rest /servicedeskapi/admin/queues/<projectkey>/cache-count - true / false body - 200 response ( default false ) Finally, to assist with understanding the current state, and whether the queue count is enabled on project, and whether it is cached. GET <contextpath>/ rest /servicedeskapi/admin/queues/<projectkey> - 200 response

            David Yu added a comment -

            Taking away the counts completely would defeat some of the functionality of the queues. Would be similar to just using favorited filters in JIRA. If the queues count could update in the background periodically without affecting the UI navigation speed, that would be ideal.

            David Yu added a comment - Taking away the counts completely would defeat some of the functionality of the queues. Would be similar to just using favorited filters in JIRA. If the queues count could update in the background periodically without affecting the UI navigation speed, that would be ideal.

            As of 3.4.2 there will be a new experimental admin REST api available to set whether the queue count should be included on the Queues page.

            A global setting that can be set by JIRA admin is available:

            PUT <contextpath>/rest/servicedeskapi/admin/queues/include-count - true/false body - 200 response (default true)
            

            There is also a project level setting that can be set by either a JIRA or Project Admin.

            PUT <contextpath>/rest/servicedeskapi/admin/queues/<projectkey>/include-count - true/false body - 200 response (default true)
            

            NB: If the global setting is off, then queue count will not be enabled for any project, as it will take precedence.

            This is an experimental api and therefore as it may be moved, or replaced in the future, until it becomes public, will require opt-in with the Experimental-API header

            Matthew McMahon (Inactive) added a comment - As of 3.4.2 there will be a new experimental admin REST api available to set whether the queue count should be included on the Queues page. A global setting that can be set by JIRA admin is available: PUT <contextpath>/ rest /servicedeskapi/admin/queues/include-count - true / false body - 200 response ( default true ) There is also a project level setting that can be set by either a JIRA or Project Admin. PUT <contextpath>/ rest /servicedeskapi/admin/queues/<projectkey>/include-count - true / false body - 200 response ( default true ) NB: If the global setting is off, then queue count will not be enabled for any project, as it will take precedence. This is an experimental api and therefore as it may be moved, or replaced in the future, until it becomes public, will require opt-in with the Experimental-API header

              Unassigned Unassigned
              vkharisma vkharisma
              Votes:
              5 Vote for this issue
              Watchers:
              22 Start watching this issue

                Created:
                Updated:
                Resolved:

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 4h
                  4h
                  Logged:
                  Time Spent - 0h
                  0h