Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-39375

Enable JQL functions to opt out of heavy processing

XMLWordPrintable

    • 10
    • 23
    • We collect Jira feedback from various sources, and we evaluate what we've collected when planning our product roadmap. To understand how this piece of feedback will be reviewed, see our Implementation of New Features Policy.

      NOTE: This suggestion is for JIRA Server. Using JIRA Cloud? See the corresponding suggestion.

      Since JQL functions work as macros, whenever they return a big list JIRA needs to go through the whole list and validate the interim results before proceeding with the query.
      Even though query is limited by other clauses, but JQL doesn't have smarts and calculates the context, even though a much more restricted context is provided by another clause.
      Some improvements have been made to reduce the pain by batching and doing less work, but it doesn't scale beyond 100-200K issues.
      We need to provide means to functions to override some of this behaviour.

      Partial fix

      First step to address this problem was done in JRASERVER-63665
      JIRA prepares a search with a small limit set (1000) and check if totalHits is bigger. If it's bigger then we do a second search and we exactly know the limit now (totalHits from the previous search).

      Additional logging

      Additional logs can help to find a plugin which is doing such searches. To enable this:

      • Set com.atlassian.jira.issue.search.providers.LuceneSearchProvider to DEBUG level to capture expensive JQLs:
        • It logs only queries which try to fetch more than 1000 documents.
        • Example of the logging: 
          "Search for the query <> was performed and results count <> exceeds max allowed search limit 
Large result sets could lead to OutOfMemoryError. If you're sure you need to do the query you can remove the limit by enabling feature."

or 

"Search for the query [<>] was performed and results count (<>) exceeds default limit"

            [JRASERVER-39375] Enable JQL functions to opt out of heavy processing

            Abe.Roba added a comment -

            10 + years and it is still gathering interest!  

            Abe.Roba added a comment - 10 + years and it is still gathering interest!  

            Andriy Yakovlev [Atlassian] added a comment -

            Note on ScriptRunner, since 3.1.3 they added pagination to the JQL functions code :

            All included JQL functions should now run in constant memory. Previously, certain functions such as linkedIssuesOf, when using against a very large number of issues, would consume too much memory.

            see: https://jamieechlin.atlassian.net/wiki/display/GRV/ScriptRunner+3.1.3

            Andriy Yakovlev [Atlassian] added a comment - Note on ScriptRunner, since 3.1.3 they added pagination to the JQL functions code : All included JQL functions should now run in constant memory. Previously, certain functions such as linkedIssuesOf, when using against a very large number of issues, would consume too much memory. see: https://jamieechlin.atlassian.net/wiki/display/GRV/ScriptRunner+3.1.3

            Jobin Kuruvilla [Adaptavist] added a comment -

            Thanks David for reaching out. I have responded to the mail. Hopefully, we can get to the bottom of it

            Jobin Kuruvilla [Adaptavist] added a comment - Thanks David for reaching out. I have responded to the mail. Hopefully, we can get to the bottom of it

            Philip Schlesinger added a comment -

            I should also note that ScriptRunner 3.x's "issueFunction in parentsOf()" function runs in nearly the same amount of time as JQL Trick 5.3.x's "issue in parent()" function. I would assume both options are affected by the same underlying JIRA limitation.

            Philip Schlesinger added a comment - I should also note that ScriptRunner 3.x's "issueFunction in parentsOf()" function runs in nearly the same amount of time as JQL Trick 5.3.x's "issue in parent()" function. I would assume both options are affected by the same underlying JIRA limitation.

            Dave C added a comment -

            Hey jobinkk@gmail.com, I've sent you an email about this to see if we can discuss it further.

            Dave C added a comment - Hey jobinkk@gmail.com , I've sent you an email about this to see if we can discuss it further.

            Jobin Kuruvilla [Adaptavist] added a comment -

            Hello Atlassian,

            Is there any progress on this issue? Unfortunately, this has become a big problem for lot of our customers and there is nothing we can do in a plugin to tackle this. Disabling the plugins/jql functions will help stabilizing the system but that takes away the precious functionality too.

            Given that JQL and searching is fundamental to the business for many customers, can we bump up the priority for this? Or, are there any other workarounds that you can suggest?

            Appreciate all the help. We have seen improvements in the later versions of JIRA but none providing marked improvements. This one seems to be the key issue. Hopefully, you can get this in one of the upcoming minor versions and that will make searching in our favorite project management system all the more fun

            Jobin Kuruvilla [Adaptavist] added a comment - Hello Atlassian, Is there any progress on this issue? Unfortunately, this has become a big problem for lot of our customers and there is nothing we can do in a plugin to tackle this. Disabling the plugins/jql functions will help stabilizing the system but that takes away the precious functionality too. Given that JQL and searching is fundamental to the business for many customers, can we bump up the priority for this? Or, are there any other workarounds that you can suggest? Appreciate all the help. We have seen improvements in the later versions of JIRA but none providing marked improvements. This one seems to be the key issue. Hopefully, you can get this in one of the upcoming minor versions and that will make searching in our favorite project management system all the more fun

            boardtc added a comment - - edited

            As reported by Philip Schlesinger we have seen the exact same timeout issue on a kanban board using issue in parent("issuetype = task"))) with 6.3.15

            boardtc added a comment - - edited As reported by Philip Schlesinger we have seen the exact same timeout issue on a kanban board using issue in parent("issuetype = task"))) with 6.3.15

            Adolfo Guzman added a comment -

            Atlassian please fix.

            Our JIRA instance was going down twice a week when users were running heavy queries and doubled when our PMO started sprint planning. Temporary workaround for us was to contact our heavy JQL users and modify their query/filters and asked them to run non peak hours.

            Adolfo Guzman added a comment - Atlassian please fix. Our JIRA instance was going down twice a week when users were running heavy queries and doubled when our PMO started sprint planning. Temporary workaround for us was to contact our heavy JQL users and modify their query/filters and asked them to run non peak hours.

            Philip Schlesinger added a comment -

            BTW this is affecting 6.4.4

            Philip Schlesinger added a comment - BTW this is affecting 6.4.4

            Philip Schlesinger added a comment -

            Using the parent() function from JQL Tricks, it's more than doubling the time for some of our Agile boards to load. Sometimes those boards report a timeout error.

            Please fix this ASAP.

            Philip Schlesinger added a comment - Using the parent() function from JQL Tricks, it's more than doubling the time for some of our Agile boards to load. Sometimes those boards report a timeout error. Please fix this ASAP.

              Unassigned Unassigned
              natashbar Norman Atashbar
              Votes:
              66 Vote for this issue
              Watchers:
              58 Start watching this issue

                Created:
                Updated: