Date-Range Lucene queries very slow - Makes Dashboard very slow if Saved Filters contains a date-range search

XMLWordPrintable

    • Type: Suggestion
    • Resolution: Fixed
    • 3.3
    • Component/s: Indexing
    • None

      Currently you are doing a after/before date-range search as two separate RangeQueries (the intersection of the two).

      This really hammers lucene. The memory utilisation goes through the roof and takes ages to complete - e.g. 2000ms to complete instead of 70ms.

      This particularly hits the Saved Filters hard - as they now display the number of hits.
      If you have several filters with date-ranges, then you have a multiplier effect on the performance impact.

      To fix this, you will need to treat both elements of a date range as a single search term - and pass a single range query term to Lucene.

      I will attach sample code to illustrate the problem:

      Here is the log file from jira (displayin the dashboard)
      You can see from the amount of GC activity that there are 3 very expensive queries ...

      [GC 60526K->51003K(129728K), 0.0211332 secs]
      2005-07-22 14:37:11,503 DEBUG [issue.search.providers.LuceneSearchProvider] Lucene boolean Query:+(status:3) +projid:10516
      2005-07-22 14:37:11,534 DEBUG [issue.search.providers.LuceneSearchProvider] Lucene boolean Query:+created:[0e8xl8bk0 TO null] +created:[ TO 0e9q61cw0] +projid:10516
      [GC 62779K->58760K(129728K), 0.0668692 secs]
      [GC 70536K->67769K(129728K), 0.0412392 secs]
      [GC 79544K->78870K(129728K), 0.0407540 secs]
      [GC 90645K->89971K(129728K), 0.0358531 secs]
      [GC 101747K->101073K(129728K), 0.0392850 secs]
      [GC 112848K->112174K(129728K), 0.0373389 secs]
      [GC 123949K->123275K(135104K), 0.0440918 secs]
      [Full GC 123275K->114191K(135104K), 0.7666367 secs]
      2005-07-22 14:37:14,034 DEBUG [issue.search.providers.LuceneSearchProvider] Lucene boolean Query:+(status:6) +updated:[0e8xl8bk0 TO null] +updated:[ TO 0e9q61cw0] +projid:10516
      [GC 133327K->128253K(211824K), 0.0755934 secs]
      [GC 147389K->146292K(211824K), 0.0891377 secs]
      [GC 165428K->164333K(211824K), 0.0850359 secs]
      [GC 183468K->182371K(211824K), 0.0919707 secs]
      2005-07-22 14:37:15,659 DEBUG [issue.search.providers.LuceneSearchProvider] Lucene boolean Query:+(priority:1) +created:[0e8xl8bk0 TO null] +created:[ TO 0e9q61cw0] +projid:10516
      [GC 201506K->196536K(215792K), 0.1144801 secs]
      [Full GC 196536K->58595K(215792K), 0.4573202 secs]
      [GC 77731K->76082K(211824K), 0.0507644 secs]
      [GC 95218K->93700K(211824K), 0.0694637 secs]
      [GC 112835K->111322K(211824K), 0.0576076 secs]
      2005-07-22 14:37:17,940 DEBUG [issue.search.providers.LuceneSearchProvider] Lucene boolean Query:+(status:1 status:4) +issue_assignee_group:ici-developers +projid:10516
      2005-07-22 14:37:17,956 DEBUG [issue.search.providers.LuceneSearchProvider] Lucene boolean Query:+(status:1 status:4) +issue_assignee:unassigned +projid:10516
      2005-07-22 14:37:18,112 DEBUG [issue.search.providers.LuceneSearchProvider] Lucene boolean Query:+issue_assignee:nminutello +(resolution:-1)
      [GC 130458K->120798K(211824K), 0.0497319 secs]
      2005-07-22 14:37:18,362 DEBUG [issue.search.providers.LuceneSearchProvider] Lucene boolean Query:+issue_assignee:nminutello +(status:3) +(resolution:-1)
      [7688ms] - /secure/Dashboard.jspa
      [562ms] - PortletColumn
      [15ms] - Running portlet: /portlets/dashboard/introduction.jsp
      [188ms] - Running portlet: /portlets/dashboard/project.jsp
      [16ms] - Searching with HitCollector
      [62ms] - Running portlet: /portlets/dashboard/project.jsp
      [16ms] - Searching with HitCollector
      [281ms] - Running portlet: /portlets/dashboard/projecttable.jsp
      [6985ms] - PortletColumn
      [6469ms] - Running velocity portlet: 13241
      [15ms] - Lucene Search
      [2484ms] - Lucene Search
      [1625ms] - Lucene Search
      [2266ms] - Lucene Search
      [406ms] - Running velocity portlet: 10269
      [265ms] - Lucene Query
      [140ms] - Permission Query
      [125ms] - Lucene Search
      [78ms] - Retrieve From cache/db and filter
      [78ms] - IssueManager.getIssues()
      [110ms] - Running velocity portlet: 10270
      [94ms] - Lucene Query
      [94ms] - Lucene Search

            Assignee:
            MarkC
            Reporter:
            Nick Minutello
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

              Created:
              Updated:
              Resolved: