Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-22453

Regression: Comment searching causes too many clauses error

    XMLWordPrintable

Details

    Description

      Please resurrect IssueIdFilter rather than using a BooleanQuery of individual TermQuery's of issue id's inside CommentClauseQueryFactory.

      Searching for Comment contains 'bug' on Jira 3.13 returns 53000 issues and works fine.
      Searching for comment ~ "bug" on our Jira 4.1.2 test server that has the same data as the Jira 3.13 server returns no results and causes

      The 'comment ~ "bug"' clause in your search matches too many results. If a clause returns too many results, the entire search will fail. Please refine or remove the clause and try the search again.

      This is not acceptable.

      Diving into the code with remote debug running, I see that the problem is that a search is being made on the comment index and then each hit from this is converted to an issue_id TermQuery. Each of these TermQuery's are collected in one massive BooleanQuery. That query with over 65000 term queries is then run against the issue index. Since the word bug appears in more than 65000 comments in our Jira, this causes the TooManyClauses exception.

      In Jira 3 the same query would search the comment index and then the hits were converted to an IssueIdFilter which was wrapped with a FilteredQuery.
      This is faster and doesn't cause the TooManyClauses exception.

      Please resurrect IssueIdFilter rather than using a BooleanQuery of individual TermQuery's of issue id's inside CommentClauseQueryFactory.

      Another improvement would be to collect the issue id's in a set since issues can have more than one comment.

      With the improvement that I suggested in JRA-22448 to speed up comment searching by using a HitCollector rather than looping over the hits returned from the comment index, our CommentClauseQueryFactory.generateIssueIdQueryFromCommentQuery now looks like this

         QueryFactoryResult generateIssueIdQueryFromCommentQuery(final Query commentIndexQuery, final QueryCreationContext creationContext) {
            try 
            {
               // Now we need to execute the search and build up a new query that enumerates the issue id's found by the comment search
               final Searcher commentSearcher = searchProviderFactory.getSearcher(SearchProviderFactory.COMMENT_INDEX);
               final Filter filter = getPermissionsFilter(creationContext.isSecurityOverriden(), creationContext.getUser());
               final Collection<String> ids = new HashSet<String>(10000);
               final HitCollector hitCollector = new DocumentHitCollector(commentSearcher) {
                  @Override
                  public void collect(Document doc) {
                     ids.add(doc.get(DocumentConstants.ISSUE_ID));
                  }
               };
               // JRA-19306 - lets restrict this search by the permissions filter so we can try to avoid the MAX_CLAUSES limit
               // Perform the search against the comment index
               if(filter == null) 
               {
                  commentSearcher.search(commentIndexQuery, hitCollector);
               }
               else 
               {
                  commentSearcher.search(commentIndexQuery, filter, hitCollector);
               }
               // Create a query that will be run against the issue index with the issue ids from the search hits
               return new QueryFactoryResult(new ConstantScoreQuery(new IssueIdFilter(ids)));
            }
            catch(final IOException e) 
            {
               log.error("Unable to search the comment index - returning false result.", e);
               return QueryFactoryResult.createFalseResult();
            }
         }
      

      Attachments

        Issue Links

          Activity

            People

              pleschev Peter Leschev
              adc6ee404f6d Jeff Kirby
              Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: