Uploaded image for project: 'Atlassian Intelligence'
  1. Atlassian Intelligence
  2. AI-394

CQL search giving 500 called by CQ on Hello

XMLWordPrintable

    • Severity 3 - Minor

      Issue Summary

      Environment

      Hello, or any instance with number of spaces that user has permission to add CQ to exceeding 1024.

      Steps to Reproduce

      1. Open CQ on Hello
      2. Click Ask a Question

      Expected Results

      No error on the page, User should also be able to see Question Space dropdown pre-populated with a list of spaces.

      Actual Results

      Popup error saying Something went wrong (attached img)

       

      Because CQL search is giving 500:

      https://hello.atlassian.net/wiki/rest/api/search?cql=type%3D%22space%22%20AND%20space%20in%20recentlyViewedSpaces(6)%20AND%20space%20in%20spacesWithCreatePermission(%22ac%3Acom.atlassian.confluence.plugins.confluence-questions%3Aquestion%22)&_r=1557965408709
      {"statusCode":500,"data":
{"authorized":false,"valid":true,"errors":[],"successful":false}
,"message":"com.atlassian.confluence.api.service.exceptions.ServiceException: Exception executing cql as async search: type=\"space\" AND space in recentlyViewedSpaces(6) AND space in spacesWithCreatePermission(\"ac:com.atlassian.confluence.plugins.confluence-questions:question\")"}

       

      Error seen in splunk:

      ...
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
com.atlassian.confluence.impl.search.elasticsearch.rest.ResponseErrorParser.handleSearchPhaseExecutionException(ResponseErrorParser.java:80)
com.atlassian.confluence.impl.search.elasticsearch.rest.ResponseErrorParser.parseErrorResponse(ResponseErrorParser.java:51)
com.atlassian.confluence.impl.search.elasticsearch.rest.ESRestClient.handleErrorResponse(ESRestClient.java:369)
com.atlassian.confluence.impl.search.elasticsearch.rest.ESRestClient.parseSearchResponse(ESRestClient.java:302)
com.atlassian.confluence.impl.search.elasticsearch.rest.ESRestClient.doSearch(ESRestClient.java:259)
com.atlassian.confluence.impl.search.elasticsearch.rest.RestActionFuture.lambda$new$0(RestActionFuture.java:23)
com.google.common.base.Suppliers$MemoizingSupplier.get(Suppliers.java:131)
com.atlassian.confluence.impl.search.elasticsearch.rest.RestActionFuture.actionGet(RestActionFuture.java:28)
com.atlassian.confluence.impl.search.elasticsearch.ESAsyncSearchCommand.run(ESAsyncSearchCommand.java:80)
com.atlassian.confluence.impl.search.elasticsearch.ESAsyncSearchCommand.run(ESAsyncSearchCommand.java:43)
com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:302)
com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:298)
rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)
... 29 more

Caused by: org.elasticsearch.ElasticsearchException: too_many_clauses:too_many_clauses: maxClauseCount is set to 1024
com.atlassian.confluence.impl.search.elasticsearch.rest.ResponseErrorParser.handleOtherException(ResponseErrorParser.java:113)
com.atlassian.confluence.impl.search.elasticsearch.rest.ResponseErrorParser.parseErrorResponse(ResponseErrorParser.java:61)
com.atlassian.confluence.impl.search.elasticsearch.rest.ResponseErrorParser.handleShardSearchFailure(ResponseErrorParser.java:93)
com.atlassian.confluence.impl.search.elasticsearch.rest.ResponseErrorParser.lambda$handleShardSearchFailures$0(ResponseErrorParser.java:88)
java.lang.Iterable.forEach(Iterable.java:75)
com.atlassian.confluence.impl.search.elasticsearch.rest.ResponseErrorParser.handleShardSearchFailures(ResponseErrorParser.java:88)
... 42 more

      Notes

      (Optional - If Necessary)

      Workaround

      Required, if there is no workaround please state:
      Currently there is no known workaround for this behavior. A workaround will be added here when available

        1. Screen Shot 2019-05-15 at 5.09.04 PM.png
          Screen Shot 2019-05-15 at 5.09.04 PM.png
          39 kB

            [AI-394] CQL search giving 500 called by CQ on Hello

            Matt added a comment -

            It's 2024 and this is still not resolved. C'mon Atlassian, QUIT CLOSING THINGS AND JUST FIX SOMETHING!!!!!!!!

            AI-913 Using SpacesWithCreatePermission in CQL can cause 500 Internal Server Error - Create and track feature requests for Atlassian products.

            Matt added a comment - It's 2024 and this is still not resolved. C'mon Atlassian, QUIT CLOSING THINGS AND JUST FIX SOMETHING!!!!!!!! AI-913 Using SpacesWithCreatePermission in CQL can cause 500 Internal Server Error - Create and track feature requests for Atlassian products.

            Alice Wang (Inactive) added a comment - - edited

            A couple of things that should be addressed:

            1. Search should handle the IN clause case where there are more than 1024 items. (addressed by disco)
            2. CQL should provide a limit to functions evals (this is especially important as confluence scales cc yerel@atlassian.com)
              1. Perhaps also provide ability to combine / chain evals - eg recent spaces & permission to create 
            3. Optimizations at the consumer level - in this case CQ can optimise by only checking create space permissions for those recent spaces. (Ecosystem cc lpanda@atlassian.com)

            Alice Wang (Inactive) added a comment - - edited A couple of things that should be addressed: Search should handle the IN clause case where there are more than 1024 items. (addressed by disco) CQL should provide a limit to functions evals (this is especially important as confluence scales cc yerel@atlassian.com ) Perhaps also provide ability to combine / chain evals - eg recent spaces & permission to create  Optimizations at the consumer level - in this case CQ can optimise by only checking create space permissions for those recent spaces. (Ecosystem cc lpanda@atlassian.com )

            Lorien Hou added a comment -

            ES has a limit on max_clause_count with default of 1024, basically the values in IN clause can't exceed that number.

            The SpacesWithCreatePermissionFunction returns all space keys that a user has permission to add a content type in, which can be huge (>1024), causing that error.

            Potentially this can happen to all such cql functions:

             

            public SearchQuery visitEntityExpr(@NotNull AqlParser.EntityExprContext ctx) {
    String fieldName = ctx.entityField().getText();

    @SuppressWarnings("unchecked")
    EqualityFieldHandler<String, V2SearchQueryWrapper> fieldHandler = registry.getEqualityFieldHandler(fieldName);

    // "field IN set"
    if (ctx.setOp() != null) {
        Iterable<String> setOperandValues = cqlIterableStringValueParseTreeVisitor.visitSetOperand(ctx.setOperand());
        if (!setOperandValues.iterator().hasNext()) {
            return MatchNoDocsQuery.getInstance();
        }
        return convertToV2SearchQuery(fieldHandler.build(expressionDataFactory.create(fieldName, ctx.setOp()), Lists.newArrayList(setOperandValues)));
    } else if (ctx.eqOp() != null) { // "field = value" query
        return convertToV2SearchQuery(fieldHandler.build(expressionDataFactory.create(fieldName, ctx.eqOp()), cqlStringValueParseTreeVisitor.visitValue(ctx.value())));
    } else {
        throw new IllegalStateException("Unrecognized operator" + ctx);
    }
}

            we need to partition the "setOperandValues" so that each IN clause doesn't have more than 1024 operands.

             

            My understanding is we could probably split the final query from space IN (a,b,c,d...) into space IN (a,b) OR space IN (c,d) by using something like this: 

            public BooleanQuery(Collection<? extends SearchQuery> must,
                    Collection<? extends SearchQuery> should,
                    Collection<? extends SearchQuery> mustNot)

            where "should" would be the split queries.

             

            Lorien Hou added a comment - ES has a limit on max_clause_count with default of 1024, basically the values in IN clause can't exceed that number. The SpacesWithCreatePermissionFunction returns all space keys that a user has permission to add a content type in, which can be huge (>1024), causing that error. Potentially this can happen to all such cql functions:   public SearchQuery visitEntityExpr(@NotNull AqlParser.EntityExprContext ctx) { String fieldName = ctx.entityField().getText(); @SuppressWarnings( "unchecked" ) EqualityFieldHandler< String , V2SearchQueryWrapper> fieldHandler = registry.getEqualityFieldHandler(fieldName); // "field IN set" if (ctx.setOp() != null ) { Iterable< String > setOperandValues = cqlIterableStringValueParseTreeVisitor.visitSetOperand(ctx.setOperand()); if (!setOperandValues.iterator().hasNext()) { return MatchNoDocsQuery.getInstance(); } return convertToV2SearchQuery(fieldHandler.build(expressionDataFactory.create(fieldName, ctx.setOp()), Lists.newArrayList(setOperandValues))); } else if (ctx.eqOp() != null ) { // "field = value" query return convertToV2SearchQuery(fieldHandler.build(expressionDataFactory.create(fieldName, ctx.eqOp()), cqlStringValueParseTreeVisitor.visitValue(ctx.value()))); } else { throw new IllegalStateException( "Unrecognized operator " + ctx); } } we need to partition the "setOperandValues" so that each IN clause doesn't have more than 1024 operands.   My understanding is we could probably split the final query from space IN (a,b,c,d...) into space IN (a,b) OR space IN (c,d) by using something like this:  public BooleanQuery(Collection<? extends SearchQuery> must, Collection<? extends SearchQuery> should, Collection<? extends SearchQuery> mustNot) where "should" would be the split queries.  

              Unassigned Unassigned
              lhou1 Lorien Hou
              Affected customers:
              0 This affects my team
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: