-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Low
-
Affects Version/s: 8.20.0, 9.11.0, 9.12.0, 9.13.0, 9.14.0
-
Component/s: JQL Search
-
8.2
-
9
-
Severity 1 - Critical
-
5
Issue Summary
Jira doesn't seem to have a control on how many clauses a user can add to a JQL search within the advanced search.
Using thousands of text ~ <matching_string> clauses within the JQL search may lead to an out of memory error.
Steps to Reproduce
- Install a vanilla Jira Software Data Center instances.
- This was tested on versions 9.13 and 9.14.
- Create some SCRUM projects with sample data.
- The following shell script will create 300 projects with sample data. After running it the instance might have 300 projects and ~6,800 issues.
JIRA_BASE_URL=Jira-Base-URL JIRA_ADMIN_USERNAME=admin-username JIRA_ADMIN_PASSWORD=admin-password JIRA_PROJECT_NAME=SCRUM JIRA_PROJECT_KEY=SCRUM for i in $(seq 1 300); do curl -v -u ${JIRA_ADMIN_USERNAME}:${JIRA_ADMIN_PASSWORD} -X POST -o /dev/null \ -H 'X-Atlassian-Token: no-check' \ ${JIRA_BASE_URL}'/rest/jira-importers-plugin/1.0/demo/create' \ --data-raw 'name='${JIRA_PROJECT_NAME}${i}'&key='${JIRA_PROJECT_KEY}${i}'&keyEdited=false&projectTemplateWebItemKey=software-demo-project-scrum&projectTemplateModuleKey=undefined' done
- The following shell script will create 300 projects with sample data. After running it the instance might have 300 projects and ~6,800 issues.
- Go to the advanced search and run a JQL with thousands of text ~ scrum OR clauses.
- Check the attached file for an example of such a JQL query – killer-jql-query.txt
.
- Check the attached file for an example of such a JQL query – killer-jql-query.txt
Expected Results
Jira identifies this is a JQL with too many clauses and blocks running it on the backend, providing a meaningful message to the user on the UI.
The performance of the backend isn't affected.
Actual Results
Jira accepts the JQL query and runs it on the backend.
The query takes a long time to complete and eventually fails on the frontend.
Inspecting the browser developer tools, there are two requests when a search is made:
- POST /secure/QueryComponent!Jql.jspa, which fails with an HTTP 400 response and a jqlTooComplex message.
- POST /rest/issueNav/1/issueTable, which may timeout on the load balancer or fail with an HTTP 500 response.
Eventually, the application may be unresponsive to users because of high GC activity, leading to an out of memory (OOM) error.
Collecting thread dumps while the request is on-going, there may be long running thread with a stack similar to the below.
"http-nio-45130-exec-25 url: /rest/issueNav/1/issueTable; user: admin" #45 daemon prio=5 os_prio=31 cpu=29470.34ms elapsed=5146.91s allocated=17005M defined_classes=131 tid=0x000000013d043800 nid=0xe103 runnable [0x00000002ab527000] java.lang.Thread.State: RUNNABLE at java.util.Collections$UnmodifiableCollection$1.next(java.base@11.0.21/Collections.java:1047) at org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:328) at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:714) at org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:329) at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:714) at org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:329) at org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:714) at org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:329) at org.apache.lucene.search.BooleanWeight.scorer(BooleanWeight.java:295) at org.apache.lucene.search.Weight.bulkScorer(Weight.java:147) at org.apache.lucene.search.BooleanWeight.bulkScorer(BooleanWeight.java:289) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:657) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:462) at com.atlassian.jira.index.stats.IndexSearcherWithStats.search(IndexSearcherWithStats.java:88) at com.atlassian.jira.index.DelegateSearcher.search(DelegateSearcher.java:169) at com.atlassian.jira.index.DelegateSearcher.search(DelegateSearcher.java:169) at com.atlassian.jira.index.UnmanagedIndexSearcher.search(UnmanagedIndexSearcher.java:55) at com.atlassian.jira.index.DelegateSearcher.search(DelegateSearcher.java:169) at com.atlassian.jira.index.ManagedIndexSearcher.search(ManagedIndexSearcher.java:15) at org.apache.lucene.search.join.JoinUtil.createJoinQuery(JoinUtil.java:376) at org.apache.lucene.search.join.JoinUtil.createJoinQuery(JoinUtil.java:107) at com.atlassian.jira.jql.query.IssueIdJoinQueryFactory.createIssueIdJoinQuery(IssueIdJoinQueryFactory.java:39) at com.atlassian.jira.jql.query.CommentClauseQueryFactory.getQuery(CommentClauseQueryFactory.java:50) at com.atlassian.jira.jql.query.AllTextClauseQueryFactory.getQuery(AllTextClauseQueryFactory.java:51) at com.atlassian.jira.jql.query.ContextAwareQueryVisitor.visit(ContextAwareQueryVisitor.java:98) at com.atlassian.jira.jql.query.ContextAwareQueryVisitor.visit(ContextAwareQueryVisitor.java:27) at com.atlassian.query.clause.TerminalClauseImpl.accept(TerminalClauseImpl.java:143) at com.atlassian.jira.jql.query.ContextAwareQueryVisitor.visit(ContextAwareQueryVisitor.java:73) at com.atlassian.jira.jql.query.ContextAwareQueryVisitor.visit(ContextAwareQueryVisitor.java:27) at com.atlassian.query.clause.OrClause.accept(OrClause.java:28) at com.atlassian.jira.jql.query.QueryVisitor.createQuery(QueryVisitor.java:51) at com.atlassian.jira.jql.query.DefaultLuceneQueryBuilder.createLuceneQuery(DefaultLuceneQueryBuilder.java:29) at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.createLuceneQuery(LuceneSearchProvider.java:343) at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.getHits(LuceneSearchProvider.java:229) at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:375) at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:138) at jdk.internal.reflect.GeneratedMethodAccessor1423.invoke(Unknown Source) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.21/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(java.base@11.0.21/Method.java:566) at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26) at com.sun.proxy.$Proxy710.search(Unknown Source) at jdk.internal.reflect.GeneratedMethodAccessor1423.invoke(Unknown Source) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.21/DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(java.base@11.0.21/Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56) at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70) at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241) at com.sun.proxy.$Proxy3423.search(Unknown Source) at com.atlassian.jira.plugin.issuenav.service.issuetable.AbstractIssueTableCreator.collectIssues(AbstractIssueTableCreator.java:164) at com.atlassian.jira.plugin.issuenav.service.issuetable.AbstractIssueTableCreator.executeNormalSearch(AbstractIssueTableCreator.java:235)
Inspecting a heap dump generated automatically with the OOM error we may see the following characteristics:
- One or more Lucene objects taking most of the heap.
- One or more user threads making requests to /rest/issueNav/1/issueTable taking most of the heap.
- Inspecting the user thread we may find the exact JQL query executed by the user.
Here is an example of an OOM error in production:
- Two user threads consuming more than 80% of the 50GB heap.
- The 2 top consumer threads were running request to /rest/issueNav/1/issueTable.
- Drilling down the thread, the majority of the heap is taken by an array of org.apache.lucene.search.Boolean2ScorerSupplier objects.
Workaround
Currently there is no known workaround for this behavior. A workaround will be added here when available.
