Details
-
Bug
-
Resolution: Fixed
-
High
-
5.0, 4.4.5, 5.1
-
None
-
4.04
-
Description
We had a SAC outage last night and this was due to the HistoryFieldValueValidator trying to find all possible values for the status field on SAC.
This means it runs a query through the ChangeItem table for the field 'status'. In the code we use a DefaultOfBizListIterator (great) but then call getCompleteList() on it which loads everything from the result set into memory. This caused an OOME since on SAC there's ~2.2 million rows for the status field in the change item table (with 4 GB of heap).
Here's the stacktrace from the heapdump:
http-172.16.3.44-8080-134 at java.nio.CharBuffer.wrap([CII)Ljava/nio/CharBuffer; (CharBuffer.java:350) at java.nio.CharBuffer.wrap([C)Ljava/nio/CharBuffer; (CharBuffer.java:373) at java.lang.StringCoding$StringDecoder.decode([BII)[C (StringCoding.java:138) at java.lang.StringCoding.decode(Ljava/lang/String;[BII)[C (StringCoding.java:173) at java.lang.String.<init>([BIILjava/lang/String;)V (String.java:443) at org.postgresql.core.Encoding.decode([BII)Ljava/lang/String; (Encoding.java:193) at org.postgresql.core.Encoding.decode([B)Ljava/lang/String; (Encoding.java:205) at org.postgresql.jdbc2.AbstractJdbc2ResultSet.getString(I)Ljava/lang/String; (AbstractJdbc2ResultSet.java:1879) at org.apache.tomcat.dbcp.dbcp.DelegatingResultSet.getString(I)Ljava/lang/String; (DelegatingResultSet.java:175) at org.ofbiz.core.entity.jdbc.SqlJdbcUtil.getValue(Ljava/sql/ResultSet;ILorg/ofbiz/core/entity/model/ModelField;Lorg/ofbiz/core/entity/GenericEntity;Lorg/ofbiz/core/entity/model/ModelFieldTypeReader;)V (SqlJdbcUtil.java:529) at org.ofbiz.core.entity.EntityListIterator.currentGenericValue()Lorg/ofbiz/core/entity/GenericValue; (EntityListIterator.java:169) at org.ofbiz.core.entity.EntityListIterator.next()Lorg/ofbiz/core/entity/GenericValue; (EntityListIterator.java:246) at org.ofbiz.core.entity.EntityListIterator.getCompleteList()Ljava/util/List; (EntityListIterator.java:308) at com.atlassian.jira.ofbiz.DefaultOfBizListIterator.getCompleteList()Ljava/util/List; (DefaultOfBizListIterator.java:170) at com.atlassian.jira.issue.changehistory.DefaultChangeHistoryManager.getAllChangeItems(Ljava/lang/String;)Ljava/util/List; (DefaultChangeHistoryManager.java:285) at com.atlassian.jira.issue.changehistory.DefaultChangeHistoryManager.findAllPossibleValues(Ljava/lang/String;)Ljava/util/Map; (DefaultChangeHistoryManager.java:271) at com.atlassian.jira.jql.validator.HistoryFieldValueValidator.stringValueExists(Lcom/opensymphony/user/User;Ljava/lang/String;Ljava/lang/String;)Z (HistoryFieldValueValidator.java:69) at com.atlassian.jira.jql.validator.HistoryFieldValueValidator.validateValues(Lcom/opensymphony/user/User;Ljava/lang/String;Ljava/util/List;)Lcom/atlassian/jira/util/MessageSet; (HistoryFieldValueValidator.java:85) at com.atlassian.jira.jql.validator.HistoryPredicateValidator.validateTerminalPredicate(Lcom/atlassian/crowd/embedded/api/User;Lcom/atlassian/jira/util/I18nHelper;Lcom/atlassian/jira/util/MessageSet;Lcom/atlassian/query/history/TerminalHistoryPredicate;Ljava/lang/String;)V (HistoryPredicateValidator.java:95) at com.atlassian.jira.jql.validator.HistoryPredicateValidator.validate(Lcom/atlassian/query/history/HistoryPredicate;Ljava/lang/String;Lcom/atlassian/crowd/embedded/api/User;)Lcom/atlassian/jira/util/MessageSet; (HistoryPredicateValidator.java:69) at com.atlassian.jira.jql.validator.HistoryPredicateValidator.validate(Lcom/atlassian/crowd/embedded/api/User;Lcom/atlassian/query/clause/ChangedClause;Lcom/atlassian/query/history/HistoryPredicate;)Lcom/atlassian/jira/util/MessageSet; (HistoryPredicateValidator.java:58) at com.atlassian.jira.jql.validator.ChangedClauseValidator.validate(Lcom/atlassian/crowd/embedded/api/User;Lcom/atlassian/query/clause/ChangedClause;)Lcom/atlassian/jira/util/MessageSet; (ChangedClauseValidator.java:72) at com.atlassian.jira.jql.validator.ValidatorVisitor.visit(Lcom/atlassian/query/clause/ChangedClause;)Lcom/atlassian/jira/util/MessageSet; (ValidatorVisitor.java:118) at com.atlassian.jira.jql.validator.ValidatorVisitor.visit(Lcom/atlassian/query/clause/ChangedClause;)Ljava/lang/Object; (ValidatorVisitor.java:32) at com.atlassian.query.clause.ChangedClauseImpl.accept(Lcom/atlassian/query/clause/ClauseVisitor;)Ljava/lang/Object; (ChangedClauseImpl.java:73) at com.atlassian.jira.bc.issue.search.DefaultSearchService.validateQuery(Lcom/atlassian/crowd/embedded/api/User;Lcom/atlassian/query/Query;)Lcom/atlassian/jira/util/MessageSet; (DefaultSearchService.java:305) at com.atlassian.jira.bc.issue.search.DefaultSearchService.validateQuery(Lcom/opensymphony/user/User;Lcom/atlassian/query/Query;)Lcom/atlassian/jira/util/MessageSet; (DefaultSearchService.java:216) at com.atlassian.jira.web.action.issue.IssueNavigator.validateQuery(Lcom/atlassian/query/Query;)Z (IssueNavigator.java:848) at com.atlassian.jira.web.action.issue.IssueNavigator.updateSearchRequestWithJqlOrParams(Lcom/atlassian/jira/issue/search/SearchRequest;)V (IssueNavigator.java:691) at com.atlassian.jira.web.action.issue.IssueNavigator._doExecuteAdvanced()Ljava/lang/String; (IssueNavigator.java:594) at com.atlassian.jira.web.action.issue.IssueNavigator.doExecuteAdvanced()Ljava/lang/String; (IssueNavigator.java:544) at sun.reflect.GeneratedMethodAccessor3951.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (Method.java:597) at webwork.util.InjectionUtils$DefaultInjectionImpl.invoke(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (InjectionUtils.java:70) at webwork.util.InjectionUtils.invoke(Ljava/lang/reflect/Method;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (InjectionUtils.java:56) at webwork.action.ActionSupport.invokeCommand()Ljava/lang/String; (ActionSupport.java:433) at webwork.action.ActionSupport.execute()Ljava/lang/String; (ActionSupport.java:157) at com.atlassian.jira.action.JiraActionSupport.execute()Ljava/lang/String; (JiraActionSupport.java:76) at webwork.interceptor.DefaultInterceptorChain.proceed()Ljava/lang/String; (DefaultInterceptorChain.java:39) at webwork.interceptor.NestedInterceptorChain.proceed()Ljava/lang/String; (NestedInterceptorChain.java:31) at webwork.interceptor.ChainedInterceptor.intercept(Lwebwork/interceptor/InterceptorChain;)Ljava/lang/String; (ChainedInterceptor.java:16) at webwork.interceptor.DefaultInterceptorChain.proceed()Ljava/lang/String; (DefaultInterceptorChain.java:35) at webwork.dispatcher.GenericDispatcher.executeAction(Lwebwork/action/Action;)Ljava/lang/String; (GenericDispatcher.java:205) at webwork.dispatcher.GenericDispatcher.executeAction()V (GenericDispatcher.java:143) at com.atlassian.jira.web.dispatcher.JiraWebworkActionDispatcher.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (JiraWebworkActionDispatcher.java:152) at javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (ApplicationFilterChain.java:206) at com.atlassian.jira.web.filters.steps.ChainedFilterStepRunner.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V (ChainedFilterStepRunner.java:74) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (ApplicationFilterChain.java:235)
Attached a screenshot of the heap.
Raising this as critical since this has the potential for huge support cost (heap dumps are hard to diagnose) and it's really quite easy to cause (the changeitem table is huge and only keeps growing over time). We really need to avoid loading unbounded tables fromt the DB into memory at all cost. In my mind that getCompleteList() method should be taken out the back and shot.
