Description
NOTE: This bug report is for JIRA Server. Using JIRA Cloud? See the corresponding bug report.
TL;DR: JIC needs a "scalable" mode where collectors can be deployed into tens of thousands of websites without crashing the JIRA instance hosting JIC. And neither should JIC represent a single point of failure for those other websites.
When JIC has a "scalable" mode, we have to update the jira-feedback-plugin in JIRA to use that mode.
I'm currently investigating performance issues on JAC and the issue collector is causing quite a bit of overhead. (https://extranet.atlassian.com/jira/browse/ADM-33614)
In particular the request for /rest/collectors/1.0/configuration/trigger/d3de7fb5 (which any client with an issue collector will send) causes quite a bit of load on the server hosting issue collectors.
When this resource is hit it causes DB lookups:
"http-127.0.0.1-9080-119" daemon prio=10 tid=0x0b4db000 nid=0x1d01 runnable [0x55699000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:135) at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:104) at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73) at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:259) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1620) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257) - locked <0x78e4b558> (a org.postgresql.core.v3.QueryExecutorImpl) at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:367) at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:271) at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96) at org.ofbiz.core.entity.jdbc.SQLProcessor.executeQuery(SQLProcessor.java:597) at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(GenericDAO.java:1061) at org.ofbiz.core.entity.GenericDAO.selectByAnd(GenericDAO.java:608) at org.ofbiz.core.entity.GenericHelperDAO.findByAnd(GenericHelperDAO.java:131) at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:788) at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:773) at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:750) at com.opensymphony.module.propertyset.ofbiz.OFBizPropertySet.findPropertyEntry(OFBizPropertySet.java:298) at com.opensymphony.module.propertyset.ofbiz.OFBizPropertySet.getType(OFBizPropertySet.java:159) at com.atlassian.jira.propertyset.JiraCachingPropertySet.getType(JiraCachingPropertySet.java:661) at sun.reflect.GeneratedMethodAccessor277.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.atlassian.sal.jira.pluginsettings.LazyProjectMigratingPropertySet$PropertySetInvocationHandler.invoke(LazyProjectMigratingPropertySet.java:57) at $Proxy1841.getType(Unknown Source) at com.atlassian.sal.jira.pluginsettings.JiraPluginSettings.getActual(JiraPluginSettings.java:47) at com.atlassian.sal.core.pluginsettings.AbstractStringPluginSettings.get(AbstractStringPluginSettings.java:126) at com.atlassian.jira.collector.plugin.components.CollectorStoreImpl.getCollector(CollectorStoreImpl.java:103) at com.atlassian.jira.collector.plugin.components.CollectorServiceImpl.getCollector(CollectorServiceImpl.java:51) at com.atlassian.jira.collector.plugin.rest.ConfigurationResource.findCollectorById(ConfigurationResource.java:55) at com.atlassian.jira.collector.plugin.rest.ConfigurationResource.getTriggerConfiguration(ConfigurationResource.java:41) at sun.reflect.GeneratedMethodAccessor315.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$ResponseOutInvoker$1.invoke(DispatchProviderHelper.java:234) at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$1.intercept(DispatchProviderHelper.java:100) at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61) at com.atlassian.plugins.rest.common.expand.interceptor.ExpandInterceptor.intercept(ExpandInterceptor.java:38) at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61) at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper.invokeMethodWithInterceptors(DispatchProviderHelper.java:132) at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$ResponseOutInvoker._dispatch(DispatchProviderHelper.java:230) at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImp
Also the response is returned with no-cache headers:
Cache-Control:no-cache, no-store, no-transform
This means that if an issue collector is enabled in OnDemand for example, every single OnDemand user will trigger a request to JAC for every single page-pop which will potentially hit the DB if the propertysets aren't cached in memory.
I think we should consider:
- Moving issue collectors to ActiveObjects which may be more efficient than the currently ugly pluginsettings storage (where everything is split across multiple DB rows)
- Adding temporary cache headers to this response so not every single end-user page-pop hits JAC. I think we don't have cache headers to allow editing of collectors, but surely several hours to see edits on the client is acceptable for a feature like this.
Effectively the issue collector in OnDemand is launching a DDOS attack on JAC currently.
Attachments
Issue Links
- is duplicated by
-
JRASERVER-34087 The Issue Collector should cache its state and not reload it from the database on every request
-
- Closed
-
- relates to
-
-
- Closed
-
-
JRADEV-18956 Loading...
-
- is related to
-
- mentioned in
-
-
-
-
-
-
-
-
-
-