[JRASERVER-31971] Issue collector causes performance issues on host JIRA instance - Create and track feature requests for Atlassian products.

Type: Bug
Resolution: Fixed
Priority: Low (View bug fix roadmap)
Fix Version/s: 6.0.3
Affects Version/s: None
Component/s: None
Labels:
- expedite

Bug Fix Policy:
View Atlassian Server bug fix policy

NOTE: This bug report is for JIRA Server. Using JIRA Cloud? See the corresponding bug report.

TL;DR: JIC needs a "scalable" mode where collectors can be deployed into tens of thousands of websites without crashing the JIRA instance hosting JIC. And neither should JIC represent a single point of failure for those other websites.

When JIC has a "scalable" mode, we have to update the jira-feedback-plugin in JIRA to use that mode.

I'm currently investigating performance issues on JAC and the issue collector is causing quite a bit of overhead. (https://extranet.atlassian.com/jira/browse/ADM-33614)

In particular the request for /rest/collectors/1.0/configuration/trigger/d3de7fb5 (which any client with an issue collector will send) causes quite a bit of load on the server hosting issue collectors.

When this resource is hit it causes DB lookups:

"http-127.0.0.1-9080-119" daemon prio=10 tid=0x0b4db000 nid=0x1d01 runnable [0x55699000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:135)
	at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:104)
	at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
	at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:259)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1620)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
	- locked <0x78e4b558> (a org.postgresql.core.v3.QueryExecutorImpl)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:367)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:271)
	at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
	at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
	at org.ofbiz.core.entity.jdbc.SQLProcessor.executeQuery(SQLProcessor.java:597)
	at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(GenericDAO.java:1061)
	at org.ofbiz.core.entity.GenericDAO.selectByAnd(GenericDAO.java:608)
	at org.ofbiz.core.entity.GenericHelperDAO.findByAnd(GenericHelperDAO.java:131)
	at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:788)
	at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:773)
	at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:750)
	at com.opensymphony.module.propertyset.ofbiz.OFBizPropertySet.findPropertyEntry(OFBizPropertySet.java:298)
	at com.opensymphony.module.propertyset.ofbiz.OFBizPropertySet.getType(OFBizPropertySet.java:159)
	at com.atlassian.jira.propertyset.JiraCachingPropertySet.getType(JiraCachingPropertySet.java:661)
	at sun.reflect.GeneratedMethodAccessor277.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.atlassian.sal.jira.pluginsettings.LazyProjectMigratingPropertySet$PropertySetInvocationHandler.invoke(LazyProjectMigratingPropertySet.java:57)
	at $Proxy1841.getType(Unknown Source)
	at com.atlassian.sal.jira.pluginsettings.JiraPluginSettings.getActual(JiraPluginSettings.java:47)
	at com.atlassian.sal.core.pluginsettings.AbstractStringPluginSettings.get(AbstractStringPluginSettings.java:126)
	at com.atlassian.jira.collector.plugin.components.CollectorStoreImpl.getCollector(CollectorStoreImpl.java:103)
	at com.atlassian.jira.collector.plugin.components.CollectorServiceImpl.getCollector(CollectorServiceImpl.java:51)
	at com.atlassian.jira.collector.plugin.rest.ConfigurationResource.findCollectorById(ConfigurationResource.java:55)
	at com.atlassian.jira.collector.plugin.rest.ConfigurationResource.getTriggerConfiguration(ConfigurationResource.java:41)
	at sun.reflect.GeneratedMethodAccessor315.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$ResponseOutInvoker$1.invoke(DispatchProviderHelper.java:234)
	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$1.intercept(DispatchProviderHelper.java:100)
	at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61)
	at com.atlassian.plugins.rest.common.expand.interceptor.ExpandInterceptor.intercept(ExpandInterceptor.java:38)
	at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61)
	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper.invokeMethodWithInterceptors(DispatchProviderHelper.java:132)
	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$ResponseOutInvoker._dispatch(DispatchProviderHelper.java:230)
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImp

Also the response is returned with no-cache headers:

Cache-Control:no-cache, no-store, no-transform

This means that if an issue collector is enabled in OnDemand for example, every single OnDemand user will trigger a request to JAC for every single page-pop which will potentially hit the DB if the propertysets aren't cached in memory.

I think we should consider:

Moving issue collectors to ActiveObjects which may be more efficient than the currently ugly pluginsettings storage (where everything is split across multiple DB rows)
Adding temporary cache headers to this response so not every single end-user page-pop hits JAC. I think we don't have cache headers to allow editing of collectors, but surely several hours to see edits on the client is acceptable for a feature like this.

Effectively the issue collector in OnDemand is launching a DDOS attack on JAC currently.

is duplicated by

JRASERVER-34087 The Issue Collector should cache its state and not reload it from the database on every request

Closed

relates to

JRACLOUD-31971 Issue collector causes performance issues on host JIRA instance

Closed

JRADEV-18956 Failed to load

JRADEV-19583 Failed to load

is related to: JRADEV-21993 Failed to load

mentioned in: Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Wiki Page Loading...; Wiki Page Loading...; Wiki Page Loading...; Wiki Page Loading...; Wiki Page Loading...

(10 mentioned in)

Form Name

Loading...

Type: Bug
Resolution: Fixed
Priority: Low
Fix Version/s: 6.0.3
Affects Version/s: None
Component/s: None
Labels:
- expedite

Bug Fix Policy:
View Atlassian Server bug fix policy

NOTE: This bug report is for JIRA Server. Using JIRA Cloud? See the corresponding bug report.

TL;DR: JIC needs a "scalable" mode where collectors can be deployed into tens of thousands of websites without crashing the JIRA instance hosting JIC. And neither should JIC represent a single point of failure for those other websites.

When JIC has a "scalable" mode, we have to update the jira-feedback-plugin in JIRA to use that mode.

I'm currently investigating performance issues on JAC and the issue collector is causing quite a bit of overhead. (https://extranet.atlassian.com/jira/browse/ADM-33614)

In particular the request for /rest/collectors/1.0/configuration/trigger/d3de7fb5 (which any client with an issue collector will send) causes quite a bit of load on the server hosting issue collectors.

When this resource is hit it causes DB lookups:

"http-127.0.0.1-9080-119" daemon prio=10 tid=0x0b4db000 nid=0x1d01 runnable [0x55699000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:135)
	at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:104)
	at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
	at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:259)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1620)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
	- locked <0x78e4b558> (a org.postgresql.core.v3.QueryExecutorImpl)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:367)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:271)
	at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
	at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
	at org.ofbiz.core.entity.jdbc.SQLProcessor.executeQuery(SQLProcessor.java:597)
	at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(GenericDAO.java:1061)
	at org.ofbiz.core.entity.GenericDAO.selectByAnd(GenericDAO.java:608)
	at org.ofbiz.core.entity.GenericHelperDAO.findByAnd(GenericHelperDAO.java:131)
	at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:788)
	at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:773)
	at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:750)
	at com.opensymphony.module.propertyset.ofbiz.OFBizPropertySet.findPropertyEntry(OFBizPropertySet.java:298)
	at com.opensymphony.module.propertyset.ofbiz.OFBizPropertySet.getType(OFBizPropertySet.java:159)
	at com.atlassian.jira.propertyset.JiraCachingPropertySet.getType(JiraCachingPropertySet.java:661)
	at sun.reflect.GeneratedMethodAccessor277.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.atlassian.sal.jira.pluginsettings.LazyProjectMigratingPropertySet$PropertySetInvocationHandler.invoke(LazyProjectMigratingPropertySet.java:57)
	at $Proxy1841.getType(Unknown Source)
	at com.atlassian.sal.jira.pluginsettings.JiraPluginSettings.getActual(JiraPluginSettings.java:47)
	at com.atlassian.sal.core.pluginsettings.AbstractStringPluginSettings.get(AbstractStringPluginSettings.java:126)
	at com.atlassian.jira.collector.plugin.components.CollectorStoreImpl.getCollector(CollectorStoreImpl.java:103)
	at com.atlassian.jira.collector.plugin.components.CollectorServiceImpl.getCollector(CollectorServiceImpl.java:51)
	at com.atlassian.jira.collector.plugin.rest.ConfigurationResource.findCollectorById(ConfigurationResource.java:55)
	at com.atlassian.jira.collector.plugin.rest.ConfigurationResource.getTriggerConfiguration(ConfigurationResource.java:41)
	at sun.reflect.GeneratedMethodAccessor315.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$ResponseOutInvoker$1.invoke(DispatchProviderHelper.java:234)
	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$1.intercept(DispatchProviderHelper.java:100)
	at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61)
	at com.atlassian.plugins.rest.common.expand.interceptor.ExpandInterceptor.intercept(ExpandInterceptor.java:38)
	at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61)
	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper.invokeMethodWithInterceptors(DispatchProviderHelper.java:132)
	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$ResponseOutInvoker._dispatch(DispatchProviderHelper.java:230)
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImp

Also the response is returned with no-cache headers:

Cache-Control:no-cache, no-store, no-transform

This means that if an issue collector is enabled in OnDemand for example, every single OnDemand user will trigger a request to JAC for every single page-pop which will potentially hit the DB if the propertysets aren't cached in memory.

I think we should consider:

Moving issue collectors to ActiveObjects which may be more efficient than the currently ugly pluginsettings storage (where everything is split across multiple DB rows)
Adding temporary cache headers to this response so not every single end-user page-pop hits JAC. I think we don't have cache headers to allow editing of collectors, but surely several hours to see edits on the client is acceptable for a feature like this.

Effectively the issue collector in OnDemand is launching a DDOS attack on JAC currently.

is duplicated by

JRASERVER-34087 The Issue Collector should cache its state and not reload it from the database on every request

Closed

relates to

JRACLOUD-31971 Issue collector causes performance issues on host JIRA instance

Closed

JRADEV-18956 Loading...

JRADEV-19583 Loading...

is related to: JRADEV-21993 Loading...

mentioned in: Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Wiki Page Loading...; Wiki Page Loading...; Wiki Page Loading...; Wiki Page Loading...; Wiki Page Loading...

(10 mentioned in)

Assignee:: Oswaldo Hernandez (Inactive)

Reporter:: Andreas Knecht (Inactive)

Votes:: 1 Vote for this issue

Watchers:: 15 Start watching this issue

Created:: 05/Mar/2013 12:22 PM

Updated:: 03/Feb/2021 11:56 PM

Resolved:: 25/Jun/2013 1:37 AM

Details

Description

Attachments

Issue Links

Forms

Activity