We couldn't load the project sidebar. Refresh the page to try again.
If the problem persists, contact your Jira admin.
IMPORTANT: JAC is a Public system and anyone on the internet will be able to view the data in the created JAC tickets. Please don’t include Customer or Sensitive data in the JAC ticket.
Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-31971

Issue collector causes performance issues on host JIRA instance

      NOTE: This bug report is for JIRA Server. Using JIRA Cloud? See the corresponding bug report.

      TL;DR: JIC needs a "scalable" mode where collectors can be deployed into tens of thousands of websites without crashing the JIRA instance hosting JIC. And neither should JIC represent a single point of failure for those other websites.

      When JIC has a "scalable" mode, we have to update the jira-feedback-plugin in JIRA to use that mode.


      I'm currently investigating performance issues on JAC and the issue collector is causing quite a bit of overhead. (https://extranet.atlassian.com/jira/browse/ADM-33614)

      In particular the request for /rest/collectors/1.0/configuration/trigger/d3de7fb5 (which any client with an issue collector will send) causes quite a bit of load on the server hosting issue collectors.

      When this resource is hit it causes DB lookups:

      "http-127.0.0.1-9080-119" daemon prio=10 tid=0x0b4db000 nid=0x1d01 runnable [0x55699000]
         java.lang.Thread.State: RUNNABLE
      	at java.net.SocketInputStream.socketRead0(Native Method)
      	at java.net.SocketInputStream.read(SocketInputStream.java:129)
      	at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:135)
      	at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:104)
      	at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
      	at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:259)
      	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1620)
      	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
      	- locked <0x78e4b558> (a org.postgresql.core.v3.QueryExecutorImpl)
      	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
      	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:367)
      	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:271)
      	at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
      	at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
      	at org.ofbiz.core.entity.jdbc.SQLProcessor.executeQuery(SQLProcessor.java:597)
      	at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(GenericDAO.java:1061)
      	at org.ofbiz.core.entity.GenericDAO.selectByAnd(GenericDAO.java:608)
      	at org.ofbiz.core.entity.GenericHelperDAO.findByAnd(GenericHelperDAO.java:131)
      	at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:788)
      	at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:773)
      	at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:750)
      	at com.opensymphony.module.propertyset.ofbiz.OFBizPropertySet.findPropertyEntry(OFBizPropertySet.java:298)
      	at com.opensymphony.module.propertyset.ofbiz.OFBizPropertySet.getType(OFBizPropertySet.java:159)
      	at com.atlassian.jira.propertyset.JiraCachingPropertySet.getType(JiraCachingPropertySet.java:661)
      	at sun.reflect.GeneratedMethodAccessor277.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at com.atlassian.sal.jira.pluginsettings.LazyProjectMigratingPropertySet$PropertySetInvocationHandler.invoke(LazyProjectMigratingPropertySet.java:57)
      	at $Proxy1841.getType(Unknown Source)
      	at com.atlassian.sal.jira.pluginsettings.JiraPluginSettings.getActual(JiraPluginSettings.java:47)
      	at com.atlassian.sal.core.pluginsettings.AbstractStringPluginSettings.get(AbstractStringPluginSettings.java:126)
      	at com.atlassian.jira.collector.plugin.components.CollectorStoreImpl.getCollector(CollectorStoreImpl.java:103)
      	at com.atlassian.jira.collector.plugin.components.CollectorServiceImpl.getCollector(CollectorServiceImpl.java:51)
      	at com.atlassian.jira.collector.plugin.rest.ConfigurationResource.findCollectorById(ConfigurationResource.java:55)
      	at com.atlassian.jira.collector.plugin.rest.ConfigurationResource.getTriggerConfiguration(ConfigurationResource.java:41)
      	at sun.reflect.GeneratedMethodAccessor315.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$ResponseOutInvoker$1.invoke(DispatchProviderHelper.java:234)
      	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$1.intercept(DispatchProviderHelper.java:100)
      	at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61)
      	at com.atlassian.plugins.rest.common.expand.interceptor.ExpandInterceptor.intercept(ExpandInterceptor.java:38)
      	at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61)
      	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper.invokeMethodWithInterceptors(DispatchProviderHelper.java:132)
      	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$ResponseOutInvoker._dispatch(DispatchProviderHelper.java:230)
      	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
      	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
      	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
      	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
      	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
      	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImp
      

      Also the response is returned with no-cache headers:

      Cache-Control:no-cache, no-store, no-transform

      This means that if an issue collector is enabled in OnDemand for example, every single OnDemand user will trigger a request to JAC for every single page-pop which will potentially hit the DB if the propertysets aren't cached in memory.

      I think we should consider:

      • Moving issue collectors to ActiveObjects which may be more efficient than the currently ugly pluginsettings storage (where everything is split across multiple DB rows)
      • Adding temporary cache headers to this response so not every single end-user page-pop hits JAC. I think we don't have cache headers to allow editing of collectors, but surely several hours to see edits on the client is acceptable for a feature like this.

      Effectively the issue collector in OnDemand is launching a DDOS attack on JAC currently.

          Form Name

            Loading...
            IMPORTANT: JAC is a Public system and anyone on the internet will be able to view the data in the created JAC tickets. Please don’t include Customer or Sensitive data in the JAC ticket.
            Uploaded image for project: 'Jira Data Center'
            1. Jira Data Center
            2. JRASERVER-31971

            Issue collector causes performance issues on host JIRA instance

              • Icon: Bug Bug
              • Resolution: Fixed
              • Icon: Low Low
              • 6.0.3
              • None
              • None

                NOTE: This bug report is for JIRA Server. Using JIRA Cloud? See the corresponding bug report.

                TL;DR: JIC needs a "scalable" mode where collectors can be deployed into tens of thousands of websites without crashing the JIRA instance hosting JIC. And neither should JIC represent a single point of failure for those other websites.

                When JIC has a "scalable" mode, we have to update the jira-feedback-plugin in JIRA to use that mode.


                I'm currently investigating performance issues on JAC and the issue collector is causing quite a bit of overhead. (https://extranet.atlassian.com/jira/browse/ADM-33614)

                In particular the request for /rest/collectors/1.0/configuration/trigger/d3de7fb5 (which any client with an issue collector will send) causes quite a bit of load on the server hosting issue collectors.

                When this resource is hit it causes DB lookups:

                "http-127.0.0.1-9080-119" daemon prio=10 tid=0x0b4db000 nid=0x1d01 runnable [0x55699000]
                   java.lang.Thread.State: RUNNABLE
                	at java.net.SocketInputStream.socketRead0(Native Method)
                	at java.net.SocketInputStream.read(SocketInputStream.java:129)
                	at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:135)
                	at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:104)
                	at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:73)
                	at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:259)
                	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1620)
                	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
                	- locked <0x78e4b558> (a org.postgresql.core.v3.QueryExecutorImpl)
                	at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:479)
                	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:367)
                	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:271)
                	at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
                	at org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
                	at org.ofbiz.core.entity.jdbc.SQLProcessor.executeQuery(SQLProcessor.java:597)
                	at org.ofbiz.core.entity.GenericDAO.selectListIteratorByCondition(GenericDAO.java:1061)
                	at org.ofbiz.core.entity.GenericDAO.selectByAnd(GenericDAO.java:608)
                	at org.ofbiz.core.entity.GenericHelperDAO.findByAnd(GenericHelperDAO.java:131)
                	at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:788)
                	at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:773)
                	at org.ofbiz.core.entity.GenericDelegator.findByAnd(GenericDelegator.java:750)
                	at com.opensymphony.module.propertyset.ofbiz.OFBizPropertySet.findPropertyEntry(OFBizPropertySet.java:298)
                	at com.opensymphony.module.propertyset.ofbiz.OFBizPropertySet.getType(OFBizPropertySet.java:159)
                	at com.atlassian.jira.propertyset.JiraCachingPropertySet.getType(JiraCachingPropertySet.java:661)
                	at sun.reflect.GeneratedMethodAccessor277.invoke(Unknown Source)
                	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                	at java.lang.reflect.Method.invoke(Method.java:597)
                	at com.atlassian.sal.jira.pluginsettings.LazyProjectMigratingPropertySet$PropertySetInvocationHandler.invoke(LazyProjectMigratingPropertySet.java:57)
                	at $Proxy1841.getType(Unknown Source)
                	at com.atlassian.sal.jira.pluginsettings.JiraPluginSettings.getActual(JiraPluginSettings.java:47)
                	at com.atlassian.sal.core.pluginsettings.AbstractStringPluginSettings.get(AbstractStringPluginSettings.java:126)
                	at com.atlassian.jira.collector.plugin.components.CollectorStoreImpl.getCollector(CollectorStoreImpl.java:103)
                	at com.atlassian.jira.collector.plugin.components.CollectorServiceImpl.getCollector(CollectorServiceImpl.java:51)
                	at com.atlassian.jira.collector.plugin.rest.ConfigurationResource.findCollectorById(ConfigurationResource.java:55)
                	at com.atlassian.jira.collector.plugin.rest.ConfigurationResource.getTriggerConfiguration(ConfigurationResource.java:41)
                	at sun.reflect.GeneratedMethodAccessor315.invoke(Unknown Source)
                	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                	at java.lang.reflect.Method.invoke(Method.java:597)
                	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$ResponseOutInvoker$1.invoke(DispatchProviderHelper.java:234)
                	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$1.intercept(DispatchProviderHelper.java:100)
                	at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61)
                	at com.atlassian.plugins.rest.common.expand.interceptor.ExpandInterceptor.intercept(ExpandInterceptor.java:38)
                	at com.atlassian.plugins.rest.common.interceptor.impl.DefaultMethodInvocation.invoke(DefaultMethodInvocation.java:61)
                	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper.invokeMethodWithInterceptors(DispatchProviderHelper.java:132)
                	at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$ResponseOutInvoker._dispatch(DispatchProviderHelper.java:230)
                	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
                	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
                	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
                	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
                	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
                	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
                	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
                	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
                	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
                	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImp
                

                Also the response is returned with no-cache headers:

                Cache-Control:no-cache, no-store, no-transform

                This means that if an issue collector is enabled in OnDemand for example, every single OnDemand user will trigger a request to JAC for every single page-pop which will potentially hit the DB if the propertysets aren't cached in memory.

                I think we should consider:

                • Moving issue collectors to ActiveObjects which may be more efficient than the currently ugly pluginsettings storage (where everything is split across multiple DB rows)
                • Adding temporary cache headers to this response so not every single end-user page-pop hits JAC. I think we don't have cache headers to allow editing of collectors, but surely several hours to see edits on the client is acceptable for a feature like this.

                Effectively the issue collector in OnDemand is launching a DDOS attack on JAC currently.

                        ohernandez@atlassian.com Oswaldo Hernandez (Inactive)
                        andreask@atlassian.com Andreas Knecht (Inactive)
                        Votes:
                        1 Vote for this issue
                        Watchers:
                        15 Start watching this issue

                          Created:
                          Updated:
                          Resolved:

                            ohernandez@atlassian.com Oswaldo Hernandez (Inactive)
                            andreask@atlassian.com Andreas Knecht (Inactive)
                            Affected customers:
                            1 This affects my team
                            Watchers:
                            15 Start watching this issue

                              Created:
                              Updated:
                              Resolved: