-
Bug
-
Resolution: Fixed
-
Medium (View bug fix roadmap)
-
6.0.6, 6.1.7, 6.2, 6.3-OD-2, 6.3-OD-6
-
None
-
6
-
NOTE: This bug report is for JIRA Server. Using JIRA Cloud? See the corresponding bug report.
Summary
We're seeing an increasing number of issues where customers are reporting initial page load times in excess of 10-15 seconds, and sometimes more.
Cause
When accessing a page within JIRA, a set of batch.js and batch.css files are provided that contain the batched Javascript and CSS resources for the page. Upon first load of a page this cache is created (if it does not already exist on the server), and is then stored for future loads of the page. This cache is a server-side cache and is separate from the client-side cache (ie the cached batch.js and batch.css files in the browser). The creation of the cache can take some time - 10 to 15 seconds - so on first load of a page it will take some time for the page to load.
On first load of any page in JIRA it does download quite a lot of content, which is then cached in the browser, so it is expected that the first page-load will be longer than any subsequent page loads. This is exacerbated when the cache is being recreated as it adds even more time onto that page load.
The cache is created on a language/locale basis, so for example if users access a page in English it will be generated, and other users access pages in Russian it will be generated again. The cache is invalidated when there are Look & Feel changes, or plugin changes occur. This means if these changes occur the cache will need to be recreated and will cause page load times to increase until the cache is created and stored server-side. If there are users with a variety of different languages using the server it will mean the cache is recreated multiple times.
In certain cases where multiple concurrent requests take place they can end up all running the batch generation simultaneously, leading to the CPU spiking and the instance potentially becoming unresponsive while all the requests are processed.
Once a type of page has been visited per user (e.g. the dashboard, a JIRA issue or a project page) the batch.css will be cached for that type of page until the next restart or plugin change. So the first time you view one of these pages the performance will be slow, but it will improve significantly on each subsequent visit.
Steps to reproduce
- Go to any JIRA instance - this can be easily replicated with a jira-dev instance
- Disable or enable a plugin
- Load the dashboard
Looking at the Network tab of developer tools, you'll see that there's a long delay in loading those batch files.
Workaround
Schedule plugin changes during off-peak periods, and then visit a page in JIRA to ensure the cache is recreated during that off-peak period.
Note: It is always best practice to schedule plugin installations and upgrades in off-peak periods or during maintenance windows. For JIRA Download customers it is also recommended to perform major plugin changes on a staging instance before it is done on a live environment.
Diagnosis
For one of the ways of identifying this problem, this will show us the number of threads waiting to process LESS compilation. A higher number over a series of thread dumps can be considered bad.
The below is being executed in a folder with a significant number of thread dumps labelled jstack that were taken over a series of days.
$ grep -B1 "com.atlassian.plugin.cache.filecache.impl.CachedFile.stream" jstack* |grep "waiting to lock" |awk '{print $1}' | sort |uniq -c
31 jstack.13007.110759.851358689-
58 jstack.13007.110810.528822009-
54 jstack.13007.110821.172461523-
42 jstack.4440.threadHigh.103649.939600529-
44 jstack.4440.threadHigh.103701.620551473-
48 jstack.4440.threadHigh.103713.178859435-
51 jstack.4440.threadHigh.103724.736087099-
57 jstack.4440.threadHigh.103736.376659166-
Here we see the thread dumps taken over time continue to consistently have a higher number of threads most likely waiting to compile LESS. The below thread dump shows samples us the entire stack trace.
Running thread
"http-bio-8080-exec-281" daemon prio=10 tid=0x00007fe2041f0000 nid=0x349c runnable [0x00007fe4645d4000] java.lang.Thread.State: RUNNABLE at org.mozilla.javascript.ScriptableObject.getSlot(ScriptableObject.java:2714) at org.mozilla.javascript.ScriptableObject.get(ScriptableObject.java:452) at org.mozilla.javascript.IdScriptableObject.get(IdScriptableObject.java:329) at org.mozilla.javascript.ScriptableObject.getProperty(ScriptableObject.java:2184) at org.mozilla.javascript.ScriptRuntime.getPropFunctionAndThisHelper(ScriptRuntime.java:2261) at org.mozilla.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2251) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_50(less-concat.js:895) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97) at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) at org.mozilla.javascript.gen.less_concat_js_2._c_$_18(less-concat.js:261) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_55(less-concat.js:1115) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97) at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) at org.mozilla.javascript.gen.less_concat_js_2._c_$_18(less-concat.js:261) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_79(less-concat.js:1602) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97) at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) at org.mozilla.javascript.gen.less_concat_js_2._c_$_18(less-concat.js:261) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_71(less-concat.js:1507) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97) at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) at org.mozilla.javascript.gen.less_concat_js_2._c_$_18(less-concat.js:261) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_64(less-concat.js:1297) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97) at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) at org.mozilla.javascript.gen.less_concat_js_2._c_$_18(less-concat.js:261) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_35(less-concat.js:655) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97) at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) at org.mozilla.javascript.gen.less_concat_js_2._c_$_18(less-concat.js:261) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_28(less-concat.js:503) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42) at org.mozilla.javascript.gen.setup_env_js_1._c_anonymous_2(setup-env.js:29) at org.mozilla.javascript.gen.setup_env_js_1.call(setup-env.js) at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_12(less-concat.js:221) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_223(less-concat.js:2702) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.BaseFunction.construct(BaseFunction.java:338) at org.mozilla.javascript.ScriptRuntime.newObject(ScriptRuntime.java:2349) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_65(less-concat.js:1330) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97) at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) at org.mozilla.javascript.gen.less_concat_js_2._c_$_18(less-concat.js:261) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_69(less-concat.js:1408) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97) at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) at org.mozilla.javascript.gen.less_concat_js_2._c_$_18(less-concat.js:261) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_35(less-concat.js:655) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97) at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) at org.mozilla.javascript.gen.less_concat_js_2._c_$_18(less-concat.js:261) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_28(less-concat.js:503) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42) at org.mozilla.javascript.gen.setup_env_js_1._c_anonymous_2(setup-env.js:29) at org.mozilla.javascript.gen.setup_env_js_1.call(setup-env.js) at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_12(less-concat.js:221) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_223(less-concat.js:2702) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.BaseFunction.construct(BaseFunction.java:338) at org.mozilla.javascript.ScriptRuntime.newObject(ScriptRuntime.java:2349) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_65(less-concat.js:1330) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97) at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) at org.mozilla.javascript.gen.less_concat_js_2._c_$_18(less-concat.js:261) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_69(less-concat.js:1408) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97) at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) at org.mozilla.javascript.gen.less_concat_js_2._c_$_18(less-concat.js:261) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_35(less-concat.js:655) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97) at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) at org.mozilla.javascript.gen.less_concat_js_2._c_$_18(less-concat.js:261) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_28(less-concat.js:503) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42) at org.mozilla.javascript.gen.setup_env_js_1._c_anonymous_2(setup-env.js:29) at org.mozilla.javascript.gen.setup_env_js_1.call(setup-env.js) at org.mozilla.javascript.optimizer.OptRuntime.callN(OptRuntime.java:52) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_12(less-concat.js:221) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_223(less-concat.js:2702) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.BaseFunction.construct(BaseFunction.java:338) at org.mozilla.javascript.ScriptRuntime.newObject(ScriptRuntime.java:2349) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_65(less-concat.js:1330) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97) at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) at org.mozilla.javascript.gen.less_concat_js_2._c_$_18(less-concat.js:261) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_69(less-concat.js:1408) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97) at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) at org.mozilla.javascript.gen.less_concat_js_2._c_$_18(less-concat.js:261) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_35(less-concat.js:655) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.ScriptRuntime.applyOrCall(ScriptRuntime.java:2430) at org.mozilla.javascript.BaseFunction.execIdCall(BaseFunction.java:269) at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:97) at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32) at org.mozilla.javascript.gen.less_concat_js_2._c_$_18(less-concat.js:261) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63) at org.mozilla.javascript.gen.less_concat_js_2._c_anonymous_28(less-concat.js:503) at org.mozilla.javascript.gen.less_concat_js_2.call(less-concat.js) at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42) at org.mozilla.javascript.gen.setup_env_js_1._c_anonymous_11(setup-env.js:94) at org.mozilla.javascript.gen.setup_env_js_1.call(setup-env.js) at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:394) at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3091) at org.mozilla.javascript.gen.setup_env_js_1.call(setup-env.js) at com.atlassian.jira.web.less.LessTransformer$LessCompiler.compile(LessTransformer.java:438) at com.atlassian.jira.web.less.LessTransformer$LessResource.transform(LessTransformer.java:220) at com.atlassian.plugin.webresource.transformer.CharSequenceDownloadableResource$1.apply(CharSequenceDownloadableResource.java:49) at com.atlassian.plugin.webresource.transformer.CharSequenceDownloadableResource$1.apply(CharSequenceDownloadableResource.java:46) at com.atlassian.plugin.webresource.transformer.TransformerUtils.transformAndStreamResource(TransformerUtils.java:40) at com.atlassian.plugin.webresource.transformer.CharSequenceDownloadableResource.streamResource(CharSequenceDownloadableResource.java:45) at com.atlassian.plugin.webresource.transformer.TransformerUtils.transformAndStreamResource(TransformerUtils.java:38) at com.atlassian.plugin.webresource.AbstractTransformResource.streamResource(AbstractTransformResource.java:60) at com.atlassian.plugin.webresource.transformer.TransformerUtils.transformAndStreamResource(TransformerUtils.java:38) at com.atlassian.plugin.webresource.AbstractTransformResource.streamResource(AbstractTransformResource.java:60) at com.atlassian.plugin.webresource.AbstractBatchDownloadableResource$2.writeStream(AbstractBatchDownloadableResource.java:115) at com.atlassian.plugin.cache.filecache.impl.CachedFile.streamToCache(CachedFile.java:188) at com.atlassian.plugin.cache.filecache.impl.CachedFile.doEnter(CachedFile.java:120) - locked <0x00007fecb8a2b398> (a java.lang.Object) at com.atlassian.plugin.cache.filecache.impl.CachedFile.stream(CachedFile.java:82) at com.atlassian.plugin.cache.filecache.impl.FileCacheImpl.stream(FileCacheImpl.java:92) at com.atlassian.plugin.webresource.cache.CacheHandle$Builder$2.stream(CacheHandle.java:48) at com.atlassian.plugin.webresource.AbstractBatchDownloadableResource.streamResourceInternal(AbstractBatchDownloadableResource.java:125) at com.atlassian.plugin.webresource.AbstractBatchDownloadableResource.serveResource(AbstractBatchDownloadableResource.java:97) at com.atlassian.plugin.webresource.servlet.PluginResourceDownload.serveFile(PluginResourceDownload.java:96) at com.atlassian.plugin.servlet.AbstractFileServerServlet.doGet(AbstractFileServerServlet.java:33) ...
Blocked thread sample
"http-bio-8080-exec-283" daemon prio=10 tid=0x00007fe2041f4000 nid=0x349e waiting for monitor entry [0x00007fe4643d9000] java.lang.Thread.State: BLOCKED (on object monitor) at com.atlassian.plugin.cache.filecache.impl.CachedFile.doEnter(CachedFile.java:115) - waiting to lock <0x00007fecb8a2b398> (a java.lang.Object) at com.atlassian.plugin.cache.filecache.impl.CachedFile.stream(CachedFile.java:82) at com.atlassian.plugin.cache.filecache.impl.FileCacheImpl.stream(FileCacheImpl.java:92) at com.atlassian.plugin.webresource.cache.CacheHandle$Builder$2.stream(CacheHandle.java:48) at com.atlassian.plugin.webresource.AbstractBatchDownloadableResource.streamResourceInternal(AbstractBatchDownloadableResource.java:125) at com.atlassian.plugin.webresource.AbstractBatchDownloadableResource.serveResource(AbstractBatchDownloadableResource.java:97) at com.atlassian.plugin.webresource.servlet.PluginResourceDownload.serveFile(PluginResourceDownload.java:96) at com.atlassian.plugin.servlet.AbstractFileServerServlet.doGet(AbstractFileServerServlet.java:33) at javax.servlet.http.HttpServlet.service(HttpServlet.java:621) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at com.atlassian.jira.web.filters.XContentTypeOptionsNoSniffFilter.doFilter(XContentTypeOptionsNoSniffFilter.java:22) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) ...
- is duplicated by
-
-
- Closed
-
- relates to
-
-
- Closed
-
-
DESK-2911 Failed to load
-
FUSE-1251 Failed to load
-
JDEV-28460 Failed to load
- is related to
-
PLUGWEB-102 Loading...
- mentioned in
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
[JRASERVER-37337] Batch CSS and Javascript generation causes long delays in page loading after plugin modification
Hi.
We've also faced this issue in ver. 5.2.11(https, sa server), and it seems like intranet users don't met this issue, only external users. Also this issue relates to number of active jira users, ours is about 300.
Hi yassine.mansour, please reach out to us at http://support.atlassian.com, that way we will able to troubleshoot and investigate what is wrong with your instance.
Thanks in advance.
Regards,
Oswaldo Hernández.
JIRA Bugmaster.
[Atlassian].
Problem Reoccurs on my Production environment (400 USERS Blocked) in 6.3.1. NOT NORMAL
Unfortunately, it's totally random, as long as there is an activity on the instance, of course. It's not admin changes that cause the problem, even if an action like a reindex or a "System info" increase the problem.
MarkW
added a comment - @Nicolas Do you know what actions you were performing when it happened? Can you check your audit logs to see if that can help tell you what admin changes were occurring?
MarkW
added a comment - @Nicolas Do you know what actions you were performing when it happened? Can you check your audit logs to see if that can help tell you what admin changes were occurring? Hello,
I encounter a similar problem (bad performance caused by batch.css) but without a plugin modification. What do you think? Is it related? JIRA 6.3.15 / 6.3.10
Regards,
Nicolas.
econnats It's possible that two users will have two different batches, thus requiring the batches to be built and cached for each user. An example of when this would happen is if the two users have a different locale set. However if two users have the same configuration, they can share batches.
Hi Richie, our company is running JIRA 6.3.15 in dev in prep for upgrading to this version, but we are experiencing the delays for the batch scripts that are mentioned in this JIRA.
Are batch.js and batch.css cached per page type? OR Are batch.js and batch.css cached per page type per user?
I'm wondering if let's say the "dashboard" batch.css and batch.js files are cached for user "A", if they will also need to be cached for user "B" (meaning each user has his/her own batch..css/js files cached on the server).
Thanks,
Eddie
Hey james.dempster , roman30, aich,
Thanks for the feedback, but before we conclude that this is contributed by the said bug, our support engineers would like to help you further to identify the root cause.
Please head towards the link below to create the ticket so that our support team can work on it:
Select OnDemand Support if you are using JIRA Cloud and JIRA Support for JIRA Server.
Looking forward to hear from you soon.
PS: Thanks richard36 for sharing the workaround to benefit the community, and yes james.dempster, only JIRA Cloud has this feature and only we can make the changes.
Regards,
Richie Gee
James Dempster
added a comment - Would we need a member of the support staff to make the change? Don't think there's an option in JIRA for that.
James Dempster
added a comment - Would we need a member of the support staff to make the change? Don't think there's an option in JIRA for that. 
Richard Donkin
added a comment - For JIRA OnDemand users - I logged a ticket that was eventually resolved, similar to this. After a lot of interaction inc sending Chrome HAR files showing slow page loads, the solution was to disable the CDN for our JIRA instance. Seems like cache headers / behaviour was also involved.
For any Atlassian staff, see https://support.atlassian.com/servicedesk/customer/portal/23/JST-106553
Richard Donkin
added a comment - For JIRA OnDemand users - I logged a ticket that was eventually resolved, similar to this. After a lot of interaction inc sending Chrome HAR files showing slow page loads, the solution was to disable the CDN for our JIRA instance. Seems like cache headers / behaviour was also involved.
For any Atlassian staff, see https://support.atlassian.com/servicedesk/customer/portal/23/JST-106553 James Dempster
added a comment - I agree. This issue is far from fixed.
We use Jira ondemand, and the performance is awful, and the only reason is this issue with batch.css and batch.js taking so long.
James Dempster
added a comment - I agree. This issue is far from fixed.
We use Jira ondemand, and the performance is awful, and the only reason is this issue with batch.css and batch.js taking so long. 
Roman Semko
added a comment - We started using jira few weeks ago (version 6 dot something). batch.css and batch.js take always +24 seconds to load. This ticket is by far not solved. Does it make sense re-opening or is there a new ticket somewhere?
Roman Semko
added a comment - We started using jira few weeks ago (version 6 dot something). batch.css and batch.js take always +24 seconds to load. This ticket is by far not solved. Does it make sense re-opening or is there a new ticket somewhere? I don't think this is fixed either, not sure why it is marked as resolved.
Rahul Aich [Nagra]
added a comment - I am on 6.3.10, but i can still see batch.js file taking 40 sec to load, when accessing an issue.Sometimes agile boards can take 80sec to load (almost entirely the slowness caused bybatch.js)
Is there a workaround to this? This looks like not been fixed.
Is any of the other users facing the same problem?
Rahul
Rahul Aich [Nagra]
added a comment - I am on 6.3.10, but i can still see batch.js file taking 40 sec to load, when accessing an issue.Sometimes agile boards can take 80sec to load (almost entirely the slowness caused bybatch.js)
Is there a workaround to this? This looks like not been fixed.
Is any of the other users facing the same problem?
Rahul Given what Markdw just added here, I start to understand why when we promote such configuration changes to customers on 6.2 we get a lot of complaints about performance. Seems more and more like a critical bug that needed to be addressed also on 6.2 maintenance releases.
MarkW
added a comment - In addition to Look & Feel changes, changes to Field Configuration schemes (making fields required) triggered this bug. Found out from a support issue I submitted.
MarkW
added a comment - In addition to Look & Feel changes, changes to Field Configuration schemes (making fields required) triggered this bug. Found out from a support issue I submitted. Hi David,
Missed that in the cause paragraph, sorry. Thanks for pointing out. I suppose there is no chance to see this fixed in 6.2.x codeline, right? 6.3 drops support for IE8 which is not yet replaced company-wide at the customer, so upgrading is not an option yet.
Cheers.
Hey aggelosp, yes that's correct - a look & feel change will invalidate the cache as in the description:
The cache is invalidated when there are Look & Feel changes, or plugin changes occur.
I believe this LESS compilation issue is not only related to plugin updates. We had a similar pattern when changing the logo (color scheme) on a JIRA 6.2.4 instance. Attaching the snapshot.
Practically the instance stops servicing requests. All threads blocked by LESS compiling thread
"http-bio-8880-exec-77" - Thread t@4273 java.lang.Thread.State: BLOCKED at com.atlassian.plugin.cache.filecache.impl.CachedFile.doEnter(CachedFile.java:115) - waiting to lock <36428d17> (a java.lang.Object) owned by "http-bio-8880-exec-111" t@4664 at com.atlassian.plugin.cache.filecache.impl.CachedFile.stream(CachedFile.java:82) at com.atlassian.plugin.cache.filecache.impl.FileCacheImpl.stream(FileCacheImpl.java:92) "http-bio-8880-exec-111" - Thread t@4664 java.lang.Thread.State: RUNNABLE at org.mozilla.javascript.regexp.NativeRegExp.executeREBytecode(NativeRegExp.java:1916) at org.mozilla.javascript.regexp.NativeRegExp.matchRegExp(NativeRegExp.java:2381) at org.mozilla.javascript.regexp.NativeRegExp.executeRegExp(NativeRegExp.java:2412) at org.mozilla.javascript.regexp.NativeRegExp.execSub(NativeRegExp.java:269)
Hi Oswaldo,
I have only seen these issues on my test/dev instance (on which I admittedly do a lot of plugin installs), so I actually don't particularly care if it's broken (or need to see it resolved)...the offer to do any further analysis was just in case it was hard to reproduce and your team needed a little extra help catching it in the wild. If you'd still like me to contact support even in that scenario, I am happy to do so.
Hi sdudley,
Can you please raise a support request at https://support.atlassian.com when you observe a problem, so that analysis can be done on logs/thread dump/memory dump to understand the root cause of what you are observing in your instance.
As the underlying cause needs to be understood and might be particular to the specifics of your JIRA instance.
Thanks a lot for your help on this.
Regards,
Oswaldo Hernández.
JIRA Bugmaster.
[Atlassian].
Fixes for the issue have been implemented and delivered by the team in JIRA 6.3.
A significant amount of developer time was invested in them, and so far according to the evidence we have been able to gather the customers suffering from the symptoms described here has significantly decreased which is great news.
Having said that, this doesn't preclude that there might be other performance problems in need of fixing for some customer scenarios on plugin install / uninstall. So far we have been unable to get any reproducible confirmed report out of this though.
As soon as this happens, a clear targeted issue with a reproducible steps/scenario will get raised to tackle those problems if necessary.
Hope the information clears up the state of this.
Regards,
Oswaldo Hernández.
JIRA Bugmaster.
[Atlassian].
Intel CHD Jira Admin
added a comment - If this issue is in fact not fixed, wouldn't it make sense to reopen this issue - or at least create a new issue and divert people's attention there? Continuing to comment on a resolved issue is not an effective way to work...
-David
Intel CHD Jira Admin
added a comment - If this issue is in fact not fixed, wouldn't it make sense to reopen this issue - or at least create a new issue and divert people's attention there? Continuing to comment on a resolved issue is not an effective way to work...
-David I've been seeing this occasionally in JIRA 6.3 too.
In case it helps, I usually run my instance attached to a debugger. If there is any specific information that would be helpful to Atlassian (eg. the names of specific files being compiled, or perhaps the values of local or class variables within a specific scope), I would be happy to dig that information out of the debugger for you the next time it happens. Just email me if that's the case.
Thanks gregory.hoggarth, we'll investigate and troubleshoot with you further in the support ticket 
Regards,
Oswaldo Hernández.
JIRA Bugmaster.
[Atlassian].
I already have, which is where I learned about this issue ticket and subsequently upgraded to 6.3 because it was said to solve the problem. You can find thread dumps and a thorough description of the issues I am seeing.
Ticket is JSP-197170
Hi gregory.hoggarth,
I am glad that the performance hit of a plugin change has been improved for you after 6.3.
I can confirm that the performance issues caused by the factors described here are indeed fixed (less compilation and web-resource caching).
Having said that, this does not preclude that there might be other performance issues in the system that might manifest themselves by a perceived decrease in overall system performance during a plugin upgrade for some customers/instances/configuration scenarios.
Following on, If there remain performance issues in your instance after a plugin deployment, we need to understand exactly what those issues are in order for us to be able to reproduce them and then be able to fix them.
Consequently, I believe that in order to pursue that further we would need for you to help us by creating a support ticket at https://support.atlassian.com so that we can get the details of what's happening in your instance and do further diagnosis and troubleshooting.
Thanks in advance.
Regards,
Oswaldo Hernández.
JIRA Bugmaster.
[Atlassian].
Even after upgrading to Jira 6.3, we're still having the problem of poor performance after a plugin is changed. It does appear to be better than it was in 6.2, in that Jira will recover faster (or before, it never recovered at all), but it's still not completely fixed.
I have entered a ticket and have been working with support for 2 weeks now with no resolution. My support ticket is JST-102329.
The fix for the issue I mentioned above (https://ecosystem.atlassian.net/browse/PLUGWEB-142) is live in both Confluence and JIRA OnDemand. The LESS compilation improvements from Bradley Ayers are also live in JIRA OnDemand.
We are continuing to investigate further performance improvements for both JIRA and Confluence OnDemand.
If you are experiencing continued performance issues with your OnDemand instance, please submit a problem report through https://support.atlassian.com/servicedesk/customer/jst so further investigation can be performed.
Cheers,
Ian.
Neither for me, very frustrating issue, especially when trying to evaluate Confluence OnDemand. Instance is basically unusable.
James Dempster
added a comment - If this fix has been rolled out to onDemand, indeed it didn't fix it for us either.
James Dempster
added a comment - If this fix has been rolled out to onDemand, indeed it didn't fix it for us either. 
Anders
added a comment - This didn't fix it for me! 
Ankur Dhawan
added a comment - Can you tell when will be the official release of JIRA 6.3, we are on 6.0.8 and want to upgrade to 6.3 as soon as it is available for download.
Ankur Dhawan
added a comment - Can you tell when will be the official release of JIRA 6.3, we are on 6.0.8 and want to upgrade to 6.3 as soon as it is available for download. I have identified some large superfluous CPU work being performed when serving context batch resources from the file cache. This additional CPU work has been removed for JIRA 6.3 and OD8. We believe this is the cause of most of the performance issues experienced on this issue, where the browser is waiting for the first byte of the response for the server for a long period of time.
I measured the impact through local requests on an ondemand instance to itself, to reduce the impact of network variation. I made 100 requests to the rapid board javascript, which is a context batch containing contexts greehopper-rapid-non-gadget,atl.general,gh-rapid and jira.global.
Concurrency 1, Mean time per request:
Before: 893.572ms
After: 55.608ms
Speedup: ~16x
Concurrency 10, Mean time per request:
Before: 6379.514ms
After: 522.127ms
Speedup: ~12x - though the variation is much lower (slowest request reduced from 11549ms to 666ms, and GC activity is significantly lower.)
LESS precompilation has been added to JIRA 6.3, this means where possible, LESS is compiled to CSS when we are building JIRA, rather than at run time. Because there are dynamic inputs to some of the LESS in JIRA it's not possible to precompile all of it (e.g. plugins, look and feel). The motivation behind introducing LESS precompilation is to reduce the runtime load on JIRA when these dynamic inputs change (which invalidate the web resource cache).
What follows is the results of the change.The way benchmark was done was:
1. Created a new <web-resource> in the plugin that included all the .less resources that were declared in the atlassian-plugin.xml.
2. Start JIRA (./jmake debug clean -sh --override-dev-mode) then make 30 or so requests for the <web-resource> until the variance in response time becomes low, then take the average of a sample (size=8).
3. Enable LESS precompilation (via the maven plugin) and take another sample as per #2
4. Compare results.
The following is the change in compilation time for LESS resources:
jira-project-config-plugin: 9% of original
fusion: 21% of original
agile: 29% of original
issue search: 26% of original
superbatch: 78% of original
And here's a graph of ⤴︎:
It looks like there's still probably some wins to be had in the superbatch, going by what was possible in the other plugins. The strategy for enabling LESS precompilation will be standard practice for future development and refactoring.
Ashok Manthina
added a comment - It is related to LESS compilation. Please read https://support.atlassian.com/servicedesk/customer/jsp/problem-report-190560
Ashok Manthina
added a comment - It is related to LESS compilation. Please read https://support.atlassian.com/servicedesk/customer/jsp/problem-report-190560 derek_sheeman The patch that has been made as a result of this issue is only compatible with JIRA ≥6.3, due to its reliance on the new LESS compiler plugin that's being used.
ashok.manthina this issue is specific to delays caused by LESS compilation, which is performed the first time pages are accessed after startup, and again after plugins are enabled/disabled/installed/uninstalled and look and feel settings are changed. If you're experiencing page load delays at times that don't fit into these categories, then it's possible that the problems you're experiencing are unrelated to this problem.
Ashok Manthina
added a comment - This issue happens even without any plugin management. So far no idea what's causing it. We are using Atlassian JIRA (v6.0.8#6109-sha1:57cf099)
Ashok Manthina
added a comment - This issue happens even without any plugin management. So far no idea what's causing it. We are using Atlassian JIRA (v6.0.8#6109-sha1:57cf099) 
Derek Sheeman
added a comment - This effectively crashes my instances of JIRA 6.2.5 whenever I touch plugins.
my UPM updates this evening hung 2 instances which had to be restarted.
It would be nice if a patch existed for 6.2 so we don't have to suffer this for several more months waiting for plugins to catch up to 6.3, whenever that comes out.
Derek Sheeman
added a comment - This effectively crashes my instances of JIRA 6.2.5 whenever I touch plugins.
my UPM updates this evening hung 2 instances which had to be restarted.
It would be nice if a patch existed for 6.2 so we don't have to suffer this for several more months waiting for plugins to catch up to 6.3, whenever that comes out. Ashok Manthina
added a comment - We are running into same problems. Any update on when the fix will be made available?
-am
Ashok Manthina
added a comment - We are running into same problems. Any update on when the fix will be made available?
-am Hi Mark,
We have made an important amount of progress on this issue and should be fixing in JIRA 6.3 a series of significant improvements to reduce the time spent by JIRA during compilation of LESS resources on plugin install / uninstall.
These improvements have been obtained by pre-compiling as many LESS resource files in JIRA and its bundled plugins and therefore reduce the cost of compiling by moving it to build time.
Unfortunately, this is dependent on the new cross-product LESS compiler used by JIRA in 6.3, which in turn depends on a series of libraries that can not be upgraded in JIRA 6.2, so as much as we would like to we won't be able to deliver these improvements to JIRA 6.2.x
Hope the information helps, and that you are able to move to JIRA 6.3 soon to obtain these fixes.
Regards,
Oswaldo Hernández.
JIRA Bugmaster.
[Atlassian].
The fix version = 6.3
Can I ask that the defect also be fixed in earlier versions, eg: 6.2.x
The problem with a "v6.3-only" fix is that v6.3 won't be released for a month or two yet... and even when it is released, I would not be given permission to upgrade until at least v6.3.1 or maybe v6.3.2.
Aha - I thought this was the old felix bug, but this very closely matches what I'm seeing after our Jira 6.1.x upgrade.
Our experience is exactly like Mark Symons' above - and given we are a 24/7 shop we can't make many guarantees about finding slack time for plugin updates.
Bruce Cran
added a comment - We're seeing this problem on our 6.2.3 installation: when updating plugins, the progress bar stops and pages start taking several minutes to load and support tickets start coming in. I've been having to restart JIRA to fix the problem.
Bruce Cran
added a comment - We're seeing this problem on our 6.2.3 installation: when updating plugins, the progress bar stops and pages start taking several minutes to load and support tickets start coming in. I've been having to restart JIRA to fix the problem. I'm having this bug as well - highly reproducible on JIRA OnDemand: just clear your browser cache to get very slow loading (10 to 20 sec typically) then reload with cache populated, and all is much faster (3 to 6 sec typically). I haven't had any of the >20 sec load times, that may well be a different issue.
I submitted a case to JIRA support with HAR files from Chrome as requested to show exact timing of requests, and they agreed it was this bug (even though not quite the same issue).
For any Atlassian support staff reading this, please see https://support.atlassian.com/servicedesk/customer/jst/problem-report-96241 for the case.
General links on slow page loading that I found:
- many issues with slow page loads with OnDemand, for a least a year: https://answers.atlassian.com/questions/143420/ondemand-products-are-very-slow
I would really like an assurance that this will be fixed very soon, and a target date for the fix.
For anyone reading this who has the same issue - please Vote and Watch this bug, and add a comment with your details.
Richard Donkin
added a comment - - edited I'm having this bug as well - highly reproducible on JIRA OnDemand: just clear your browser cache to get very slow loading (10 to 20 sec typically) then reload with cache populated, and all is much faster (3 to 6 sec typically). I haven't had any of the >20 sec load times, that may well be a different issue.
I submitted a case to JIRA support with HAR files from Chrome as requested to show exact timing of requests, and they agreed it was this bug (even though not quite the same issue).
For any Atlassian support staff reading this, please see https://support.atlassian.com/servicedesk/customer/jst/problem-report-96241 for the case.
General links on slow page loading that I found:
many issues with slow page loads with OnDemand, for a least a year: https://answers.atlassian.com/questions/143420/ondemand-products-are-very-slow
I would really like an assurance that this will be fixed very soon, and a target date for the fix.
For anyone reading this who has the same issue - please Vote and Watch this bug, and add a comment with your details. This is a problem for our installation as well. We used to see it only after server restarts and it would clear up after users loaded various pages, but I installed the Jitter Notify plugin and now the batch.css file is not loading what seems every hour and causing a lot of slow page loads for our users. Firefox gives up on batch.css after 60 seconds, allowing the page to load but the menu bar takes up half of the screen because of it.
I would also like to be assured that this issue will be fixed as soon as possible.
I've installed an Apache proxy so that I can deny access to JIRA during maintenance.
I also use Javamelody to terminate user all users sessions but my own before upgrading a plugin
...but I still end up with CPU peaking at 90% and the system unresponsive for up to 20 minutes... meaning that it's quicker to restart the JIRA service rather than wait.
If I have multiple plugins that need to be upgraded then I have to rinse and repeat. Most frustrating... and expensive. Time is money.
We also have a customer who has that problem since upgrading from JIRA 5.x to JIRA 6.1.6.
For example, yesterday some plugins were updated and immediately after that the batch resources (mostly CSS) where slow (>10s) again, after being fast for many days. The weird thing is, that 24 hours later there are still many slow requests for batch resources. Reading this issue, I thought that only the first requests after plugin modification would be slow. See slow-batch-resource-requests.txt
for the slow (>10s) requests of the last 2 hours.
What is the status of this? How long before it is assigned and fixed?
Hi figarick,
This issue was addressed in JIRA 6.3. Consequently, you will need to upgrade to at least JIRA 6.3 (or above) to see any notable differences here. If you need assistance/advice with the upgrade please contact us at https://support.atlassian.com and we'll be glad to help you there.
Regards,
Oswaldo Hernández.
JIRA Bugmaster.
[Atlassian].