Loading...

XML

Word

Printable

Type: Bug
Resolution: Fixed
Priority: Highest
Fix Version/s: 8.15.0
Affects Version/s: 8.5.0, 8.5.1, 8.13.0, 8.13.1
Component/s: REST API
Labels:

Introduced in Version:
8.05
Support reference count:
32
Symptom Severity:
Severity 1 - Critical
UIS:
280
Bug Fix Policy:
View Atlassian Server bug fix policy

Issue Summary
Adding Jira Issues Dates event types to Team Calendars is extremely non performant, and can cause outages by consuming threads that can take hours to return. It causes high CPU on the processing nodes which can also result in performance impacts and system outages, as system resources cannot be used to respond to other requests.

This issue was recently resolved in Jira Cloud by upgrading the Fortuna ical library. The issue is caused by the library creating thousands of date objects per date, and checking each against a call to the system date, to determine timezones.

Steps to Reproduce

Confluence and Jira

Create a custom field in JIRA of time Date/Time Picker.
Set this field for a few issues (5 was enough for production instance).
In Confluence open Calendars and add a new calendar
Click '...' and Add event
Select Jira Issue Dates, and the Jira project with the custom fields you set
Select some date fields including the custom field you created and submit
It will take a long time to add the events, or if the project is large they may never be added to the calendar

Jira only

Create a custom field in JIRA of time Date/Time Picker.
Set this field for a few issues (5 was enough for production instance).
Call /rest/ical/1.0/ical/config/fields?jql=project=<project_key> Jira endpoint to find out the custom field id (customfield_XXXXX).
Call /rest/ical/1.0/ical/search/jql/events.ics?jql=project+%3D+%22<project_key>%22&dateFieldName=<custom_field_id>&includeFixVersions=false&start=1598572800000&end=1609977600000&maxIssue=1000 endpoint to retrieve a list of events in iCal format (set Jira project key, custom field id, and start/stop timestamps in parameters). Events are generated based on the value of the custom field. So make sure that this date is within the timeframe set by start and end arguments.
You will see a 20+ second delay even though the number of issues/events is small.

Expected Results

Requests to Jira REST endpoints return in a timely manner.

Actual Results

Threads are consumed and don't return for hours, and cause high CPU, which leads to performance degradation and outages.

There is no exception in the logfile as the threads are still processing. However, threads similar to those below will be visible in thread dumps:

http-nio-8080-exec-1 url:/rest/ical/1.0/...rch/jql/events.ics
priority:5 - threadId:0x00007f44f1450000 - nativeId:0x47a8 - nativeId (decimal):18344 - state:RUNNABLE
stackTrace:
java.lang.Thread.State: RUNNABLE
at java.util.GregorianCalendar.computeTime(GregorianCalendar.java:2818)
at java.util.Calendar.updateTime(Calendar.java:3393)
at java.util.Calendar.getTimeInMillis(Calendar.java:1782)
at net.fortuna.ical4j.util.Dates.round(Dates.java:300)
at net.fortuna.ical4j.model.Iso8601.<init>(Iso8601.java:74)
at net.fortuna.ical4j.model.Time.<init>(Time.java:97)
at net.fortuna.ical4j.model.Time.<init>(Time.java:88)
at net.fortuna.ical4j.model.DateTime.<init>(DateTime.java:237)
at net.fortuna.ical4j.util.Dates.getInstance(Dates.java:223)
at net.fortuna.ical4j.model.Recur.getMonthDayVariants(Recur.java:935)
at net.fortuna.ical4j.model.Recur.getCandidates(Recur.java:781)
at net.fortuna.ical4j.model.Recur.getDates(Recur.java:620)
at net.fortuna.ical4j.model.Recur.getDates(Recur.java:516)
at net.fortuna.ical4j.model.component.Observance.getLatestOnset(Observance.java:242)
at net.fortuna.ical4j.model.component.VTimeZone.getApplicableObservance(VTimeZone.java:313)
at net.fortuna.ical4j.model.TimeZone.inDaylightTime(TimeZone.java:120)
at java.util.TimeZone.getOffset(TimeZone.java:209)
at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2342)
at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2312)
at java.util.Calendar.setTimeInMillis(Calendar.java:1804)
at net.fortuna.ical4j.util.Dates.round(Dates.java:287)
at net.fortuna.ical4j.model.Iso8601.<init>(Iso8601.java:74)
at net.fortuna.ical4j.model.Time.<init>(Time.java:116)
at net.fortuna.ical4j.model.DateTime.setTimeZone(DateTime.java:408)
at net.fortuna.ical4j.model.property.DateProperty.updateTimeZone(DateProperty.java:188)
at net.fortuna.ical4j.model.property.DateProperty.setTimeZone(DateProperty.java:163)
at com.atlassian.jira.extra.icalfeed.IcalendarResource.toVEvent(IcalendarResource.java:479)
at com.atlassian.jira.extra.icalfeed.IcalendarResource.toIcalendar(IcalendarResource.java:364)
at com.atlassian.jira.extra.icalfeed.IcalendarResource.search(IcalendarResource.java:323)
at com.atlassian.jira.extra.icalfeed.IcalendarResource.getIcalendarByJql(IcalendarResource.java:273)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)

http-nio-8080-exec-110 url:/rest/ical/1.0/...rch/jql/events.ics
java.util.Hashtable.get(Hashtable.java:363)
java.util.Properties.getProperty(Properties.java:969)
net.fortuna.ical4j.util.Configurator.getProperty(Configurator.java:75)
net.fortuna.ical4j.util.CompatibilityHints.isHintEnabled(CompatibilityHints.java:118)
net.fortuna.ical4j.model.Iso8601.<init>(Iso8601.java:78)
net.fortuna.ical4j.model.Time.<init>(Time.java:97)
net.fortuna.ical4j.model.Time.<init>(Time.java:88)
net.fortuna.ical4j.model.DateTime.<init>(DateTime.java:237)
net.fortuna.ical4j.util.Dates.getInstance(Dates.java:223)
net.fortuna.ical4j.model.Recur.getMonthDayVariants(Recur.java:935)
net.fortuna.ical4j.model.Recur.getCandidates(Recur.java:781)
net.fortuna.ical4j.model.Recur.getDates(Recur.java:620)
net.fortuna.ical4j.model.Recur.getDates(Recur.java:516)
net.fortuna.ical4j.model.component.Observance.getLatestOnset(Observance.java:242)
net.fortuna.ical4j.model.component.VTimeZone.getApplicableObservance(VTimeZone.java:313)
net.fortuna.ical4j.model.TimeZone.inDaylightTime(TimeZone.java:120)
java.util.TimeZone.getOffset(TimeZone.java:209)
java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2342)
java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2312)
java.util.Calendar.setTimeInMillis(Calendar.java:1804)
net.fortuna.ical4j.util.Dates.round(Dates.java:287)
net.fortuna.ical4j.model.Iso8601.<init>(Iso8601.java:74)
net.fortuna.ical4j.model.Time.<init>(Time.java:116)
net.fortuna.ical4j.model.DateTime.setTimeZone(DateTime.java:408)
net.fortuna.ical4j.model.property.DateProperty.updateTimeZone(DateProperty.java:188)
net.fortuna.ical4j.model.property.DateProperty.setTimeZone(DateProperty.java:163)
com.atlassian.jira.extra.icalfeed.IcalendarResource.toVEvent(IcalendarResource.java:479)
com.atlassian.jira.extra.icalfeed.IcalendarResource.toIcalendar(IcalendarResource.java:364)
com.atlassian.jira.extra.icalfeed.IcalendarResource.search(IcalendarResource.java:323)
com.atlassian.jira.extra.icalfeed.IcalendarResource.getIcalendarByJql(IcalendarResource.java:273)
sun.reflect.GeneratedMethodAccessor7727.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)

Notes

Currently we are aware of couple of problems related to ical4j:

High CPU load (current ticket - ~~JRASERVER-71803~~)
- this is a contention due to frequent getProperty calls
Thread being blocked/stuck trying to contact http://tzurl.org, this is not related to current ticket
- this is a contention in the network and there is a feature flag to prevent the call by disabling the TZ update definition (see https://ical4j.github.io/ical4j-user-guide/timezones/ for more details), to do that:
- Set the property "-Dnet.fortuna.ical4j.timezone.update.enabled=false" on Jira startup options.
- Restart Jira

Workaround

Restarting the affected nodes kills the threads and will return the system to performing normally.
* As an extreme measure, the /rest/ical/1.0/ical/search/jql/events.ics REST endpoint can be blocked at the proxy layer (or disable "JIRA iCalendar Plugin"). This will break Team Calendar functionality that relies on this endpoint.

Resolution

Upgrade to Jira 8.15 once it is released.

Alternatively, for Jiras on 8.5.10 and above, update Jira iCalendar Plugin through the Universal Plugin Manager

Go to Admin > Manage Apps
Search for JIRA iCalendar Plugin (you may need to expand the list to All Apps)
Confirm there is an update to v1.8.2 or above
Click Update

relates to

JRASERVER-41041 The Calendar Gadget can cause high CPU and memory usage for large filters

Closed

HOT-93048 Loading...

PSR-605 Loading...

causes: ITPLAT-271 Loading...

is detailed by: HL-1888 Loading...

links to

PSR-540

Mentioned in

CSS Top Asks - Jira On Prem: JRASERVER-71803 | Adding Jira Issues Dates event types to Team Calendars is extremely non performant

mentioned in: Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...

(1 links to, 1 Mentioned in, 17 mentioned in)

Assignee:: Artur Gniadzik

Reporter:: Denise Unterwurzacher [Atlassian] (Inactive)

Votes:: 14 Vote for this issue

Watchers:: 39 Start watching this issue

Created:: 13/Nov/2020 7:19 PM

Updated:: 10/Oct/2024 4:49 AM

Resolved:: 02/Feb/2021 9:16 AM

Details

Description

Steps to Reproduce

Confluence and Jira

Jira only

Expected Results

Actual Results

Notes

Workaround

Resolution

Attachments

Issue Links

Forms

Activity

People

Dates