Details
-
Bug
-
Resolution: Fixed
-
Highest
-
8.5.0, 8.5.1, 8.13.0, 8.13.1
-
8.05
-
32
-
Severity 1 - Critical
-
280
-
Description
Issue Summary
Adding Jira Issues Dates event types to Team Calendars is extremely non performant, and can cause outages by consuming threads that can take hours to return. It causes high CPU on the processing nodes which can also result in performance impacts and system outages, as system resources cannot be used to respond to other requests.
This issue was recently resolved in Jira Cloud by upgrading the Fortuna ical library. The issue is caused by the library creating thousands of date objects per date, and checking each against a call to the system date, to determine timezones.
Steps to Reproduce
Confluence and Jira
- Create a custom field in JIRA of time Date/Time Picker.
- Set this field for a few issues (5 was enough for production instance).
- In Confluence open Calendars and add a new calendar
- Click '...' and Add event
- Select Jira Issue Dates, and the Jira project with the custom fields you set
- Select some date fields including the custom field you created and submit
- It will take a long time to add the events, or if the project is large they may never be added to the calendar
Jira only
- Create a custom field in JIRA of time Date/Time Picker.
- Set this field for a few issues (5 was enough for production instance).
- Call /rest/ical/1.0/ical/config/fields?jql=project=<project_key> Jira endpoint to find out the custom field id (customfield_XXXXX).
- Call /rest/ical/1.0/ical/search/jql/events.ics?jql=project+%3D+%22<project_key>%22&dateFieldName=<custom_field_id>&includeFixVersions=false&start=1598572800000&end=1609977600000&maxIssue=1000 endpoint to retrieve a list of events in iCal format (set Jira project key, custom field id, and start/stop timestamps in parameters). Events are generated based on the value of the custom field. So make sure that this date is within the timeframe set by start and end arguments.
- You will see a 20+ second delay even though the number of issues/events is small.
Expected Results
Requests to Jira REST endpoints return in a timely manner.
Actual Results
Threads are consumed and don't return for hours, and cause high CPU, which leads to performance degradation and outages.
There is no exception in the logfile as the threads are still processing. However, threads similar to those below will be visible in thread dumps:
http-nio-8080-exec-1 url:/rest/ical/1.0/...rch/jql/events.ics priority:5 - threadId:0x00007f44f1450000 - nativeId:0x47a8 - nativeId (decimal):18344 - state:RUNNABLE stackTrace: java.lang.Thread.State: RUNNABLE at java.util.GregorianCalendar.computeTime(GregorianCalendar.java:2818) at java.util.Calendar.updateTime(Calendar.java:3393) at java.util.Calendar.getTimeInMillis(Calendar.java:1782) at net.fortuna.ical4j.util.Dates.round(Dates.java:300) at net.fortuna.ical4j.model.Iso8601.<init>(Iso8601.java:74) at net.fortuna.ical4j.model.Time.<init>(Time.java:97) at net.fortuna.ical4j.model.Time.<init>(Time.java:88) at net.fortuna.ical4j.model.DateTime.<init>(DateTime.java:237) at net.fortuna.ical4j.util.Dates.getInstance(Dates.java:223) at net.fortuna.ical4j.model.Recur.getMonthDayVariants(Recur.java:935) at net.fortuna.ical4j.model.Recur.getCandidates(Recur.java:781) at net.fortuna.ical4j.model.Recur.getDates(Recur.java:620) at net.fortuna.ical4j.model.Recur.getDates(Recur.java:516) at net.fortuna.ical4j.model.component.Observance.getLatestOnset(Observance.java:242) at net.fortuna.ical4j.model.component.VTimeZone.getApplicableObservance(VTimeZone.java:313) at net.fortuna.ical4j.model.TimeZone.inDaylightTime(TimeZone.java:120) at java.util.TimeZone.getOffset(TimeZone.java:209) at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2342) at java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2312) at java.util.Calendar.setTimeInMillis(Calendar.java:1804) at net.fortuna.ical4j.util.Dates.round(Dates.java:287) at net.fortuna.ical4j.model.Iso8601.<init>(Iso8601.java:74) at net.fortuna.ical4j.model.Time.<init>(Time.java:116) at net.fortuna.ical4j.model.DateTime.setTimeZone(DateTime.java:408) at net.fortuna.ical4j.model.property.DateProperty.updateTimeZone(DateProperty.java:188) at net.fortuna.ical4j.model.property.DateProperty.setTimeZone(DateProperty.java:163) at com.atlassian.jira.extra.icalfeed.IcalendarResource.toVEvent(IcalendarResource.java:479) at com.atlassian.jira.extra.icalfeed.IcalendarResource.toIcalendar(IcalendarResource.java:364) at com.atlassian.jira.extra.icalfeed.IcalendarResource.search(IcalendarResource.java:323) at com.atlassian.jira.extra.icalfeed.IcalendarResource.getIcalendarByJql(IcalendarResource.java:273) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498)
http-nio-8080-exec-110 url:/rest/ical/1.0/...rch/jql/events.ics
java.util.Hashtable.get(Hashtable.java:363)
java.util.Properties.getProperty(Properties.java:969)
net.fortuna.ical4j.util.Configurator.getProperty(Configurator.java:75)
net.fortuna.ical4j.util.CompatibilityHints.isHintEnabled(CompatibilityHints.java:118)
net.fortuna.ical4j.model.Iso8601.<init>(Iso8601.java:78)
net.fortuna.ical4j.model.Time.<init>(Time.java:97)
net.fortuna.ical4j.model.Time.<init>(Time.java:88)
net.fortuna.ical4j.model.DateTime.<init>(DateTime.java:237)
net.fortuna.ical4j.util.Dates.getInstance(Dates.java:223)
net.fortuna.ical4j.model.Recur.getMonthDayVariants(Recur.java:935)
net.fortuna.ical4j.model.Recur.getCandidates(Recur.java:781)
net.fortuna.ical4j.model.Recur.getDates(Recur.java:620)
net.fortuna.ical4j.model.Recur.getDates(Recur.java:516)
net.fortuna.ical4j.model.component.Observance.getLatestOnset(Observance.java:242)
net.fortuna.ical4j.model.component.VTimeZone.getApplicableObservance(VTimeZone.java:313)
net.fortuna.ical4j.model.TimeZone.inDaylightTime(TimeZone.java:120)
java.util.TimeZone.getOffset(TimeZone.java:209)
java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2342)
java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2312)
java.util.Calendar.setTimeInMillis(Calendar.java:1804)
net.fortuna.ical4j.util.Dates.round(Dates.java:287)
net.fortuna.ical4j.model.Iso8601.<init>(Iso8601.java:74)
net.fortuna.ical4j.model.Time.<init>(Time.java:116)
net.fortuna.ical4j.model.DateTime.setTimeZone(DateTime.java:408)
net.fortuna.ical4j.model.property.DateProperty.updateTimeZone(DateProperty.java:188)
net.fortuna.ical4j.model.property.DateProperty.setTimeZone(DateProperty.java:163)
com.atlassian.jira.extra.icalfeed.IcalendarResource.toVEvent(IcalendarResource.java:479)
com.atlassian.jira.extra.icalfeed.IcalendarResource.toIcalendar(IcalendarResource.java:364)
com.atlassian.jira.extra.icalfeed.IcalendarResource.search(IcalendarResource.java:323)
com.atlassian.jira.extra.icalfeed.IcalendarResource.getIcalendarByJql(IcalendarResource.java:273)
sun.reflect.GeneratedMethodAccessor7727.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
Notes
Currently we are aware of couple of problems related to ical4j:
- High CPU load (current ticket -
JRASERVER-71803)- this is a contention due to frequent getProperty calls
- Thread being blocked/stuck trying to contact http://tzurl.org, this is not related to current ticket
- this is a contention in the network and there is a feature flag to prevent the call by disabling the TZ update definition (see https://ical4j.github.io/ical4j-user-guide/timezones/ for more details), to do that:
- Set the property "-Dnet.fortuna.ical4j.timezone.update.enabled=false" on Jira startup options.
- Restart Jira
Workaround
- Restarting the affected nodes kills the threads and will return the system to performing normally.
- * As an extreme measure, the /rest/ical/1.0/ical/search/jql/events.ics REST endpoint can be blocked at the proxy layer (or disable "JIRA iCalendar Plugin"). This will break Team Calendar functionality that relies on this endpoint.
Resolution
Upgrade to Jira 8.15 once it is released.
Alternatively, for Jiras on 8.5.10 and above, update Jira iCalendar Plugin through the Universal Plugin Manager
- Go to Admin > Manage Apps
- Search for JIRA iCalendar Plugin (you may need to expand the list to All Apps)
- Confirm there is an update to v1.8.2 or above
- Click Update
Attachments
Issue Links
- relates to
-
-
- Closed
-
-
HOT-93048 Loading...
-
- causes
-
- is detailed by
-
- links to
- Mentioned in
- mentioned in
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-