Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-49494

Teamcal causing the old gen to fill and GC to become ineffective (and CPU effective) for over 20 minutes

    XMLWordPrintable

Details

    Description

      NOTE: This bug report is for Confluence Server. Using Confluence Cloud? See the corresponding bug report.

      There were reports of EAC being unavailable. As per usual it was actually just slow/busy. Checking the GC logs showed the typical pattern we get associated with many outages although this time it actually recovered and started to free memory from the old generation -

      Looking at the Simon monitoring in Confluence we see a single request to .../calendar/mycalendar.action with a start time of 20:17:23 and an end time of 20:39:21. Allowing for the difference timezones between us and the server this correlates very closely with the GC "incident".

      The application logs for around the time are showing a lot of occurrences of calendar related problems such as -

      2013-11-14 20:14:42,906 WARN [http-9081-13] [extra.calendar3.rest.CalendarResource] getSubcalendarsInternal Unable to retrieve information of sub-calendar with ID d3fff381-f819-4ce2-9734-bc020c302d2f for user 2c9082c93f36441c013f364a80690298. Please ignore if the mycalendars page is appearing alright for the user.
 -- url: /rest/calendar-services/1.0/calendar/subcalendars.json | userName: mhodges | referer: https://extranet.atlassian.com/dashboard.action
com.atlassian.confluence.extra.calendar3.exception.CalendarException
        at com.atlassian.confluence.extra.calendar3.calendarstore.DefaultJiraAccessor$16.handle(DefaultJiraAccessor.java:1286)
        at com.atlassian.confluence.extra.calendar3.calendarstore.DefaultJiraAccessor$16.handle(DefaultJiraAccessor.java:1255)
        at com.atlassian.applinks.core.auth.oauth.OAuthApplinksResponseHandler.handle(OAuthApplinksResponseHandler.java:61)
        at com.atlassian.applinks.core.auth.ApplicationLinkRequestAdaptor$1.handle(ApplicationLinkRequestAdaptor.java:87)
        at com.atlassian.plugins.rest.module.jersey.JerseyRequest$2.handle(JerseyRequest.java:166)
        at com.atlassian.sal.core.net.HttpClientRequest.executeAndReturn(HttpClientRequest.java:336)
        at com.atlassian.plugins.rest.module.jersey.JerseyRequest.executeAndReturn(JerseyRequest.java:161)
        at com.atlassian.applinks.core.auth.ApplicationLinkRequestAdaptor.execute(ApplicationLinkRequestAdaptor.java:84)
        at com.atlassian.applinks.core.auth.oauth.ThreeLeggedOAuthRequest.execute(ThreeLeggedOAuthRequest.java:54)
        at com.atlassian.confluence.extra.calendar3.calendarstore.DefaultJiraAccessor.getIssuesReturnedByJql(DefaultJiraAccessor.java:1253)
        at com.atlassian.confluence.extra.calendar3.calendarstore.AbstractJiraSubCalendarDataStore.isJqlReturningTooManyIssues(AbstractJiraSubCalendarDataStore.java:953)
        at com.atlassian.confluence.extra.calendar3.calendarstore.AbstractJiraSubCalendarDataStore.getSubCalendarWarnings(AbstractJiraSubCalendarDataStore.java:903)
        at com.atlassian.confluence.extra.calendar3.calendarstore.AbstractJiraSubCalendarDataStore.getSubCalendarWarnings(AbstractJiraSubCalendarDataStore.java:70)
        at com.atlassian.confluence.extra.calendar3.calendarstore.CalendarDataStoreCachingDecorator.getSubCalendarWarnings(CalendarDataStoreCachingDecorator.java:339)
        at com.atlassian.confluence.extra.calendar3.calendarstore.DelegatingCalendarDataStore.getSubCalendarWarnings(DelegatingCalendarDataStore.java:172)
        at com.atlassian.confluence.extra.calendar3.DefaultCalendarManager.getSubCalendarWarnings(DefaultCalendarManager.java:191)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.getSubCalendarWarnings(CalendarResource.java:297)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.toExtendedSubCalendar(CalendarResource.java:260)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.toExtendedSubCalendar(CalendarResource.java:255)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.access$100(CalendarResource.java:95)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource$2.apply(CalendarResource.java:217)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource$2.apply(CalendarResource.java:213)
        at com.google.common.collect.Iterators$8.next(Iterators.java:782)
        at com.google.common.collect.Iterators$7.computeNext(Iterators.java:644)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
        at com.google.common.collect.Lists.newArrayList(Lists.java:138)
        at com.google.common.collect.Collections2$FilteredCollection.toArray(Collections2.java:228)
        at java.util.ArrayList.<init>(ArrayList.java:151)
        at com.google.common.collect.Lists.newArrayList(Lists.java:119)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.getSubcalendarsInternal(CalendarResource.java:208)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.getSubCalendarsResponse(CalendarResource.java:165)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.access$000(CalendarResource.java:95)
...

      2013-11-14 20:14:43,017 ERROR [http-9081-13] [extra.calendar3.calendarstore.JiraCalendarDataStore] getSubCalendarJql Unable to get details of search filter 18296
 -- url: /rest/calendar-services/1.0/calendar/subcalendars.json | userName: mhodges | referer: https://extranet.atlassian.com/dashboard.action
com.atlassian.confluence.extra.calendar3.calendarstore.JiraAccessor$JiraPreConditionUnmetException
        at com.atlassian.confluence.extra.calendar3.calendarstore.DefaultJiraAccessor$15.handle(DefaultJiraAccessor.java:1227)
        at com.atlassian.confluence.extra.calendar3.calendarstore.DefaultJiraAccessor$15.handle(DefaultJiraAccessor.java:1195)
        at com.atlassian.applinks.core.auth.oauth.OAuthApplinksResponseHandler.handle(OAuthApplinksResponseHandler.java:61)
        at com.atlassian.applinks.core.auth.ApplicationLinkRequestAdaptor$1.handle(ApplicationLinkRequestAdaptor.java:87)
        at com.atlassian.plugins.rest.module.jersey.JerseyRequest$2.handle(JerseyRequest.java:166)
        at com.atlassian.sal.core.net.HttpClientRequest.executeAndReturn(HttpClientRequest.java:336)
        at com.atlassian.plugins.rest.module.jersey.JerseyRequest.executeAndReturn(JerseyRequest.java:161)
        at com.atlassian.applinks.core.auth.ApplicationLinkRequestAdaptor.execute(ApplicationLinkRequestAdaptor.java:84)
        at com.atlassian.applinks.core.auth.oauth.ThreeLeggedOAuthRequest.execute(ThreeLeggedOAuthRequest.java:54)
        at com.atlassian.confluence.extra.calendar3.calendarstore.DefaultJiraAccessor.getSearchFilterJql(DefaultJiraAccessor.java:1195)
        at com.atlassian.confluence.extra.calendar3.calendarstore.AbstractJiraSubCalendarDataStore.getSubCalendarJql(AbstractJiraSubCalendarDataStore.java:968)
        at com.atlassian.confluence.extra.calendar3.calendarstore.AbstractJiraSubCalendarDataStore.getSubCalendarWarnings(AbstractJiraSubCalendarDataStore.java:903)
        at com.atlassian.confluence.extra.calendar3.calendarstore.AbstractJiraSubCalendarDataStore.getSubCalendarWarnings(AbstractJiraSubCalendarDataStore.java:70)
        at com.atlassian.confluence.extra.calendar3.calendarstore.CalendarDataStoreCachingDecorator.getSubCalendarWarnings(CalendarDataStoreCachingDecorator.java:339)
        at com.atlassian.confluence.extra.calendar3.calendarstore.DelegatingCalendarDataStore.getSubCalendarWarnings(DelegatingCalendarDataStore.java:172)
        at com.atlassian.confluence.extra.calendar3.calendarstore.InternalSubscriptionCalendarDataStore.getSubCalendarWarnings(InternalSubscriptionCalendarDataStore.java:663)
        at com.atlassian.confluence.extra.calendar3.calendarstore.InternalSubscriptionCalendarDataStore.getSubCalendarWarnings(InternalSubscriptionCalendarDataStore.java:50)
        at com.atlassian.confluence.extra.calendar3.calendarstore.DelegatingCalendarDataStore.getSubCalendarWarnings(DelegatingCalendarDataStore.java:172)
        at com.atlassian.confluence.extra.calendar3.DefaultCalendarManager.getSubCalendarWarnings(DefaultCalendarManager.java:191)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.getSubCalendarWarnings(CalendarResource.java:297)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.toExtendedSubCalendar(CalendarResource.java:260)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.toExtendedSubCalendar(CalendarResource.java:255)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.access$100(CalendarResource.java:95)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource$2.apply(CalendarResource.java:217)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource$2.apply(CalendarResource.java:213)
...

      2013-11-14 20:14:43,028 WARN [http-9081-13] [extra.calendar3.rest.CalendarResource] getSubcalendarsInternal Unable to retrieve information of sub-calendar with ID 05ae4bab-8a95-4efa-8b9b-c124bc8ee4af for user 2c9082c93f36441c013f364a80690298. Please ignore if the mycalendars page is appearing alright for the user.
 -- url: /rest/calendar-services/1.0/calendar/subcalendars.json | userName: mhodges | referer: https://extranet.atlassian.com/dashboard.action
java.lang.IllegalArgumentException: Arguments to addParameter(String, String) cannot be null
        at org.apache.commons.httpclient.methods.PostMethod.addParameter(PostMethod.java:268)
        at com.atlassian.sal.core.net.HttpClientRequest.processParameters(HttpClientRequest.java:614)
        at com.atlassian.sal.core.net.HttpClientRequest.executeAndReturn(HttpClientRequest.java:323)
        at com.atlassian.plugins.rest.module.jersey.JerseyRequest.executeAndReturn(JerseyRequest.java:161)
        at com.atlassian.applinks.core.auth.ApplicationLinkRequestAdaptor.execute(ApplicationLinkRequestAdaptor.java:84)
        at com.atlassian.applinks.core.auth.oauth.ThreeLeggedOAuthRequest.execute(ThreeLeggedOAuthRequest.java:54)
        at com.atlassian.confluence.extra.calendar3.calendarstore.DefaultJiraAccessor.getIssuesReturnedByJql(DefaultJiraAccessor.java:1253)
        at com.atlassian.confluence.extra.calendar3.calendarstore.AbstractJiraSubCalendarDataStore.isJqlReturningTooManyIssues(AbstractJiraSubCalendarDataStore.java:953)
        at com.atlassian.confluence.extra.calendar3.calendarstore.AbstractJiraSubCalendarDataStore.getSubCalendarWarnings(AbstractJiraSubCalendarDataStore.java:903)
        at com.atlassian.confluence.extra.calendar3.calendarstore.AbstractJiraSubCalendarDataStore.getSubCalendarWarnings(AbstractJiraSubCalendarDataStore.java:70)
        at com.atlassian.confluence.extra.calendar3.calendarstore.CalendarDataStoreCachingDecorator.getSubCalendarWarnings(CalendarDataStoreCachingDecorator.java:339)
        at com.atlassian.confluence.extra.calendar3.calendarstore.DelegatingCalendarDataStore.getSubCalendarWarnings(DelegatingCalendarDataStore.java:172)
        at com.atlassian.confluence.extra.calendar3.calendarstore.InternalSubscriptionCalendarDataStore.getSubCalendarWarnings(InternalSubscriptionCalendarDataStore.java:663)
        at com.atlassian.confluence.extra.calendar3.calendarstore.InternalSubscriptionCalendarDataStore.getSubCalendarWarnings(InternalSubscriptionCalendarDataStore.java:50)
        at com.atlassian.confluence.extra.calendar3.calendarstore.DelegatingCalendarDataStore.getSubCalendarWarnings(DelegatingCalendarDataStore.java:172)
        at com.atlassian.confluence.extra.calendar3.DefaultCalendarManager.getSubCalendarWarnings(DefaultCalendarManager.java:191)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.getSubCalendarWarnings(CalendarResource.java:297)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.toExtendedSubCalendar(CalendarResource.java:260)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.toExtendedSubCalendar(CalendarResource.java:255)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.access$100(CalendarResource.java:95)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource$2.apply(CalendarResource.java:217)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource$2.apply(CalendarResource.java:213)
        at com.google.common.collect.Iterators$8.next(Iterators.java:782)
...

      2013-11-14 20:14:43,310 WARN [http-9081-13] [extra.calendar3.rest.CalendarResource] getSubcalendarsInternal Unable to retrieve information of sub-calendar with ID 54525f20-de74-472e-a5c0-cd30e13b422f for user 2c9082c93f36441c013f364a80690298. Please ignore if the mycalendars page is appearing alright for the user.
 -- url: /rest/calendar-services/1.0/calendar/subcalendars.json | userName: mhodges | referer: https://extranet.atlassian.com/dashboard.action
com.atlassian.confluence.extra.calendar3.exception.CalendarException
        at com.atlassian.confluence.extra.calendar3.calendarstore.DefaultJiraAccessor$16.handle(DefaultJiraAccessor.java:1286)
        at com.atlassian.confluence.extra.calendar3.calendarstore.DefaultJiraAccessor$16.handle(DefaultJiraAccessor.java:1255)
        at com.atlassian.applinks.core.auth.oauth.OAuthApplinksResponseHandler.handle(OAuthApplinksResponseHandler.java:61)
        at com.atlassian.applinks.core.auth.ApplicationLinkRequestAdaptor$1.handle(ApplicationLinkRequestAdaptor.java:87)
        at com.atlassian.plugins.rest.module.jersey.JerseyRequest$2.handle(JerseyRequest.java:166)
        at com.atlassian.sal.core.net.HttpClientRequest.executeAndReturn(HttpClientRequest.java:336)
        at com.atlassian.plugins.rest.module.jersey.JerseyRequest.executeAndReturn(JerseyRequest.java:161)
        at com.atlassian.applinks.core.auth.ApplicationLinkRequestAdaptor.execute(ApplicationLinkRequestAdaptor.java:84)
        at com.atlassian.applinks.core.auth.oauth.ThreeLeggedOAuthRequest.execute(ThreeLeggedOAuthRequest.java:54)
        at com.atlassian.confluence.extra.calendar3.calendarstore.DefaultJiraAccessor.getIssuesReturnedByJql(DefaultJiraAccessor.java:1253)
        at com.atlassian.confluence.extra.calendar3.calendarstore.AbstractJiraSubCalendarDataStore.isJqlReturningTooManyIssues(AbstractJiraSubCalendarDataStore.java:953)
        at com.atlassian.confluence.extra.calendar3.calendarstore.AbstractJiraSubCalendarDataStore.getSubCalendarWarnings(AbstractJiraSubCalendarDataStore.java:903)
        at com.atlassian.confluence.extra.calendar3.calendarstore.AbstractJiraSubCalendarDataStore.getSubCalendarWarnings(AbstractJiraSubCalendarDataStore.java:70)
        at com.atlassian.confluence.extra.calendar3.calendarstore.CalendarDataStoreCachingDecorator.getSubCalendarWarnings(CalendarDataStoreCachingDecorator.java:339)
        at com.atlassian.confluence.extra.calendar3.calendarstore.DelegatingCalendarDataStore.getSubCalendarWarnings(DelegatingCalendarDataStore.java:172)
        at com.atlassian.confluence.extra.calendar3.DefaultCalendarManager.getSubCalendarWarnings(DefaultCalendarManager.java:191)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.getSubCalendarWarnings(CalendarResource.java:297)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.toExtendedSubCalendar(CalendarResource.java:260)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.toExtendedSubCalendar(CalendarResource.java:255)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource.access$100(CalendarResource.java:95)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource$2.apply(CalendarResource.java:217)
        at com.atlassian.confluence.extra.calendar3.rest.CalendarResource$2.apply(CalendarResource.java:213)
...

      These don't appear to stem from the .../calendar/mycalendar.action so may not actually be relevant to the GC "event" we are worried about here.

      Attachments

        1. team-cal-gc.png
          team-cal-gc.png
          235 kB
        2. TeamCal-wow.png
          TeamCal-wow.png
          136 kB

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. CONFCLOUD-49494 Teamcal causing the old gen to fill and GC to become ineffective (and CPU effective) for over 20 minutes

          • High - High priority issues
          • Closed
          mentioned in

          Page Loading...

          Activity

            People

              merickson Matthew Erickson
              pcurren Paul Curren
              Votes:
              4 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: