Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-55506

Deleting repeating Events in Team Calendar causes duplicates in database and java.lang.OutOfMemoryError

      Summary

      Deleting Events in Team Calendar that are part of a repeating Event cause duplicates in the database, which causes performance issues and java.lang.OutOfMemoryError when viewing Team Calendars as the duplicates are loaded back into memory when re-loading the Calendar.

      Environment

      • Confluence 6.9.0
      • Team Calendar 6.0.0 or 5.5.4

      Steps to Reproduce

      This is quite tricky to reproduce with clear, concise steps. It may differ but in general

      1. In Team Calendars, create an all day Event that repeats every day and never ends
      2. Delete one event, at least a week after the first one
      3. Check the Team Calendar Exclusion table for duplicates (this is for Postgres, adjust as necessary)
        select "EXCLUSION", count(*) from "AO_950DC3_TC_EVENTS_EXCL" group by "EXCLUSION";
        

        There should be one event listed, the deleted one

      1. Click on a date before the deleted event, and select All Future Events
      2. Check the Team Calendar Exclusion table for all duplicates (this is for Postgres, adjust as necessary)
        select "EXCLUSION", count(*) from "AO_950DC3_TC_EVENTS_EXCL" group by "EXCLUSION";
        

        The Events are not removed.

      Diagnosis

      Run the following query to see if you may be affected by this issue (this is for Postgres, adjust as necessary):

      select "ALL_DAY", "EVENT_ID", "EXCLUSION", count(*) from "AO_950DC3_TC_EVENTS_EXCL" group by "ALL_DAY", "EVENT_ID", "EXCLUSION" having count(*) > 1;

      If there are rows returned, it's possible you are experiencing this issue, especially if the count for any row is high.

      Expected Results

      There should be only be two rows in AO_950DC3_TC_EVENTS_EXCL and the Events should be removed.

      Actual Results

      The below exception is thrown in the atlassian-confluence.log file:

      atlassian-confluence.log
      2018-05-09 16:55:31,339 ERROR [http-nio-8090-exec-7] [common.error.jersey.ThrowableExceptionMapper] toResponse Uncaught exception thrown by REST service: Java heap space
       -- referer: http://confluence.mycompany.com:8090/calendar/mycalendar.action | url: /rest/calendar-services/1.0/calendar/events.json | traceId: b85eaf70252053ab | userName: admin
      java.lang.OutOfMemoryError: Java heap space
      

      Resolution

      Upgrade to Team Calendars 6.0.12 which contains a fix for this issue.

      Notes

      Reviewing the heap dump from the java.lang.OutOfMemory error we can see that there are a few very large threads loaded with duplicates of the data, for example

      object cause size
      com.mysql.jdbc.JDBC42ResultSet AO_950DC3_TC_EVENTS_EXCL 946M
      http-nio-8090-exec-2 /rest/calendar-services/1.0/calendar/events.json 716M

      The com.mysql.jdbc.JDBC42ResultSet is pulling all the data from AO_950DC3_TC_EVENTS_EXCL. http-nio-8090-exec-2 is the associated user thread.

      The URL in the HAR fill shows it as

      And this returns all the data, or generates the java.lang.OutOfMemoryError.

      The error is caused by this code

      com.atlassian.confluence.extra.calendar3.DefaultCalendarManager#excludeEvent
                  excludeRecurrence(
                          baseEvent,
                          !(baseEvent.getStartDate().getDate() instanceof net.fortuna.ical4j.model.DateTime || baseEvent.getEndDate().getDate() instanceof net.fortuna.ical4j.model.DateTime)
                                  ? excludeDate.withZoneRetainFields(DateTimeZone.forID(srcSubCalendar.getTimeZoneId()))
                                  : excludeDate.withZone(DateTimeZone.forID(srcSubCalendar.getTimeZoneId()))
                  );
      
                  calendarDataStore.updateEvent(srcSubCalendar, baseEvent);
      

      Workaround

      The only option is to delete the duplicates in the AO_950DC3_TC_EVENTS_EXCL table and informing users to not delete All Future Events.

      IMPORTANT

      Always back up your data before performing any modifications to the database. If possible, test any alter, insert, update, or delete SQL commands on a staging server first.

      MySQL

      You can remove the duplicates by adding then removing this index

      MySQL 5.6 and below
      ALTER IGNORE TABLE AO_950DC3_TC_EVENTS_EXCL ADD UNIQUE INDEX AO_950DC3_TC_EVENTS_EXCL_TEMP_INDEX (ALL_DAY, EVENT_ID, EXCLUSION);
      ALTER TABLE AO_950DC3_TC_EVENTS_EXCL DROP INDEX AO_950DC3_TC_EVENTS_EXCL_TEMP_INDEX;
      
      MySQL 5.7 and above
      CREATE TABLE AO_950DC3_TC_EVENTS_EXCL_NEW LIKE AO_950DC3_TC_EVENTS_EXCL;
      ALTER TABLE AO_950DC3_TC_EVENTS_EXCL_NEW ADD UNIQUE INDEX (EVENT_ID);
      INSERT IGNORE INTO AO_950DC3_TC_EVENTS_EXCL_NEW SELECT * FROM AO_950DC3_TC_EVENTS_EXCL;
      DROP TABLE AO_950DC3_TC_EVENTS_EXCL;
      RENAME TABLE AO_950DC3_TC_EVENTS_EXCL_NEW TO AO_950DC3_TC_EVENTS_EXCL;
      

      PostgreSQL

      Delete the duplicates with (this may take some time)

      PostgreSQL
      delete from "AO_950DC3_TC_EVENTS_EXCL" where "ID" in (select "ID" from (select "ID", row_number() over (partition BY "EXCLUSION" order by "ID") from "AO_950DC3_TC_EVENTS_EXCL") excl_table WHERE excl_table.row_number > 1);
      

            [CONFSERVER-55506] Deleting repeating Events in Team Calendar causes duplicates in database and java.lang.OutOfMemoryError

            Hi GRPG04162,

            I tested this bug in TC 6.0.0 and confirmed the fix worked with TC 6.0.12. You may have had a different issue to what this bug was reporting.

            James.

            James Richards added a comment - Hi GRPG04162 , I tested this bug in TC 6.0.0 and confirmed the fix worked with TC 6.0.12. You may have had a different issue to what this bug was reporting. James.

            LAS added a comment -

            Was anyone able to replicate this bug and did the upgrade fix the issue? For us, we were only able to fix it by deleting one particular calendar which was corrupt and caused this heap space error whenever it was accessed.

             

            LAS added a comment - Was anyone able to replicate this bug and did the upgrade fix the issue? For us, we were only able to fix it by deleting one particular calendar which was corrupt and caused this heap space error whenever it was accessed.  

            I can see this bug mentioned on the latest release note - https://confluence.atlassian.com/teamcal/team-calendars-6-0-12-release-notes-951398756.html , but this bug status still showing as "Awaiting Release" , what it means ??? Is the latest version has the fix for this issue? or Still waiting for it?? We cannot afford one more outage again with this buggy product.

            Anoop K Baby added a comment - I can see this bug mentioned on the latest release note - https://confluence.atlassian.com/teamcal/team-calendars-6-0-12-release-notes-951398756.html , but this bug status still showing as "Awaiting Release" , what it means ??? Is the latest version has the fix for this issue? or Still waiting for it?? We cannot afford one more outage again with this buggy product.

            John Sibo added a comment -

            Atlassian just confirmed this bug is what has been bringing our system down for literally a day and a half straight. We run an enterprise instance and approximately every 30 minutes to 1 hour our CPU/memory would spike & kill our service - making Confluence completely unusable throughout the day. We have had a waste of 2 days because of this bug.. Please fix this ASAP. 

            Support issue # CSP-229219

            John Sibo added a comment - Atlassian just confirmed this bug is what has been bringing our system down for literally a day and a half straight. We run an enterprise instance and approximately every 30 minutes to 1 hour our CPU/memory would spike & kill our service - making Confluence completely unusable throughout the day. We have had a waste of 2 days because of this bug.. Please fix this ASAP.   Support issue # CSP-229219

            G added a comment -

            Thanks Anoop.  I cleared all plugin cache and tried the sql fix provided and i can still replicate the problem. Will deal with the Atlassian support ticket. 

             

            G added a comment - Thanks Anoop.  I cleared all plugin cache and tried the sql fix provided and i can still replicate the problem. Will deal with the Atlassian support ticket.   

            Gaj, please dont try downgrade, that made more worst on my situation. I had discussed it with Atlassian support team, as per them downgrade is not advisable, eventhough you try to downgrade there should be some reference on db tables for the upgrade. Otherwise you need to delete entire calendar plugin installation by running the following query (its mysql query) :

            delete from plugindata where pluginkey = 'com.atlassian.confluence.extra.team-calendars';

            But you will loose all your events datas. Anyway the workaround mentioned on this first thread, stabilize my Confluence Datacenter performance. So far no issues other than the delete "All Future Instance" option not working. But didn't notice an issue even if somebody click on that link yet.

            Anoop K Baby added a comment - Gaj, please dont try downgrade, that made more worst on my situation. I had discussed it with Atlassian support team, as per them downgrade is not advisable, eventhough you try to downgrade there should be some reference on db tables for the upgrade. Otherwise you need to delete entire calendar plugin installation by running the following query (its mysql query) : delete from plugindata where pluginkey = 'com.atlassian.confluence.extra.team-calendars'; But you will loose all your events datas. Anyway the workaround mentioned on this first thread, stabilize my Confluence Datacenter performance. So far no issues other than the delete "All Future Instance" option not working. But didn't notice an issue even if somebody click on that link yet.

            G added a comment -

            Can you confirm if it's safe to downgrade from 5.5.4 -> 5.3.20 ? 

            G added a comment - Can you confirm if it's safe to downgrade from 5.5.4 -> 5.3.20 ? 

            G added a comment - - edited

            CSP-228702 We had a day without crash with the Calendar disabled, then once the Calendar was enabled we just had another crash. 

            My other account is LAS.

            G added a comment - - edited CSP-228702 We had a day without crash with the Calendar disabled, then once the Calendar was enabled we just had another crash.  My other account is LAS.

            Hi chris_doughty,

            We've already got you logged in this ticket but thanks for the quick response.

            James (the same James as on your ticket ).

            James Richards added a comment - Hi chris_doughty , We've already got you logged in this ticket but thanks for the quick response. James (the same James as on your ticket ).

            Here's my support issue number: CSP-228342

            Chris Doughty added a comment - Here's my support issue number: CSP-228342

              dluong Duy Truong Luong
              jrichards@atlassian.com James Richards
              Affected customers:
              9 This affects my team
              Watchers:
              23 Start watching this issue

                Created:
                Updated:
                Resolved: