Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-87298

Soft and Hard Retention cleanup jobs take long time to run

      Issue Summary

      This is reproducible on Data Center: yes

      Steps to Reproduce

      1. Create an instance with many content items with many versions (multiple spaces)
      2. Set a retention rule on a single space and run the job

      Expected Results

      The rules applied and the retention job cleans up content every 10 minutes, efficiently clearing all content that applies to the rules

      Actual Results

      The job can run upwards of 30 minutes per cycle, causing the cleanup to to take a long time. Note the time difference

      2023-05-04 13:06:22,683 DEBUG [Caesium-1-4] [confluence.impl.retention.DefaultHistoricalVersionService] find Finding Historical ATTACHMENT Content for content IDs starting with 73652368, result limit [3000]
      2023-05-04 13:35:58,861 DEBUG [Caesium-1-4] [impl.retention.manager.DefaultSpaceRetentionPolicyManager] getPolicy No SpaceRetentionPolicies found for space with key TEST

      This is because the find operation scans the entire content table. This is run twice (for page and attachments) for each job run

      Workaround

      Currently, there is no known workaround for this behaviour. A workaround will be added here when available.

      Atlassian Update - 16th January 2024

      Hi everyone,

      Thank you for sharing your interest and feedback on this issue.

      We are aware of some scalability limitations in the way the retention rules are currently operating. Resolving these issues is on our backlog and is something we plan on addressing later this calendar year, following our current focus on security.

      Please continue to vote on and watch this ticket to share the importance of this bug with us and to receive updates on our progress.

      To learn more about our recent investments in Confluence Data Center, please check our public roadmap and our dashboards containing recently resolved issues, and current work and future plans.

      Kind regards,
      Confluence Data Center

            [CONFSERVER-87298] Soft and Hard Retention cleanup jobs take long time to run

            Hi cba81dde1a75 ,

            We do have plan to backport this fix to previous LTS. The current fix is still on soaking for 9.1. We will create backport tickets after that. cc ephillips@atlassian.com 

            Jeffery Xie added a comment - Hi cba81dde1a75 , We do have plan to backport this fix to previous LTS. The current fix is still on soaking for 9.1. We will create backport tickets after that. cc  ephillips@atlassian.com  

            will this fix be applied to the 8.5 (LTS) version?

            kathy reidinger added a comment - will this fix be applied to the 8.5 (LTS) version?

            A fix for this issue is available in Confluence Data Center 9.1.0.
            Upgrade now or check out the Release Notes to see what other issues are resolved.

            Jordan Anslow added a comment - A fix for this issue is available in Confluence Data Center 9.1.0. Upgrade now or check out the Release Notes to see what other issues are resolved.

            We are also affected, not just by it running for long time initial few runs, but sadly(different bug) ours also does encounter errors before finishing the procedure.
            Effectively not fully cleaning our instance after being enabled for month/s.

            Working approach to avoid new job run while old is ongoing is to adjust the schedule to off work hours 1-4 times a day for example. Granted it does not solve how long it's running for.

            We have ongoing ticket with ATL for both issues. But mainly the aspect of not fully cleaning revisions/etc.

            Tomas Karas added a comment - We are also affected, not just by it running for long time initial few runs, but sadly(different bug) ours also does encounter errors before finishing the procedure. Effectively not fully cleaning our instance after being enabled for month/s. Working approach to avoid new job run while old is ongoing is to adjust the schedule to off work hours 1-4 times a day for example. Granted it does not solve how long it's running for. We have ongoing ticket with ATL for both issues. But mainly the aspect of not fully cleaning revisions/etc.

            We are affected by this.

            Right now, the implementation of this feature is highly requested by us.

            We have a really large instance and expected from this feature to solve a lot of our issues.

            With the current state, it is unusable.

            Marie Endres added a comment - We are affected by this. Right now, the implementation of this feature is highly requested by us. We have a really large instance and expected from this feature to solve a lot of our issues. With the current state, it is unusable.

            Atlassian Update - 16th January 2024

            Hi everyone,

            Thank you for sharing your interest and feedback on this issue.

            We are aware of some scalability limitations in the way the retention rules are currently operating. Resolving these issues is on our backlog and is something we plan on addressing later this calendar year, following our current focus on security.

            Please continue to vote on and watch this ticket to share the importance of this bug with us and to receive updates on our progress.

            To learn more about our recent investments in Confluence Data Center, please check our public roadmap and our dashboards containing recently resolved issues, and current work and future plans.

            Kind regards,
            Confluence Data Center

            Charlie Marriott added a comment - Atlassian Update - 16th January 2024 Hi everyone, Thank you for sharing your interest and feedback on this issue. We are aware of some scalability limitations in the way the retention rules are currently operating. Resolving these issues is on our backlog and is something we plan on addressing later this calendar year, following our current focus on security. Please continue to vote on and watch this ticket to share the importance of this bug with us and to receive updates on our progress. To learn more about our recent investments in Confluence Data Center, please check our public roadmap and our dashboards containing recently resolved issues , and current work and future plans . Kind regards, Confluence Data Center

            Since a half year it is in "Gathering Impact" status. This nice feature is especially for large instances, but it is not properly working on large instances. Any status update here?

            Sinan Yildirim added a comment - Since a half year it is in "Gathering Impact" status. This nice feature is especially for large instances, but it is not properly working on large instances. Any status update here?

            Hi, we are also affected by this error. We have an instance with around 700,000 pages and the retention rules would be a valuable function for us to shrink our instance. Unfortunately, the current implementation is completely useless for us because even with retention rules for just one medium-sized space, nothing gets deleted for weeks.

            Atlassian, please make the function usable for large instances because that is where it is needed most!

            Marco Leist added a comment - Hi, we are also affected by this error. We have an instance with around 700,000 pages and the retention rules would be a valuable function for us to shrink our instance. Unfortunately, the current implementation is completely useless for us because even with retention rules for just one medium-sized space, nothing gets deleted for weeks. Atlassian, please make the function usable for large instances because that is where it is needed most!

            is this issue resolved in confluence 8.5.4 ?

             

            Arjun Hazari added a comment - is this issue resolved in confluence 8.5.4 ?  

            Alexander added a comment - - edited

            Hi, we're also affected by this bug. We recently updated to 7.19 and were especially excited to use the retention rules to shrinken our big instance a little bit, but now this feature is useless to us, because it doesn't work!

            Atlassian, please fix.

            Alexander added a comment - - edited Hi, we're also affected by this bug. We recently updated to 7.19 and were especially excited to use the retention rules to shrinken our big instance a little bit, but now this feature is useless to us, because it doesn't work! Atlassian, please fix.

              5339cdd01cf4 Jeffery Xie
              ephillips@atlassian.com Edward
              Affected customers:
              45 This affects my team
              Watchers:
              61 Start watching this issue

                Created:
                Updated:
                Resolved: