Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-13004

Attachment directory needs to be hashed to avoid filesystem limitations

XMLWordPrintable

    • We collect Confluence feedback from various sources, and we evaluate what we've collected when planning our product roadmap. To understand how this piece of feedback will be reviewed, see our Implementation of New Features Policy.

      Confluence stores all attachments in a as subdirectories the data/attachments folder, the name of the subdirectory is the page id. This is bad for the following reasons:

      • On linux few filesystems can handle directories with more than 2^15 sub directories. Once that limit is reached, new directories will fail to be created. The limit for ext3, the default file system for common linux distros is 31998 directory entries. More esoteric filesystems may raise this limit (please, not filesystem wars, ext2/3 is the most common filesystem) but it is still not a practice to be emulated.
      • Traversing a directory this size slows file access and most tools are not designed to handle directories that large

      This will become an issue in confluence hosted very soon.

      The attachment directory should be hashed, similar to the way squid hashes its cache directory.

            [CONFSERVER-13004] Attachment directory needs to be hashed to avoid filesystem limitations

            Michael Gettes added a comment -

            I'd appreciate it if someone could get a look at https://support.atlassian.com/browse/CSP-32954 - we are having trouble upgrading to 3.0 as the problem report indicates and it is due to the attachment re-organization.

            thanks

            Michael Gettes added a comment - I'd appreciate it if someone could get a look at https://support.atlassian.com/browse/CSP-32954 - we are having trouble upgrading to 3.0 as the problem report indicates and it is due to the attachment re-organization. thanks

            Mark Hrynczak (Inactive) added a comment -

            Tested that attachments are working as expected after upgrading from 2.10.2 to 3.0 trunk.

            Further testing will be done as part of the 3.0 cycle upgrade testing.

            Mark Hrynczak (Inactive) added a comment - Tested that attachments are working as expected after upgrading from 2.10.2 to 3.0 trunk. Further testing will be done as part of the 3.0 cycle upgrade testing.

            Paul Curren added a comment -

            No review comments requiring modification

            Paul Curren added a comment - No review comments requiring modification

            Per Fragemann [Atlassian] added a comment -

            Andrew, will you review this? If so, please try to do it this week - the number of open reviews is way too high at the moment. thanks,per

            Per Fragemann [Atlassian] added a comment - Andrew, will you review this? If so, please try to do it this week - the number of open reviews is way too high at the moment. thanks,per

            Paul Curren added a comment -

            Turns out that some time back in 2003 Confluence used to have an id generation scheme that could create really small id's like 10.

            I wasn't expecting that and more work will be required.

            Paul Curren added a comment - Turns out that some time back in 2003 Confluence used to have an id generation scheme that could create really small id's like 10. I wasn't expecting that and more work will be required.

            Paul Curren added a comment -

            See the documentation at http://confluence.atlassian.com/display/DOC/Hierarchical+File+System+Attachment+Storage for more details.

            Paul Curren added a comment - See the documentation at http://confluence.atlassian.com/display/DOC/Hierarchical+File+System+Attachment+Storage for more details.

            Paul Curren added a comment -

            Added the documentation issue CONF-14254

            Paul Curren added a comment - Added the documentation issue CONF-14254

            Paul Curren added a comment -

            This fix supercedes CONF-8298 and will be automatically applied to customers that upgrade. For Confluence 3.0 users the upgrade will be automatic regardless of whether the patch for CONF-8298 was previously applied or not. New users of Confluence will automatically get the new and improved attachments directory structure.

            Paul Curren added a comment - This fix supercedes CONF-8298 and will be automatically applied to customers that upgrade. For Confluence 3.0 users the upgrade will be automatic regardless of whether the patch for CONF-8298 was previously applied or not. New users of Confluence will automatically get the new and improved attachments directory structure.

            Paul Curren added a comment -

            Assigned back to me. Turns out I will be working on this improvement within the immediate future

            Paul Curren added a comment - Assigned back to me. Turns out I will be working on this improvement within the immediate future

            jens added a comment -

            We should keep in mind that we will require some solution soon. If we can't go all the way now, maybe we will need to find a middle ground that we can extend on later.

            Thoughts?

            jens added a comment - We should keep in mind that we will require some solution soon. If we can't go all the way now, maybe we will need to find a middle ground that we can extend on later. Thoughts?

              pcurren Paul Curren
              dcheney David Cheney (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: