Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-19873

attachment directory structure has exceeded maximum file handles

XMLWordPrintable

    • We collect Jira feedback from various sources, and we evaluate what we've collected when planning our product roadmap. To understand how this piece of feedback will be reviewed, see our Implementation of New Features Policy.

      NOTE: This suggestion is for JIRA Server. Using JIRA Cloud? See the corresponding suggestion.

      One of our projects now has over 32000 issues that each have attachments.
      When a user went to give the 32,001 issue an attachment, the file system refused it because it has a maximum of 32000 file handles per directory.
      The problem is that each issue gets its own sub-directory underneath the project key in the attachment path.
      There needs to be some sort of arbitrary middle directory between the project key and the issue directory, similar to how photos are stored on SIM cards. Maybe group issues in groups by the fourth power of ten?

      For example, instead of

      /usr/local/jira/attachments/JRA/JRA-123456

      have

      /usr/local/jira/attachments/JRA/120000/JRA-123456

      For now we've moved the 1,000 oldest attachments to another directory, effectively deleting them from the Jira web site.
      So we have a couple weeks to come up with some patch for this or we keep effectively archiving or deleting old attachments.

      I understand from our OS folks that it would be possible to recompile the kernel to have a higher maximum file handle setting. However, my management wants to pursue an application solution rather than risk the OS recompilation solution.

      Here is the error reported in the logs: 

      2009-11-23 10:15:37,371 TP-Processor94 ERROR [jira.issue.fields.AttachmentSystemField] Error occurred while creating attachment.
com.atlassian.jira.web.util.AttachmentException: Cannot write to attachment directory. Check that the application server and JIRA have permissions to write to: /usr/local/jira/attachments/NOV/NOV-83753
        at com.atlassian.jira.util.AttachmentUtils.checkValidAttachmentDirectory(AttachmentUtils.java:109)
        at com.atlassian.jira.issue.managers.DefaultAttachmentManager.createAttachment(DefaultAttachmentManager.java:277)
        at com.atlassian.jira.issue.managers.DefaultAttachmentManager.createAttachment(DefaultAttachmentManager.java:294)
        at com.atlassian.jira.issue.fields.AttachmentSystemField.addAttachment(AttachmentSystemField.java:162)
        at com.atlassian.jira.issue.fields.AttachmentSystemField.createValue(AttachmentSystemField.java:141)
        at com.atlassian.jira.workflow.function.issue.IssueCreateFunction.execute(IssueCreateFunction.java:85)
        at com.opensymphony.workflow.AbstractWorkflow.executeFunction(AbstractWorkflow.java:869)
        at com.opensymphony.workflow.AbstractWorkflow.transitionWorkflow(AbstractWorkflow.java:1265)
        at com.opensymphony.workflow.AbstractWorkflow.initialize(AbstractWorkflow.java:618)
        at com.atlassian.jira.workflow.SimpleWorkflowManager.createIssue(SimpleWorkflowManager.java:217)
        at com.atlassian.jira.issue.managers.DefaultIssueManager.createIssue(DefaultIssueManager.java:378)
        at com.atlassian.jira.issue.managers.DefaultIssueManager.createIssue(DefaultIssueManager.java:326)
        at sun.reflect.GeneratedMethodAccessor999.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:71)
        at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28)
        at $Proxy3.createIssue(Unknown Source)
        at com.atlassian.jira.web.action.issue.CreateIssueDetails.createIssue(CreateIssueDetails.java:141)
        at com.atlassian.jira.web.action.issue.CreateIssueDetails.doExecute(CreateIssueDetails.java:110)
        at webwork.action.ActionSupport.execute(ActionSupport.java:153)
        at com.atlassian.jira.action.JiraActionSupport.execute(JiraActionSupport.java:54)
        at webwork.dispatcher.GenericDispatcher.executeAction(GenericDispatcher.java:132)
        at com.atlassian.jira.web.dispatcher.JiraServletDispatcher.service(JiraServletDispatcher.java:178)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.atlassian.jira.web.filters.AccessLogFilter.doFilter(AccessLogFilter.java:73)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:119)
        at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:55)
        at com.atlassian.jira.web.filters.SitemeshExcludePathFilter.doFilter(SitemeshExcludePathFilter.java:38)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:192)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.atlassian.seraph.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:120)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:125)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.atlassian.util.profiling.filters.ProfilingFilter.doFilter(ProfilingFilter.java:132)

        1. attachment-patch-binary.tar
          70 kB
        2. attachment-patch-source.tar
          90 kB

            [JRASERVER-19873] attachment directory structure has exceeded maximum file handles

            Robin Reidl (Inactive) added a comment -

            fmotglongorder We added a new subdirectory to the attachments directory that allows more subdirectories to be created in the file system. In 7.x, attachments are stored in $JIRA_Home/data/jira/data/attachments/PROJECT_KEY/x0000/ISSUE_KEY/ID, where x0000 is a subdirectory that is created and named for each 10,000 issues. For example, attachments for issues 1-10,000 are stored in /PROJECT_KEY/10000/ISSUE_KEY/ID; issues 10001-20,000 are stored in 20000, etc.

            • If you migrate to 7.x, your existing attachments stay in the original location of $JIRA_Home/data/jira/data/attachments/PROJECT_KEY/ISSUE_KEY/ID unless:
              someone views a migrated issue and clicks an attachment thumbnail. JIRA then copies the attachment to the 7.x directory structure.
            • someone adds a new attachment to a migrated issue. The new attachment is stored in the 7.x directory structure.

            This blog explains in more detail.

            Robin Reidl (Inactive) added a comment - fmotglongorder We added a new subdirectory to the attachments directory that allows more subdirectories to be created in the file system. In 7.x, attachments are stored in $JIRA_Home/data/jira/data/attachments/PROJECT_KEY/x0000/ISSUE_KEY/ID, where x0000 is a subdirectory that is created and named for each 10,000 issues. For example, attachments for issues 1-10,000 are stored in /PROJECT_KEY/10000/ISSUE_KEY/ID; issues 10001-20,000 are stored in 20000, etc. If you migrate to 7.x, your existing attachments stay in the original location of $JIRA_Home/data/jira/data/attachments/PROJECT_KEY/ISSUE_KEY/ID unless: someone views a migrated issue and clicks an attachment thumbnail. JIRA then copies the attachment to the 7.x directory structure. someone adds a new attachment to a migrated issue. The new attachment is stored in the 7.x directory structure. This blog  explains in more detail.

            BBC Application Admin Team added a comment -

            This is marked as resolved in v7.0.0 but there isn't any detail of how it's been resolved. If we're migrating from v5.x to v7.x can we safely assume that the issue now goes away?

            BBC Application Admin Team added a comment - This is marked as resolved in v7.0.0 but there isn't any detail of how it's been resolved. If we're migrating from v5.x to v7.x can we safely assume that the issue now goes away?

            Roy Krishna (Inactive) added a comment -

            Thanks all. This is on our backlog, but we're not committing to have this in 6.3 at this stage.

            We will let you know of any progress via this issue.

            Roy Krishna (Inactive) added a comment - Thanks all. This is on our backlog, but we're not committing to have this in 6.3 at this stage. We will let you know of any progress via this issue.

            Jozsef Kozell added a comment -

            In the 6.1.x version, the attachment handling is pretty unified (in 5.x, you had to workaround in 4 files), so the change itself would be relatively easy:

            /jira-project/jira-components/jira-core/src/main/java/com/atlassian/jira/issue/attachment/DefaultAttachmentStore.java:

            New method
            public static String getIssuePathPart(String issueKey) {
	String number = String.format("%07d", IssueKey.from(issueKey).getIssueNumber());
	int len = number.length();
	String MMM = number.substring(len - 6, len - 3);
	String UUUU = number.substring(0, len - 6);

	return UUUU + File.separator + MMM;
}
            getAttachmentDirectory(final String attachmentDirectory, final String projectKey, final String issueKey)
            	// Replace the "return" in the else block.
	return new File(projectDirectory + File.separator + getIssuePathPart(issueKey), computeIssueKeyForOriginalProjectKey( originalProjectKey, issueKey));

            Build, deploy.

            This will split up the directory tree to attachment_root/U/MMM/<PKEY>-<NUMBER>/<attachment_id>_<filename>, this way we will have 2 additional directory level, and at the lowest level we store maximum 1000 issue per directory.
            If the NUMBER is smaller than 1000 the issuePathPart will be 0/000.

            Warning: I haven't checked, that this rearranges or not the existing attachments!

            How does it scale? We have ~1.5M attachments in ~1.3M issues.

            Jozsef Kozell added a comment - In the 6.1.x version, the attachment handling is pretty unified (in 5.x, you had to workaround in 4 files), so the change itself would be relatively easy: /jira-project/jira-components/jira-core/src/main/java/com/atlassian/jira/issue/attachment/DefaultAttachmentStore.java: New method public static String getIssuePathPart( String issueKey) { String number = String .format( "%07d" , IssueKey.from(issueKey).getIssueNumber()); int len = number.length(); String MMM = number.substring(len - 6, len - 3); String UUUU = number.substring(0, len - 6); return UUUU + File.separator + MMM; } getAttachmentDirectory(final String attachmentDirectory, final String projectKey, final String issueKey) // Replace the " return " in the else block. return new File(projectDirectory + File.separator + getIssuePathPart(issueKey), computeIssueKeyForOriginalProjectKey( originalProjectKey, issueKey)); Build, deploy. This will split up the directory tree to attachment_root/U/MMM/<PKEY>-<NUMBER>/<attachment_id>_<filename>, this way we will have 2 additional directory level, and at the lowest level we store maximum 1000 issue per directory. If the NUMBER is smaller than 1000 the issuePathPart will be 0/000. Warning: I haven't checked, that this rearranges or not the existing attachments! How does it scale? We have ~1.5M attachments in ~1.3M issues.

            intersol_old added a comment -

            I hope that Atlassian will realize the importance of problem and solve this on the fast-track, clearly this must be implemented in 6.2 in order to support clustering. It would not make any sense to have a different file-structure for clustered and non-clustered versions.

            In fact Atlassian already have experience doing this on Confluence 3.x when they did the same. I just hope they will perform OS moves instead of trying to copy files as we have terabytes of attachments.

            intersol_old added a comment - I hope that Atlassian will realize the importance of problem and solve this on the fast-track, clearly this must be implemented in 6.2 in order to support clustering. It would not make any sense to have a different file-structure for clustered and non-clustered versions. In fact Atlassian already have experience doing this on Confluence 3.x when they did the same. I just hope they will perform OS moves instead of trying to copy files as we have terabytes of attachments.

            Jason Brison added a comment - - edited

            @matt.doar - I just changed the URL to access the page ID and was directed to JIRA Blue Label Pioneer Program.

            Jason Brison added a comment - - edited @ matt.doar - I just changed the URL to access the page ID and was directed to JIRA Blue Label Pioneer Program .

            MattS added a comment -

            Sorin, that link failed for me, perhaps because there are two comments with the same content? Anyway, I can imagine other schemes being used for clustered JIRA but you're right, not the current one unchanged

            MattS added a comment - Sorin, that link failed for me, perhaps because there are two comments with the same content? Anyway, I can imagine other schemes being used for clustered JIRA but you're right, not the current one unchanged

            intersol_old added a comment -

            This bug priority should be reconsidered as it will be blocker for implementing a clustered JIRA - https://confluence.atlassian.com/pages/doaddcomment.action?pageId=441745954

            intersol_old added a comment - This bug priority should be reconsidered as it will be blocker for implementing a clustered JIRA - https://confluence.atlassian.com/pages/doaddcomment.action?pageId=441745954

            Jozsef Kozell added a comment -

            Look, what I just found: http://pastebin.com/YKaQBdnJ
            It is for 6.1.6, seems they(Atlassian) unified the attachment storage handling, so there is only one place to change things.
            If you backport it to 6.1, be sure to check that BulkMoveOperationImpl.java and MoveIssueConfirm.java are using attachmentStorage() or not.
            In 5.*, there is no AttachmentManager and AttachmentStorage, you should fix things in AttachmentUtils.

            Happy hacking.

            Jozsef Kozell added a comment - Look, what I just found: http://pastebin.com/YKaQBdnJ It is for 6.1.6, seems they(Atlassian) unified the attachment storage handling, so there is only one place to change things. If you backport it to 6.1, be sure to check that BulkMoveOperationImpl.java and MoveIssueConfirm.java are using attachmentStorage() or not. In 5.*, there is no AttachmentManager and AttachmentStorage, you should fix things in AttachmentUtils. Happy hacking.

            Jozsef Kozell added a comment -

            I'll do what I can do

            Jozsef Kozell added a comment - I'll do what I can do

              ldurkan Skywalker
              adc6ee404f6d Jeff Kirby
              Votes:
              35 Vote for this issue
              Watchers:
              40 Start watching this issue

                Created:
                Updated:
                Resolved: