• Icon: Support Request Support Request
    • Resolution: Support Request
    • Icon: High High (View bug fix roadmap)
    • None
    • 5.2.4
    • None

      We have a customer with a large JIRA (1M issues) and every two weeks they have to restart JIRA because it is running out of file descriptors. We increased the system setting from 30K to 300K descriptors but it still happens eventually. We're using SSD for the JIRA home directory

      I've done some investigation and found that on the production JIRA using lsof and the counts on the Lucene issue index files increase every 5 minutes or so. On staging it's only once or twice per day. The rate is lower at weekends, so it is human-related. What happens is that the same 60 different index files have new file descriptors added, but the old file descriptors are not closed. There is also an error that appears at the same time in the JIRA log file:

      "Tried to reopen the IndexReader, but it threw AlreadyClosedException. Opening a fresh IndexReader."

      This comes from DefaultIndexEngine and is where a new Lucene IndexReader is created if the previous one is believed closed. I suspect that the previous IndexReader may not have been really closed.

      There is a comment right in that place in the source code. It's the "don't worry" that makes me think there is still a problem around here!

      // JRADEV-7825: Really this shouldn't happen unless someone closes the reader from outside all
      // the inscrutable code in this class (and its friends) but
      // don't worry, we will just open a new one in that case.

      This is similar symptoms but apparently a different cause than https://confluence.atlassian.com/display/JIRAKB/Loss+of+Functionality+due+to+Too+Many+Open+Files+Error

            [JRASERVER-38039] JIRA IndexReader is leaking file descriptors

            Monique Khairuliana (Inactive) made changes -
            Workflow Original: GreenHopper Kanban Workflow 20141014 - Restricted [ 2587475 ] New: Support Request Workflow [ 3435531 ]
            Status Original: Resolved [ 5 ] New: Closed [ 6 ]
            Ignat (Inactive) made changes -
            Workflow Original: JRA workflow with restrictions [ 1652283 ] New: GreenHopper Kanban Workflow 20141014 - Restricted [ 2587475 ]
            jonah (Inactive) made changes -
            Link New: This issue relates to JRACLOUD-38039 [ JRACLOUD-38039 ]
            Katherine Yabut made changes -
            Workflow Original: classic default workflow with restrictions [ 1543609 ] New: JRA workflow with restrictions [ 1652283 ]
            Confluence Escalation Bot (Inactive) made changes -
            Labels New: affects-server
            Owen made changes -
            Workflow Original: classic default workflow [ 683332 ] New: classic default workflow with restrictions [ 1543609 ]

            crf added a comment -

            andrew.morin1:

            Leaking Lucene indexes can be caused by any number of things. While there have been some versions of JIRA with bugs that triggered this, it is almost always caused by a plugin that is doing something wrong, such as running searches from its own thread pool without using JiraThreadLocalUtil to take care of the cleanup.

            Given that there are a wide variety of potential culprits and that it is usually related to a plugin that we do not ourselves test, there is no simple resolution for this issue. Customers that encounter it should open a support request so that we can work with you to track down what is causing it on your particular instance.

            crf added a comment - andrew.morin1 : Leaking Lucene indexes can be caused by any number of things. While there have been some versions of JIRA with bugs that triggered this, it is almost always caused by a plugin that is doing something wrong, such as running searches from its own thread pool without using JiraThreadLocalUtil to take care of the cleanup. Given that there are a wide variety of potential culprits and that it is usually related to a plugin that we do not ourselves test, there is no simple resolution for this issue. Customers that encounter it should open a support request so that we can work with you to track down what is causing it on your particular instance.

            What was the resolution for this?

            Andrew Morin added a comment - What was the resolution for this?
            ElaineA made changes -
            Remote Link New: This issue links to "Page (Extranet)" [ 83317 ]

            Hi jasbris,

            Could you please file a support case at https://support.atlassian.com? that way we can take a look and troubleshoot what is happening in your instance.

            Regards,

            Oswaldo Hernández.
            JIRA Bugmaster.
            [Atlassian].

            Oswaldo Hernandez (Inactive) added a comment - Hi jasbris , Could you please file a support case at https://support.atlassian.com? that way we can take a look and troubleshoot what is happening in your instance. Regards, Oswaldo Hernández. JIRA Bugmaster. [Atlassian] .

              Unassigned Unassigned
              73d805a2526b MattS
              Votes:
              1 Vote for this issue
              Watchers:
              15 Start watching this issue

                Created:
                Updated:
                Resolved: