This is a possible regression of JRA-29587

      Steps to reproduce:

      1. install JIRA 5.2.11 in linux and create mass number of projects and issues
      2. edit or comment on some issue. Run the "lsof +L1" command.
      3. deleted opened files still being display. example (snippet of it):
        java      6337 janetalbion  644r   REG   0,19  1147106     0  410788 /home/janetalbion/supportinstance/jsp/jira5211/home/caches/indexes/comments/_g1.tis (deleted)
        java      6337 janetalbion  681r   REG   0,19  2715619     0  410805 /home/janetalbion/supportinstance/jsp/jira5211/home/caches/indexes/comments/_g1.frq (deleted)
        java      6337 janetalbion  682r   REG   0,19  1318707     0  410807 /home/janetalbion/supportinstance/jsp/jira5211/home/caches/indexes/comments/_g1.prx (deleted)
        java      6337 janetalbion  683r   REG   0,19 12637805     0  410793 /home/janetalbion/supportinstance/jsp/jira5211/home/caches/indexes/comments/_g1.fdt (deleted)
        java      6337 janetalbion  684r   REG   0,19   289524     0  410794 /home/janetalbion/supportinstance/jsp/jira5211/home/caches/indexes/comments/_g1.fdx (deleted)
        
      • even though the ERROR "too many opened" files might not be thrown immediately, after some times, JIRA still need a restart to clear the opened files

      Additional notes:

      • re-indexing while locking the instance will not reproduce this behaviour
      • not easy to reproduce with small instance. It's become easier to reproduce when instance is larger. So far tested with vanilla instance with approximately 5000 issues and it can be reproduce more frequently compare to 100 issues.
      • the list of open files might disappear on it's own (without performing any operation, some of the open files are no longer appearing when "lsof +L1" is executed
      • when Test Disk Speed is run, the result is Excellent
      • this is so far tested with JIRA 5.2.11

      Workaround

      Please see our Loss of Functionality due to Too Many Open Files Error KB for further information.

          Form Name

            [JRASERVER-35726] Jira does not close handles of old index properly

            Hi german.rodriguez,

            For those JIRA 5.2.x instances who don't have the SVN plugin installed, but still experience the leakage, would you recommend that the 'ulimit' parameter is increased once again in the server's OS? In our case, we have increased it twice (from the default 1024 to 4096, and then from 4096 to 8192), and the issue still came up a few weeks ago. Somewhere in the knowledge base I read that a recommended workaround for large instances (ours has 200,000+ issues) is to perform periodic JIRA restarts, to mitigate the impact of the index leakage.

            It is not possible to make further recommendations / troubleshooting without more specifics of your environment and data.

            Therefore the way forward here is for you to please create a support ticket at https://support.atlassian.com? That way we can provide you with support specific to your circumstances and with higher security. The credentials should be the same as for this site (https://jira.atlassian.com).

            If we upgrade to JIRA 6.x in the future, would the issue be definitely fixed?

            The known and confirmed leakage issue in JIRA was originally raised as JRA-29587 and it should be fixed in JIRA 5.1.6, 5.2 and later versions of JIRA.

            Hope the information helps.

            Regards,

            Oswaldo Hernández.
            JIRA Bugmaster.
            [Atlassian].

            Oswaldo Hernandez (Inactive) added a comment - - edited Hi german.rodriguez , For those JIRA 5.2.x instances who don't have the SVN plugin installed, but still experience the leakage, would you recommend that the 'ulimit' parameter is increased once again in the server's OS? In our case, we have increased it twice (from the default 1024 to 4096, and then from 4096 to 8192), and the issue still came up a few weeks ago. Somewhere in the knowledge base I read that a recommended workaround for large instances (ours has 200,000+ issues) is to perform periodic JIRA restarts, to mitigate the impact of the index leakage. It is not possible to make further recommendations / troubleshooting without more specifics of your environment and data. Therefore the way forward here is for you to please create a support ticket at https://support.atlassian.com? That way we can provide you with support specific to your circumstances and with higher security. The credentials should be the same as for this site ( https://jira.atlassian.com ). If we upgrade to JIRA 6.x in the future, would the issue be definitely fixed? The known and confirmed leakage issue in JIRA was originally raised as JRA-29587 and it should be fixed in JIRA 5.1.6, 5.2 and later versions of JIRA. Hope the information helps. Regards, Oswaldo Hernández. JIRA Bugmaster. [Atlassian] .

            Hi Oswaldo,

            For those JIRA 5.2.x instances who don't have the SVN plugin installed, but still experience the leakage, would you recommend that the 'ulimit' parameter is increased once again in the server's OS? In our case, we have increased it twice (from the default 1024 to 4096, and then from 4096 to 8192), and the issue still came up a few weeks ago. Somewhere in the knowledge base I read that a recommended workaround for large instances (ours has 200,000+ issues) is to perform periodic JIRA restarts, to mitigate the impact of the index leakage. Are there any other trouble points that we should check? If we upgrade to JIRA 6.x in the future, would the issue be definitely fixed?

            Thanks in advance for your comments.
            GERMAN

            german.rodriguez added a comment - Hi Oswaldo, For those JIRA 5.2.x instances who don't have the SVN plugin installed, but still experience the leakage, would you recommend that the 'ulimit' parameter is increased once again in the server's OS? In our case, we have increased it twice (from the default 1024 to 4096, and then from 4096 to 8192), and the issue still came up a few weeks ago. Somewhere in the knowledge base I read that a recommended workaround for large instances (ours has 200,000+ issues) is to perform periodic JIRA restarts, to mitigate the impact of the index leakage. Are there any other trouble points that we should check? If we upgrade to JIRA 6.x in the future, would the issue be definitely fixed? Thanks in advance for your comments. GERMAN

            Hi all,

            After all the investigative work done here, we are fairly confident there is no regression here after the initial fix done at JRA-29587.

            From the investigation done in support cases there's suspected leakage issues in the subversion plugin available at: https://marketplace.atlassian.com/plugins/com.atlassian.jira.plugin.ext.subversion

            If this is confirmed then an issue needs to be raised at https://ecosystem.atlassian.net/browse/SVN to fix any problems that might exist in this plugin.

            Many thanks to jalbion and cfuller for all the investigative work done on this issue.

            Regards,

            Oswaldo Hernández.
            JIRA Bugmaster.
            [Atlassian].

            Oswaldo Hernandez (Inactive) added a comment - Hi all, After all the investigative work done here, we are fairly confident there is no regression here after the initial fix done at JRA-29587 . From the investigation done in support cases there's suspected leakage issues in the subversion plugin available at: https://marketplace.atlassian.com/plugins/com.atlassian.jira.plugin.ext.subversion If this is confirmed then an issue needs to be raised at https://ecosystem.atlassian.net/browse/SVN to fix any problems that might exist in this plugin. Many thanks to jalbion and cfuller for all the investigative work done on this issue. Regards, Oswaldo Hernández. JIRA Bugmaster. [Atlassian] .

            crf added a comment -

            german.rodriguez:

            Bulk edits could cause the numbers to spike, but as I stated above, this should be a temporary condition, and the file handles should be reclaimed afterwards.

            I am not very familiar with the JIRA Agile (formerly GreenHopper) code, but to the best of my knowledge, it uses the normal, well-behaved ways of accessing Lucene indexes and would therefore not be likely to contribute to the problem. The SVN plugin works with Lucene directly, so it is more likely to cause problems.

            crf added a comment - german.rodriguez : Bulk edits could cause the numbers to spike, but as I stated above, this should be a temporary condition, and the file handles should be reclaimed afterwards. I am not very familiar with the JIRA Agile (formerly GreenHopper) code, but to the best of my knowledge, it uses the normal, well-behaved ways of accessing Lucene indexes and would therefore not be likely to contribute to the problem. The SVN plugin works with Lucene directly, so it is more likely to cause problems.

            Chris,

            Thanks so much for the information. In the particular case of my company, we haven't installed the SVN plugin yet, although we are planning to test it out and, if the management agrees, we will set it up permanently. I forgot to mention that the people in charge of supporting the server in which our JIRA instance resides have already increased the 'ulimit' parameter twice (current value is 8192); actually, between the first and second times, only a few weeks passed, and the second time the limit was increased by 2x (from 4096 to 8192).

            The remark that you made about bulk edits might make sense for our instance, because I know that many of our users use that feature. Can I ask you if possibly Greenhopper (our current version is 6.2) could also be causing index file leaks?

            Thanks so much for your support.

            german.rodriguez added a comment - Chris, Thanks so much for the information. In the particular case of my company, we haven't installed the SVN plugin yet, although we are planning to test it out and, if the management agrees, we will set it up permanently. I forgot to mention that the people in charge of supporting the server in which our JIRA instance resides have already increased the 'ulimit' parameter twice (current value is 8192); actually, between the first and second times, only a few weeks passed, and the second time the limit was increased by 2x (from 4096 to 8192). The remark that you made about bulk edits might make sense for our instance, because I know that many of our users use that feature. Can I ask you if possibly Greenhopper (our current version is 6.2) could also be causing index file leaks? Thanks so much for your support.

            crf added a comment -

            german.rodriguez (and any other watchers):

            While deleted index files will show up in LSOF output from time to time, to a certain extent this can be normal for how JIRA interacts with Lucene. Due to how Lucene organizes things internally, this is particularly likely to see on larger instances when you edit or delete an old issue. I would also expect large bulk edits to spike the number of deleted file handles temporarily. The way that these old index files get collected can be "slow", as it relies on searches completing and possibly several garbage collection cycles, but they should eventually be reclaimed, and you should only be concerned if the number of index files in the "deleted" list grows noticeably over time.

            Although this problem is occasionally reported for specific instances, as far as I am aware, it cannot be reproduced on a standard JIRA installation and happens to very few customers. This suggests that add-ons are likely to be responsible. I am aware of only one existing support case related to this issue with an installed version of 5.2 or higher, and in that particular case I suspect that the JIRA Subversion Plugin is responsible for leaking the indexes when it loses its connection to the svn server. This has not been confirmed; however, given the rarity of the problem, it seems very likely that some add-on is ultimately responsible.

            If that is true, then this issue should probably be closed as invalid, but there still isn't enough information to make that decision at this time. Unfortunately, since the problem seems to have different causes for each individual customer, the only generic workaround available is to restart JIRA on a regular basis to force the deleted file handles to be released. Obviously, this is not a good answer, but it is all I can offer without more information.

            The best course of action for anyone experiencing this problem is to open a support request if you have not already done so. In order for developers to track down the problem, we need a heap dump and lsof +L1 output from the time that the failure occurred. If you need help collecting these, the support team will be able to provide detailed instructions.

            crf added a comment - german.rodriguez (and any other watchers): While deleted index files will show up in LSOF output from time to time, to a certain extent this can be normal for how JIRA interacts with Lucene. Due to how Lucene organizes things internally, this is particularly likely to see on larger instances when you edit or delete an old issue. I would also expect large bulk edits to spike the number of deleted file handles temporarily. The way that these old index files get collected can be "slow", as it relies on searches completing and possibly several garbage collection cycles, but they should eventually be reclaimed, and you should only be concerned if the number of index files in the "deleted" list grows noticeably over time. Although this problem is occasionally reported for specific instances, as far as I am aware, it cannot be reproduced on a standard JIRA installation and happens to very few customers. This suggests that add-ons are likely to be responsible. I am aware of only one existing support case related to this issue with an installed version of 5.2 or higher, and in that particular case I suspect that the JIRA Subversion Plugin is responsible for leaking the indexes when it loses its connection to the svn server. This has not been confirmed; however, given the rarity of the problem, it seems very likely that some add-on is ultimately responsible. If that is true, then this issue should probably be closed as invalid, but there still isn't enough information to make that decision at this time. Unfortunately, since the problem seems to have different causes for each individual customer, the only generic workaround available is to restart JIRA on a regular basis to force the deleted file handles to be released. Obviously, this is not a good answer, but it is all I can offer without more information. The best course of action for anyone experiencing this problem is to open a support request if you have not already done so. In order for developers to track down the problem, we need a heap dump and lsof +L1 output from the time that the failure occurred. If you need help collecting these, the support team will be able to provide detailed instructions.

            Hi,

            My company has been experiencing this issue sporadically for the last few months. We're using JIRA 5.2.4.

            Are there any other workarounds that Atlassian might have discovered lately? After reading the comments posted in this issue and in JRA-29587, it's not 100% clear to me if re-indexing helps as a temporary workaround, or it actually induces more index file leaks. Our instance has gotten a bit large (about 200,000+ issues), so if it hasn't been manually reindexed frequently for the last several weeks, does that cause the deleted index file handles to be maintained in memory or disk?

            Thanks.

            german.rodriguez added a comment - Hi, My company has been experiencing this issue sporadically for the last few months. We're using JIRA 5.2.4. Are there any other workarounds that Atlassian might have discovered lately? After reading the comments posted in this issue and in JRA-29587 , it's not 100% clear to me if re-indexing helps as a temporary workaround, or it actually induces more index file leaks. Our instance has gotten a bit large (about 200,000+ issues), so if it hasn't been manually reindexed frequently for the last several weeks, does that cause the deleted index file handles to be maintained in memory or disk? Thanks.

            MattS added a comment -

            As Chris Fuller [Atlassian] noted on the original bug about at https://jira.atlassian.com/browse/JRA-29587?focusedCommentId=338881&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-338881

            "keep in mind that every time they reindex will leak space from the filesystem that is equivalent to the entire index, so the space can disappear very quickly."

            and that scheduling a JIRA restart is a complex process for many enterprise customers.

            MattS added a comment - As Chris Fuller [Atlassian] noted on the original bug about at https://jira.atlassian.com/browse/JRA-29587?focusedCommentId=338881&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-338881 "keep in mind that every time they reindex will leak space from the filesystem that is equivalent to the entire index, so the space can disappear very quickly." and that scheduling a JIRA restart is a complex process for many enterprise customers.

              Unassigned Unassigned
              jalbion Janet Albion (Inactive)
              Affected customers:
              2 This affects my team
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: