Issue Details (XML | Word | Printable)

Key: CONF-7401
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: Unassigned
Reporter: Ivan Benko [Atlassian]
Votes: 2
Watchers: 8
Operations

Add/Edit UI Mockup to this issue
If you were logged in you would be able to see more operations.
Confluence

too many files open error due to bonnie.LuceneException

Created: 30/Nov/06 07:26 PM   Updated: 30/Aug/07 02:23 AM
Component/s: Searching / Indexing
Affects Version/s: 2.2.10
Fix Version/s: 2.3.x

Time Tracking:
Not Specified

File Attachments: 1. Java Archive File bonnie-2006.03.30.1.jar (41 kB)

Issue Links:
Part
 
Regression
 

Participants: Christopher Owen [Atlassian], Einar Mykletun, Greg Smith, Ivan Benko [Atlassian], Matt Ryall [Atlassian], Michael McKeown, Nicholas Ilacqua [Atlassian], Paul Schifferer and Tom Davies [Atlassian]
Since last comment: 1 year, 44 weeks, 1 day ago
Resolution Date: 18/Feb/07 10:14 PM
Labels:


 Description  « Hide
The present indexing process creates 'too many file handles'. When more FHs are needed, it runs out of resources and terminates with an error:

2006-11-13 10:59:11,991 ERROR [confluence.search.lucene.DefaultConfluenceIndexManager] processTasks Failed to refresh the lucene searcher.
java.io.FileNotFoundException: /opt/confluence-data/index/_16oo.f139 (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.lucene.store.FSIndexInput$Descriptor.<init>(FSDirectory.java:425)
at org.apache.lucene.store.FSIndexInput.<init>(FSDirectory.java:434)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:324)
at org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:525)
at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:157)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:129)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:115)
at org.apache.lucene.index.IndexReader$1.doBody(IndexReader.java:150)
at org.apache.lucene.store.Lock$With.run(Lock.java:109)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:143)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:138)
at org.apache.lucene.search.IndexSearcher.<init>(IndexSearcher.java:47)
at org.apache.lucene.search.DelayCloseIndexSearcher.<init>(DelayCloseIndexSearcher.java:88)
at org.apache.lucene.search.DateFilterScoringSearcher.<init>(DateFilterScoringSearcher.java:36)
at com.atlassian.confluence.search.lucene.ConfluenceLuceneConnection.createSearcher(ConfluenceLuceneConnection.java:63)
at com.atlassian.confluence.search.lucene.ConfluenceLuceneConnection.refreshSearcher(ConfluenceLuceneConnection.java:102)
at com.atlassian.confluence.search.lucene.DefaultConfluenceIndexManager.processTasks(DefaultConfluenceIndexManager.java:208)
at com.atlassian.confluence.search.lucene.DefaultConfluenceIndexManager.flushQueue(DefaultConfluenceIndexManager.java:93)
at sun.reflect.GeneratedMethodAccessor479.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.springframework.aop.framework.AopProxyUtils.invokeJoinpointUsingReflection(AopProxyUtils.java:61)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:149)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:116)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:56)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:138)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:152)
at $Proxy15.flushQueue(Unknown Source)
at com.atlassian.confluence.search.lucene.IndexQueueFlusher.executeInternal(IndexQueueFlusher.java:38)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:66)
at org.quartz.core.JobRunShell.run(JobRunShell.java:191)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:516)



 All   Comments   Work Log   Change History      Sort Order: Ascending order - Click to sort in descending order
Ivan Benko [Atlassian] added a comment - 04/Dec/06 06:36 AM
Unfortunately this is a known issue described in our documentation already. You can find a workaround on this page as well.

http://confluence.atlassian.com/x/Jd4C


Michael McKeown added a comment - 07/Dec/06 08:19 PM
That comment doesn't indicate how to fix it on a Windows server - we're running Windows server 2003 and experiencing this problem.

Ivan Benko [Atlassian] added a comment - 08/Dec/06 01:30 AM
Try to address this issue by switching over to using a compound index that uses less file handles. Please find attached a patched jar file (bonnie-2006.03.30-patched.jar) and copy to your confluence/WEB-INF/lib directory and restart.

Ivan Benko [Atlassian] added a comment - 08/Jan/07 05:57 PM
The newest release of Confluence 2.3 already includes this patch.

Michael McKeown added a comment - 09/Jan/07 07:20 PM
I have encountered the problem again, even with the bonnie-2006.03.30-patched.jar in place.

Ivan Benko [Atlassian] added a comment - 09/Jan/07 11:17 PM
Hi Michael,

Could you please create a support issue for your request at http://support.atlassian.com
Kindly attach your log files found in <confluence-install>/logs directory. Please zip-up this directory and attach it to the issue.

I shall deal with it separately - there may be something else going on.

Thanks,
Ivan


Michael McKeown added a comment - 09/Jan/07 11:46 PM
Already have - it's issue CSP-6493

Nicholas Ilacqua [Atlassian] added a comment - 11/Jan/07 06:14 PM
Has been taken over by above CSP

Einar Mykletun added a comment - 14/Feb/07 03:42 PM
We are using Confluence 2.2.8. When installing the patch jar file in WEB-INF/lib, should one delete the old bonnie.jar file?

Greg Smith added a comment - 14/Feb/07 03:53 PM
Einar,

The correct answer is yes, remove the old one however, If your up for the upgrade, I'd suggest a move to 2.3.1 or better.

Our 'too many open files' issue was never resolved with this patch, it only helped to prolong how often the Index was optimized and then when it did optimize it caused just as much latency. Eventually our server wouls still crash out with 'too many open files'. It was not until we moved to 2.3.1 that this issue went away. The compound indexing is 2.3.1 also really changed our server performance.

Obviously every environment is different, I only speak of my experiences.

  • Greg Smith

Ivan Benko [Atlassian] added a comment - 14/Feb/07 05:26 PM
Hi Einar,

Let me please just clarify this matter. Up to the version of Confluence 2.3, Lucene java text search engine module was not using compound indexing and thus some users were experiencing problems with 'too many files opened'. Since this problem was resolved by applying the compound indexing, we have had no enquiries forwarded to us regarding problems with indexing.
Hence, you may try to apply the patch as per the instructions given on this page under the heading "When getting support for this error" http://confluence.atlassian.com/x/Jd4C
Of course if your licence permits, you are more than welcome to perform an upgrade to the newest release of Confluence version 2.3.3 (as of yesterday)
Please follow these instructions:
http://confluence.atlassian.com/x/4hE

Let me know how you go and in case you need our support, please open a support ticket in http://support.atlassian.com
Thanks,
Ivan


Einar Mykletun added a comment - 15/Feb/07 10:23 AM
Greg and Ivan,

Thank you both for your comments and clarifications. Very much appreciated. Ok, I have removed the old bonnie.jar file and rebuilt the index – will see how it works. Also planning on upgrading to 2.3.x in about a month, assuming that all our critical macros are compatable (currently some are not, like the calendar).

Ivan – perhaps it's obvious to most everyone, but it may be an idea to mention that one should remove the old bonnie.jar file in the second bullet (bottom of page) here: http://confluence.atlassian.com/display/DOC/Fix+%27Too+many+open+files%27+error+on+Linux+by+increasing+filehandles?showComments=true#comments

Thanks again,

Einar


Tom Davies [Atlassian] added a comment - 18/Feb/07 10:14 PM
I have attached a corrected version of the patched bonnie jar. This should be used with any 2.2.x instance to make Confluence reduce file handle usage by using a compound Lucene index.

Michael McKeown added a comment - 18/Feb/07 10:18 PM
What's changed from the previous version? I'd like to know how significant the change is to see if it's worth scheduling an outage for.

Has this change been reflected in the 2.2.3 build?


Christopher Owen [Atlassian] added a comment - 18/Feb/07 10:27 PM
Hi Michael,

If you are still experiencing out of file handles on an instance of Confluence 2.2.x using the original patched bonnie jar then this patch should resolve it and I would consider it worth scheduling an outage for. If you aren't having file handle issues anymore then I wouldn't bother at this stage.

This patch is for an issue in Confluence 2.2.x that does not occur in Confluence 2.3 or higher, so this patch should not be used for 2.3.x instances (I assume that is what you meant)

Cheers,
Chris


Paul Schifferer added a comment - 23/Jul/07 05:10 PM
I have an instance of Confluence 2.5.3 where this issue is happening.

Greg Smith added a comment - 24/Jul/07 09:01 AM
I have also experienced this again with 2.5.3 while the index was being optimized.

It only happened once but figure it's worth noting. The log files looked very similar to the ones I had seen in the past with this issue.


Matt Ryall [Atlassian] added a comment - 30/Aug/07 02:23 AM
This issue has a regression in 2.5.x which was fixed in 2.5.7. See CONF-9251 for more details.