Medium Original Description
Method: retrieveObjectsIndividuallyAndLogFailures in ObjectQueue.java:
the 'handles' attribute is not synced and we have reported cases that causing the DefaultObjectQueueWorker (ie indexing threads to stop working because of this. (ie indexing grinds to a halt as more and more index threads a terminated)
Looking at at the log files in the case:
untitled text 3:4169: Exception in thread "Indexer: 10" java.util.ConcurrentModificationException untitled text 3:4181: Exception in thread "Indexer: 9" java.util.ConcurrentModificationException untitled text 3:4201: Exception in thread "Indexer: 4" java.util.ConcurrentModificationException untitled text 3:4289: Exception in thread "Indexer: 8" java.util.ConcurrentModificationException untitled text 3:4306: Exception in thread "Indexer: 7" java.util.ConcurrentModificationException untitled text 3:4329: Exception in thread "Indexer: 1" java.util.ConcurrentModificationException untitled text 3:4401: Exception in thread "Indexer: 5" java.util.ConcurrentModificationException untitled text 3:4838: Exception in thread "Indexer: 2" java.util.ConcurrentModificationException untitled text 3:4863: "Indexer: 6" daemon prio=10 tid=0x0000002bb5d2fc00 nid=0x508d runnable [0x0000000045f86000..0x0000000045f87c30] untitled text 3:5850: "Indexer: 6" daemon prio=10 tid=0x0000002bb5d2fc00 nid=0x508d runnable [0x0000000045f86000..0x0000000045f87c30] untitled text 3:6601: "Indexer: 6" daemon prio=10 tid=0x0000002bb5d2fc00 nid=0x508d runnable [0x0000000045f86000..0x0000000045f87c30] untitled text 3:7362: "Indexer: 6" daemon prio=10 tid=0x0000002bb5d2fc00 nid=0x508d runnable [0x0000000045f86000..0x0000000045f87c30] untitled text 3:8127: "Indexer: 6" daemon prio=10 tid=0x0000002bb5d2fc00 nid=0x508d runnable [0x0000000045f86000..0x0000000045f87c30]
- all threads experience this ConcurrentModEx...and when a threaddump is peformed by the customer, there is only one thread left.
The solution may be to sync the handles, as we expect this happens rarely (as it is only called in line 68 catch clause: (HibernateObjectRetrievalFailureException) and therefore less likely to affect the indexing performance.
For consistency: sync handles is is also in Pop(), we should do the same here?
The fixed code is this:
private Collection retrieveObjectsIndividuallyAndLogFailures() { List objects = new ArrayList(handles.size()); synchronized (handles) { for (HibernateHandle handle : handles) { try { objects.add(anyTypeDao.findByHandle(handle)); } catch (HibernateObjectRetrievalFailureException e) { if(log.isDebugEnabled()) { log.debug("Unable to retrieve single object: " + e.getMessage(), e); } else { log.warn("Unable to retrieve single object: " + e.getMessage()); } } }
hope this helps.
Workaround
Attached is a single class patch for this issue. To apply it, download this file in <CONFLUENCE HOME>/WEB-INF/classes/com/atlassian/confluence/search/lucene/ (creating any missing directories as appropriate) and restart Confluence. This patch is (currently) required to resolve this issue for all versions of Confluence prior to 3.4.
- mentioned in
-
Page Loading...