> McAfee Viruscan active, but not set to scan index file-types.

Not sure if it's relevant, but we have a report from another user that McAfee's NetShield 4.5 claims to let one exclude folders from scanning, but doesn't actually do so. Upgrading to 7.0 apparently fixes this.

Are we sure its nothing to do with the way indexing is done now (it now uses less files)

I meant 17 minutes.

We have about 17500 issues
We have 26000 comments

Issue cache size is 10000

Server is running on linux (so no virusscan issues)

Need to work out why this is taking so long....

a) its practically 100% IO-bound
b) its running in only 1 thread

What you want to do is

a) get Doug Lea's concurrency Library
b) have a number of worker pools loading issues, loading comments, indexing issues, indexing comments.

My suggestion:
Have 4 worker pools (see note # below)

Issue Loading Worker Pool
Comment Loading Worker Pool
Issue Indexing Worker Pool
Comment Indexing Worker Pool

• main thread dispatches issue-batch-load jobs (load 100 at a time) to Issue Loading Worker Pool
• The workers in Issue Loading Worker Pool loads, say, 100 issues and sends issue-indexing jobs to Issue Indexing Worker Pool - and sends comment-loading jobs to Comment Loading Worker Pool
• Issue Indexing Worker Pool ... takes an issue and indexes the bugger
• Comment Loading Worker Pool loads comments for a given issue and sends comment-indexing jobs to the Comment Indexing Worker Pool
• Comment Indexing Worker Pool ... takes a comment and indexes the bugger.

(#) Really speaking, you can fold the 2 loaders into one worker pool and the two indexers into another. The primary thing is to give the database access and the lucene indexing their own thread pools so one task doesnt starve the other of threads.

Given the io-bound nature of the reindexing, you can probably tune it up to a very high number of threads.

-Nick

Roughly speaking it looks like this:

PooledExecutor executor = new PooledExecutor(new LinkedQueue, 10);
executor.createThreads(10)

foreach issue: {

executor.execute(new Runnable() {
... index issue
});

executor.execute(new Runnable() { ... load comments ... index comments }); }
executor.shutdownAfterProcessingCurrentlyQueuedTasks();
executor.awaitTerminationAfterShutdown();