Lucene reindexing fails and goes to 100% CPU on multi-processor systems

After upgrading from 1.4-DR6 to 1.4-RC2, I decided to "Rebuild the Search Index". The reindexing process hung and CPU utilization was at 100%. I stopped Tomcat, removed all the files from the index directory and tried again. It hung at a different place. I repeated this many times with the hang always occurring on a different document. There were many different types of exceptions appearing on the console, and one of them mentioned a concurrency error. I cleared the index directory again and restarted Tomcat, but set "CPU Affinity" to just a single CPU – and that allowed the reindexing process to run successfully without any errors. This indicates that the reindex process has a critical problem on multi-processor servers. I do not yet know if this problem occurs during normal indexer updates or whether it is limited to the reindex process.