This is a development task which should involve George, Giles and maybe IT. It must be driven by a developer, don't make Giles do this on his own.

Some bulletpointed ideas for best practices:

• upgrade and reindex one cluster at a time, shut down again. this avoids cluster communication during reindex
• lock out users from a server that is reindexing after an upgrade. this helps reindexing be fast
• try reindexing out on staging server. if the time it takes feels long compared the size of the installation, or if it takes longer than 30 minutes, investigate: reindexing should max out all cores. the more you cores you have, the faster reindexing should be. if the cores are not maxed out, check other parameters like disk IO, database, etc. It could be that your 8 cores can't work fast enought because your database is on a slow (or saturared) network link, or generally much weaker hardware.

• Possibly try concmarksweep. We have made good experiences with xxxxx. we wouldnt set this as the default setting, because all jvms are different.

See Paul for some examples on trying this on QA-CAC.