Confluence ver003 to v4 migration stalls if there are too many subfolders

Problem

When upgrading to Confluence 8.1.x version from lower versions such as 7.x.x, the ver003 to v4 attachment migration job moves the files from v3 to v4 file structure. This process becomes very slow if there are more than 300k subfolders within a folder.

Environment

.Upgrading from 7.x.x or lower to 8.1.x

Steps to Reproduce

1. Create a Confluence instance with version 7.x.x or lower
2. Ensure that there are more than 300k Subfolders in nonspaced folder of this instance. This happens in real life only if there are too many users using the system. Attachments such as profile pictures and unsaved drafts are saved here.
3. Upgrade this instance to 8.1.x
4. The migration job will run fine for smaller folders, but will become very slow, such as more than one hour to migrate one attachment.
5. We can notice the following logs (after setting class atlassian.confluence.upgrade.AttachmentMigratorToV4 to DEBUG):

   moveAttachmentToV4 Moving file from /appl/confluence/shared-home/attachments/ver003/nonspaced/xxx/xxx/7890xxxxx/xxxxxxxx/1 to /appl/confluence/shared-home/attachments/v4/82/92/xxxxxxxx/xxxxxx.1
   

1. The frequency of occurrence these logs will be very slow such as once an hour etc., eventhough the file size is small

Expected Results

The migration should happen even if the subfolders contain large number of files

Actual Results

The migration process becomes very slow such as moving one attachment per hour.

Workaround

No workaround at this point. We could temporarily disable migration during upgrade:

CATALINA_OPTS="-Datlassian.darkfeature.confluence.disable-attachments-ver004=true ${CATALINA_OPTS}"

Notes