AWS S3: Bulk Attachment operations - "Unable to execute HTTP request: Timeout waiting for connection from pool"

Issue Summary

This is reproducible on Data Center: yes

Confluence DC configured for Amazon S3 becomes temporarily unresponsive and slow when performing a page copy involving a large number of attachments. 

During this period navigating through Confluence becomes slow and attachment data and avatars are not rendered. This behaviour lasts for ~10mins on the node on which the operation was performed, after which things returns to normal. 

Initial analysis shows that this issue is encountered when performing simultaneous attachment related operations that involve > 49 attachments, for example, page copies containing more than 49 attachments. 

Steps to Reproduce

For a page comprising of more than 49 attachments perform a page copy. Doing this invokes the issue, and Confluence becomes slow to respond when simply interacting with it through the UX.

Expected Results

Page copies regardless of attachment count should not impact Confluence in this way. 

Actual Results

1. Copying a page comprising of more than 49 attachments invokes the issue, and Confluence becomes slow to respond when simply interacting with it through the UX.
2. In this state attachment data is not loaded and nor can it be added.
3. In a multi node cluster, the behaviour is limited to the node that the attachment copy was invoked from. Other nodes in the cluster are not impacted.
4. In this state, when trying to add an attachment to another page, this message was rendered via UX:
   Unable to execute HTTP request: Timeout waiting for connection from pool.
5. After ~10 mins the problematic node resolves itself and attachment related operations return to normal, where avatar data is once again loaded and attachments can be viewed and added.

The below exception is thrown in the confluence.log file:

caused by: com.atlassian.confluence.pages.persistence.dao.filesystem.AttachmentDataFileSystemException: Failed to open InputStream for new S3 object storage attachment: confluence/attachments/v4/192/148/4618617205/4618617205.1
at com.atlassian.confluence.impl.pages.attachments.objectstorage.ObjectStorageAttachmentDataFileSystem.saveAttachmentData(ObjectStorageAttachmentDataFileSystem.java:100)
caused by: java.io.IOException: Unable to execute HTTP request: Timeout waiting for connection from pool
at com.atlassian.dc.filestore.impl.s3.Client.translate(Client.java:92)
caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316) 

Workaround

Currently not known.