Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-82499

AWS S3: Bulk Attachment operations - "Unable to execute HTTP request: Timeout waiting for connection from pool"

XMLWordPrintable

      Issue Summary

      This is reproducible on Data Center: yes

      Confluence DC configured for Amazon S3 becomes temporarily unresponsive and slow when performing a page copy involving a large number of attachments. 

      During this period navigating through Confluence becomes slow and attachment data and avatars are not rendered. This behaviour lasts for ~10mins on the node on which the operation was performed, after which things returns to normal. 

      Initial analysis shows that this issue is encountered when performing simultaneous attachment related operations that involve > 49 attachments, for example, page copies containing more than 49 attachments. 

      Steps to Reproduce

      For a page comprising of more than 49 attachments perform a page copy. Doing this invokes the issue, and Confluence becomes slow to respond when simply interacting with it through the UX.

      Expected Results

      Page copies regardless of attachment count should not impact Confluence in this way. 

      Actual Results

      1. Copying a page comprising of more than 49 attachments invokes the issue, and Confluence becomes slow to respond when simply interacting with it through the UX.
      2. In this state attachment data is not loaded and nor can it be added.
      3. In a multi node cluster, the behaviour is limited to the node that the attachment copy was invoked from. Other nodes in the cluster are not impacted.
      4. In this state, when trying to add an attachment to another page, this message was rendered via UX:
        Unable to execute HTTP request: Timeout waiting for connection from pool.
      5. After ~10 mins the problematic node resolves itself and attachment related operations return to normal, where avatar data is once again loaded and attachments can be viewed and added.

      The below exception is thrown in the confluence.log file:

      caused by: com.atlassian.confluence.pages.persistence.dao.filesystem.AttachmentDataFileSystemException: Failed to open InputStream for new S3 object storage attachment: confluence/attachments/v4/192/148/4618617205/4618617205.1
      at com.atlassian.confluence.impl.pages.attachments.objectstorage.ObjectStorageAttachmentDataFileSystem.saveAttachmentData(ObjectStorageAttachmentDataFileSystem.java:100)
      caused by: java.io.IOException: Unable to execute HTTP request: Timeout waiting for connection from pool
      at com.atlassian.dc.filestore.impl.s3.Client.translate(Client.java:92)
      caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
      at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
      caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
      at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316) 

      Workaround

      Currently not known.

        1. attach-file.png
          attach-file.png
          27 kB
        2. oops-an-error.png
          oops-an-error.png
          112 kB

              b56a0fbdbbcd Dylan Rathbone
              b56a0fbdbbcd Dylan Rathbone
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: