Improve S3 Artifact Handler download throughput in AWS SDK v2

XMLWordPrintable

    • Type: Suggestion
    • Resolution: Unresolved
    • None
    • Component/s: Artifacts, AWS
    • None

      Issue Summary

      Following the upgrade to AWS SDK v2 in Bamboo Data Center, S3 artifact downloads show a noticeable throughput reduction compared to older versions.

      • AWS SDK v1 scheduled multipart download tasks across concurrent threads in parallel. 
      • AWS SDK v2 uses an asynchronous Netty HTTP client where multipart download chunks are requested sequentially via the subscriber stream model rather than concurrently.

      This is a suggestion to explore optimizations to improve S3 artifact download concurrency/throughput within the AWS SDK v2 architecture (e.g. evaluating AWS CRT client tuning or concurrency configurations) for file transfers.

      Steps to Reproduce

      1. Configure the Amazon S3 Artifact Handler in Bamboo Administration (Overview > Artifact Handlers).
      1. Run a build or deployment plan on a remote agent that downloads large artifacts from S3.
      1. Compare download throughput against Bamboo 9.6.x or direct S3 concurrent downloads.

      Expected Results

      S3 artifact downloads download concurrently with high multi-part throughput comparable to AWS SDK v1 or optimized concurrent S3 transfer clients.

      Actual Results

      Artifact download throughput is lower due to sequential multi-part chunk retrieval in the default AWS SDK v2 Netty HTTP client.

      Workaround

      Currently there is no known workaround for this behavior. A workaround will be added here when available

              Assignee:
              Unassigned
              Reporter:
              AP
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: