The segmented message solution implemented for transferring artifacts from remote agents to the server has a performance bottleneck on the server side that should be investigated.
Performance I was seeing in a test environment was in the order of 100 KB/s.
[BAM-1889] Improve server-side performance of artifact transfer
Adrian Hempel [Atlassian]
added a comment - I have tried adding each of the following settings to the TCP broker URLs on both the server and agent side:
- wireformat.tcpNoDelayEnabled=true
- jms.useAsyncSend=true
- socket.tcpNoDelay=true
- dispatchAsync=true
None have yielded any significant gain in throughput.
I would like to see an improvement of at least 20x.
My next approach is to implement a very simple REST-based mechanism for transferring artifacts to the server. I should be able to implement this by the end of the day.
Adrian Hempel [Atlassian]
added a comment - I have tried adding each of the following settings to the TCP broker URLs on both the server and agent side:
wireformat.tcpNoDelayEnabled=true
jms.useAsyncSend=true
socket.tcpNoDelay=true
dispatchAsync=true
None have yielded any significant gain in throughput.
I would like to see an improvement of at least 20x.
My next approach is to implement a very simple REST-based mechanism for transferring artifacts to the server. I should be able to implement this by the end of the day. Created a test setup that reproduces the problem described above.
On the same LAN, 10 MB is transferred in < 3s. From EC2, this rises to ~12 mins.
It appears that TCP packets are not being dispatched until the previous message is acknowledged.
I tried enabling TCP_NODELAY, to no avail.
Tomorrow, I will try using asynchronous sends.
Adrian Hempel [Atlassian]
added a comment - - edited Created a test setup that reproduces the problem described above.
On the same LAN, 10 MB is transferred in < 3s. From EC2, this rises to ~12 mins.
It appears that TCP packets are not being dispatched until the previous message is acknowledged.
I tried enabling TCP_NODELAY, to no avail.
Tomorrow, I will try using asynchronous sends . Some local testing with ipfw suggests that throughput is heavily tied with latency. The numbers below are for transferring a 40MB zip file.
| Bandwidth (Mbit/s) | Latency (ms) | Artifact time | Artifact (min) | Approx MB/s |
|---|---|---|---|---|
| 0 | 0 | 5 | 0.08 | 8.000 |
| 0 | 10 | 123 | 2.05 | 0.325 |
| 0 | 20 | 240 | 4 | 0.167 |
| 0 | 30 | 357 | 5.95 | 0.112 |
| 0 | 50 | 589 | 9.82 | 0.068 |
| 0 | 60 | 703 | 11.72 | 0.057 |
| 0 | 70 | 819 | 13.65 | 0.049 |
| 0 | 80 | 936 | 15.6 | 0.043 |
| 0 | 90 | 1054 | 17.57 | 0.038 |
| 0 | 100 | 1169 | 19.48 | 0.034 |
| 0 | 150 | 1749 | 29.15 | 0.023 |
| 0 | 200 | 2329 | 38.82 | 0.017 |
| 0 | 250 | 2910 | 48.5 | 0.014 |
| 10 | 0 | 85 | 1.42 | 0.471 |
| 10 | 10 | 202 | 3.37 | 0.198 |
| 10 | 20 | 318 | 5.3 | 0.126 |
| 10 | 30 | 434 | 7.23 | 0.092 |
| 10 | 50 | 666 | 11.1 | 0.060 |
| 10 | 60 | 784 | 13.07 | 0.051 |
| 10 | 70 | 899 | 14.98 | 0.044 |
| 10 | 80 | 1015 | 16.92 | 0.039 |
| 10 | 90 | 1131 | 18.85 | 0.035 |
| 10 | 100 | 1247 | 20.78 | 0.032 |
| 10 | 150 | 1827 | 30.45 | 0.022 |
| 10 | 200 | 2409 | 40.15 | 0.017 |
| 10 | 250 | 2987 | 49.78 | 0.013 |
Artifact transfer from EC2 seems to be around 14k/s which is about right (I think latency was around 240ms last we checked?)
I've implemented the REST-based mechanism, and it now takes 18s to transfer a 10MB artifact from EC2. Previously, this took around 12 minutes.
I need to do a little more work to protect against attackers from submitting malicious artifacts, and I should be able to complete this tomorrow.