Uploaded image for project: 'Atlassian Cloud'
  1. Atlassian Cloud
  2. CLOUD-8384

Cloud instances are receiving a timeout when connecting to Blobstore

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Medium
    • None
    • None

    Description

      It's reproducible by adding attachments on issues.
      It's also intermittent.
      By the time JIRA tries to sync with Blobstore, the following errors appears:

      2015-11-07 01:52:19.881416500 2015-11-07 01:52:19,881 blobstore-client-http:thread-612 WARN sysadmin 111x144162x2 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.InterruptedIOException: timeout
      2015-11-07 01:52:20.084498500 2015-11-07 01:52:20,084 blobstore-client-http:thread-612 WARN sysadmin 111x144162x2 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.IOException: Stream closed
      2015-11-07 01:52:20.091263500 2015-11-07 01:52:20,091 blobstore-client-http:thread-613 WARN sysadmin 111x144163x3 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.InterruptedIOException: timeout
      2015-11-07 01:52:20.234413500 2015-11-07 01:52:20,234 blobstore-client-http:thread-614 WARN sysadmin 111x144165x4 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.InterruptedIOException: timeout
      2015-11-07 01:52:20.293592500 2015-11-07 01:52:20,293 blobstore-client-http:thread-613 WARN sysadmin 111x144163x3 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.IOException: Stream closed
      2015-11-07 01:52:20.436838500 2015-11-07 01:52:20,436 blobstore-client-http:thread-614 WARN sysadmin 111x144165x4 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.IOException: Stream closed
      2015-11-07 01:52:20.486762500 2015-11-07 01:52:20,486 blobstore-client-http:thread-612 WARN sysadmin 111x144162x2 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.IOException: Stream closed
      2015-11-07 01:52:20.696227500 2015-11-07 01:52:20,696 blobstore-client-http:thread-613 WARN sysadmin 111x144163x3 1c0qgs8 104.192.142.137,104.192.140.114 /rest/internal/2/AttachTemporaryFile [i.a.b.client.impl.RemoteBlobStoreImpl] Retrying after recoverable exception java.io.IOException: Stream closed
      
      2015-11-06 12:46:09.768474500 2015-11-06 12:46:09,764 ajp-nio-127.0.0.104-8009-exec-137 WARN sysadmin 764x38599x5 pp5zew 104.192.142.137,104.192.141.18 /rest/jddap/1.0/attachment [c.a.j.issue.thumbnail.DefaultThumbnailManager] Error writing to thumbnail file: /data/jirastudio/jira/home/data/attachments/NEE/10000/NEE-7175/thumbs/_thumb_24718.png7966518938032165601.tmp
      2015-11-06 12:46:09.768476500 com.atlassian.jira.issue.attachment.AttachmentReadException: Remote blobstore couldn't provide an input stream for attachment 24718: java.net.SocketException: Socket closed
      2015-11-06 12:46:09.768483500 	at java.net.SocketInputStream.read(SocketInputStream.java:203)
      2015-11-06 12:46:09.768483500 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
      2015-11-06 12:46:09.768484500 	at okio.Okio$2.read(Okio.java:137)
      2015-11-06 12:46:09.768484500 	at okio.AsyncTimeout$2.read(AsyncTimeout.java:217)
      2015-11-06 12:46:09.768486500 	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:291)
      2015-11-06 12:46:09.768486500 	at okio.RealBufferedSource.indexOf(RealBufferedSource.java:285)
      2015-11-06 12:46:09.768487500 	at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:196)
      2015-11-06 12:46:09.768487500 	at com.squareup.okhttp.internal.http.HttpConnection.readResponse(HttpConnection.java:190)
      2015-11-06 12:46:09.768491500 	at com.squareup.okhttp.internal.http.HttpTransport.readResponseHeaders(HttpTransport.java:80)
      2015-11-06 12:46:09.768491500 	at com.squareup.okhttp.internal.http.HttpEngine.readNetworkResponse(HttpEngine.java:909)
      2015-11-06 12:46:09.768492500 	at com.squareup.okhttp.internal.http.HttpEngine.access$300(HttpEngine.java:95)
      2015-11-06 12:46:09.768492500 	at com.squareup.okhttp.internal.http.HttpEngine$NetworkInterceptorChain.proceed(HttpEngine.java:902)
      2015-11-06 12:46:09.768494500 	at com.squareup.okhttp.internal.http.HttpEngine.readResponse(HttpEngine.java:760)
      2015-11-06 12:46:09.768494500 	at com.squareup.okhttp.Call.getResponse(Call.java:274)
      2015-11-06 12:46:09.768495500 	at com.squareup.okhttp.Call$ApplicationInterceptorChain.proceed(Call.java:230)
      2015-11-06 12:46:09.768495500 	at com.squareup.okhttp.Call.getResponseWithInterceptorChain(Call.java:201)
      2015-11-06 12:46:09.768497500 	at com.squareup.okhttp.Call.execute(Call.java:81)
      2015-11-06 12:46:09.768497500 	at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl.lambda$doGet$15(RemoteBlobStoreImpl.java:225)
      2015-11-06 12:46:09.768498500 	at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl$$Lambda$559/12511424.get(Unknown Source)
      2015-11-06 12:46:09.768498500 	at com.atlassian.fugue.retry.RetrySupplier.get(RetrySupplier.java:93)
      2015-11-06 12:46:09.768500500 	at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl.lambda$retried$20(RemoteBlobStoreImpl.java:364)
      2015-11-06 12:46:09.768508500 	at io.atlassian.blobstore.client.impl.RemoteBlobStoreImpl$$Lambda$560/256677.get(Unknown Source)
      2015-11-06 12:46:09.768509500 	at com.atlassian.util.concurrent.Suppliers$CallableAdapter.call(Suppliers.java:119)
      2015-11-06 12:46:09.768509500 	at com.atlassian.util.concurrent.Executors$DefaultSubmitter$CallableRunner.run(Executors.java:62)
      2015-11-06 12:46:09.768510500 	at com.atlassian.util.concurrent.LimitedExecutor$Runner.run(LimitedExecutor.java:93)
      2015-11-06 12:46:09.768512500 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      2015-11-06 12:46:09.768512500 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      2015-11-06 12:46:09.768513500 	at java.lang.Thread.run(Thread.java:745)
      2015-11-06 12:46:09.768513500 	at com.atlassian.jira.issue.attachment.store.BlobStoreAttachmentStore$3.apply(BlobStoreAttachmentStore.java:107)
      2015-11-06 12:46:09.768515500 	at com.atlassian.jira.issue.attachment.store.BlobStoreAttachmentStore$3.apply(BlobStoreAttachmentStore.java:101)
      2015-11-06 12:46:09.768515500 	at com.google.common.base.Functions$FunctionComposition.apply(Functions.java:216)
      2015-11-06 12:46:09.768516500 	at com.atlassian.fugue.Either$Left.fold(Either.java:508)
      2015-11-06 12:46:09.768516500 	at com.atlassian.jira.issue.attachment.store.BlobStoreAttachmentStore$15.apply(BlobStoreAttachmentStore.java:367)
      2015-11-06 12:46:09.768519500 	at com.atlassian.jira.issue.attachment.store.BlobStoreAttachmentStore$15.apply(BlobStoreAttachmentStore.java:363)
      2015-11-06 12:46:09.768520500 	at com.atlassian.util.concurrent.Promises$Of$1.apply(Promises.java:263)
      2015-11-06 12:46:09.768520500 	at com.atlassian.util.concurrent.Promises$2.onSuccess(Promises.java:185)
      2015-11-06 12:46:09.768521500 	at com.google.common.util.concurrent.Futures$6.run(Futures.java:1319)
      2015-11-06 12:46:09.768522500 	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
      

      This is the public report of an internal incident.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              aborzzatto Andre Borzzatto
              Votes:
              5 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: