Uploaded image for project: 'Bitbucket Server'
  1. Bitbucket Server
  2. BSERV-12202

Mirror synchronization on large repositories fail to sync due to idle timeout

    XMLWordPrintable

    Details

      Description

      Issue Summary

      The mirror synchronization idle timeout (git) is too aggressive at 5 minutes, compared to the upstream default at 30 minutes. With large repositories the upstream can spend over 5 minutes to gather data and send a response back to the client. This causes the sync operation to fail with a timeout before the upstream can send anything to the mirror.

      If the ustream has populated the clone cache then the response will be much faster, which will then succeed.

      Steps to Reproduce

      1. Mirror a large repository

      Expected Results

      The repository completes synchronization.

      Actual Results

      The repository fails to sync with the following:

      2020-01-16 11:50:20,084 DEBUG [initial-sync-thread-pool:thread-5]  c.a.b.i.m.m.f.s.InitialSyncHelper Exception processing SyncRequest{externalRepositoryId=333}
      com.atlassian.bitbucket.ServerException: An error occurred while executing an external process: process timed out
      	at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.evaluateThrowable(GitCommandExitHandler.java:120)
      	at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.onError(GitCommandExitHandler.java:208)
      	at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onExit(DefaultCommandExitHandler.java:32)
      	at com.atlassian.stash.internal.scm.git.command.fetch.FetchExitHandler.onExit(FetchExitHandler.java:39)
      	at com.atlassian.bitbucket.scm.BaseCommand.callExitHandler(BaseCommand.java:214)
      

      On the upstream we see:
      Application logs:

      2020-01-16 04:43:17,902 @ESXSIAx274x691295x3 c.a.bitbucket.scm.BaseCommand Executed /usr/local/git/bin/git upload-pack /var/bitbucket/shared/data/repositories/333
      2020-01-16 04:43:17,902 @ESXSIAx274x691295x3 c.a.b.scm.ssh.SshCommandExitHandler /usr/local/git/bin/git upload-pack /var/bitbucket/shared/data/repositories/333: (Exit code: 0)
      

      Access logs:

      10.0.0.85 | ssh | i@ESXSIAx274x691295x3 | addon_bitbucket.mirror.b3z8-rvjf-m97z-im8o | 2020-01-16 04:34:28,459 | SSH - git-upload-pack '/prj/test.git' | "SSH-2.0-OpenSSH_7.4" | - | - | - | - | - | 16xc27y | 
      10.0.0.85 | ssh | o@ESXSIAx274x691295x3 | addon_bitbucket.mirror.b3z8-rvjf-m97z-im8o | 2020-01-16 06:34:17,796 | SSH - git-upload-pack '/prj/test.git' | "SSH-2.0-OpenSSH_7.4" | 0 | 219 | 5284055822 | cache:miss, clone, protocol:1, ssh:user:id:1234 | 7189337 | 16xc27y | 
      

      Note the long time difference between access log lines vs the application log. The request continues to run on the upstream until the clone cache has been populated. If the mirror retries after this point and the cache is still valid, it will succeed.

      Workaround

      Place the following in the mirror server bitbucket.properties file to set the timeout to 30 minutes:

      plugin.mirroring.remote.command.timeout.idle=1800
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              fdoherty@atlassian.com Frank Doherty
              Reporter:
              ephillips@atlassian.com Edward
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: