Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-12202

Mirror synchronization on large repositories fail to sync due to idle timeout


      Issue Summary

      The mirror synchronization idle timeout (git) is too aggressive at 5 minutes, compared to the upstream default at 30 minutes. With large repositories the upstream can spend over 5 minutes to gather data and send a response back to the client. This causes the sync operation to fail with a timeout before the upstream can send anything to the mirror.

      If the ustream has populated the clone cache then the response will be much faster, which will then succeed.

      Steps to Reproduce

      1. Mirror a large repository

      Expected Results

      The repository completes synchronization.

      Actual Results

      The repository fails to sync with the following:

      2020-01-16 11:50:20,084 DEBUG [initial-sync-thread-pool:thread-5]  c.a.b.i.m.m.f.s.InitialSyncHelper Exception processing SyncRequest{externalRepositoryId=333}
      com.atlassian.bitbucket.ServerException: An error occurred while executing an external process: process timed out
      	at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.evaluateThrowable(GitCommandExitHandler.java:120)
      	at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.onError(GitCommandExitHandler.java:208)
      	at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onExit(DefaultCommandExitHandler.java:32)
      	at com.atlassian.stash.internal.scm.git.command.fetch.FetchExitHandler.onExit(FetchExitHandler.java:39)
      	at com.atlassian.bitbucket.scm.BaseCommand.callExitHandler(BaseCommand.java:214)

      On the upstream we see:
      Application logs:

      2020-01-16 04:43:17,902 @ESXSIAx274x691295x3 c.a.bitbucket.scm.BaseCommand Executed /usr/local/git/bin/git upload-pack /var/bitbucket/shared/data/repositories/333
      2020-01-16 04:43:17,902 @ESXSIAx274x691295x3 c.a.b.scm.ssh.SshCommandExitHandler /usr/local/git/bin/git upload-pack /var/bitbucket/shared/data/repositories/333: (Exit code: 0)

      Access logs: | ssh | i@ESXSIAx274x691295x3 | addon_bitbucket.mirror.b3z8-rvjf-m97z-im8o | 2020-01-16 04:34:28,459 | SSH - git-upload-pack '/prj/test.git' | "SSH-2.0-OpenSSH_7.4" | - | - | - | - | - | 16xc27y | | ssh | o@ESXSIAx274x691295x3 | addon_bitbucket.mirror.b3z8-rvjf-m97z-im8o | 2020-01-16 06:34:17,796 | SSH - git-upload-pack '/prj/test.git' | "SSH-2.0-OpenSSH_7.4" | 0 | 219 | 5284055822 | cache:miss, clone, protocol:1, ssh:user:id:1234 | 7189337 | 16xc27y | 

      Note the long time difference between access log lines vs the application log. The request continues to run on the upstream until the clone cache has been populated. If the mirror retries after this point and the cache is still valid, it will succeed.


      Place the following in the mirror server bitbucket.properties file to set the timeout to 30 minutes:


            fdoherty@atlassian.com Frank Doherty
            ephillips@atlassian.com Edward
            1 Vote for this issue
            8 Start watching this issue