Details
-
Bug
-
Resolution: Fixed
-
High
-
6.7.0
-
8
-
Severity 3 - Minor
-
7
-
Description
Issue Summary
The mirror synchronization idle timeout (git) is too aggressive at 5 minutes, compared to the upstream default at 30 minutes. With large repositories the upstream can spend over 5 minutes to gather data and send a response back to the client. This causes the sync operation to fail with a timeout before the upstream can send anything to the mirror.
If the ustream has populated the clone cache then the response will be much faster, which will then succeed.
Steps to Reproduce
- Mirror a large repository
Expected Results
The repository completes synchronization.
Actual Results
The repository fails to sync with the following:
2020-01-16 11:50:20,084 DEBUG [initial-sync-thread-pool:thread-5] c.a.b.i.m.m.f.s.InitialSyncHelper Exception processing SyncRequest{externalRepositoryId=333}
com.atlassian.bitbucket.ServerException: An error occurred while executing an external process: process timed out
at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.evaluateThrowable(GitCommandExitHandler.java:120)
at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.onError(GitCommandExitHandler.java:208)
at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onExit(DefaultCommandExitHandler.java:32)
at com.atlassian.stash.internal.scm.git.command.fetch.FetchExitHandler.onExit(FetchExitHandler.java:39)
at com.atlassian.bitbucket.scm.BaseCommand.callExitHandler(BaseCommand.java:214)
On the upstream we see:
Application logs:
2020-01-16 04:43:17,902 @ESXSIAx274x691295x3 c.a.bitbucket.scm.BaseCommand Executed /usr/local/git/bin/git upload-pack /var/bitbucket/shared/data/repositories/333 2020-01-16 04:43:17,902 @ESXSIAx274x691295x3 c.a.b.scm.ssh.SshCommandExitHandler /usr/local/git/bin/git upload-pack /var/bitbucket/shared/data/repositories/333: (Exit code: 0)
Access logs:
10.0.0.85 | ssh | i@ESXSIAx274x691295x3 | addon_bitbucket.mirror.b3z8-rvjf-m97z-im8o | 2020-01-16 04:34:28,459 | SSH - git-upload-pack '/prj/test.git' | "SSH-2.0-OpenSSH_7.4" | - | - | - | - | - | 16xc27y | 10.0.0.85 | ssh | o@ESXSIAx274x691295x3 | addon_bitbucket.mirror.b3z8-rvjf-m97z-im8o | 2020-01-16 06:34:17,796 | SSH - git-upload-pack '/prj/test.git' | "SSH-2.0-OpenSSH_7.4" | 0 | 219 | 5284055822 | cache:miss, clone, protocol:1, ssh:user:id:1234 | 7189337 | 16xc27y |
Note the long time difference between access log lines vs the application log. The request continues to run on the upstream until the clone cache has been populated. If the mirror retries after this point and the cache is still valid, it will succeed.
Workaround
Place the following in the mirror server bitbucket.properties file to set the timeout to 30 minutes:
plugin.mirroring.remote.command.timeout.idle=1800
