Mirror synchronization idle timeout causes storage leak

Issue Summary

If a mirror sync (git fetch) idle times out, fetched items are left on-disk as temporary files. Due to the sync failure, the mirror will schedule a new sync. This retry process is indefinite with a incremental retry-wait time. This causes a buildup of objects/pack/tmp_pack_* files which can lead to disk exhaustion.

The idle timeout commonly occurs after objects are received (during/before the resolving deltas stage) and therefore the full repository is fetched multiple times, which can lead to disk space exhaustion.

BSERV-12202 can aggravate this issue for smaller repositories, leading to longer leak times.

Steps to Reproduce

1. Set plugin.mirroring.remote.command.timeout.idle=2
2. Mirror the Linux repository

Expected Results

1. Sync will timeout and retry, cleaning up past attempts

Actual Results

1. Sync times out
2. Git will leave the fetch files on disk objects/pack/tmp_pack_*
   • These files follow unreachable object timelines (default 2 weeks) and will be pruned in 2 weeks automatically by git GC
3. After a retry-wait time, the mirror retries the sync causing a new tmp_pack_* file. Each retry has a longer retry-wait time.
4. Cycle repeats

The below exception is thrown in the atlassian-bitbucket.log file:

2020-01-16 11:50:20,084 DEBUG [initial-sync-thread-pool:thread-5]  c.a.b.i.m.m.f.s.InitialSyncHelper Exception processing SyncRequest{externalRepositoryId=333}
com.atlassian.bitbucket.ServerException: An error occurred while executing an external process: process timed out
	at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.evaluateThrowable(GitCommandExitHandler.java:120)
	at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.onError(GitCommandExitHandler.java:208)
	at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onExit(DefaultCommandExitHandler.java:32)
	at com.atlassian.stash.internal.scm.git.command.fetch.FetchExitHandler.onExit(FetchExitHandler.java:39)
	at com.atlassian.bitbucket.scm.BaseCommand.callExitHandler(BaseCommand.java:214)

When disk space is exhausted:

fatal: write error: No space left on device
fatal: index-pack failed

Please note, due to disk space exhaustion logs can be incomplete

Workaround

Increase plugin.mirroring.remote.command.timeout.idle=1800 to a larger number to accommodate your repository.