If a mirror sync (git fetch) idle times out, fetched items are left on-disk as temporary files. Due to the sync failure, the mirror will schedule a new sync. This retry process is indefinite with a incremental retry-wait time. This causes a buildup of objects/pack/tmp_pack_* files which can lead to disk exhaustion.
The idle timeout commonly occurs after objects are received (during/before the resolving deltas stage) and therefore the full repository is fetched multiple times, which can lead to disk space exhaustion.
BSERV-12202 can aggravate this issue for smaller repositories, leading to longer leak times.
- Set plugin.mirroring.remote.command.timeout.idle=2
- Mirror the Linux repository
- Sync will timeout and retry, cleaning up past attempts
- Sync times out
- Git will leave the fetch files on disk objects/pack/tmp_pack_*
- These files follow unreachable object timelines (default 2 weeks) and will be pruned in 2 weeks automatically by git GC
- After a retry-wait time, the mirror retries the sync causing a new tmp_pack_* file. Each retry has a longer retry-wait time.
- Cycle repeats
The below exception is thrown in the atlassian-bitbucket.log file:
When disk space is exhausted:
Please note, due to disk space exhaustion logs can be incomplete
Increase plugin.mirroring.remote.command.timeout.idle=1800 to a larger number to accommodate your repository.