Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-12278

Mirror synchronization idle timeout causes storage leak


      Issue Summary

      If a mirror sync (git fetch) idle times out, fetched items are left on-disk as temporary files. Due to the sync failure, the mirror will schedule a new sync. This retry process is indefinite with a incremental retry-wait time. This causes a buildup of objects/pack/tmp_pack_* files which can lead to disk exhaustion.

      The idle timeout commonly occurs after objects are received (during/before the resolving deltas stage) and therefore the full repository is fetched multiple times, which can lead to disk space exhaustion.

      BSERV-12202 can aggravate this issue for smaller repositories, leading to longer leak times.

      Steps to Reproduce

      1. Set plugin.mirroring.remote.command.timeout.idle=2
      2. Mirror the Linux repository

      Expected Results

      1. Sync will timeout and retry, cleaning up past attempts

      Actual Results

      1. Sync times out
      2. Git will leave the fetch files on disk objects/pack/tmp_pack_*
        • These files follow unreachable object timelines (default 2 weeks) and will be pruned in 2 weeks automatically by git GC
      3. After a retry-wait time, the mirror retries the sync causing a new tmp_pack_* file. Each retry has a longer retry-wait time.
      4. Cycle repeats

      The below exception is thrown in the atlassian-bitbucket.log file:

      2020-01-16 11:50:20,084 DEBUG [initial-sync-thread-pool:thread-5]  c.a.b.i.m.m.f.s.InitialSyncHelper Exception processing SyncRequest{externalRepositoryId=333}
      com.atlassian.bitbucket.ServerException: An error occurred while executing an external process: process timed out
      	at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.evaluateThrowable(GitCommandExitHandler.java:120)
      	at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.onError(GitCommandExitHandler.java:208)
      	at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onExit(DefaultCommandExitHandler.java:32)
      	at com.atlassian.stash.internal.scm.git.command.fetch.FetchExitHandler.onExit(FetchExitHandler.java:39)
      	at com.atlassian.bitbucket.scm.BaseCommand.callExitHandler(BaseCommand.java:214)

      When disk space is exhausted:

      fatal: write error: No space left on device
      fatal: index-pack failed

      Please note, due to disk space exhaustion logs can be incomplete


      Increase plugin.mirroring.remote.command.timeout.idle=1800 to a larger number to accommodate your repository.

            ysun Yingran Sun
            ephillips@atlassian.com Edward
            1 Vote for this issue
            10 Start watching this issue