Uploaded image for project: 'Bitbucket Server'
  1. Bitbucket Server
  2. BSERV-12278

Mirror synchronization idle timeout causes storage leak

    XMLWordPrintable

    Details

      Description

      Issue Summary

      If a mirror sync (git fetch) idle times out, fetched items are left on-disk as temporary files. Due to the sync failure, the mirror will schedule a new sync. This retry process is indefinite with a incremental retry-wait time. This causes a buildup of objects/pack/tmp_pack_* files which can lead to disk exhaustion.

      The idle timeout commonly occurs after objects are received (during/before the resolving deltas stage) and therefore the full repository is fetched multiple times, which can lead to disk space exhaustion.

      BSERV-12202 can aggravate this issue for smaller repositories, leading to longer leak times.

      Steps to Reproduce

      1. Set plugin.mirroring.remote.command.timeout.idle=2
      2. Mirror the Linux repository

      Expected Results

      1. Sync will timeout and retry, cleaning up past attempts

      Actual Results

      1. Sync times out
      2. Git will leave the fetch files on disk objects/pack/tmp_pack_*
        • These files follow unreachable object timelines (default 2 weeks) and will be pruned in 2 weeks automatically by git GC
      3. After a retry-wait time, the mirror retries the sync causing a new tmp_pack_* file. Each retry has a longer retry-wait time.
      4. Cycle repeats

      The below exception is thrown in the atlassian-bitbucket.log file:

      2020-01-16 11:50:20,084 DEBUG [initial-sync-thread-pool:thread-5]  c.a.b.i.m.m.f.s.InitialSyncHelper Exception processing SyncRequest{externalRepositoryId=333}
      com.atlassian.bitbucket.ServerException: An error occurred while executing an external process: process timed out
      	at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.evaluateThrowable(GitCommandExitHandler.java:120)
      	at com.atlassian.bitbucket.scm.git.command.GitCommandExitHandler.onError(GitCommandExitHandler.java:208)
      	at com.atlassian.bitbucket.scm.DefaultCommandExitHandler.onExit(DefaultCommandExitHandler.java:32)
      	at com.atlassian.stash.internal.scm.git.command.fetch.FetchExitHandler.onExit(FetchExitHandler.java:39)
      	at com.atlassian.bitbucket.scm.BaseCommand.callExitHandler(BaseCommand.java:214)

      When disk space is exhausted:

      fatal: write error: No space left on device
      fatal: index-pack failed
      

      Please note, due to disk space exhaustion logs can be incomplete

      Workaround

      Increase plugin.mirroring.remote.command.timeout.idle=1800 to a larger number to accommodate your repository.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              ysun Yingran Sun
              Reporter:
              ephillips@atlassian.com Edward
              Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: