Uploaded image for project: 'Bitbucket Server'
  1. Bitbucket Server
  2. BSERV-12319

Pruning stale git objects fails on Windows operating systems

    XMLWordPrintable

    Details

      Description

      Issue Summary

      When the Bitbucket GitGarbageTruck thread attempts to prune stale git objects on Windows Bitbucket instances, the attempt to delete the file fails because git places a read only lock on all git objects - preventing the Java process from being able to delete the file.

      Steps to Reproduce

      1. Create a new project/repository
      2. Push changes such that a temporary pack file is eventually created that needs to be pruned
      3. Wait the required amount of time for the pack to be considered stale

      Issue can also be reproduced by creating a file with the name tmp_pack_TEST in the ...\objects\pack directory, and manually set the read only lock and last modified date to well past the garbage collection threshold by executing the following Powershell commands within a repository's pack directory:

      echo "test" > tmp_pack_TEST
      $(Get-Item tmp_pack_TEST).lastwritetime=$(Get-Date "01/01/1999 12:00 am")
      Get-ItemProperty -Path tmp_pack_TEST | Select-Object IsReadOnly
      Set-ItemProperty -Path tmp_pack_TEST -Name IsReadOnly -Value $true
      Get-ItemProperty -Path tmp_pack_TEST | Select-Object IsReadOnly
      

      Expected Results

      The stale tmp pack file, along with the temporary pack directory, are successfully deleted

      Actual Results

      Java is unable to delete the git object file, as git puts a read only lock on git objects so that only git processes are capable of manipulating these files. The following exception is then thrown in the logs at time of garbage collection, and the file is not removed:

      2020-03-11 15:12:50,860 WARN  [git:gc:thread-3]  c.a.s.i.scm.git.gc.GitGarbageTruck <Repository Name>: Stale entries in the objects/ directory could not be pruned
      java.nio.file.AccessDeniedException: C:\atlassian\ApplicationData\Stash\shared\data\repositories\272\objects\incoming-a26820\pack\tmp_pack_yk5Km9
          at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
          at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
          at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
          at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:274)
          at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
          at java.base/java.nio.file.Files.delete(Files.java:1141)
          at com.atlassian.stash.internal.scm.git.gc.GcStaleFileVisitor.prune(GcStaleFileVisitor.java:188)
          at com.atlassian.stash.internal.scm.git.gc.GcStaleFileVisitor.visitFile(GcStaleFileVisitor.java:167)
          at com.atlassian.stash.internal.scm.git.gc.GcStaleFileVisitor.visitFile(GcStaleFileVisitor.java:35)
          at java.base/java.nio.file.Files.walkFileTree(Files.java:2724)
          at java.base/java.nio.file.Files.walkFileTree(Files.java:2796)
          at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck$GcOperation.pruneStaleObjects(GitGarbageTruck.java:562)
          at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck$GcOperation.perform(GitGarbageTruck.java:408)
          at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck$GcOperation.perform(GitGarbageTruck.java:365)
          at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck.lambda$withLock$3(GitGarbageTruck.java:296)
          at com.atlassian.stash.internal.scm.git.DefaultGitRepositoryLayout.withLock(DefaultGitRepositoryLayout.java:181)
          at com.atlassian.stash.internal.scm.git.DefaultGitRepositoryLayout.editFile(DefaultGitRepositoryLayout.java:96)
          at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck.withLock(GitGarbageTruck.java:281)
          at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck.withLock(GitGarbageTruck.java:285)
          at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck.collectGarbage(GitGarbageTruck.java:197)
          at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck$GcBootstrapper.runGc(GitGarbageTruck.java:345)
          at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck$GcBootstrapper.run(GitGarbageTruck.java:329)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.lang.Thread.run(Thread.java:834)
          ... 1 frame trimmed
      

      Workaround

      Go and manually delete the file on the file-system, or manually remove the read-only file lock - which then allows Java to successfully delete the file/directory. Some example Powershell to remove the read only lock from any tmp pack files can be set up as a scheduled task to automatically remove these locks:

      $Path = "C:\atlassian\ApplicationData\Stash\shared\data\repositories" 
      $Files = Get-ChildItem $Path -Recurse | Where-Object { $_.FullName -match 'tmp_pack*' }
      ForEach ($File in $Files) 
      {
         Write-Host "File: " $File "IsReadOnly: " $File.IsReadOnly 
         if ($File.Attributes -ne "Directory" -and $File.Attributes -ne"Directory, Archive")
         {
            if ($File.IsReadOnly -eq $true )
            {
                try  
                {
                     Set-ItemProperty -path $File.FullName -name IsReadOnly -value $false 
                     write-host "file:" $File.FullName "is now false read only"  -foregroundcolor "magenta"
                }
                catch [Exception] 
                { 
                     Write-Host "Error at file " $Path "\" $File 
                     Write-Host $_.Exception.Message
                }
            }
          } 
      }
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              bturner Bryan Turner
              Reporter:
              eslaughter@atlassian.com Evan Slaughter
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: