Details
-
Bug
-
Resolution: Fixed
-
Highest
-
5.16.0, 7.1.2
-
5
-
Severity 3 - Minor
-
26
-
Description
Issue Summary
When the Bitbucket GitGarbageTruck thread attempts to prune stale git objects on Windows Bitbucket instances, the attempt to delete the file fails because git places a read only lock on all git objects - preventing the Java process from being able to delete the file.
Steps to Reproduce
- Create a new project/repository
- Push changes such that a temporary pack file is eventually created that needs to be pruned
- Wait the required amount of time for the pack to be considered stale
Issue can also be reproduced by creating a file with the name tmp_pack_TEST in the ...\objects\pack directory, and manually set the read only lock and last modified date to well past the garbage collection threshold by executing the following Powershell commands within a repository's pack directory:
echo "test" > tmp_pack_TEST $(Get-Item tmp_pack_TEST).lastwritetime=$(Get-Date "01/01/1999 12:00 am") Get-ItemProperty -Path tmp_pack_TEST | Select-Object IsReadOnly Set-ItemProperty -Path tmp_pack_TEST -Name IsReadOnly -Value $true Get-ItemProperty -Path tmp_pack_TEST | Select-Object IsReadOnly
Expected Results
The stale tmp pack file, along with the temporary pack directory, are successfully deleted
Actual Results
Java is unable to delete the git object file, as git puts a read only lock on git objects so that only git processes are capable of manipulating these files. The following exception is then thrown in the logs at time of garbage collection, and the file is not removed:
2020-03-11 15:12:50,860 WARN [git:gc:thread-3] c.a.s.i.scm.git.gc.GitGarbageTruck <Repository Name>: Stale entries in the objects/ directory could not be pruned
java.nio.file.AccessDeniedException: C:\atlassian\ApplicationData\Stash\shared\data\repositories\272\objects\incoming-a26820\pack\tmp_pack_yk5Km9
at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:274)
at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
at java.base/java.nio.file.Files.delete(Files.java:1141)
at com.atlassian.stash.internal.scm.git.gc.GcStaleFileVisitor.prune(GcStaleFileVisitor.java:188)
at com.atlassian.stash.internal.scm.git.gc.GcStaleFileVisitor.visitFile(GcStaleFileVisitor.java:167)
at com.atlassian.stash.internal.scm.git.gc.GcStaleFileVisitor.visitFile(GcStaleFileVisitor.java:35)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2724)
at java.base/java.nio.file.Files.walkFileTree(Files.java:2796)
at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck$GcOperation.pruneStaleObjects(GitGarbageTruck.java:562)
at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck$GcOperation.perform(GitGarbageTruck.java:408)
at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck$GcOperation.perform(GitGarbageTruck.java:365)
at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck.lambda$withLock$3(GitGarbageTruck.java:296)
at com.atlassian.stash.internal.scm.git.DefaultGitRepositoryLayout.withLock(DefaultGitRepositoryLayout.java:181)
at com.atlassian.stash.internal.scm.git.DefaultGitRepositoryLayout.editFile(DefaultGitRepositoryLayout.java:96)
at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck.withLock(GitGarbageTruck.java:281)
at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck.withLock(GitGarbageTruck.java:285)
at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck.collectGarbage(GitGarbageTruck.java:197)
at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck$GcBootstrapper.runGc(GitGarbageTruck.java:345)
at com.atlassian.stash.internal.scm.git.gc.GitGarbageTruck$GcBootstrapper.run(GitGarbageTruck.java:329)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.lang.Thread.run(Thread.java:834)
... 1 frame trimmed
Workaround
Go and manually delete the file on the file-system, or manually remove the read-only file lock - which then allows Java to successfully delete the file/directory. Some example Powershell to remove the read only lock from any tmp pack files can be set up as a scheduled task to automatically remove these locks:
$Path = "C:\atlassian\ApplicationData\Stash\shared\data\repositories" $Files = Get-ChildItem $Path -Recurse | Where-Object { $_.FullName -match 'tmp_pack*' } ForEach ($File in $Files) { Write-Host "File: " $File "IsReadOnly: " $File.IsReadOnly if ($File.Attributes -ne "Directory" -and $File.Attributes -ne"Directory, Archive") { if ($File.IsReadOnly -eq $true ) { try { Set-ItemProperty -path $File.FullName -name IsReadOnly -value $false write-host "file:" $File.FullName "is now false read only" -foregroundcolor "magenta" } catch [Exception] { Write-Host "Error at file " $Path "\" $File Write-Host $_.Exception.Message } } } }
Attachments
Issue Links
- relates to
-
-
- Closed
-