-
Bug
-
Resolution: Fixed
-
High
-
2.3.1
-
None
Since upgrading to svnkit 1.3 there are svn authentication file access problems during svn authentication:
file not found
com.atlassian.bamboo.repository.RepositoryException: Unable to retrieve source code to 'null' for 'SECURITY-TRUNK': svn: Cannot read from 'C:\Documents and Settings\Default User\Application Data\Subversion\auth\svn.simple\2c897b0a1ee600f4f74b2ee6c4fe585b': C:\Documents and Settings\Default User\Application Data\Subversion\auth\svn.simple\2c897b0a1ee600f4f74b2ee6c4fe585b (The system cannot find the file specified) at com.atlassian.bamboo.repository.svn.SvnRepository.retrieveSourceCode(SvnRepository.java:416) ... Caused by: java.io.FileNotFoundException: C:\Documents and Settings\Default User\Application Data\Subversion\auth\svn.simple\2c897b0a1ee600f4f74b2ee6c4fe585b (The system cannot find the file specified) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.(Unknown Source) at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.createFileInputStream(SVNFileUtil.java:1339) at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.openFileForReading(SVNFileUtil.java:1325)
access denied
com.atlassian.bamboo.repository.RepositoryException: Build 'WFJWIZCOMP-TRUNK' failed to check SVN repository at com.atlassian.bamboo.repository.svn.SvnRepository.collectChangesSinceLastBuild(SvnRepository.java:257) ... Caused by: org.tmatesoft.svn.core.SVNException: svn: Cannot read from 'C:\Users\bamboo\AppData\Roaming\Subversion\auth\svn.simple\c06ce622ee195a0f7777879507af2761': C:\Users\bamboo\AppData\Roaming\Subversion\auth\svn.simple\c06ce622ee195a0f7777879507af2761 (Zugriff verweigert) at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:82) ... Caused by: java.io.FileNotFoundException: C:\Users\bamboo\AppData\Roaming\Subversion\auth\svn.simple\c06ce622ee195a0f7777879507af2761 (Zugriff verweigert)
[BAM-4517] svnkit authentication fails due to authentication file access problems to local cache on all platforms
I get the same issue on Linux (RHEL, 32 bit JVM, with Bamboo 2.3.1), here is my stack trace:
com.atlassian.bamboo.repository.RepositoryException: Build 'XXX-XXX' failed to check SVN repository
at com.atlassian.bamboo.repository.svn.SvnRepository.collectChangesSinceLastBuild(SvnRepository.java:257)
at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectChangesSinceLastBuild(DefaultChangeDetectionManager.java:56)
at com.atlassian.bamboo.v2.trigger.ChangeDetectionListenerAction.process(ChangeDetectionListenerAction.java:65)
at com.atlassian.bamboo.build.DefaultBuildExecutionManager.tryToDetectAndBuild(DefaultBuildExecutionManager.java:86)
at com.atlassian.bamboo.v2.trigger.ChangeDetectionListener.handleEvent(ChangeDetectionListener.java:41)
at com.atlassian.bamboo.event.BambooEventManager$2.run(BambooEventManager.java:46)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at com.atlassian.bamboo.build.pipeline.concurrent.NamedThreadFactory$1.run(NamedThreadFactory.java:34)
at edu.emory.mathcs.backport.java.util.concurrent.helpers.ThreadHelpers$1.run(ThreadHelpers.java:34)
at java.lang.Thread.run(Unknown Source)
Caused by: org.tmatesoft.svn.core.SVNException: svn: Cannot read from '/home/bamboo/.subversion/auth/svn.simple/dd81f60b72a6b058e18716c4185af149': /home/bamboo/.subversion/auth/svn.simple/dd81f60b72a6b058e18716c4185af149 (No such file or directory)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:82)
at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.openFileForReading(SVNFileUtil.java:1329)
at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.openFileForReading(SVNFileUtil.java:1317)
at org.tmatesoft.svn.core.internal.wc.SVNWCProperties.setPropertyValue(SVNWCProperties.java:288)
at org.tmatesoft.svn.core.internal.wc.SVNWCProperties.setPropertyValue(SVNWCProperties.java:276)
at org.tmatesoft.svn.core.internal.wc.DefaultSVNAuthenticationManager$PersistentAuthenticationProvider.saveAuthentication(DefaultSVNAuthenticationManager.java:628)
at org.tmatesoft.svn.core.internal.wc.DefaultSVNAuthenticationManager.acknowledgeAuthentication(DefaultSVNAuthenticationManager.java:217)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:594)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:273)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:261)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:516)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:98)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:999)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getLatestRevision(DAVRepository.java:178)
at com.atlassian.bamboo.repository.svn.SvnRepository.detectCommitsForUrl(SvnRepository.java:305)
at com.atlassian.bamboo.repository.svn.SvnRepository.collectChangesSinceLastBuild(SvnRepository.java:223)
... 10 more
Caused by: java.io.FileNotFoundException: /home/bamboo/.subversion/auth/svn.simple/dd81f60b72a6b058e18716c4185af149 (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(Unknown Source)
at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.createFileInputStream(SVNFileUtil.java:1339)
at org.tmatesoft.svn.core.internal.wc.SVNFileUtil.openFileForReading(SVNFileUtil.java:1325)
... 24 more
I also have gotten a few of these:
com.atlassian.bamboo.repository.RepositoryException: Build 'XXX-XXX' failed to check SVN repository
at com.atlassian.bamboo.repository.svn.SvnRepository.collectChangesSinceLastBuild(SvnRepository.java:257)
at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectChangesSinceLastBuild(DefaultChangeDetectionManager.java:56)
at com.atlassian.bamboo.v2.trigger.ChangeDetectionListenerAction.process(ChangeDetectionListenerAction.java:65)
at com.atlassian.bamboo.build.DefaultBuildExecutionManager.tryToDetectAndBuild(DefaultBuildExecutionManager.java:86)
at com.atlassian.bamboo.v2.trigger.ChangeDetectionListener.handleEvent(ChangeDetectionListener.java:41)
at com.atlassian.bamboo.event.BambooEventManager$2.run(BambooEventManager.java:46)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at com.atlassian.bamboo.build.pipeline.concurrent.NamedThreadFactory$1.run(NamedThreadFactory.java:34)
at edu.emory.mathcs.backport.java.util.concurrent.helpers.ThreadHelpers$1.run(ThreadHelpers.java:34)
at java.lang.Thread.run(Unknown Source)
Caused by: org.tmatesoft.svn.core.SVNException: svn: invalid properties file format
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:82)
at org.tmatesoft.svn.core.internal.wc.SVNErrorManager.error(SVNErrorManager.java:69)
at org.tmatesoft.svn.core.internal.wc.SVNWCProperties.copyProperties(SVNWCProperties.java:466)
at org.tmatesoft.svn.core.internal.wc.SVNWCProperties.setPropertyValue(SVNWCProperties.java:291)
at org.tmatesoft.svn.core.internal.wc.SVNWCProperties.setPropertyValue(SVNWCProperties.java:276)
at org.tmatesoft.svn.core.internal.wc.DefaultSVNAuthenticationManager$PersistentAuthenticationProvider.saveAuthentication(DefaultSVNAuthenticationManager.java:628)
at org.tmatesoft.svn.core.internal.wc.DefaultSVNAuthenticationManager.acknowledgeAuthentication(DefaultSVNAuthenticationManager.java:217)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:594)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:273)
at org.tmatesoft.svn.core.internal.io.dav.http.HTTPConnection.request(HTTPConnection.java:261)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.exchangeCapabilities(DAVConnection.java:516)
at org.tmatesoft.svn.core.internal.io.dav.DAVConnection.open(DAVConnection.java:98)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.openConnection(DAVRepository.java:999)
at org.tmatesoft.svn.core.internal.io.dav.DAVRepository.getLatestRevision(DAVRepository.java:178)
at com.atlassian.bamboo.repository.svn.SvnRepository.detectCommitsForUrl(SvnRepository.java:305)
at com.atlassian.bamboo.repository.svn.SvnRepository.collectChangesSinceLastBuild(SvnRepository.java:223)
... 10 more
Caused by: java.io.IOException: invalid properties file format
at org.tmatesoft.svn.core.internal.wc.SVNWCProperties.readLength(SVNWCProperties.java:519)
at org.tmatesoft.svn.core.internal.wc.SVNWCProperties.copyProperties(SVNWCProperties.java:438)
... 23 more
I recently upgraded from a 2.0.x version to 2.3.1, and I also went from a url like http://hostname to http://hostname.domain.com which caused a few other issues about "detecting" changes, but were solved with a manual build (so it would recheck it out).
Ulrich Kuhnhardt [Atlassian]
added a comment - sysinternals process monitor is a great tool to monitor the file access - see screenshot how to set it up for subversion authentication cache.
After activating the monitor reproduce the problem, save and attach process monitor log file.
Ulrich Kuhnhardt [Atlassian]
added a comment - sysinternals process monitor is a great tool to monitor the file access - see screenshot how to set it up for subversion authentication cache.
After activating the monitor reproduce the problem, save and attach process monitor log file. Ulrich Kuhnhardt [Atlassian]
added a comment - I was not able to reproduce this issue - must be on a win platform...
will keep trying.
attached the latest svnkit zip
Ulrich Kuhnhardt [Atlassian]
added a comment - I was not able to reproduce this issue - must be on a win platform...
will keep trying.
attached the latest svnkit zip Ulrich Kuhnhardt [Atlassian]
added a comment - I have filed a bug with svnkit http://svnkit.com/tracker/view.php?id=331
see also
https://support.atlassian.com/browse/BSP-2125
https://support.atlassian.com/browse/BSP-2059
https://support.atlassian.com/browse/BSP-2130
email from svnkit
From the stack trace in the issue provided I do not see that there are
two threads competing for the access to the file. However, there are
good news - at least I think I understand what could be the reason of
the problem and could suggest ways to try to fix it.
First, I manage to reproduce "access denined" on Windows with the
following test:
1. create file with some contents.
2. delete it.
3. goto 1.
In case above is repeated long enough time, launching external viewer
(another process) on that very file causes test to fail with "access
denined" on attempt to write to the file. The reason is that reader
process (viewer) blocks file deletion and new file couldn't be created
on place of a file that is in "deletion queue", but not yet deleted.
So, the first advice would be to make sure that there is no other
process running in background and accessing files on the Bamboo server.
In particular these could be antiviral software, Tortoise daemon
(TSVNCache.exe), probably some other monitors.
Then, in the user's report exception occurs in attempt to read a file -
this I didn't manage to reproduce. My colleague said, that even in a
context of a single thread on Windows, deleting a file adds it to the
system "deletion queue" and next attempt to read the file (which is what
happens in SVNKit) may result in "access denied" because file at that
moment exists, but is in "pending deletion" state (like Schredinger's
cat 
. This is not easy to reproduce, but again some users rarely have
encountered that problem with SVNKit in other applications (on Windows
only).
Also I've changed the code to avoid calling "delete" on a file. Instead
rename operation is performed so that old file is replaced with the new
contents. One may think that target file is still "deleted" by rename
operation, but surprisingly my test (see above) worked fine
independently on whether I used external viewer to open the file or not.
That means that rename is not just "delete+write" sequence ran atomically.
So, second advice would be to get latest version of SVNKit from 1.3.x
branch and ask user to try it.
Another thing that somehow related to that issue is usage of
"sun.io.useCanonCaches" system property. By default it is set to "true",
but AFAIK Tomcat sets it to "false". If possible, please make sure that
Bamboo also sets it to "false" - disabling these cache makes JDK file
system API work much better when files could be modified by different
processes.
Last, but not least, is the fact that Bamboo uses
DefaultSVNAuthenticationManager. Personally, I think that it would be
not only safer, but also more "correct" to use custom implementation of
ISVNAuthenticationManager - this will allow to store all credentials
within Bamboo application, not as something belonging to the user. It
would be also much easier to configure repository access credentials
from Bamboo user interface and there will be no problems like one we're
discussing currently (though of course it should be fixed!).
Alexander Kitaev,
TMate Software,
http://svnkit.com/ - Java [Sub]Versioning Library!
Ulrich Kuhnhardt [Atlassian]
added a comment - I have filed a bug with svnkit http://svnkit.com/tracker/view.php?id=331
see also
https://support.atlassian.com/browse/BSP-2125
https://support.atlassian.com/browse/BSP-2059
https://support.atlassian.com/browse/BSP-2130
email from svnkit
From the stack trace in the issue provided I do not see that there are
two threads competing for the access to the file. However, there are
good news - at least I think I understand what could be the reason of
the problem and could suggest ways to try to fix it.
First, I manage to reproduce "access denined" on Windows with the
following test:
1. create file with some contents.
2. delete it.
3. goto 1.
In case above is repeated long enough time, launching external viewer
(another process) on that very file causes test to fail with "access
denined" on attempt to write to the file. The reason is that reader
process (viewer) blocks file deletion and new file couldn't be created
on place of a file that is in "deletion queue", but not yet deleted.
So, the first advice would be to make sure that there is no other
process running in background and accessing files on the Bamboo server.
In particular these could be antiviral software, Tortoise daemon
(TSVNCache.exe), probably some other monitors.
Then, in the user's report exception occurs in attempt to read a file -
this I didn't manage to reproduce. My colleague said, that even in a
context of a single thread on Windows, deleting a file adds it to the
system "deletion queue" and next attempt to read the file (which is what
happens in SVNKit) may result in "access denied" because file at that
moment exists, but is in "pending deletion" state (like Schredinger's
cat . This is not easy to reproduce, but again some users rarely have
encountered that problem with SVNKit in other applications (on Windows
only).
Also I've changed the code to avoid calling "delete" on a file. Instead
rename operation is performed so that old file is replaced with the new
contents. One may think that target file is still "deleted" by rename
operation, but surprisingly my test (see above) worked fine
independently on whether I used external viewer to open the file or not.
That means that rename is not just "delete+write" sequence ran atomically.
So, second advice would be to get latest version of SVNKit from 1.3.x
branch and ask user to try it.
Another thing that somehow related to that issue is usage of
"sun.io.useCanonCaches" system property. By default it is set to "true",
but AFAIK Tomcat sets it to "false". If possible, please make sure that
Bamboo also sets it to "false" - disabling these cache makes JDK file
system API work much better when files could be modified by different
processes.
Last, but not least, is the fact that Bamboo uses
DefaultSVNAuthenticationManager. Personally, I think that it would be
not only safer, but also more "correct" to use custom implementation of
ISVNAuthenticationManager - this will allow to store all credentials
within Bamboo application, not as something belonging to the user. It
would be also much easier to configure repository access credentials
from Bamboo user interface and there will be no problems like one we're
discussing currently (though of course it should be fixed!).
Alexander Kitaev,
TMate Software,
http://svnkit.com/ - Java [Sub] Versioning Library!
Brydie to investigate our options for the 2.4 timeframe. That is anything we can do this week with low risk.