Bryan Turner (Inactive) added a comment - 28/Aug/2014 12:33 PM Tomas, Glad to hear it. I've confirmed that there is some work pending for a 1.3.2 release of the backup client, but it's focused on handling some edge cases we've uncovered. Any backups you take with 1.3.1 and Stash 3.2.4 should restore without issue, however. Best regards, Bryan Turner Atlassian Stash

Tomas Vala added a comment - 28/Aug/2014 11:10 AM Hi Bryan, thanks again for detailed response, much appreciated. I can confirm running Stash Backup Client 1.3.1 against Stash 3.2.4 with -noverify option as advised succeeded to finish normally and produced a backup file. Cheers, Tomas

Bryan Turner (Inactive) added a comment - 28/Aug/2014 10:27 AM Tomas, Correct, backup client 1.1 is not compatible with Stash 3.2 due to the new home directory layout. 1.3, as documented on Marketplace, adds support for the new layout. 1.3 is compatible with all previous versions of Stash as well. Not all new releases of Stash require a new backup client; we do our best to keep things compatible from release to release. However, as the product moves forward, occasionally the home directory layout or database schema require updates which also require matching updates to the backup client. As noted above, you should be able to run the 1.3 client using the -noverify flag, and it can backup Stash 3.2. Please ensure you are using 1.3.1, not 1.3.0. Also, note that bugs in Stash 3.2 and 3.2.2 result in unrestorable backups. Stash 3.2.4 should include fixes for that, so Stash 3.2.4 and backup client 1.3.1 is the current sweet spot. I believe version 1.3.2 of the backup client will also be released soon fixing some additional issues we've turned up in internal testing. I'll confirm that with the developer who has been working on it and update this issue. Best regards, Bryan Turner Atlassian Stash

Tomas Vala added a comment - 28/Aug/2014 9:35 AM Hi Bryan, thanks for the comment. Understood. A little off topic. Am I right assuming Stash Backup Client 1.1.0 is no longer suitable for performing backup of Stash 3.2.x? I have been using it no problem up to recently but currently (after upgrading Stash to the most recent version) it keeps failing like this: 2014-08-28 11:31:13,701 INFO Initializing 2014-08-28 11:31:16,001 ERROR A backup could not be created. Reason: The configured STASH_HOME, /opt/atlassian/stash_data, does not appear to be a valid home directory. Creating a backup requires direct access to the home directory for the running instance to allow it to interact with the repositories. com.atlassian.stash.internal.backup.client.BackupException: The configured STASH_HOME, /opt/atlassian/stash_data, does not appear to be a valid home directory. Creating a backup requires direct access to the home directory for the running instance to allow it to interact with the repositories. at com.atlassian.stash.internal.backup.client.layout.DefaultStashHome.verify(DefaultStashHome.groovy:53) ~[stash-backup-core-1.1.0.jar:na] at com.atlassian.stash.internal.backup.client.DefaultBackupClient.backup(DefaultBackupClient.groovy:44) ~[stash-backup-client.jar:1.1.0] ... 2 more frames available in the log file

Bryan Turner (Inactive) added a comment - 28/Aug/2014 9:14 AM Tomas, Thanks for your report. As noted previously, this is a bug in Java, not in the backup client. The backup client is using libraries, like Groovy and Javassist, which run afoul of a bug in Java 7u65 and 8u9 and higher. 1.3.0 uses newer versions of those libraries than 1.1.0 does, and is also compiled as Java 7 bytecode (1.1.0 still supports Java 6). Oracle's JIRA indicates that 8u20 includes a fix for the issue, though I haven't yet been able to confirm that. So far there is no version of Java 7 which includes a backport of the fix. Best regards, Bryan Turner Atlassian Stash

Tomas Vala added a comment - 28/Aug/2014 8:48 AM - edited Same problem here with Stash Backup Client 1.3.0 and 1.3.1 on Oracle Java 1.7.0_65 and 1.7.0_67 . Stash Backup Client 1.1.0 runs fine in the same environment. [root@stash atlassian]# which java /usr/java/ default /bin/java [root@stash atlassian]# java -version java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

Deric Lee (Inactive) added a comment - 13/Aug/2014 3:24 PM Error thrown for future references Exception in thread "main" java.lang.VerifyError: Bad <init> method call from inside of a branch Exception Details: Location: com/atlassian/stash/internal/backup/client/spring/SpringMain$ClassPathXmlWithSingletonsApplicationContext.<init>(Ljava/lang/String;Ljava/util/Map;)V @157: invokespecial Reason: Error exists in the bytecode Bytecode: 0000000: b800 184e 05bd 001a 5903 04bd 001a 5903 0000010: 2b53 b800 2012 22b8 0026 c000 2212 22b8 0000020: 002a 5359 0403 b800 3053 5910 ff12 04b8 0000030: 0034 2a5f ab00 0000 0000 016e 0000 000a 0000040: aad3 b1ff 0000 005c bafc 019a 0000 006f 0000050: bd55 5f31 0000 009a 0000 9b75 0000 00c2 0000060: 1407 608a 0000 00cb 260d d8f3 0000 00e7 0000070: 3231 c38a 0000 0106 3382 f81c 0000 011b 0000080: 560d c231 0000 0130 5a72 031e 0000 0151 0000090: 5f5a 5903 32b8 003a c000 3c5f 57b7 003f 00000a0: a701 0c5f 5a59 0332 1222 b800 42c0 0022 00000b0: 5f59 0432 b800 46c0 0048 5f59 0532 124a 00000c0: b800 42c0 004a 5f57 b700 4da7 00e1 5f5a 00000d0: 5903 3212 22b8 0042 c000 225f 5904 32b8 00000e0: 0053 5f59 0532 124a b800 42c0 004a 5f57 00000f0: b700 56a7 00b9 5f5a 57b7 0059 a700 b05f 0000100: 5a59 0332 1222 b800 42c0 0022 5f59 0432 0000110: b800 535f 57b7 005c a700 945f 5a59 0332 0000120: 1222 b800 42c0 0022 5f59 0432 b800 46c0 0000130: 0048 5f57 b700 5fa7 0075 5f5a 5903 3212 0000140: 22b8 0042 c000 225f 57b7 0062 a700 605f 0000150: 5a59 0332 124a b800 42c0 004a 5f57 b700 0000160: 65a7 004b 5f5a 5903 3212 22b8 0042 c000 0000170: 225f 5904 3212 4ab8 0042 c000 4a5f 57b7 0000180: 0068 a700 2a5f 5a59 0332 b800 3ac0 003c 0000190: 5f59 0432 b800 46c0 0048 5f57 b700 6ba7 00001a0: 000d bb00 6d59 126f b700 70bf 572a b600 00001b0: 743a 0419 042a 5fb5 0076 1904 572c 3a05 00001c0: 1905 1278 b800 42c0 0078 2a5f b500 7a19 00001d0: 0557 b1 Stackmap Table: full_frame(@144,{UninitializedThis,Object[#60],Object[#120],Object[#128]},{Object[#130],UninitializedThis}) full_frame(@163,{UninitializedThis,Object[#60],Object[#120],Object[#128]},{Object[#130],UninitializedThis}) full_frame(@206,{UninitializedThis,Object[#60],Object[#120],Object[#128]},{Object[#130],UninitializedThis}) full_frame(@246,{UninitializedThis,Object[#60],Object[#120],Object[#128]},{Object[#130],UninitializedThis}) full_frame(@255,{UninitializedThis,Object[#60],Object[#120],Object[#128]},{Object[#130],UninitializedThis}) full_frame(@283,{UninitializedThis,Object[#60],Object[#120],Object[#128]},{Object[#130],UninitializedThis}) full_frame(@314,{UninitializedThis,Object[#60],Object[#120],Object[#128]},{Object[#130],UninitializedThis}) full_frame(@335,{UninitializedThis,Object[#60],Object[#120],Object[#128]},{Object[#130],UninitializedThis}) full_frame(@356,{UninitializedThis,Object[#60],Object[#120],Object[#128]},{Object[#130],UninitializedThis}) full_frame(@389,{UninitializedThis,Object[#60],Object[#120],Object[#128]},{Object[#130],UninitializedThis}) full_frame(@418,{UninitializedThis,Object[#60],Object[#120],Object[#128]},{Object[#130],UninitializedThis}) full_frame(@428,{Object[#2],Object[#60],Object[#120],Object[#128]},{Object[#130]}) at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2532) at java.lang.Class.getDeclaredConstructors(Class.java:1901) at org.codehaus.groovy.reflection.CachedClass$2$1.run(CachedClass.java:69) at java.security.AccessController.doPrivileged(Native Method) at org.codehaus.groovy.reflection.CachedClass$2.initValue(CachedClass.java:66) at org.codehaus.groovy.reflection.CachedClass$2.initValue(CachedClass.java:64) at org.codehaus.groovy.util.LazyReference.getLocked(LazyReference.java:46) at org.codehaus.groovy.util.LazyReference.get(LazyReference.java:33) at org.codehaus.groovy.reflection.CachedClass.getConstructors(CachedClass.java:258) at groovy.lang.MetaClassImpl.<init>(MetaClassImpl.java:213) at groovy.lang.MetaClassImpl.<init>(MetaClassImpl.java:223) at groovy.lang.MetaClassRegistry$MetaClassCreationHandle.createNormalMetaClass(MetaClassRegistry.java:168) at groovy.lang.MetaClassRegistry$MetaClassCreationHandle.createWithCustomLookup(MetaClassRegistry.java:158) at groovy.lang.MetaClassRegistry$MetaClassCreationHandle.create(MetaClassRegistry.java:141) at org.codehaus.groovy.reflection.ClassInfo.getMetaClassUnderLock(ClassInfo.java:209) at org.codehaus.groovy.reflection.ClassInfo.getMetaClass(ClassInfo.java:241) at org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.getMetaClass(MetaClassRegistryImpl.java:255) at org.codehaus.groovy.runtime.InvokerHelper.getMetaClass(InvokerHelper.java:859) at org.codehaus.groovy.runtime.callsite.CallSiteArray.createCallConstructorSite(CallSiteArray.java:84) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:194) at com.atlassian.stash.internal.backup.client.spring.SpringMain.run(SpringMain.groovy:26) at com.atlassian.stash.internal.backup.client.spring.SpringMain.run(SpringMain.groovy) at com.atlassian.stash.internal.backup.client.spring.SpringMain$run.call(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) at com.atlassian.stash.internal.backup.client.BackupMain.main(BackupMain.groovy:38)

mplaneck added a comment - 05/Aug/2014 6:57 AM Thank you guys for diggin' into it! I'll stay at the -noverify option for now - it seems to work fine best regards Marc

Bryan Turner (Inactive) added a comment - 05/Aug/2014 4:32 AM Bradley, That is indeed the issue. I double-checked the test I performed prior to releasing 1.3.0, running a full backup and restore against in-development Stash 3.3, and both Java 7u60 and Java 8u5 work without issue (and are the versions I tested with prior to releasing 1.3.0 last week). I also tested Java 7u67, which some sources claim fixes the issue, and confirmed that that's not the case (it's still broken). Looks like a downgrade or -noverify are the only options at the moment. I'd recommend caution with Stash. Javassist, which is used by Hibernate for proxying, also has issues on Java 7u65 and 8u11 . I've run Stash's full suite of unit and functional tests without issue, suggesting the system should be stable, but quirks may arise. As for this issue, unfortunately there's not anything I can do. I'll monitor for whether Groovy, similar to ZeroTurnaround, introduces a workaround that fixes the issue and, if they do, I'll release a new version of the backup client with that. Best regards, Bryan Turner Atlassian Stash

Bradley Baetz added a comment - 05/Aug/2014 3:11 AM Ran into this myself on my local dev box running Fedora 20, with: java version "1.7.0_65" OpenJDK Runtime Environment (fedora-2.5.1.3.fc20-x86_64 u65-b17) OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode) I think that this is a JDK bug - https://bugs.openjdk.java.net/browse/JDK-8051012 - from the groovy stack trace / https://jira.codehaus.org/browse/GROOVY-6951 (this also breaks powermock and a few other dev tools) Note that stash itself seems to run fine; I was just testing the backups before upgrading our prod 3.1 version to 3.2.