Summary

Restoring a backup of Stash (any 2.x or 3.x) version using the Bitbucket Server Backup Client 2.0+ recreates some columns with an incorrect type. The incorrect column types are created on restore regardless of the database Stash was using when the backup was taken, and regardless of whether Stash was backed up with the Stash Backup Client (1.x) or the Bitbucket Server Backup Client (2.x). The important detail is that the restore is done with the Bitbucket Server Backup Client (2.x).

Steps to Reproduce

1. Backup a Stash instance with any backup client
2. Restore the backup with the latest backup client

Expected Results

When checking the AO_02A6C0_REJECTED_REF table in the restored database, it should look like this:

Actual Results

REF_ID and REF_DISPLAY_ID are created with the ntext instead of varchar(767) (on SQL Server; on other databases they will be created as that DBMS's CLOB type).

The exception below is thrown and logged in the atlassian-bitbucket.log or atlassian-stash.log file of an instance experiencing this issue:

2015-12-16 15:36:27,956 ERROR [http-nio-7990-exec-1] jdoe @1RR82X8x936x444x0 d15xgs 0.0.0.0 "GET /rest/sync/latest/projects/KEY/repos/slug HTTP/1.1" c.a.s.i.r.e.DefaultUnhandledExceptionMapperHelper Unhandled exception while processing REST request: "GET /rest/sync/latest/projects/KEY/repos/slug HTTP/1.1"
com.atlassian.activeobjects.internal.ActiveObjectsInitException: bundle [com.atlassian.bitbucket.server.bitbucket-repository-ref-sync]
Caused by: com.atlassian.activeobjects.internal.ActiveObjectsSqlException: There was a SQL exception thrown by the Active Objects library:
Database:
	- name:Microsoft SQL Server
	- version:10.50.1600
	- minor version:50
	- major version:10
Driver:
	- name:Microsoft JDBC Driver 4.2 for SQL Server
	- version:4.2.6420.100

com.microsoft.sqlserver.jdbc.SQLServerException: Argument data type ntext is invalid for argument 1 of len function.

This error indicates fork syncing cannot function because its database schema is incorrect.

Workaround

We've identified that the issue occurs on the ActiveObjects library we ship with our client.

We've tested it and restoring a backup with Stash Backup Client 1.9.1 (which contains ActiveObjects 0.28 as opposed to 0.29 on the latest version) works successfully.

There are a few alternatives:

• Alternative 1 (Stash 3.11 or earlier ONLY):
  • Backup and restore with a backup/restore client that contains an older version of the Active objects. In order to check that, you should check the following directory of your unzipped backup client:

    $ ls -ltra stash-backup-client-1.9.1/lib | grep active
    -rw-rw-r--@   1 tbomfim  staff    23717  6 Mar  2015 activeobjects-spi-0.28.1.jar
    -rw-rw-r--@   1 tbomfim  staff  2238224  6 Mar  2015 activeobjects-plugin-0.28.1.jar
    

• Alternative 2:
  • use the DIY backup scripts to backup and restore your instance.

• Alternative 3:
  • If the scripts above are intimidating or not an easy solution to be worked on by you (they need to be rewritten as they are written to support PostgreSQL), alternatively you could also simply:
    • Stop Stash
    • Make a cold backup of the STASH_HOME (i.e. copy the directory to another storing device)
    • Make a cold backup of the database (i.e. with your own database native backup tool)
    • additional note: the Alternative 3 should be enough for you to restore in case the restore fails. It is actually what the DIY backup does but in a more elegant approach by locking your instance and implementing all the steps described on Advanced - writing your own DIY Backup using the REST APIs.