Introduce DatabaseUpgradeLock mechanism to prevent accidentally upgrading live instance DB's from test environments

Problem Definition

It's common for users to set up test instances of new versions of Bitbucket Server by copying their $BITBUCKET_HOME to a test instance, not realising that the test instance will still be pointing to the production database

In these cases, the newer Bitbucket Server instance will upgrade the schema of the production instance, whilst the production instance is running.

In either case this typically results in a customer having to contact support to diagnose the issue, and then needing to either roll back their instance, or upgrade production.

Steps to reproduce

1. Start up a new Stash instance and connect it to an external DB (e.g. postgres)
2. Whilst it's running, install Bitbucket Server on a new server, and copy over the contents of $STASH_HOME from the live instance to $BITBUCKET_HOME on your new test instance
3. Start Bitbucket Server, whilst Stash is still running

Bitbucket server upgrades the DB, and the Stash instance starts exhibiting failures every time a DB inconsistency is found

Suggested Solution

Introduce some mechanism whereby database upgrade tasks cannot be triggered whilst an instance of Bitbucket Server is actively running on that database. For example some kind of lock that is set on startup and cleared on shutdown, but that only blocks upgrade tasks (so if the app was killed without unsetting the lock, normal startup would not be blocked). That way, DB upgrade tasks would only be run if no other app was already using the DB, which should prevent the vast majority of cases

Workaround

• Configure the database so that connections from other IPs (not matching the production Bitbucket Server site) are blocked
• Configure the network to no allow such connections