When Confluence starts up, it verifies schemas for all tables with active objects (AO), and modifies / adds fields if required. For example, if a new version of a plugin is installed, and this plugin introduces a new field in AO class, the corresponding table will be updated and new column will be created.
For MySQL, migration procedure has a bug and updates several tables on every boot like this:
It is not a serious issue for Confluence with one node (it only increases boot time), but in case of 2+ nodes it could lead Confluence to deadlock. Example:
- The first node is executing slow query (which takes a lot of time) on AO_BAF3AA_AOINLINE_TASK table
- The second node is being restarted and is trying to modify AO_BAF3AA_AOINLINE_TASK. But this operation is blocked while the first node does not finish all SQL queries on AO_BAF3AA_AOINLINE_TASK
- The first node tries to run more SQL queries on AO_BAF3AA_AOINLINE_TASK, but all of them are blocked by DDL operation on the second node.
As a result, all threads could be blocked.
Confluence should not run any DDL operations on boot if tables do not require changes.
It happens in SchemaGenerator.migrate