Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-78741

Webhooks upgrade task fails when upgrading to Jira 10 if there are webhooks with more than 255 characters in the JQL filter

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: High High
    • 10.3.8, 10.7.2
    • 10.0.0, 10.1.0, 10.2.0, 10.3.0, 10.4.0, 10.5.2
    • Upgrade, Webhooks
    • 10
    • 22
    • Severity 3 - Minor
    • 154
    • Hide
      Atlassian Update – 3 Jul 2025

      Dear Customers,

      This bug is fixed in Jira 10.3.8 LTS, 10.7.2 and above versions and I confirmed the functionality to be working as expected.

      Best regards

      Tomasz Ziółkowski
      Principal Software Engineer, Jira Platform

      Show
      Atlassian Update – 3 Jul 2025 Dear Customers, This bug is fixed in Jira 10.3.8 LTS, 10.7.2 and above versions and I confirmed the functionality to be working as expected. Best regards Tomasz Ziółkowski Principal Software Engineer, Jira Platform

      Issue Summary

      As part of upgrading to Jira 10, the database schema upgrade task will fail to migrate webhooks from the old AO_4AEACD_WEBHOOK_DAO table to AO_A0B856_WEBHOOK if the JQL filter has more than 255 characters.

      This occurs since the VALUE column of the AO_A0B856_WEBHOOK_CONFIG table is VARCHAR(255). The corresponding FILTER column of the AO_4AEACD_WEBHOOK_DAO table is TEXT (unlimited length). This problem is described in JRASERVER-78654.

      Steps to Reproduce

      1. Install Jira 9 and configure a webhook with JQL defined in the "Events" section with more than 255 characters.
      2. Upgrade to an affected Jira 10 version.

      Expected Results

      The webhook upgrade task will execute successfully, and the webhooks will be displayed in the webhook menu.

      Actual Results

      1. An incomplete list of webhook configurations is found in ⚙️ (gear icon) > System > Webhooks.
      2. The webhook migration task failure is found in the startup logs during the upgrade:
      2025-03-19 18:20:14,258-0400 active-objects-init-0 INFO anonymous   [c.a.j.p.w.ao.upgrade.MigrateToWebhooks7Task] Migrating webhooks to version external atlassian-webhooks-plugin.
      2025-03-19 18:20:14,350-0400 active-objects-init-0 INFO anonymous   [c.a.j.p.w.ao.upgrade.MigrateToWebhooks7Task] Webhooks migration; 17 items to migrate
      2025-03-19 18:20:14,350-0400 active-objects-init-0 INFO anonymous   [c.a.j.p.w.ao.upgrade.MigrateToWebhooks7Task] Delegating webhooks data migration to atlassian-webhooks ao.
      2025-03-19 18:20:14,528-0400 active-objects-init-0 ERROR anonymous   [c.a.j.p.w.ao.upgrade.MigrateToWebhooks7Task] Webhooks migration failed
      com.atlassian.activeobjects.internal.ActiveObjectsSqlException: There was a SQL exception thrown by the Active Objects library:
      Database:
        - name:PostgreSQL
        - version:14.12
        - minor version:12
        - major version:14
      Driver:
        - name:PostgreSQL JDBC Driver
        - version:42.7.3
      org.postgresql.util.PSQLException: ERROR: value too long for type character varying(255)
        at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.create(EntityManagedActiveObjects.java:103)
        at com.atlassian.webhooks.internal.dao.ao.v1.WebhookV1MigrateService.lambda$migrate$0(WebhookV1MigrateService.java:45)
        at com.google.common.collect.RegularImmutableMap.forEach(RegularImmutableMap.java:300) 
      

      Note

      In case of a MSSQL Database Server, the following error will be thrown:

      2025-06-25 06:59:24,362-0400 active-objects-init-0 ERROR anonymous     [c.a.j.p.w.ao.upgrade.MigrateToWebhooks7Task] Webhooks migration failed
      com.atlassian.activeobjects.internal.ActiveObjectsSqlException: There was a SQL exception thrown by the Active Objects library:
      Database:
      	- name:Microsoft SQL Server
      	- version:16.00.1000
      	- minor version:0
      	- major version:16
      Driver:
      	- name:Microsoft JDBC Driver 9.2 for SQL Server
      	- version:9.2.1.0
      com.microsoft.sqlserver.jdbc.SQLServerException: String or binary data would be truncated.
      	at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.create(EntityManagedActiveObjects.java:104)
      	at com.atlassian.webhooks.internal.dao.ao.v1.WebhookV1MigrateService.lambda$migrate$0(WebhookV1MigrateService.java:46)
      	at com.google.common.collect.RegularImmutableMap.forEach(RegularImmutableMap.java:300)
      	at com.atlassian.webhooks.internal.dao.ao.v1.WebhookV1MigrateService.migrate(WebhookV1MigrateService.java:46)
      	at com.atlassian.webhooks.external.OsgiMigrationToV1.lambda$registerExternalMigrationTask$0(OsgiMigrationToV1.java:39)
      	at java.base/java.util.ArrayList.forEach(Unknown Source)
      	at com.atlassian.webhooks.external.OsgiMigrationToV1.lambda$registerExternalMigrationTask$1(OsgiMigrationToV1.java:39)
      	at com.atlassian.webhooks.internal.dao.ao.v1.ExternalMigrationTaskInvokerGuard.migrationTask(ExternalMigrationTaskInvokerGuard.java:42)
      	at com.atlassian.webhooks.external.OsgiMigrationToV1.registerExternalMigrationTask(OsgiMigrationToV1.java:36)
      
      ...
      
      
      Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: String or binary data would be truncated.
      

      Workaround

      1. Stop Jira node

      Atlassian Support strongly recommends conducting this workaround with a complete outage. This workaround can be accomplished by stopping only a single node. However, webhooks will briefly malfunction on sister nodes during the process.

      2. Backup the broken webhooks

      Since we're about to delete these webhooks' JQL filters, you must save the results (to a CSV). You'll need them for a later step.

      PostgreSQL Database

      SELECT *
      FROM "AO_4AEACD_WEBHOOK_DAO"
      WHERE LENGTH("FILTER") >= 255;
      

      MSSQL Database

      SELECT *
      FROM AO_4AEACD_WEBHOOK_DAO
      WHERE LEN(FILTER) >= 255;
      

      3. Remove the broken webhooks' filters

      PostgreSQL Database

      UPDATE "AO_4AEACD_WEBHOOK_DAO"
      SET "FILTER" = ''
      WHERE LENGTH("FILTER") >= 255;
      

      MSSQL Database

      UPDATE AO_4AEACD_WEBHOOK_DAO
      SET FILTER = ''
      WHERE LEN(FILTER) >= 255;
      

      4. Wipe the partially migrated webhooks

      This will override any webhook configuration changes you made since the Jira 10 upgrade. This step is needed to prevent duplicates and avoid other complications.

      PostgreSQL Database

      TRUNCATE TABLE "AO_A0B856_WEBHOOK",
          "AO_A0B856_WEBHOOK_CONFIG",
          "AO_A0B856_WEBHOOK_EVENT",
          "AO_A0B856_HIST_INVOCATION",
          "AO_A0B856_DAILY_COUNTS";
      

      MSSQL Database

      TRUNCATE TABLE AO_A0B856_WEBHOOK;
      TRUNCATE TABLE AO_A0B856_WEBHOOK_CONFIG;
      TRUNCATE TABLE AO_A0B856_WEBHOOK_EVENT;
      TRUNCATE TABLE AO_A0B856_HIST_INVOCATION;
      TRUNCATE TABLE AO_A0B856_DAILY_COUNTS;
      

      5. Reset the upgrade tasks

      Set Atlassian Webhooks Plugin back to database schema version 0

      PostgreSQL Database

      UPDATE propertystring ps
      SET propertyvalue = 0
      WHERE
          ps.id = (SELECT pe.id FROM propertyentry pe WHERE entity_name = 'jira.properties' AND property_key = 'AO_A0B856_#');
      

      MSSQL Database

      UPDATE propertystring
      SET propertyvalue = '0'
      WHERE
          id = (SELECT id FROM propertyentry WHERE entity_name = 'jira.properties' AND property_key = 'AO_A0B856_#');
      
      Set Jira Webhooks Plugin back to database schema version 1

      PostgreSQL Database

      UPDATE propertystring ps
      SET propertyvalue = 1
      WHERE
          ps.id = (SELECT pe.id FROM propertyentry pe WHERE entity_name = 'jira.properties' AND property_key = 'AO_4AEACD_#');
      

      MSSQL Database

      UPDATE propertystring
      SET propertyvalue = '1'
      WHERE
          id = (SELECT id FROM propertyentry WHERE entity_name = 'jira.properties' AND property_key = 'AO_4AEACD_#');
      
      5a. You can optionally validate this with:

      All Databases

      SELECT pe.id, pe.entity_name, pe.property_key, ps.propertyvalue
      FROM propertyentry pe
               JOIN propertystring ps ON pe.id = ps.id
      WHERE pe.entity_name = 'jira.properties'
        AND pe.property_key = 'AO_A0B856_#';
      

      All Databases

      SELECT pe.id, pe.entity_name, pe.property_key, ps.propertyvalue
      FROM propertyentry pe
               JOIN propertystring ps ON pe.id = ps.id
      WHERE pe.entity_name = 'jira.properties'
        AND pe.property_key = 'AO_4AEACD_#';
      

      6. Start node

      Observe the migration task running during the ActiveObjects initialization:

      2025-05-12 16:05:39,302-0700 active-objects-init-0 INFO anonymous     [c.a.j.p.w.ao.upgrade.MigrateToWebhooks7Task] Migrating webhooks to version external atlassian-webhooks-plugin.
      2025-05-12 16:05:39,331-0700 active-objects-init-0 INFO anonymous     [c.a.j.p.w.ao.upgrade.MigrateToWebhooks7Task] Webhooks migration; 10 items to migrate
      2025-05-12 16:05:39,331-0700 active-objects-init-0 INFO anonymous     [c.a.j.p.w.ao.upgrade.MigrateToWebhooks7Task] Delegating webhooks data migration to atlassian-webhooks ao.
      2025-05-12 16:05:39,355-0700 active-objects-init-0 INFO anonymous     [c.a.j.p.w.ao.upgrade.MigrateToWebhooks7Task] Migration data received from atlassian-webhooks by jira-webhooks.
      

      7. Start remaining nodes

      The migration upgrade task only needs to be performed once. When the other nodes come online, they'll detect that it's already been completed and skip it.

      If the other nodes weren't stopped, they'll detect the new webhooks the next time an event is fired or an admin visits the webhooks configuration page. Webhook configurations are not cached.

      8. Validate migration and restore missing JQL filters

      Log in to the UI and use the CSV you generated in Step 2 to reapply the removed JQL filters. Remember that the total length must still be less than 255 characters until JRASERVER-78654 is patched. You may be able to use a filter = filterID clause in the webhook's JQL parameter as a workaround.

              c05631dc8c5f Tomasz Ziółkowski
              44f973aba2e6 David McCoy
              Votes:
              11 Vote for this issue
              Watchers:
              22 Start watching this issue

                Created:
                Updated:
                Resolved: