Uploaded image for project: 'Jira Software Data Center'
  1. Jira Software Data Center
  2. JSWSERVER-16473

Reindex All FAILED - Caused by null issuenum entry in jiraissue table

    • 7.02
    • 80
    • Severity 2 - Major
    • 229
    • Hide

      Hi all,

      I'm glad to announce that Jira versions 8.9.1, 8.10.0, and higher come with a sanity check that will prevent Jira from trying to index such corrupted issue entries.

      While this doesn't fix or prevent the corrupted data in the jiraissue table (see JRASERVER-69826 for that), it prevents the index corruption.

      The sanity check was only added to the latest Jira versions because Jira 8.9.0 comes with a huge rewrite of the affected classes. However, this fix alone is quite small and is a likely candidate for backporting to the 8.5 ER.

      Daniel Rauf

      Show
      Hi all, I'm glad to announce that Jira versions 8.9.1, 8.10.0, and higher come with a sanity check that will prevent Jira from trying to index such corrupted issue entries. While this doesn't fix or prevent the corrupted data in the jiraissue table (see JRASERVER-69826 for that), it prevents the index corruption. The sanity check was only added to the latest Jira versions because Jira 8.9.0 comes with a huge rewrite of the affected classes. However, this fix alone is quite small and is a likely candidate for backporting to the 8.5 ER. Daniel Rauf

      Summary

      Under a specific scenario, when doing a Bulk Change -> Edit Issues, Jira ends up with a NULL value under the issuenum column of jiraissue table, causing indexing failures with Reindex All FAILED error.

      Messages like the one below appear in the logs during reindex:

      2020-04-16 23:21:15,522+1000 JiraTaskExecutionThread-2 WARN admin 1401x13091x1 1jldanz 127.0.0.1 /secure/admin/IndexReIndex!reindex.jspa [c.a.jira.index.AccumulatingResultBuilder] com.atlassian.jira.util.dbc.Assertions$NullArgumentException: parameter should not be null!
      java.lang.RuntimeException: com.atlassian.jira.util.dbc.Assertions$NullArgumentException: parameter should not be null!
      	at com.atlassian.jira.index.DefaultIndex$Failure.<init>(DefaultIndex.java:100)
      	at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:469)
      	at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:438)
      	at com.atlassian.jira.issue.index.DefaultIssueIndexer.lambda$null$2(DefaultIssueIndexer.java:365)
      	at com.atlassian.jira.index.SimpleIndexingStrategy.apply(SimpleIndexingStrategy.java:7)
      	at com.atlassian.jira.index.SimpleIndexingStrategy.apply(SimpleIndexingStrategy.java:5)
      	at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call(MultiThreadedIndexingStrategy.java:33)
      	at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call(MultiThreadedIndexingStrategy.java:31)
      	at com.atlassian.jira.util.concurrent.BoundedExecutor$2.call(BoundedExecutor.java:68)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: com.atlassian.jira.util.dbc.Assertions$NullArgumentException: parameter should not be null!
      	at com.atlassian.jira.util.dbc.Assertions.notNull(Assertions.java:32)
      	at com.atlassian.jira.issue.changehistory.ChangeHistoryItem$Builder.inProject(ChangeHistoryItem.java:326)
      	at com.atlassian.jira.issue.history.AbstractDateRangeBuilder.changeItemBuilder(AbstractDateRangeBuilder.java:95)
      	at com.atlassian.jira.issue.history.VersionDateRangeBuilder.createInitialChangeItem(VersionDateRangeBuilder.java:69)
      	at com.atlassian.jira.issue.history.VersionDateRangeBuilder.buildDateRanges(VersionDateRangeBuilder.java:110)
      	at com.atlassian.jira.issue.index.DefaultChangeHistoryRetriever.filterChangeItems(DefaultChangeHistoryRetriever.java:74)
      	at com.atlassian.jira.issue.index.DefaultChangeHistoryRetriever.createChangeGroupsFromChangeItems(DefaultChangeHistoryRetriever.java:47)
      	at com.atlassian.jira.issue.index.DefaultChangeHistoryRetriever.apply(DefaultChangeHistoryRetriever.java:41)
      	at com.atlassian.jira.issue.index.DefaultChangeHistoryRetriever.apply(DefaultChangeHistoryRetriever.java:25)
      	at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy$1.apply(DefaultIssueIndexer.java:626)
      	at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy$1.apply(DefaultIssueIndexer.java:623)
      	at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy.get(DefaultIssueIndexer.java:616)
      	at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:450)
      	... 11 more 

      Steps to Reproduce

      1. Create a clean Jira installation;
      2. Create a Jira project;
      3. Create 2+ issues;
      4. Goto Issues -> Search for issues;
      5. On a new browser window or tab, open one of the recently created issues;
      6. Back to the original window or tab (search for issues page), click the Tools button and choose Bulk Change: all 2+ issue(s);
      7. Select all 2+ and click next;
      8. Select Edit Issues and click next;
      9. At this point, delete the issue you opened in a different window or tab;
      10. Go back to the original window or tab, edit something, click next and confirm the change;
      11. Wait for the operation to complete;
      12. Run the following query, a record will be returned with almost all field set as null.
        SELECT * FROM jiraissue WHERE issuenum IS NULL;

      Expected Results

      It was expected to have the deleted record/row completely gone from jiraissue table, and only the remaining records updated, or maybe have Jira showing a friendly error message to the user.

      Actual Results

      After the bulk operation is 100% complete, the deleted record remains in the table jiraissue, but with null values for all fields, with exception of the id and updated fields. Fields updated in the bulk process may also have a value different from NULL 

      Workaround

      1. Remove the NULL entries
        1. Stop Jira;
        2. Perform a complete database backup in case you need to rollback;
        3. Run the following query:
          DELETE FROM jiraissue WHERE issuenum IS NULL;
          
        4. Start Jira;
      2. Do a full Re-index, by choosing the option to 'Lock and Re-Index'.

          Form Name

            [JSWSERVER-16473] Reindex All FAILED - Caused by null issuenum entry in jiraissue table

            In my opinion it is a bug that Jira produces such nearly empty lines in jiraissue table and should be treated with a high priority as an critical bug because it can happen every day and keep administrators from reindexing.
            The workaround mentioned in JSWSERVER-16473 (to delete the corrupt lines from jiraissue table) is no real solution.

            We always avoid manual database manipulation because they are “open-heart operations” where you never know whether there are dependent records in other tables and you can get inconsistencies.

            So why is JSWSERVER-16473 solved by preventing “Jira from trying to index such corrupted issue entries” instead of solving the real bug (additionally!) which is obviously already known for at least 2,5 years?

            JRASERVER-69826 is a „suggestion“ to help to find the cause for the bug and has been created more than 2 years ago and still is in status “Gathering interest” which is totally unintelligible in my eyes.

            What about the “Atlassian Data Center and Server Bug Fix Policy”?:
            “We'll generally fix critical bugs in the next maintenance release”

            Gisela Lassahn added a comment - In my opinion it is a bug that Jira produces such nearly empty lines in jiraissue table and should be treated with a high priority as an critical bug because it can happen every day and keep administrators from reindexing. The workaround mentioned in JSWSERVER-16473 (to delete the corrupt lines from jiraissue table) is no real solution. We always avoid manual database manipulation because they are “open-heart operations” where you never know whether there are dependent records in other tables and you can get inconsistencies. So why is JSWSERVER-16473 solved by preventing “Jira from trying to index such corrupted issue entries” instead of solving the real bug (additionally!) which is obviously already known for at least 2,5 years? JRASERVER-69826 is a „suggestion“ to help to find the cause for the bug and has been created more than 2 years ago and still is in status “Gathering interest” which is totally unintelligible in my eyes. What about the “Atlassian Data Center and Server Bug Fix Policy”?: “We'll generally fix critical bugs in the next maintenance release”

            This happened with us for Jira 8.13.6.

            Vedant Kulkarni_Trundl added a comment - This happened with us for Jira 8.13.6.

            April added a comment -

            Great news, Daniel!

            We would be very grateful if this were backported to the 8.5 ER. Please comment again if that happens.

            April added a comment - Great news, Daniel! We would be very grateful if this were backported to the 8.5 ER. Please comment again if that happens.

            I'm glad to announce that Jira versions 8.9.1, 8.10.0, and higher come with a sanity check that will prevent Jira from trying to index such corrupted issue entries.

            While this doesn't fix or prevent the corrupted data in the jiraissue table (see JRASERVER-69826 for that), it prevents the index corruption.

            The sanity check was only added to the latest Jira versions because Jira 8.9.0 comes with a huge rewrite of the affected classes. However, this fix alone is quite small and is a likely candidate for backporting to the 8.5 ER.

            Daniel Rauf added a comment - I'm glad to announce that Jira versions 8.9.1, 8.10.0, and higher come with a sanity check that will prevent Jira from trying to index such corrupted issue entries. While this doesn't fix or prevent the corrupted data in the jiraissue table (see JRASERVER-69826 for that), it prevents the index corruption. The sanity check was only added to the latest Jira versions because Jira 8.9.0 comes with a huge rewrite of the affected classes. However, this fix alone is quite small and is a likely candidate for backporting to the 8.5 ER.

            Please fix this issue, we are facing it in 7.6.10.

            Dibyandu Roy added a comment - Please fix this issue, we are facing it in 7.6.10.

            Same issue in 7.13.0. 

            Sami Ahmed Shaik added a comment - Same issue in 7.13.0. 

            Esteban Casuscelli added a comment - https://getsupport.atlassian.com/browse/PS-56222

            This happens quite frequently on our platform with 40k users. +1

            Matthias Krauss added a comment - This happens quite frequently on our platform with 40k users. +1

            Murakami added a comment -

            Murakami added a comment - https://getsupport.atlassian.com/browse/PS-50421

            Divya TV added a comment -

            This problem also affects Jira 7.13 datacenter.

            Divya TV added a comment - This problem also affects Jira 7.13 datacenter.

              drauf Daniel Rauf
              amacedo@atlassian.com Alessandro Macedo
              Affected customers:
              59 This affects my team
              Watchers:
              74 Start watching this issue

                Created:
                Updated:
                Resolved: