-
Bug
-
Resolution: Fixed
-
Low (View bug fix roadmap)
-
7.13.2, 8.0.2, 8.1.1
-
7.13
-
35
-
Severity 2 - Major
-
200
-
-
Issue Summary
Jira Datacenter is reporting wrong numbers after finishing foreground re-Indexing.
The re-Indexing finishes unexpectedly before it reaches 100% and no errors are being logged.
The following appears in the atlassian-jira.log:
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing started
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All starting...
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.index.ha.DefaultNodeReindexService] Pausing node re-index service
2019-01-01 00:00:23,647 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 0% complete. Current index: Issue
2019-01-01 00:00:23,647 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.issue.index.DefaultIndexManager] ReindexAll in foreground: {indexIssues=true, indexChangeHistory=true, indexComments=true, indexWorklogs=true, forceReloadFromDatabase=false}
.....
.....
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 68% complete. Current index:
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All complete. Total time: 3840094ms. Reindex run: 101
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing finished
Also the below could be seen in the logs:
2019-01-1 12:35:00,166 http-nio-8080-exec-1 DEBUG user2 111x11111x2 1111aab xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/jira/IndexAdmin.jspa [c.a.j.issue.index.IndexConsistencyUtils] isIndexConsistent: Issue: expectedCount=235000; actualCount=159000 2019-01-1 12:35:00,166 http-nio-8080-exec-1 WARN user2 111x11111x2 1111aab xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/jira/IndexAdmin.jspa [c.a.j.issue.index.IndexConsistencyUtils] Index consistency check failed for index 'Issue': expectedCount=235000; actualCount=159000 As can be seen the Index is having a big discrepancy from the number of issues got from the database (expectedCount).
Environment
- Jira Datacenter 7.10.x or later
- Having 1 or more projects in Archive.
- Unsure if you have Archived projects? See List of Archived Projects from Database
Steps to Reproduce
No known steps to reproduce this issue so far.
Expected Results
Re-Indexing should finish at 100% and Jira should report the right number of issues responding to the call:
GET <JIRA_BASE_URL>/rest/api/2/index/summary
Actual Results
- Jira reports wrong progress while doing Re-Indexing
- and the above call shows zero issues in archive ("countInArchive": 0) which is wrong:
{ "indexReadable": true, "countInDatabase": 235000, "countInIndex": 159000, "countInArchive": 0, "lastUpdatedInDatabase": "2019-01-1T00:40:00.357+0100", "lastUpdatedInIndex": "2019-01-1T12:35:00.000+0100" }
Workaround
The workaround for this issue is simple, we need to correct the values of the cache by flushing it and rebuilding the number.
This can be done by adding or removing a project.
Please follow the below steps:
- As a workaround to clear the cache, create a new empty project, an issue on that project, then archive the project.
- verify if countInArchive has changed in /rest/api/2/index/summary response.
The above issue is also described in the KB: ReIndexing reporting wrong numbers on Jira Datacenter with Archived Projects.
[JRASERVER-69296] ReIndexing reporting wrong numbers on Jira Datacenter with Archived projects
We are on Jira Data Center 7.13.9 with around 600 archived projects. In our case a mere restart of just one node triggers this bug, PS-53158. It also affects the Structure plugin as it makes use of JQL search and thus dependent upon the Lucene index. Structure's error: /rest/structure/2.0/poll [c.a.j.issue.index.IndexConsistencyUtils] Index consistency check failed for index 'Issue': expectedCount=xxxxx; actualCount=xxxx.
Dear Customers,
problem is caused by multiple factors and at this point we believe we fixed the cause of the cache inconsistency. We will introduce a bugfix in 8.5.0 and into 8.4.2 versions.
That being said, we are not fully sure if we covered all possible cases. Therefore, we would like to monitor the problem and won’t be closing issue yet.
Regards,
Marcin (Jira Developer)
Any update on this issue? This has now started affecting our clients.
How do we find out which issue is causing the foreground reindex to fail? I can't enable logging to show every issue indexed in a very large Jira instance. Shouldn't some error be logged with the issue key and field causing the problem?
@Anton Can get read access to your ticket PS-53158