[JRASERVER-69296] ReIndexing reporting wrong numbers on Jira Datacenter with Archived projects

Type: Bug
Resolution: Fixed
Priority: Low (View bug fix roadmap)
Fix Version/s: 8.4.2, 8.5.0
Affects Version/s: 7.13.2, 8.0.2, 8.1.1
Component/s: Data Center - Archiving, Data Center - Index
Labels:
- er85
- flash
- pse-request
- warranty

Fixed in Long Term Support Release/s:

Download 8.5
Introduced in Version:
7.13
Support reference count:
35
Symptom Severity:
Severity 2 - Major
UIS:
200
Bug Fix Policy:
View Atlassian Server bug fix policy
Current Status:

Hide

Dear Customers,

problem is caused by multiple factors and at this point we believe we fixed the cause of the cache inconsistency. We will introduce a bugfix in 8.5.0 and into 8.4.2 versions.
That being said, we are not fully sure if we covered all possible cases. Therefore, we would like to monitor the problem and won’t be closing issue yet.

Regards,
Marcin (Jira Developer)

Show
Dear Customers, problem is caused by multiple factors and at this point we believe we fixed the cause of the cache inconsistency. We will introduce a bugfix in 8.5.0 and into 8.4.2 versions. That being said, we are not fully sure if we covered all possible cases. Therefore, we would like to monitor the problem and won’t be closing issue yet. Regards, Marcin (Jira Developer)

Issue Summary

Jira Datacenter is reporting wrong numbers after finishing foreground re-Indexing.

The re-Indexing finishes unexpectedly before it reaches 100% and no errors are being logged.

The following appears in the atlassian-jira.log:

2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing started
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All starting...
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.index.ha.DefaultNodeReindexService] Pausing node re-index service
2019-01-01 00:00:23,647 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 0% complete. Current index: Issue
2019-01-01 00:00:23,647 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.issue.index.DefaultIndexManager] ReindexAll in foreground: {indexIssues=true, indexChangeHistory=true, indexComments=true, indexWorklogs=true, forceReloadFromDatabase=false}
.....
.....
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 68% complete. Current index: 
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All complete. Total time: 3840094ms. Reindex run: 101
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing finished

Also the below could be seen in the logs:

2019-01-1 12:35:00,166 http-nio-8080-exec-1 DEBUG user2 111x11111x2 1111aab xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/jira/IndexAdmin.jspa [c.a.j.issue.index.IndexConsistencyUtils] isIndexConsistent: Issue: expectedCount=235000; actualCount=159000
2019-01-1 12:35:00,166 http-nio-8080-exec-1 WARN user2 111x11111x2 1111aab xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/jira/IndexAdmin.jspa [c.a.j.issue.index.IndexConsistencyUtils] Index consistency check failed for index 'Issue': expectedCount=235000; actualCount=159000
As can be seen the Index is having a big discrepancy from the number of issues got from the database (expectedCount).

Environment

Jira Datacenter 7.10.x or later
Having 1 or more projects in Archive.
- Unsure if you have Archived projects? See List of Archived Projects from Database

Steps to Reproduce

No known steps to reproduce this issue so far.

Expected Results

Re-Indexing should finish at 100% and Jira should report the right number of issues responding to the call:

GET <JIRA_BASE_URL>/rest/api/2/index/summary

Actual Results

Jira reports wrong progress while doing Re-Indexing

and the above call shows zero issues in archive ("countInArchive": 0) which is wrong:

{
  "indexReadable": true,
  "countInDatabase": 235000,
  "countInIndex": 159000,
  "countInArchive": 0,
  "lastUpdatedInDatabase": "2019-01-1T00:40:00.357+0100",
  "lastUpdatedInIndex": "2019-01-1T12:35:00.000+0100"
}

Workaround

The workaround for this issue is simple, we need to correct the values of the cache by flushing it and rebuilding the number.
This can be done by adding or removing a project.

Please follow the below steps:

As a workaround to clear the cache, create a new empty project, an issue on that project, then archive the project.
verify if countInArchive has changed in /rest/api/2/index/summary response.

The above issue is also described in the KB: ReIndexing reporting wrong numbers on Jira Datacenter with Archived Projects.

is caused by: FLASH-1547 Loading...

mentioned in: Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...

relates to: FLASH-1411 Loading...

(7 mentioned in, 1 relates to)

Adrian Castillo made changes - 25/Jul/2023 10:22 PM

Remote Link

Original: This issue links to "FLASH-1411 (Bulldog)" [ 431035 ]

New: This issue links to "FLASH-1411 (JIRA Server (Bulldog))" [ 431035 ]

Eduard M made changes - 05/Nov/2020 3:24 PM

Description

Original: h3. Issue Summary

Jira Datacenter is reporting wrong numbers after finishing foreground re-Indexing.

The re-Indexing finishes unexpectedly before it reaches 100% and no errors are being logged.

The following appears in the atlassian-jira.log:
{code:java}
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing started
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All starting...
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.index.ha.DefaultNodeReindexService] Pausing node re-index service
2019-01-01 00:00:23,647 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 0% complete. Current index: Issue
2019-01-01 00:00:23,647 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.issue.index.DefaultIndexManager] ReindexAll in foreground: {indexIssues=true, indexChangeHistory=true, indexComments=true, indexWorklogs=true, forceReloadFromDatabase=false}
.....
.....
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 68% complete. Current index:
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All complete. Total time: 3840094ms. Reindex run: 101
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing finished
{code}
Also the below could be seen in the logs:
{code:java}
2019-01-1 12:35:00,166 http-nio-8080-exec-1 DEBUG user2 111x11111x2 1111aab xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/jira/IndexAdmin.jspa [c.a.j.issue.index.IndexConsistencyUtils] isIndexConsistent: Issue: expectedCount=235000; actualCount=159000
2019-01-1 12:35:00,166 http-nio-8080-exec-1 WARN user2 111x11111x2 1111aab xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/jira/IndexAdmin.jspa [c.a.j.issue.index.IndexConsistencyUtils] Index consistency check failed for index 'Issue': expectedCount=235000; actualCount=159000
As can be seen the Index is having a big discrepancy from the number of issues got from the database (expectedCount).
{code}
h3. Environment
* Jira Datacenter 7.10.x or later
* Having 1 or more projects in Archive.
** Unsure if you have Archived projects? See [List of Archived Projects from Database|https://confluence.atlassian.com/jirakb/list-of-archived-projects-from-database-976763219.html]

h3. Steps to Reproduce

No known steps to reproduce this issue so far.
h3. Expected Results

Re-Indexing should finish at 100% and Jira should report the right number of issues responding to the call:
{code:java}
GET <JIRA_BASE_URL>/rest/api/2/index/summary
{code}
h3. Actual Results
* Jira reports wrong progress while doing Re-Indexing
* and the above call shows zero issues in archive (_"countInArchive": 0_) which is wrong:
{code:java}
{
  "indexReadable": true,
  "countInDatabase": 235000,
  "countInIndex": 159000,
  "countInArchive": 0,
  "lastUpdatedInDatabase": "2019-01-1T00:40:00.357+0100",
  "lastUpdatedInIndex": "2019-01-1T12:35:00.000+0100"
}
{code}

h3. Workaround

The workaround for this issue is simple, we need to correct the values of the cache by flushing it and rebuilding the number.
This can be done by adding or removing a project.

Please follow the below steps:
# As a workaround to clear the cache, create a new empty project
# verify if countInArchive has changed in */rest/api/2/index/summary* response.

The above issue is also described in the KB: [ReIndexing reporting wrong numbers on Jira Datacenter with Archived Projects|https://confluence.atlassian.com/display/JIRAKB/ReIndexing+reporting+wrong+numbers+on+Jira+Datacenter+with+Archived+projects].

New: h3. Issue Summary

Jira Datacenter is reporting wrong numbers after finishing foreground re-Indexing.

The re-Indexing finishes unexpectedly before it reaches 100% and no errors are being logged.

The following appears in the atlassian-jira.log:
{code:java}
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing started
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All starting...
2019-01-01 00:00:23,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.index.ha.DefaultNodeReindexService] Pausing node re-index service
2019-01-01 00:00:23,647 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 0% complete. Current index: Issue
2019-01-01 00:00:23,647 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.issue.index.DefaultIndexManager] ReindexAll in foreground: {indexIssues=true, indexChangeHistory=true, indexComments=true, indexWorklogs=true, forceReloadFromDatabase=false}
.....
.....
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 68% complete. Current index:
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All complete. Total time: 3840094ms. Reindex run: 101
2019-01-01 01:04:26,624 JiraTaskExectionThread-3 INFO user1 111x11111x1 1111aaa xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/IndexReIndex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing finished
{code}
Also the below could be seen in the logs:
{code:java}
2019-01-1 12:35:00,166 http-nio-8080-exec-1 DEBUG user2 111x11111x2 1111aab xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/jira/IndexAdmin.jspa [c.a.j.issue.index.IndexConsistencyUtils] isIndexConsistent: Issue: expectedCount=235000; actualCount=159000
2019-01-1 12:35:00,166 http-nio-8080-exec-1 WARN user2 111x11111x2 1111aab xx.xx.xx.xx,xx.xx.xx.xx /secure/admin/jira/IndexAdmin.jspa [c.a.j.issue.index.IndexConsistencyUtils] Index consistency check failed for index 'Issue': expectedCount=235000; actualCount=159000
As can be seen the Index is having a big discrepancy from the number of issues got from the database (expectedCount).
{code}
h3. Environment
* Jira Datacenter 7.10.x or later
* Having 1 or more projects in Archive.
** Unsure if you have Archived projects? See [List of Archived Projects from Database|https://confluence.atlassian.com/jirakb/list-of-archived-projects-from-database-976763219.html]

h3. Steps to Reproduce

No known steps to reproduce this issue so far.
h3. Expected Results

Re-Indexing should finish at 100% and Jira should report the right number of issues responding to the call:
{code:java}
GET <JIRA_BASE_URL>/rest/api/2/index/summary
{code}
h3. Actual Results
* Jira reports wrong progress while doing Re-Indexing
* and the above call shows zero issues in archive (_"countInArchive": 0_) which is wrong:
{code:java}
{
  "indexReadable": true,
  "countInDatabase": 235000,
  "countInIndex": 159000,
  "countInArchive": 0,
  "lastUpdatedInDatabase": "2019-01-1T00:40:00.357+0100",
  "lastUpdatedInIndex": "2019-01-1T12:35:00.000+0100"
}
{code}

h3. Workaround

The workaround for this issue is simple, we need to correct the values of the cache by flushing it and rebuilding the number.
This can be done by adding or removing a project.

Please follow the below steps:
# As a workaround to clear the cache, create a new empty project, an issue on that project, then archive the project.
# verify if countInArchive has changed in */rest/api/2/index/summary* response.

The above issue is also described in the KB: [ReIndexing reporting wrong numbers on Jira Datacenter with Archived Projects|https://confluence.atlassian.com/display/JIRAKB/ReIndexing+reporting+wrong+numbers+on+Jira+Datacenter+with+Archived+projects].

set-jac-bot made changes - 22/May/2020 8:25 AM

Fixed in Enterprise Release/s

New: [Download 8.5|https://confluence.atlassian.com/enterprise/atlassian-enterprise-releases-948227420.html]

Tony D. made changes - 13/May/2020 9:30 PM

Remote Link

New: This issue links to "Page (Confluence)" [ 483886 ]

Michal Warkocz made changes - 08/May/2020 8:37 AM

Remote Link

Original: This issue links to "Page (Confluence)" [ 482622 ]

Michal Warkocz made changes - 07/May/2020 7:08 AM

Remote Link

New: This issue links to "Page (Confluence)" [ 482622 ]

Artur Faruga made changes - 15/Apr/2020 6:48 AM

Remote Link

New: This issue links to "Page (Confluence)" [ 479048 ]

Marcin Sasinowski made changes - 25/Feb/2020 5:35 AM

Resolution		New: Fixed [ 1 ]
Status	Original: In Progress [ 3 ]	New: Closed [ 6 ]

SET Analytics Bot made changes - 14/Feb/2020 12:35 AM

UIS

Original: 218

New: 200

Tony D. made changes - 05/Feb/2020 9:22 PM

Remote Link

New: This issue links to "Page (Confluence)" [ 469972 ]

Assignee:: Unassigned

Reporter:: Sherif Abdelfattah (Inactive)

Affected customers:: 25 This affects my team

Watchers:: 41 Start watching this issue

Created:: 07/May/2019 2:29 PM

Updated:: 25/Jul/2023 10:22 PM

Resolved:: 25/Feb/2020 5:35 AM

Details

Description

Issue Summary

Environment

Steps to Reproduce

Expected Results

Actual Results

Workaround

Attachments

Issue Links

Forms

Activity

People

Dates