-
Bug
-
Resolution: Fixed
-
Low (View bug fix roadmap)
-
7.2.10, 7.2.15, 7.13.0, 7.6.9, 7.6.10, 8.5.4, 8.13.6
-
7.02
-
11
-
Severity 3 - Minor
-
1
-
Summary
When an issue is deleted on one node, it can still appear in searches on some other node. The key and summary of the issue are visible, but after clicking on the issue they change to "This issue can no longer be opened" message.
This happens when an "UPDATE" replication event is placed into the database after a "DELETE" event and they are processed in the immediate following batches by the node.
For example, with the following database entries, if one batch is processed at 20:22:48, it will reply the DELETE event, and then the next batch will process the UPDATE event, reviving the deleted issue.
In production instances, this usually happens due to some automation performing very quick operations, e.g. changing the issue's description and immediately deleting it.
Sometimes this can lead to the DELETE event being stored in the database before the UPDATE event that was performed earlier.
Even though both of the operations will be added to the database milliseconds apart, with some unlucky timing they can be processed in separate batches, as described above.
Environment
Jira Data Center
Steps to Reproduce
It's very hard to reproduce this in the UI. The easiest way is to create the scenario directly in the database.
1. Create some test issue and grab its id (e.g. by running select on the jiraissue table), here it's 10005
2. Delete the issue from the database:
delete from jiraissue where id = 10005;
3. Create a DELETE replication event, like the one below. Note that the node's id (c44634) has to belong to a running node, and the operation's id (11035) needs to be larger than the last processed replication event from that node.
insert into REPLICATEDINDEXOPERATION values (11035, now(), 'c44634', 'ISSUE', 'NONE', 10005, 'DELETE', '');
4. After ~5 seconds, create an UPDATE replication event, with the same constraints as above. Waiting too little can make this event be processed in the same batch as the previous event, or 2 batches later - in both of those scenarios the bug seems to not be reproducible.
insert into REPLICATEDINDEXOPERATION values (11036, now(), 'c44634', 'ISSUE', 'NONE', 10005, 'UPDATE', '');
Expected Results
The issue does not appear in the search results.
Nice to have: some message is logged in the application's log about the race occurring.
Actual Results
The issue's key and summary are visible in searches. Users are unable to view issue details.
Workaround
To fix the index inconsistency, run a project reindex, or a full reindex.
How to remove from Jira lucene index, issues that have been deleted in the database, but still present in index offers a workaround that does not require a project or full re-index.
The the occurrences can also be limited by modifying automation scripts to avoid updating issues right before deleting them.
- causes
-
-
- Gathering Impact
-
- is related to
-
JRASERVER-70422 Create Lucene document based replication for Jira DC for index replication
- Closed
- relates to
-
-
- Closed
-
-
PSR-122 You do not have permission to view this issue
[JRASERVER-68642] Deleted issues can be shown in searches in Jira DC
Hi 2684737ab56a
Could you please create a support ticket for this?
Most important question:
- are you on Jira server or Jira DC (the implementation of this functionality is different)
Please also provide the following information:
- please find the the issue id
- please paste row describing this issue in the version table (issue_version) (issueId, parentIssueId, updateTime, indexVersion, deleted)
If deleted==false we would need to investigate why deleting (actually de-indexing) the issue did not mark the issue as deleted in the version table.
If deleted==true we would need to investigate why the job re-applying deletes failed to detect this issue not being deleted.
All Jira logs +-5min around the time the delete happened would be very useful.
- we have added versioning of all entities - now every change which triggers an issue, comment or worklog reindex bumps this entity version (bumping the version is only triggered by user actions, not full/background re-index)
- all index updates (except full-foreground re-index) are now doing a conditional update: we only update the index if the entity in the index is older/equal (in therms of versioning - not time) to the new entity: this gives us a guarantee that all update conflicts will be resolved on all nodes with the same result; when doing a full re-index we have a clean index and there are no concurrent index updates so the index updates are non-conditional;
- on both server and DC there is a task running every 5sec which checks the consistency of the recent (last 15 sec) changes in the index and re-applies deletes if there was a late update which has overridden the delete (conditional updates do not solve this problem); this task checks for the consistency of updates as well to handle corner cases like thread was gone after updating the DB (for DC this is the index replication table, for server the entity version table) but before updating the local index
Let me know if you need any more information.
This issue is mentioned in https://confluence.atlassian.com/jirakb/how-to-remove-from-jira-lucene-index-issues-that-have-been-deleted-in-the-database-but-still-present-in-index-998896911.html but not linked
Hi, d083a6e71f02 -
This bug is quite specific and versioning guarantees that UPDATE index operation can not be created after DELETE operation.
However there is a bug with similar symptoms specific to multi-node Jira DC deployment where the replication of index can cause a DELETE to be overridden by an UPDATE. This problem is covered by: https://jira.atlassian.com/browse/JRASERVER-74348
Is it possible you are affected by
JRASERVER-74348?mac