Deleted issues can be shown in searches in Jira DC

Summary

When an issue is deleted on one node, it can still appear in searches on some other node. The key and summary of the issue are visible, but after clicking on the issue they change to "This issue can no longer be opened" message.

This happens when an "UPDATE" replication event is placed into the database after a "DELETE" event and they are processed in the immediate following batches by the node.
For example, with the following database entries, if one batch is processed at 20:22:48, it will reply the DELETE event, and then the next batch will process the UPDATE event, reviving the deleted issue.
 

In production instances, this usually happens due to some automation performing very quick operations, e.g. changing the issue's description and immediately deleting it.
Sometimes this can lead to the DELETE event being stored in the database before the UPDATE event that was performed earlier.
Even though both of the operations will be added to the database milliseconds apart, with some unlucky timing they can be processed in separate batches, as described above.

Environment

Jira Data Center

Steps to Reproduce

It's very hard to reproduce this in the UI. The easiest way is to create the scenario directly in the database.
1. Create some test issue and grab its id (e.g. by running select on the jiraissue table), here it's 10005
2. Delete the issue from the database:

delete from jiraissue where id = 10005;

3. Create a DELETE replication event, like the one below. Note that the node's id (c44634) has to belong to a running node, and the operation's id (11035) needs to be larger than the last processed replication event from that node.

insert into REPLICATEDINDEXOPERATION values (11035, now(), 'c44634', 'ISSUE', 'NONE', 10005, 'DELETE', '');

4. After ~5 seconds, create an UPDATE replication event, with the same constraints as above. Waiting too little can make this event be processed in the same batch as the previous event, or 2 batches later - in both of those scenarios the bug seems to not be reproducible.

insert into REPLICATEDINDEXOPERATION values (11036, now(), 'c44634', 'ISSUE', 'NONE', 10005, 'UPDATE', '');

Expected Results

The issue does not appear in the search results.
Nice to have: some message is logged in the application's log about the race occurring.

Actual Results

The issue's key and summary are visible in searches. Users are unable to view issue details.

Workaround

To fix the index inconsistency, run a project reindex, or a full reindex.

How to remove from Jira lucene index, issues that have been deleted in the database, but still present in index offers a workaround that does not require a project or full re-index. 

The the occurrences can also be limited by modifying automation scripts to avoid updating issues right before deleting them.