Issue Summary

Jira DC can insert stale custom field values into its Lucene index while replicating index changes from another node(s).
This happens due to receiving node reading from the stale EagerLoadingOfBizCustomFieldPersister cache.
This will cause JQL search not returning the expected result on the affected node.

Notes:

• System fields are not affected since they are not using same cache.

Environment

The problem is visible in DC setup with 1+ nodes.

Steps to Reproduce

• create the issueA on a node A
  • the less load DC has the more likely the problem will be reproduced.
• edit a custom field with "value A"
• wait until this issue is replicated to other node(s)
• edit the same custom field once again and set "value B"
• wait until this issue is replicated to other node(s)

Expected Results

• All nodes see "value B" in the index.
• JQL search "CF = value B" shows the issueA

Actual Results

• Other nodes see stale CF value - "value A"
• JQL search "CF = value B" doen't show the issueA
• JQL search "CF = value A" shows the issueA
  • Opening issueA will show the correct value, since it will be loaded directly from DB

Notes

• NodeReindexServiceThread doesn't clear thread-local request caches on each consecutive run of reIndex.
• Problem is caused by JRASERVER-68806. NodeReindexServiceThread is caching 10 last issues, so when reindexing replication happens for the same issue it uses the old value to create Lucene doc. Since Lucene contains the old value, JQL search doesn't reflect the change. 
• Since EagerLoadingOfBizCustomFieldPersister cache stores 10 issues by default, creating or modifying 10 distinct issues with CF on nodeA will kick issueA out of the cache on all nodes. The next update of issueA on any other node will be replicated properly to the index.
• Jira 8+ is not affected.

Workaround

• Running Full reindex will temporarily fix the consistency problem.