-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Low
-
Affects Version/s: 5.10.0
-
Component/s: Search - Indexing
-
1
-
Severity 3 - Minor
-
1
Issue Summary
When a node joins a Bitbucket Data Center cluster, the internal search client used by both the search and indexing subsystems is unconditionally closed and rebuilt. This refresh exists to recover from a network partition ("split brain"), where a node that was cut off from the cluster may be holding stale search configuration. However, the refresh currently runs on every node join, including ordinary node startups where no partition ever occurred and nothing needs to be recovered.
The refresh needs to be more conservative: it should only trigger when there is a genuine possibility that the joining node was actually affected by a network partition, not on every single node join. As it stands today, the client is rebuilt on every node join in a cluster, regardless of whether a partition was ever possible.
Closing and replacing the client while it may be in active use is not expected to be safe in all cases. There is a possibility that this refresh can interrupt an indexing operation that happens to be in progress at the moment the refresh occurs.
Steps to Reproduce
- Set up a Bitbucket Data Center cluster with two or more nodes sharing a search server (Elasticsearch/OpenSearch).
- Start node 1 and allow it to fully initialize, with search indexing enabled.
- Start (or restart) an additional node so that it joins the cluster normally, with no network interruption between nodes at any point.
- Observe atlassian-bitbucket.log on the already-running node(s) at the moment the new node joins.
Expected Results
The search client should only be refreshed when the join event indicates that a network partition may actually have been resolved. An ordinary node join/startup, with no prior loss of connectivity between nodes, should not cause the search client to be rebuilt.
Actual Results
The search client is refreshed every time any node joins the cluster, even when there is no indication that a network partition ever occurred. atlassian-bitbucket.log shows a new search client being created immediately after every node-join log entry, even on clusters where nodes have never lost contact with one another:
<node join log entry> DefaultSearchClientFactory - Create new search client using server URL '...' DefaultSearchClientFactory - Create new search client using server URL '...'
No exception is thrown as part of this behavior. The concern is the unnecessary refresh itself, and the possibility that it could interrupt an in-flight search or indexing request if one happens to be in progress at the same moment.
Workaround
Currently there is no known workaround for this behavior. A workaround will be added here when available