JIRA Datacenter start generates large network traffic result in higher response time

Summary

JIRA Data Center uses a caching library called EHCache, and contains an in-memory cache of certain project, user and group data. On the very first node start (if no others are running) this cache is built from querying the database. When additional nodes are started, the cache is built from querying the existing nodes in-memory cache and propagating that data into the node that is starting up. Other caches are built only locally by filling the node's own copy of the cache directly from the database.

Additionally when JIRA Data Center restarts a node reindex thread runs which is designed to synchronise changes between the nodes - this can also contribute to the increase in network traffic.

Expected Behaviour

Starting up a node should not affect other nodes performance

Actual Behaviour

When adding node during busy hour we can notice surge in network traffic below

Which result in increased of the overall response time of all other nodes in the cluster for 3 minutes.

node-1	node-2	node-3
​	​	​

Note

There some improvements related to the cache replication during start-up:

• JRASERVER-63515
• JRASERVER-66392

Also in recent version 7.6.4 there is a fix JRASERVER-66839 which mostly contributing to this problem.

Workaround

Avoid adding/removing nodes during peak hour