Problem Definition

JIRA Datacenter (JDC) uses Ehcache as caching technology, cache server which exposes available cache operations remotely through RMI. Ehcache requires two port for it's network operations, RMICacheManagerPeerListener paramets

• hostName - may be null, in which case the hostName will be looked up.
• port - a port in the range 1025 - 65536
• remoteObjectPort - the port number on which the remote objects bound in the registry receive calls. This defaults to a free port if not specified.
• cacheManager - the CacheManager this listener belongs to
• socketTimeoutMillis - TCP/IP Socket timeout when waiting on response

see: http://www.ehcache.org/apidocs/2.6.9/net/sf/ehcache/distribution/RMICacheManagerPeerListener.html

Currently in JDC you can specify only port (default = 40001, setting: ehcache.listener.port), see Cluster.properties file parameters
Another remoteObjectPort is dynamically allocated and TCP ephemeral port is selected. So you need to open wide range of port at firewall.

Without opening the port, the following error is displayed as other nodes start up :

2017-03-22 11:53:38,023 ehcache-replicator-0 WARN      [c.a.j.cluster.distribution.JiraCacheManagerPeerProvider] Looking up rmiUrl //ip-x-x-x-x:40001/com.atlassian.jira.project.util.CachingProjectKeyStore.cache threw a connection exception. This could mean that a node has gone offline  or it may indicate network connectivity difficulties. Details: Connection refused to host: ip-x-x-x-x; nested exception is:
        java.net.ConnectException: Connection timed out

Suggested Solution

Add option to specify remoteObjectPort (RMI port)

See related links:

• https://answers.atlassian.com/questions/42692523/what-is-the-random-port-opened-by-jira-datacenter-used-for-and-what-is-the-range-of-choices
• https://en.wikipedia.org/wiki/Java_remote_method_invocation
• https://en.wikipedia.org/wiki/Ephemeral_port

Workaround

Open all Ephemeral_port range in firewall and allow connection from all JDC nodes.

Feature introduced in JIRA 7.3

New variable: ehcache.object.port

Note: If not defined an ephemeral port is used, see JRASERVER-66608

Ticket to update docs: JRA-64976