Summary

When initiating a new replication run via the Migration Connector, there is currently no preflight validation or automated recovery mechanism to handle stale, aborted, or orphaned replication sessions. If a previous replication crashes (e.g., due to disk exhaustion) or if nodes restart/cycle, the connector leaves an active session lock registered against a dead or non-existent node ID. Subsequent replication attempts fail immediately with java.lang.IllegalStateException: Attempting to create new session when previous session is active.

We propose adding a Preflight Check and Stale Session Resolution mechanism in the Migration Connector to detect and handle existing replication states before starting a new session.

Observation:

During migration cutovers and test runs, if a replication session is interrupted abnormally:

• Orphaned Node Locks: In clustered Data Center environments, a previous session may remain tied to a nodeId that has since restarted or been decommissioned/replaced (e.g., Engine is not null... but active session was started on node <old-node-id>).
• Hard Block on Retries: When a retry or new replication command is issued from the control plane (PRC), the connector rejects it outright:

ERROR [c.a.m.c.service.ReplicationServiceImpl] Error creating new session when previous session is active - <previous-session-id>java.lang.IllegalStateException: Attempting to create new session when previous session is active
 

• Lack of Visibility: The admin receives a generic failure without an automated preflight check indicating that a stale session exists or how to safely terminate/clear it.

Proposed Solution & Requirements

1. Preflight Check (Pre-Replication Validation)

• Active Session & Cluster Health Check: Before accepting or dispatching a startReplication command, inspect the connector's shared state:

• • Check if an existing session ID is marked as RUNNING or ACTIVE.

• • Verify if the node holding the session is still an active member of the cluster (via Hazelcast / cluster membership).

• Clear User Guidance in UI: If a previous session is detected:

• • Display the active Session ID, starting Node ID, start timestamp, and last heartbeat/progress timestamp.

• • Warn if the session appears stale/dead (no heartbeat within threshold or node is dead).

2. Automated Stale Session Detection & Healing

• Heartbeat / TTL Timeout: Implement a lease/heartbeat mechanism for replication sessions. If the active node stops sending heartbeats or drops out of the cluster, mark the session state as ABORTED / STALE.

• Safe Force-Reset / Clean-Up Action: Provide an admin/support mechanism (via CCMA UI, REST API, or automated preflight option) to safely abort/clean up orphaned replication sessions and release locks in shared home without requiring manual DB surgery or full cluster restarts.

3. Pre-Replication Shared Home Storage Check

• Add a preflight disk space check on <shared-home>/connector/sessions to ensure adequate free disk space and inode availability before launching initial Debezium database snapshots.