CP Subsystem

Data structures in the CP subsystem always maintain linearizability and prefer consistency over availability during network partitions. These data structures are CP with respect to the CAP theorem.

CP Subsystem supports the following Hazelcast data structures:

Because CP data structures do not maintain large states, not all members of a Hazelcast cluster need to take part in the CP Subsystem. The members that do take part in the CP Subsystem are called CP members.

To enable the CP Subsystem, you need to configure the number of CP members in the cluster. When a cluster is configured with N CP members, the first N members to start form the CP Subsystem.

After CP Subsystem is initialized, more CP members can be added at runtime and the number of active CP members can go beyond the configured CP member count. The number of CP members can be smaller than the total size of the Hazelcast cluster. For instance, you can run 5 CP members in a Hazelcast cluster of 20 members.

CP members can also contain data for the AP data structures, such as map and set.

When a member becomes a CP member, it generates an additional UUID that other CP members can use to identify it. You will see this CP UUID in the following places:

CP data structures are stored in CP groups. The CP Subsystem includes two CP groups by default:

For CP data structures that involve resource ownership management, such as fenced locks or semaphores, sessions are required to keep track of liveliness of callers.

A caller initially creates a session before sending its first request to the CP group, such as to acquire a fenced lock. After creating a session on the CP group, the caller stores its session ID locally and sends it alongside its session-based operations. A single session is used for all lock and semaphore proxies of the caller.

When a CP group receives a session-based operation, it checks the validity of the session using the session ID information available in the operation. A session is valid if it is still open in the CP group.

An operation with a valid session ID is accepted as a new session heartbeat.

To keep its session alive, a caller commits a periodic heartbeat to the CP group in the background.

A session is closed when the caller does not touch the session during a configurable duration. In this case, the caller is assumed to be crashed and all its resources are released automatically.

When CP members start, they initiate a discovery process to find each other. Other Hazelcast members skip this process.

The CP discovery process runs out of the box without requiring any custom configuration for different environments. It is completed when each CP member initializes its local CP member list and commits it to the METADATA CP group. A soon-to-be CP member terminates itself if any of the following conditions occur before the CP discovery process is completed:

When CP Subsystem is reset, the CP discovery process is triggered again. However, it does not terminate Hazelcast members if a soon-to-be CP member terminates itself, because Hazelcast members are likely to contain data for AP data structures and their termination can cause data loss. Hence, you need to observe the cluster and check if the CP discovery process completes successfully on the CP Subsystem reset. See the CP Subsystem Management APIs section for more details.

By default, the CP Subsystem works only in memory without persisting any state to disk. This means that a crashed CP member is not able to rejoin the cluster by restoring its previous state. Therefore, crashed CP members increase the risk of gradually losing the majority of CP groups and eventually the total loss of the CP Subsystem. To prevent such situations, crashed CP members can be removed from the CP Subsystem and replaced in CP groups with other available CP members. This flexibility provides a good degree of fault tolerance at runtime.

Deployment of CP within Kubernetes is supported from Hazelcast Enterprise 5.5 and covers the following scenarios when using Hazelcast Platform Operator or our hazelcast/hazelcast-enterprise Helm Chart.

We support 3, 5- and 7-CP member deployments under the constraints discussed in this section.

The method by which deployment, pause, resume and rolling update are performed will vary according to the way that CP was deployed. See Deploying in Kubernetes for more information.

We recommend setting data-load-timeout-seconds to a value that spans the duration from when the first pod is running to when last pod is running and has completed its CP intialisation procedure. This is particularly important if you intend to perform _resume scenarios. Currently the only way to determine when a CP member has completed its initialisation is to consult the logs. Therefore, we recommend the following to determine a reasonable value for data-load-timeout-seconds:

If you are using a log aggregation service and want to filter key startup events within CP, you can use the INFO level patterns emitted by CPPersistenceServiceImpl as detailed below.