A newer version of IMDG is available.

Want to try Hazelcast Platform?

We’ve combined the in-memory storage of IMDG with the stream processing power of Jet to bring you the all new Hazelcast Platform.

CP Subsystem Management

Unlike the dynamic nature of Hazelcast clusters, the CP subsystem requires manual intervention while expanding/shrinking its size, or when a CP member crashes or becomes unreachable. When a CP member becomes unreachable, it cannot be automatically removed from the CP subsystem because it could be still alive and partitioned away.

Moreover, the current CP subsystem implementation works only in memory without persisting any state to disk. It means that a crashed CP member will not be able to recover by reloading its previous state. Therefore, crashed CP members create a danger for gradually losing the majority of CP groups and eventually total loss of the availability of the CP subsystem. To prevent such situations, CPSubsystemManagementService offers APIs for dynamic management of the CP members.

The CP subsystem relies on Hazelcast’s failure detectors to test the reachability of CP members. Before removing a CP member from the CP subsystem, please make sure that it is declared as unreachable by Hazelcast’s failure detector and removed from the Hazelcast’s member list.

CP member additions and removals are internally handled by performing a single membership change at a time. When multiple CP members are shutting down concurrently, their shutdown process is executed serially. First, the Metadata CP group creates a membership change plan for CP groups. Then, the scheduled changes are applied to the CP groups one by one. After all removals are done, the shutting down CP member is removed from the active CP members list and its shutdown process is completed.

When a CP member is being shut down, it is replaced with another available CP member in all of its CP groups, including the Metadata group, in order not to decrease or more importantly not to lose the majority of CP groups. If there is no available CP member to replace a shutting down CP member in a CP group, that group’s size is reduced by 1 and its majority value is recalculated.

A new CP member can be added to the CP subsystem to either increase the number of available CP members for new CP groups or to fulfill the missing slots in the existing CP groups. After the initial Hazelcast cluster startup is done, an existing Hazelcast member can be be promoted to the CP member role. This new CP member automatically joins to CP groups that have missing members, and the majority value of these CP groups is recalculated.

A CP member may crash due to hardware problems or a defect in user code, or it may become unreachable because of connection problems, such as network partitions, network hardware failures, etc. If a CP member is known to be alive but only has temporary communication issues, it will catch up the other CP members and continue to operate normally after its communication issues are resolved. If it is known to be crashed or communication issues cannot be resolved in a short time, it can be preferable to remove this CP member from the CP subsystem, hence from all its CP groups. In this case, the unreachable CP member should be terminated to prevent any accidental communication with the rest of the CP subsystem.

When the majority of a CP group is lost for any reason, that CP group cannot make progress anymore. Even a new CP member cannot join to this CP group, because membership changes also go through the Raft consensus algorithm. For this reason, the only option is to force-destroy the CP group via the CPSubsystemManagementService.forceDestroyCPGroup() API. When this API is used, the CP group is terminated non-gracefully, without the Raft algorithm mechanics. Then, all CP data structure proxies that talk to this CP group fail with CPGroupDestroyedException. However, if a new proxy is created afterwards, then this CP group will be recreated from the scratch with a new set of CP members. Losing the majority of a CP group can be likened to partition-loss scenario of AP Hazelcast.

Please note that the CP groups that have lost their majority must be force-destroyed immediately, because they can block the Metadata CP group to perform membership changes.

Loss of the majority of Metadata CP group is the doomsday scenario for the CP subsystem. It is a fatal failure and the only solution is to reset the whole CP subsystem state via the CPSubsystemManagementService.restart() API. To be able to reset the CP subsystem, the initial size of the CP subsystem must be satisfied, which is defined by CPSubsystemConfig.getCPMemberCount(). For instance, assuming that CPSubsystemConfig.getCPMemberCount() is 5 and only 1 CP member is currently alive, when CPSubsystemManagementService.restart() is called, additional 4 regular Hazelcast members should exist in the cluster. New Hazelcast members can be started to satisfy CPSubsystemConfig.getCPMemberCount().

There is a subtle point about graceful shutdown of CP members. If there are N CP members in the cluster, HazelcastInstance.shutdown() can be called on N-2 CP members concurrently. Once these N-2 CP members complete their shutdown, the remaining 2 CP members must be shut down serially. Even though the shutdown API is called concurrently on multiple members, the Metadata CP group handles shutdown requests serially. Therefore, it would be simpler to shut down CP members one by one, by calling HazelcastInstance.shutdown() on the next CP member once the current CP member completes its shutdown. The reason behind this limitation is, each shutdown request internally requires a Raft commit to the Metadata CP group. A CP member proceeds to shutdown after it receives a response of its commit to the Metadata CP group. To be able to perform a Raft commit, the Metadata CP group must have its majority available. When there are only 2 CP members left after graceful shutdowns, the majority of the Metadata CP group becomes 2. If the last 2 CP members shut down concurrently, one of them is likely to perform its Raft commit faster than the other one and leave the cluster before the other CP member completes its Raft commit. In this case, the last CP member waits for a response of its commit attempt on the Metadata group, and times out eventually. This situation causes an unnecessary delay on shutdown process of the last CP member. On the other hand, when the last 2 CP members shut down serially, the `N-1`th member receives response of its commit after its shutdown request is committed also on the last CP member. Then, the last CP member checks its local data to notice that it is the last CP member alive, and proceeds its shutdown without attempting a Raft commit on the Metadata CP group.

CP Subsystem Management APIs

You can access the CP Subsystem management APIs using the Java API or REST interface. To communicate with the REST interface there are two options; one is to access REST endpoint URL directly or using the cp-subsystem.sh shell script, which comes with the Hazelcast package.

The cp-subsystem.sh script uses curl command, and curl must be installed to be able to use the script.

Get Local CP Member:

Returns the local CP member if this Hazelcast member is a part of CP Subsystem.

Java API

        CPMember localMember = cpSubsystem.getLocalCPMember();

REST API

> curl http://127.0.0.1:5701/hazelcast/rest/cp-subsystem/members/local
OR
> sh cp-subsystem.sh -o get-local-member --address 127.0.0.1 --port 5701
+
Sample Response:
{
    "uuid": "6428d7fd-6079-48b2-902c-bdf6a376051e",
    "address": "[127.0.0.1]:5701"
}

Get CP Groups:

Returns the list of active CP groups.

Java API

        CPSubsystemManagementService managementService = cpSubsystem.getCPSubsystemManagementService();
        ICompletableFuture<Collection<CPGroupId>> future = managementService.getCPGroupIds();
        Collection<CPGroupId> groups = future.get();

REST API

> curl http://127.0.0.1:5701/hazelcast/rest/cp-subsystem/groups
OR
> sh cp-subsystem.sh -o get-groups --address 127.0.0.1 --port 5701
+
Sample Response:
[{
    "name": "METADATA",
    "id": 0
}, {
    "name": "atomics",
    "id": 8
}, {
    "name": "locks",
    "id": 14
}]

Get a single CP Group:

Returns the active CP group with the given name. There can be at most one active CP group with a given name.

Java API

        CPSubsystemManagementService managementService = cpSubsystem.getCPSubsystemManagementService();
        ICompletableFuture<CPGroup> future = managementService.getCPGroup(groupName);
        CPGroup group = future.get();

REST API

> curl http://127.0.0.1:5701/hazelcast/rest/cp-subsystem/groups/$\{CPGROUP_NAME}
OR
> sh cp-subsystem.sh -o get-group --group $\{CPGROUP_NAME} --address 127.0.0.1 --port 5701
+
Sample Response:
{
    "id": {
        "name": "locks",
        "id": 14
    },
    "status": "ACTIVE",
    "members": [{
        "uuid": "33f84b0f-46ba-4a41-9e0a-29ee284c1c2a",
        "address": "[127.0.0.1]:5703"
    }, {
        "uuid": "59ca804c-312c-4cd6-95ff-906b2db13acb",
        "address": "[127.0.0.1]:5704"
    }, {
        "uuid": "777ff6ea-b8a3-478d-9642-47d1db019b37",
        "address": "[127.0.0.1]:5705"
    }, {
        "uuid": "c7856e0f-25d2-4717-9919-88fb3ecb3384",
        "address": "[127.0.0.1]:5702"
    }, {
        "uuid": "c6229b44-8976-4602-bb57-d13cf743ccef",
        "address": "[127.0.0.1]:5701"
    }]
}

Get CP Members:

Returns the list of active CP members in the cluster.

Java API

        CPSubsystemManagementService managementService = cpSubsystem.getCPSubsystemManagementService();
        ICompletableFuture<Collection<CPMember>> future = managementService.getCPMembers();
        Collection<CPMember> members = future.get();

REST API

> curl http://127.0.0.1:5701/hazelcast/rest/cp-subsystem/members
OR
> sh cp-subsystem.sh -o get-members --address 127.0.0.1 --port 5701
+
Sample Response:
[{
    "uuid": "33f84b0f-46ba-4a41-9e0a-29ee284c1c2a",
    "address": "[127.0.0.1]:5703"
}, {
    "uuid": "59ca804c-312c-4cd6-95ff-906b2db13acb",
    "address": "[127.0.0.1]:5704"
}, {
    "uuid": "777ff6ea-b8a3-478d-9642-47d1db019b37",
    "address": "[127.0.0.1]:5705"
}, {
    "uuid": "c6229b44-8976-4602-bb57-d13cf743ccef",
    "address": "[127.0.0.1]:5701"
}, {
    "uuid": "c7856e0f-25d2-4717-9919-88fb3ecb3384",
    "address": "[127.0.0.1]:5702"
}]

Force Destroy a CP Group:

Unconditionally destroys the given active CP group without using the Raft algorithm mechanics. This method must be used only when a CP group loses its majority and cannot make progress anymore. Normally, membership changes in CP groups, such as CP member promotion or removal, are done via the Raft consensus algorithm. However, when a CP group permanently loses its majority, it will not be able to commit any new operation. Therefore, this method ungracefully terminates the remaining members of the given CP group on the remaining CP group members. It also performs a Raft commit to the METADATA CP group in order to update the status of the destroyed group. Once a CP group is destroyed, all CP data structure proxies created before the destroy fails with CPGroupDestroyedException. However, if a new proxy is created afterwards, then this CP group is re-created from scratch with a new set of CP members.

This method is idempotent. It has no effect if the given CP group is already destroyed.
Java API
```
        CPSubsystemManagementService managementService = cpSubsystem.getCPSubsystemManagementService();
        ICompletableFuture<Void> future = managementService.forceDestroyCPGroup(groupName);
        future.get();
```
REST API
```
> curl -X POST --data "${GROUPNAME}&$\{PASSWORD}" http://127.0.0.1:5701/hazelcast/rest/cp-subsystem/groups/$\{CPGROUP_NAME}/remove
OR
> sh cp-subsystem.sh -o force-destroy-group --group $\{CPGROUP_NAME} --address 127.0.0.1 --port 5701 --groupname ${GROUPNAME} --password $\{PASSWORD}
```

Remove a CP Member:

Removes the given unreachable CP member from the active CP members list and all CP groups it belongs to. If any other active CP member is available, it replaces the removed CP member in its CP groups. Otherwise, CP groups which the removed CP member is a member of shrinks and their majority values are recalculated.

Before removing a CP member from CP Subsystem, please make sure that it is declared as unreachable by Hazelcast’s failure detector and removed from Hazelcast’s member list. The behavior is undefined when a running CP member is removed from CP Subsystem.

Java API

        CPSubsystemManagementService managementService = cpSubsystem.getCPSubsystemManagementService();
        ICompletableFuture<Void> future = managementService.removeCPMember(memberUUID);
        future.get();

REST API

> curl -X POST --data "${GROUPNAME}&$\{PASSWORD}" http://127.0.0.1:5701/hazelcast/rest/cp-subsystem/members/$\{CPMEMBER_UUID}/remove
OR
> sh cp-subsystem.sh -o remove-member --member $\{CPMEMBER_UUID} --address 127.0.0.1 --port 5701 --groupname ${GROUPNAME} --password $\{PASSWORD}

Promote Local Member to a CP Member

Promotes the local Hazelcast member to the CP member. If the local member is already in the active CP members list, i.e., it is already a CP member, then this method has no effect. When the local member is promoted to the CP role, its member UUID is assigned as CP member UUID. The promoted CP member will be added to the CP groups that have missing members, i.e., whose current size is smaller than CPSubsystemConfig.getGroupSize().
Java API
```
        CPSubsystemManagementService managementService = cpSubsystem.getCPSubsystemManagementService();
        ICompletableFuture<Void> future = managementService.promoteToCPMember();
        future.get();
```
REST API
```
> curl -X POST --data "${GROUPNAME}&$\{PASSWORD}" http://127.0.0.1:5701/hazelcast/rest/cp-subsystem/members
OR
> sh cp-subsystem.sh -o promote-member --address 127.0.0.1 --port 5701 --groupname ${GROUPNAME} --password $\{PASSWORD}
```

Wipe and Restart CP Subsystem

Wipes and resets the whole CP subsystem and initializes it as if the Hazelcast cluster is starting up initially. This method must be used only when the Metadata CP group loses its majority and cannot make progress anymore.

After this method is called, all CP state and data are wiped and the CP members start with empty state.

This method can be invoked only from the Hazelcast master member. Moreover, the Hazelcast cluster must have at least CPSubsystemConfig.getCPMemberCount() members.

This method must not be called while there are membership changes in the cluster. Before calling this method, please make sure that there is no new member joining and all existing Hazelcast members have seen the same member list.

This method is NOT idempotent and multiple invocations can break the whole system! After calling this API, you must observe the system to see if the restart process is successfully completed or failed before making another call.

Java API

        CPSubsystemManagementService managementService = cpSubsystem.getCPSubsystemManagementService();
        ICompletableFuture<Void> future = managementService.restart();
        future.get();

REST API

> curl -X POST --data "${GROUPNAME}&$\{PASSWORD}" http://127.0.0.1:5701/hazelcast/rest/cp-subsystem/reset
OR
> sh cp-subsystem.sh -o reset --address 127.0.0.1 --port 5701 --groupname ${GROUPNAME} --password $\{PASSWORD}

Session Management API

There are two management API methods for session management.

Get CP Group Sessions:

Returns all CP sessions that are currently active in a CP group.

Java API

        CPSessionManagementService sessionManagementService = cpSubsystem.getCPSessionManagementService();
        ICompletableFuture<Collection<CPSession>> future = sessionManagementService.getAllSessions(groupName);
        Collection<CPSession> sessions = future.get();

REST API

> curl http://127.0.0.1:5701/hazelcast/rest/cp-subsystem/groups/$\{CPGROUP_NAME}/sessions
OR
> sh cp-subsystem.sh -o get-sessions --group $\{CPGROUP_NAME} --address 127.0.0.1 --port 5701
+
Sample Response:
[{
    "id": 1,
    "creationTime": 1549008095530,
    "expirationTime": 1549008766630,
    "version": 73,
    "endpoint": "[127.0.0.1]:5701",
    "endpointType": "SERVER",
    "endpointName": "hz-member-1"
}, {
    "id": 2,
    "creationTime": 1549008115419,
    "expirationTime": 1549008765425,
    "version": 71,
    "endpoint": "[127.0.0.1]:5702",
    "endpointType": "SERVER",
    "endpointName": "hz-member-2"
}]

Force Close a Session:

If a Hazelcast instance that owns a CP session crashes, its CP session is not terminated immediately. Instead, the session is closed after CPSubsystemConfig.getSessionTimeToLiveSeconds() passes. If it is known for sure that the session owner is not partitioned and definitely crashed, this method can be used for closing the session and releasing its resources immediately.

Java API

        CPSessionManagementService sessionManagementService = cpSubsystem.getCPSessionManagementService();
        ICompletableFuture<Boolean> future = sessionManagementService.forceCloseSession(groupName, sessionId);
        future.get();

REST API

> curl -X POST --data "${GROUPNAME}&$\{PASSWORD}" http://127.0.0.1:5701/hazelcast/rest/cp-subsystem/groups/$\{CPGROUP_NAME}/sessions/$\{CP_SESSION_ID}/remove
OR
> sh cp-subsystem.sh -o force-close-session --group $\{CPGROUP_NAME} --session-id $\{CP_SESSION_ID} --address 127.0.0.1 --port 5701 --groupname ${GROUPNAME} --password $\{PASSWORD}

Edit this page Report an issue

CP Subsystem Management

CP Subsystem Management APIs

Session Management API

Help and support