A newer version of IMDG is available.

View latest

Want to try Hazelcast Platform?

We’ve combined the in-memory storage of IMDG with the stream processing power of Jet to bring you the all new Hazelcast Platform.

CP Subsystem Persistence

4.2 4.1 4.0 3.12 3.11 3.10 3.9 3.8 3.7 3.6 3.5

Hazelcast IMDG Pro and Enterprise Feature

CP Subsystem Persistence Overview

CP Subsystem Persistence enables CP members to recover from crash scenarios. This capability significantly improves the overall reliability of CP Subsystem. When it is enabled via CPSubsystemConfig.setPersistenceEnabled(boolean), CP members persist their local state to stable storage. When you restart the crashed CP members, they restore their local state and resume working as if they have never crashed. CP Subsystem Persistence enables you to handle single or multiple CP member crashes, or even whole cluster crashes and guarantees that committed operations are not lost after recovery. In other words, CP member crashes and restarts do not create any consistency problem. As long as majority of CP members are available after recovery, CP Subsystem remains operational.

Please see the CP Subsystem Configuration section for the configuration options of CP Subsystem Persistence.

When CP Subsystem Persistence is enabled, all Hazelcast cluster members create a sub-directory under the base persistence directory which is specified via CPSubsystemConfig.getBaseDir(). This means that AP Hazelcast members, which are the ones not marked as CP members during the CP discovery process, create their persistence directories as well. Those members persist only the information that they are not CP members. This is done because when a Hazelcast member starts with CP Subsystem Persistence enabled, it checks if there is a CP persistence directory belonging to itself. If it founds one, it skips the CP discovery process and initializes its CP member identity from the persisted data. If it was an AP member before shutdown or crash, it restores this information and starts as an AP member. Otherwise, it could think that the CP discovery process has not been executed and trigger it, which would break CP Subsystem.

In light of this information, if you have both CP and AP members in your cluster when CP Subsystem Persistence is enabled, and if you want to perform a cluster-wide restart, you need to ensure that AP members are also restarted with their CP persistence directories.

You can check the code sample below to see how CP Subsystem Persistence works in general. In this code sample, we configure CP Subsystem with 3 CP members and also enable CP Subsystem Persistence. We start 3 Hazelcast members with this configuration and update a CP IAtomicLong instance. Each member creates a sub-directory for itself inside the default base CP Subsystem Persistence directory and stores its local CP state there. Then, we terminate two of these members as if they crash and restart only 1 of them back. When we fetch the same IAtomicLong instance from the restarted members and get its current value, we see that it returns the update that we made before terminating these members. Please note that we make sure that we have the majority of CP members alive to keep CP Subsystem available after restart.

        Config config = new Config();
        config.setLicenseKey("your-license-key");
        NetworkConfig networkConfig = config.getNetworkConfig();
        JoinConfig join = networkConfig.getJoin();
        TcpIpConfig tcpIpConfig = join.getTcpIpConfig();
        tcpIpConfig.setEnabled(true);
        tcpIpConfig.addMember("127.0.0.1");
       // config.getCPSubsystemConfig().setCPMemberCount(3).setPersistenceEnabled(true);

        HazelcastInstance instance1 = Hazelcast.newHazelcastInstance(config);
        HazelcastInstance instance2 = Hazelcast.newHazelcastInstance(config);
        HazelcastInstance instance3 = Hazelcast.newHazelcastInstance(config);

        IAtomicLong counter = instance1.getCPSubsystem().getAtomicLong("counter");
        counter.set(0);
        counter.incrementAndGet();

        instance1.getLifecycleService().terminate();
        instance2.getLifecycleService().terminate();

        instance1 = Hazelcast.newHazelcastInstance(config);

        counter = instance1.getCPSubsystem().getAtomicLong("counter");

        long val = counter.get();
        assert val == 1L;

CP Subsystem Persistence Behavior During CP Subsystem Reset

If the majority of CP members are permanently lost, CP Subsystem becomes unavailable. There is no solution to recover from this failure case with strong consistency guarantee. CP Subsystem Management API contains a method to delete all CP Subsystem state on the remaining CP members and start from scratch. CPSubsystemManagementService.reset() wipes and resets the whole CP Subsystem state and initializes it as if the Hazelcast cluster is starting up for the first time. This method deletes the persisted CP member states as well.

Interaction with Hot Restart Persistence

Hazelcast offers another persistence capability which is called Hot Restart Persistence. Hot Restart Persistence is used for restarting a cluster with large AP data after a planned cluster shutdown or a whole cluster-wide crash. Please note that CP Subsystem Persistence and Hot Restart Persistence are separate features with different behaviors and reliability guarantees. For instance, CP Subsystem Persistence guarantees that committed operations will be restored and the linearizability semantics of the CP Subsystem data structures will be preserved on restarts. However, Hot Restart Persistence may lose some of the acknowledged updates on AP data structures, based on how you configure the fsync behavior for your persisted AP data structures. Moreover, if you store AP and CP data in a single Hazelcast cluster and use both of the persistence features, Hazelcast member restarts or cluster restarts can fail because of the Hot Restart Persistence recovery semantics, even if the CP Subsystem Persistence recovery procedure is successful, or vice-versa.

lifebuoy

Help and support