Restart a cluster

5.7-SNAPSHOT 5.6 5.5 5.4 5.3 5.2 5.1 5.0

Static configuration is configuration that cannot be changed while members are running. To apply updates to static configuration, you must restart the cluster.

Restart options

There are two options for restarting a cluster:

Where possible, you should perform a rolling restart. In a rolling restart, members are stopped one at a time and each member’s data is automatically repartitioned across the remaining cluster members before shutting down. This minimizes the risk of service interruptions and data loss. When the replacement member joins the cluster, data is repartitioned again.

Depending on the scope of your configuration changes, you may need to update a single member, a cluster, or multiple clusters. The process is the same in all cases.

You can also use a rolling restart for canarying: testing changes on a single member before updating the rest of the cluster.

Some configuration needs to be consistent across all members in a cluster. To update this configuration, you must restart the whole cluster simultaneously. By default, this causes all data in the cluster to be lost. To avoid data loss, you must enable persistence. To avoid downtime, you must divert traffic to another Hazelcast cluster before restarting. Hazelcast supports blue-green deployments to facilitate this.

Note that member processes are not reused. In all cases, members are terminated, and new members are created to replace them.

Persistence

If persistence is enabled and the cluster enters NO_MIGRATION or FROZEN state, replacement members will adopt the UUIDs of the terminated members and attempt to load persisted data from disk. This avoids the repartitioning process, helping the cluster recover more quickly during a rolling restart.

Persistence is required to avoid data loss during a whole cluster restart. See Configuring Persistence for details.

Embedded mode

If your cluster is operating in embedded mode, you should restart your host application instead of trying to restart the embedded Hazelcast members independently.

Perform a restart

This section describes how to restart a cluster using a choice of common tools. Hazelcast supports a wide range of deployment options that may require other tools or allow you to automate parts of the process, but the required steps will be similar.

You should perform restarts in a low utilization or maintenance window to reduce the risk of service impacts.

Deprecation Notice for the Community Edition REST API

The Community Edition REST API has been deprecated and will be removed as of Hazelcast version 7.0. An improved Enterprise version of this feature is available and actively being developed. For more information, see Enterprise REST API.

Perform a rolling restart

You should check the cluster is healthy before and after making changes. The best way to do this is to run a Healthcheck in Management Center. For other options, see Monitoring.

To perform a rolling restart:

  1. Update the static configuration as required.

  2. Gracefully shut down the first member:

    • CLI

    • Management Center

    • Docker

    This command shuts down all running members on the local machine. 
    hz-stop

    Go to Members, select the member’s row in the table and select Shutdown.

    docker stop <CONTAINER_NAME>

  3. Wait until all partition migrations are completed. You can monitor the progress of migrations on the cluster dashboard in Management Center.

  4. Start a new member.

    • CLI

    • Docker

    hz start
    docker start <CONTAINER_NAME>

  5. Wait until the new member joins the cluster and all partition migrations are completed. Again, you can monitor progress using the cluster dashboard in Management Center.

  6. Repeat this process until all members have been restarted with the updated configuration.

Perform a whole cluster restart

You should check the cluster is healthy before and after making changes. The best way to do this is to run a Healthcheck in Management Center. For other options, see Monitoring.

To perform a whole cluster restart:

  1. Update the static configuration as required.

  2. Divert traffic to another Hazelcast cluster. How to do this depends on how your Hazelcast deployment is architected. Management Center provides an easy way to manage client connections using filtering rules.

  3. If you have enabled persistence, change the cluster state to NO_MIGRATION or FROZEN:

    • CLI

    • Management Center

    hz-cluster-admin -a <MEMBER_ADDRESS> -c <CLUSTER_NAME> -s no_migration

    You can specify the IP address of any member in the cluster.

    Go to Administration > Cluster State and select No Migration or Frozen.

  4. Gracefully shut down the cluster:

    • CLI

    • Management Center

    • Docker

    This command shuts down all running members on the local machine. 
    hz-stop

    Go to Administration > Cluster State and select Shutdown.

    Use docker stop to shut down each container.

    docker stop <CONTAINER_NAME>

  5. Check that all members are shut down. The cluster dashboard in Management Center shows which members are in SHUT_DOWN state. If all members are shut down, Management Center will disconnect from the cluster.

  6. Recreate the cluster:

    • CLI

    • Docker

    Use hz start to create each new member. They will form a cluster automatically.

    hz start

    Use docker start to restart the containers and create each new member. They will form a cluster automatically.

    docker start <CONTAINER_NAME>

  7. Check that all members are in ACTIVE state using the cluster dashboard in Management Center or change state to ACTIVE if NO_MIGRATION or FROZEN was used for restart.

  8. Confirm the cluster is healthy, for example by running a Healthcheck.

  9. Restore traffic to the cluster.

lifebuoy

Help and support