Restart a cluster

Static configuration is configuration that cannot be changed while members are running. To apply updates to static configuration, you must restart the cluster.

Restart options

There are two options for restarting a cluster:

• Perform a rolling restart
• Perform a whole cluster restart

Where possible, you should perform a rolling restart. In a rolling restart, members are stopped one at a time and each member's data is automatically repartitioned across the remaining cluster members before shutting down. This minimizes the risk of service interruptions and data loss. When the replacement member joins the cluster, data is repartitioned again.

Depending on the scope of your configuration changes, you may need to update a single member, a cluster, or multiple clusters. The process is the same in all cases.

tip

You can also use a rolling restart for canarying: testing changes on a single member before updating the rest of the cluster.

Some configuration needs to be consistent across all members in a cluster. To update this configuration, you must restart the whole cluster simultaneously. By default, this causes all data in the cluster to be lost. To avoid data loss, you must enable persistence. To avoid downtime, you must divert traffic to another Hazelcast cluster before restarting. Hazelcast supports blue-green deployments to facilitate this.

Note that member processes are not reused. In all cases, members are terminated, and new members are created to replace them.

Persistence

If persistence is enabled and the cluster enters NO_MIGRATION or FROZEN state, replacement members will adopt the UUIDs of the terminated members and attempt to load persisted data from disk. This avoids the repartitioning process, helping the cluster recover more quickly during a rolling restart.

Persistence is required to avoid data loss during a whole cluster restart. See Configuring Persistence for details.

Embedded mode

If your cluster is operating in embedded mode, you should restart your host application instead of trying to restart the embedded Hazelcast members independently.

Perform a restart

This section describes how to restart a cluster using a choice of common tools. Hazelcast supports a wide range of deployment options that may require other tools or allow you to automate parts of the process, but the required steps will be similar.

You should perform restarts in a low utilization or maintenance window to reduce the risk of service impacts.

Deprecation Notice for the Community Edition REST API

The Community Edition REST API has been deprecated and will be removed as of Hazelcast version 7.0. An improved Enterprise version of this feature is available and actively being developed. For more information, see Enterprise REST API.

Perform a rolling restart

You should check the cluster is healthy before and after making changes. The best way to do this is to run a Healthcheck in Management Center. For other options, see Monitoring.

To perform a rolling restart:

1. Update the static configuration as required.

2. Gracefully shut down the first member:

   CLI

   important

   This command shuts down all running members on the local machine.

   bash

   hz-stop

   Management Center

   Go to Members, select the member's row in the table and select Shutdown.

   Docker

   bash

   docker stop <CONTAINER_NAME>

3. Wait until all partition migrations are completed. You can monitor the progress of migrations on the cluster dashboard in Management Center.

4. Start a new member.

   CLI

   bash

   hz start

   Docker

   bash

   docker start <CONTAINER_NAME>

5. Wait until the new member joins the cluster and all partition migrations are completed. Again, you can monitor progress using the cluster dashboard in Management Center.

6. Repeat this process until all members have been restarted with the updated configuration.

Perform a whole cluster restart

You should check the cluster is healthy before and after making changes. The best way to do this is to run a Healthcheck in Management Center. For other options, see Monitoring.

To perform a whole cluster restart:

1. Update the static configuration as required.

2. Divert traffic to another Hazelcast cluster. How to do this depends on how your Hazelcast deployment is architected. Management Center provides an easy way to manage client connections using filtering rules.

3. If you have enabled persistence, change the cluster state to NO_MIGRATION or FROZEN:

   CLI

   bash

   hz-cluster-admin -a <MEMBER_ADDRESS> -c <CLUSTER_NAME> -s no_migration

   You can specify the IP address of any member in the cluster.

   Management Center

   Go to Administration > Cluster State and select No Migration or Frozen.

4. Gracefully shut down the cluster:

   CLI

   important

   This command shuts down all running members on the local machine.

   bash

   hz-stop

   Management Center

   Go to Administration > Cluster State and select Shutdown.

   Docker

   Use docker stop to shut down each container.

   bash

   docker stop <CONTAINER_NAME>

5. Check that all members are shut down. The cluster dashboard in Management Center shows which members are in SHUT_DOWN state. If all members are shut down, Management Center will disconnect from the cluster.

6. Recreate the cluster:

   CLI

   Use hz start to create each new member. They will form a cluster automatically.

   bash

   hz start

   Docker

   Use docker start to restart the containers and create each new member. They will form a cluster automatically.

   bash

   docker start <CONTAINER_NAME>

7. Check that all members are in ACTIVE state using the cluster dashboard in Management Center or change state to ACTIVE if NO_MIGRATION or FROZEN was used for restart.

8. Confirm the cluster is healthy, for example by running a Healthcheck.

9. Restore traffic to the cluster.