Rolling Upgrades

6.0-SNAPSHOT 5.5 5.4 5.3 5.2 5.1 5.0

You can use rolling upgrades to upgrade the version of Hazelcast members in a running cluster without interrupting the operation of the cluster.

If you are moving members from a 5.5 Community Edition cluster to an Enterprise Edition cluster, use an in-place rolling restart instead of a rolling upgrade. For further information on rolling restarts, see Move to Enterprise Edition from Community Edition.

Best Practices

Before starting a rolling upgrade, consider the following best practices:

  • Ensure that Upgrade Compatibility covers your requirements

  • If the minimum version of JDK has changed between releases, upgrade the JDK version before starting a rolling upgrade

  • If you used Hazelcast Operator to deploy your cluster, refer to the Operator documentation

  • If using Hazelcast Management Center to trigger your rolling upgrade, upgrade Management Center before upgrading your members. For information on upgrading Management Center, refer to the Management Center documentation, or if you deployed Management Center using the Hazelcast Operator, refer to the Operator documentation

  • If using the hz-cluster-admin script to update your cluster, enable the CLUSTER_WRITE endpoints in the REST API. For further information on the Community Edition REST API, see Community Edition REST API. For further information on the Enterprise Edition REST API, see Enterprise Edition REST API

  • Plan sufficient time for the migration and upgrade to complete, and remember that the members cannot change during the upgrade

  • Avoid issues where an automatic upgrade is triggered before the last member has been upgraded by setting a minimum cluster size

Before you Begin

Hazelcast uses semantic versioning, where versions appear in the following format:

major.minor.patch

For example, in version 5.1.0, 5 is the major version, 1 is the minor version, and 0 is the patch version.

Table 1. Glossary
Term Definition

member codebase version

The major.minor.patch version of the Hazelcast binary on which the member executes. For example, when running on hazelcast-5.2.jar, your member’s codebase version is 5.2.0.

cluster version

The major.minor version at which the cluster operates. This ensures that cluster members are able to communicate using the same cluster protocol and determines the feature set exposed by the cluster.

Upgrade Compatibility

To allow you to use rolling upgrades, Hazelcast provides the following GA (general availability) release compatibility:

  • Hazelcast members operating on the same major and minor cluster version are always compatible, regardless of patch version.

  • Each minor version is compatible with the previous one.

As a result, you can use rolling upgrades to upgrade clusters to the following member codebase versions:

  • A new patch version of the same major and minor version (Community Edition and Enterprise Edition)

  • The next minor version of the same major version (Enterprise Edition only)

    This also includes the ability of rolling upgrade from Hazelcast IMDG 4.2 to Platform 5.0, since IMDG is part of the Platform and 5.0 is considered to be the next minor version after IMDG 4.2.

  • The 5.5 Long-term Support (LTS) release direct from 5.2 or later. For further information on additional compatibilities for direct-to-LTS rolling upgrades, see Long-term Support Releases.

Following the rolling upgrade, you must resubmit jobs as they don’t run during the upgrade. For further information on the compatibility of job states in upgrades, see Versioning and Compatibility.

Upgrade to LTS Release

With the introduction of Hazelcast Platform LTS release, you can upgrade directly from 5.2.x, or later, to 5.5. This means that, if using a supported earlier version, you do not need to upgrade to each version between your current version and the LTS release.

For more information on LTS releases, and any specific considerations when upgrading, see Long-term Support Releases.

JDK Version Upgrade

As newer JDK releases contain improvements, we recommend that you upgrade to a newer JDK version where supported.

Upgrade the JDK version before upgrading your members if support for the JDK version has changed between your current and planned cluster versions. For example, in 5.2 JDK 8 or later was supported, but 5.5 supports only JDK 17 or later.

Performing a Rolling Upgrade

If you deployed your cluster using the Hazelcast Operator, follow the steps in the Operator documentation.

To upgrade your cluster after deploying with Hazelcast Platform, complete the following steps:

  1. Gracefully shut down an existing member.

    See Shutting Down a Hazelcast Member.

  2. Wait until all partition migrations are completed.

    During migrations, members cannot join or leave the cluster.

  3. Upgrade the member’s codebase to a new patch version or the next minor version.

    To upgrade the codebase to the next minor version, you need an Enterprise license.

  4. Start the member and wait until it joins the cluster.

    You should see something like the following in your logs:

     ...
 INFO: [192.168.2.2]:5701 [cluster] [{new codebase version}] (1)
 Hazelcast Enterprise {new codebase version} (20170630 - a67dc3a) starting at [192.168.2.2]:5701
 ...
 INFO: [192.168.2.2]:5701 [cluster] [{new codebase version}] Cluster version set to {previous codebase version} (2)
    1 The member’s codebase version. Once the member locates the existing cluster members, it sends its join request to the master member. The master member validates that the new member is allowed to join the cluster and lets the new member know that the cluster is currently operating on the previous cluster version.
    2 The member sets its cluster version to the previous one and starts operating normally.
    Do not run members on different codebase versions for prolonged periods of time.

  5. Repeat this process for each member in the cluster.

    When all members of the cluster have been upgraded to the new codebase version, the cluster will still be operating on the previous cluster version. In order to use the new codebase version the cluster version must be upgraded to that version, using a rolling upgrade.

  6. Trigger a rolling upgrade on the cluster, using one of the following:

    • Management Center.

      To use Management Center, you need to upgrade your version of Management Center before upgrading the member version. Management Center is compatible with the previous minor version of Hazelcast. For example, Management Center 5.4 works with both Hazelcast 5.3 and 5.4. To change your cluster version to 5.4, you need Management Center 5.4.

    • hz-cluster-admin script.

      To use this script, you must enable the CLUSTER_WRITE REST endpoint group (its default is disabled). See REST Endpoint Groups.
      Deprecation Notice for the Community Edition REST API

      The Community Edition REST API has been deprecated and will be removed as of Hazelcast version 7.0. An improved Enterprise version of this feature is available and actively being developed. For more info, see Enterprise REST API

    • Allow the cluster to auto-upgrade.

The cluster is now operating on the new codebase version: Both the codebase version and the cluster version have been upgraded.

Enabling Auto-Upgrading

The cluster can automatically upgrade its version. As soon as it detects that all its members have a version higher than the current cluster version, it upgrades the cluster version to match it. This feature is disabled by default. To enable it, set the system property hazelcast.cluster.version.auto.upgrade.enabled to true.

There is one tricky detail here: as you are shutting down and upgrading the members one by one, when you shut down the last one, all the members in the remaining cluster have the newer version, but you don’t want the auto-upgrade to kick in before you have successfully upgraded the last member as well. To avoid this, you can use the hazelcast.cluster.version.auto.upgrade.min.cluster.size system property. You should set it to the size of your cluster, and then Hazelcast will wait for the last member to join before it can proceed with the auto-upgrade.

Network Partitions and Rolling Upgrades

In the event of network partitions which split your cluster into two subclusters, split-brain handling works as explained in the Network Partitioning chapter, with the additional constraint that two subclusters only merge as long as they operate on the same cluster version. This is a requirement to ensure that all members participating in each one of the subclusters are able to operate as members of the merged cluster at the same cluster version.

With regards to rolling upgrades, the above constraint implies that if a network partition occurs while a change of cluster version is in progress, then with some unlucky timing, one subcluster may be upgraded to the new cluster version and another subcluster may have upgraded members but still operate at the old cluster version.

In order for the two subclusters to merge, it is necessary to change the cluster version of the subcluster that still operates on the old cluster version, so that both subclusters will be operating at the same, upgraded cluster version and able to merge as soon as the network partition is fixed.

Using Rolling Restarts to Switch to High-Density Memory Store

As a Hazelcast Platform Enterprise user, you can perform a rolling restart on your cluster to start using High-Density Memory Store in your maps.

For this you need to:

  1. Update your cluster and map configurations:

  2. Perform a rolling restart of your members, i.e., restart each cluster member by shutting down and starting them again.

Your maps are now backed with High-Density Memory Store.

Rolling Upgrade FAQ

The following provide answers to the frequently asked questions related to rolling member upgrades.

How is the cluster version set?

When a new member starts, it is not yet joined to a cluster; therefore its cluster version is still undetermined. In order for the cluster version to be set, one of the following must happen:

  • the member cannot locate any members of the cluster to join or is configured without a joiner: in this case, the member appoints itself as the master of a new single-member cluster and its cluster version is set to the major.minor version of its own codebase version. So a standalone member running on codebase version 5.2.0 sets its own cluster version to 5.2.

  • the member that is starting locates members of the cluster and identifies which is the master: in this case, the master validates that the joining member’s codebase version is compatible with the current cluster version. If it is found to be compatible, then the member joins and the master sends the cluster version, which is set on the joining member. Otherwise, the starting member fails to join and shuts down.

What if a new Hazelcast minor version changes fundamental cluster protocol communication, like join messages?

The version numbers used in the paragraph below are only used as an example.

On startup, as answered in the above question (How is the cluster version set?), the cluster version is not yet known to a member that has not joined any cluster. By default the newly started member uses the cluster protocol that corresponds to its codebase version until this member joins a cluster (so for codebase 5.2.0 this means implicitly assuming cluster version 5.2). If, hypothetically, major changes in discovery & join operations have been introduced which do not allow the member to join a 5.1 cluster, then the member should be explicitly configured to start assuming a 5.1 cluster version.

Do I have to upgrade clients to work with rolling upgrades?

Clients which implement the Open Binary Client Protocol are compatible with Hazelcast version 3.6 and newer minor versions. Thus, older client versions are compatible with next minor versions. Newer clients connected to a cluster operate at the lower version of capabilities until all members are upgraded and the cluster version upgrade occurs.

Can I stop and start multiple members at once during a rolling member upgrade?

It is not recommended due to potential network partitions. It is advised to always stop and start one member in each upgrade step.

Can I upgrade my business app together with Hazelcast while doing a rolling member upgrade?

Yes, but make sure to make the new version of your app compatible with the old one since there will be a timespan when both versions interoperate. Checking if two versions of your app are compatible includes verifying binary and algorithmic compatibility and some other steps.

It is worth mentioning that a business app upgrade is orthogonal to a rolling member upgrade. A rolling business app upgrade may be done without upgrading the members.

Are features marked as Beta in my current version upgraded?

Features marked as Beta in the source version are upgraded, but are supported only in a cluster of the same version as the source version due to changes that can occur between releases.

This means that you can encounter failures if, for example, changes have been made to the field format or where there is a SQL dependency. Following any failure, where the Beta feature has been updated or made generally available in the target version, the updated version of the feature is used.

Can I upgrade directly to the latest release?

You can upgrade directly to the latest release in the following circumstances:

  • Your source version is no more than one minor version below the latest (target) version

  • The target version is an LTS release and your source version is among those supported for direct-to-LTS upgrades. For further information on LTS releases and the supported source versions, see Long-term Releases

lifebuoy

Help and support