Running Highly Available Hazelcast Clusters

In the Hazelcast Platform Operator, you can use partition groups with High Availability Mode to add resilience to your clusters. Avoid data loss even when a Kubernetes node or whole availability zone is down, and all related Hazelcast members are terminated.

Partition groups let you choose where Hazelcast members store backups of a data partition. You can configure a partition group to backup data inside a member in a different availability zone (ZONE_AWARE) or on a different Kubernetes node (NODE_AWARE). See Partition Group Configuration for more details.

When using either type of partition grouping (ZONE_AWARE or NODE_AWARE) with a Hazelcast cluster that spans multiple availability zones and nodes, you must have an equal number of members in each zone or node. Otherwise, it results in the uneven distribution of partitions amongst the members.

The highAvailabilityMode parameter allows you to specify partition groups and to automatically distribute members across availability zones and nodes, using the Kubernetes policy topologySpreadConstraints.

Configuring High Availability Mode

Below are the configuration options for the High Availability Mode feature.

Field Description

highAvailabilityMode

Configuration for partition groups and Kubernetes scheduling policy
  • NODE: the partition-group is configured as NODE_AWARE and topologySpreadConstraints is added into the statefulset:

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
          matchLabels:
            app.kubernetes.io/name: hazelcast
            app.kubernetes.io/instance: hazelcast
            app.kubernetes.io/managed-by: hazelcast-platform-operator
  • ZONE: the partition-group is configured as ZONE_AWARE and topologySpreadConstraints is added into the statefulset:

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
          matchLabels:
            app.kubernetes.io/name: hazelcast
            app.kubernetes.io/instance: hazelcast
            app.kubernetes.io/managed-by: hazelcast-platform-operator

Example Configuration

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast
spec:
  highAvailabilityMode: ZONE