This is a prerelease version.

View latest

Deploy a Cluster with the Enterprise Operator (Preview)

Hazelcast Enterprise Operator is currently in the early alpha preview stage, but it will soon become the preferred way of installing Hazelcast in Kubernetes/OpenShift environments.

The simplest way of installing and operating your Hazelcast Enterprise cluster in the Kubernetes (and OpenShift) environments is to use Hazelcast Enterprise Operator.

Deploy Hazelcast Enterprise Operator

You can deploy Hazelcast Enterprise Operator by creating the following bundle.yaml file.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.4.1
  creationTimestamp: null
  name: hazelcasts.hazelcast.com
spec:
  group: hazelcast.com
  names:
    kind: Hazelcast
    listKind: HazelcastList
    plural: hazelcasts
    singular: hazelcast
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - description: Current state of the Hazelcast deployment
      jsonPath: .status.phase
      name: Status
      type: string
    - description: Current numbers of ready Hazelcast members
      jsonPath: .status.hazelcastClusterStatus.readyMembers
      name: Members
      type: string
    name: v1alpha1
    schema:
      openAPIV3Schema:
        description: Hazelcast is the Schema for the hazelcasts API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: HazelcastSpec defines the desired state of Hazelcast
            properties:
              clusterName:
                default: dev
                description: Name of the Hazelcast cluster.
                type: string
              clusterSize:
                default: 3
                description: Number of Hazelcast members in the cluster.
                format: int32
                minimum: 1
                type: integer
              exposeExternally:
                description: Configuration to expose Hazelcast cluster to external clients.
                properties:
                  discoveryServiceType:
                    description: Type of the service used to discover Hazelcast cluster.
                    type: string
                  memberAccess:
                    description: 'How each member is accessed from the external client. Valid values are: - "NodePortExternalIP" (default): each member is accessed by the NodePort service and the node external IP/hostname - "NodePortNodeName": each member is accessed by the NodePort service and the node name - "LoadBalancer": each member is accessed by the LoadBalancer service external address'
                    enum:
                    - NodePortExternalIP
                    - NodePortNodeName
                    - LoadBalancer
                    type: string
                  type:
                    description: 'Specifies how members are exposed. Valid values are: - "Smart" (default): each member pod is exposed with a separate external address - "Unisocket": all member pods are exposed with one external address'
                    enum:
                    - Smart
                    - Unisocket
                    type: string
                type: object
              licenseKeySecret:
                default: hazelcast-license-key
                description: Name of the secret with Hazelcast Enterprise License Key.
                type: string
              repository:
                default: docker.io/hazelcast/hazelcast-enterprise
                description: Repository to pull the Hazelcast Platform image from.
                type: string
              version:
                default: '5.1-SNAPSHOT-slim'
                description: Version of Hazelcast Platform.
                type: string
            type: object
          status:
            description: HazelcastStatus defines the observed state of Hazelcast
            properties:
              hazelcastClusterStatus:
                description: HazelcastClusterStatus defines the status of the Hazelcast cluster
                properties:
                  readyMembers:
                    description: ReadyMembers represents the number of members that are connected to cluster from the desired number of members in the format <ready>/<desired>
                    type: string
                required:
                - readyMembers
                type: object
              phase:
                description: Phase represents the current state of the cluster
                type: string
            required:
            - hazelcastClusterStatus
            - phase
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.4.1
  creationTimestamp: null
  name: managementcenters.hazelcast.com
spec:
  group: hazelcast.com
  names:
    kind: ManagementCenter
    listKind: ManagementCenterList
    plural: managementcenters
    singular: managementcenter
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - description: Current state of the Management Center deployment
      jsonPath: .status.phase
      name: Status
      type: string
    name: v1alpha1
    schema:
      openAPIV3Schema:
        description: ManagementCenter is the Schema for the managementcenters API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: ManagementCenterSpec defines the desired state of ManagementCenter.
            properties:
              externalConnectivity:
                description: Configuration to expose Management Center to outside.
                properties:
                  type:
                    default: LoadBalancer
                    description: 'How Management Center is exposed. Valid values are: - "ClusterIP" - "NodePort" - "Loadbalancer" (default)'
                    enum:
                    - ClusterIP
                    - NodePort
                    - LoadBalancer
                    type: string
                type: object
              hazelcastClusters:
                description: Connection configuration for the Hazelcast clusters that Management Center will monitor.
                items:
                  properties:
                    address:
                      description: IP address or DNS name of the Hazelcast cluster. If the cluster is exposed with a service name in a different namespace, use the following syntax "<service-name>.<service-namespace>".
                      type: string
                    name:
                      default: dev
                      description: Name of the Hazelcast cluster that Management Center will connect to, default is dev.
                      type: string
                  required:
                  - address
                  type: object
                type: array
              licenseKeySecret:
                default: hazelcast-license-key
                description: Name of the secret with Hazelcast Enterprise License Key.
                type: string
              persistence:
                description: Configuration for Management Center persistence.
                properties:
                  enabled:
                    default: true
                    type: boolean
                  size:
                    anyOf:
                    - type: integer
                    - type: string
                    default: 10Gi
                    pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                    x-kubernetes-int-or-string: true
                  storageClass:
                    type: string
                type: object
              repository:
                default: docker.io/hazelcast/management-center
                description: Repository to pull the Management Center image from.
                type: string
              version:
                default: '5.1-snapshot'
                description: Version of Management Center.
                type: string
            type: object
          status:
            description: ManagementCenterStatus defines the observed state of ManagementCenter.
            properties:
              phase:
                description: Phase represents the current state of the cluster
                type: string
            required:
            - phase
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: hazelcast-enterprise-controller-manager
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  creationTimestamp: null
  name: hazelcast-enterprise-manager-role
  namespace: default
rules:
- apiGroups:
  - ""
  resources:
  - events
  - serviceaccounts
  - services
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - hazelcast.com
  resources:
  - hazelcasts
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - hazelcast.com
  resources:
  - hazelcasts/finalizers
  verbs:
  - update
- apiGroups:
  - hazelcast.com
  resources:
  - hazelcasts/status
  verbs:
  - get
  - patch
  - update
- apiGroups:
  - hazelcast.com
  resources:
  - managementcenters
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - hazelcast.com
  resources:
  - managementcenters/status
  verbs:
  - get
  - patch
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: hazelcast-enterprise-manager-role
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - nodes
  - pods
  - services
  verbs:
  - get
  - list
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - clusterrolebindings
  - clusterroles
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: hazelcast-enterprise-manager-rolebinding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: hazelcast-enterprise-manager-role
subjects:
- kind: ServiceAccount
  name: hazelcast-enterprise-controller-manager
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: hazelcast-enterprise-manager-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: hazelcast-enterprise-manager-role
subjects:
- kind: ServiceAccount
  name: hazelcast-enterprise-controller-manager
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    control-plane: controller-manager
  name: hazelcast-enterprise-controller-manager
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      control-plane: controller-manager
  template:
    metadata:
      labels:
        control-plane: controller-manager
    spec:
      containers:
      - args:
        - --leader-elect=false
        command:
        - /manager
        env:
        - name: WATCH_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        image: hazelcast/hazelcast-enterprise-operator:5-preview-snapshot
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8081
          initialDelaySeconds: 15
          periodSeconds: 20
        name: manager
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8081
          initialDelaySeconds: 5
          periodSeconds: 10
        resources:
          limits:
            cpu: 100m
            memory: 30Mi
          requests:
            cpu: 100m
            memory: 20Mi
        securityContext:
          allowPrivilegeEscalation: false
      securityContext:
        runAsNonRoot: true
      serviceAccountName: hazelcast-enterprise-controller-manager
      terminationGracePeriodSeconds: 10

Then, you can apply it to your Kubernetes cluster with the following command.

kubectl apply -f bundle.yaml

At this point, your Hazelcast Enteprise Operator should be up and running. You can check it with the following command.

kubectl logs deployment.apps/hazelcast-enterprise-controller-manager
2021-07-14T09:03:40.713Z        INFO    setup   Watching namespace: default
2021-07-14T09:03:41.524Z        INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2021-07-14T09:03:41.524Z        INFO    setup   starting manager
2021-07-14T09:03:41.525Z        INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}

Customize Namespace (optional)

By default, Hazelcast Operator is installed into the default namespace. If you want to change it, you can either replace all namespace: default parameters in bundle.yaml or use Kustomize to achieve it by creating the following kustomization.yaml file.

namespace: <your-namespace>
resources:
- bundle.yaml

Then, use Kustomize to apply bundle.yaml with the changed namespace.

kubectl apply -k .

Note that you’ll need to add a parameter -n <your-namespace> to all further commands.

Create Secret with Hazelcast License Key

Hazelcast Enteprise requires having the license key specified as Kubernetes Secret.

If you don’t have Hazelcast Enterprise License Key, please check Requesting a License Key
kubectl create secret generic hazelcast-license-key --from-literal=license-key=<YOUR LICENSE KEY>

Start Hazelcast Enterprise Cluster

Finally, you can create Hazelcast Custom Resource file as hazelcast.yaml.

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast-sample
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  version: '5.1-SNAPSHOT-slim'
  licenseKeySecret: hazelcast-license-key

Apply it with the following command to start the Hazelcast cluster.

kubectl apply -f hazelcast.yaml

After a moment, you can verify that Hazelcast cluster is up and running by checking the Hazelcast member logs.

kubectl logs pod/hazelcast-sample-0
Members {size:3, ver:3} [
        Member [10.36.8.3]:5701 - ccf31703-de3b-4094-9faf-7b5d0dc145b2 this
        Member [10.36.7.2]:5701 - e75bd6e2-de4b-4360-8113-040773d858b7
        Member [10.36.6.2]:5701 - c3d105d2-0bca-4a66-8519-1cacffc05c98
]

Connect to Hazelcast from Outside Kubernetes

To connect to a Hazelcast cluster from outside Kubernetes, you need to configure the expose-externally field of your custom resource.

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast-sample
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  version: '5.1-SNAPSHOT-slim'
  licenseKeySecret: hazelcast-license-key
  exposeExternally:
    type: Smart
    discoveryServiceType: LoadBalancer
    memberAccess: NodePortExternalIP

After applying the configuration change in Kubernetes, each Hazelcast member will be exposed as a separate service.

kubectl get service
NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)          AGE
hazelcast-sample     LoadBalancer   10.219.246.19    35.230.92.217   5701:30560/TCP   2m11s
hazelcast-sample-0   NodePort       10.219.254.192   <none>          5701:31890/TCP   2m11s
hazelcast-sample-1   NodePort       10.219.247.43    <none>          5701:32310/TCP   2m11s
hazelcast-sample-2   NodePort       10.219.243.16    <none>          5701:32585/TCP   2m11s

To connect to the cluster, you can add the external IP address of the LoadBalancer service to your Hazelcast client configuration.

You can configure the expose-externally field with the following properties:

  • type: The way in which Hazelcast members are exposed to clients.

    • Unisocket: Members are exposed via one Kubernetes service.

    • Smart: Each member is exposed via a separate Kubernetes service.

  • discoveryServiceType: Type of Kubernetes service used for discovering members in a Hazelcast cluster.

    • LoadBalancer (default): Hazelcast client configuration <load-balancer-external-ip>:5701.

    • NodePort: Hazelcast client configuration <node-address>:<node-port>.

  • memberAccess: The way in which clients access Hazelcast members.

    • NodePortExternalIP (default): Hazelcast clients are outside the Kubernetes network, so a client can access members only via the public IP address that’s assigned to their Kubernetes nodes.

    • NodePortNodeName: Hazelcast clients are inside the same network as the Kubernetes nodes, so clients can access a member by the hostname of its Kubernetes node.

    • LoadBalancer: Hazelcast clients are outside the Kubernetes network and can access members via the the external IP assigned to the LoadBalancer service.

Your Kubernetes cluster should be able to assign public IPs to LoadBalancer services to connect to Hazelcast from outside Kubernetes.
If NodePorts are used (discoveryServiceType: NodePort or memberAccess: NodePortExternalIP), then the Kubernetes nodes should have access from outside Kubernetes. This includes both the node IPs and ports.

Google Kubernetes Engine

If you are using Google Kubernetes Engine (GKE) and trying to connect to Hazelcast from outside Kubernetes, then you need to consider the following:

  • The Kubernetes cluster should not be private.

  • If NodePorts are used (discoveryServiceType: NodePort or memberAccess: NodePortExternalIP), a firewall rule for each Hazelcast member should be added to allow TCP traffic on node ports.

Amazon Elastic Kubernetes Service

If you are using Amazon Elastic Kubernetes Service (EKS) and trying to connect to Hazelcast from outside Kubernetes, then you need to consider the following:

  • Having the VPC of the EKS cluster configured only with private subnets is not recommended.

  • If NodePorts are used (discoveryServiceType: NodePort or memberAccess: NodePortExternalIP), the VPC of the EKS cluster should be configured only with public subnets. Additionally, inbound rules for the node ports should be added to the nodes' security groups.

  • AWS load balancers are assigned with hostnames by default. Since it takes some time to resolve these hostnames, the member pods might fail to start with CrashLoopBackOff status in the beginning. But the pods will be running fine eventually after a couple of restarts.

Azure Kubernetes Service

If you are using Azure Kubernetes Service (AKS) and trying to connect to Hazelcast from outside Kubernetes, then you need to consider the following:

  • The Kubernetes cluster should not be private.

  • If NodePorts are used (discoveryServiceType: NodePort or memberAccess: NodePortExternalIP), the node pools should be created by using --enable-node-public-ip flag. Additionally, the required inbound rules should be added for network security groups (NSGs) to allow access to node IPs and ports.

Check that the Hazelcast Cluster is Running

To check if a cluster is running, see the status field of the Hazelcast resource.

status:
  hazelcastClusterStatus:
    readyMembers: 3/3
  phase: Running

The phase field represents the current status of the cluster, and can contain any of the following values:

  • Running: The cluster is up and running.

  • Pending: The cluster is in the process of starting.

  • Failed: An error occurred while starting the cluster.

The readyMembers field represents the number of Hazelcast members that are connected to the cluster.

Use the readyMembers field only for informational purposes. This field is not always accurate. Some members may have joined or left the cluster since this field was last updated.

Start Management Center

You can monitor the Hazelcast cluster by starting the Management Center. You can create ManagementCenter custom resource file as management-center.yaml.

apiVersion: hazelcast.com/v1alpha1
kind: ManagementCenter
metadata:
  name: managementcenter-sample
spec:
  repository: 'hazelcast/management-center'
  version: '5.1-snapshot'
  licenseKeySecret: hazelcast-license-key
  externalConnectivity:
    type: LoadBalancer
  hazelcastClusters:
    - address: hazelcast-sample
      name: dev
  persistence:
    enabled: true
    size: 10Gi
StatefulSet does not support updates to volumeClaimTemplates field, so persistence field should be set only at the creation of the custom resource. Any update to the persistence field will not affect the Management Center.
hazelcastClusters field does not support deleting clusters from the custom resource. If you want to remove a cluster from the Management Center, you can do it from the Management Center UI.

Apply it with the following command to start Management Center.

kubectl apply -f management-center.yaml

After a moment, you can verify that Management Center is up and running by checking the Management Center logs.

kubectl logs managementcenter-sample-0
2021-08-26 15:21:04,842 [ INFO] [MC-Client-dev.lifecycle-1] [c.h.w.s.MCClientManager]: MC Client connected to cluster dev.
2021-08-26 15:21:05,241 [ INFO] [MC-Client-dev.event-1] [c.h.w.s.MCClientManager]: Started communication with member: Member [10.36.8.3]:5701 - ccf31703-de3b-4094-9faf-7b5d0dc145b2
2021-08-26 15:21:05,245 [ INFO] [MC-Client-dev.event-1] [c.h.w.s.MCClientManager]: Started communication with member: Member [10.36.7.2]:5701 - e75bd6e2-de4b-4360-8113-040773d858b7
2021-08-26 15:21:05,251 [ INFO] [MC-Client-dev.event-1] [c.h.w.s.MCClientManager]: Started communication with member: Member [10.36.6.2]:5701 - c3d105d2-0bca-4a66-8519-1cacffc05c98
2021-08-26 15:21:07,234 [ INFO] [main] [c.h.w.Launcher]: Hazelcast Management Center successfully started at http://localhost:8080/

To access the Management Center dashboard, open the browser at address http://$MANCENTER_IP:8080.

MANCENTER_IP=$( kubectl get service managementcenter-sample -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Clean up

Run the following commands to remove Hazelcast cluster, Management Center, Hazelcast License Key Secret, and Hazelcast Enterprise Operator.

kubectl delete -f hazelcast.yaml
kubectl delete -f management-center.yaml
kubectl delete secret hazelcast-license-key
kubectl delete -f bundle.yaml