Deploy a Cluster with the Enterprise Operator (Preview)

Hazelcast Enterprise Operator is currently in the early alpha preview stage, but it will soon become the preferred way of installing Hazelcast in Kubernetes/OpenShift environments.

The simplest way of installing and operating your Hazelcast Enterprise cluster in the Kubernetes (and OpenShift) environments is to use Hazelcast Enterprise Operator.

Deploy Hazelcast Enterprise Operator

You can deploy Hazelcast Enterprise Operator by creating the following bundle.yaml file.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.4.1
  creationTimestamp: null
  name: hazelcasts.hazelcast.com
spec:
  group: hazelcast.com
  names:
    kind: Hazelcast
    listKind: HazelcastList
    plural: hazelcasts
    singular: hazelcast
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - description: Current state of the Hazelcast deployment
      jsonPath: .status.phase
      name: Status
      type: string
    name: v1alpha1
    schema:
      openAPIV3Schema:
        description: Hazelcast is the Schema for the hazelcasts API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: HazelcastSpec defines the desired state of Hazelcast
            properties:
              clusterSize:
                default: 3
                description: Number of Hazelcast members in the cluster.
                format: int32
                minimum: 1
                type: integer
              exposeExternally:
                description: Configuration to expose Hazelcast cluster to external clients.
                properties:
                  discoveryServiceType:
                    description: Type of the service used to discover Hazelcast cluster.
                    enum:
                    - LoadBalancer
                    - NodePort
                    type: string
                  memberAccess:
                    description: 'Method of how each member is accessed from the external client. Valid values are: - "NodePortExternalIP" (default): each member is accessed by the NodePort service and the node external IP/hostname - "NodePortNodeName": each member is accessed by the NodePort service and the node name - "LoadBalancer": each member is accessed by the LoadBalancer service external address'
                    enum:
                    - NodePortExternalIP
                    - NodePortNodeName
                    - LoadBalancer
                    type: string
                  type:
                    description: 'Specifies how members are exposed. Valid values are: - "Smart" (default): each member pod is exposed with a separate external address - "Unisocket": all member pods are exposed with one external address'
                    enum:
                    - Smart
                    - Unisocket
                    type: string
                type: object
              licenseKeySecret:
                default: hazelcast-license-key
                description: Name of the secret with Hazelcast Enterprise License Key.
                type: string
              repository:
                default: docker.io/hazelcast/hazelcast-enterprise
                description: Repository to pull the Hazelcast Platform image from.
                type: string
              version:
                default: 5.0-SNAPSHOT
                description: Version of Hazelcast Platform.
                type: string
            type: object
          status:
            description: HazelcastStatus defines the observed state of Hazelcast
            properties:
              phase:
                description: Phase represents the current state of the cluster
                type: string
            required:
            - phase
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: hazelcast-enterprise-controller-manager
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  creationTimestamp: null
  name: hazelcast-enterprise-manager-role
  namespace: default
rules:
- apiGroups:
  - ""
  resources:
  - events
  - serviceaccounts
  - services
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - hazelcast.com
  resources:
  - hazelcasts
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - hazelcast.com
  resources:
  - hazelcasts/finalizers
  verbs:
  - update
- apiGroups:
  - hazelcast.com
  resources:
  - hazelcasts/status
  verbs:
  - get
  - patch
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: hazelcast-enterprise-manager-role
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - nodes
  - pods
  - services
  verbs:
  - get
  - list
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - clusterrolebindings
  - clusterroles
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: hazelcast-enterprise-manager-rolebinding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: hazelcast-enterprise-manager-role
subjects:
- kind: ServiceAccount
  name: hazelcast-enterprise-controller-manager
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: hazelcast-enterprise-manager-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: hazelcast-enterprise-manager-role
subjects:
- kind: ServiceAccount
  name: hazelcast-enterprise-controller-manager
  namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    control-plane: controller-manager
  name: hazelcast-enterprise-controller-manager
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      control-plane: controller-manager
  template:
    metadata:
      labels:
        control-plane: controller-manager
    spec:
      containers:
      - args:
        - --leader-elect=false
        command:
        - /manager
        env:
        - name: WATCH_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        image: hazelcast/hazelcast-enterprise-operator:5-preview-snapshot
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8081
          initialDelaySeconds: 15
          periodSeconds: 20
        name: manager
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8081
          initialDelaySeconds: 5
          periodSeconds: 10
        resources:
          limits:
            cpu: 100m
            memory: 30Mi
          requests:
            cpu: 100m
            memory: 20Mi
        securityContext:
          allowPrivilegeEscalation: false
      securityContext:
        runAsNonRoot: true
      serviceAccountName: hazelcast-enterprise-controller-manager
      terminationGracePeriodSeconds: 10

Then, you can apply it to your Kubernetes cluster with the following command.

kubectl apply -f bundle.yaml

At this point, your Hazelcast Enteprise Operator should be up and running. You can check it with the following command.

kubectl logs deployment.apps/hazelcast-enterprise-controller-manager

2021-07-14T09:03:40.713Z        INFO    setup   Watching namespace: default
2021-07-14T09:03:41.524Z        INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2021-07-14T09:03:41.524Z        INFO    setup   starting manager
2021-07-14T09:03:41.525Z        INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
...

Customize Namespace (optional)

By default, Hazelcast Operator is installed into the default namespace. If you want to change it, you can either replace all namespace: default parameters in bundle.yaml or use Kustomize to achieve it by creating the following kustomization.yaml file.

namespace: <your-namespace>
resources:
- bundle.yaml

Then, use Kustomize to apply bundle.yaml with the changed namespace.

kubectl apply -k .

Note that you’ll need to add a parameter -n <your-namespace> to all further commands.

Create Secret with Hazelcast License Key

Hazelcast Enteprise requires having the license key specified as Kubernetes Secret.

If you don’t have Hazelcast Enterprise License Key, please check Requesting a License Key
kubectl create secret generic hazelcast-license-key --from-literal=license-key=<YOUR LICENSE KEY>

Start Hazelcast Enterprise Cluster

Finally, you can create Hazelcast Custom Resource file as hazelcast.yaml.

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast-sample
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  version: '{full-version}-slim'
  licenseKeySecret: hazelcast-license-key

Apply it with the following command to start the Hazelcast cluster.

kubectl apply -f hazelcast.yaml

After a moment, you can verify that Hazelcast cluster is up and running by checking the Hazelcast member logs.

kubectl logs pod/hazelcast-sample-0
...
Members {size:3, ver:3} [
        Member [10.36.8.3]:5701 - ccf31703-de3b-4094-9faf-7b5d0dc145b2 this
        Member [10.36.7.2]:5701 - e75bd6e2-de4b-4360-8113-040773d858b7
        Member [10.36.6.2]:5701 - c3d105d2-0bca-4a66-8519-1cacffc05c98
]
...

Connect to Hazelcast from Outside Kubernetes

To connect to a Hazelcast cluster from outside Kubernetes, you need to configure the expose-externally field of your custom resource.

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast-sample
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  version: '{full-version}-slim'
  licenseKeySecret: hazelcast-license-key
  exposeExternally:
    type: Smart
    discoveryServiceType: LoadBalancer
    memberAccess: NodePortExternalIP

After applying the configuration change in Kubernetes, each Hazelcast member will be exposed as a separate service.

kubectl get service
NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)          AGE
hazelcast-sample     LoadBalancer   10.219.246.19    35.230.92.217   5701:30560/TCP   2m11s
hazelcast-sample-0   NodePort       10.219.254.192   <none>          5701:31890/TCP   2m11s
hazelcast-sample-1   NodePort       10.219.247.43    <none>          5701:32310/TCP   2m11s
hazelcast-sample-2   NodePort       10.219.243.16    <none>          5701:32585/TCP   2m11s

To connect to the cluster, you can add the external IP address of the LoadBalancer service to your Hazelcast client configuration.

You can configure the expose-externally field with the following properties:

  • type: The way in which Hazelcast members are exposed to clients.

    • Unisocket: Members are exposed via one Kubernetes service.

    • Smart: Each member is exposed via a separate Kubernetes service.

  • discoveryServiceType: Type of Kubernetes service used for discovering members in a Hazelcast cluster.

    • LoadBalancer (default): Hazelcast client configuration <load-balancer-external-ip>:5701.

    • NodePort: Hazelcast client configuration <node-address>:<node-port>.

  • memberAccess: The way in which clients access Hazelcast members.

    • NodePortExternalIP (default): Hazelcast clients are outside the Kubernetes network, so a client can access members only via the public IP address that’s assigned to their Kubernetes nodes.

    • NodePortNodeName: Hazelcast clients are inside the same network as the Kubernetes nodes, so clients can access a member by the hostname of its Kubernetes node.

    • LoadBalancer: Hazelcast clients are outside the Kubernetes network and can access members via the the external IP assigned to the LoadBalancer service.

Clean up

Run the following commands to remove Hazelcast cluster, Hazelcast License Key Secret, and Hazelcast Enterprise Operator.

kubectl delete -f hazelcast.yaml
kubectl delete secret hazelcast-license-key
kubectl delete -f bundle.yaml