Connect To Hazelcast Running on Kubernetes from Outside

This guide uses Helm and Kubectl for deploying the Hazelcast cluster. If you want to use the Hazelcast Platform Operator instead, see Connect to Hazelcast from Outside Kubernetes.

What You’ll Learn

Deploy a Hazelcast Kubernetes cluster and connect to it, using a client outside Kubernetes.

Before you Begin

Introduction

There are two approaches you may want to use when setting up on-premises Hazelcast cluster in Kubernetes:

  • Unisocket Client - client sends requests to a random Hazelcast member

  • Smart Client - client connects to all members and sends requests directly to the members owning the data

Let’s see both approaches.

Unisocket Client

The simplest possible scenario is to deploy the Hazelcast cluster on Kubernetes and expose all Hazelcast pods with one LoadBalancer (or NodePort) service. With that approach, we piggyback on the standard Kubernetes mechanism, which automatically load balances the traffic to Hazelcast members.

Hazelcast Unisocket Client
Figure 1. Hazelcast Unisocket Client

Starting Hazelcast cluster

There are different ways of deploying Hazelcast to Kubernetes. For the production environment we recommend using Helm.

  • Kubectl

  • Helm

kubectl apply -f https://raw.githubusercontent.com/hazelcast/hazelcast-kubernetes/master/rbac.yaml

kubectl run hz-hazelcast-0 --image=hazelcast/hazelcast:$HAZELCAST_VERSION -l "role=hazelcast"
kubectl run hz-hazelcast-1 --image=hazelcast/hazelcast:$HAZELCAST_VERSION -l "role=hazelcast"
kubectl run hz-hazelcast-2 --image=hazelcast/hazelcast:$HAZELCAST_VERSION -l "role=hazelcast"

kubectl create service loadbalancer hz-hazelcast --tcp=5701 -o yaml --dry-run=client | kubectl set selector --local -f - "role=hazelcast" -o yaml | kubectl create -f -
helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/
helm repo update
helm install hz-hazelcast --set service.type=LoadBalancer hazelcast/hazelcast

Verifying Hazelcast cluster

You can check that the Hazelcast cluster is up and running.

kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
hz-hazelcast-0   1/1     Running   0          32s
hz-hazelcast-1   1/1     Running   0          30s
hz-hazelcast-2   1/1     Running   0          29s

You can also check that all Hazelcast members formed a cluster.

kubectl logs hz-hazelcast-0
...
Members {size:3, ver:3} [
        Member [10.216.6.7]:5701 - 6d2100e0-8dcf-4e7c-ab40-8e98e23475e3 this
        Member [10.216.5.6]:5701 - 5ab4d554-fd7d-4929-8475-0ddf79a21076
        Member [10.216.8.6]:5701 - 7f7dd5f4-e732-4575-89d6-a6e823da38da
]

At this point you have a Hazelcast cluster exposed with a single LoadBalancer service called hz-hazelcast. You can find it’s address with the following command.

kubectl get service hz-hazelcast
NAME           TYPE           CLUSTER-IP       EXTERNAL-IP       PORT(S)          AGE
hz-hazelcast   LoadBalancer   10.108.141.178   10.96.184.178     5701:31434/TCP   5m44s

If you are using Minikube, you need to execute minikube tunnel now in order to get LoadBalancer External IPs assigned.

The field EXTERNAL-IP is the address of your Hazelcast cluster.

Connecting with Hazelcast Client

Configure the Hazelcast client with the external address and disable smart routing to use the unisocket connection.

  • Java

  • NodeJS

  • Go

  • Python

  • C++

ClientConfig config = new ClientConfig();
config.getNetworkConfig().addAddress("<EXTERNAL-IP>")
                         .setSmartRouting(false);
const { Client } = require('hazelcast-client');

const clientConfig = {
    network: {
        clusterMembers: [
            '<EXTERNAL-IP>'
        ],
        smartRouting: false
    }
};
const client = await Client.newHazelcastClient(clientConfig);
import (
	"log"

	"github.com/hazelcast/hazelcast-go-client"
)

func main() {
	config := hazelcast.Config{}
	cc := &config.Cluster
	cc.Network.SetAddresses("<EXTERNAL-IP>:5701")
	cc.Discovery.UsePublicIP = true
	cc.Unisocket = true
	ctx := context.TODO()
	client, err := hazelcast.StartNewClientWithConfig(ctx, config)
	if err != nil {
		panic(err)
	}
}
import logging

import hazelcast

logging.basicConfig(level=logging.INFO)

client = hazelcast.HazelcastClient(
    cluster_members=["<EXTERNAL-IP>"],
    use_public_ip=True,
    smart_routing=False,
)
#include <hazelcast/client/hazelcast_client.h>

int main(int argc, char **argv) {
    hazelcast::client::client_config config;
    config.get_network_config().use_public_address(true)
                            .add_address(hazelcast::client::address{"<EXTERNAL-IP>", 5701})
                            .set_smart_routing(false);
    auto client = hazelcast::new_client(std::move(config)).get();

    return 0;
}

Finally, start the client application using the following command.

  • Java

  • NodeJS

  • Go

  • Python

  • C++

cd java-unisocket
mvn package
java -jar target/*jar-with-dependencies*.jar
cd nodejs-unisocket
npm install
npm start
cd go-unisocket
go run main.go
cd python-unisocket
pip install -r requirements.txt
python main.py
cd cpp-unisocket
cmake -B build -S . -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=/path/to/vcpkg/scripts/buildsystems/vcpkg.cmake
cmake --build build --verbose
./build/cpp-unisocket

You should see the following output.

Successful connection!
Starting to fill the map with random entries.
Current map size: 2
Current map size: 3
Current map size: 4
Current map size: 5
Current map size: 6
Current map size: 7
Current map size: 8
Current map size: 9
Current map size: 10

Clean Up

To clean up the environment execute the following commands.

  • Kubectl

  • Helm

kubectl delete pod/hz-hazelcast-0 pod/hz-hazelcast-1 pod/hz-hazelcast-2 service/hz-hazelcast
kubectl delete -f https://raw.githubusercontent.com/hazelcast/hazelcast-kubernetes/master/rbac.yaml
helm uninstall hz-hazelcast

Unisocket is very simple to configure; however, it has one significant drawback - low performance. Using a load balancer is perfect for traffic distribution across stateless services; however, Hazelcast is not stateless. Actually, Hazelcast is more like a sharded database in which each member contains a different part of the data. That is why it’s suboptimal to randomly load balance the traffic. It effectively means that each operation needs to be internally migrated, because your data may be load balanced to member 1, while the partition for its key is stored in member 2. All in all, if performance is important for your use case, then you need to use smart client.

Smart Client

Hazelcast smart client is capable of mapping the given key with its owner member, which means that it sends the data directly to the member which contains the right data partition. This fact implies that in the Kubernetes environment, we need to provide access to each Hazelcast pod from the outside. A dynamic approach to this problem is to expose each pod with a separate service. Again, the service can be either LoadBalancer or NodePort. In a real-life scenario, that would probably be NodePort, since having a separate public IP for each pod is expensive.

Hazelcast Smart Client
Figure 2. Hazelcast Smart Client

Kubernetes does not offer a feature for automatically creating a service for each pod. That is why to set up a cluster this way we need to either expose each pod manually with kubectl or use the project called Metacontroller, which enables such functionality.

If you are using Minikube, you need to execute minikube tunnel now in order to get LoadBalancer External IPs assigned.

  • Kubectl

  • Helm and Metacontroller

To create a loadbalancer for each running Hazelcast pod you need to run the following commands:

kubectl apply -f https://raw.githubusercontent.com/hazelcast/hazelcast-kubernetes/master/rbac.yaml

kubectl create service loadbalancer hz-hazelcast-0 --tcp=5701
kubectl run hz-hazelcast-0 --image=hazelcast/hazelcast:$HAZELCAST_VERSION --port=5701 -l "app=hz-hazelcast-0,role=hazelcast"
kubectl create service loadbalancer hz-hazelcast-1 --tcp=5701
kubectl run hz-hazelcast-1 --image=hazelcast/hazelcast:$HAZELCAST_VERSION --port=5701 -l "app=hz-hazelcast-1,role=hazelcast"
kubectl create service loadbalancer hz-hazelcast-2 --tcp=5701
kubectl run hz-hazelcast-2 --image=hazelcast/hazelcast:$HAZELCAST_VERSION --port=5701 -l "app=hz-hazelcast-2,role=hazelcast"

kubectl create service loadbalancer hz-hazelcast --tcp=5701 -o yaml --dry-run=client | kubectl set selector --local -f - "role=hazelcast" -o yaml | kubectl create -f -
  • Install Metacontroller plugin

To install Metacontroller plugin, execute the following commands:

# Create metacontroller namespace.
kubectl apply -f https://raw.githubusercontent.com/metacontroller/metacontroller/v2.1.1/manifests/production/metacontroller-namespace.yaml
# Create metacontroller service account and role/binding.
kubectl apply -f https://raw.githubusercontent.com/metacontroller/metacontroller/v2.1.1/manifests/production/metacontroller-rbac.yaml
# Create CRDs for Metacontroller APIs
kubectl apply -f https://raw.githubusercontent.com/metacontroller/metacontroller/v2.1.1/manifests/production/metacontroller-crds-v1.yaml
# Create Metacontroller StatefulSet.
kubectl apply -f https://raw.githubusercontent.com/metacontroller/metacontroller/v2.1.1/manifests/production/metacontroller.yaml

If you have any issues while creating Metacontroller, it may mean that you don’t have ClusterRole access to your cluster. Please check this for details.

  • Install Service-Per-Pod DecoratorController

To install Service-Per-Pod Decorator Controller, you need to execute the following commands.

kubectl apply -k service-per-pod

This Decorator Controller automatically creates a service for each POD marked with the specific annotations.

  • Start Hazelcast cluster

values.yaml
service:
  type: "LoadBalancer"
annotations:
  service-per-pod-label: "statefulset.kubernetes.io/pod-name"
  service-per-pod-ports: "5701:5701"
initContainers:
  - name: wait-for-lb
    image: bitnami/kubectl:1.22
    env:
      - name: POD_NAMESPACE
        valueFrom:
          fieldRef:
            fieldPath: metadata.namespace
      - name: POD_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.name
    command:
      - "sh"
      - "-c"
    args:
      - until [ -n "$$(kubectl get svc -n $${POD_NAMESPACE} -l app=service-per-pod -ojsonpath="{.items[?(@.spec.selector.statefulset\.kubernetes\.io/pod-name==\"$${POD_NAME}\")].status.loadBalancer.ingress[0].ip}")" ]; do sleep 8; done
helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/
helm repo update
helm install hz-hazelcast -f service-per-pod/values.yaml hazelcast/hazelcast

Note that each service created per pod must start before the Hazelcast pod itself. Otherwise Hazelcast won’t be able to resolve its public addresses.

Verifying Hazelcast cluster

You can check that the Hazelcast cluster is up and running.

kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
hz-hazelcast-0   1/1     Running   0          32s
hz-hazelcast-1   1/1     Running   0          30s
hz-hazelcast-2   1/1     Running   0          29s

At this point, you should also have every Hazelcast member exposed with a separate externally accessible address.

kubectl get service
NAME             TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)          AGE
hz-hazelcast     LoadBalancer   10.219.246.40    35.230.84.127    5701:30443/TCP   4m2s
hz-hazelcast-0   LoadBalancer   10.219.255.141   34.145.108.167   5701:30091/TCP   4m7s
hz-hazelcast-1   LoadBalancer   10.219.241.203   34.82.71.106     5701:30687/TCP   4m5s
hz-hazelcast-2   LoadBalancer   10.219.247.106   35.247.93.190    5701:32452/TCP   4m4s

We’ll use the hz-hazelcast service for the discovery.

kubectl get service hz-hazelcast
NAME           TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
hz-hazelcast   LoadBalancer   10.219.246.40   35.230.84.127   5701:30443/TCP   5m29s

The field EXTERNAL-IP is the address of your Hazelcast cluster.

Connecting with Hazelcast Client

Configure the Hazelcast client to connect to the cluster external address.

  • Java

  • NodeJS

  • Go

  • Python

  • C++

ClientConfig config = new ClientConfig();
config.getNetworkConfig().addAddress("<EXTERNAL-IP>");
const { Client } = require('hazelcast-client');

const clientConfig = {
    network: {
        clusterMembers: [
            '<EXTERNAL-IP>'
        ]
    }
};
const client = await Client.newHazelcastClient(clientConfig);
import (
	"log"

	"github.com/hazelcast/hazelcast-go-client"
)

func main() {
	config := hazelcast.Config{}
	cc := &config.Cluster
	cc.Network.SetAddresses("<EXTERNAL-IP>:5701")
	cc.Discovery.UsePublicIP = true
	ctx := context.TODO()
	client, err := hazelcast.StartNewClientWithConfig(ctx, config)
	if err != nil {
		panic(err)
	}
}
import logging

import hazelcast

logging.basicConfig(level=logging.INFO)

client = hazelcast.HazelcastClient(
    cluster_members=["<EXTERNAL-IP>"],
    use_public_ip=True,
)
#include <hazelcast/client/hazelcast_client.h>

int main(int argc, char **argv) {
    hazelcast::client::client_config config;
    config.get_network_config().use_public_address(true)
                            .add_address(hazelcast::client::address{"<EXTERNAL-IP>", 5701});
    auto client = hazelcast::new_client(std::move(config)).get();

    return 0;
}

Finally, start the client application using the following command.

  • Java

  • NodeJS

  • Go

  • Python

  • C++

cd java
mvn package
java -jar target/*jar-with-dependencies*.jar
cd nodejs
npm install
npm start
cd go
go run main.go
cd python
pip install -r requirements.txt
python main.py
cd cpp
cmake -B build -S . -DCMAKE_BUILD_TYPE=Release -DCMAKE_TOOLCHAIN_FILE=/path/to/vcpkg/scripts/buildsystems/vcpkg.cmake
cmake --build build --verbose
./build/cpp

You should see the following output.

Successful connection!
Starting to fill the map with random entries.
Current map size: 2
Current map size: 3
Current map size: 4
Current map size: 5
Current map size: 6
Current map size: 7
Current map size: 8
Current map size: 9
Current map size: 10

Clean Up

To clean up the environment execute the following commands.

  • Kubectl

  • Helm and Metacontroller

kubectl delete pod/hz-hazelcast-0 service/hz-hazelcast-0 pod/hz-hazelcast-1 service/hz-hazelcast-1 pod/hz-hazelcast-2 service/hz-hazelcast-2 service/hz-hazelcast
kubectl delete -f https://raw.githubusercontent.com/hazelcast/hazelcast-kubernetes/master/rbac.yaml
helm uninstall hz-hazelcast

kubectl delete -k service-per-pod

kubectl delete -f https://raw.githubusercontent.com/metacontroller/metacontroller/v2.1.1/manifests/production/metacontroller.yaml
kubectl delete -f https://raw.githubusercontent.com/metacontroller/metacontroller/v2.1.1/manifests/production/metacontroller-crds-v1.yaml
kubectl delete -f https://raw.githubusercontent.com/metacontroller/metacontroller/v2.1.1/manifests/production/metacontroller-rbac.yaml
kubectl delete -f https://raw.githubusercontent.com/metacontroller/metacontroller/v2.1.1/manifests/production/metacontroller-namespace.yaml