This is a prerelease version.

View latest

Connect to Hazelcast from outside Kubernetes

latest-snapshot 5.16 5.15 5.14 5.13 5.12

Hazelcast supports connections from both smart and unisocket clients that are outside your Kubernetes cluster. To allow these clients to connect to Hazelcast, you need to configure the `exposeExternally` field of the Hazelcast custom resource.

For a worked example, see the Connect from outside Kubernetes tutorial.

Prerequisites

The kubectl command line tool must be configured to communicate with your cluster.

Google Kubernetes Engine (GKE)

If you use GKE, consider the following:

  • Make sure the Kubernetes cluster is public.

  • If you plan on using NodePorts, (discoveryServiceType: NodePort or memberAccess: NodePortExternalIP), set up a firewall rule for each Hazelcast member to allow TCP traffic on node ports.

Amazon Elastic Kubernetes Service (EKS)

If you use EKS, consider the following:

  • Having the VPC of the EKS cluster configured only with private subnets is not recommended.

  • If you plan on using NodePorts, (discoveryServiceType: NodePort or memberAccess: NodePortExternalIP), the VPC of the EKS cluster should be configured only with public subnets. Additionally, inbound rules for the node ports should be added to the nodes' security groups.

For Smart clients with LoadBalancer discovery and member access, the default in-tree controller may create a Classic Load Balancer with TCP health checks that can cause noisy member logs. The recommended approach is to use the AWS Load Balancer Controller with Network Load Balancer (NLB) IP targets and HTTP health checks on /hazelcast/health. Install the controller using AWS Load Balancer Controller documentation or your EKS add-on workflow. Configuration of the Hazelcast custom resource is described in Amazon EKS: Network Load Balancer with IP targets (recommended).
Azure Kubernetes Service (AKS)

If you use AKS, consider the following:

  • Make sure the Kubernetes cluster is public.

  • If you plan on using NodePorts, (discoveryServiceType: NodePort or memberAccess: NodePortExternalIP), the node pools should be created by using the --enable-node-public-ip flag. Additionally, the required inbound rules should be added for network security groups (NSGs) to allow access to node IP addresses and ports.

Configure exposeExternally

To allow connections from clients, you need to configure the exposeExternally field of the Hazelcast custom resource.

If using Istio, you cannot use exposeExternally.

This field has the following properties:

Property Description

type

The way in which Hazelcast members are exposed to clients:

  • Unisocket: Members are exposed through one Kubernetes service that automatically load balances the traffic to Hazelcast members. Use this property if your clients use the unisocket operation mode.

  • Smart: Each member is exposed through a separate Kubernetes service that allows clients to send requests directly to the members that own a particular partition. Use this property if your clients use the smart operation mode.

discoveryServiceType

Type of Kubernetes service used for discovering members in a Hazelcast cluster:

  • LoadBalancer (default): Hazelcast client configuration <load-balancer-external-ip>:5701.

    Your Kubernetes cluster must be able to assign public IP addresses to LoadBalancer services.

  • NodePort: Hazelcast client configuration <node-address>:<node-port>.

memberAccess

The way in which clients access Hazelcast members:

  • LoadBalancer: Hazelcast clients are outside the Kubernetes network and can access members, using the external IP that is assigned to the LoadBalancer service.

  • NodePortExternalIP (default): Hazelcast clients are outside the Kubernetes network, so a client can access members only via the public IP address that’s assigned to their Kubernetes nodes.

  • NodePortNodeName: Hazelcast clients are inside the same network as the Kubernetes nodes, so clients can access a member by the hostname of its Kubernetes node.
If NodePorts are used (discoveryServiceType: NodePort or memberAccess: NodePortExternalIP), the Kubernetes nodes must allow external TCP traffic on node IP addresses and ports.

Expose Hazelcast to smart clients

Smart clients access partitioned data in a cluster by communicating with the partition owner. This operation mode increases the overall throughput and efficiency of client-cluster communication.

For non-partitioned data, smart clients can communicate with any cluster member.

To support smart clients, configure the type property as smart.

In this configuration, the standard Kubernetes mechanism routes client traffic to the correct member. Or, if the client is not accessing partitioned data, the traffic is automatically load balanced.

Amazon EKS: Network Load Balancer with IP targets (recommended)

On Amazon EKS, when you use Smart clients with discoveryServiceType: LoadBalancer and memberAccess: LoadBalancer, the default cloud controller may provision a Classic Load Balancer that performs TCP health checks on the member port. That can produce noisy member logs (for example connectionType=NONE) and does not use HTTP health checks on the Hazelcast REST endpoint.

Recommended: Install the AWS Load Balancer Controller in your cluster (see AWS Load Balancer Controller documentation or the EKS add-ons console). Then add annotations under spec.annotations on the Hazelcast custom resource as described below. The operator copies spec.annotations to every Service it creates for that cluster (the discovery service and each per-member service when using Smart + LoadBalancer).

With NLB IP targets, health checks reach each pod on port 8081 (REST) directly. You do not need to expose REST as a NodePort on nodes for load balancer health checks.

Prerequisites:

  • EKS cluster with AWS Load Balancer Controller installed and able to reconcile LoadBalancer Services.

  • Hazelcast Enterprise and exposeExternally configured for Smart + LoadBalancer as shown below.

Add the following annotations under spec.annotations on the Hazelcast custom resource:

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast
spec:
  clusterSize: 3
  repository: "docker.io/hazelcast/hazelcast-enterprise"
  version: <your-version>
  licenseKeySecretName: hazelcast-license-key
  exposeExternally:
    type: Smart
    discoveryServiceType: LoadBalancer
    memberAccess: LoadBalancer
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: HTTP
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "8081"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: /hazelcast/health
Annotation Purpose

service.beta.kubernetes.io/aws-load-balancer-type: external

Defers provisioning to the AWS Load Balancer Controller instead of the in-tree AWS cloud controller.

service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip

Registers pod IPs as NLB targets (not EC2 instance / NodePort targets).

service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: HTTP

Uses HTTP for the load balancer health check.

service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "8081"

Health checks use the Hazelcast REST port on the pod.

service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: /hazelcast/health

Standard Hazelcast health endpoint.
If the AWS Load Balancer Controller is not installed, Services using aws-load-balancer-type: external may not receive an external hostname or IP. Use annotation sets supported by your cluster’s cloud integration only.
Some clusters use the GA prefix service.kubernetes.io/ instead of service.beta.kubernetes.io/ for the same keys; follow your Kubernetes and AWS documentation for the correct prefix.

You can combine these annotations with other Service annotations supported by AWS (subnets, internal scheme, cross-zone load balancing, and so on) in the same spec.annotations map.

Expose Hazelcast to unisocket clients

Unisocket clients communicate with a single member in the cluster. To enable these clients to connect to the Hazelcast cluster, configure the type property as unisocket.

In this configuration, the standard Kubernetes mechanism automatically load balances client traffic across the Hazelcast cluster.

Example configuration

In this example, the Hazelcast cluster is configured to load balance connections from smart clients.

Example configuration
apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast-sample
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  version: '5.6.0-slim'
  licenseKeySecretName: hazelcast-license-key
  exposeExternally:
    type: Smart
    discoveryServiceType: LoadBalancer
    memberAccess: NodePortExternalIP

This configuration creates a load balancer for the discovery service and one service for each member.

kubectl get service
NAME                 TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)          AGE
hazelcast-sample     LoadBalancer   10.219.246.19    192.0.2.0	     5701:30560/TCP   2m11s (1)
hazelcast-sample-0   NodePort       10.219.254.192   <none>          5701:31890/TCP   2m11s (2)
hazelcast-sample-1   NodePort       10.219.247.43    <none>          5701:32310/TCP   2m11s (2)
hazelcast-sample-2   NodePort       10.219.243.16    <none>          5701:32585/TCP   2m11s (2)
1 Discovery service
2 Member access service

The command below allows you to list the external addresses of the cluster.

kubectl get hazelcastendpoint --selector="app.kubernetes.io/instance=hazelcast-sample"
NAME                   TYPE        ADDRESS
hazelcast-sample       Discovery   198.51.100.0:5701
hazelcast-sample-0     Member      203.0.113.18:30108
hazelcast-sample-1     Member      203.0.113.200:32012
hazelcast-sample-2     Member      203.0.113.209:30512
hazelcast-sample-wan   WAN         198.51.100.0:5710

Smart clients can use the external IP of the load balancer to connect to the cluster. For example:

  • Java

  • Node.js

  • Go

  • Python

ClientConfig config = new ClientConfig();
config.getNetworkConfig().addAddress("192.0.2.0");
config.getProperties().setProperty(ClientProperty.DISCOVERY_SPI_PUBLIC_IP_ENABLED.toString(), "true");
HazelcastInstance client = HazelcastClient.newHazelcastClient(config);
const { Client } = require('hazelcast-client');

const clientConfig = {
  network: {
    clusterMembers: [
      '192.0.2.0'
    ]
  },
  properties: {
    ['hazelcast.discovery.public.ip.enabled']: true
  }
};
const client = await Client.newHazelcastClient(clientConfig);
import (
	"context"

	"github.com/hazelcast/hazelcast-go-client"
)

func main() {
	config := hazelcast.Config{}
	cc := &config.Cluster
	cc.Network.SetAddresses("192.0.2.0")
	cc.Discovery.UsePublicIP = true
	ctx := context.TODO()
	client, err := hazelcast.StartNewClientWithConfig(ctx, config)
	if err != nil {
		panic(err)
	}
}
import logging
import hazelcast

logging.basicConfig(level=logging.INFO)

client = hazelcast.HazelcastClient(
    cluster_members=["192.0.2.0"],
    use_public_ip=True,
)

Help and support