Hazelcast with SSL in Kubernetes

Enable SSL between Hazelcast members and Hazelcast clients in Kubernetes.

Before you Begin

This example focuses on the security features and assumes that you have some general knowledge about the following topics:

This tutorial also assumes you have a running Kubernetes cluster, and the kubectl tool installed.

1. Create Secrets

To secure the connection between Hazelcast Server and Hazelcast Client we need to make keys available as secrets.

kubectl create secret generic keystore --from-file=server/keystore --from-file=server/truststore

2. Start Hazelcast Cluster

Hazelcast SSL is an enterprise feature, so you need to create a secret with Hazelcast Enterprise License (if you don’t have one, get a trial license key from this link).

kubectl create secret generic hz-license-key --from-literal key=<hz-license-key>

Then, you can start Hazelcast Cluster with the following command.

  • Helm

  • Kubectl

helm repo add hazelcast https://hazelcast-charts.s3.amazonaws.com/
helm repo update
kubectl create configmap hazelcast-configuration --from-file=server/helm/hazelcast.yaml
helm install hz -f server/helm/values.yaml hazelcast/hazelcast-enterprise
kubectl apply -f https://raw.githubusercontent.com/hazelcast/hazelcast/master/kubernetes-rbac.yaml
kubectl create configmap hazelcast-configuration --from-file=server/kubectl/hazelcast.yaml
kubectl apply -f server/kubectl/deployment.yaml

You can check in the logs that Hazelcast cluster has been formed and that SSL is enabled for the communication.

$ kubectl logs pod/hz-hazelcast-enterprise-0
...
INFO: [10.172.2.29]:5701 [dev] [4.0] SSL is enabled
...
Members {size:3, ver:3} [
        Member [10.244.1.3]:5702 - 8eb71540-5be6-4a06-aa2f-78492816230b this
        Member [10.244.2.4]:5702 - 53d5a2a9-f1f5-4bea-b881-0ffe48f0eea0
        Member [10.244.1.4]:5702 - f8b42a13-6944-4e94-970a-030c69e75a5b
]

3. Starting Hazelcast Client

As a client side, you’ll deploy a Spring Boot application which connects to the Hazelcast cluster.

The application already includes the client configuration and truststore. All you need to do is to compile, package, create Docker image, and push it into your Docker Hub account. You may do it all with the following commands.

mvn clean package -f client/pom.xml
docker build -t hazelcastguides/hazelcast-client-ssl client
docker push hazelcastguides/hazelcast-client-ssl

If you want to build your own Docker image, then please make sure you change hazelcastguides to your Docker Hub name and update client/deployment.yaml accordingly. Otherwise, you can use hazelcastguides/hazelcast-client-ssl.

Finally, you can deploy the client application.

kubectl apply -f client/deployment.yaml

You can see that the client connected to the cluster using SSL by checking the logs.

$ kubectl logs deployment.apps/hazelcast-client
  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.2.6.RELEASE)
...
2020-11-11 13:01:14.576  INFO 1 --- [           main] c.hazelcast.client.impl.ClientExtension  : SSL is enabled
...
Members [3] {
        Member [10.244.1.3]:5701 - 8eb71540-5be6-4a06-aa2f-78492816230b
        Member [10.244.2.4]:5701 - 53d5a2a9-f1f5-4bea-b881-0ffe48f0eea0
        Member [10.244.1.4]:5701 - f8b42a13-6944-4e94-970a-030c69e75a5b
}
...
2020-11-11 13:01:17.226  INFO 1 --- [           main] com.hazelcast.kubernetes.Application     : Started Application in 8.705 seconds (JVM running for 10.03)

Then, assuming you have Load Balancer configured for your Kubernetes environment, you can check the public IP of your client application.

$ kubectl get all | grep service/hazelcast-client
service/hazelcast-client   LoadBalancer   10.175.254.124   35.241.133.251   8080:31220/TCP   10m

Now, to check that everything works fine by executing the following commands.

$ curl "35.241.133.251:8080/put?key=some-key&value=some-value"
{"response":null}

$ curl "35.241.133.251:8080/get?key=some-key"
{"response":"some-value"}

Note: In the example, we didn’t use Mutual Authentication, so Hazelcast client itself was not being authorized. For more information check Mutual Authentication section.

4. Clean up

To delete all created resources, exucte the following commands.

  • Helm

  • Kubectl

kubectl delete -f client/deployment.yaml
helm uninstall hz
kubectl delete configmap hazelcast-configuration
kubectl delete secret keystore
kubectl delete secret hz-license-key
kubectl delete -f client/deployment.yaml
kubectl delete -f server/kubectl/deployment.yaml
kubectl delete configmap hazelcast-configuration
kubectl delete -f https://raw.githubusercontent.com/hazelcast/hazelcast/master/kubernetes-rbac.yaml
kubectl delete secret keystore
kubectl delete secret hz-license-key

More Information

How KeyStore and TrustStore were Generated

KeyStore and TrustStore files for this example were generated using the following commands:

$ keytool -genkey -alias client -keyalg RSA -keystore keystore -keysize 2048 -storepass 123456
What is your first and last name?
  [Unknown]:  hazelcast-mancenter
What is the name of your organizational unit?
  [Unknown]:
What is the name of your organization?
  [Unknown]:
What is the name of your City or Locality?
  [Unknown]:
What is the name of your State or Province?
  [Unknown]:
What is the two-letter country code for this unit?
  [Unknown]:
Is CN=my-release-hazelcast-enterprise-mancenter, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown correct?
  [no]:  yes

$ keytool -export -alias client -file client.crt -keystore keystore -storepass 123456
Certificate stored in file <client.crt>

$ keytool -import -v -trustcacerts -alias client -file client.crt -keystore truststore -storepass 123456
Owner: CN=my-release-hazelcast-enterprise-mancenter, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown
Issuer: CN=my-release-hazelcast-enterprise-mancenter, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown
Serial number: 7c8af8f7
Valid from: Wed Nov 28 13:41:29 GMT 2018 until: Tue Feb 26 13:41:29 GMT 2019
Certificate fingerprints:
         SHA1: 0B:8B:B2:F2:BA:DA:4F:3E:88:90:A7:7E:47:4A:DE:18:BE:DD:7E:5D
         SHA256: A9:A4:EE:BB:1E:FB:A2:0F:18:D0:34:09:07:0A:63:AE:62:4E:F6:1B:A0:4F:E1:D2:6A:CD:EB:2B:91:D2:EE:29
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: F1 CC 48 90 06 75 D0 51   1D 75 D8 E0 16 DC 66 04  ..H..u.Q.u....f.
0010: FC 4D A3 9B                                        .M..
]
]

Trust this certificate? [no]:  yes
Certificate was added to keystore
[Storing truststore]

$ rm client.crt

We used hazelcast-mancenter as the hostname, which means that if you start Management Center, its service must be named hazelcast-mancenter (otherwise the hostname verification fails).

Mutual Authentication section

SSL Mutual Authentication can be enabled to increase the security. To enable it, you need to configure it in both Hazelcast Server and Hazelcast Client.

Currently, Mutual Authentication does not work with livenessProbe/readinessProbe enabled.

  • Hazelcast Server

Add the following line to the ssl properties section (in hazelcast.yaml):

mutualAuthentication: REQUIRED

  • Hazelcast Client

Add the following lines to the SSLConfig object in hazelcastConfig() (in the file Application.java):

.setProperty("keyStore", "keystore")
.setProperty("keyStorePassword", System.getEnv("KEYSTORE_PASSWORD"))

You also need to add keystore into resources by appending the following line to Dockerfile:

COPY src/main/resources/keystore keystore