Restore a Cluster from Cloud Storage with Hazelcast Platform Operator
Learn how to back up data in Hazelcast maps to cloud storage and restore a cluster from that backup data.
Context
In this tutorial, you’ll do the following:
-
Deploy Hazelcast with persistence enabled.
-
Create a Hazelcast map that has persistence enabled.
-
Back up all map entries to external storage in the cloud.
-
Restart the Hazelcast cluster and restore the backup map entries from the cloud.
Before you Begin
Before starting this tutorial, make sure that you meet the following prerequisites:
-
Up and running Kubernetes cluster
-
Kubernetes command-line tool, kubectl
-
Deployed Hazelcast Platform Operator
-
Created blob storage and access credentials in one of the cloud providers: AWS - GCP - Azure
Step 1. Start the Hazelcast Cluster
-
Create a license secret
Create a secret with your Hazelcast Enterprise License.
kubectl create secret generic hazelcast-license-key --from-literal=license-key=<hz-license-key>
-
Create the Hazelcast Cluster
Run the following command to create the Hazelcast cluster with Persistence enabled.
kubectl apply -f - <<EOF apiVersion: hazelcast.com/v1alpha1 kind: Hazelcast metadata: name: my-hazelcast spec: clusterSize: 3 licenseKeySecretName: hazelcast-license-key persistence: pvc: accessModes: ["ReadWriteOnce"] requestStorage: 8Gi exposeExternally: type: Unisocket discoveryServiceType: LoadBalancer EOF
-
Check the Cluster Status
Run the following commands to see the cluster status
$ kubectl get hazelcast my-hazelcast NAME STATUS MEMBERS my-hazelcast Running 3/3
$ kubectl get pods -l app.kubernetes.io/instance=my-hazelcast NAME READY STATUS RESTARTS AGE my-hazelcast-0 2/2 Running 0 3m43s my-hazelcast-1 2/2 Running 0 3m16s my-hazelcast-2 2/2 Running 0 2m50s
As you can see from the pod states, the agent container will be deployed with the Hazelcast container in the same Pod. The agent is responsible for backing data up into the external storage.
-
Get the Address of the Hazelcast Cluster
After verifying that the cluster is
Running
and all the members are ready, run the following command to find the discovery address.$ kubectl get hazelcastendpoint my-hazelcast NAME TYPE ADDRESS my-hazelcast Discovery 34.30.60.128:5701
The
ADDRESS
column displays the external address of your Hazelcast cluster.
Step 2. Create Persistent Map and Put Data
-
Create Persistent Map
Run the following command to create the Map resource with Persistence enabled.
kubectl apply -f - <<EOF apiVersion: hazelcast.com/v1alpha1 kind: Map metadata: name: persistent-map spec: hazelcastResourceName: my-hazelcast persistenceEnabled: true EOF
-
Check the Status of the Backup
Run the following command to check the status of the map configuration.
kubectl get map persistent-map
The status of the map is displayed in the output.
NAME STATUS persistent-map Success
-
Configure the Hazelcast client to connect to the cluster.
To access all sample clients, clone the following repository:
git clone https://github.com/hazelcast-guides/hazelcast-platform-operator-external-backup-restore.git cd hazelcast-platform-operator-external-backup-restore
The sample code(excluding CLC) for this tutorial is in the
docs/modules/ROOT/examples/operator-external-backup
directory.Before using CLC, it should be installed in your system. Check the installation instructions for CLC: Installing the Hazelcast CLC. Run the following command for adding the cluster config to the CLC.
clc config add hz cluster.name=dev cluster.address=<EXTERNAL-IP>
package com.hazelcast; import com.hazelcast.client.HazelcastClient; import com.hazelcast.client.config.ClientConfig; import com.hazelcast.client.config.ClientNetworkConfig; import com.hazelcast.client.impl.connection.tcp.RoutingMode; import com.hazelcast.core.HazelcastInstance; import com.hazelcast.map.IMap; import java.util.Random; public class Main { public static void main(String[] args) throws Exception { if(args.length == 0) { System.out.println("You should pass an argument to run: fill or size"); } else if (!((args[0].equals("fill") || args[0].equals("size")))) { System.out.println("Wrong argument, you should pass: fill or size"); } else{ ClientConfig clientConfig = new ClientConfig(); ClientNetworkConfig networkConfig = clientConfig.getNetworkConfig(); networkConfig.addAddress("<EXTERNAL-IP>"); networkConfig.getClusterRoutingConfig().setRoutingMode(RoutingMode.SINGLE_MEMBER); HazelcastInstance client = HazelcastClient.newHazelcastClient(clientConfig); System.out.println("Successful connection!"); IMap<String, String> map = client.getMap("persistent-map"); if (args[0].equals("fill")) { System.out.println("Starting to fill the map with random entries."); Random random = new Random(); while (true) { int randomKey = random.nextInt(100_000); map.put("key-" + randomKey, "value-" + randomKey); System.out.println("Current map size: " + map.size()); } } else { System.out.println("Current map size: " + map.size()); client.shutdown(); } } } }
'use strict'; const { Client } = require('hazelcast-client'); const clientConfig = { network: { clusterMembers: [ '<EXTERNAL-IP>' ], smartRouting: false } }; (async () => { try { if (process.argv.length === 2) { console.error('You should pass an argument to run: fill or size'); } else if (!(process.argv[2] === 'fill' || process.argv[2] === 'size')) { console.error('Wrong argument, you should pass: fill or size'); } else { const client = await Client.newHazelcastClient(clientConfig); const map = await client.getMap('persistent-map'); await map.put('key', 'value'); const res = await map.get('key'); if (res !== 'value') { throw new Error('Connection failed, check your configuration.'); } console.log('Successful connection!'); if (process.argv[2] === 'fill'){ console.log('Starting to fill the map with random entries.'); while (true) { const randomKey = Math.floor(Math.random() * 100000); await map.put('key' + randomKey, 'value' + randomKey); const size = await map.size(); console.log(`Current map size: ${size}`); } } else { const size = await map.size(); console.log(`Current map size: ${size}`); } } } catch (err) { console.error('Error occurred:', err); } })();
package main import ( "context" "fmt" "math/rand" "os" "github.com/hazelcast/hazelcast-go-client" ) func main() { if len(os.Args) != 2 { fmt.Println("You should pass an argument to run: fill or size") return } if !(os.Args[1] == "fill" || os.Args[1] == "size") { fmt.Println("Wrong argument, you should pass: fill or size") return } config := hazelcast.Config{} cc := &config.Cluster cc.Network.SetAddresses("<EXTERNAL-IP>:5701") cc.Unisocket = true ctx := context.TODO() client, err := hazelcast.StartNewClientWithConfig(ctx, config) if err != nil { panic(err) } fmt.Println("Successful connection!") m, err := client.GetMap(ctx, "persistent-map") if err != nil { panic(err) } if os.Args[1] == "fill" { fmt.Println("Starting to fill the map with random entries.") for { num := rand.Intn(100_000) key := fmt.Sprintf("key-%d", num) value := fmt.Sprintf("value-%d", num) if _, err = m.Put(ctx, key, value); err != nil { fmt.Println("ERR:", err.Error()) continue } mapSize, err := m.Size(ctx) if err != nil { fmt.Println("ERR:", err.Error()) continue } fmt.Println("Current map size:", mapSize) } return } mapSize, err := m.Size(ctx) if err != nil { fmt.Println("ERR:", err.Error()) return } fmt.Println("Current map size:", mapSize) }
import logging import random import sys import hazelcast logging.basicConfig(level=logging.INFO) if len(sys.argv) != 2: print("You should pass an argument to run: fill or size") elif not (sys.argv[1] == "fill" or sys.argv[1] == "size"): print("Wrong argument, you should pass: fill or size") else: client = hazelcast.HazelcastClient( cluster_members=["<EXTERNAL-IP>"], smart_routing=False, ) print("Successful connection!", flush=True) m = client.get_map("persistent-map").blocking() if sys.argv[1] == "fill": print("Starting to fill the map with random entries.", flush=True) while True: random_number = str(random.randrange(0, 100000)) m.put("key-" + random_number, "value-" + random_number) print("Current map size:", m.size()) else: print("Current map size:", m.size())
using System; using System.Threading.Tasks; using Hazelcast; using Microsoft.Extensions.Logging; namespace Client { public class Program { static async Task Main(string[] args) { if (args.Length != 1) { Console.WriteLine("You should pass an argument to run: fill or size"); return; } if (!(args[0] == "fill" || args[0] == "size")) { Console.WriteLine("Wrong argument, you should pass: fill or size"); return; } var options = new HazelcastOptionsBuilder() .With(args) .With((configuration, options) => { options.LoggerFactory.Creator = () => LoggerFactory.Create(loggingBuilder => loggingBuilder .AddConfiguration(configuration.GetSection("logging")) .AddConsole()); options.Networking.Addresses.Add("<EXTERNAL-IP>:5701"); options.Networking.SmartRouting = false; }) .Build(); await using var client = await HazelcastClientFactory.StartNewClientAsync(options); Console.WriteLine("Successful connection!"); Console.WriteLine("Starting to fill the map with random entries."); var map = await client.GetMapAsync<string, string>("persistent-map"); var random = new Random(); if (args[0] == "fill") { Console.WriteLine("Starting to fill the map with random entries."); while (true) { var num = random.Next(100_000); var key = $"key-{num}"; var value = $"value-{num}"; await map.PutAsync(key, value); var mapSize = await map.GetSizeAsync(); Console.WriteLine($"Current map size: {mapSize}"); } } else { var mapSize = await map.GetSizeAsync(); Console.WriteLine($"Current map size: {mapSize}"); await client.DisposeAsync(); } } } }
-
Start the client to fill the map.
Run the following command to fill a map.
for i in {1..10}; do clc -c hz map set --name persistent-map key-$i value-$i; done
Run the following command to check the map size.
clc -c hz map size --name persistent-map
cd java mvn package java -jar target/*jar-with-dependencies*.jar fill
You should see the following output.
Successful connection! Starting to fill the map with random entries. Current map size: 2 Current map size: 3 Current map size: 4 .... ....
cd nodejs npm install npm start fill
You should see the following output.
Successful connection! Starting to fill the map with random entries. Current map size: 2 Current map size: 3 Current map size: 4 .... ....
cd go go run main.go fill
You should see the following output.
Successful connection! Starting to fill the map with random entries. Current map size: 2 Current map size: 3 Current map size: 4 .... ....
cd python pip install -r requirements.txt python main.py fill
You should see the following output.
Successful connection! Starting to fill the map with random entries. Current map size: 2 Current map size: 3 Current map size: 4 .... ....
cd dotnet dotnet build dotnet run fill
Successful connection! Starting to fill the map with random entries. Current map size: 2 Current map size: 3 Current map size: 4 .... ....
Step 3. Trigger External Backup
For triggering backup, you need bucketURI
where backup data will be stored in and secret
with credentials for accessing the given Bucket URI.
-
Create Secret
Run one of the following command to create the secret according to the cloud provider you want to backup.
kubectl create secret generic <external-bucket-secret-name> --from-literal=region=<region> \ --from-literal=access-key-id=<access-key-id> \ --from-literal=secret-access-key=<secret-access-key>
kubectl create secret generic <external-bucket-secret-name> --from-file=google-credentials-path=<service_account_json_file>
kubectl create secret generic <external-bucket-secret-name> \ --from-literal=storage-account=<storage-account> \ --from-literal=storage-key=<storage-key>
-
Trigger Backup
Run the following command to trigger backup
kubectl apply -f - <<EOF apiVersion: hazelcast.com/v1alpha1 kind: HotBackup metadata: name: hot-backup spec: hazelcastResourceName: my-hazelcast bucketURI: "s3://operator-backup" secret: "br-secret-s3" EOF
Example URI → "s3://operator-backup?prefix=hazelcast/2022-06-08-17-01-20/"
-
Check the Status of the Backup
Run the following command to check the status of the backup
kubectl get hotbackup hot-backup
The status of the backup is displayed in the output.
NAME STATUS hot-backup Success
Step 4. Restore from External Backup
-
Delete the Hazelcast Cluster
Run the following command to delete the Hazelcast cluster
kubectl delete hazelcast my-hazelcast
-
Delete the PersistentVolumeClaims
The data will be restored from external buckets, so you do not keep persistent volumes during restore operation.
Run the following command to list PVCs which were created for Hazelcast cluster
$ kubectl get pvc -l "app.kubernetes.io/managed-by=hazelcast-platform-operator" NAME STATUS VOLUME CAPACITY persistence-my-hazelcast-0 Bound pvc-a77fcd6e-e64d-4aeb-9708-835d66b4c5b9 8Gi persistence-my-hazelcast-1 Bound pvc-8548a4a5-fa8e-45bd-9909-8e6bb6a6c3e8 8Gi persistence-my-hazelcast-2 Bound pvc-c94edb25-5a72-482a-bf4e-84a99319d509 8Gi
Run the following command to delete the PVCs
kubectl delete pvc -l "app.kubernetes.io/managed-by=hazelcast-platform-operator"
-
Create new Hazelcast Cluster
For restoring you will use the
HotBackup
resource you have created.Run the following command to create the Hazelcast cluster. Before the Hazelcast cluster is started, the operator starts the agent(as InitContainer) which restores the backup data from the external bucket.
kubectl apply -f - <<EOF apiVersion: hazelcast.com/v1alpha1 kind: Hazelcast metadata: name: my-hazelcast spec: clusterSize: 3 licenseKeySecretName: hazelcast-license-key persistence: pvc: accessModes: ["ReadWriteOnce"] requestStorage: 8Gi restore: hotBackupResourceName: hot-backup exposeExternally: type: Unisocket discoveryServiceType: LoadBalancer EOF
As you may see, the agent configuration is not set. Thus, the operator directly use the latest stable version of the agent.
-
Check the Cluster Status
Run the following commands to see the cluster status
$ kubectl get hazelcast my-hazelcast NAME STATUS MEMBERS my-hazelcast Running 3/3
After verifying that the cluster is
Running
and all the members are ready, run the following command to find the discovery address.$ kubectl get hazelcastendpoint my-hazelcast NAME TYPE ADDRESS my-hazelcast Discovery 34.28.159.226:5701
Since we recreate the Hazelcast cluster, services are also recreated. The
ADDRESS
may change. -
Check the Map Size
Configure the Hazelcast client to connect to the cluster external address as you did in Configure the Hazelcast Client.
Start the client to check the map size and see if the restore is successful.
clc -c hz map size --name persistent-map
cd java mvn package java -jar target/*jar-with-dependencies*.jar size
You should see the following output.
Successful connection! Current map size: 12
cd nodejs npm install npm start size
You should see the following output.
Successful connection! Current map size: 12
cd go go run main.go size
You should see the following output.
Successful connection! Current map size: 12
cd python pip install -r requirements.txt python main.py size
You should see the following output.
Successful connection! Current map size: 12
cd dotnet dotnet run size
You should see the following output.
Successful connection! Current map size: 12
Clean Up
To clean up the created resources remove the all Custom Resources and PVCs.
kubectl delete secret <external-bucket-secret-name>
kubectl delete secret hazelcast-license-key
kubectl delete $(kubectl get hazelcast,hotbackup -o name)
kubectl delete pvc -l "app.kubernetes.io/managed-by=hazelcast-platform-operator"