Restore a Cluster from Cloud Storage with Hazelcast Platform Operator

Learn how to back up data in Hazelcast maps to cloud storage and restore a cluster from that backup data.

Context

In this tutorial, you’ll do the following:

  • Deploy Hazelcast with persistence enabled.

  • Create a Hazelcast map that has persistence enabled.

  • Back up all map entries to external storage in the cloud.

  • Restart the Hazelcast cluster and restore the backup map entries from the cloud.

Before you Begin

Before starting this tutorial, make sure that you meet the following prerequisites:

Step 1. Start the Hazelcast Cluster

  1. Create a license secret

    Create a secret with your Hazelcast Enterprise License.

    kubectl create secret generic hazelcast-license-key --from-literal=license-key=<hz-license-key>
  2. Create the Hazelcast Cluster

    Run the following command to create the Hazelcast cluster with Persistence enabled.

    kubectl apply -f - <<EOF
    apiVersion: hazelcast.com/v1alpha1
    kind: Hazelcast
    metadata:
      name: my-hazelcast
    spec:
      clusterSize: 3
      licenseKeySecretName: hazelcast-license-key
      persistence:
        pvc:
          accessModes: ["ReadWriteOnce"]
          requestStorage: 8Gi
      exposeExternally:
        type: Unisocket
        discoveryServiceType: LoadBalancer
    EOF
  3. Check the Cluster Status

    Run the following commands to see the cluster status

    $ kubectl get hazelcast my-hazelcast
    NAME           STATUS    MEMBERS
    my-hazelcast   Running   3/3
    $ kubectl get pods -l app.kubernetes.io/instance=my-hazelcast
    NAME             READY   STATUS    RESTARTS   AGE
    my-hazelcast-0   2/2     Running   0          3m43s
    my-hazelcast-1   2/2     Running   0          3m16s
    my-hazelcast-2   2/2     Running   0          2m50s

    As you can see from the pod states, the agent container will be deployed with the Hazelcast container in the same Pod. The agent is responsible for backing data up into the external storage.

  4. Get the Address of the Hazelcast Cluster

    After verifying that the cluster is Running and all the members are ready, run the following command to find the discovery address.

    $ kubectl get hazelcastendpoint my-hazelcast
    NAME               TYPE        ADDRESS
    my-hazelcast       Discovery   34.30.60.128:5701

    The ADDRESS column displays the external address of your Hazelcast cluster.

Step 2. Create Persistent Map and Put Data

  1. Create Persistent Map

    Run the following command to create the Map resource with Persistence enabled.

    kubectl apply -f - <<EOF
    apiVersion: hazelcast.com/v1alpha1
    kind: Map
    metadata:
      name: persistent-map
    spec:
      hazelcastResourceName: my-hazelcast
      persistenceEnabled: true
    EOF
  2. Check the Status of the Backup

    Run the following command to check the status of the map configuration.

    kubectl get map persistent-map

    The status of the map is displayed in the output.

    NAME             STATUS
    persistent-map   Success
  3. Configure the Hazelcast client to connect to the cluster.

    To access all sample clients, clone the following repository:

    git clone https://github.com/hazelcast-guides/hazelcast-platform-operator-external-backup-restore.git
    cd hazelcast-platform-operator-external-backup-restore

    The sample code(excluding CLC) for this tutorial is in the docs/modules/ROOT/examples/operator-external-backup directory.

    • CLC

    • Java

    • NodeJS

    • Go

    • Python

    • .NET

    Before using CLC, it should be installed in your system. Check the installation instructions for CLC: Installing the Hazelcast CLC.

    Run the following command for adding the cluster config to the CLC.

    clc config add hz cluster.name=dev cluster.address=<EXTERNAL-IP>
    package com.hazelcast;
    
    import com.hazelcast.client.HazelcastClient;
    import com.hazelcast.client.config.ClientConfig;
    import com.hazelcast.client.config.ClientNetworkConfig;
    import com.hazelcast.client.impl.connection.tcp.RoutingMode;
    import com.hazelcast.core.HazelcastInstance;
    import com.hazelcast.map.IMap;
    
    import java.util.Random;
    
    public class Main {
        public static void main(String[] args) throws Exception {
            if(args.length == 0) {
                System.out.println("You should pass an argument to run: fill or size");
            } else if (!((args[0].equals("fill") || args[0].equals("size")))) {
                System.out.println("Wrong argument, you should pass: fill or size");
            } else{
                ClientConfig clientConfig = new ClientConfig();
                ClientNetworkConfig networkConfig = clientConfig.getNetworkConfig();
                networkConfig.addAddress("<EXTERNAL-IP>");
                networkConfig.getClusterRoutingConfig().setRoutingMode(RoutingMode.SINGLE_MEMBER);
                HazelcastInstance client = HazelcastClient.newHazelcastClient(clientConfig);
                System.out.println("Successful connection!");
    
                IMap<String, String> map = client.getMap("persistent-map");
    
                if (args[0].equals("fill")) {
                    System.out.println("Starting to fill the map with random entries.");
    
                    Random random = new Random();
                    while (true) {
                        int randomKey = random.nextInt(100_000);
                        map.put("key-" + randomKey, "value-" + randomKey);
                        System.out.println("Current map size: " + map.size());
                    }
                } else {
                    System.out.println("Current map size: " + map.size());
                    client.shutdown();
                }
            }
    
        }
    }
    'use strict';
    
    const { Client } = require('hazelcast-client');
    
    const clientConfig = {
        network: {
            clusterMembers: [
                '<EXTERNAL-IP>'
            ],
            smartRouting: false
        }
    };
    
    (async () => {
        try {
            if (process.argv.length === 2) {
                console.error('You should pass an argument to run: fill or size');
            } else if (!(process.argv[2] === 'fill' || process.argv[2] === 'size')) {
                console.error('Wrong argument, you should pass: fill or size');
            } else {
                const client = await Client.newHazelcastClient(clientConfig);
                const map = await client.getMap('persistent-map');
                await map.put('key', 'value');
                const res = await map.get('key');
                if (res !== 'value') {
                    throw new Error('Connection failed, check your configuration.');
                }
                console.log('Successful connection!');
                if (process.argv[2] === 'fill'){
                    console.log('Starting to fill the map with random entries.');
                    while (true) {
                        const randomKey = Math.floor(Math.random() * 100000);
                        await map.put('key' + randomKey, 'value' + randomKey);
                        const size = await map.size();
                        console.log(`Current map size: ${size}`);
                    }
                } else {
                    const size = await map.size();
                    console.log(`Current map size: ${size}`);
                }
            }
        } catch (err) {
            console.error('Error occurred:', err);
        }
    })();
    package main
    
    import (
    	"context"
    	"fmt"
    	"math/rand"
    	"os"
    
    	"github.com/hazelcast/hazelcast-go-client"
    )
    
    func main() {
    	if len(os.Args) != 2 {
    		fmt.Println("You should pass an argument to run: fill or size")
    		return
    	}
    	if !(os.Args[1] == "fill" || os.Args[1] == "size") {
    		fmt.Println("Wrong argument, you should pass: fill or size")
    		return
    	}
    
    	config := hazelcast.Config{}
    	cc := &config.Cluster
    	cc.Network.SetAddresses("<EXTERNAL-IP>:5701")
    	cc.Unisocket = true
    	ctx := context.TODO()
    	client, err := hazelcast.StartNewClientWithConfig(ctx, config)
    	if err != nil {
    		panic(err)
    	}
    	fmt.Println("Successful connection!")
    	m, err := client.GetMap(ctx, "persistent-map")
    	if err != nil {
    		panic(err)
    	}
    	if os.Args[1] == "fill" {
    		fmt.Println("Starting to fill the map with random entries.")
    		for {
    			num := rand.Intn(100_000)
    			key := fmt.Sprintf("key-%d", num)
    			value := fmt.Sprintf("value-%d", num)
    			if _, err = m.Put(ctx, key, value); err != nil {
    				fmt.Println("ERR:", err.Error())
    				continue
    			}
    			mapSize, err := m.Size(ctx)
    			if err != nil {
    				fmt.Println("ERR:", err.Error())
    				continue
    			}
    			fmt.Println("Current map size:", mapSize)
    		}
    		return
    	}
    	mapSize, err := m.Size(ctx)
    	if err != nil {
    		fmt.Println("ERR:", err.Error())
    		return
    	}
    	fmt.Println("Current map size:", mapSize)
    
    }
    import logging
    import random
    import sys
    
    import hazelcast
    
    logging.basicConfig(level=logging.INFO)
    
    if len(sys.argv) != 2:
        print("You should pass an argument to run: fill or size")
    elif not (sys.argv[1] == "fill" or sys.argv[1] == "size"):
        print("Wrong argument, you should pass: fill or size")
    else:
        client = hazelcast.HazelcastClient(
            cluster_members=["<EXTERNAL-IP>"],
            smart_routing=False,
        )
        print("Successful connection!", flush=True)
        m = client.get_map("persistent-map").blocking()
    
        if sys.argv[1] == "fill":
            print("Starting to fill the map with random entries.", flush=True)
            while True:
                random_number = str(random.randrange(0, 100000))
                m.put("key-" + random_number, "value-" + random_number)
                print("Current map size:", m.size())
        else:
            print("Current map size:", m.size())
    using System;
    using System.Threading.Tasks;
    using Hazelcast;
    using Microsoft.Extensions.Logging;
    
    namespace Client
    {
        public class Program
        {
            static async Task Main(string[] args)
            {
                if (args.Length != 1)
                {
                    Console.WriteLine("You should pass an argument to run: fill or size");
                    return;
                }
                if (!(args[0] == "fill" || args[0] == "size"))
                {
                    Console.WriteLine("Wrong argument, you should pass: fill or size");
                    return;
                }
                var options = new HazelcastOptionsBuilder()                
                    .With(args)                
                    .With((configuration, options) =>
                    {
                        options.LoggerFactory.Creator = () => LoggerFactory.Create(loggingBuilder =>
                            loggingBuilder
                                .AddConfiguration(configuration.GetSection("logging"))
                                .AddConsole());
    
                        options.Networking.Addresses.Add("<EXTERNAL-IP>:5701");
                        options.Networking.SmartRouting = false;
                        
                    })
                    .Build();
    
    
                await using var client = await HazelcastClientFactory.StartNewClientAsync(options);
    
                Console.WriteLine("Successful connection!");
                Console.WriteLine("Starting to fill the map with random entries.");
    
                var map = await client.GetMapAsync<string, string>("persistent-map");
                var random = new Random();
    
                if (args[0] == "fill")
                {
                    Console.WriteLine("Starting to fill the map with random entries.");
                    while (true)
                    {
                        var num = random.Next(100_000);
                        var key = $"key-{num}";
                        var value = $"value-{num}";
                        await map.PutAsync(key, value);
                        var mapSize = await map.GetSizeAsync();
                        Console.WriteLine($"Current map size: {mapSize}");
                    }
                }
                else
                {
                    var mapSize = await map.GetSizeAsync();
                    Console.WriteLine($"Current map size: {mapSize}");
                    await client.DisposeAsync();
                }
            }
        } 
    }
  4. Start the client to fill the map.

    • CLC

    • Java

    • NodeJS

    • Go

    • Python

    • .NET

    Run the following command to fill a map.

    for i in {1..10};
    do
       clc -c hz map set --name persistent-map key-$i value-$i;
    done

    Run the following command to check the map size.

    clc -c hz map size --name persistent-map
    cd java
    mvn package
    java -jar target/*jar-with-dependencies*.jar fill

    You should see the following output.

    Successful connection!
    Starting to fill the map with random entries.
    Current map size: 2
    Current map size: 3
    Current map size: 4
    ....
    ....
    cd nodejs
    npm install
    npm start fill

    You should see the following output.

    Successful connection!
    Starting to fill the map with random entries.
    Current map size: 2
    Current map size: 3
    Current map size: 4
    ....
    ....
    cd go
    go run main.go fill

    You should see the following output.

    Successful connection!
    Starting to fill the map with random entries.
    Current map size: 2
    Current map size: 3
    Current map size: 4
    ....
    ....
    cd python
    pip install -r requirements.txt
    python main.py fill

    You should see the following output.

    Successful connection!
    Starting to fill the map with random entries.
    Current map size: 2
    Current map size: 3
    Current map size: 4
    ....
    ....
    cd dotnet
    dotnet build
    dotnet run fill
    Successful connection!
    Starting to fill the map with random entries.
    Current map size: 2
    Current map size: 3
    Current map size: 4
    ....
    ....

Step 3. Trigger External Backup

For triggering backup, you need bucketURI where backup data will be stored in and secret with credentials for accessing the given Bucket URI.

  1. Create Secret

    Run one of the following command to create the secret according to the cloud provider you want to backup.

    • AWS

    • GCP

    • Azure

    kubectl create secret generic <external-bucket-secret-name> --from-literal=region=<region> \
    	--from-literal=access-key-id=<access-key-id> \
    	--from-literal=secret-access-key=<secret-access-key>
    kubectl create secret generic <external-bucket-secret-name> --from-file=google-credentials-path=<service_account_json_file>
    kubectl create secret generic <external-bucket-secret-name> \
    	--from-literal=storage-account=<storage-account> \
    	--from-literal=storage-key=<storage-key>
  2. Trigger Backup

    Run the following command to trigger backup

    kubectl apply -f - <<EOF
    apiVersion: hazelcast.com/v1alpha1
    kind: HotBackup
    metadata:
      name: hot-backup
    spec:
      hazelcastResourceName: my-hazelcast
      bucketURI: "s3://operator-backup"
      secret: "br-secret-s3"
    EOF

    Example URI → "s3://operator-backup?prefix=hazelcast/2022-06-08-17-01-20/"

  3. Check the Status of the Backup

    Run the following command to check the status of the backup

    kubectl get hotbackup hot-backup

    The status of the backup is displayed in the output.

    NAME         STATUS
    hot-backup   Success

Step 4. Restore from External Backup

  1. Delete the Hazelcast Cluster

    Run the following command to delete the Hazelcast cluster

    kubectl delete hazelcast my-hazelcast
  2. Delete the PersistentVolumeClaims

    The data will be restored from external buckets, so you do not keep persistent volumes during restore operation.

    Run the following command to list PVCs which were created for Hazelcast cluster

    $ kubectl get pvc -l "app.kubernetes.io/managed-by=hazelcast-platform-operator"
    NAME                         STATUS   VOLUME                                     CAPACITY
    persistence-my-hazelcast-0   Bound    pvc-a77fcd6e-e64d-4aeb-9708-835d66b4c5b9   8Gi
    persistence-my-hazelcast-1   Bound    pvc-8548a4a5-fa8e-45bd-9909-8e6bb6a6c3e8   8Gi
    persistence-my-hazelcast-2   Bound    pvc-c94edb25-5a72-482a-bf4e-84a99319d509   8Gi

    Run the following command to delete the PVCs

    kubectl delete pvc -l "app.kubernetes.io/managed-by=hazelcast-platform-operator"
  3. Create new Hazelcast Cluster

    For restoring you will use the HotBackup resource you have created.

    Run the following command to create the Hazelcast cluster. Before the Hazelcast cluster is started, the operator starts the agent(as InitContainer) which restores the backup data from the external bucket.

    kubectl apply -f - <<EOF
    apiVersion: hazelcast.com/v1alpha1
    kind: Hazelcast
    metadata:
      name: my-hazelcast
    spec:
      clusterSize: 3
      licenseKeySecretName: hazelcast-license-key
      persistence:
        pvc:
          accessModes: ["ReadWriteOnce"]
          requestStorage: 8Gi
        restore:
          hotBackupResourceName: hot-backup
      exposeExternally:
        type: Unisocket
        discoveryServiceType: LoadBalancer
    EOF

    As you may see, the agent configuration is not set. Thus, the operator directly use the latest stable version of the agent.

  4. Check the Cluster Status

    Run the following commands to see the cluster status

    $ kubectl get hazelcast my-hazelcast
    NAME           STATUS    MEMBERS
    my-hazelcast   Running   3/3

    After verifying that the cluster is Running and all the members are ready, run the following command to find the discovery address.

    $ kubectl get hazelcastendpoint my-hazelcast
    NAME           TYPE        ADDRESS
    my-hazelcast   Discovery   34.28.159.226:5701

    Since we recreate the Hazelcast cluster, services are also recreated. The ADDRESS may change.

  5. Check the Map Size

    Configure the Hazelcast client to connect to the cluster external address as you did in Configure the Hazelcast Client.

    Start the client to check the map size and see if the restore is successful.

    • CLC

    • Java

    • NodeJS

    • Go

    • Python

    • .NET

    clc -c hz map size --name persistent-map
    cd java
    mvn package
    java -jar target/*jar-with-dependencies*.jar size

    You should see the following output.

    Successful connection!
    Current map size: 12
    cd nodejs
    npm install
    npm start size

    You should see the following output.

    Successful connection!
    Current map size: 12
    cd go
    go run main.go size

    You should see the following output.

    Successful connection!
    Current map size: 12
    cd python
    pip install -r requirements.txt
    python main.py size

    You should see the following output.

    Successful connection!
    Current map size: 12
    cd dotnet
    dotnet run size

    You should see the following output.

    Successful connection!
    Current map size: 12

Clean Up

To clean up the created resources remove the all Custom Resources and PVCs.

kubectl delete secret <external-bucket-secret-name>
kubectl delete secret hazelcast-license-key
kubectl delete $(kubectl get hazelcast,hotbackup -o name)
kubectl delete pvc -l "app.kubernetes.io/managed-by=hazelcast-platform-operator"