Tutorial: Restore a cluster from cloud storage

Learn how to back up data in Hazelcast maps to cloud storage and restore a cluster from that backup data.

Overview

In this tutorial, you'll do the following:

• Deploy Hazelcast with persistence enabled.
• Create a Hazelcast map that has persistence enabled.
• Back up all map entries to external storage in the cloud.
• Restart the Hazelcast cluster and restore the backup map entries from the cloud.

The tutorial should take approximately 20 minutes to complete.

Prerequisites

Before you begin, make sure that you have:

• An up and running Kubernetes cluster.

• Kubernetes command line tool, kubectl.

• Deployed Hazelcast Platform Operator.

• Created blob storage and access credentials in one of the Cloud providers:

  • AWS
  • GCP
  • Azure

• The Hazelcast Code Samples repository https://github.com/hazelcast/hazelcast-code-samples cloned to your local machine.

Start the Hazelcast cluster

1. Create a secret with your Hazelcast Enterprise License.

   shell

   kubectl create secret generic hazelcast-license-key --from-literal=license-key=<hz-license-key>

2. Create a Hazelcast cluster with Persistence enabled:

   shell

   kubectl apply -f - <<EOF
   apiVersion: hazelcast.com/v1alpha1
   kind: Hazelcast
   metadata:
     name: my-hazelcast
   spec:
     clusterSize: 3
     licenseKeySecretName: hazelcast-license-key
     persistence:
       pvc:
         accessModes: ["ReadWriteOnce"]
         requestStorage: 8Gi
     exposeExternally:
       type: Unisocket
       discoveryServiceType: LoadBalancer
   EOF

3. Check the cluster status:

   shell

   kubectl get hazelcast my-hazelcast
   NAME           STATUS    MEMBERS
   my-hazelcast   Running   3/3

   shell

   kubectl get pods -l app.kubernetes.io/instance=my-hazelcast
   NAME             READY   STATUS    RESTARTS   AGE
   my-hazelcast-0   2/2     Running   0          3m43s
   my-hazelcast-1   2/2     Running   0          3m16s
   my-hazelcast-2   2/2     Running   0          2m50s

   As you can see from the pod states, the agent container will be deployed with the Hazelcast container in the same Pod. The agent is responsible for backing data up into the external storage.

4. After verifying that the cluster is Running and all the members are ready, find the discovery address:

   shell

   kubectl get hazelcastendpoint my-hazelcast
   NAME               TYPE        ADDRESS
   my-hazelcast       Discovery   198.51.100.128:5701

   The ADDRESS column displays the external address of your Hazelcast cluster.

Create persistent map and put data

1. Create a Map resource with persistence enabled:

   shell

   kubectl apply -f - <<EOF
   apiVersion: hazelcast.com/v1alpha1
   kind: Map
   metadata:
     name: persistent-map
   spec:
     hazelcastResourceName: my-hazelcast
     persistenceEnabled: true
   EOF

2. Check the status of the map configuration:

   bash

   kubectl get map persistent-map

   The status of the map is displayed in the output.

   bash

   NAME             STATUS
   persistent-map   Success

3. Configure the Hazelcast client to connect to the cluster.

   Open the examples folder to access all sample clients:

   shell

   cd hazelcast-platform-operator-external-backup-restore

   The sample code for this tutorial is in the docs/modules/ROOT/examples/operator-external-backup directory.

   CLC

   note

   CLC must be installed in your system. See the CLC installation instructions.

   Add the cluster configuration to CLC:

   bash

   clc config add hz cluster.name=dev cluster.address=<EXTERNAL-IP>

   Java

   java

   package com.hazelcast;
   
   import com.hazelcast.client.HazelcastClient;
   import com.hazelcast.client.config.ClientConfig;
   import com.hazelcast.client.config.ClientNetworkConfig;
   import com.hazelcast.client.impl.connection.tcp.RoutingMode;
   import com.hazelcast.core.HazelcastInstance;
   import com.hazelcast.map.IMap;
   
   import java.util.Random;
   
   public class Main {
       public static void main(String[] args) throws Exception {
           if(args.length == 0) {
               System.out.println("You should pass an argument to run: fill or size");
           } else if (!((args[0].equals("fill") || args[0].equals("size")))) {
               System.out.println("Wrong argument, you should pass: fill or size");
           } else{
               ClientConfig clientConfig = new ClientConfig();
               ClientNetworkConfig networkConfig = clientConfig.getNetworkConfig();
               networkConfig.addAddress("<EXTERNAL-IP>");
               networkConfig.getClusterRoutingConfig().setRoutingMode(RoutingMode.SINGLE_MEMBER);
               HazelcastInstance client = HazelcastClient.newHazelcastClient(clientConfig);
               System.out.println("Successful connection!");
   
               IMap<String, String> map = client.getMap("persistent-map");
   
               if (args[0].equals("fill")) {
                   System.out.println("Starting to fill the map with random entries.");
   
                   Random random = new Random();
                   while (true) {
                       int randomKey = random.nextInt(100_000);
                       map.put("key-" + randomKey, "value-" + randomKey);
                       System.out.println("Current map size: " + map.size());
                   }
               } else {
                   System.out.println("Current map size: " + map.size());
                   client.shutdown();
               }
           }
   
       }
   }

   NodeJS

   javascript

   'use strict';
   
   const { Client } = require('hazelcast-client');
   
   const clientConfig = {
       network: {
           clusterMembers: [
               '<EXTERNAL-IP>'
           ],
           smartRouting: false
       }
   };
   
   (async () => {
       try {
           if (process.argv.length === 2) {
               console.error('You should pass an argument to run: fill or size');
           } else if (!(process.argv[2] === 'fill' || process.argv[2] === 'size')) {
               console.error('Wrong argument, you should pass: fill or size');
           } else {
               const client = await Client.newHazelcastClient(clientConfig);
               const map = await client.getMap('persistent-map');
               await map.put('key', 'value');
               const res = await map.get('key');
               if (res !== 'value') {
                   throw new Error('Connection failed, check your configuration.');
               }
               console.log('Successful connection!');
               if (process.argv[2] === 'fill'){
                   console.log('Starting to fill the map with random entries.');
                   while (true) {
                       const randomKey = Math.floor(Math.random() * 100000);
                       await map.put('key' + randomKey, 'value' + randomKey);
                       const size = await map.size();
                       console.log(`Current map size: ${size}`);
                   }
               } else {
                   const size = await map.size();
                   console.log(`Current map size: ${size}`);
               }
           }
       } catch (err) {
           console.error('Error occurred:', err);
       }
   })();

   Go

   go

   package main
   
   import (
   	"context"
   	"fmt"
   	"math/rand"
   	"os"
   
   	"github.com/hazelcast/hazelcast-go-client"
   )
   
   func main() {
   	if len(os.Args) != 2 {
   		fmt.Println("You should pass an argument to run: fill or size")
   		return
   	}
   	if !(os.Args[1] == "fill" || os.Args[1] == "size") {
   		fmt.Println("Wrong argument, you should pass: fill or size")
   		return
   	}
   
   	config := hazelcast.Config{}
   	cc := &config.Cluster
   	cc.Network.SetAddresses("<EXTERNAL-IP>:5701")
   	cc.Unisocket = true
   	ctx := context.TODO()
   	client, err := hazelcast.StartNewClientWithConfig(ctx, config)
   	if err != nil {
   		panic(err)
   	}
   	fmt.Println("Successful connection!")
   	m, err := client.GetMap(ctx, "persistent-map")
   	if err != nil {
   		panic(err)
   	}
   	if os.Args[1] == "fill" {
   		fmt.Println("Starting to fill the map with random entries.")
   		for {
   			num := rand.Intn(100_000)
   			key := fmt.Sprintf("key-%d", num)
   			value := fmt.Sprintf("value-%d", num)
   			if _, err = m.Put(ctx, key, value); err != nil {
   				fmt.Println("ERR:", err.Error())
   				continue
   			}
   			mapSize, err := m.Size(ctx)
   			if err != nil {
   				fmt.Println("ERR:", err.Error())
   				continue
   			}
   			fmt.Println("Current map size:", mapSize)
   		}
   		return
   	}
   	mapSize, err := m.Size(ctx)
   	if err != nil {
   		fmt.Println("ERR:", err.Error())
   		return
   	}
   	fmt.Println("Current map size:", mapSize)
   
   }

   Python

   python

   import logging
   import random
   import sys
   
   import hazelcast
   
   logging.basicConfig(level=logging.INFO)
   
   if len(sys.argv) != 2:
       print("You should pass an argument to run: fill or size")
   elif not (sys.argv[1] == "fill" or sys.argv[1] == "size"):
       print("Wrong argument, you should pass: fill or size")
   else:
       client = hazelcast.HazelcastClient(
           cluster_members=["<EXTERNAL-IP>"],
           smart_routing=False,
       )
       print("Successful connection!", flush=True)
       m = client.get_map("persistent-map").blocking()
   
       if sys.argv[1] == "fill":
           print("Starting to fill the map with random entries.", flush=True)
           while True:
               random_number = str(random.randrange(0, 100000))
               m.put("key-" + random_number, "value-" + random_number)
               print("Current map size:", m.size())
       else:
           print("Current map size:", m.size())

   .NET

   cs

   using System;
   using System.Threading.Tasks;
   using Hazelcast;
   using Microsoft.Extensions.Logging;
   
   namespace Client
   {
       public class Program
       {
           static async Task Main(string[] args)
           {
               if (args.Length != 1)
               {
                   Console.WriteLine("You should pass an argument to run: fill or size");
                   return;
               }
               if (!(args[0] == "fill" || args[0] == "size"))
               {
                   Console.WriteLine("Wrong argument, you should pass: fill or size");
                   return;
               }
               var options = new HazelcastOptionsBuilder()                
                   .With(args)                
                   .With((configuration, options) =>
                   {
                       options.LoggerFactory.Creator = () => LoggerFactory.Create(loggingBuilder =>
                           loggingBuilder
                               .AddConfiguration(configuration.GetSection("logging"))
                               .AddConsole());
   
                       options.Networking.Addresses.Add("<EXTERNAL-IP>:5701");
                       options.Networking.SmartRouting = false;
                       
                   })
                   .Build();
   
   
               await using var client = await HazelcastClientFactory.StartNewClientAsync(options);
   
               Console.WriteLine("Successful connection!");
               Console.WriteLine("Starting to fill the map with random entries.");
   
               var map = await client.GetMapAsync<string, string>("persistent-map");
               var random = new Random();
   
               if (args[0] == "fill")
               {
                   Console.WriteLine("Starting to fill the map with random entries.");
                   while (true)
                   {
                       var num = random.Next(100_000);
                       var key = $"key-{num}";
                       var value = $"value-{num}";
                       await map.PutAsync(key, value);
                       var mapSize = await map.GetSizeAsync();
                       Console.WriteLine($"Current map size: {mapSize}");
                   }
               }
               else
               {
                   var mapSize = await map.GetSizeAsync();
                   Console.WriteLine($"Current map size: {mapSize}");
                   await client.DisposeAsync();
               }
           }
       } 
   }

4. Start the client to fill the map.

   CLC

   Fill a map:

   bash

   for i in {1..10};
   do
      clc -c hz map set --name persistent-map key-$i value-$i;
   done

   Check the map size:

   bash

   clc -c hz map size --name persistent-map

   Java

   bash

   cd java
   mvn package
   java -jar target/*jar-with-dependencies*.jar fill

   You should see the following output:

   shell

   Successful connection!
   Starting to fill the map with random entries.
   Current map size: 2
   Current map size: 3
   Current map size: 4
   ....
   ....

   NodeJS

   bash

   cd nodejs
   npm install
   npm start fill

   You should see the following output:

   shell

   Successful connection!
   Starting to fill the map with random entries.
   Current map size: 2
   Current map size: 3
   Current map size: 4
   ....
   ....

   Go

   bash

   cd go
   go run main.go fill

   You should see the following output:

   shell

   Successful connection!
   Starting to fill the map with random entries.
   Current map size: 2
   Current map size: 3
   Current map size: 4
   ....
   ....

   Python

   bash

   cd python
   pip install -r requirements.txt
   python main.py fill

   You should see the following output:

   shell

   Successful connection!
   Starting to fill the map with random entries.
   Current map size: 2
   Current map size: 3
   Current map size: 4
   ....
   ....

   .NET

   bash

   cd dotnet
   dotnet build
   dotnet run fill

   shell

   Successful connection!
   Starting to fill the map with random entries.
   Current map size: 2
   Current map size: 3
   Current map size: 4
   ....
   ....

Trigger an external backup

To trigger a backup, you need a bucketURI where backup data will be stored and a secret with credentials for accessing the bucket URI.

1. Run one of the following commands to create the secret according to the cloud provider you want to back up:

   AWS

   bash

   kubectl create secret generic <external-bucket-secret-name> --from-literal=region=<region> \
   	--from-literal=access-key-id=<access-key-id> \
   	--from-literal=secret-access-key=<secret-access-key>

   GCP

   bash

   kubectl create secret generic <external-bucket-secret-name> --from-file=google-credentials-path=<service_account_json_file>

   Azure

   bash

   kubectl create secret generic <external-bucket-secret-name> \
   	--from-literal=storage-account=<storage-account> \
   	--from-literal=storage-key=<storage-key>

2. Trigger a backup:

   shell

   kubectl apply -f - <<EOF
   apiVersion: hazelcast.com/v1alpha1
   kind: HotBackup
   metadata:
     name: hot-backup
   spec:
     hazelcastResourceName: my-hazelcast
     bucketURI: "s3://operator-backup"
     secret: "br-secret-s3"
   EOF

   Example URI → "s3://operator-backup?prefix=hazelcast/2022-06-08-17-01-20/"

3. Check the status of the backup:

   bash

   kubectl get hotbackup hot-backup

   The status of the backup is displayed in the output.

   bash

   NAME         STATUS
   hot-backup   Success

Restore from an external backup

1. Delete the Hazelcast cluster:

   bash

   kubectl delete hazelcast my-hazelcast

2. Delete the PersistentVolumeClaims.

   The data is restored from external buckets, so you don't keep persistent volumes during a restore operation.

   List PVCs that were created for the Hazelcast cluster:

   bash

   kubectl get pvc -l "app.kubernetes.io/managed-by=hazelcast-platform-operator"
   NAME                         STATUS   VOLUME                                     CAPACITY
   persistence-my-hazelcast-0   Bound    pvc-a77fcd6e-e64d-4aeb-9708-835d66b4c5b9   8Gi
   persistence-my-hazelcast-1   Bound    pvc-8548a4a5-fa8e-45bd-9909-8e6bb6a6c3e8   8Gi
   persistence-my-hazelcast-2   Bound    pvc-c94edb25-5a72-482a-bf4e-84a99319d509   8Gi

   Delete the PVCs:

   bash

   kubectl delete pvc -l "app.kubernetes.io/managed-by=hazelcast-platform-operator"

3. Create a new Hazelcast cluster.

   For restoring, you will use the HotBackup resource you have created.

   Create the Hazelcast cluster. Before the Hazelcast cluster is started, Operator starts the agent (as InitContainer), which restores the backup data from the external bucket.

   shell

   kubectl apply -f - <<EOF
   apiVersion: hazelcast.com/v1alpha1
   kind: Hazelcast
   metadata:
     name: my-hazelcast
   spec:
     clusterSize: 3
     licenseKeySecretName: hazelcast-license-key
     persistence:
       pvc:
         accessModes: ["ReadWriteOnce"]
         requestStorage: 8Gi
       restore:
         hotBackupResourceName: hot-backup
     exposeExternally:
       type: Unisocket
       discoveryServiceType: LoadBalancer
   EOF

   As you may see, the agent configuration is not set. Operator therefore uses the latest stable version of the agent.

4. Check the cluster status:

   shell

   kubectl get hazelcast my-hazelcast
   NAME           STATUS    MEMBERS
   my-hazelcast   Running   3/3

   After verifying that the cluster is Running and all the members are ready, find the discovery address:

   shell

   kubectl get hazelcastendpoint my-hazelcast
   NAME           TYPE        ADDRESS
   my-hazelcast   Discovery   34.28.159.226:5701

   Since we recreated the Hazelcast cluster, services are also recreated. The ADDRESS may change.

5. Check the map size.

   Configure the Hazelcast client to connect to the cluster external address as you did in Configure the Hazelcast Client.

   Start the client to check the map size and check the restore is successful.

   CLC

   bash

   clc -c hz map size --name persistent-map

   Java

   bash

   cd java
   mvn package
   java -jar target/*jar-with-dependencies*.jar size

   You should see the following output:

   shell

   Successful connection!
   Current map size: 12

   NodeJS

   bash

   cd nodejs
   npm install
   npm start size

   You should see the following output:

   shell

   Successful connection!
   Current map size: 12

   Go

   bash

   cd go
   go run main.go size

   You should see the following output:

   shell

   Successful connection!
   Current map size: 12

   Python

   bash

   cd python
   pip install -r requirements.txt
   python main.py size

   You should see the following output:

   shell

   Successful connection!
   Current map size: 12

   .NET

   bash

   cd dotnet
   dotnet run size

   You should see the following output:

   shell

   Successful connection!
   Current map size: 12

Clean up

To clean up the created resources, remove all Custom Resources and PVCs.

shell

kubectl delete secret <external-bucket-secret-name>
kubectl delete secret hazelcast-license-key
kubectl delete $(kubectl get hazelcast,hotbackup -o name)
kubectl delete pvc -l "app.kubernetes.io/managed-by=hazelcast-platform-operator"

Summary

In this tutorial, you learned how to:

• Deploy Hazelcast with persistence enabled.
• Create a Hazelcast map that has persistence enabled.
• Back up all map entries to external storage in the cloud.
• Restart the Hazelcast cluster and restore the backup map entries from the cloud.

Next steps

See also:

• Persistence, backup and restore
• Tutorial: Connect to Hazelcast from outside Kubernetes