Restore a cluster from cloud storage with Platform Operator

Learn how to back up data in Hazelcast maps to cloud storage and restore a cluster from that backup data.

Overview

In this tutorial, you’ll do the following:

Deploy Hazelcast with persistence enabled
Create a Hazelcast map that has persistence enabled
Back up all map entries to external storage in the Cloud
Restart the Hazelcast cluster and restore the backup map entries from the Cloud

The tutorial should take approximately 20 minutes to complete.

Prerequisites

Before you begin, make sure that you have:

An up and running Kubernetes cluster
Kubernetes command-line tool, kubectl
Deployed Hazelcast Platform Operator
Created blob storage and access credentials in one of the Cloud providers: AWS - GCP - Azure
The Hazelcast Code Samples repository https://github.com/hazelcast/hazelcast-code-samples cloned to your local machine

Start the Hazelcast cluster

Create a license secret

Create a secret with your Hazelcast Enterprise License.

kubectl create secret generic hazelcast-license-key --from-literal=license-key=<hz-license-key>

Create the Hazelcast cluster

Create the Hazelcast cluster with Persistence enabled:

kubectl apply -f - <<EOF
apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: my-hazelcast
spec:
  clusterSize: 3
  licenseKeySecretName: hazelcast-license-key
  persistence:
    pvc:
      accessModes: ["ReadWriteOnce"]
      requestStorage: 8Gi
  exposeExternally:
    type: Unisocket
    discoveryServiceType: LoadBalancer
EOF

Check the cluster status

Run the following commands to see the cluster status:

$ kubectl get hazelcast my-hazelcast
NAME           STATUS    MEMBERS
my-hazelcast   Running   3/3

$ kubectl get pods -l app.kubernetes.io/instance=my-hazelcast
NAME             READY   STATUS    RESTARTS   AGE
my-hazelcast-0   2/2     Running   0          3m43s
my-hazelcast-1   2/2     Running   0          3m16s
my-hazelcast-2   2/2     Running   0          2m50s

As you can see from the pod states, the agent container will be deployed with the Hazelcast container in the same Pod. The agent is responsible for backing data up into the external storage.

Get the address of the Hazelcast cluster

After verifying that the cluster is Running and all the members are ready, find the discovery address:
```
$ kubectl get hazelcastendpoint my-hazelcast
NAME               TYPE        ADDRESS
my-hazelcast       Discovery   34.30.60.128:5701
```
The ADDRESS column displays the external address of your Hazelcast cluster.

Create persistent map and put data

Create persistent map

Create the Map resource with Persistence enabled:

kubectl apply -f - <<EOF
apiVersion: hazelcast.com/v1alpha1
kind: Map
metadata:
  name: persistent-map
spec:
  hazelcastResourceName: my-hazelcast
  persistenceEnabled: true
EOF

Check the status of the backup

Check the status of the map configuration:
```
kubectl get map persistent-map
```
The status of the map is displayed in the output.
```
NAME             STATUS
persistent-map   Success
```

Configure the Hazelcast client to connect to the cluster.

Open the examples folder to access all sample clients:

cd hazelcast-platform-operator-external-backup-restore

The sample code (excluding CLC) for this tutorial is in the docs/modules/ROOT/examples/operator-external-backup directory.

CLC
Java
NodeJS
Go
Python
.NET

CLC must be installed in your system. See the CLC installation instructions.

Add the cluster config to the CLC:

clc config add hz cluster.name=dev cluster.address=<EXTERNAL-IP>

package com.hazelcast;

import com.hazelcast.client.HazelcastClient;
import com.hazelcast.client.config.ClientConfig;
import com.hazelcast.client.config.ClientNetworkConfig;
import com.hazelcast.client.impl.connection.tcp.RoutingMode;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.map.IMap;

import java.util.Random;

public class Main {
    public static void main(String[] args) throws Exception {
        if(args.length == 0) {
            System.out.println("You should pass an argument to run: fill or size");
        } else if (!((args[0].equals("fill") || args[0].equals("size")))) {
            System.out.println("Wrong argument, you should pass: fill or size");
        } else{
            ClientConfig clientConfig = new ClientConfig();
            ClientNetworkConfig networkConfig = clientConfig.getNetworkConfig();
            networkConfig.addAddress("<EXTERNAL-IP>");
            networkConfig.getClusterRoutingConfig().setRoutingMode(RoutingMode.SINGLE_MEMBER);
            HazelcastInstance client = HazelcastClient.newHazelcastClient(clientConfig);
            System.out.println("Successful connection!");

            IMap<String, String> map = client.getMap("persistent-map");

            if (args[0].equals("fill")) {
                System.out.println("Starting to fill the map with random entries.");

                Random random = new Random();
                while (true) {
                    int randomKey = random.nextInt(100_000);
                    map.put("key-" + randomKey, "value-" + randomKey);
                    System.out.println("Current map size: " + map.size());
                }
            } else {
                System.out.println("Current map size: " + map.size());
                client.shutdown();
            }
        }

    }
}

'use strict';

const { Client } = require('hazelcast-client');

const clientConfig = {
    network: {
        clusterMembers: [
            '<EXTERNAL-IP>'
        ],
        smartRouting: false
    }
};

(async () => {
    try {
        if (process.argv.length === 2) {
            console.error('You should pass an argument to run: fill or size');
        } else if (!(process.argv[2] === 'fill' || process.argv[2] === 'size')) {
            console.error('Wrong argument, you should pass: fill or size');
        } else {
            const client = await Client.newHazelcastClient(clientConfig);
            const map = await client.getMap('persistent-map');
            await map.put('key', 'value');
            const res = await map.get('key');
            if (res !== 'value') {
                throw new Error('Connection failed, check your configuration.');
            }
            console.log('Successful connection!');
            if (process.argv[2] === 'fill'){
                console.log('Starting to fill the map with random entries.');
                while (true) {
                    const randomKey = Math.floor(Math.random() * 100000);
                    await map.put('key' + randomKey, 'value' + randomKey);
                    const size = await map.size();
                    console.log(`Current map size: ${size}`);
                }
            } else {
                const size = await map.size();
                console.log(`Current map size: ${size}`);
            }
        }
    } catch (err) {
        console.error('Error occurred:', err);
    }
})();

package main

import (
	"context"
	"fmt"
	"math/rand"
	"os"

	"github.com/hazelcast/hazelcast-go-client"
)

func main() {
	if len(os.Args) != 2 {
		fmt.Println("You should pass an argument to run: fill or size")
		return
	}
	if !(os.Args[1] == "fill" || os.Args[1] == "size") {
		fmt.Println("Wrong argument, you should pass: fill or size")
		return
	}

	config := hazelcast.Config{}
	cc := &config.Cluster
	cc.Network.SetAddresses("<EXTERNAL-IP>:5701")
	cc.Unisocket = true
	ctx := context.TODO()
	client, err := hazelcast.StartNewClientWithConfig(ctx, config)
	if err != nil {
		panic(err)
	}
	fmt.Println("Successful connection!")
	m, err := client.GetMap(ctx, "persistent-map")
	if err != nil {
		panic(err)
	}
	if os.Args[1] == "fill" {
		fmt.Println("Starting to fill the map with random entries.")
		for {
			num := rand.Intn(100_000)
			key := fmt.Sprintf("key-%d", num)
			value := fmt.Sprintf("value-%d", num)
			if _, err = m.Put(ctx, key, value); err != nil {
				fmt.Println("ERR:", err.Error())
				continue
			}
			mapSize, err := m.Size(ctx)
			if err != nil {
				fmt.Println("ERR:", err.Error())
				continue
			}
			fmt.Println("Current map size:", mapSize)
		}
		return
	}
	mapSize, err := m.Size(ctx)
	if err != nil {
		fmt.Println("ERR:", err.Error())
		return
	}
	fmt.Println("Current map size:", mapSize)

}

import logging
import random
import sys

import hazelcast

logging.basicConfig(level=logging.INFO)

if len(sys.argv) != 2:
    print("You should pass an argument to run: fill or size")
elif not (sys.argv[1] == "fill" or sys.argv[1] == "size"):
    print("Wrong argument, you should pass: fill or size")
else:
    client = hazelcast.HazelcastClient(
        cluster_members=["<EXTERNAL-IP>"],
        smart_routing=False,
    )
    print("Successful connection!", flush=True)
    m = client.get_map("persistent-map").blocking()

    if sys.argv[1] == "fill":
        print("Starting to fill the map with random entries.", flush=True)
        while True:
            random_number = str(random.randrange(0, 100000))
            m.put("key-" + random_number, "value-" + random_number)
            print("Current map size:", m.size())
    else:
        print("Current map size:", m.size())

using System;
using System.Threading.Tasks;
using Hazelcast;
using Microsoft.Extensions.Logging;

namespace Client
{
    public class Program
    {
        static async Task Main(string[] args)
        {
            if (args.Length != 1)
            {
                Console.WriteLine("You should pass an argument to run: fill or size");
                return;
            }
            if (!(args[0] == "fill" || args[0] == "size"))
            {
                Console.WriteLine("Wrong argument, you should pass: fill or size");
                return;
            }
            var options = new HazelcastOptionsBuilder()                
                .With(args)                
                .With((configuration, options) =>
                {
                    options.LoggerFactory.Creator = () => LoggerFactory.Create(loggingBuilder =>
                        loggingBuilder
                            .AddConfiguration(configuration.GetSection("logging"))
                            .AddConsole());

                    options.Networking.Addresses.Add("<EXTERNAL-IP>:5701");
                    options.Networking.SmartRouting = false;
                    
                })
                .Build();


            await using var client = await HazelcastClientFactory.StartNewClientAsync(options);

            Console.WriteLine("Successful connection!");
            Console.WriteLine("Starting to fill the map with random entries.");

            var map = await client.GetMapAsync<string, string>("persistent-map");
            var random = new Random();

            if (args[0] == "fill")
            {
                Console.WriteLine("Starting to fill the map with random entries.");
                while (true)
                {
                    var num = random.Next(100_000);
                    var key = $"key-{num}";
                    var value = $"value-{num}";
                    await map.PutAsync(key, value);
                    var mapSize = await map.GetSizeAsync();
                    Console.WriteLine($"Current map size: {mapSize}");
                }
            }
            else
            {
                var mapSize = await map.GetSizeAsync();
                Console.WriteLine($"Current map size: {mapSize}");
                await client.DisposeAsync();
            }
        }
    } 
}

Start the client to fill the map.

CLC
Java
NodeJS
Go
Python
.NET

Fill a map:

for i in {1..10};
do
   clc -c hz map set --name persistent-map key-$i value-$i;
done

Check the map size:

clc -c hz map size --name persistent-map

cd java
mvn package
java -jar target/*jar-with-dependencies*.jar fill

You should see the following output:

Successful connection!
Starting to fill the map with random entries.
Current map size: 2
Current map size: 3
Current map size: 4
....
....

cd nodejs
npm install
npm start fill

You should see the following output:

Successful connection!
Starting to fill the map with random entries.
Current map size: 2
Current map size: 3
Current map size: 4
....
....

cd go
go run main.go fill

You should see the following output:

Successful connection!
Starting to fill the map with random entries.
Current map size: 2
Current map size: 3
Current map size: 4
....
....

cd python
pip install -r requirements.txt
python main.py fill

You should see the following output:

Successful connection!
Starting to fill the map with random entries.
Current map size: 2
Current map size: 3
Current map size: 4
....
....

cd dotnet
dotnet build
dotnet run fill

Successful connection!
Starting to fill the map with random entries.
Current map size: 2
Current map size: 3
Current map size: 4
....
....

Trigger external backup

For triggering backup, you need bucketURI where backup data will be stored in, and a secret with credentials for accessing the given Bucket URI.

Create Secret

Run one of the following commands to create the secret according to the Cloud provider you want to backup:

AWS
GCP
Azure

kubectl create secret generic <external-bucket-secret-name> --from-literal=region=<region> \
	--from-literal=access-key-id=<access-key-id> \
	--from-literal=secret-access-key=<secret-access-key>

kubectl create secret generic <external-bucket-secret-name> --from-file=google-credentials-path=<service_account_json_file>

kubectl create secret generic <external-bucket-secret-name> \
	--from-literal=storage-account=<storage-account> \
	--from-literal=storage-key=<storage-key>

Trigger backup

Run the following command to trigger backup:

kubectl apply -f - <<EOF
apiVersion: hazelcast.com/v1alpha1
kind: HotBackup
metadata:
  name: hot-backup
spec:
  hazelcastResourceName: my-hazelcast
  bucketURI: "s3://operator-backup"
  secret: "br-secret-s3"
EOF

Example URI → "s3://operator-backup?prefix=hazelcast/2022-06-08-17-01-20/"

Check the status of the backup

Run the following command to check the status of the backup:
```
kubectl get hotbackup hot-backup
```
The status of the backup is displayed in the output.
```
NAME         STATUS
hot-backup   Success
```

Restore from external backup

Delete the Hazelcast cluster

Run the following command to delete the Hazelcast cluster:
```
kubectl delete hazelcast my-hazelcast
```

Delete the PersistentVolumeClaims

The data is restored from external buckets, so you don’t keep persistent volumes during a restore operation.

List PVCs which were created for the Hazelcast cluster:

$ kubectl get pvc -l "app.kubernetes.io/managed-by=hazelcast-platform-operator"
NAME                         STATUS   VOLUME                                     CAPACITY
persistence-my-hazelcast-0   Bound    pvc-a77fcd6e-e64d-4aeb-9708-835d66b4c5b9   8Gi
persistence-my-hazelcast-1   Bound    pvc-8548a4a5-fa8e-45bd-9909-8e6bb6a6c3e8   8Gi
persistence-my-hazelcast-2   Bound    pvc-c94edb25-5a72-482a-bf4e-84a99319d509   8Gi

Delete the PVCs:

kubectl delete pvc -l "app.kubernetes.io/managed-by=hazelcast-platform-operator"

Create new Hazelcast cluster

For restoring, you will use the HotBackup resource you have created.

Create the Hazelcast cluster. Before the Hazelcast cluster is started, the Operator starts the agent (as InitContainer) which restores the backup data from the external bucket.

kubectl apply -f - <<EOF
apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: my-hazelcast
spec:
  clusterSize: 3
  licenseKeySecretName: hazelcast-license-key
  persistence:
    pvc:
      accessModes: ["ReadWriteOnce"]
      requestStorage: 8Gi
    restore:
      hotBackupResourceName: hot-backup
  exposeExternally:
    type: Unisocket
    discoveryServiceType: LoadBalancer
EOF

As you may see, the agent configuration is not set. Thus, the Operator directly uses the latest stable version of the agent.

Check the cluster status

Run the following commands to see the cluster status:
```
$ kubectl get hazelcast my-hazelcast
NAME           STATUS    MEMBERS
my-hazelcast   Running   3/3
```
After verifying that the cluster is Running and all the members are ready, find the discovery address:
```
$ kubectl get hazelcastendpoint my-hazelcast
NAME           TYPE        ADDRESS
my-hazelcast   Discovery   34.28.159.226:5701
```
Since we recreate the Hazelcast cluster, services are also recreated. The ADDRESS may change.

Check the map size

Configure the Hazelcast client to connect to the cluster external address as you did in Configure the Hazelcast Client.

Start the client to check the map size and see if the restore is successful.

CLC
Java
NodeJS
Go
Python
.NET

clc -c hz map size --name persistent-map

cd java
mvn package
java -jar target/*jar-with-dependencies*.jar size

You should see the following output:

Successful connection!
Current map size: 12

cd nodejs
npm install
npm start size

You should see the following output:

Successful connection!
Current map size: 12

cd go
go run main.go size

You should see the following output:

Successful connection!
Current map size: 12

cd python
pip install -r requirements.txt
python main.py size

You should see the following output:

Successful connection!
Current map size: 12

cd dotnet
dotnet run size

You should see the following output:

Successful connection!
Current map size: 12

Clean up

To clean up the created resources, remove all Custom Resources and PVCs.

kubectl delete secret <external-bucket-secret-name>
kubectl delete secret hazelcast-license-key
kubectl delete $(kubectl get hazelcast,hotbackup -o name)
kubectl delete pvc -l "app.kubernetes.io/managed-by=hazelcast-platform-operator"

Summary

In this tutorial, you’ve learned how to:

Deploy Hazelcast with persistence enabled
Create a Hazelcast map that has persistence enabled
Back up all map entries to external storage in the Cloud
Restart the Hazelcast cluster and restore the backup map entries from the Cloud

Restore a cluster from cloud storage with Platform Operator

Overview

Prerequisites

Start the Hazelcast cluster

Create persistent map and put data

Trigger external backup

Restore from external backup

Clean up

Summary

Next steps

Send us your feedback

Help and support