Get Started with Embedded Hazelcast on ECS

What You’ll Learn

In this tutorial, you’ll deploy an application with embedded Hazelcast on an ECS cluster. Hazelcast members from each application replica will automatically discover themselves and form one consistent Hazelcast cluster. Thanks to Hazelcast AWS discovery plugin, there is no static configuration needed.

Before you Begin

Create an Application

You can embed Hazelcast into any JVM-based application and use any web framework you want. As an example, this tutorial uses the application from the Get Started with Hazelcast using Spring Boot tutorial. To download it, execute the following command.

git clone https://github.com/hazelcast-guides/hazelcast-embedded-springboot

Use Hazelcast AWS ECS Configuration

Hazelcast provides the dedicated Hazelcast AWS plugin which allows to automatically form Hazelcast cluster in the AWS environment. It supports both EC2 discovery and ECS discovery. To enabled it, use the following Hazelcast configuration.

hazelcast:
  network:
    interfaces:
      enabled: true
      interfaces:
        - 10.0.*.*
    join:
      multicast:
        enabled: false
      aws:
        enabled: true

Note that you need to change 10.0.*.* to the VPC you plan to use for your ECS Tasks. To list all your available VPCs, execute the following command.

aws ec2 describe-vpcs --query 'Vpcs[*].[VpcId,CidrBlock]'
[
    [
        "vpc-0ae005a4a2835bbb5",
        "172.18.0.0/16"
    ],
    [
        "vpc-0681043d6f49b039a",
        "10.0.0.0/16"
    ]
]

For the next steps, let’s assume that we always use the VPC with ID vpc-0681043d6f49b039a (CIDR vpc-10.0.0.0/16). For the sake of simplicity, let’s put it as an environment variable:

export VPC_ID='vpc-0681043d6f49b039a'

To include hazelcast.yaml inside your Spring Boot project, copy it into hazelcast-embedded-springboot/src/main/resources/.

cp hazelcast.yaml hazelcast-embedded-springboot/src/main/resources/

Now, you can build the project with the following command.

mvn package -f hazelcast-embedded-springboot/pom.xml

As an output, the JAR file with our application should be created at hazelcast-embedded-springboot/target/*.jar.

Containerize the Application

To containerize the application, you need to have Docker installed. Then, you can use the following Dockerfile.

FROM openjdk:8-jre-alpine
COPY hazelcast-embedded-springboot/target/*.jar app.jar
ENTRYPOINT ["java","-jar","app.jar"]

In order to build the Docker image, run the following command.

docker build -t hazelcastguides/hazelcast-embedded-ecs .

If you build the image by yourself, then you need to use your Docker Hub account instead of hazelcastguides. Then, you can push the image into your Docker Hub registry with the following command.

docker push hazelcastguides/hazelcast-embedded-ecs

If you want to use your image in the following steps, please also make sure your Docker Hub registry is public. However, for the purpose of this tutorial, you can also use the already built hazelcastguides/hazelcast-embedded-ecs Docker image.

Create AWS ECS Cluster

Assuming you have an AWS Account and AWS command line tool configured, you can create an ECS cluster with the following command.

aws ecs create-cluster --cluster-name test-cluster

We will use this cluster for all further steps in this tutorial.

Create AWS Security Group

By default, Hazelcast uses port 5701 for all the communication. Therefore, we need to create an appropriate security group.

aws ec2 create-security-group \
  --group-name hazelcast-security-group \
  --description "Hazelcast Security Group" \
  --vpc-id ${VPC_ID}
{
    "GroupId": "sg-0cf1116753b96cdce"
}

Let’s define the security group id as an environment variable.

export SECURITY_GROUP_ID='sg-0cf1116753b96cdce'

Next, we need to open port 5701 for the TCP communication.

aws ec2 authorize-security-group-ingress \
  --group-id ${SECURITY_GROUP_ID} \
  --protocol tcp \
  --port 5701 \
  --cidr 0.0.0.0/0

That is enough for the Hazelcast communication, however if you also want to play with the Spring Boot service, then you need to whitelist its port 8080.

aws ec2 authorize-security-group-ingress \
  --group-id ${SECURITY_GROUP_ID} \
  --protocol tcp \
  --port 8080 \
  --cidr 0.0.0.0/0

Create IAM Role

Hazelcast ECS Discovery uses AWS API to discover Hazelcast members. That is why we need to grant certain permissions. Here are the commands to create an IAM Role we will later use by ECS tasks.

aws iam create-policy \
  --policy-name hazelcast-ecs-policy \
  --policy-document file://policy.json
{
    "Policy": {
        "PolicyName": "hazelcast-ecs-policy",
        "PolicyId": "ANPAZV4HIPQ4QUIH45BXB",
        "Arn": "arn:aws:iam::665466731577:policy/hazelcast-ecs-policy",
        "Path": "/",
        "DefaultVersionId": "v1",
        "AttachmentCount": 0,
        "PermissionsBoundaryUsageCount": 0,
        "IsAttachable": true,
        "CreateDate": "2020-06-29T09:52:05Z",
        "UpdateDate": "2020-06-29T09:52:05Z"
    }
}

Let’s define this policy ARN as an environment variable.

export POLICY_ARN='arn:aws:iam::665466731577:policy/hazelcast-ecs-policy'

Next, let’s define a new IAM Role.

aws iam create-role \
  --role-name hazelcast-ecs-role \
  --assume-role-policy-document file://role-policy.json
{
    "Role": {
        "Path": "/",
        "RoleName": "hazelcast-ecs-role",
        "RoleId": "AROAZV4HIPQ47NJGCHJ2A",
        "Arn": "arn:aws:iam::665466731577:role/hazelcast-ecs-role",
        "CreateDate": "2020-06-29T09:58:06Z",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "ecs-tasks.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}

Define an environment variable with the role ARN.

export TASK_ROLE_ARN='arn:aws:iam::665466731577:role/hazelcast-ecs-role'

Finally, let’s attach the defined policy to the created role.

aws iam attach-role-policy --role-name hazelcast-ecs-role --policy-arn ${POLICY_ARN}

Configure AWS CloudWatch

If you want to read logs from your application, then you need to create AWS CloudWatch Group. Note that this step is not required, but highly recommended. If you skip it, you also need to remove CloudWatch entry from task-definition.json and then, in the further steps, you won’t be able to see any logs from your application.

To create CloudWatch group, execute the following command.

aws logs create-log-group --log-group-name /ecs/hazelcast

To allow ECS task to write to CloudWatch log group, you also need to create the following ECS role.

aws iam create-role \
  --role-name ecs-execution-role \
  --assume-role-policy-document file://role-policy.json
{
    "Role": {
        "Path": "/",
        "RoleName": "ecs-execution-role",
        "RoleId": "AROAZV4HIPQ4SGDIWGYRK",
        "Arn": "arn:aws:iam::665466731577:role/ecs-execution-role",
        "CreateDate": "2020-06-29T10:44:01Z",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "ecs-tasks.amazonaws.com"
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}

Export this role as an environment variable.

export EXECUTION_ROLE_ARN='arn:aws:iam::665466731577:role/ecs-execution-role'

Finally, attach AmazonECSTaskExecutionRolePolicy to the created role.

aws iam attach-role-policy \
  --role-name ecs-execution-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

Create Task Definition

To create ECS task definition, you need first to update task-definition.json with the following values:

  • TASK_ROLE_ARN - ARN of the role created in the "Create IAM Role" step

  • EXECUTION_ROLE_ARN - ARN of the Role created in the "Configure AWS CloudWatch" step

  • REGION - Region that you use for your CloudWatch and ECS Cluster

export ECS_REGION=$(aws configure get region)

sed -i.bak "s~TASK_ROLE_ARN~${TASK_ROLE_ARN}~g" task-definition.json
sed -i.bak "s~EXECUTION_ROLE_ARN~${EXECUTION_ROLE_ARN}~g" task-definition.json
sed -i.bak "s~REGION~${ECS_REGION}~g" task-definition.json

rm task-definition.json.bak

Then, you can create task definition with the following command.

aws ecs register-task-definition --cli-input-json file://task-definition.json

Create ECS Service

Finally, when all is set and done, we can start an ECS service which in turn creates ECS tasks. First, choose the subnet in which your service should operate.

aws ec2 describe-subnets \
  --filters "Name=vpc-id,Values=${VPC_ID}" \
  --query 'Subnets[*].[SubnetId,CidrBlock]'
[
    [
        "subnet-0f042c997bad8e2b9",
        "10.0.1.0/24"
    ]
]

export SUBNET_ID='subnet-0f042c997bad8e2b9'

Then, create a service with 3 application replicas.

aws ecs create-service --cluster test-cluster \
  --service-name hazelcast-embedded \
  --task-definition hazelcast-embedded \
  --launch-type=FARGATE \
  --network-configuration "awsvpcConfiguration={subnets=["${SUBNET_ID}"],securityGroups=["${SECURITY_GROUP_ID}"],assignPublicIp=ENABLED}" \
  --desired-count 3

You can check that the related tasks were created. If you don’t see any tasks, then wait a moment and check the tasks again.

aws ecs list-tasks --cluster test-cluster --service hazelcast-embedded
{
    "taskArns": [
        "arn:aws:ecs:eu-central-1:665466731577:task/2154b675-df19-459e-a95e-8466f5d4bb59",
        "arn:aws:ecs:eu-central-1:665466731577:task/7c3adb1d-6850-4f86-9eb1-7eed5bc75235",
        "arn:aws:ecs:eu-central-1:665466731577:task/99de6977-949d-489c-9c5a-44c763edfaaf"
    ]
}

You can also examine the logs of one of the tasks and you should see that Hazelcast cluster was successfully formed.

aws logs get-log-events \
  --log-group-name /ecs/hazelcast \
  --log-stream-name ecs/hazelcast-embedded/2154b675-df19-459e-a95e-8466f5d4bb59 \
  --output text --query 'events[*].[message]'

...
Members {size:3, ver:3} [
        Member [10.0.1.4]:5701 - dae44d3d-882b-4f3a-aff3-09721b737276
        Member [10.0.1.6]:5701 - 3874314e-30f9-4f10-9f7d-11a0a32dc16b
        Member [10.0.1.220]:5701 - 9009d19a-63b1-4e90-b4de-48e877bb7086 this
]
...

Test the Application

To test that the application works correctly, you can check the public IPs of your tasks and make REST calls to the application replicas.

aws ecs list-tasks --cluster test-cluster --service hazelcast-embedded
{
    "taskArns": [
        "arn:aws:ecs:eu-central-1:665466731577:task/2154b675-df19-459e-a95e-8466f5d4bb59",
        "arn:aws:ecs:eu-central-1:665466731577:task/7c3adb1d-6850-4f86-9eb1-7eed5bc75235",
        "arn:aws:ecs:eu-central-1:665466731577:task/99de6977-949d-489c-9c5a-44c763edfaaf"
    ]
}

aws ecs describe-tasks --cluster test-cluster \
  --tasks 2154b675-df19-459e-a95e-8466f5d4bb59 \
          7c3adb1d-6850-4f86-9eb1-7eed5bc75235 \
          99de6977-949d-489c-9c5a-44c763edfaaf \
  --query "tasks[*].attachments[*].details"
[
    [
        [
            {
                "name": "subnetId",
                "value": "subnet-0f042c997bad8e2b9"
            },
            {
                "name": "networkInterfaceId",
                "value": "eni-0464651cd2cdf1b45"
            },
            {
                "name": "macAddress",
                "value": "02:e3:4b:c3:53:04"
            },
            {
                "name": "privateIPv4Address",
                "value": "10.0.1.220"
            }
        ]
    ],
    [
        [
            {
                "name": "subnetId",
                "value": "subnet-0f042c997bad8e2b9"
            },
            {
                "name": "networkInterfaceId",
                "value": "eni-061b0e2414cfb1440"
            },
            {
                "name": "macAddress",
                "value": "02:e6:04:12:00:70"
            },
            {
                "name": "privateIPv4Address",
                "value": "10.0.1.4"
            }
        ]
    ],
    [
        [
            {
                "name": "subnetId",
                "value": "subnet-0f042c997bad8e2b9"
            },
            {
                "name": "networkInterfaceId",
                "value": "eni-0050d027099aeb812"
            },
            {
                "name": "macAddress",
                "value": "02:47:51:40:b2:72"
            },
            {
                "name": "privateIPv4Address",
                "value": "10.0.1.6"
            }
        ]
    ]
]

aws ec2 describe-network-interfaces \
  --network-interface-ids eni-0050d027099aeb812 \
                          eni-061b0e2414cfb1440 \
                          eni-0464651cd2cdf1b45 \
  --query "NetworkInterfaces[*].Association.PublicIp"
[
    "3.123.22.224",
    "18.185.12.209",
    "18.184.248.123"
]

Knowing the public IPs of tasks, we can make calls to the application replicas.

curl --data "key=key1&value=hazelcast" "3.123.22.224:8080/put"
{"value":"hazelcast"}

curl "18.185.12.209:8080/get?key=key1"
{"value":"hazelcast"}

Tear Down the Deployment

To delete the service, execute the following commands.

aws ecs update-service \
  --cluster test-cluster \
  --service hazelcast-embedded \
  --desired-count 0
aws ecs delete-service \
  --cluster test-cluster \
  --service hazelcast-embedded

To delete all other resources we created in the tutorial, execute the following commands.

aws ecs deregister-task-definition --task-definition hazelcast-embedded:1
aws iam detach-role-policy \
  --role-name ecs-execution-role \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
aws iam delete-role --role-name ecs-execution-role
aws logs delete-log-group --log-group-name /ecs/hazelcast
aws iam detach-role-policy --role-name hazelcast-ecs-role --policy-arn ${POLICY_ARN}
aws iam delete-role --role-name hazelcast-ecs-role
aws iam delete-policy --policy-arn ${POLICY_ARN}
aws ec2 delete-security-group --group-id ${SECURITY_GROUP_ID}
aws ecs delete-cluster --cluster test-cluster