Deploying a Cluster on Amazon AWS

6.0-SNAPSHOT 5.5 5.4 5.3 5.2 5.1 5.0

Deploy Hazelcast clusters on Amazon AWS EC2 and AWS ECS/Fargate environments and allow them to discover each other automatically.

Before you Begin

Before deploying Hazelcast on AWS, you must have the following:

  • Linux Kernel 3.19+

    TCP connections may get stuck when used with older Kernel versions, resulting in undefined timeouts.

Discovering Members Automatically

To make it easier to set up clusters on Amazon AWS, Hazelcast allows members to discover each other automatically, using discovery strategies.

When a member starts on AWS, it fetches a list of all running instances filtered by the member’s AWS configuration settings. Then, each instance is checked one-by-one with its IP and each of the ports defined in the hz-port property (see the example configurations in this section for the property’s description). When a member is discovered under IP:PORT, then it joins the cluster.

Choose from one of the following environments to configure a discovery strategy:

Configuring Hazelcast for EC2

You can configure both Hazelcast members and Hazelcast clients to automatically discover clusters deployed in Amazon EC2.

EC2 Hazelcast Member Discovery

The following is an example AWS EC2 configuration snippet in a member, to enable automatic member discoveries in EC2. Each member of your cluster should have the same configuration.

  • XML

  • YAML

  • Java

<hazelcast>
  <network>
    <join>
      <multicast enabled="false"/> (1)
      <aws enabled="true"> (2)
        <tag-key>my-ec2-instance-tag-key</tag-key> (3)
        <tag-value>my-ec2-instance-tag-value</tag-value> (4)
      </aws>
    </join>
  </network>
</hazelcast>
hazelcast:
  network:
    join:
      multicast:
        enabled: false (1)
      aws:
        enabled: true (2)
        tag-key: my-ec2-instance-tag-key (3)
        tag-value: my-ec2-instance-tag-value (4)
config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false); (1)
config.getNetworkConfig().getJoin().getAwsConfig().setEnabled(true) (2)
      .setProperty("tag-key", "my-ec2-instance-tag-key") (3)
      .setProperty("tag-value", "my-ec2-instance-tag-value"); (4)
1 Multicast discovery strategy must be disabled. It is mandatory to set.
2 AWS discovery strategy must be enabled. It is mandatory to set.
3 Filter to look only for EC2 Instances with the given tag-key; comma-separated multiple values are supported, e.g., KeyA,KeyB; comma-separated values behave as AND conditions. It is mandatory to set.
4 Filter to look only for EC2 Instances with the given tag-value; comma-separated multiple values are supported, e.g., ValueA,ValueB; comma-separated values behave as AND conditions. It is mandatory to set.

The following are the optional properties to be configured:

  • access-key, secret-key: Access and secret keys of your AWS account; if not set, iam-role is used.

  • iam-role: IAM Role attached to the EC2 instance used to fetch credentials (if access-key/secret-key are not specified); if not set, the default IAM Role attached to the EC2 instance is used.

  • region: Region where the Hazelcast members are running; default is the current region.

  • host-header: ec2, ecs, or the URL of a EC2/ECS API endpoint; automatically detected by default.

  • security-group-name: Filter to look only for the EC2 instances with the given security group.

  • connection-timeout-seconds, read-timeout-seconds: Connection and read timeouts when making a call to the AWS API; its default is 10 seconds.

  • connection-retries: Number of retries while connecting to AWS API; its default is 3.

  • hz-port: Range of ports where the discovery strategy looks for Hazelcast members; its default is 5701-5708.

Note that if you don’t specify any of the optional properties, then Hazelcast uses the IAM Role assigned to the EC2 Instance to form a cluster from all Hazelcast members running in the same region.

Client Discovery of EC2 Hazelcast Cluster

Hazelcast clients can automatically discover the Hazelcast cluster running in EC2 environment. This section is for the Hazelcast Java clients. See C++ client and Go client documentations to learn how to configure Hazelcast C++ and Go clients to automatically discover members running in EC2 environment.

  • Client running in EC2:

To discover the Hazelcast cluster running on EC2 using the client running on EC2, you can use the same discovery parameters as mentioned in the previous section.

The following is an example configuration snippet in a Java client. Each client should have the same configuration.

  • XML

  • YAML

  • Java

<hazelcast-client>
  <network>
    <aws enabled="true">
      <tag-key>my-ec2-instance-tag-key</tag-key>
      <tag-value>my-ec2-instance-tag-value</tag-value>
    </aws>
  </network>
</hazelcast>
hazelcast-client:
  network:
    aws:
      enabled: true
      tag-key: my-ec2-instance-tag-key
      tag-value: my-ec2-instance-tag-value
config.getNetworkConfig().getAwsConfig().setEnabled(true)
      .setProperty("tag-key", "my-ec2-instance-tag-key")
      .setProperty("tag-value", "my-ec2-instance-tag-value");

  • Client running in ECS:

To discover the Hazelcast cluster running on EC2 using the client running on ECS, you need to specify the needed EC2 related properties (iam-role, security-group-name) and to set the ec2:DescribeInstances permission. If none of the ECS or EC2 related properties are specified, the AWS discovery strategy tries to discover the ECS members. If none is found, then it tries to discover EC2 members.

The following is an example configuration snippet in a Java client. Each client should have the same configuration.

  • XML

  • YAML

  • Java

<hazelcast-client>
  <network>
    <aws enabled="true">
      <iam-role>my-iam-role</iam-role>
      <security-group-name>my-security-group-name</security-group-name>
      <tag-key>my-ec2-instance-tag-key</tag-key>
      <tag-value>my-ec2-instance-tag-value</tag-value>
    </aws>
  </network>
</hazelcast>
hazelcast-client:
  network:
    aws:
      enabled: true
      iam-role: my-iam-role
      security-group-name: my-security-group-name
      tag-key: my-ec2-instance-tag-key
      tag-value: my-ec2-instance-tag-value
config.getNetworkConfig().getAwsConfig().setEnabled(true)
      .setProperty("iam-role", "my-iam-role")
      .setProperty("security-group-name", "my-security-group-name")
      .setProperty("tag-key", "my-ec2-instance-tag-key")
      .setProperty("tag-value", "my-ec2-instance-tag-value");

  • Client running outside AWS:

For the Java clients running outside AWS, you always need to specify the following parameters:

  • access-key, secret-key - IAM role cannot be used from outside AWS.

  • region - it cannot be detected automatically.

  • use-public-ip - must be set to true.

Note also that your EC2 instances must have a public IP assigned.

The following is an example configuration snippet.

  • XML

  • YAML

  • Java

<hazelcast-client>
  <network>
    <aws enabled="true">
      <access-key>my-access-key</access-key>
      <secret-key>my-secret-key</secret-key>
      <region>us-west-1</region>
      <tag-key>my-ec2-instance-tag-key</tag-key>
      <tag-value>my-ec2-instance-tag-value</tag-value>
      <use-public-ip>true</use-public-ip>
    </aws>
  </network>
</hazelcast-client>
hazelcast-client:
  network:
    aws:
      enabled: true
      access-key: my-access-key
      secret-key: my-secret-key
      region: us-west-1
      tag-key: my-ec2-instance-tag-key
      tag-value: my-ec2-instance-tag-value
      use-public-ip: true
clientConfig.getNetworkConfig().getAwsConfig()
      .setEnabled(true)
      .setProperty("access-key", "my-access-key")
      .setProperty("secret-key", "my-secret-key")
      .setProperty("region", "us-west-1")
      .setProperty("tag-key", "my-ec2-instance-tag-key")
      .setProperty("tag-value", "my-ec2-instance-tag-value")
      .setProperty("use-public-ip", "true");

ECS/Fargate Configuration

The plugin works both for Hazelcast Member Discovery (forming Hazelcast cluster) and Hazelcast Client Discovery.

For the detailed description, check out Hazelcast Guides: Getting Started with Embedded Hazelcast on ECS.

ECS Hazelcast Member Discovery

Make sure that your IAM Task Role has the following permissions:

  • ecs:ListTasks

  • ecs:DescribeTasks

  • ec2:DescribeNetworkInterfaces (needed only if task have public IPs)

Then, you can configure Hazelcast in one of the following manners. Please note that 10.0.. value depends on your VPC CIDR block definition.

  • XML

  • YAML

  • Java

<hazelcast>
  <network>
    <join>
      <multicast enabled="false"/>
      <aws enabled="true" />
    </join>
    <interfaces enabled="true">
      <interface>10.0.*.*</interface>
    </interfaces>
  </network>
</hazelcast>
hazelcast:
  network:
    join:
      multicast:
        enabled: false
      aws:
        enabled: true
    interfaces:
      enabled: true
      interfaces:
        - 10.0.*.*
config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
config.getNetworkConfig().getJoin().getAwsConfig().setEnabled(true);
config.getNetworkConfig().getInterfaces().setEnabled(true).addInterface("10.0.*.*");

The following optional properties can be configured:

  • access-key, secret-key: access and secret keys of AWS your account; if not set, IAM Task Role is used

  • region: region where Hazelcast members are running; default is the current region

  • cluster: ECS cluster short name or ARN; default is the current cluster

  • family: filter to look only for ECS tasks with the given family name; mutually exclusive with service-name

  • service-name: filter to look only for ECS tasks from the given service; mutually exclusive with family

  • tag-key, tag-value: filter to look only for ECS Tasks with the given tag-key/tag-value; comma-separated multiple values are supported, e.g., KeyA,KeyB; comma-separated values behave as AND conditions

  • host-header: ecs or the URL of an ECS API endpoint; automatically detected by default

  • connection-timeout-seconds, read-timeout-seconds: connection and read timeouts when making a call to AWS API; default to 10

  • connection-retries: number of retries while connecting to AWS API; default to 3

  • hz-port: a range of ports where the plugin looks for Hazelcast members; default is 5701-5708

If you don’t specify any of the properties, then the plugin discovers all Hazelcast members running in the current ECS cluster.
ECS discovery can use IAM Role assigned to the ECS Task instead of using access-key, secret-key.

Client Discovery of ECS Hazelcast Cluster

Hazelcast Client can automatically discover the Hazelcast cluster running in ECS environment.

  • Client running in ECS:

To discover the Hazelcast Cluster running on ECS using the client running on the ECS, use the same discovery parameters as mentioned above.

Following are example declarative and programmatic configuration snippets.

  • XML

  • YAML

  • Java

<hazelcast-client>
  <network>
    <aws enabled="true">
    </aws>
  </network>
</hazelcast-client>
hazelcast-client:
  network:
    aws:
      enabled: true
clientConfig.getNetworkConfig().getAwsConfig()
      .setEnabled(true);

  • Client running in EC2:

Hazelcast Client on EC2 instances can discover the members running on ECS. For this you need to specify the needed ECS related properties (cluster, family, service-name) and make sure to set the ec2:DescribeInstances permission. If none of the ECS or EC2 related properties are specified, the AWS discovery tries to discover the EC2 members. If none is found, then it tries to discover ECS members.

Following are example declarative and programmatic configuration snippets.

  • XML

  • YAML

  • Java

<hazelcast-client>
  <network>
    <aws enabled="true">
      <cluster>my-ecs-cluster</cluster>
      <service-name>my-ecs-service</service-name>
    </aws>
  </network>
</hazelcast-client>
hazelcast-client:
  network:
    aws:
      enabled: true
      cluster: my-ecs-cluster
      service-name: my-ecs-service
clientConfig.getNetworkConfig().getAwsConfig()
      .setEnabled(true)
      .setProperty("cluster", "my-ecs-cluster")
      .setProperty("service-name", "my-ecs-service");

  • Client running outside AWS:

If Hazelcast Client runs outside AWS, then you need to always specify the following parameters:

  • access-key, secret-key - IAM role cannot be used from outside AWS

  • region - it cannot be detected automatically

  • cluster - it cannot be detected automatically

  • use-public-ip - must be set to true

Your ECS Tasks must have public IPs assigned and your IAM Task Role must have ec2:DescribeNetworkInterfaces permission.

Following are example declarative and programmatic configuration snippets.

  • XML

  • YAML

  • Java

<hazelcast-client>
  <network>
    <aws enabled="true">
      <access-key>my-access-key</access-key>
      <secret-key>my-secret-key</secret-key>
      <region>eu-central-1</region>
      <cluster>my-cluster</cluster>
      <use-public-ip>true</use-public-ip>
    </aws>
  </network>
</hazelcast-client>
hazelcast-client:
  network:
    aws:
      enabled: true
      access-key: my-access-key
      secret-key: my-secret-key
      region: eu-central-1
      cluster: my-cluster
      use-public-ip: true
clientConfig.getNetworkConfig().getAwsConfig()
      .setEnabled(true)
      .setProperty("access-key", "my-access-key")
      .setProperty("secret-key", "my-secret-key")
      .setProperty("region", "eu-central-1")
      .setProperty("cluster", "my-cluster")
      .setProperty("use-public-ip", "true");

  • Client running on Lambda:

Hazelcast Clients launched on Lambda functions can discover the members running on ECS in the same way as connecting outside of AWS. Since the Lambda function does not provide the Metadata API at execution time, the required properties cannot be inferred by the Discovery Plugin. That is why you must declare all the necessary properties in the Hazelcast Client configuration.

  • access-key, secret-key - IAM role cannot be used in Lambda function

  • region - it cannot be detected automatically

  • cluster - it cannot be detected automatically

  • use-public-ip - must be set to true

The access and secret keys generated for a session are not allowed for use in Discovery Plugins. Instead, you should use the Access and Secret keys generated from the persistent IAM Role.

The following example demonstrates how to establish a connection to a Hazelcast Cluster in a Lambda function.

That is the hazelcast-client.yaml file located within the JAR file of the Lambda function.

hazelcast-client:
  network:
    aws:
      enabled: true
      access-key: my-access-key
      secret-key: my-secret-key
      region: ca-central-1
      cluster: my-cluster
      hz-port: 5701
      use-public-ip: true

This is the simple Lambda handler in Java, which creates a Hazelcast Client by loading the given configuration file above.

// Handler value: example.HazelcastClientHandler
public class HazelcastClientHandler implements RequestHandler<Map<String, String>, Void> {

    @Override
    public Void handleRequest(Map<String, String> input, Context context) {
        var clientConfig = ClientConfig.load();
        var hz = HazelcastClient.newHazelcastClient(clientConfig);

        // ---snip---

        return null;
    }
}

ECS Environment with EC2 Discovery

If you use ECS on EC2 instances (not Fargate), you may also set up your ECS Tasks to use host network mode and then use EC2 discovery mode instead of ECS. In that case, your Hazelcast configuration would look as follows.

hazelcast:
  network:
    join:
      multicast:
        enabled: false
      aws:
        enabled: true
        host-header: ec2
    interfaces:
      enabled: true
      interfaces:
        - 10.0.*.*

All other parameters can be used exactly the same as described in the EC2-related section.

Enabling IMDSv2 for EC2 Instances

Hazelcast automatically handles the usage of IMDSv2 method to access instance metadata, provided that your EC2 instance is configured accordingly:

  1. Enable IMDSv2 on the instances you have; see the AWS documentation on how to do that.

  2. Start Hazelcast with AWS auto-discovery enabled as explained in EC2 Hazelcast Member Discovery.

AWS Elastic Beanstalk

While deploying your application into the Java Platform, please make sure your Elastic Beanstalk Environment Configuration satisfies the following requirements:

  • EC2 security groups contain a group which allows the port 5701

  • IAM instance profile contains IAM role which has ec2:DescribeInstances permission (or your Hazelcast configuration contains access-key and secret-key)

  • Deployment policy is Rolling (instead of the default All at once which may cause the whole Hazelcast members to restart at the same time and therefore lose data)

Preventing Data Loss

By default, Hazelcast distributes partition replicas (backups) randomly and equally among cluster members. However, this is not safe in terms of high availability when a partition and its replicas are stored on the same rack, using the same network, or power source. To deal with that, Hazelcast offers logical partition grouping, so that a partition itself and its backups would not be stored within the same group. This way Hazelcast guarantees that a possible failure affecting more than one member at a time will not cause data loss. For more details about partition groups, see Partition Group Configuration.

In addition to two built-in grouping options ZONE_AWARE and PLACEMENT_AWARE, you can customize the formation of these groups based on the network interfaces of members. For more details about custom groups, see Custom Partition Groups.

Multi-Zone Deployments

If ZONE_AWARE partition group is enabled, the backups of a partition are always stored in a different availability zone. Hazelcast supports the ZONE_AWARE feature for both EC2 and ECS.

When using the ZONE_AWARE partition grouping, a cluster spanning multiple Availability Zones (AZ) should have an equal number of members in each AZ. Otherwise, it will result in uneven partition distribution among the members.

  • XML

  • YAML

  • Java

<partition-group enabled="true" group-type="ZONE_AWARE" />
hazelcast:
  partition-group:
    enabled: true
    group-type: ZONE_AWARE
config.getPartitionGroupConfig()
    .setEnabled(true)
    .setGroupType(MemberGroupType.ZONE_AWARE);

Partition Placement Group Deployments

AWS Partition Placement Group (PPG) ensures low latency between the instances in the same partition of a placement group and also provides availability since no two partitions share the same underlying hardware. As long as the partitions of a PPG contain an equal number of instances, it will be good practice for Hazelcast clusters formed within a single zone.

If EC2 instances belong to a PPG and PLACEMENT_AWARE partition group is enabled, then Hazelcast members will be grouped by the partitions of the PPG. For instance, the Hazelcast members in the first partition of a PPG named ppg will belong to the partition group of ppg-1, and those in the second partition will belong to ppg-2 and so on. Furthermore, these groups will be specific to each availability zone. That is, they are formed with zone names as well: us-east-1-ppg-1, us-east-2-ppg-1, and the like. However, if a Hazelcast cluster spans multiple availability zones then you should consider using ZONE_AWARE.

Cluster Placement Group Deployments

AWS Cluster Placement Group (CPG) ensures low latency by packing instances close together inside an availability zone. If you favor latency over availability, then CPG will serve your purpose.

In the case of CPG, using PLACEMENT_AWARE has no effect, so can use the default Hazelcast partition group strategy.

Spread Placement Group Deployments

AWS Spread Placement Groups (SPG) ensures high availability in a single zone by placing each instance in a group on a distinct rack. It provides better latency than multi-zone deployment, but worse than Cluster Placement Group. SPG is limited to 7 instances, so if you need a larger Hazelcast cluster within a single zone, you should use PPG instead.

In the case of SPG, using PLACEMENT_AWARE has no effect, so can use the default Hazelcast partition group strategy.

  • XML

  • YAML

  • Java

<partition-group enabled="true" group-type="PLACEMENT_AWARE" />
hazelcast:
  partition-group:
    enabled: true
    group-type: PLACEMENT_AWARE
config.getPartitionGroupConfig()
    .setEnabled(true)
    .setGroupType(MemberGroupType.PLACEMENT_AWARE);

Autoscaling

Hazelcast is prepared to work correctly within the autoscaling environments. Note that there are two specific requirements to prevent Hazelcast from losing data:

  • The number of members in a cluster must not change by more than one at a time

  • When a member is launched or terminated, the cluster must be in a safe state

Read about details in the blog post: AWS Auto Scaling with Hazelcast.

lifebuoy

Help and support