Capacity Planning

A Hazelcast cluster is a network of members that run Hazelcast software. The cluster performance depends on multiple factors, including data size, number of backups, queries, and which features are used. Therefore, planning the cluster remains a complex task that requires knowledge of Hazelcast’s architecture and concepts. Here, we introduce some basic guidelines that help to properly size a cluster.

We recommend always benchmarking your setup before deploying it to production. We also recommend that benchmarking systems resemble the production system as much as possible to avoid unexpected results. We provide a bechmarking example that you can use as a starting point.

Hazelcast clusters will run both data processing and data storage workloads, so planning for both types of workload is important.

In order to correctly size the cluster for your use case, answers to as many of the following questions as possible are necessary:

Once you know the size, access patterns, throughput, latency, and fault tolerance requirements of your application, you can use the following guidelines to help you determine the size of your cluster. Also, if using WAN Replication, the WAN Replication queue sizes need to be taken into consideration for sizing.

Even a single Hazelcast instance running on a recommended server can host hundreds of jobs at a time. A clustered setup improves the performance (throughput and latency) of hosted jobs and increases resilience.

To make fault tolerance possible, your cluster must have at least three members. Generally, you need n+1 cluster members to tolerate n simultaneous member failures.

If you use the Jet stream processing engine, it can use hundreds of CPU cores efficiently by exploiting data and task parallelism. When you add more members to the cluster, the capacity for CPU-bound computation rises. You can achieve better performance by distributing the data partitions across the cluster to process them in parallel.

Benchmark your jobs in a clustered setup to see the differences in performance, as in the sizing example.

Once you know the size of your working set of data, you can start sizing your memory requirements.

When speaking of "data" in Hazelcast, this includes both active data and backup data for high availability. The total memory footprint will be the size of your active data plus the size of your backup data and a small memory overhead of Hazelcast internals.

If your fault tolerance allows for just a single backup, then each member of the Hazelcast cluster will contain a 1:1 ratio of active data to backup data for a total memory footprint of two times the active data.

If your fault tolerance requires two backups, then that ratio climbs to 1:2 active to backup data for a total memory footprint of three times your active data set.

If you use only heap memory, each Hazelcast member with a 4 GB heap should accommodate a maximum of 3.5 GB of total data (active and backup). If you use the High-Density Data Store, up to 75% of your physical memory footprint may be used for active and backup data, with headroom of 25% for normal fragmentation. In both cases, however, the best practice is to keep some memory headroom available to handle any member failure or explicit member shutdown. When a member leaves the cluster, the data previously owned by the newly offline member will be redistributed across the remaining members. For this reason, we recommend that you plan to use only 60% of available memory, with 40% headroom to handle member failure or shutdown.

If you use High-Density Memory Store, Hazelcast automatically assigns a percentage of available off-heap memory to the internal memory manager. Since allocation happens lazily, if you want to be informed about how much off-heap memory is being used by the memory manager, you can set the metadata-space-percentage property to a percentage value and Hazelcast will log warnings when that value is reached. By default, this property is set to 12.5%.

If you’re using jobs, all operational data must fit within available memory. This design leads to a predictable performance but requires enough RAM not to run out. Estimate the memory requirements and plan with a headroom of 25% for normal memory fragmentation. For fault-tolerant operations, we recommend reserving extra memory to survive any failures. See sizing for failures.

If your computation is memory-bound, consider:

Memory consumption is affected by:

Hazelcast offers lite members to prevent memory usage on these members. Lite members do not own any partitions, but they can access partitions that are owned by other members in the cluster. If there is no specific advantage to using non-homogeneous cluster members, we do not recommend using lite members as they increase network calls and thus increase the latency. See Enabling Lite Members for information about lite members.

Hazelcast’s default partition count is 271. This is a good choice for clusters of up to 50 members and ~25–30 GB of data. Up to this threshold, partitions are small enough that any rebalancing of the partition map when members join or leave the cluster doesn’t disturb the smooth operation of the cluster. With larger clusters and/or bigger data sets, a larger partition count helps to maintain an efficient rebalancing of data across members.

An optimum partition size is between 50MB – 100MB. Therefore, when designing the cluster, determine the size of the data that will be distributed across all members, and then determine the number of partitions such that no partition size exceeds 100MB. If the default count of 271 results in heavily loaded partitions, increase the partition count to the point where data load per-partition is under 100MB. Remember to factor in headroom for projected data growth.

To change the partition count, use the system property hazelcast.partition.count.

Hazelcast performs scaling tests for each version of the software. Based on this testing we specify some scaling maximums. These are defined for each version of the software. We recommend staying below these numbers. Please contact Hazelcast if you plan to use higher limits.

Clients that use the ALL_MEMBERS cluster routing mode maintain a connection to each member. Clients that use the SINGLE_MEMBER cluster routing mode have a single connection to the entire cluster. You can also choose to connect to a partition group which provides direct connections to members in that group with those members acting as a gateway to the other members in the cluster. You can find more information about the cluster routing modes here: Java Cluster Routing Modes.

Hazelcast clusters are elastic to deal with failures and performance spikes.

When a cluster member fails, this reduces available resources and increases stress on the remaining members until recovery. The data previously owned by the failed member gets distributed among the surviving ones. The cluster must catch up with the data that has accumulated while it was adapting to the new size, and it must keep up with the head of the stream without the CPU capacity of the lost member.

To tolerate the failure of one member, we recommend sizing your cluster so it can operate well with n-1 members.

Another approach to improve fault-tolerance is to separate the concerns of data storage and computation into two separate clusters. As an example, you could use one cluster for Hazelcast IMaps and their event journals and another one for running Hazelcast jobs. This way a single failure doesn’t simultaneously hurt both the storage and the computation capacity.

The jobs running in one cluster share the resources to maximize hardware utilization. This is efficient for setups without the risk of noisy neighbors such as:

For stronger resource isolation (multi-tenant environments, strict SLAs), consider starting multiple smaller clusters with resources allocated at the OS level or using a resource manager such as Kubernetes.

Consider the capacity of data sources and sinks when planning the Hazelcast cluster.

Each Hazelcast job participates in a larger data pipeline: it continuously reads the data from the sources and writes the results to the sinks. The capacity of all components of the data pipeline must be balanced to avoid bottlenecks.

If a data sink is slow, Hazelcast applies backpressure all the way to the source, slowing down data consumption. The data sources should be designed to participate by reducing the pace of data production or by buffering the data.

On the other hand, if the data source is slow, i.e., it can’t produce or transmit the data fast enough, adding more resources to the Hazelcast cluster won’t bring any performance benefits.

Test your setup on a dataset that represents the characteristics of the production data, notably:

Hazelcast splits the data across the cluster to process it in parallel. It is designed to perform well under the assumption of balanced partitions. Imbalanced partitions may create a "hot spot" (a segment of data accessed far more often than other data) in your cluster. Factors that affect partitioning are the data source and the grouping keys used in the Hazelcast application.

A frequent source of a partition imbalance are special cases. For example, in a payment processing application, there might be a small number of accounts with very high activity. Imagine a retail company account with thousands of payments per minute vs. personal accounts with just a few payments in a day. Using account as a grouping key leads to imbalanced partitions. Consider such special cases when designing your pipelines and the test datasets.

See the following caching and streaming use cases to see sample benchmarking and sizing exercises.

Based on an extensive testing, we extracted some points of advice on how to choose the right JDK/GC (Garbage Collection) combination and how to tune your setup to the workload of your Hazelcast data pipeline.