Monitoring

Management Center allows you to execute scripts that can automate interactions with the cluster.

By default, scripting is disabled for security. Scripting engines give code access to the underlying system on the members (files and other resources) and run with the same permissions as the current user. You can enable scripting in the member configuration file:

Note that the JSR 223 API is used in Hazelcast to support scripting.

Management Center allows you to execute client operations using built-in Hazelcast Command-Line Client (CLC) in the user interface. This console is useful for testing and development purposes. You can enable the console in the member configuration file:

See the Management Center documentation for more information.

Management Center allows you to access contents of Hazelcast data structures (for instance map entries) via SQL Browser or Map Browser. It may be useful to restrict data access for Management Center if sensitive financial or personal information is stored in the cluster. Management Center can’t access the data if at least one member has the data access disabled. You can disable data access for Management Center in the member configuration file:

By default, any instance of Management Center can connect to a cluster as long as it can be authenticated. To restrict access only to trusted instances of Management Center, you can define the trusted IP addresses in the trusted-interfaces configuration setting. This setting supports wildcards (*) and ranges (-).

Here is an example of programmatic member-side Java configuration:

The equivalent declarative configuration is as follows:

You can use this configuration to enable the instance tracking feature, specify the file name and the pattern for the file contents. By default, the feature is disabled, the file name is Hazelcast.process in the OS temporary directory as returned by System.getProperty("java.io.tmpdir") and the file contents are JSON-formatted key-value pairs of all available metadata.

The client configuration is analogous and only differs in the name of the outer configuration block or configuration instance containing the instance tracking configuration.

Here is an example when running a client instance:

Here is an example when running a member instance in the "server" mode:

And here is an example when running a member instance in the "embedded" mode:

You can specify a custom format by using a predefined set of available metadata keys an example of which is shown below:

This should produce a file with the following content:

As you can see, once we encounter a broken placeholder, all subsequent placeholders are ignored. On the other hand, missing placeholders are skipped and subsequent placeholders are resolved.

The currently valid metadata placeholders and their possible values are as follows:

The possible values for the product placeholder: Hazelcast, Hazelcast Enterprise, Hazelcast Client, Hazelcast Client Enterprise.

The possible values for the mode placeholder:

There are a wide range of metrics and statistics provided by Hazelcast:

See the full list of Hazelcast metrics in List of Metrics appendix.

User-defined metrics are actually a subset of job metrics. What distinguishes them from regular job-specific metrics is exactly what their name implies: they are not built-in, but defined when processing pipelines are written.

Since user-defined metrics are also job metrics, they will have all the tags job metrics have. They also have an extra tag, called user which is of type boolean and is set to true.

Let’s see how one would go about defining such metrics. For example if you would like to monitor your filtering step you could write code like this:

User-defined metrics can be used anywhere in pipeline definitions where custom code can be added. This means (just to name the most important ones): filtering, mapping and flat-mapping functions, various constituent functions of aggregations (accumulate, create, combine, deduct, export & finish), key extraction function when grouping, in custom batch sources, custom stream sources, custom sinks, processors and so on.

The following are the tools and interfaces to expose the metrics to the outside world:

The metrics collection is enabled by default. You can configure the metrics system declaratively or programmatically. The following is an example declarative configuration with the default values, on the member side:

Note that all the metrics configuration values can be overridden with system properties. The properties are listed below:

The client configuration is very similar, it just lacks the Management Center configuration block (management-center configuration element), as shown below. This is because the clients are not connected to Management Center and the client metrics are sent to Management Center through a member to which the client is connected.

Similarly to the member configuration, the client metrics configuration can be overridden with the following system properties:

Note that the metric names may change between MINOR versions but not between PATCH versions.

The metrics system is designed with care to make the least possible impact on the performance of the cluster. Since the metrics collection takes place periodically with a few seconds frequency, the main focus is keeping allocation rates and memory footprint at minimum. Therefore, the blobs that store the metrics for Management Center are stored in the memory in a compressed format. The measurements, that use multiple IMaps to scale up the number of metrics, show that one blob occupies only a few KBs and it grows above 10KB only if there are more than 1000 IMaps.

The allocation rate of a metric collection cycle is also low. With both Management Center and JMX consumers enabled, the allocation rate with 100 IMaps is below 256KB per cycle, and it grows above 1MB with 1000 IMaps. This means that metrics collection does not increase the frequency of the garbage collection (GC) noticeably.

While the metrics collection is considered GC friendly, it should be noted that the blobs are not recycled: configuring the retention time should be done with taking the frequency of the GC into account to prevent the blobs from getting promoted into the tenured region of the heap that in the end contributes to major GCs after time.

To get local map statistics, use the getLocalMapStats() method from the IMap interface. This method returns a LocalMapStats object that holds local map statistics.

Below is an example code where the getLocalMapStats() method and the getOwnedEntryCount() method get the number of entries owned by this member.

The following are some of the metrics that you can access via the LocalMapStats object:

See the LocalMapStats Javadoc to see all the metrics.

To access map index statistics, if you are using indexes to speed up map queries, use the getIndexStats() method of the LocalMapStats interface returned by IMap.getLocalMapStats().

Below is an example where the getIndexStats() method is used to examine an average selectivity of index hits:

The following are some of the metrics that you can obtain via the LocalIndexStats interface:

See the LocalIndexStats Javadoc to see all the metrics.

To compute an aggregated value of getAverageHitSelectivity() for all cluster members, you can use a simple averaging computation as shown below:

In this computation, s(i) is an average hit selectivity on the member i and n is the total number of cluster members.

A more advanced solution is to compute a weighted average as shown below:

Here, s(i) is an average hit selectivity on the member i, h(i) is a hit count (getHitCount()) on the member i and n is the total number of cluster members. This more advanced solution may produce more precise results in unstable dynamic clusters where new members do not have enough statistics accumulated. The same technique may be applied to the getAverageHitLatency() metric.

Accuracy and reliability notes:

To get Near Cache statistics, use the getNearCacheStats() method from the LocalMapStats object. This method returns a NearCacheStats object that holds Near Cache statistics.

Below is an example code where the getNearCacheStats() method and the getRatio method from NearCacheStats get a Near Cache hit/miss ratio.

The following are some of the metrics that you can access via the NearCacheStats object (applies to both client and member Near Caches):

See the NearCacheStats Javadoc to see all the metrics.

To get MultiMap statistics, use the getLocalMultiMapStats() method from the MultiMap interface. This method returns a LocalMultiMapStats object that holds local MultiMap statistics.

Below is an example code where the getLocalMultiMapStats() method and the getLastUpdateTime method from LocalMultiMapStats get the last update time.

The following are some of the metrics that you can access via the LocalMultiMapStats object:

See the LocalMultiMapStats Javadoc to see all the metrics.

To get local queue statistics, use the getLocalQueueStats() method from the IQueue interface. This method returns a LocalQueueStats object that holds local queue statistics.

Below is an example code where the getLocalQueueStats() method and the getAverageAge method from LocalQueueStats get the average age of items.

The following are some of the metrics that you can access via the `LocalQueueStats ` object:

See the LocalQueueStats Javadoc to see all the metrics.

To get local topic statistics, use the getLocalTopicStats() method from the ITopic interface. This method returns a LocalTopicStats object that holds local topic statistics.

Below is an example code where the getLocalTopicStats() method and the getPublishOperationCount method from LocalTopicStats get the number of publish operations.

The following are the metrics that you can access via the `LocalTopicStats ` object:

See the LocalTopicStats Javadoc to see all the metrics.

To get local executor statistics, use the getLocalExecutorStats() method from the IExecutorService interface. This method returns a LocalExecutorStats object that holds local executor statistics.

Below is an example code where the getLocalExecutorStats() method and the getCompletedTaskCount method from LocalExecutorStats get the number of completed operations of the executor service.

The following are some of the metrics that you can access via the `LocalExecutorStats ` object:

See the LocalExecutorStats Javadoc to see all the metrics.

To use the health check endpoint and script, enable either of the following configuration options:

You can use Hazelcast’s HTTP-based health check implementation to get basic information about the cluster and member on which it is launched.

To use the HTTP-based health check:

The hz-healthcheck script comes with the Hazelcast package. Internally, it uses the HTTP-based health check endpoint.

To run the hz-healthcheck script:

You can use the following parameters to perform checks and operations on your Hazelcast clusters:

Example 1: Checking the State of Members in a Healthy Cluster:

If the member is deployed under the address 127.0.0.1:5701 and it is in the healthy state, the following output is expected:

Example 2: Checking the Safety of a Non-Existent Cluster:

If the cluster has no members running under the address 127.0.0.1:5701, the following output is expected:

The health monitor periodically prints logs in your console to provide information about your member’s state. By default, it is enabled when you start your cluster.

You can set the interval of health monitoring using the hazelcast.health.monitoring.delay.seconds system property. Its default value is 20 seconds.

The system property hazelcast.health.monitoring.level is used to configure the monitoring’s log level. If it is set to OFF, the monitoring is disabled. If it is set to NOISY, monitoring logs are always printed for the defined intervals. When it is SILENT, which is the default value, monitoring logs are printed only when the values exceed some predefined thresholds. These thresholds are related to memory and CPU percentages, and can be configured using the hazelcast.health.monitoring.threshold.memory.percentage and hazelcast.health.monitoring.threshold.cpu.percentage system properties, whose default values are both 70.

The following is an example monitoring output

The F5® BIG-IP® Local Traffic Manager™ (LTM) can be used as a load balancer for Hazelcast cluster members. This section describes how you can configure a health monitor to check the Hazelcast member states.

To enable diagnostics logging, you should specify the following properties on the member side:

On the Java client side, it is enough to set the following properties:

You can use the following property to specify the location of the diagnostics log file:

The name of the log file has the following format:

You can set a custom string prefix for the name of log file using the following property.

The content format of the diagnostics log file is depicted below:

A rolling file approach is used to prevent creating too much data. By default 10 files of 50MB each are allowed to exist. You can set the size of each file and number of files using the following properties.

You can use the following property to specify the output type of diagnostics.

Available types:

Hazelcast does not enforce any logging framework. You can always use your logging framework to configure the diagnostics. You can forward the logs to a logging framework by setting the hazelcast.diagnostics.stdout property to LOGGER:

Above configuration forwards the logs to the com.hazelcast.diagnostics logger so you can write them to a file by referencing one of the configured appenders. The following is an example for Log4j2.

By configuring TimeBasedTriggeringPolicy and SizeBasedTriggeringPolicy for the appender, you can control the size and rolling behavior as you want.

As it is stated in the introduction of this section and shown in the log file content above, diagnostics utility consists of plugins such as BuildInfo, SystemProperties and HazelcastInstance.

Hazelcast set the naming convention for MBeans as follows:

The MBeans name consists of the Hazelcast instance name, the type of the data structure and that data structure’s name. In the above example, _hzInstance_1_dev is the instance name, we connect to an IMap with the name trial.

One of the ways you can connect to JMX agent is using jconsole, jvisualvm (with MBean plugin) or another JMX compliant monitoring tool.

The other way to connect is to use a custom JMX client.

First, you need to specify the URL where the Hazelcast JMX service is running. See the following code snippet:

The port in the above example should be the one that you define while setting the JMX remote port number (if different than the default port 1099).

Then use the URL you acquired to connect to the JMX service and get the JMXConnector object. Using this object, get the MBeanServerConnection object. The MBeanServerConnection object enables you to use the MBean methods. See the example code below.

Once you get the MBeanServerConnection object, you can call the getter methods of MBeans as follows:

Hazelcast members expose various management beans which include statistics about distributed data structures and the states of Hazelcast member internals.

The metrics are local to the members, i.e., they do not reflect cluster wide values.

You can find the JMX API definition below with descriptions and the API methods in parenthesis.

Atomic Long (IAtomicLong)

Atomic Reference ( IAtomicReference )

Countdown Latch ( ICountDownLatch )

Executor Service ( IExecutorService )

List ( IList )

Lock ( ILock )

Map ( IMap )

MultiMap ( MultiMap )

Replicated Map ( ReplicatedMap )

Queue ( IQueue )

Semaphore ( ISemaphore )

Set ( ISet )

Topic ( ITopic )

Hazelcast Instance ( HazelcastInstance )

Hazelcast Management Center can expose the metrics collected from cluster members to Prometheus. This feature can be turned on by setting the hazelcast.mc.prometheusExporter.enabled system property to true.

Prometheus can be configured to scrape Management Center in prometheus.yml as follows:

After starting Prometheus with this configuration, all metrics will be exported to Prometheus with the hz_ prefix. The metrics are also available via the member JMX API. With the default configuration, Management Center exports all metrics reported by the cluster members. Since it can be overly verbose for some use cases, the metrics can be filtered with the hazelcast.mc.prometheusExporter.filter.metrics.included or the hazelcast.mc.prometheusExporter.filter.metrics.excluded system properties, both being comma-separated lists of metric names.

Example of starting Management Center with specifying the metrics exported to Prometheus:

Example of starting Management Center with specifying the metrics to be excluded from the Prometheus export:

By default, Prometheus connects via the same IP and port as the Management Center web interface. It is possible to override the port number using the -Dhazelcast.mc.prometheusExporter.port system property. Let’s say you have started Management Center as shown below:

Then, the Prometheus endpoint will be available at http://localhost:2222/metrics, which should be reflected by the Prometheus configuration as below:

You can use the Clustered JMX interface to integrate the Hazelcast Management Center with AppDynamics. To perform this integration, attach the AppDynamics Java agent to the Management Center.

For agent installation, see the Install the App Agent for Java page.

For monitoring on AppDynamics, see the Using AppDynamics for JMX Monitoring page.

After installing AppDynamics agent, you can start the Management Center as shown below:

When the Management Center starts, you should see the logs below:

You can use the Clustered JMX interface to integrate the Hazelcast Management Center with New Relic. To perform this integration, attach the New Relic Java agent and provide an extension file that describes which metrics will be sent to New Relic.

See Custom JMX instrumentation by YAML on the New Relic webpage.

The following is an example Map monitoring .yml file for New Relic:

Put the .yml file in the extensions directory in your New Relic installation. If an extensions directory does not exist there, create one.

After you set your extension, attach the New Relic Java agent and start the Management Center as shown below.

If your logging level is set to FINER, you should see the log listing in the file newrelic_agent.log, which is located in the logs directory in your New Relic installation. The following is an example log listing:

Then you can navigate to your New Relic account and create custom dashboards. See Get Started with Dashboards.

While you are creating the dashboard, you should see the metrics that you are sending to New Relic from the Management Center in the Metrics section under the JMX directory.