Prometheus Monitoring

6.0-SNAPSHOT 5.7 5.6 5.5 5.4 5.3 5.2 5.1 5.0 4.2022.01 4.2021.12

Prometheus is an open source monitoring system for metrics. You can export Hazelcast cluster metrics to a scrape endpoint in Management Center that Prometheus can consume. This allows you to monitor Hazelcast along with your other applications in the same monitoring tool.

Management Center’s Prometheus Exporter automatically updates with any cluster changes and allows you to easily collect metrics for the whole cluster, or multiple clusters, from a single endpoint. Without it, you would have to scrape each member individually, manage any changes from scaling, restarts, and IP configuration, and collate the metrics from all members.

Enabling the Prometheus Exporter

To start exposing metrics via the Prometheus Exporter, set the hazelcast.mc.prometheusExporter.enabled system property to true.

Metrics are exposed to the /metrics endpoint, which is the Prometheus default. The endpoint provides metrics for all clusters that Management Center is connected to.

Management Center also reformats the metrics to align with Prometheus best practice recommendations. See Prometheus metrics for details.

Disabling Timestamps

Management Center collects metrics with the timestamp of individual members, and sends this timestamp to Prometheus. Prometheus discards metrics with the oldest timestamps. You can disable sending timestamps so that your monitoring system continues to operate when clock shifts happen in the cluster.

To stop sending timestamps to Prometheus, set the hazelcast.mc.prometheusExporter.timestamp.enabled property to false.

Configuring Prometheus

After enabling the Prometheus Exporter, you can configure your Prometheus YAML file to scrape metrics from Management Center.

For guidance, see the Prometheus configuration documentation.

To allow Prometheus to scrape metrics from Management Center, set the static_configs.targets parameter to the URL of your Management Center instance.

scrape_configs:
  - job_name: 'HZ MC'
    # scheme defaults to HTTP.
    static_configs:
    - targets: ['localhost:8080'] # replace this address with the network address of Hazelcast Management Center

Authenticating the Scrape Endpoint

To secure the scrape endpoint, you can set up basic authentication.

To set up basic authentication, use the set prometheus-auth task of the hz-mc conf tool. Replace the <my-user> and <my-pass> placeholders with your chosen username and password.

  • Linux and Mac

  • Windows

hz-mc conf set prometheus-auth <my-user> <my-pass>
mc-conf.bat set prometheus-auth <my-user> <my-pass>

Then, add the credentials to the basic_auth parameter of the Prometheus YAML file:

  - job_name: 'HZ MC'
    static_configs:
    - targets: ['localhost:8080']
    basic_auth:
      username: my-user
      password: my-pass

Resetting the Authentication Credentials

To reset authentication credentials for the scrape endpoint, use the security reset task of the hz-mc conf tool. You must stop Management Center to apply this change.

This task will also delete all users from the database.

  • Linux and Mac

  • Windows

hz-mc conf security reset
mc-conf.bat security reset

Filtering Metrics

By default, Management Center exports all metrics that are reported by the cluster.

To filter the metrics that are exposed to the scrape endpoint, use the hazelcast.mc.prometheusExporter.filter.metrics.included and hazelcast.mc.prometheusExporter.filter.metrics.excluded system properties.

For example, this configuration exposes only the following metrics:

  • hz_topic_totalReceivedMessages

  • hz_map_totalPutLatency

  • Linux and Mac

  • Windows

hz-mc start -Dhazelcast.mc.prometheusExporter.enabled=true \
  -Dhazelcast.mc.prometheusExporter.filter.metrics.included=hz_topic_totalReceivedMessages,hz_map_totalPutLatency
mc-start.cmd -Dhazelcast.mc.prometheusExporter.enabled=true ^
  -Dhazelcast.mc.prometheusExporter.filter.metrics.included=hz_topic_totalReceivedMessages,hz_map_totalPutLatency

This configuration exposes all metrics except the following:

  • hz_os_systemLoadAverage,hz_memory_freeHeap

  • Linux and Mac

  • Windows

hz-mc start -Dhazelcast.mc.prometheusExporter.enabled=true \
  -Dhazelcast.mc.prometheusExporter.filter.metrics.excluded=hz_os_systemLoadAverage,hz_memory_freeHeap
mc-start.cmd -Dhazelcast.mc.prometheusExporter.enabled=true ^
  -Dhazelcast.mc.prometheusExporter.filter.metrics.excluded=hz_os_systemLoadAverage,hz_memory_freeHeap
As described in Prometheus metrics, Management Center reformats metrics to align with Prometheus best practice recommendations. Filtering matches against metrics in any format. For example, if you exclude map.getCount, you will exclude hz_map_getCount and hazelcast_map_operations_total.

Overriding the Default Port

By default, metrics are exposed on the same port as the Management Center web interface. To override the port number, use the -Dhazelcast.mc.prometheusExporter.port system property.

  • Linux and Mac

  • Windows

hz-mc start -Dhazelcast.mc.prometheusExporter.enabled=true \
  -Dhazelcast.mc.prometheusExporter.port=2222
mc-start.cmd -Dhazelcast.mc.prometheusExporter.enabled=true ^
  -Dhazelcast.mc.prometheusExporter.port=2222

In this example, the Prometheus endpoint will be available at http://localhost:2222/metrics. To allow Prometheus to scrape metrics from Management Center, make sure to set the static_configs.targets parameter to this port.

scrape_configs:
  - job_name: 'HZ MC'
    static_configs:
    - targets: ['localhost:2222']

Visualizing Metrics in Grafana

Grafana is an open source data visualization solution that allows you to build monitoring dashboards from your Prometheus metrics. To start visualizing Prometheus metrics in Grafana, refer to the Grafana documentation. Hazelcast provides an example dashboard to help you get started.

Next Steps

Learn more about the hz-mc conf tool.

Explore system properties and their definitions.

lifebuoy

Help and support