This is a prerelease version.

View latest

Prometheus metrics

6.0-SNAPSHOT 5.8 5.7 5.6 5.5 5.4 5.3 5.2 5.1 5.0 4.2022.01

Management Center automatically reformats all Hazelcast cluster metrics for compatibility with Prometheus. In addition, Management Center updates a subset of metrics to align with Prometheus best practice recommendations. This makes it easier to construct complex queries, build dashboards, and work alongside metrics from other applications.

By default, Management Center exposes two versions of metrics based on Hazelcast’s JMX metrics system:

  • Version 1 (V1) metrics apply a basic transformation for compatibility with Prometheus:

    • Convert . to _

    • Add hz_ prefix

  • Version 2 (V2) metrics (currently in BETA) apply an advanced transformation for conformity with Prometheus naming conventions:

    • Use snake_case

    • Add hazelcast_ prefix

    • Update metric names where appropriate

    • Combine related metrics and add labels

    • Convert to recommended base units:

      • Time in seconds

      • Memory in bytes

      • Percentages as ratios (for example, 50% converts to 0.5)

    • Append units to metric names

    • Remove unit labels

    • Append _total to accumulated metrics

You can optionally change which set of metrics is exposed.

The following examples illustrate the differences between the V1 and V2 metrics.

  • Rename and remove unit label, add _total suffix:

    # V1
hz_map_queryCount{name="map-1",mc_member="127.0.0.1:5701",mc_cluster="Cluster-1",unit="COUNT",} 0.0 1737715903399
# V2
hazelcast_map_queries_total{name="map-1",mc_member="127.0.0.1:5701",mc_cluster="Cluster-1",} 0.0 1737715903399

  • Rename and append unit, remove unit label, convert value (1.7x1012 milliseconds to 1.7x109 seconds):

    # V1
hz_map_creationTime{name="map-1",mc_member="127.0.0.1:5701",mc_cluster="Cluster-1",unit="MS",} 1.737715861118E12 1737715903399
# V2
hazelcast_map_creation_timestamp_seconds{name="map-1",mc_member="127.0.0.1:5701",mc_cluster="Cluster-1",} 1.737715861118E9 1737715903399

  • Rename and append unit, remove unit label, add remove label:

    # V1
hz_map_totalMaxRemoveLatency{name="map-1",mc_member="127.0.0.1:5701",mc_cluster="Cluster-1",unit="MS",} 0.0 1737715903399
# V2
hazelcast_map_latency_max_seconds{name="map-1",mc_member="127.0.0.1:5701",mc_cluster="Cluster-1",operation="remove",} 0.0 1737715903399

  • Rename, append unit suffix and _total suffix, remove unit label, add remove label, convert value from milliseconds to seconds:

    # V1
hz_map_totalPutLatency{name="map-1",mc_member="127.0.0.1:5701",mc_cluster="Cluster-1",unit="MS",} 2019.0 1743601193973
# V2
hazelcast_map_latency_seconds_total{name="map-1",mc_member="127.0.0.1:5701",mc_cluster="Cluster-1",operation="put",} 2.019 1743601193973

Change the metrics version

To change which set of metrics is exposed to Prometheus, update the hazelcast.mc.prometheusExporter.printers system property:

  • V1: hazelcast.mc.prometheusExporter.printers=V1

  • V2: hazelcast.mc.prometheusExporter.printers=V2

  • V1 and V2 (default): hazelcast.mc.prometheusExporter.printers=V1,V2

See System properties and environment variables.

If you set the property to V2, Management Center will only expose the subset of metrics that have updated versions available.

Metrics list

The following table lists all V2 metrics with their V1 equivalents. The full list of Hazelcast metrics and descriptions is provided in the Hazelcast Platform documentation.

In some cases, several V1 metrics are combined into a single V2 metric differentiated by labels. For example, hazelcast_map_operations_total replaces hz_map_getCount and hz_map_setCount by adding operation="get" and operation="set" labels.

Map
V1 metric hz_ V2 metric hazelcast_ Additional labels

map_backupCount

map_backups_total

n/a

map_backupEntryCount

map_backup_entries_total

n/a

map_backupEntryMemoryCost

map_backup_entries_memory_cost_bytes

n/a

map_creationTime

map_creation_timestamp_seconds

n/a

map_dirtyEntryCount

map_dirty_entries_total

n/a

map_evictionCount

map_evictions_total

n/a

map_expirationCount

map_expirations_total

n/a

map_getCount

map_operations_total

operation=get

map_heapCost

map_heap_cost_bytes

n/a

map_indexedQueryCount

map_indexed_queries_total

n/a

map_lastAccessTime

map_last_access_timestamp_seconds

n/a

map_lastUpdateTime

map_last_update_timestamp_seconds

n/a

map_lockedEntryCount

map_locked_entries_total

n/a

map_merkleTreesCost

map_merkle_trees_cost_bytes

n/a

map_numberOfEvents

map_events_total

n/a

map_numberOfOtherOperations

map_operations_total

operation=other

map_ownedEntryCount

map_owned_entries_total

n/a

map_ownedEntryMemoryCost

map_owned_entry_memory_cost_bytes

n/a

map_putCount

map_operations_total

operation=put

map_queryCount

map_queries_total

n/a

map_removeCount

map_operations_total

operation=remove

map_setCount

map_operations_total

operation=set

map_hits

map_hits_total

n/a

map_totalGetLatency

map_latency_seconds_total

operation=get

map_totalMaxGetLatency

map_latency_max_seconds

operation=get

map_totalPutLatency

map_latency_seconds_total

operation=put

map_totalMaxPutLatency

map_latency_max_seconds

operation=put

map_totalRemoveLatency

map_latency_seconds_total

operation=remove

map_totalMaxRemoveLatency

map_latency_max_seconds

operation=remove

map_totalSetLatency

map_latency_seconds_total

operation=set

map_totalMaxSetLatency

map_latency_max_seconds

operation=set

map_index_creationTime

map_index_creation_timestamp_seconds

n/a

map_index_hitCount

map_index_hits_total

n/a

map_index_insertCount

map_index_inserts_total

n/a

map_index_memoryCost

map_index_memory_cost_bytes

n/a

map_index_queryCount

map_index_queries_total

n/a

map_index_removeCount

map_index_removes_total

n/a

map_index_totalInsertLatency

map_index_latency_seconds_total

operation=insert

map_index_totalRemoveLatency

map_index_latency_seconds_total

operation=remove

map_index_totalUpdateLatency

map_index_latency_seconds_total

operation=update

map_index_updateCount

map_index_updates_total

n/a
Set
V1 metric hz_ V2 metric hazelcast_ Additional labels

set_creationTime

set_creation_timestamp_seconds

n/a

set_lastAccessTime

set_last_access_timestamp_seconds

n/a

set_lastUpdateTime

set_last_update_timestamp_seconds

n/a
List
V1 metric hz_ V2 metric hazelcast_ Additional labels

list_creationTime

list_creation_timestamp_seconds

n/a

list_lastAccessTime

list_last_access_timestamp_seconds

n/a

list_lastUpdateTime

list_last_update_timestamp_seconds

n/a
Operating System
V1 metric hz_ V2 metric hazelcast_ Additional labels

os.committedVirtualMemorySize

os_committed_virtual_memory_size_bytes

n/a

os.freePhysicalMemorySize

os_free_physical_memory_size_bytes

n/a

os.freeSwapSpaceSize

os_free_swap_space_size_bytes

n/a

os.maxFileDescriptorCount

os_max_file_descriptor_total

n/a

os.openFileDescriptorCount

os_open_file_descriptor_total

n/a

os.processCpuLoad

os_process_cpu_load_ratio

n/a

os.processCpuTime

os_process_cpu_time_seconds

n/a

os.systemCpuLoad

os_system_cpu_load_ratio

n/a

os.systemLoadAverage

os_system_load_average_ratio

n/a

os.totalPhysicalMemorySize

os_total_physical_memory_size_bytes

n/a

os.totalSwapSpaceSize

os_total_swap_space_size_bytes

n/a
Events
V1 metric hz_ V2 metric hazelcast_ Additional labels

event.eventQueueSize

event_queue_size_total

n/a

event.eventsProcessed

event_processed_total

n/a

event.listenerCount

event_listeners_total

n/a

event.publicationCount

event_publications_total

n/a

event.queueCapacity

event_queue_capacity_total

n/a

event.rejectedCount

event_rejected_total

n/a

event.syncDeliveryFailureCount

event_sync_delivery_failures_total

n/a

event.threadCount

event_threads_total

n/a

event.totalFailureCount

event_failures_total

n/a
Listeners
V1 metric hz_ V2 metric hazelcast_ Additional labels

listeners.eventsProcessed

listeners_events_processed_total

n/a

listeners.eventQueueSize

listeners_event_queue_size_total

n/a
Capacity
V1 metric hz_ V2 metric hazelcast_ Additional labels

file.partition.freeSpace

file_partition_free_space_bytes

n/a

file.partition.totalSpace

file_partition_total_space_bytes

n/a

file.partition.usableSpace

file_partition_usable_space_bytes

n/a
Garbage Collection
V1 metric hz_ V2 metric hazelcast_ Additional labels

gc.majorCount

gc_major_total

n/a

gc.majorTime

gc_major_time_seconds_total

n/a

gc.minorCount

gc_minor_total

n/a

gc.minorTime

gc_minor_time_seconds_total

n/a

gc.unknownCount

gc_unknown_total

n/a

gc.unknownTime

gc_unknown_time_seconds_total

n/a
Memory
V1 metric hz_ V2 metric hazelcast_ Additional labels

memory.committedHeap

memory_committed_heap_bytes

n/a

memory.committedNative

memory_committed_native_bytes

n/a

memory.freeHeap

memory_free_heap_bytes

n/a

memory.freeNative

memory_free_native_bytes

n/a

memory.freePhysical

memory_free_physical_bytes

n/a

memory.maxHeap

memory_max_heap_bytes

n/a

memory.maxMetadata

memory_max_metadata_bytes

n/a

memory.maxNative

memory_max_native_bytes

n/a

memory.totalPhysical

memory_total_physical_bytes

n/a

memory.usedHeap

memory_used_heap_bytes

n/a

memory.usedMetadata

memory_used_metadata_bytes

n/a

memory.usedNative

memory_used_native_bytes

n/a
Operations
V1 metric hz_ V2 metric hazelcast_ Additional labels

operation.adhoc.executedOperationsCount

operation_adhoc_executed_operations_total

n/a

operation.asyncOperations

operation_async_operations_total

n/a

operation.completedCount

operation_completed_total

n/a

operation.failedBackups

operation_failed_backups_total

n/a

operation.generic.executedOperationsCount

operation_generic_executed_operations_total

n/a

operation.genericPriorityQueueSize

operation_generic_priority_queue_size_total

n/a

operation.genericQueueSize

operation_generic_queue_size_total

n/a

operation.genericThreadCount

operation_generic_threads_total

n/a

operation.invocations.backupTimeoutMillis

operation_invocations_backup_timeout_seconds

n/a

operation.invocations.backupTimeouts

operation_invocations_backup_timeouts_total

n/a

operation.invocations.delayedExecutionCount

operation_invocations_delayed_executions_total

n/a

operation.invocations.heartbeatBroadcastPeriodMillis

operation_invocations_heartbeat_broadcast_period_seconds

n/a

operation.invocations.heartbeatPacketsReceived

operation_invocations_heartbeat_packets_received_total

n/a

operation.invocations.heartbeatPacketsSent

operation_invocations_heartbeat_packets_sent_total

n/a

operation.invocations.invocationScanPeriodMillis

operation_invocations_invocation_scan_period_seconds

n/a

operation.invocations.invocationTimeoutMillis

operation_invocations_invocation_timeout_seconds

n/a

operation.invocations.lastCallId

operation_invocations_last_call_id_total

n/a

operation.invocations.normalTimeouts

operation_invocations_normal_timeouts_total

n/a

operation.invocations.pending

operation_invocations_pending_total

n/a

operation.invocations.usedPercentage

operation_invocations_used_ratio

n/a

operation.parker.parkQueueCount

operation_parker_park_queue_total

n/a

operation.parker.totalParkedOperationCount

operation_parker_parked_operations_total

n/a

operation.partition.executedOperationsCount

operation_partition_executed_operations_total

n/a

operation.partitionThreadCount

operation_partition_threads_total

n/a

operation.priorityQueueSize

operation_priority_queue_size_total

n/a

operation.queueSize

operation_queue_size_total

n/a

operation.responseQueueSize

operation_response_queue_size_total

n/a

operation.responses.backupCount

operation_responses_backups_total

n/a

operation.responses.errorCount

operation_responses_errors_total

n/a

operation.responses.missingCount

operation_responses_missing_total

n/a

operation.responses.normalCount

operation_responses_normal_total

n/a

operation.responses.timeoutCount

operation_responses_timeouts_total

n/a

operation.retryCount

operation_retries_total

n/a

operation.runningCount

operation_running_total

n/a

operation.runningGenericCount

operation_running_generic_total

n/a

operation.runningPartitionCount

operation_running_partition_total

n/a

operation.thread.completedOperationCount

operation_thread_completed_operations_total

n/a

operation.thread.completedOperationBatchCount

operation_thread_completed_operation_batches_total

n/a

operation.thread.completedPacketCount

operation_thread_completed_packets_total

n/a

operation.thread.completedPartitionSpecificRunnableCount

operation_thread_completed_partition_specific_runnables_total

n/a

operation.thread.completedRunnableCount

operation_thread_completed_runnables_total

n/a

operation.thread.completedTotalCount

operation_thread_completed_total

n/a

operation.thread.errorCount

operation_thread_errors_total

n/a

operation.thread.normalPendingCount

operation_thread_normal_pending_total

n/a

operation.thread.priorityPendingCount

operation_thread_priority_pending_total

n/a
Runtime
V1 metric hz_ V2 metric hazelcast_ Additional labels

runtime.availableProcessors

runtime_available_processors_total

n/a

runtime.freeMemory

runtime_free_memory_bytes

n/a

runtime.maxMemory

runtime_max_memory_bytes

n/a

runtime.totalMemory

runtime_total_memory_bytes

n/a

runtime.uptime

runtime_up_time_seconds_total

n/a

runtime.usedMemory

runtime_used_memory_bytes

n/a
TCP
V1 metric hz_ V2 metric hazelcast_ Additional labels

tcp.acceptor.eventCount

tcp_acceptor_events_total

n/a

tcp.acceptor.exceptionCount

tcp_acceptor_exceptions_total

n/a

tcp.acceptor.idleTimeMillis

tcp_acceptor_idle_time_seconds

n/a

tcp.acceptor.selectorRecreateCount

tcp_acceptor_selector_recreate_total

n/a

tcp.balancer.imbalanceDetectedCount

tcp_balancer_imbalance_detected_total

n/a

tcp.balancer.migrationCompletedCount

tcp_balancer_migration_completed_total

n/a

tcp.bytesReceived

tcp_received_bytes

n/a

tcp.bytesSend

tcp_sent_bytes

n/a

tcp.connection.acceptedSocketCount

tcp_connection_accepted_socket_total

n/a

tcp.connection.activeCount

tcp_connection_active_total

n/a

tcp.connection.clientCount

tcp_connection_clients_total

n/a

tcp.connection.closedCount

tcp_connection_closed_total

n/a

tcp.connection.connectionListenerCount

tcp_connection_listeners_total

n/a

tcp.connection.count

tcp_connection_total

n/a

tcp.connection.inProgressCount

tcp_connection_in_progress_total

n/a

tcp.connection.openedCount

tcp_connection_opened_total

n/a

tcp.connection.textCount

tcp_connection_text_total

n/a

tcp.connection.in.completedMigrations

tcp_connection_completed_migrations_total

direction=in

tcp.connection.out.completedMigrations

tcp_connection_completed_migrations_total

direction=out

tcp.connection.in.opsInterested

tcp_connection_ops_interested_total

direction=in

tcp.connection.out.opsInterested

tcp_connection_ops_interested_total

direction=out

tcp.connection.in.opsReady

tcp_connection_ops_ready_total

direction=in

tcp.connection.out.opsReady

tcp_connection_ops_ready_total

direction=out

tcp.connection.in.ownerId

tcp_connection_owner_id_total

direction=in

tcp.connection.out.ownerId

tcp_connection_owner_id_total

direction=out

tcp.connection.in.processCount

tcp_connection_process_total

direction=in

tcp.connection.out.processCount

tcp_connection_process_total

direction=out

tcp.connection.in.startedMigrations

tcp_connection_started_migrations_total

direction=in

tcp.connection.out.startedMigrations

tcp_connection_started_migrations_total

direction=out

tcp.connection.in.bytesRead

tcp_connection_in_read_bytes

n/a

tcp.connection.in.idleTimeMs

tcp_connection_in_idle_time_seconds

n/a

tcp.connection.in.normalFramesRead

tcp_connection_in_normal_frames_read_total

n/a

tcp.connection.in.priorityFramesRead

tcp_connection_in_priority_frames_read_total

n/a

tcp.connection.out.bytesWritten

tcp_connection_out_written_bytes

n/a

tcp.connection.out.idleTimeMillis

tcp_connection_out_idle_time_seconds

n/a

tcp.connection.out.normalFramesWritten

tcp_connection_out_normal_frames_written_total

n/a

tcp.connection.out.priorityFramesWritten

tcp_connection_out_priority_frames_written_total

n/a

tcp.connection.out.priorityWriteQueuePendingBytes

tcp_connection_out_priority_write_queue_pending_bytes

n/a

tcp.connection.out.priorityWriteQueueSize

tcp_connection_out_priority_write_queue_size_total

n/a

tcp.connection.out.scheduled

tcp_connection_out_scheduled_total

n/a

tcp.connection.out.writeQueuePendingBytes

tcp_connection_out_write_queue_pending_bytes

n/a

tcp.connection.out.writeQueueSize

tcp_connection_out_write_queue_size_total

n/a

tcp.inputThread.bytesTransceived

tcp_transceived_bytes

thread=input

tcp.outputThread.bytesTransceived

tcp_transceived_bytes

thread=output

tcp.inputThread.completedTaskCount

tcp_completed_tasks_total

thread=input

tcp.outputThread.completedTaskCount

tcp_completed_tasks_total

thread=output

tcp.inputThread.eventCount

tcp_events_total

thread=input

tcp.outputThread.eventCount

tcp_events_total

thread=output

tcp.inputThread.framesTransceived

tcp_transceived_frames_total

thread=input

tcp.outputThread.framesTransceived

tcp_transceived_frames_total

thread=output

tcp.inputThread.idleTimeMillis

tcp_idle_seconds

thread=input

tcp.outputThread.idleTimeMillis

tcp_idle_seconds

thread=output

tcp.inputThread.ioThreadId

tcp_io_thread_id_total

thread=input

tcp.outputThread.ioThreadId

tcp_io_thread_id_total

thread=output

tcp.inputThread.priorityFramesTransceived

tcp_priority_frames_transceived_total

thread=input

tcp.outputThread.priorityFramesTransceived

tcp_priority_frames_transceived_total

thread=output

tcp.inputThread.processCount

tcp_process_total

thread=input

tcp.outputThread.processCount

tcp_process_total

thread=output

tcp.inputThread.selectorIOExceptionCount

tcp_selector_io_exceptions_total

thread=input

tcp.outputThread.selectorIOExceptionCount

tcp_selector_io_exceptions_total

thread=output

tcp.inputThread.taskQueueSize

tcp_task_queue_size_total

thread=input

tcp.outputThread.taskQueueSize

tcp_task_queue_size_total

thread=output
Threads
V1 metric hz_ V2 metric hazelcast_ Additional labels

thread.daemonThreadCount

thread_daemon_total

n/a

thread.peakThreadCount

thread_peak_total

n/a

thread.threadCount

thread_total

n/a

thread.totalStartedThreadCount

thread_started_total

n/a
lifebuoy

Help and support