List of Hazelcast Metrics
The table below lists the metrics with their explanations in grouped by their relevant subjects.
The metrics are collected per member and specific to the local member from which you collect them. For example, the distributed data structure metrics reflect the local statistics of that data structure for the portion held in that member.
Some metrics may store cluster-wide agreed value, that is, they may show the values obtained
by communicating with other members in the cluster. This type of
metrics reflect the member’s local view of the cluster (consider split-brain scenarios). The clusterStartTime is an example of this type of
metrics, and its value in the local member is obtained by communicating
with the master member.
Streaming Engine Cluster-Wide Metrics
| Name | Description | Tags |
|---|---|---|
blockingWorkerCount |
Number of non-cooperative workers employed. |
none Each Hazelcast member will have one instance of this metric. |
jobs.submitted |
Number of computational jobs submitted. |
|
jobs.completedSuccessfully |
Number of computational jobs successfully completed. |
|
jobs.completedWithFailure |
Number of computational jobs that have failed. |
|
jobs.executionStarted |
Number of computational job executions started. Each job can execute multiple times, for example when it’s restarted or suspended and then resumed. |
|
jobs.executionTerminated |
Number of computational job executions finished. Each job can execute multiple times, for example when it’s restarted or suspended and then resumed. |
|
iterationCount |
The total number of iterations the driver of tasklets in cooperative thread N made. It should increase by at least 250 iterations/s. Lower value means some of the cooperative processors blocks for too long. Somewhat lower value is normal if there are many tasklets assigned to the processor. Lower value affects the latency. |
cooperativeWorker Each Hazelcast member will have one of this metric for each of its cooperative worker threads. |
taskletCount |
The number of assigned tasklets to cooperative thread N. |
Streaming Engine Job-specific Metrics
All job specific metrics have their job (name and ID of the job) and
exec(job execution ID) tags set, and most also have the
vertex (vertex name) tag set (with very few exceptions). This means
that most of these metrics will have at least one instance for each
vertex of each current job execution.
Additionally, if the vertex sourcing them is a data source or data sink,
then the source or sink tags will also be set to true.
| Name | Description | Tags |
|---|---|---|
executionStartTime |
Start time of the current execution of the job (epoch time in milliseconds). |
job, exec There will be a single instance of these metrics for each job execution. |
executionCompletionTime |
Completion time of the current execution of the job (epoch time in milliseconds). |
|
status |
Information regarding the job’s status. Potential values:
|
|
userCancelled |
Details on whether the user canceled the job. Potential values:
Exposed via Job API only. |
|
snapshotBytes |
Total number of bytes written out in the last snapshot. |
job, exec, vertex There will be a single instance of these metrics for each vertex. |
snapshotKeys |
Total number of keys written out in the last snapshot. |
|
distributedBytesIn |
Total number of bytes received from remote members. |
job, exec, vertex, ordinal Each Hazelcast member will have an instance of these metrics for each ordinal of each vertex of each job execution. Note: These metrics are only present for distributed edges, i.e., edges producing network traffic. |
distributedBytesOut |
Total number of bytes sent to remote members. |
|
distributedItemsIn |
Total number of items received from remote members. |
|
distributedItemsOut |
Total number of items sent to remote members. |
|
topObservedWm |
This value is equal to the highest coalescedWm on any input edge of this processor. |
job, exec, vertex, proc Each Hazelcast member will have one instances of these metrics for each processor instance N, the N denotes the global processor index. Processor is the parallel worker doing the work of the vertex. |
coalescedWm |
The highest watermark received from all inputs that was sent to the processor to handle. |
|
lastForwardedWm |
Last watermark emitted by the processor to output. |
|
lastForwardedWmLatency |
The difference between <i>lastForwardedWn</i> and the system time at the moment when metrics were collected. |
|
queuesCapacity |
The total capacity of input queues. |
|
queuesSize |
The total number of items waiting in input queues. |
|
topObservedWm |
The highest received watermark from any input on edge N. |
job, exec, vertex, proc, ordinal Each Hazelcast member will have one instance of these metrics for each edge M (input or output) of each processor N. N is the global processor index and M is either the ordinal of the edge or has the value snapshot for output items written to state snapshot. |
coalescedWm |
The highest watermark received from all upstream processors on edge N. |
|
emittedCount |
The number of emitted items. This number includes watermarks, snapshot barriers etc. Unlike distributedItemsOut, it includes items emitted items to local processors. |
|
receivedCount |
The number of received items. This number does not include watermarks, snapshot barriers etc. It’s the number of items the Processor.process method will receive. |
|
receivedBatches |
The number of received batches. |
|
numInFlightOps |
The number of pending (in flight) operations when using asynchronous mapping processors. See Processors.mapUsingServiceAsyncP. |
job, exec, vertex, proc, procType Processor specific metrics, only certain types of processors have them. The procType tag can be used to identify the exact type of processor sourcing them. Like all processor metrics, each Hazelcast member will have one instances of these metrics for each processor instance N, the N denotes the global processor index. |
totalKeys |
The number of active keys being tracked by a session window processor. |
|
totalWindows |
The number of active windows being tracked by a session window processor. See Processors.aggregateToSessionWindowP. |
|
totalFrames |
The number of active frames being tracked by a sliding window processor. |
|
totalKeysInFrames |
The number of grouping keys associated with the current active frames of a sliding window processor. See Processors.aggregateToSlidingWindowP. |
|
lateEventsDropped |
The number of late events dropped by various processor, due to the watermark already having passed their windows. |
Map
Name |
Unit |
Description |
|
count |
Number of backups per entry |
|
count |
Number of backup entries held by the member |
|
bytes |
Memory cost of backup entries in this member |
|
ms |
Creation time of the map on the member |
|
count |
Number of updated but not persisted yet entries, dirty entries, that the member owns |
|
count |
Number of evictions happened on locally owned entries, backups are not included |
|
count |
Number of expirations happened on locally owned entries, backups are not included |
|
count |
Number of local get operations on the map; it is incremented for every get operation even if the entries do not exist. |
|
count |
Total heap cost for the map on this member |
|
count |
Number of reads of the locally owned entries; it is incremented for every read by any type of operation (get, set, put). So the entries should exist. |
|
count |
Total number of indexed local queries performed on the map |
|
ms |
Last access (read) time of the locally owned entries |
|
ms |
Last update time of the locally owned entries |
|
count |
Number of locked entries that the member owns |
|
count |
Total heap cost of the Merkle trees used |
|
count |
Number of local events received on the map |
|
count |
Total number of other operations performed on this member |
|
count |
Number of map entries owned by the member |
|
bytes |
Memory cost of owned map entries on this member |
|
count |
Number of local put operations on the map |
|
count |
Number of queries executed on the map (it may be imprecise for queries involving partition predicates (PartitionPredicate) on the off-heap storage) |
|
count |
Number of local remove operations on the map |
|
count |
Number of local set operations on the map |
|
ms |
Total latency of local get operations on the map |
|
ms |
Maximum latency of local get operations on the map |
|
ms |
Maximum latency of local put operations on the map |
|
ms |
Maximum latency of local remove operations on the map |
|
ms |
Maximum latency of local set operations on the map |
|
ms |
Total latency of local put operations on the map |
|
ms |
Total latency of local remove operations on the map |
|
ms |
Total latency of local set operations on the map |
The above
|
||
|
ns |
Average hit latency for the index on this member |
|
percent |
Average selectivity of the hits served by the index on this member (the returned value is in the range from 0.0 to 1.0 - values close to 1.0 indicate a high selectivity meaning the index is efficient; values close to 0.0 indicate a low selectivity meaning the index efficiency is approaching an efficiency of a simple full scan) |
|
ms |
Creation time of the index on this member |
|
count |
Total number of index hits (the value of this metric may be greater than the |
|
count |
Number of insert operations performed on the index |
|
bytes |
Local memory cost of the index (for on-heap indexes in OBJECT or BINARY formats, the returned value is just a best-effort approximation and doesn’t indicate a precise on-heap memory usage of the index) |
|
count |
Total number of queries served by the index |
|
count |
Number of remove operations performed on the index |
|
ns |
Total latency of insert operations performed on the index |
|
ns |
Total latency of remove operations performed on the index |
|
ns |
Total latency of update operations performed on the index. |
|
count |
Number of update operations performed on the index |
Vector Collection (Beta) Enterprise Edition
Name |
Unit |
Description |
|
count |
Number of backups per entry |
|
count |
Number of backup entries held by the member |
|
bytes |
Heap memory cost of backup entries in this member |
|
count |
Number of local clear operations on the vector collection |
|
ms |
Creation time of the vector collection on the member |
|
count |
Number of local delete operations on the vector collection |
|
count |
Number of local get operations on the vector collection; it is incremented for every get operation even if the entries do not exist. |
|
count |
Total heap cost of the vector collection on this member |
|
ms |
Last access time of the locally owned entries |
|
ms |
Last update time of the locally owned entries |
|
ms |
Maximum latency of local clear operations on the vector collection |
|
ms |
Maximum latency of local optimize operations on the vector collection |
|
ms |
Maximum latency of search operations coordinated by this member on the vector collection |
|
ms |
Maximum latency of local size operations on the vector collection |
|
count |
Number of local optimize operations on the vector collection |
|
count |
Number of vector collection entries owned by the member |
|
bytes |
Heap memory cost of owned vector collection entries on this member |
|
count |
Number of local putAll operations on the vector collection. Note that for each |
|
count |
Number of entries passed to local putAll operations on the vector collection |
|
count |
Number of local put operations on the vector collection |
|
count |
Number of local remove operations on the vector collection |
|
count |
Number of search operations coordinated by this member on the vector collection |
|
count |
Number of results returned by search operations coordinated by this member on the vector collection |
|
count |
Number of local set operations on the vector collection |
|
count |
Number of local size operations on the vector collection |
|
ms |
Total latency of local clear operations on the vector collection |
|
ms |
Total latency of local delete operations on the vector collection |
|
ms |
Total latency of local get operations on the vector collection |
|
ms |
Maximum latency of local delete operations on the vector collection |
|
ms |
Maximum latency of local get operations on the vector collection |
|
ms |
Maximum latency of local putAll operations on the vector collection |
|
ms |
Maximum latency of local put operations on the vector collection |
|
ms |
Maximum latency of local remove operations on the vector collection |
|
ms |
Maximum latency of local set operations on the vector collection |
|
ms |
Total latency of local optimize operations on the vector collection |
|
ms |
Total latency of local putAll operations on the vector collection |
|
ms |
Total latency of local put operations on the vector collection |
|
ms |
Total latency of local remove operations on the vector collection |
|
ms |
Total latency of search operations coordinated by this member on the vector collection |
|
ms |
Total latency of local set operations on the vector collection |
|
ms |
Total latency of local size operations on the vector collection |
The above
|
||
MultiMap
Name |
Unit |
Description |
|
count |
Number of backups per entry |
|
count |
Number of backup entries held by the member |
|
bytes |
Memory cost of backup entries in this member |
|
ms |
Creation time of the multimap in the member |
|
count |
Number of dirty (updated but not persisted yet) entries that the member owns |
|
count |
Number of local get operations on the multimap |
|
count |
Total heap cost for the multimap on this member |
|
count |
Number of hits (reads) of the locally owned entries |
|
count |
Total number of indexed local queries performed on the multimap |
|
ms |
Last access (read) time of the locally owned entries |
|
ms |
Last update time of the locally owned entries |
|
count |
Number of locked entries that the member owns |
|
count |
Heap cost of the Merkle trees |
|
count |
Number of local events received |
|
count |
Total number of other operations |
|
count |
Number of multimap entries owned by the member |
|
bytes |
Memory cost of owned multimap entries on this member |
|
count |
Number of local put operations on the multimap |
|
count |
Number of local queries executed on the multimap (it may be imprecise for queries involving partition predicates (PartitionPredicate) on the off-heap storage) |
|
count |
Number of local remove operations on the multimap |
|
count |
Number of local set operations on the multimap |
|
ms |
Total latency of local get operations |
|
ms |
Maximum latency of local get operations |
|
ms |
Maximum latency of local put operations |
|
ms |
Maximum latency of local remove operations |
|
ms |
Maximum latency of local set operations |
|
ms |
Total latency of local put operations |
|
ms |
Total latency of local remove operations |
|
ms |
Total latency of local set operations |
Replicated Map
Name |
Unit |
Description |
|
ms |
Creation time of this replicated map on this member |
|
count |
Number of get operations on this member |
|
count |
Number of hits (reads) of the locally owned entries |
|
ms |
Last access (read) time of the locally owned entries |
|
ms |
Last update time of the locally owned entries |
|
ms |
Maximum latency of get operations |
|
ms |
Maximum latency of put operations |
|
ms |
Maximum latency of remove operations |
|
count |
Number of events received on this member |
|
count |
Total number of other operations on this member |
|
count |
Number of entries owned on this member |
|
bytes |
Memory cost of owned entries on this member |
|
count |
Number of put operations on this member |
|
count |
Number of remove operations on this member |
|
ms |
Total latency of get operations |
|
ms |
Total latency of put operations |
|
ms |
Total latency of remove operations |
|
count |
Total number of operations on this member |
Cache
Name |
Unit |
Description |
|
µs |
Mean time to execute gets on the cache |
|
µs |
Mean time to execute puts on the cache |
|
µs |
Mean time to execute removes on the cache |
|
count |
Number of evictions on the cache |
|
count |
Number of gets on the cache |
|
count |
Number of successful get operations, hits, on the cache |
|
percent |
Percentage of successful get operations, hits, out of all get operations on the cache |
|
count |
Number of puts to the cache |
|
count |
Number of removals from the cache |
|
count |
Number of missed cache accesses on the cache |
|
percent |
Percentage of missed cache accesses out of all the cache accesses/access attempts |
|
ms |
Creation time of the cache on the member |
|
ms |
Cache’s last access time |
|
ms |
Cache’s last update time |
|
count |
Locally owned entry count in the cache |
Queue
Name |
Unit |
Description |
|
ms |
Average age of the items in this member |
|
count |
Number of backup items held by the member |
|
ms |
Creation time of the topic on the member |
|
count |
Number of event operations |
|
ms |
Maximum age of the items in this member |
|
ms |
Minimum age of the items in this member |
|
count |
Number of null returning poll operations |
|
count |
Number of event operations (duplicate of eventOperationCount) |
|
count |
Number of offer/put/add operations |
|
count |
Number of other operations |
|
count |
Number of poll/take/remove operations. |
|
count |
Number of rejected offers |
|
count |
Number of owned items in this member |
|
count |
Total number of operations (numberOfOffers + numberOfPolls + numberOfOtherOperations) |
Set
Name |
Unit |
Description |
|
ms |
Creation time of the set on the member |
|
ms |
Last access (read) time of the locally owned items |
|
ms |
Last update time of the locally owned items |
List
Name |
Unit |
Description |
|
ms |
Creation time of this list on the member |
|
ms |
Last access (read) time of the locally owned items |
|
ms |
Last update time of the locally owned items |
Topic
Name |
Unit |
Description |
|
ms |
Creation time of the topic on the member |
|
count |
Total number of published messages of this topic on this member |
|
count |
Total number of received messages of this topic on this member |
Reliable Topic
Name |
Unit |
Description |
|
ms |
Creation time of this reliable topic on the member |
|
count |
Total number of published messages of this reliable topic on this member |
|
count |
Total number of received messages of this reliable topic on this member |
Flake ID Generator
Name |
Unit |
Description |
|
count |
Total number of times the Flake ID generator has been used to generate a new ID batch |
|
ms |
Creation time of this Flake ID Generator on the member |
|
count |
Total number of IDs generated (the sum of IDs for all batches) |
PN Counter
Name |
Unit |
Description |
|
ms |
Creation time of the PN counter on the member |
|
count |
Number of subtract (including decrement) operations on this PN counter |
|
count |
Number of add (including increment) operations on this PN counter |
|
count |
Current value of the PN counter |
Executor Service
Name |
Unit |
Description |
|
count |
Number of cancelled operations on the executor service |
|
count |
Number of completed operations on the executor service |
|
ms |
Creation time of this executor on the member |
|
count |
Number of pending operations on the executor service |
|
count |
Number of started operations on the executor service |
|
ms |
Total execution time of the finished operations |
|
ms |
Total start latency of operations started |
|
count |
Number of completed tasks by this executor |
|
count |
Maximum number of threads in the executor’s thread pool |
|
count |
Number of threads in the executor’s thread pool |
|
count |
Number of pending tasks in this executor’s task queue |
|
count |
Remaining capacity on the executor’s task queue |
User Code Deployment
Name |
Unit |
Description |
|
count |
Number of classes that are currently loaded |
|
count |
Total number of classes that have been loaded since the instance has started execution. |
|
count |
Total number of unloaded classes. |
Cluster
Name |
Unit |
Description |
|
ms |
|
|
ms |
Elapsed time since the master member was created ( |
|
ms |
Difference between the local time ( |
|
ms |
Uptime of the cluster (current time - |
|
ms |
Member’s local clock timestamp |
|
ms |
Maximum observed cluster time difference |
|
count |
Connection close time for this connection |
|
count |
Connection ID for this client connection |
|
count |
Number of event handlers for this client connection |
|
ms |
Last time that this member sent a heartbeat to other known cluster members |
|
count |
Number of members in the cluster |
Clients
Name |
Unit |
Description |
|
count |
Number of active client endpoints for this member |
|
count |
Total number of client endpoint registrations |
Client Invocations
Name |
Unit |
Description |
|
count |
Maximum number of concurrent client invocations |
|
count |
Number of pending client invocations on this client |
|
count |
Number of started client invocations on this client |
CP Subsystem
Name |
Unit |
Description |
|
count |
Value of this |
|
count |
Initial count of |
|
count |
Remaining number of expected countdowns |
|
count |
Round number of the |
|
count |
Maximum number of reentrant acquires of this |
|
count |
Total number of times this |
|
count |
Address of the |
|
count |
Session Id of the |
|
count |
Number of the remaining available permits |
|
count |
State value which shows whether semaphore is initialized with a value or not; in exposed metrics, it shows 0 when semaphore is not initialized, a positive value otherwise |
|
count |
Number of keys in this |
|
count |
Total number of bytes used by key-value pairs for this |
|
ms |
Creation time of this session |
|
Address of the endpoint which the CP session belongs to |
|
|
Type of the endpoint; either SERVER or CLIENT |
|
|
ms |
Expiration time of the CP session |
|
count |
Version number of the CP session, basically it shows how many times the session heartbeat is received |
We also have per-object type a summary section which provides live and destroyed object counts grouped by CP Group. These can be found under cp.{atomiclong,atomicref,countdownlatch,lock,map,semaphore}.summary and provide the following child attributes.
Name |
Unit |
Description |
|
count |
Number of objects for the parent object type that have been destroyed. Also referred to as the tombstoned object count. |
|
count |
Number of objects for the parent object type that are in use. |
Events
Name |
Unit |
Description |
|
count |
Total number of events pending to be processed |
|
count |
Total number of processed events |
|
count |
Number of subscribed listeners for the specified service |
|
count |
Number of published events for the specified service |
|
count |
Queue capacity of the executor processing the events (this capacity is shared for all events) |
|
count |
Number of rejected events; if the event is not accepted to the executor in |
|
count |
Number of failures of sync event delivery |
|
count |
Number of threads for the event service executor (the event thread count) |
|
count |
Number of events that fail to be published |
Listeners
Name |
Unit |
Description |
|
count |
Total number of processed events on the client listener service |
|
count |
Total number of tasks pending to be processed on the client listener service |
Capacity
Name |
Unit |
Description |
|
bytes |
Amount of free space in the given directory, user.home |
|
bytes |
Amount of total space in the given directory, user.home |
|
bytes |
Amount of usable space in the given directory, user.home |
Garbage Collection
Name |
Unit |
Description |
|
count |
Total number of major garbage collections (GCs) that have occurred |
|
ms |
Accumulated elapsed time in major GCs |
|
count |
Total number of minor GCs that have occurred |
|
ms |
Accumulated elapsed time in minor GCs |
|
count |
Number of unknown GCs that cannot be determined as minor or major (this is usually due to the lack of support of the used garbage collector) |
|
ms |
Accumulated elapsed time in unknown GCs |
Memory
Name |
Unit |
Description |
|
bytes |
Amount of heap memory that is committed for the JVM to use |
|
bytes |
Amount of native memory that is committed for current instance (member or client) to use |
|
bytes |
Amount of free memory in the JVM of current instance (member or client) |
|
bytes |
Amount of free native memory in the current instance (member or client) |
|
bytes |
Amount of free physical memory available in OS |
|
bytes |
Maximum amount of memory that the JVM will attempt to us |
|
bytes |
Amount of native memory reserved for metadata (this memory is separate and not accounted for by the NativeMemory statistics) |
|
bytes |
Maximum amount of native memory that current instance (member or client) will attempt to use |
|
bytes |
Amount of total physical memory available in OS |
|
bytes |
Amount of used memory in the JVM of the current instance (member or client) |
|
bytes |
Amount of used metadata memory by the current instance (member or client) |
|
bytes |
Amount of used native memory by the current instance (member or client) |
High-Density (HD) Memory Store
Name |
Unit |
Description |
|
bytes |
Cost of HD memory for the map to determine whether eviction of map entries is needed |
|
bytes |
Number of evictions performed for the HD memory store (when your eviction policy does not free enough native memory, operations that add entries to a map or cache such as |
|
bytes |
Number of entries removed from the HD memory store |
|
bytes |
Number of entries in the HD memory store |
Near Cache
Name |
Unit |
Description |
|
ms |
Creation time of this Near Cache on this instance (member or client) |
|
count |
Number of evictions of Near Cache entries owned by this instance (member or client) |
|
count |
Number of TTL and max-idle expirations of Near Cache entries owned by this instance (member or client) |
|
count |
Number of hits (reads) of Near Cache entries owned by this instance (member or client) |
|
count |
Number of invalidations of Near Cache entries owned by this instance (member or client). |
|
count |
Number of invalidations of Near Cache entries owned by this instance (member or client). |
|
ms |
Duration of the last Near Cache key persistence |
|
count |
Number of persisted keys of the last Near Cache key persistence |
|
ms |
Timestamp of the last Near Cache key persistence |
|
bytes |
Written bytes of the last Near Cache key persistence |
|
count |
Number of misses of Near Cache entries owned by this instance (member or client). |
|
count |
Number of Near Cache entries owned by this instance (member or client) |
|
bytes |
Memory cost of Near Cache entries owned by this instance (member or client) |
|
count |
Number of Near Cache key persistences (when the preload feature is enabled) |
Operations
Within Hazelcast context, the priority operations are the ones that are important for the stability of cluster, for example heartbeats and migration requests.
The normal operations are the ones that manipulate the data, for example map.get and map.put.
|
Name |
Unit |
Description |
|
count |
Number of executed adhoc operations |
|
count |
Number of current executing async operations on the operation service of the member |
|
count |
Number of completed operations |
|
count |
Number of failed backup operations on the operation service of the member |
|
count |
Number of executed generic operations |
|
count |
Number of priority generic operations pending (waiting in the priority queue) |
|
count |
Number of normal generic operations pending (waiting in the queue) |
|
count |
Number of generic operation handler threads in the member |
|
ms |
Operation backup timeout that specifies how long the invocation will wait for acknowledgements from the backup replicas (if acks are not received from some backups, there will not be any rollback on other successful replicas) |
|
count |
Number of operation invocations that acknowledgment from backups has timeout |
|
count |
Number of times that the operation invocations have delayed |
|
ms |
Broadcast period of operation heartbeats (this heartbeat packets sent to inform the other member about if the operation is still alive). The heartbeat period is configured to be 1/4 of the call timeout. So with default settings, every 15 seconds, every member in the cluster, will notify every other member in the cluster about all calls that are pending. |
|
count |
Number of received heartbeat packets |
|
count |
Number of sent heartbeat packets |
|
ms |
Period for scanning over pending invocations for getting rid of duplicates, checking for heartbeat timeout, and checking for backup timeout |
|
ms |
Timeout for operation invocations |
|
count |
Last issued invocation call ID |
|
count |
Number of times that the operation invocations have timed out |
|
count |
Number of pending invocations |
|
percent |
Usage percentage of the operation invocation capacity (pending invocations/ max concurrent invocations) |
|
count |
Number of separate WaitSet (set of operations waiting for some condition) |
|
count |
Total number of parked operations |
|
count |
Number of executed partition operations on the specified partition |
|
count |
Number of partition operation handler threads for given member |
|
count |
Number of priority operations pending (priority partition ops. + priority generic ops.) |
|
count |
Number of normal operations pending (normal partition operations + normal generic operations). It refers to the number of operations sent to the member that have yet to be consumed for processing by the partition operation threads.
This is the most critical queue for partition aware operations such as |
|
count |
Total number of pending responses (work queue for the response threads) to be processed. |
|
count |
Number of backup acknowledgement responses |
|
count |
Number of error responses |
|
count |
Number of responses having missing invocations |
|
count |
Number of normal responses |
|
count |
Number of call timeout responses |
|
count |
Number of retried operations |
|
count |
Number of currently running operations (runningPartitionCount + runningGenericCount) |
|
count |
Number of currently running generic (non partition specific) operations |
|
count |
Number of currently running partition operations |
|
count |
Number of completed operations by this operation thread |
|
count |
Number of completed |
|
count |
Number of packets that executed by this operation thread |
|
count |
Number of |
|
count |
Total number of runnables executed by this operation thread |
|
count |
Total number of tasks ( |
|
count |
Total number of failed tasks on this operation thread |
|
count |
Number of normal pending operations (tasks) |
|
count |
Number of priority pending operations (tasks) |
Operating System
Name |
Unit |
Description |
|
bytes |
Amount of committed virtual memory (that is, the amount of virtual memory guaranteed to be available to the running process) |
|
bytes |
Amount of free physical memory |
|
bytes |
Amount of free swap space size |
|
count |
Maximum number of open file descriptors (only for UNIX platforms) |
|
count |
Number of open file descriptors (only for UNIX platforms) |
|
percent |
Recent CPU usage for the JVM process; a negative value if not available |
|
ms |
CPU time used by the process on which the JVM is running |
|
percent |
Recent CPU usage for the whole system; a negative value if not available |
|
percent |
System load average for the last minute, or a negative value if not available |
|
bytes |
Total amount of physical memory |
|
bytes |
Total amount of swap space |
Partitions
Name |
Unit |
Description |
|
count |
Number of partitions assigned to the member |
|
count |
Number of completed migrations on the latest repartitioning round |
|
ns |
Total elapsed time of commit operations' executions to the destination endpoint on the latest repartitioning round |
|
ns |
Total elapsed time of migration & replication operations' executions from source to destination endpoints on the latest repartitioning round |
|
ns |
Total elapsed time from the start of migration tasks to their completion (successful or otherwise) on the latest repartitioning round |
|
ms |
Latest time that repartition took place |
|
count |
Number of partitions currently owned by given member |
|
count |
Maximum allowed backup count according to current cluster formation and partition group configuration |
|
count |
Number of the member groups to be used in partition assignments |
|
boolean |
Whether there are any currently active migration tasks |
|
count |
Number of migration tasks in the migration queue |
|
count |
Total partition count |
|
count |
Number of planned migrations on the latest repartitioning round |
|
count |
Number of replica sync requests |
|
count |
Permits count of this replica sync semaphore |
|
count |
Stamp value for the current partition table; stamp is calculated by hashing the individual partition versions using MurmurHash3 (if stamp has this initial value, 0L, then that means partition table is not initialized yet) |
|
count |
Total number of completed migrations |
|
ns |
Total elapsed time of commit operations' executions to the destination endpoint since the beginning |
|
ns |
Total elapsed time of migration & replication operations' executions from source to destination endpoints since the beginning |
|
ns |
Total elapsed time from the start of migration tasks to their completion (successful or otherwise) since the beginning |
Persistence
Name |
Unit |
Description |
|
count |
Number of live tombstones in the persistent store |
|
count |
Number of live values in the persistent store |
|
bytes |
Approximate size of the garbage within the tombstone chunks (it does not account for data in the active chunk, it is incremented when a record is retired or an active chunk is turned into a stable one) |
|
bytes |
Approximate size of tombstone chunks (it does not account for data in the active chunk, it is incremented when the active chunk is turned into a stable one) |
|
bytes |
Approximate size of the garbage within the value chunk (it does not account for data in the active chunk, it is incremented when a record is retired or an active chunk is turned into a stable one) |
|
bytes |
Approximate size of value chunks (it does not account for data in the active chunk, it is incremented when the active chunk is turned into a stable one) |
Proxies
Name |
Unit |
Description |
|
count |
Number of created proxies for a given service |
|
count |
Number of destroyed proxies for a given service |
|
count |
Number of active proxies for a given service |
Raft Algorithm
Name |
Unit |
Description |
|
count |
Number of destroyed raft node group IDs |
|
count |
Available log capacity for this CP group |
|
count |
Commit index of this CP group |
|
count |
Last applied index of this CP group |
|
count |
Last log index of this CP group |
|
count |
Last log term of this CP group |
|
count |
Number of members in this CP group |
|
count |
Raft snapshot index of this CP group |
|
count |
Raft term of this CP group |
|
count |
Commit index of the active CP members |
|
count |
Number of active CP members |
|
count |
Number of CP groups |
|
count |
Number of missing CP members |
|
count |
Number of local Raft nodes |
|
count |
Number of terminated raft node group IDs |
Runtime
Name |
Unit |
Description |
|
count |
Number of processors available to the JVM |
|
bytes |
Amount of free memory in the JVM |
|
bytes |
Maximum amount of memory that the JVM will attempt to use |
|
bytes |
Total amount of memory in the JVM, the value of this metric may vary over time, depending on the host environment |
|
ms |
Uptime of the JVM |
|
bytes |
Approximation to the total amount of memory currently used |
TCP
Name |
Unit |
Description |
|
count |
Total number of the connections accepted by |
|
count |
Number of thrown exceptions on this |
|
ms |
Idle time that measures how long this TcpServerAcceptor has not received any events |
|
count |
Number of times the selector was recreated |
|
count |
Number of times the |
|
count |
Number of completed NioPipeline migrations by the IOBalancer (these migrations are performed to fix the load imbalance problem on the NioThreads) |
|
bytes |
Number of bytes received over all connections (active and closed) |
|
bytes |
Number of bytes sent over all connections (active and closed) |
|
count |
Number of accepted socket channels |
|
count |
Number of active connections |
|
count |
Number of the active client connections |
|
count |
Number of closed connections |
|
count |
Number of active connection listeners |
|
count |
Number of |
|
count |
Number of connection establishments in progress |
|
count |
Number of opened connections |
|
count |
Number of connections used by text-based protocols (REST, Memcache) |
|
count |
Number of completed migrations on this pipeline (migrates this pipeline to a different NioThread) |
|
count |
|
|
count |
|
|
count |
Owner ID of this |
|
count |
Number of time the |
|
count |
Number of started migrations on this pipeline |
|
bytes |
Total size of frames read on this inbound pipeline |
|
ms |
Idle time that indicates how long since the last read on this inbound nio pipeline |
|
count |
Number of priority frames read on this inbound nio pipeline |
|
count |
Number of priority frames read |
|
bytes |
Total amount of written frames on this outbound pipeline |
|
ms |
Idle time that indicates how long since the last write on this outbound nio pipeline |
|
count |
Number of written normal frames on this outbound nio pipeline |
|
count |
Number of priority frames written into this nio pipeline |
|
bytes |
Total size of priority frames pending in the write queue |
|
count |
Number of priority frames pending in the write queue |
|
count |
Ordinal of enum state of this outbound pipeline: 0 → UNSCHEDULED, 1 → SCHEDULED, 2 → BLOCKED, 3 → RESCHEDULE |
|
bytes |
Total size of normal frames pending in the write queue |
|
count |
Number of normal frames pending in the write queue |
|
bytes |
Amount of transceived data on this NioThread |
|
count |
Total number of completed tasks on this NioThread |
|
count |
Total number of the connections accepted by TcpServerAcceptor |
|
count |
Number of transceived frames on this NioThread |
|
ms |
Idle time that indicates the duration since the last read/write |
|
count |
Thread ID of this NioThread |
|
count |
Number of transceived priority frames |
|
count |
Number of processed `NioPipeline`s on this NioThread |
|
count |
Number of times that I/O exceptions are thrown during selection |
|
count |
Number of pending tasks on the queue of NioThread |
Threads
Name |
Unit |
Description |
|
count |
Current number of live daemon thread in the JVM |
|
count |
Peak live thread count since the JVM started |
|
count |
Current number of live threads including both daemon and non-daemon threads in the JVM |
|
count |
Total number of threads started since the JVM started |
Transactions
Name |
Unit |
Description |
|
count |
Number of committed transactions |
|
count |
Number of rollbacked transactions |
|
count |
Number of started transactions |
Tiered Store
Name |
Unit |
Description |
|
bytes |
Amount of free space in the device directory |
|
bytes |
Amount of total space in the device directory |
|
bytes |
Amount of space in the device directory used by Hybrid Log files |
|
bytes |
Amount of used space in the device directory |
|
count |
Average number of Hybrid Log allocations per page |
|
count |
Maximum number of Hybrid Log allocations per page |
|
count |
Minimum number of Hybrid Log allocations per page |
|
bytes |
Average Hybrid Log allocation size |
|
bytes |
Maximum Hybrid Log allocation size |
|
bytes |
Minimum Hybrid Log allocation size |
|
bytes |
Total size of Hybrid Log allocations |
|
ns |
Average time spent on stalled allocation for Hybrid Log |
|
ns |
Maximum time spent on stalled allocation for Hybrid Log |
|
ns |
Minimum time spent on stalled allocation for Hybrid Log |
|
ns |
Total time spent on stalled allocations for Hybrid Log |
|
count |
Number of finished Hybrid Log compactions (successful and failed) |
|
count |
Number of failed Hybrid Log compactions |
|
count |
Number of Hybrid Log compactions currently in progress |
|
ns |
Time spent on I/O during compaction |
|
count |
Number of Hybrid Log compactions currently waiting in the queue |
|
ns |
Average time for which Hybrid Log compaction has been waiting in the queue |
|
ns |
Maximum time for which Hybrid Log compaction has been waiting in the queue |
|
ns |
Minimum time for which Hybrid Log compaction has been waiting in the queue |
|
ns |
Total time for which Hybrid Log compactions have been waiting in the queue |
|
ns |
Average time for which Hybrid Log compaction has been executing |
|
ns |
Maximum time for which Hybrid Log compaction has been executing |
|
ns |
Minimum time for which Hybrid Log compaction has been executing |
|
ns |
Total time for which Hybrid Log compactions have been executing |
|
bytes |
Current size of the Hybrid Log |
|
ns |
Average time it took to write a page to the device |
|
ns |
Maximum time it took to write a page to the device |
|
ns |
Minimum time it took to write a page to the device |
|
ns |
Average time between consecutive Hybrid Log page allocations |
|
ns |
Maximum time between consecutive Hybrid Log page allocations |
|
ns |
Minimum time between consecutive Hybrid Log page allocations |
|
ns |
Average time it took to read a record from the device |
|
ns |
Maximum time it took to read a record from the device |
|
ns |
Minimum time it took to read a record from the device |
|
count |
Number of times when requested record was in memory |
|
count |
Number of times when requested record was not in memory |
|
percent |
Percent of times when requested record was in memory |
|
percent |
Percent of times when requested record was not in memory |
|
bytes |
Average space wasted due to alignment of Hybrid Log allocation |
|
bytes |
Maximum space wasted due to alignment of Hybrid Log allocation |
|
bytes |
Minimum space wasted due to alignment of Hybrid Log allocation |
|
bytes |
Total space wasted due to alignment of Hybrid Log allocations |
|
bytes |
Average space wasted due to crossing page boundaries of Hybrid Log allocation |
|
bytes |
Maximum space wasted due to crossing page boundaries of Hybrid Log allocation |
|
bytes |
Minimum space wasted due to crossing page boundaries of Hybrid Log allocation |
|
bytes |
Total space wasted due to crossing page boundaries of Hybrid Log allocations |
WAN Replication
Name |
Unit |
Description |
|
ms |
Duration of ongoing delaying, -1 if there is no current delaying |
|
ms |
Timestamp of the last end of delaying the acknowledgments; if this value is bigger than |
|
ms |
Timestamp of the last start of delaying the acknowledgments |
|
count |
Total number of the triggering delaying the WAN acknowledgments (exceeding the invocation threshold) |
|
ms |
Total amount of time delaying the WAN acknowledgments was taking place |
|
count |
Number of checked partitions on the last WAN consistency check |
|
count |
Number of checked partitions on the last WAN consistency check |
|
count |
Number of different Merkle tree leaves on the last WAN consistency check |
|
count |
Number of partitions found to be inconsistent on the last WAN consistency check |
|
count |
Number of entries to synchronize to get the clusters into sync on the last WAN consistency check |
|
count |
Number of dropped entry events |
|
count |
Outbound WAN queue size on this member |
|
count |
Number of entry remove events |
|
count |
Number of entry sync events |
|
count |
Average of the number of records belong the synchronized Merkle tree nodes have |
|
count |
Maximum of the number of records belong the synchronized Merkle tree nodes have |
|
count |
Minimum of the number of records belong the synchronized Merkle tree nodes have |
|
count |
Number of the synchronized Merkle tree nodes |
|
count |
Number of synchronized partitions |
|
count |
Number of partitions to synchronize |
|
count |
Number of synchronized records |
|
ns |
Duration of the last synchronization |
|
count |
Standard deviation of the number of records belong the synchronized Merkle tree nodes have |
|
ns |
Start time of this WAN synchronization |
|
ms |
Total latency of published WAN events from this member |
|
count |
Total number of published WAN events from this member |
|
count |
Number of entry update events |
|
boolean |
The health of an individual WAN target endpoint, where 1 is healthy and 0 is not |
|
count |
Number of attempted WAN replication transmissions that have failed |
User Code Namespaces
Name |
Unit |
Description |
|
ms |
Update time of a user code namespace configuration |
|
count |
Number of resources contained in a namespace |
|
enum |
The Ordinal of the enum type of the resource contained in a namespace |
|
bytes |
The size in bytes of a resource contained in a namespace |
