Configure diagnostic logging

Hazelcast offers an extended set of diagnostics plugins for both Hazelcast members and clients. A dedicated log file is used to write the diagnostics content, and a rolling file approach is used to prevent taking up too much disk space.

Using the Operator, you can switch diagnostic logging on and off at runtime (known as dynamic configuration) without having to restart the cluster.

This section explains how to configure diagnostics for Hazelcast clusters managed by the Hazelcast Platform Operator, including enabling diagnostics, setting auto turn-off to avoid disk overuse, forwarding diagnostics to your logging framework, and persisting diagnostics to a Persistent Volume Claim (PVC).

Enable diagnostics using the CRD

To enable diagnostics, use the Hazelcast custom resource (CRD):

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast
spec:
  clusterSize: 3
  licenseKeySecretName: hazelcast-license-key
  diagnostics:
    outputType: STDOUT

The following properties are available for configuration:

outputType: choose the output destination (e.g. FILE or STDOUT). The default is STDOUT in Operator.
maxRolledFileSize / maxRolledFileCount: cap file size and retention.
pluginProperties: pass the same diagnostics system properties shown above (e.g. enable plugins or adjust their periods).
autoOffTimerInMinutes: configure automatic turn-off (see below).
pvcName: bind diagnostics to a PVC (see below).
fileNamePrefix: add a custom prefix.

For more information, see: Diagnostics. The diagnostics file rolls by default (10 files, 50 MB each) and contains plugin sections. For information about the default plugins, see Diagnostics Plugins.

Configure auto turn-off

To prevent diagnostic logs from running indefinitely and having a performance impact on the cluster, set the Operator’s autoOffTimerInMinutes to automatically disable diagnostic logging after the specified number of minutes. See Diagnostics.

You can combine this with rolling limits: maxRolledFileSize, maxRolledFileCount.

Forward diagnostics to a logging framework

Instead of file output, you can forward diagnostics to your logging framework by setting the value of outputType to LOGGER.

Then configure the com.hazelcast.diagnostics logger (e.g. a Log4j2 logger and rolling appender) to control size and rollover using time/size policies. For information, see Diagnostics using Logging Frameworks.

Persist diagnostics to a PVC

To retain diagnostics across pod restarts and isolate them from the container stdout, mount a Persistent Volume Claim (PVC) and direct diagnostics there:

Set the diagnostics output to files.
Specify the pvcName in the diagnostics config to a PVC mounted in the persistenceVolumeClaims. This instructs the Operator to use the named PVC for diagnostics storage.

Example

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast
spec:
  clusterSize: 3
  licenseKeySecretName: hazelcast-license-key
  diagnostics:
    outputType: FILE
    maxRolledFileCount: 10
    maxRolledFileSize: '50M'
    pvcName: diagnostics-pvc-volume
  persistenceVolumeClaims:
    - accessModes: [ "ReadWriteOnce" ]
      requestStorage: 1Gi
      storageClassName: "standard"
      name: diagnostics-pvc-volume

System log diagnostics (cluster activity and optional partition migration details) can be enabled/tuned via hazelcast.diagnostics.systemlog.enabled and hazelcast.diagnostics.systemlog.partitions if needed. For more information, see SystemLog.

Configure diagnostics via system properties

Diagnostic logging can also be configured statically via system properties set in Hazelcast Platform, and enabled with rolling log controls and a specified output destination.

However, if you enable or disable diagnostic logging using system properties, these settings will override any configuration settings applied using the CRD, and you will be unable to switch logging on and off dynamically at runtime.

For more information about system properties, see Enable diagnostics.