Configuring Management Center

This chapter explains how you can configure Hazelcast Management Center according to your needs.

The communication between the IMDG/Jet cluster members and the Management Center instance is based on Hazelcast open binary client protocol. The clusters that Management Center should connect to are configured within Management Center as described in the Connecting Management Center to IMDG Members chapter.

In previous versions, the URL of the Management Center instance was configured within the <hazelcast></hazelcast> section of hazelcast.xml. This configuration is now deprecated and ignored.

Providing a License

When starting the Management Center from the command line, a license can be provided using the system property hazelcast.mc.license. For example by using the command line parameter:

java -Dhazelcast.mc.license=<key> -jar hazelcast-management-center-4.2021.02.jar

When this option is used, the license provided takes precedence over any license set and stored previously using the user interface. Previously stored licenses are not affected and will be used again when the Management Center is started without the hazelcast.mc.license property. This also means no new license can be stored when the property is used.

Providing an Extra Classpath

You can also start the Management Center with an extra classpath entry (for example, when using JAAS authentication) by using the following command:

java -cp "hazelcast-management-center-4.2021.02.jar:/path/to/an/extra.jar" com.hazelcast.webmonitor.Launcher

On Windows, the command becomes as follows (semicolon instead of colon):

java -cp "hazelcast-management-center-4.2021.02.jar;/path/to/an/extra.jar" com.hazelcast.webmonitor.Launcher

Configuring the Client Used by Management Center

You can configure the client instance that is used for connecting to the cluster by using the following command line parameters:

-Dhazelcast.mc.client.initial.backoff.millis: Duration, in milliseconds, to wait after the first failure before retrying. It is in milliseconds. The default value is 1000 ms. Set values have to be in range of 1000 to 60000 ms.
-Dhazelcast.mc.client.backoff.multiplier: Factor with which to multiply backoff after a failed retry. Default value is 2. Set values have to be in range of 1 to 10.
-Dhazelcast.mc.client.max.backoff.millis: When backoff reaches this upper bound, it does not increase any more. It is in milliseconds. Default value is 32000 ms. Set values have to be in range of 32000 to 600000 ms.

Note that these parameters apply to all clients that are started by Management Center.

You can also pass in a custom client configuration file to Management Center. These parameters override any configuration you pass as part of adding a new cluster connection. Following is a list of all other client configuration parameters that are overriden by Management Center when creating clients to connect to clusters:

InstanceName: Generated based on the cluster name.
ConnectionStrategyConfig.asyncStart: Set to true. It means that the client starts without waiting to get a cluster connection.
ConnectionStrategyConfig.clusterConnectTimeoutMillis: Set to infinity. The client never gives up trying to connect to the cluster.
NetworkConfig.smartRouting: Set to true. It means that client will open connections to all members.

Configuring Metadata Polling Period of Management Center

Management Center polls the cluster members periodically for their metadata, which includes the list of data structures it has and their configurations. You can change the polling frequency by using the hazelcast.mc.state.reschedule.delay.millis command line parameter. Its default value is 1000.

Configuring Disk Usage

You can control the disk space used by the Management Center to avoid exceeding available disk space. That can be done by configuring the Metrics Persistence.

You can use the hazelcast.mc.metrics.disk.ttl.days system property to configure the Management Center’s metrics persistence. This property specifies Time-to-Live (TTL) setting for each record in the metrics persistence. It is in days and its default value is 1.

It is important to understand that the TTL setting acts as a soft limit and gives you an indirect control over Management Center disk usage. The final disk usage depends on the volume of metrics persistence data generated by your clusters, i.e., the number of cluster members and data structures with enabled statistics that you have in the connected clusters.

You may want to decrease the TTL setting in situations when the Management Center uses too much disk space.

Enabling Health Check Endpoint

When running the Management Center from the command line, you can enable the Health Check endpoint. This endpoint responds with 200 OK HTTP status code once the Management Center web application has started. The endpoint is available on port <Management Center HTTP port> + 1 with context path <Management Center context path>/health (by default, its URL is http://localhost:8081/health). Note that the HTTP protocol is always used for the Health Check endpoint, independently of TLS/SSL settings, and no additional authentication is enforced for it.

If you want to enable the Health Check endpoint, use the following command line argument:

-Dhazelcast.mc.healthCheck.enable=true

Configuring Sessions

This section provides information on how to configure the Management Center sessions for various aspects including timeouts and login/logout operations.

Configuring Session Timeout

If you have started the Management Center from the command line by using the JAR file, by default, the sessions that are inactive for 30 minutes are invalidated. To change this, you can use the -Dhazelcast.mc.session.timeout.seconds command line parameter.

For example, the following command starts the Management Center with a session timeout period of 1 minute:

java -Dhazelcast.mc.session.timeout.seconds=60 -jar hazelcast-management-center-4.2021.02.jar

If you have deployed the Management Center on an application server/container, you can configure the default session timeout period of the application server/container to change the session timeout period for the Management Center. If your server/container allows application specific configuration, you can use it to configure the session timeout period for the Management Center.

Normally, a user account on the Management Center can be used from multiple locations at the same time. If you want to forbid others from logging in, when there’s already someone logged in with the same username, you can start the Management Center with the -Dhazelcast.mc.allowMultipleLogin=false command line parameter.

In order to prevent password guessing attacks, logging in is disabled temporarily after a number of failed login attempts. When not configured explicitly, the default values are used, i.e., logging in is disabled for 5 seconds when a username is failed to log in consecutively 3 times. During this 5 seconds of period, logging in is not allowed even when the correct credentials are used. After 5 seconds, the user will be able to log in using the correct credentials.

Assuming the configuration with the default values, if the failed attempts continue (consecutively 3 times) after the period of disabled login passes, this time the disable period is multiplied by 10: logging in is disabled for 50 seconds. The whole process repeats itself until the user logs in successfully. By default, there’s no upper limit to the disable period, but can be configured by using the -Dhazelcast.mc.maxDisableLoginPeriod parameter.

Here is a scenario, in the given order, with the default values:

You try to login with your credentials consecutively 3 times but failed.
Logging in is disabled and you have to wait for 5 seconds.
After 5 seconds have passed, logging in is enabled.
You try to login with your credentials consecutively 3 times but again failed.
Logging in is disabled again and this time you have to wait for 50 seconds until your next login attempt.
And so on; each 3 consecutive login failures causes the disable period to be multiplied by 10.

You can configure the number of failed login attempts, initial and maximum duration of the disabled login and the multiplier using the following command line parameters:

-Dhazelcast.mc.failedAttemptsBeforeDisableLogin: Number of failed login attempts that cause the logging in to be disabled temporarily. Default value is 3.
-Dhazelcast.mc.initialDisableLoginPeriod: Initial duration for the disabled login in seconds. Default value is 5.
-Dhazelcast.mc.disableLoginPeriodMultiplier: Multiplier used for extending the disable period in case the failed login attempts continue after disable period passes. Default value is 10.
-Dhazelcast.mc.maxDisableLoginPeriod: Maximum amount of time for the disable login period. This parameter does not have a default value. By default, disabled login period is not limited.

If you haven’t allowed multiple simultaneous login attempts explicitly, the first user to login with a username stays logged in until that username explicitly logs out or its session expires. In the meantime, no one else can login with the same username. If you want to force logout for the first user and let the newcomer login, you need to start Management Center with the -Dhazelcast.mc.forceLogoutOnMultipleLogin=true command line parameter.

Configuring and Enabling Security

This section provides information on how to use and manage the Management Center with TLS/SSL and mutual authentication. You will also learn how to force the users to specify passwords that are hard to guess.

Using Management Center with TLS/SSL Only

To encrypt data transmitted over all channels of the Management Center using TLS/SSL, make sure you do all of the following:

Deploy the Management Center on a TLS/SSL enabled container or start it from the command line with TLS/SSL enabled. See the Enabling TLS/SSL section below.
- Another option is to place the Management Center behind a TLS-enabled reverse proxy. In that case, make sure your reverse proxy sets the necessary HTTP header (X-Forwarded-Proto) for resolving the correct protocol.
Enable TLS/SSL communication to the Management Center for your Hazelcast cluster. See the Connecting Members chapter.
If you’re using Clustered JMX on the Management center, enable TLS/SSL for it. See Enabling TLS/SSL for Clustered JMX section.
If you’re using LDAP authentication, make sure you use LDAPS or enable the "Start TLS" field. See the LDAP Authentication section.
If you’re using Active Directory authentication, make sure you use Java’s truststore related system properties. See the Active Directory Authentication section.

You can configure how Management Center treats X-Forwarded-* headers using the system property hazelcast.mc.forwarded.requests.enabled. If its value is set to true, Management Center accepts and treats them as set by a reverse proxy in front of it, otherwise, they are ignored. Its default value is true.

Enabling TLS/SSL When Starting with JAR File

When you start the Management Center from the command line, it serves the pages unencrypted by using "http", by default. To enable TLS/SSL, use the following command line parameters when starting the Management Center:

-Dhazelcast.mc.tls.enabled: Specifies whether TLS/SSL is enabled. Its default value is false (disabled).
-Dhazelcast.mc.tls.keyStore: Path to the keystore.
-Dhazelcast.mc.tls.keyStorePassword: Password of the keystore.
-Dhazelcast.mc.tls.trustStore: Path to the truststore.
-Dhazelcast.mc.tls.trustStorePassword: Password of the truststore.

You can leave the truststore and truststore password values empty to use the system JVM’s own truststore.

The following is an example on how to start the Management Center with TLS/SSL enabled from the command line:

java -Dhazelcast.mc.tls.enabled=true
-Dhazelcast.mc.tls.keyStore=/some/dir/selfsigned.jks
-Dhazelcast.mc.tls.keyStorePassword=yourpassword -jar hazelcast-management-center-4.2021.02.jar

You can access the Management Center from the following HTTPS URL on port 8443: https://localhost:8443.

On the member side, you need to configure the Management Center URL as https://localhost:8443 and also set the following JVM arguments when starting the member:

-Djavax.net.ssl.trustStore=path to your truststore
-Djavax.net.ssl.trustStorePassword=yourpassword

If you plan to use a self-signed certificate, make sure to create a certificate with the hostname of the machine you will deploy the Management Center on. Otherwise, you will see a line similar to the following in the member logs:

javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException:
No subject alternative names matching IP address 127.0.0.1 found

To override the HTTPS port, you can use the -Dhazelcast.mc.https.port command line option when starting the Management Center. For example:

java -Dhazelcast.mc.tls.enabled=true \
     -Dhazelcast.mc.tls.keyStore=/dir/to/certificate.jks \
     -Dhazelcast.mc.tls.keyStorePassword=yourpassword \
     -Dhazelcast.mc.https.port=443 \
     -jar hazelcast-management-center-4.2021.02.jar

This starts the Management Center on HTTPS port 443.

You can encrypt the keystore/truststore passwords and pass them as command line arguments in encrypted form for improved security. See the Variable Replacers section for more information.

Enabling HTTP Port

By default, HTTP port is disabled when you enable TLS. If you want to have an open HTTP port that redirects to the HTTPS port, use the following command line argument:

-Dhazelcast.mc.tls.enableHttpPort=true

Managing TLS Enabled Clusters

If a Hazelcast cluster is configured to use TLS for communication between its members using a self-signed certificate, the Management Center will not be able to perform some of the operations that use the cluster’s HTTP endpoints (such as shutting down a member or getting the thread dump of a member). This is so because self-signed certificates are not trusted by default by the JVM. For these operations to work, you need to configure a truststore containing the public key of the self-signed certificate when starting the JVM of the Management Center using the following command line parameters:

-Dhazelcast.mc.httpClient.tls.trustStore: Path to the truststore.
-Dhazelcast.mc.httpClient.tls.trustStorePassword: Password of the truststore.
-Dhazelcast.mc.httpClient.tls.trustStoreType: Type of the truststore. Its default value is JKS.
-Dhazelcast.mc.httpClient.tls.trustManagerAlgorithm: Name of the algorithm based on which the authentication keys are provided. System default is used if none is provided. You can find out the default by calling the javax.net.ssl.TrustManagerFactory#getDefaultAlgorithm method.

You can encrypt the truststore password and pass it as a command line argument in encrypted form for improved security. See the Variable Replacers section for more information.

By default, JVM also checks for the validity of the hostname of the certificate. If this test fails, you will see a line similar to the following in the Management Center logs:

javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException:
No subject alternative names matching IP address 127.0.0.1 found

If you want to disable this check, start the Management Center with the following command line parameter:

-Dhazelcast.mc.disableHostnameVerification=true

Mutual Authentication

You can configure Management Center to require mutual authentication. With this setup, any client (be it a Web browser or an HTTP client such as curl) needs to present their TLS certificate and the Management Center needs to have its truststore configured so that the Management Center can know which clients it can trust. To enable mutual authentication, you need to use the following command line parameters when starting the Management Center:

-Dhazelcast.mc.tls.mutualAuthentication=REQUIRED

See the below snippet to see the full command to start the Management Center:

java -Dhazelcast.mc.tls.enabled=true \
     -Dhazelcast.mc.tls.trustStore=path to your truststore \
     -Dhazelcast.mc.tls.trustStorePassword=password for your truststore \
     -Dhazelcast.mc.tls.mutualAuthentication=REQUIRED \
     -jar hazelcast-management-center-4.2021.02.jar

The parameter -Dhazelcast.mc.tls.mutualAuthentication has two options:

REQUIRED: If the client does not provide a keystore or the provided keys are not included in the Management Center’s truststore, the client will not be authenticated.
OPTIONAL: If the client does not provide a keystore, it will be authenticated. But if the client provides keys that are not included in the Management Center’s truststore, the client will not be authenticated.

Managing Mutual Authentication Enabled Clusters

If mutual authentication is enabled for the cluster (as described here), the Management Center needs to have a keystore to identify itself. For this, you need to start the Management Center with the following command line parameters:

-Dhazelcast.mc.httpClient.tls.keyStore: Path to the keystore.
-Dhazelcast.mc.httpClient.tls.keyStorePassword: Password of the keystore.
-Dhazelcast.mc.httpClient.tls.keyStoreType: Type of the keystore. Its default value is JKS.
-Dhazelcast.mc.httpClient.tls.keyManagerAlgorithm: Name of the algorithm based on which the authentication keys are provided. System default is used if none is provided. You can find out the default by calling the javax.net.ssl.KeyManagerFactory#getDefaultAlgorithm method.

Excluding Specific TLS/SSL Protocols

When you enable TLS on the Management Center, it will support the clients connecting with any of the TLS/SSL protocols that the JVM supports by default. In order to disable specific protocols, you need to set the -Dhazelcast.mc.tls.excludeProtocols command line argument to a comma separated list of protocols to be excluded from the list of supported protocols. For example, to allow only TLSv1.2, you need to add the following command line argument when starting the Management Center:

-Dhazelcast.mc.tls.excludeProtocols=SSLv3,SSLv2Hello,TLSv1,TLSv1.1

When you specify the above argument, you should see a line similar to the following in the Management Center log:

2017-06-21 12:35:54.856:INFO:oejus.SslContextFactory:Enabled Protocols
[TLSv1.2] of [SSLv2Hello, SSLv3, TLSv1, TLSv1.1, TLSv1.2]

Using OpenSSL

To use OpenSSL with Management Center, you need to add the following command line argument when starting the Management Center:

-Dhazelcast.mc.tls.openSsl=true

When you specify the above argument, Management Center uses Google’s Conscrypt SSL that is built on their fork of OpenSSL, BoringSSL.

If you are using Java 8 and your JVM doesn’t support TLSv1.3, you must exclude TLSv1.3 protocol by passing -Dhazelcast.mc.tls.excludeProtocols="TLSv1.3" command line argument. We recommend that you upgrade your JVM to the latest version of Java 8. Many vendors including Oracle and AdoptOpenJDK (Eclipse Adoptium as its new name) support TLSv1.3 in their latest Java 8 implementations.

Using a Dictionary to Prevent Weak Passwords

In order to prevent certain words from being included in the user passwords, you can start the Management Center with -Dhazelcast.mc.security.dictionary.path command line parameter which points to a text file that contains a word on each line. As a result, the user passwords will not contain any dictionary words, making them harder to guess.

The words in the dictionary need to be at least three characters long in order to be used for checking the passwords. The shorter words are ignored to prevent them from blocking the usage of many password combinations. You can configure the minimum length of words by starting the Management Center with -Dhazelcast.mc.security.dictionary.minWordLength command line parameter and setting it to a number.

An example to start the Management Center using the aforementioned parameters is shown below:

java -Dhazelcast.mc.security.dictionary.path=/usr/MCtext/pwd.txt \
     -Dhazelcast.mc.security.dictionary.minWordLength=3 \
     -jar hazelcast-management-center-4.2021.02.jar

Including and/or Excluding Specific Cipher Suites

When you configure TLS you also can provide which cipher suites Management Center can use for establishing TLS connection. You can include cipher suites with -Dhazelcast.mc.include.cipher.suites and exclude with -Dhazelcast.mc.exclude.cipher.suites system properties during Management Center startup. You can use the exact cipher suite name or a regular expression. For example:

"-Dhazelcast.mc.include.cipher.suites=^SSL_.*$"
"-Dhazelcast.mc.exclude.cipher.suites=^.*_(MD5|SHA|SHA1)$,^TLS_RSA_.*$,^.*_NULL_.*$"

Configuring Logging

Starting with version 4.2020.11, Management Center uses Log4j 2 for its logging. By default, it uses the following configuration:

appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d [%highlight{%5p}{FATAL=red, ERROR=red, WARN=yellow, INFO=green, DEBUG=magenta}] [%style{%t{1.}}{cyan}] [%style{%c{1.}}{blue}]: %m%n
appender.console.filter.threshold.type = ThresholdFilter
appender.console.filter.threshold.level = ${sys:hazelcast.mc.log.level:-all}

logger.hazelcast.name = com.hazelcast
logger.hazelcast.level = ${sys:hazelcast.mc.log.level:-error}
logger.hazelcast.additivity = false
logger.hazelcast.appenderRef.rolling.ref = STDOUT

logger.mc.name = com.hazelcast.webmonitor
logger.mc.level = ${sys:hazelcast.mc.log.level:-info}
logger.mc.additivity = false
logger.mc.appenderRef.rolling.ref = STDOUT

logger.jetty.name = org.eclipse.jetty
logger.jetty.level = ${sys:hazelcast.mc.log.level:-warn}
logger.jetty.additivity = false
logger.jetty.appenderRef.rolling.ref = STDOUT

logger.spring.name = org.springframework
logger.spring.level = ${sys:hazelcast.mc.log.level:-warn}
logger.spring.additivity = false
logger.spring.appenderRef.rolling.ref = STDOUT

logger.hibernatevalidator.name = org.hibernate.validator
logger.hibernatevalidator.level = ${sys:hazelcast.mc.log.level:-warn}
logger.hibernatevalidator.additivity = false
logger.hibernatevalidator.appenderRef.rolling.ref = STDOUT

logger.flywaydb.name = org.flywaydb
logger.flywaydb.level = ${sys:hazelcast.mc.log.level:-warn}
logger.flywaydb.additivity = false
logger.flywaydb.appenderRef.rolling.ref = STDOUT

logger.hikari.name = com.zaxxer.hikari
logger.hikari.level = ${sys:hazelcast.mc.log.level:-warn}
logger.hikari.additivity = false
logger.hikari.appenderRef.rolling.ref = STDOUT

# Otherwise every resolved exception is logged
logger.springerror.name = org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver
logger.springerror.level = ${sys:hazelcast.mc.log.level:-error}
logger.springerror.additivity = false
logger.springerror.appenderRef.rolling.ref = STDOUT

rootLogger.level = ${sys:hazelcast.mc.log.level:-info}
rootLogger.appenderRef.stdout.ref = STDOUT

To change the logging level for all loggers, e.g., enabling debug logs, you can start Management Center with -Dhazelcast.mc.log.level command line option. For example, to enable debug logs, start Management Center with -Dhazelcast.mc.log.level=debug command line parameter.

To further customize the logging configuration, you can create a custom Log4j configuration file and start Management Center with the -Dlog4j.configurationFile option pointing to your configuration file.

For example, you can create a file named log4j2-custom.properties with the following content and set logging level to DEBUG. To use this file as the logging configuration, you need to start Management Center with the -Dlog4j.configurationFile=/path/to/your/log4j2-custom.properties command line parameter:

appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d [%highlight{%5p}{FATAL=red, ERROR=red, WARN=yellow, INFO=green, DEBUG=magenta}] [%style{%t{1.}}{cyan}] [%style{%c{1.}}{blue}]: %m%n
appender.console.filter.threshold.type = ThresholdFilter
appender.console.filter.threshold.level = ${sys:hazelcast.mc.log.level:-all}

rootLogger.level = debug
rootLogger.appenderRef.stdout.ref = STDOUT

To write log messages into rolling log files (in parallel with printing them into the console), you can use a similar Log4j configuration file:

appender.console.type=Console
appender.console.name=STDOUT
appender.console.layout.type=PatternLayout
appender.console.layout.pattern=%d [%highlight{${LOG_LEVEL_PATTERN:-%5p}}{FATAL=red, ERROR=red, WARN=yellow, INFO=green, DEBUG=magenta}] [%style{%t{1.}}{cyan}] [%style{%c{1.}}{blue}]: %m%n

appender.rolling.type=RollingFile
appender.rolling.name=RollingFile
appender.rolling.fileName=${sys:user.home}/mc-logs/mc.log
appender.rolling.filePattern=${sys:user.home}/mc-logs/mc.%d{yyyy-MM-dd}.log
appender.rolling.layout.type=PatternLayout
appender.rolling.layout.pattern=%d [%5p] [%t] [%c{.1}]: %m%n
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy

rootLogger.level=info
rootLogger.appenderRef.stdout.ref=STDOUT
rootLogger.appenderRef.rolling.ref=RollingFile

Enabling Audit Logging

You may enable additional security audit logging by using the -Dhazelcast.mc.auditlog.enabled=true command line argument. Log entries from the audit logging will be marked with the hazelcast.auditlog logging category, abbreviated as h.auditlog in logs.

An example log entry looks like the following:

2020-10-13 09:57:54,803 [ INFO] [qtp973576304-35] [h.auditlog]: MC-2001 [Auth]:User logged in:{username=JohnHallaign}n}

MC-2001 [Auth] you see in this example represents the log’s type. The following table lists the current log categories along with their types:

Event Category Log Type/Description

Event Category	Log Type/Description
Management Center Configuration Logs	`MC-0001 [Config]`: Metrics Persistence is enabled. `MC-0002 [Config]`: Metrics Persistence is disabled. `MC-0003 [Config]`: User is created. `MC-0004 [Config]`: User is edited. `MC-0005 [Config]`: User’s password is changed. `MC-0006 [Config]`: User is deleted. `MC-0009 [Config]`: License is set.
Cluster Configuration Logs	`MC-1001 [Cluster Config]`: Map’s configuration is changed. `MC-1003 [Cluster Config]`: Cluster’s state is changed. `MC-1004 [Cluster Config]`: Cluster is shut down. `MC-1005 [Cluster Config]`: Member is shut down. `MC-1006 [Cluster Config]`: Lite member is promoted. `MC-1007 [Cluster Config]`: Cluster version is changed.
Authentication Logs	`MC-2001 [Auth]`: User logs in. `MC-2002 [Auth]`: User logs out. `MC-2003 [Auth]`: Login failures.
Scripting Logs	`MC-3001 [Script]`: Script is executed on a member.
Console Logs	`MC-4001 [Console]`: Console command is executed on the cluster.
Map/Cache Logs	`MC-5001 [Browser]`: User browses through a map screen in Management Center. `MC-5002 [Browser]`: User browses through a cache screen in Management Center.
Hot Restart Logs	`MC-6001 [Hot Restart]`: Force start is run. `MC-6002 [Hot Restart]`: Partial start is run. `MC-6003 [Hot Restart]`: Hot Restart backup operation is triggered. `MC-6004 [Hot Restart]`: Hot Restart backup operation is interrupted.
WAN Replication Logs	`MC-7001 [WAN]`: WAN configuration is added. `MC-7002 [WAN]`: WAN consistency check operation is run. `MC-7003 [WAN]`: WAN synchronization on a map is run. `MC-7004 [WAN]`: State of the WAN publisher is changed. `MC-7005 [WAN]`: Clear operation for the WAN events queue is run.
CP Subsystem Logs	`MC-8001 [CP Subsystem]`: Member is promoted to be a CP subsystem member. `MC-8002 [CP Subsystem]`: Member is removed from CP subsystem. `MC-8003 [CP Subsystem]`: CP subsystem is reset.
Jet Logs	`MC-9001 [Jet]`: Jet job is restarted. `MC-9002 [Jet]`: Jet job is suspended. `MC-9003 [Jet]`: Jet job is resumed. `MC-9004 [Jet]`: Jet job is cancelled. `MC-9005 [Jet]`: Jet snapshot is deleted. `MC-9006 [Jet]`: Jet snapshot is exported. `MC-9007 [Jet]`: Jet job is cancelled and snapshot is exported.

Management Center Configuration Logs

MC-0001 [Config]: Metrics Persistence is enabled.
MC-0002 [Config]: Metrics Persistence is disabled.
MC-0003 [Config]: User is created.
MC-0004 [Config]: User is edited.
MC-0005 [Config]: User’s password is changed.
MC-0006 [Config]: User is deleted.
MC-0009 [Config]: License is set.

Cluster Configuration Logs

MC-1001 [Cluster Config]: Map’s configuration is changed.
MC-1003 [Cluster Config]: Cluster’s state is changed.
MC-1004 [Cluster Config]: Cluster is shut down.
MC-1005 [Cluster Config]: Member is shut down.
MC-1006 [Cluster Config]: Lite member is promoted.
MC-1007 [Cluster Config]: Cluster version is changed.

Authentication Logs

MC-2001 [Auth]: User logs in.
MC-2002 [Auth]: User logs out.
MC-2003 [Auth]: Login failures.

Scripting Logs

MC-3001 [Script]: Script is executed on a member.

Console Logs

MC-4001 [Console]: Console command is executed on the cluster.

Map/Cache Logs

MC-5001 [Browser]: User browses through a map screen in Management Center.
MC-5002 [Browser]: User browses through a cache screen in Management Center.

Hot Restart Logs

MC-6001 [Hot Restart]: Force start is run.
MC-6002 [Hot Restart]: Partial start is run.
MC-6003 [Hot Restart]: Hot Restart backup operation is triggered.
MC-6004 [Hot Restart]: Hot Restart backup operation is interrupted.

WAN Replication Logs

MC-7001 [WAN]: WAN configuration is added.
MC-7002 [WAN]: WAN consistency check operation is run.
MC-7003 [WAN]: WAN synchronization on a map is run.
MC-7004 [WAN]: State of the WAN publisher is changed.
MC-7005 [WAN]: Clear operation for the WAN events queue is run.

CP Subsystem Logs

MC-8001 [CP Subsystem]: Member is promoted to be a CP subsystem member.
MC-8002 [CP Subsystem]: Member is removed from CP subsystem.
MC-8003 [CP Subsystem]: CP subsystem is reset.

Jet Logs

MC-9001 [Jet]: Jet job is restarted.
MC-9002 [Jet]: Jet job is suspended.
MC-9003 [Jet]: Jet job is resumed.
MC-9004 [Jet]: Jet job is cancelled.
MC-9005 [Jet]: Jet snapshot is deleted.
MC-9006 [Jet]: Jet snapshot is exported.
MC-9007 [Jet]: Jet job is cancelled and snapshot is exported.

To write security audit logging into separate rolling log files, you can use a similar Log4j configuration file:

appender.console.type=Console
appender.console.name=STDOUT
appender.console.layout.type=PatternLayout
appender.console.layout.pattern=%d [%highlight{${LOG_LEVEL_PATTERN:-%5p}}{FATAL=red, ERROR=red, WARN=yellow, INFO=green, DEBUG=magenta}] [%style{%t{1.}}{cyan}] [%style{%c{1.}}{blue}]: %m%n

appender.audit.type=RollingFile
appender.audit.name=AuditFile
appender.audit.fileName=${sys:user.home}/mc-logs/audit.log
appender.audit.filePattern=${sys:user.home}/mc-logs/audit.%d{yyyy-MM-dd}.log
appender.audit.layout.type=PatternLayout
appender.audit.layout.pattern=%d [%5p] [%t] [%c{.1}]: %m%n
appender.audit.policies.type = Policies
appender.audit.policies.time.type = TimeBasedTriggeringPolicy

logger.audit.name=hazelcast.auditlog
logger.audit.level=info
logger.audit.additivity=false
logger.audit.appenderRef.audit.ref=AuditFile

rootLogger.level=info
rootLogger.appenderRef.stdout.ref=STDOUT

Using Variable Replacers

Variable replacers are used to replace custom strings during loading the configuration, either passed as command line arguments, used during UI based Management Center configuration, or configured with Configuration tool. They can be used to mask sensitive information such as usernames and passwords. Of course their usage is not limited to security related information.

Variable replacers implement the interface com.hazelcast.webmonitor.configreplacer.spi.ConfigReplacer and they are configured via the following command line arguments:

-Dhazelcast.mc.configReplacer.class: Full class name of the replacer.
-Dhazelcast.mc.configReplacer.failIfValueMissing: Specifies whether the loading configuration process stops when a replacement value is missing. It is an optional attribute and its default value is true.
Additional command line arguments specific to each replacer implementation. All of the properties for the built-in replacers are explained in the upcoming sections.

The following replacer classes are provided by Hazelcast as example implementations of the ConfigReplacer interface. Note that you can also implement your own replacers.

EncryptionReplacer
PropertyReplacer

Each example replacer is explained in the following sections.

EncryptionReplacer

This example EncryptionReplacer replaces the encrypted variables with its plain form. The secret key for encryption/decryption is generated from a password which can be a value in a file and/or environment specific values, such as MAC address and actual user data.

Its full class name is com.hazelcast.webmonitor.configreplacer.EncryptionReplacer and the replacer prefix is ENC. Here are the properties used to configure this example replacer:

hazelcast.mc.configReplacer.prop.cipherAlgorithm: Cipher algorithm used for the encryption/decryption. Its default value is AES.
hazelcast.mc.configReplacer.prop.keyLengthBits: Length (in bits) of the secret key to be generated. Its default value is 128.
hazelcast.mc.configReplacer.prop.passwordFile: Path to a file whose content should be used as a part of the encryption password. When the property is not provided, no file is used as a part of the password. Its default value is null.
hazelcast.mc.configReplacer.prop.passwordNetworkInterface: Name of the network interface whose MAC address should be used as a part of the encryption password. When the property is not provided no network interface property is used as a part of the password. Its default value is null.
hazelcast.mc.configReplacer.prop.passwordUserProperties: Specifies whether the current user properties (user.name and user.home) should be used as a part of the encryption password. Its default value is true.
hazelcast.mc.configReplacer.prop.saltLengthBytes: Length (in bytes) of a random password salt. Its default value is 8.
hazelcast.mc.configReplacer.prop.secretKeyAlgorithm: Name of the secret key algorithm to be associated with the generated secret key. Its default value is AES.
hazelcast.mc.configReplacer.prop.secretKeyFactoryAlgorithm: Algorithm used to generate a secret key from a password. Its default value is PBKDF2WithHmacSHA256.
hazelcast.mc.configReplacer.prop.securityProvider: Name of a Java Security Provider to be used for retrieving the configured secret key factory and the cipher. Its default value is null.

Older Java versions may not support all the algorithms used as defaults. Use the property values supported by your Java version.

As a usage example, let’s create a password file and generate the encrypted strings out of this file as shown below:

Create the password file: echo '/Za-uG3dDfpd,5.-' > /opt/master-password

Define the encrypted variables:

java -cp hazelcast-management-center-4.2021.02.jar \
     -Dhazelcast.mc.configReplacer.prop.passwordFile=/opt/master-password \
     -Dhazelcast.mc.configReplacer.prop.passwordUserProperties=false \
     com.hazelcast.webmonitor.configreplacer.EncryptionReplacer \
     "aPasswordToEncrypt"

Output:

$ENC{wJxe1vfHTgg=:531:WkAEdSi//YWEbwvVNoU9mUyZ0DE49acJeaJmGalHHfA=}

Configure the replacer and provide the encrypted variables as command line arguments while starting the Management Center:

java \
 -Dhazelcast.mc.configReplacer.class=com.hazelcast.webmonitor.configreplacer.EncryptionReplacer \
 -Dhazelcast.mc.configReplacer.prop.passwordFile=/opt/master-password \
 -Dhazelcast.mc.configReplacer.prop.passwordUserProperties=false \
 -Dhazelcast.mc.tls.enabled=true \
 -Dhazelcast.mc.tls.keyStore=/opt/mc.keystore \
 -Dhazelcast.mc.tls.keyStorePassword='$ENC{wJxe1vfHTgg=:531:WkAEdSi//YWEbwvVNoU9mUyZ0DE49acJeaJmGalHHfA=}' \
 -jar hazelcast-management-center-4.2021.02.jar

PropertyReplacer

PropertyReplacer replaces variables by properties with the given name. Usually the system properties are used, e.g., ${user.name}.

Its full class name is com.hazelcast.webmonitor.configreplacer.PropertyReplacer and the replacer prefix is empty string ("").

Implementing Custom Replacers

You can also provide your own replacer implementations. All replacers have to implement the three methods that have the same signatures as the methods of the following interface:

import java.util.Properties;

public interface ConfigReplacer {
    void init(Properties properties);
    String getPrefix();
    String getReplacement(String maskedValue);
}