A newer version of Hazelcast Platform is available.

View latest

Ping Failure Detector

6.0-SNAPSHOT 5.5 5.4 5.3 5.2 5.1 5.0

The Ping Failure Detector may be configured in addition to one of Deadline and Phi Accrual Failure Detectors. It operates at Layer 3 of the OSI protocol and provides much quicker and more deterministic detection of hardware and other lower level events. This detector may be configured to perform an extra check after a member is suspected by one of the other detectors, or it can work in parallel, which is the default. This way hardware and network level issues are detected more quickly.

This failure detector is based on InetAddress.isReachable(). When the JVM process has enough permissions to create RAW sockets, the implementation chooses to rely on ICMP Echo requests. This is preferred.

If there are not enough permissions, it can be configured to fallback on attempting a TCP Echo on port 7. In the latter case, both a successful connection or an explicit rejection is treated as "Host is Reachable". Or, it can be forced to use only RAW sockets. This is not preferred as each call creates a heavyweight socket and moreover the Echo service is typically disabled.

For the Ping Failure Detector to rely only on ICMP Echo requests, there are some criteria that need to be met.

Requirements and Linux/Unix Configuration

  • Supported OS: as of Java 1.8 only Linux/Unix environments are supported. This detector relies on ICMP, i.e., the protocol behind the ping command. It tries to issue the ping attempts periodically, and their responses are used to determine the reachability of the remote member. However, you cannot simply create an ICMP Echo Request because these type of packets do not rely on any of the preexisting transport protocols such as TCP. In order to create such a request, you must have the privileges to create RAW sockets (see https://linux.die.net/man/7/raw). Most operating systems allow this to the root users, however Unix based ones are more flexible and allow the use of custom privileges per process instead of requiring root access. Therefore, this detector is supported only on Linux.

  • The Java executable must have the cap_net_raw capability. As described in the above requirement, on Linux, you have the ability to define extra capabilities to a single process, which would allow the process to interact with the RAW sockets. This interaction is achieved via the capability cap_net_raw (see https://linux.die.net/man/7/capabilities). To enable this capability run the following command:

    sudo setcap cap_net_raw=+ep <JDK_HOME>/jre/bin/java

  • When running with custom capabilities, the dynamic linker on Linux rejects loading the libs from untrusted paths. Since you have now cap_net_raw as a custom capability for a process, it becomes suspicious to the dynamic linker and throws an error: java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory

    • To overcome this rejection, the <JDK_HOME>/jre/lib/amd64/jli/ path needs to be added in the ld.conf. Run the following command to do this: echo "<JDK_HOME>/jre/lib/amd64/jli/" >> /etc/ld.so.conf.d/java.conf && sudo ldconfig

  • ICMP Echo Requests must not be blocked by the receiving hosts. /proc/sys/net/ipv4/icmp_echo_ignore_all set to 0. Run the following command:

    echo 0 > /proc/sys/net/ipv4/icmp_echo_ignore_all

If any of the above criteria isn’t met, then the isReachable always falls back on TCP Echo attempts on port 7.

To be able to use the Ping Failure Detector, you can configure it using the icmp element in your Hazelcast declarative configuration file, e.g., hazelcast.xml. An example is shown below:

  • XML

  • YAML

<hazelcast>
    <network>
    ...
        <failure-detector>
            <icmp enabled="true">
                <timeout-milliseconds>1000</timeout-milliseconds>
                <fail-fast-on-startup>true</fail-fast-on-startup>
                <interval-milliseconds>1000</interval-milliseconds>
                <max-attempts>3</max-attempts>
                <parallel-mode>true</parallel-mode>
                <ttl>0</ttl>
            </icmp>
        </failure-detector>
    </network>
    ...
</hazelcast>
hazelcast:
  network:
    failure-detector:
      icmp:
        enabled: true
        timeout-milliseconds: 1000
        fail-fast-on-startup: true
        interval-milliseconds: 1000
        max-attempts: 3
        parallel-mode: true
        ttl: 0

The following are the element and attribute descriptions:

  • enabled: Specifies whether the legacy ICMP detection mode is enabled; works cooperatively with the existing failure detector and only kicks-in after a pre-defined period has passed with no heartbeats from a member. Its default value is false.

  • parallel-mode: Specifies whether the parallel ping detector is enabled; works separately from the other detectors. Its default value is true.

  • timeout-milliseconds: Number of milliseconds until a ping attempt is considered failed if there was no reply. Its default value is 1000 milliseconds.

  • max-attempts: Maximum number of ping attempts before the member/node gets suspected by the detector. Its default value is 2.

  • interval-milliseconds: Interval, in milliseconds, between each ping attempt. 1000ms (1 sec) is also the minimum interval allowed. Its default value is 1000 milliseconds.

  • ttl: Maximum number of hops the packets should go through. Its default value is 0.

  • fail-fast-on-startup: Specifies whether the cluster member fails to start if it is unable to action an ICMP ping command when ICMP is enabled. Failure is usually due to OS level restrictions.

In the above example configuration, the Ping detector attempts 3 pings, one every second and waits up to 1 second for each to complete. If after 3 seconds, there was no successful ping, the member gets suspected.

To enforce the Requirements, the property hazelcast.icmp.echo.fail.fast.on.startup can also be set to true, in which case, if any of the requirements isn’t met, Hazelcast fails to start.

Below is a summary table of all possible configuration combinations of the ping failure detector.

Table 1. Ping Failure Detector Possible Configuration Combinations
ICMP Parallel Fail-Fast Description Linux Windows macOS

false

false

false

Completely disabled

N/A

N/A

N/A

true

false

false

Legacy ping mode. This works hand-to-hand with the OSI Layer 7 failure detector (see. phi or deadline in the sections above). Ping in this mode only kicks in after a period when there are no heartbeats received, in which case the remote Hazelcast member is pinged up to a configurable count of attempts. If all those attempts fail, the member gets suspected. You can configure this attempt count using the max-attempts configuration element listed above.

Supported ICMP Echo if available - Falls back on TCP Echo on port 7

Supported TCP Echo on port 7

Supported ICMP Echo if available - Falls back on TCP Echo on port 7

true

true

false

Parallel ping detector, works in parallel with the configured failure detector. Checks periodically if members are live (OSI Layer 3) and suspects them immediately, regardless of the other detectors.

Supported ICMP Echo if available - Falls back on TCP Echo on port 7

Supported TCP Echo on port 7

Supported ICMP Echo if available - Falls back on TCP Echo on port 7

true

true

true

Parallel ping detector, works in parallel with the configured failure detector. Checks periodically if members are live (OSI Layer 3) and suspects them immediately, regardless of the other detectors.

Supported - Requires OS Configuration Enforcing ICMP Echo if available - No start up if not available

Not Supported

Not Supported - Requires root privileges
lifebuoy

Help and support