Discovery SPI
By default, Hazelcast is bundled with multiple ways to define and find other members in the same network. Commonly used, especially with development, is the Multicast discovery. This sends out a multicast request to a network segment and awaits other members to answer with their IP addresses. In addition, Hazelcast supports a number of built-in discovery strategies described in the Discovery Mechanisms section.
Since there is an ever growing number of public and private cloud environments, as well as numerous Service Discovery systems in the wild, Hazelcast provides cloud or service discovery vendors with the option to implement their own discovery strategy.
Over the course of this section, we will build a simple
discovery strategy based on the /etc/hosts file.
Discovery SPI Interfaces and Classes
The Hazelcast Discovery SPI (Member Discovery Extensions) consists of multiple interfaces and abstract classes. In the following subsections, we will have a quick look at all of them and shortly introduce the idea and usage behind them. The example will follow in the next section, Discovery Strategy.
DiscoveryStrategy: Implement
The com.hazelcast.spi.discovery.DiscoveryStrategy interface is
the main entry point for vendors to implement their corresponding member discovery strategies.
Its main purpose is to return discovered members on request.
The com.hazelcast.spi.discovery.DiscoveryStrategy interface also offers
light lifecycle capabilities for setup and teardown logic (for example, opening or closing sockets or REST API clients).
DiscoveryStrategys can also do automatic registration / de-registration on
service discovery systems if necessary.
You can use the provided DiscoveryNode that is passed to
the factory method to retrieve local addresses and ports, as well as metadata.
AbstractDiscoveryStrategy: Abstract Class
The com.hazelcast.spi.discovery.AbstractDiscoveryStrategy is a convenience abstract class meant to
ease the implementation of strategies. It basically provides additional support for
reading / resolving configuration properties and empty implementations of lifecycle methods if unnecessary.
DiscoveryStrategyFactory: Factory Contract
The com.hazelcast.spi.discovery.DiscoveryStrategyFactory interface describes
the factory contract that creates a certain DiscoveryStrategy.
DiscoveryStrategyFactory s are registered automatically at startup of
a Hazelcast member or client whenever they are found in the classpath.
For automatic discovery, factories need to announce themselves as
SPI services using a resource file according to the
Java Service Provider Interface.
The service registration file must be part of the JAR file, located under
META-INF/services/com.hazelcast.spi.discovery.DiscoveryStrategyFactory, and consist of
a line with the full canonical class name of the DiscoveryStrategy per provided strategy implementation.
DiscoveryNode: Describe a Member
The com.hazelcast.spi.discovery.DiscoveryNode abstract class describes a member in the Discovery SPI.
It is used for multiple purposes, since it will be returned from strategies for discovered members.
It is also passed to DiscoveryStrategyFactorys factory method to define
the local member itself if created on a Hazelcast member; on Hazelcast clients, null is passed.
SimpleDiscoveryNode: Default DiscoveryNode
com.hazelcast.spi.discovery.SimpleDiscoveryNode is a default implementation of the DiscoveryNode.
It is meant for convenience use of the Discovery SPI and can be returned from
vendor implementations if no special needs are required.
NodeFilter: Filter Members
You can configure com.hazelcast.spi.discovery.NodeFilter before startup and
you can implement logic to do additional filtering of members.
This might be necessary if query languages for discovery strategies are not
expressive enough to describe members or to overcome inefficiencies of strategy implementations.
The DiscoveryStrategy vendor does not need to take possibly configured filters into account
as their use is transparent to the strategies.
|
DiscoveryService: Support In Integrator Systems
A com.hazelcast.spi.discovery.integration.DiscoveryService is part of the integration domain.
DiscoveryStrategy vendors do not need to implement DiscoveryService because
it is meant to support the Discovery SPI in situations where vendors integrate Hazelcast into
their own systems or frameworks. Certain needs might be necessary as part of the classloading or
Java Service Provider Interface lookup.
DiscoveryServiceProvider: Provide a DiscoveryService
Use the com.hazelcast.spi.discovery.integration.DiscoveryServiceProvider to provide
a DiscoveryService to the Hazelcast discovery subsystem. Configure the provider with the Hazelcast configuration API.
Discovery Strategy
This subsection walks through the implementation of a simple DiscoveryStrategy and its necessary setup.
Discovery Strategy Example
The example strategy uses the local /etc/hosts (and on Windows it uses
the equivalent to the *nix hosts file named %SystemRoot%\system32\drivers\etc\hosts) to
lookup IP addresses of different hosts. The strategy implementation expects hosts to be configured with
hostname sub-groups under the same domain. So far to theory, let’s get into it.
The full example’s source code can be found here.
Configuring Site Domain
As a first step we do some basic configuration setup. We want the user to be able to
configure the site domain for the discovery inside the hosts file, therefore we define
a configuration property called site-domain. The configuration is not optional:
you need to configure it before the creation of the HazelcastInstance, either
via Hazelcast’s declarative or programmatic configuration.
It is recommended that you keep all defined properties in a separate configuration class as
public constants (public static final) with sufficient documentation.
This allows users to easily look up possible configuration values.
public final class HostsDiscoveryConfiguration {
public static final PropertyDefinition DOMAIN = new SimplePropertyDefinition("site-domain", PropertyTypeConverter.STRING);
private HostsDiscoveryConfiguration() {
}
}An additional ValueValidator could be passed to the definition to make sure
the configured value looks like a domain or has a special format.
Creating Discovery
As the second step we create the very simple DiscoveryStrategyFactory implementation class.
To keep things clear we are going to name the discovery strategy after its purpose:
looking into the hosts file.
public class HostsDiscoveryStrategyFactory implements DiscoveryStrategyFactory {
private static final Collection<PropertyDefinition> PROPERTIES = singletonList(HostsDiscoveryConfiguration.DOMAIN);
@Override
public Class<? extends DiscoveryStrategy> getDiscoveryStrategyType() {
return HostsDiscoveryStrategy.class;
}
@Override
public DiscoveryStrategy newDiscoveryStrategy(DiscoveryNode discoveryNode, ILogger logger, Map<String, Comparable> properties) {
return new HostsDiscoveryStrategy(logger, properties);
}
@Override
public Collection<PropertyDefinition> getConfigurationProperties() {
return PROPERTIES;
}
}This factory now defines properties known to the discovery strategy implementation and
provides a clean way to instantiate it. While creating the HostsDiscoveryStrategy we ignore
the passed DiscoveryNode since this strategy does not support automatic registration of new members.
In cases where the strategy does not support registration, the environment has to handle this in some provided way.
Remember that, when created on a Hazelcast client, the provided DiscoveryNode is null, as there is no
local member in existence.
|
Next, we register the DiscoveryStrategyFactory to make Hazelcast pick it up automatically at startup.
As described earlier, this is done according to the
Java Service Provider Interface specification.
The filename is the name of the interface itself. Therefore we create a new resource file called
com.hazelcast.spi.discovery.DiscoveryStrategyFactory and place it under META-INF/services.
The content is the full canonical class name of our factory implementation.
com.hazelcast.examples.spi.discovery.HostsDiscoveryStrategyFactoryIf our JAR file contains multiple factories, each consecutive line can define another
full canonical DiscoveryStrategyFactory implementation class name.
Implementing Discovery Strategy
Now comes the interesting part. We are going to implement the discovery itself. The previous parts we did are normally pretty similar for all strategies aside from the configuration properties itself. However, implementing the discovery heavily depends on the way the strategy has to come up with IP addresses of other Hazelcast members.
Extending The AbstractDiscoveryStrategy
For ease of implementation, we back our implementation by extending the AbstractDiscoveryStrategy and
only implementing the absolute minimum ourselves.
public class HostsDiscoveryStrategy extends AbstractDiscoveryStrategy {
private static final String HOSTS_NIX = "/etc/hosts";
private static final String HOSTS_WINDOWS = "%SystemRoot%\\system32\\drivers\\etc\\hosts";
private final String siteDomain;
HostsDiscoveryStrategy(ILogger logger, Map<String, Comparable> properties) {
super(logger, properties);
this.siteDomain = getOrNull("discovery.hosts", HostsDiscoveryConfiguration.DOMAIN);
}
@Override
public Iterable<DiscoveryNode> discoverNodes() {
List<String> assignments = filterHosts();
return mapToDiscoveryNodes(assignments);
}
private List<String> filterHosts() {
String os = System.getProperty("os.name");
String hostsPath;
if (os.contains("Windows")) {
hostsPath = HOSTS_WINDOWS;
} else {
hostsPath = HOSTS_NIX;
}
File hosts = new File(hostsPath);
List<String> lines = readLines(hosts);
List<String> assignments = new ArrayList<String>();
for (String line : lines) {
if (matchesDomain(line)) {
assignments.add(line);
}
}
return assignments;
}
private Iterable<DiscoveryNode> mapToDiscoveryNodes(List<String> assignments) {
Collection<DiscoveryNode> discoveredNodes = new ArrayList<DiscoveryNode>();
for (String assignment : assignments) {
String address = sliceAddress(assignment);
String hostname = sliceHostname(assignment);
Map<String, String> attributes = Collections.singletonMap("hostname", hostname);
InetAddress inetAddress = mapToInetAddress(address);
Address addr = new Address(inetAddress, NetworkConfig.DEFAULT_PORT);
discoveredNodes.add(new SimpleDiscoveryNode(addr, attributes));
}
return discoveredNodes;
}
private List<String> readLines(File hosts) {
try {
List<String> lines = new ArrayList<String>();
BufferedReader reader = new BufferedReader(new FileReader(hosts));
String line;
while ((line = reader.readLine()) != null) {
line = line.trim();
if (!line.startsWith("#")) {
lines.add(line.trim());
}
}
return lines;
} catch (IOException e) {
throw new RuntimeException("Could not read hosts file", e);
}
}
private boolean matchesDomain(String line) {
if (line.isEmpty()) {
return false;
}
String hostname = sliceHostname(line);
return hostname.endsWith("." + siteDomain);
}
private String sliceAddress(String assignment) {
String[] tokens = assignment.split("\\p{javaSpaceChar}+");
if (tokens.length < 1) {
throw new RuntimeException("Could not find ip address in " + assignment);
}
return tokens[0];
}
private static String sliceHostname(String assignment) {
String[] tokens = assignment.split("(\\p{javaSpaceChar}+|\t+)+");
if (tokens.length < 2) {
throw new RuntimeException("Could not find hostname in " + assignment);
}
return tokens[1];
}
private InetAddress mapToInetAddress(String address) {
try {
return InetAddress.getByName(address);
} catch (UnknownHostException e) {
throw new RuntimeException("Could not resolve ip address", e);
}
}
}Overriding Discovery Configuration
So far our implementation retrieves the configuration property for the site-domain.
Our implementation offers the option to override the value from the configuration
(declarative or programmatic) right from the system environment or JVM properties.
That can be useful when the hazelcast.xml defines a setup for an developer system (like cluster.local) and
operations wants to override it for the real deployment.
By providing a prefix (in this case discovery.hosts) we created an
external property named discovery.hosts.site-domain which can be set as an
environment variable or passed as a JVM property from the startup script.
The lookup priority is explained in the following list, priority is from top to bottom:
-
JVM properties (or under the
propertieselement inhazelcast.xml) -
System environment
-
Configuration properties
Implementing Lookup
Since we have the value for our property now, we can implement the actual lookup and
mapping as already prepared in the discoverNodes method.
The following part is very specific to this special discovery strategy;
for completeness we are showing it anyways.
private static final String HOSTS_NIX = "/etc/hosts";
private static final String HOSTS_WINDOWS =
"%SystemRoot%\\system32\\drivers\\etc\\hosts";
private List<String> filterHosts() {
String os = System.getProperty( "os.name" );
String hostsPath;
if ( os.contains( "Windows" ) ) {
hostsPath = HOSTS_WINDOWS;
} else {
hostsPath = HOSTS_NIX;
}
File hosts = new File( hostsPath );
// Read all lines
List<String> lines = readLines( hosts );
List<String> assignments = new ArrayList<String>();
for ( String line : lines ) {
// Example:
// 192.168.0.1 host1.cluster.local
if ( matchesDomain( line ) ) {
assignments.add( line );
}
}
return assignments;
}Mapping to DiscoveryNode
After we have collected the address assignments configured in the hosts file,
we can go to the final step and map those to the DiscoveryNodes to return
them from our strategy.
private Iterable<DiscoveryNode> mapToDiscoveryNodes( List<String> assignments ) {
Collection<DiscoveryNode> discoveredNodes = new ArrayList<DiscoveryNode>();
for ( String assignment : assignments ) {
String address = sliceAddress( assignment );
String hostname = sliceHostname( assignment );
Map<String, Object> attributes =
Collections.singletonMap( "hostname", hostname );
InetAddress inetAddress = mapToInetAddress( address );
Address addr = new Address( inetAddress, NetworkConfig.DEFAULT_PORT );
discoveredNodes.add( new SimpleDiscoveryNode( addr, attributes ) );
}
return discoveredNodes;
}With that mapping, we now have a full discovery, executed whenever Hazelcast asks for IPs. So why don’t we read them in once and cache them? The answer is simple: it might happen that members go down or come up over time. Since we expect the hosts file to be injected into the running container, it also might change over time. We want to get the latest available members, therefore we read the file on request.
Configuring DiscoveryStrategy
To actually use the new DiscoveryStrategy implementation we need to
configure it like in the following example:
<hazelcast>
...
<!-- activate Discovery SPI -->
<properties>
<property name="hazelcast.discovery.enabled">true</property>
</properties>
<network>
<join>
<!-- activate our discovery strategy -->
<discovery-strategies>
<!-- class equals to the DiscoveryStrategy not the factory! -->
<discovery-strategy enabled="true"
class="com.hazelcast.examples.spi.discovery.HostsDiscoveryStrategy">
<properties>
<property name="site-domain">cluster.local</property>
</properties>
</discovery-strategy>
</discovery-strategies>
</join>
</network>
...
</hazelcast>hazelcast:
properties:
hazelcast.discovery.enabled: true
network:
join:
discovery-strategies:
discovery-strategy:
- class: com.hazelcast.examples.spi.discovery.HostsDiscoveryStrategy
enabled: true
properties:
site-domain: cluster.localTo find out further details, please have a look at the Discovery SPI Javadoc.
DiscoveryService (Framework integration)
Since the DiscoveryStrategy is meant for cloud vendors or implementors
of service discovery systems, the DiscoveryService is meant for integrators.
In this case, integrators mean people integrating Hazelcast into their own systems or frameworks.
In those situations, there may be special requirements on
how to lookup framework services like the discovery strategies or similar services.
Integrators can extend or implement their own DiscoveryService and
DiscoveryServiceProvider and inject them using the
com.hazelcast.config.DiscoveryConfig configuration API prior to instantiating the HazelcastInstance.
In any case, integrators might have to remember that a DiscoveryService might have to change
behavior based on the runtime environment (Hazelcast member or client) and
then the DiscoveryServiceSettings should provide information about the started HazelcastInstance.
Since the implementation heavily depends on one’s needs,
there is no reason to provide an example of how to implement your own DiscoveryService.
However, Hazelcast provides a default implementation which can be a good example to get started.
This default implementation is com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.
