A newer version of Hazelcast Platform is available.

View latest

Using the generic MapLoader

6.0-SNAPSHOT 5.5 5.4 5.3 5.2 5.1 5.0

With the generic MapLoader, you can configure a map to cache data from an external system. This topic includes an example of how to configure a map with a generic MapLoader that connects to a MySQL database.

The objects created in the distributed map are stored as GenericRecord. You can use the type-name property to store the data in a POJO (Plain Old Java Object).

For a list of all supported external systems, including databases, see available data connection types.

Before you begin

You need a data connection that’s configured on all cluster members.

Add dependencies

If you are using a Hazelcast JAR file, you need to ensure the following is added to your classpath:

<dependency>
     <groupId>com.hazelcast</groupId>
     <artifactId>hazelcast-sql</artifactId>
</dependency>

<dependency>
     <groupId>com.hazelcast</groupId>
     <artifactId>hazelcast-mapstore</artifactId>
</dependency>
If you are using the slim distribution, you need to add hazelcast-mapstore. If you are using MongoDb, you also need to add hazelcast-jet-mongodb.

Quickstart configuration

This example shows a basic map configuration that uses a data connection called my-mysql-database. See Distributed Map for the details of other properties that you can include in your map configuration.

  • XML

  • YAML

  • Java

<hazelcast>
    ...
    <map name="myMapName">
        <map-store enabled="true">
            <class-name>com.hazelcast.mapstore.GenericMapLoader</class-name> (1)
            <properties>
                <property name="data-connection-ref">my-mysql-database</property> (2)
            </properties>
        </map-store>
    </map>
    ...
</hazelcast>
hazelcast:
  map:
    mymapname:
      map-store:
        enabled: true
        class-name: com.hazelcast.mapstore.GenericMapLoader (1)
        properties:
            data-connection-ref: my-mysql-database (2)
MapConfig mapConfig = new MapConfig("myMapName");
MapStoreConfig mapStoreConfig = new MapStoreConfig();
mapStoreConfig.setClassName("com.hazelcast.mapstore.GenericMapLoader"); (1)
mapStoreConfig.setProperty("data-connection-ref", "my-mysql-database"); (2)
mapConfig.setMapStoreConfig(mapStoreConfig);
instance().getConfig().addMapConfig(mapConfig);
1 The class name of the generic MapLoader that’s built into Hazelcast. This implementation reads from the data connection configuration to create a SQL mapping.
2 The name of your data connection.

SQL mapping for the generic MapLoader

When you configure a map with the generic MapLoader, Hazelcast creates a SQL mapping with the JDBC connector. The name of the mapping is the same name as your map prefixed with __map-store.. This mapping is used to read data from the external system, and is removed whenever the configured map is removed. You can also configure this SQL mapping, using configuration properties.

Configuration properties for the generic MapLoader

These configuration properties allow you to configure the generic MapLoader and its SQL mapping.

As well as these properties, you can also configure general MapStore settings.
Table 1. Generic MapLoader configuration options
Option Description Default Example

data-connection-ref (required)

The name of the data connection that sets up a mapping. Use this property when you have an existing data connection to an external system.

'' (empty)

  • XML

  • YAML

  • Java

<hazelcast>
  <map name="myMapName">
    <map-store enabled="true">
      <class-name>com.hazelcast.mapstore.GenericMapLoader</class-name>
      <properties>
          <property name="data-connection-ref">my-mysql-database</property>
      </properties>
    </map-store>
</hazelcast>
hazelcast:
  map:
    mymapname:
      map-store:
        enabled: true
        class-name: com.hazelcast.mapstore.GenericMapLoader
        properties:
            data-connection-ref: my-mysql-database
MapConfig mapConfig = new MapConfig("myMapName");

MapStoreConfig mapStoreConfig = new MapStoreConfig();
mapStoreConfig.setClassName("com.hazelcast.mapstore.GenericMapLoader");
mapStoreConfig.setProperty("data-connection-ref", "my-mysql-database");

mapConfig.setMapStoreConfig(mapStoreConfig);

external-name

External name of the data (e.g. table or collection) to read from.

The name of the map.

  • XML

  • YAML

  • Java

<hazelcast>
  <map name="myMapName">
    <map-store enabled="true">
      <class-name>com.hazelcast.mapstore.GenericMapLoader</class-name>
      <properties>
          <property name="data-connection-ref">my-mysql-database</property>
          <property name="external-name">test</property>
      </properties>
    </map-store>
</hazelcast>
hazelcast:
  map:
    mymapname:
      map-store:
        enabled: true
        class-name: com.hazelcast.mapstore.GenericMapLoader
        properties:
            data-connection-ref: my-mysql-database
            external-name: test
MapConfig mapConfig = new MapConfig("myMapName");

MapStoreConfig mapStoreConfig = new MapStoreConfig();
mapStoreConfig.setClassName("com.hazelcast.mapstore.GenericMapLoader");
mapStoreConfig.setProperty("data-connection-ref", "my-mysql-database");
mapStoreConfig.setProperty("external-name", "test");

mapConfig.setMapStoreConfig(mapStoreConfig);

mapping-type

SQL connector to use for the mapping.

The SQL connector is derived from the data connection in the configuration.

  • XML

  • YAML

  • Java

<hazelcast>
  <map name="myMapName">
    <map-store enabled="true">
      <class-name>com.hazelcast.mapstore.GenericMapLoader</class-name>
      <properties>
          <property name="data-connection-ref">my-mysql-database</property>
          <property name="mapping-type">JDBC</property>
      </properties>
    </map-store>
</hazelcast>
hazelcast:
  map:
    mymapname:
      map-store:
        enabled: true
        class-name: com.hazelcast.mapstore.GenericMapLoader
        properties:
            data-connection-ref: my-mysql-database
            mapping-type: JDBC
MapConfig mapConfig = new MapConfig("myMapName");

MapStoreConfig mapStoreConfig = new MapStoreConfig();
mapStoreConfig.setClassName("com.hazelcast.mapstore.GenericMapLoader");
mapStoreConfig.setProperty("data-connection-ref", "my-mysql-database");
mapStoreConfig.setProperty("mapping-type", "JDBC");

mapConfig.setMapStoreConfig(mapStoreConfig);

id-column

Name of the column that contains the primary key.

id

  • XML

  • YAML

  • Java

<hazelcast>
  <map name="myMapName">
    <map-store enabled="true">
      <class-name>com.hazelcast.mapstore.GenericMapLoader</class-name>
      <properties>
          <property name="data-connection-ref">my-mysql-database</property>
          <property name="id-column">id</property>
      </properties>
    </map-store>
</hazelcast>
hazelcast:
  map:
    mymapname:
      map-store:
        enabled: true
        class-name: com.hazelcast.mapstore.GenericMapLoader
        properties:
            data-connection-ref: my-mysql-database
            id-column: id
MapConfig mapConfig = new MapConfig("myMapName");

MapStoreConfig mapStoreConfig = new MapStoreConfig();
mapStoreConfig.setClassName("com.hazelcast.mapstore.GenericMapLoader");
mapStoreConfig.setProperty("data-connection-ref", "my-mysql-database");
mapStoreConfig.setProperty("id-column", "id");

mapConfig.setMapStoreConfig(mapStoreConfig);

columns

Names of the columns to map. This value must include a subset of columns in the table. Missing columns must have a default value defined.

  • XML

  • YAML

  • Java

<hazelcast>
  <map name="myMapName">
    <map-store enabled="true">
      <class-name>com.hazelcast.mapstore.GenericMapLoader</class-name>
      <properties>
          <property name="data-connection-ref">my-mysql-database</property>
          <property name="columns">name</property>
      </properties>
    </map-store>
</hazelcast>
hazelcast:
  map:
    mymapname:
      map-store:
        enabled: true
        class-name: com.hazelcast.mapstore.GenericMapLoader
        properties:
            data-connection-ref: my-mysql-database
            columns: name
MapConfig mapConfig = new MapConfig("myMapName");

MapStoreConfig mapStoreConfig = new MapStoreConfig();
mapStoreConfig.setClassName("com.hazelcast.mapstore.GenericMapLoader");
mapStoreConfig.setProperty("data-connection-ref", "my-mysql-database");
mapStoreConfig.setProperty("columns", "name");

mapConfig.setMapStoreConfig(mapStoreConfig);

Supported backends

The generic MapStore needs a SQL Connector that supports SELECT, UPDATE, SINK INTO and DELETE statements.

Officially supported connectors:

  • JDBC Connector

    • supports MySQL, PostgreSQL.

    • requires JDBC driver on the classpath

  • MongoDB Connector

    • make sure you have hazelcast-jet-mongodb artifact included on the classpath.

Next steps

See the MapStore configuration guide for details about configuration options, including caching behaviors.

lifebuoy

Help and support