This is a prerelease version.

IMap SQL Connector

The IMap connector supports batch and stream processing by reading from maps as well as adding entries to maps in a local cluster. Maps in remote clusters are not supported yet.

Serialization Options

The keyFormat and valueFormat options are mandatory. Currently, if you create the mapping explicitly, we can’t resolve these from a sample.

Possible values for keyFormat and valueFormat:

  • any of the supported SQL data types, except for OBJECT

  • portable

  • json

  • java

The key and value formats can be different.

Primitive Serialization

If the format is one of the primitive type names, then the Java class representing that type will be stored in the map. By primitive type we mean any supported SQL data type, except for OBJECT.

For example, to create a mapping for IMap<Integer, String>, use:

CREATE MAPPING my_map
TYPE IMap
OPTIONS (
    'keyFormat'='int',
    'valueFormat'='varchar'
)

Portable Serialization

For this format, you need to specify additional options:

  • keyPortableFactoryId, valuePortableFactoryId

  • keyPortableClassId, valuePortableClassId

  • keyPortableVersion, valuePortableVersion: optional, default is 0

If you omit a column list from the CREATE MAPPING command, Hazelcast will resolve column names and types by looking at the ClassDefinition found using the given factory ID, class ID, and version.

If the ClassDefinition with the given IDs is not known to the cluster, the column list is mandatory and Hazelcast will create the ClassDefinition based on the column list.

The benefit of this format is that it doesn’t deserialize the whole key or value when reading only a subset of fields. Also it doesn’t require a custom Java class to be defined on the cluster, so it’s usable for non-Java clients.

Example mapping where both key and value are Portable:

CREATE MAPPING my_map
TYPE IMap
OPTIONS (
    'keyFormat' = 'portable',
    'keyPortableFactoryId' = '123',
    'keyPortableClassId' = '456',
    'keyPortableVersion' = '0',  -- optional
    'valueFormat' = 'portable',
    'valuePortableFactoryId' = '123',
    'valuePortableClassId' = '789',
    'valuePortableVersion' = '0'  -- optional
)

For more information on Portable see Implementing Portable Serialization.

JSON Serialization

To store values in JSON, you must declare the field names. For example, here, we create a mapping to a distributed map that stores a JSON object with the ticker and amount fields.

CREATE MAPPING my_map(
    __key BIGINT,
    ticker VARCHAR,
    amount INT)
TYPE IMap
OPTIONS (
    'keyFormat' = 'bigint',
    'valueFormat' = 'json')

There are no additional options for this format.

By default, Hazelcast serializes JSON into HazelcastJsonValue objects, which allows you to query its fields.

JSON’s type system doesn’t match SQL’s exactly. For example, JSON numbers have unlimited precision, but such numbers are typically not portable. We convert SQL integer and floating-point types into JSON numbers. We convert the DECIMAL type, as well as all temporal types, to JSON strings.

We don’t yet support the JSON type from the SQL standard. That means you can’t use functions like JSON_VALUE or JSON_QUERY. If your JSON documents don’t all have the same fields or if they contain nested objects, the usability is limited.

Java Serialization

Java serialization is the last-resort serialization option. It uses the Java object exactly as map.get() returns it. You can use it for objects serialized using the Java serialization or Hazelcast custom serialization (DataSerializable or IdentifiedDataSerializable).

For this format you must specify the class name using keyJavaClass and valueJavaClass options, for example:

CREATE MAPPING my_map
TYPE IMap
OPTIONS (
    'keyFormat' = 'java',
    'keyJavaClass' = 'java.lang.Long',
    'valueFormat' = 'java',
    'valueJavaClass' = 'com.example.Person')

If the Java class corresponds to one of the basic data types (numbers, dates, strings), that type will directly be used for the key or value and mapped as a column named __key for keys and this for values. In the example above, the key will be mapped with the BIGINT type. In fact, the above keyFormat and keyJavaClass duo is equivalent to 'keyFormat'='bigint'.

If the Java class is not one of the basic types, Hazelcast will analyze the class using reflection and use its properties as column names. It recognizes public fields and JavaBean-style getters. If some property has a non-primitive type, it will be mapped under the OBJECT type.

The class must be available to the cluster. You can either add it to the members class paths by creating a JAR file and adding it to the lib folder, or you can use User Code Deployment. The user code deployment has to be enabled on the members; add the following section to the config/hazelcast.yaml file:

hazelcast:
  user-code-deployment:
    enabled: true

Then use a client to upload the class:

ClientConfig clientConfig = new ClientConfig();
clientConfig.getUserCodeDeploymentConfig()
            .setEnabled(true)
            .addClass(Trade.class);
HazelcastInstance hz = HazelcastClient.newHazelcastClient(clientConfig);