A newer version of Hazelcast Platform is available.

View latest

Mapping to MongoDB

6.0-SNAPSHOT 5.5 5.4 5.3 5.2 5.1 5.0

To query MongoDB data connections, you can create a mapping to them with the Mongo connector.

What is the Mongo Connector

The Mongo connector allows you to read from/write to a MongoDB database, and to execute SQL queries on Mongo collections directly from Hazelcast.

Supported SQL Statements

For the batch mode:

For the streaming mode:

Installing the Connector

The Mongo Connector artifacts are published on the Maven repositories. Add the following lines to your pom.xml to include it as a dependency to your project:

<dependency>
    <groupId>com.hazelcast.jet</groupId>
    <artifactId>hazelcast-jet-mongodb</artifactId>
    <version>$5.3.8</version>
</dependency>

or if you are using Gradle:

compile group: 'com.hazelcast.jet', name: 'hazelcast-jet-mongodb', version: $5.3.8.
To be able to use SQL over MongoDB, you have to include hazelcast-sql as well as a dependency.

Permissions

Enterprise

The Mongo connector does not yet support permissions.

Before you Begin

Before you can create a mapping to a MongoDB, you must have the following:

  • A $jsonSchema validation in the collection (see the schema documentation), or you have to have at least one element in the collection you want to create mapping for (for property type validation).

  • Enabled operations log (oplog) if you want to use streaming mappings.

Creating a MongoDB Mapping

The following example creates a mapping to a MongoDB database.

  1. In a MongoDB database, create a people collection. For example in Java, you would run the following command.

    CreateCollectionOptions options = new CreateCollectionOptions();
ValidationOptions validationOptions = new ValidationOptions();
validationOptions.validator(BsonDocument.parse(
       "{\n" +
               "    $jsonSchema: {\n" +
               "      bsonType: \"object\",\n" +
               "      title: \"Object Validation\",\n" +
               "      properties: {" +
               "        \"personId\": { \"bsonType\": \"int\" },\n" +
               "        \"name\": { \"bsonType\": \"string\" }\n" +
               "      }\n" +
               "    }\n" +
               "  }\n"
));
options.validationOptions(validationOptions);
database.createCollection(collectionName, options);

    The ValidationOptions are not required, but recommended.

  2. Configure the data connection so that the client can be reused by multiple mappings.

    • XML

    • YAML

    • Java

    <hazelcast>
    <data-connection name="myMongo">
        <type>Mongo</type>
        <properties>
            <property name="connectionString">stringForMongo</property> (1)
        </properties>
        <shared>false</shared> (2)
    </data-connection>
</hazelcast>
    data-connection:
  name: myMongo
  type: Mongo
  properties:
    connectionString: stringforMongo (1)
  shared: false (2)
    DataConnectionConfig dataConnectionConfig = new DataConnectionConfig()
        .setName("myMongo")
        .setType("Mongo")
        .setProperty("connectionString", connectionStringToMongo) (1)
        .setShared(false); (2)
config.addDataConnectionConfig(dataConnectionConfig);
    1 Your connection string.
    2 Set to true if the connection is reusable.

    Instead of providing a single connectionString parameter, you may also want to provide host, username, password and (optionally) authDb.

    Instead of providing the configuration in YAML or XML, you can also run the following SQL query.

    CREATE DATA CONNECTION myMongo type Mongo SHARED
OPTIONS (
	‘connectionString’ = ‘<your connection string>’
)

  3. Create the mapping.

    CREATE MAPPING people
TYPE MONGO (1)
DATA CONNECTION myMongo; (2)
    1 The type of the connector.
    2 The name of the data connection configuration on your members (see Step 2 above).

    In the above case, automatic schema inference will be used. You may also want to provide the schema explicitly as shown below.

    CREATE MAPPING people (
    firstName VARCHAR(100),
    lastName VARCHAR(100),
    age INT
)
DATA CONNECTION myMongo

    Notice that there is no mention of TYPE MONGO this time; it’s automatically assumed by the SQL engine when you provide MongoDB data connection. This works with both schema provided or not.

Type Mapping

The type system in MongoDB and SQL is not exactly the same. That leads to potential confusions and the need of type coercion.

Table 1. MongoDB Type Conversion
BSON Type SQL Type Java Type

DOUBLE

DOUBLE

DOUBLE

STRING

VARCHAR

STRING

OBJECT

OBJECT

org.bson.Document

ARRAY

OBJECT

LIST

BINDATA

-

-

UNDEFINED

-

-

OBJECTID

OBJECT

org.bson.ObjectId

BOOL

BOOLEAN

BOOLEAN

DATE

This represents seconds from Unix epoch in UTC timezone. Therefore, it’s not mapped to pure DATE SQL type nor LOCALDATE in Java (nor any formats with timezones).

DATE_TIME or TIMESTAMP

LOCALDATETIME

TIMESTAMP

DATE_TIME or TIMESTAMP

LOCALDATETIME

NULL

-

-

REGEX

OBJECT

org.bson.BsonRegularExpression

DBPOINTER

-

-

JAVASCRIPT

VARCHAR

STRING

JAVASCRIPTWITHSCOPE

OBJECT

org.bson.CodeWithScope

SYMBOL

-

-

INT (32 BIT)

INT

INT

LONG (64 BIT)

BIGINT

LONG

DECIMAL (128 BIT)

DECIMAL

BIGDECIMAL

MINKEY

OBJECT

org.bson.MinKey

MAXKEY

OBJECT

org.bson.MaxKey

The Java Type column represents an object returned by the SQL query if the object put into the collection is of given BSON type.

Note that, while Hazelcast is able to convert MongoDB type to the requested SQL type in the projection, the argument binding will not always work the same due to technical limitations. For example, you can have an object with the type TIMESTAMP represented as DATE_TIME, that after execution of SELECT it will give you LocalDateTime in Java client. However, binding LocalDateTime as an argument will not work, as only native MongoDB types will work for arguments. Same applies to, for example, having BSON column of type STRING mapped to INTEGER in SQL.

Type Coercion

The following table shows the possible and supported type coercions. All the default mappings from the previous section are always valid.

Table 2. MongoDB Type Conversion
Type of Provided Argument Resolved Insertion Type

LOCALDATETIME

BSONDATETIME

OFFSETDATETIME

BSONDATETIME

HazelcastJsonValue (JSON column)

DOCUMENT
lifebuoy

Help and support