This is a prerelease version.

View latest

Connector Guides

Hazelcast comes with a wide range of built-in connectors for accessing data in other platforms. You can also use Hazelcast with pre-built connectors for importing data from external systems. Both types of connector are configured to read and write data in the most efficient way for their respective platform.

Built-in connectors are available as either sources or sinks. Sources allow Hazelcast to read data from the required platform either for queries or as part of a larger data pipeline. Sinks allow Hazelcast to write data, such as query results, to a target system.

You can also use pre-built Kafka Connect Source connectors with Hazelcast. These low-code connectors allow you to stream events from non-Kafka external systems into a data pipeline ready for stream processing. Kafka Connect Source connectors are available for many popular platforms, including RabbitMQ, Neo4j, Couchbase, Scylla, SAP and Redis.

Explore the available connectors and choose one for your use case.

Processing Guarantees

To ensure that all data is processed correctly, connectors that support streaming have either an at-least-once or exactly-once processing guarantee.

Connectors that do not support streaming do not have any processing guarantees. Instead, during intermittent failures, you must restart any queries or jobs that use these connectors.

Connectors for SQL

You can use the following connectors in SQL by configuring them in the CREATE MAPPING statement.

Sources

Connector Module Batch/Streaming Processing guarantee

Apache Kafka

hazelcast-jet-kafka

streaming

exactly-once

File

Depends on the file format and/or storage system.

Depends on the storage system.

Depends on the storage system.

IMap

hazelcast-sql

batch

N/A

JDBC

hazelcast-sql

batch

N/A

MongoDB

hazelcast-sql and hazelcast-jet-mongodb

batch, streaming

exactly-once

Sinks

Connector Module Batch/Streaming Processing guarantee

Apache Kafka

hazelcast-jet-kafka

streaming

exactly-once

IMap

hazelcast-sql

batch

N/A

JDBC

hazelcast-sql

batch

N/A

MongoDB

hazelcast-sql and hazelcast-jet-mongodb

batch

exactly-once

Kafka Connect Connectors

Source

Integrate the Kafka Connect Source connector for your external system with Hazelcast by calling the KafkaConnectSources.connect() method in a pipeline.

Connector Module Batch/Streaming Processing Guarantee

KafkaConnectSources.connect

hazelcast-jet-kafka-connect

streaming

at-least-once

Connectors for the Jet API

The Jet API supports more connectors than SQL.

Sources

Connector Module Batch/Streaming Processing guarantee

AvroSources.files

hazelcast-jet-avro

batch

N/A

DebeziumCdcSources.debezium

hazelcast-jet-cdc-debezium

streaming

at-least-once

ElasticSources.elastic

hazelcast-jet-elasticsearch-7

batch

N/A

HadoopSources.inputFormat

hazelcast-jet-hadoop

batch

N/A

KafkaSources.kafka

hazelcast-jet-kafka

streaming

exactly-once

KinesisSources.kinesis

hazelcast-jet-kinesis

streaming

exactly-once

MongoSources.batch

hazelcast-jet-mongodb

batch

exactly-once

MongoSources.stream

hazelcast-jet-mongodb

streaming

exactly-once

MySqlCdcSources.mysql

hazelcast-jet-cdc-mysql

streaming

exactly-once

PostgresCdcSources.postgres

hazelcast-jet-cdc-postgres

streaming

exactly-once

PulsarSources.pulsarConsumer

hazelcast-jet-contrib-pulsar

streaming

N/A

PulsarSources.pulsarReader

hazelcast-jet-contrib-pulsar

streaming

exactly-once

S3Sources.s3

hazelcast-jet-s3

batch

N/A

Sources.cache

hazelcast-jet

batch

N/A

Sources.cacheJournal

hazelcast-jet

streaming

exactly-once

Sources.files

hazelcast-jet

batch

N/A

Sources.fileWatcher

hazelcast-jet

streaming

none

Sources.json

hazelcast-jet

batch

N/A

Sources.jsonWatcher

hazelcast-jet

streaming

none

Sources.jdbc

hazelcast-jet

batch

N/A

Sources.jmsQueue

hazelcast-jet

streaming

exactly-once

Sources.list

hazelcast-jet

batch

N/A

Sources.map

hazelcast-jet

batch

N/A

Sources.mapJournal

hazelcast-jet

streaming

exactly-once

Sources.socket

hazelcast-jet

streaming

none

TestSources.items

hazelcast-jet

batch

N/A

TestSources.itemStream

hazelcast-jet

streaming

none

Sinks

Connector Module Batch/Streaming Processing Guarantee

AvroSinks.files

hazelcast-jet-avro

batch

N/A

CdcSinks.map

hazelcast-jet-cdc-debezium

streaming

at-least-once

ElasticSinks.elastic

hazelcast-jet-elasticsearch-7

streaming

at-least-once

HadoopSinks.outputFormat

hazelcast-jet-hadoop

batch

N/A

KafkaSinks.kafka

hazelcast-jet-kafka

streaming

exactly-once

KinesisSinks.kinesis

hazelcast-jet-kinesis

streaming

at-least-once

MongoSinks.mongodb

hazelcast-jet-kinesis

batch, streaming

exactly-once

PulsarSources.pulsarSink

hazelcast-jet-contrib-pulsar

streaming

at-least-once

S3Sinks.s3

hazelcast-jet-s3

batch

N/A

Sinks.cache

hazelcast-jet

streaming

at-least-once

Sinks.files

hazelcast-jet

streaming

exactly-once

Sinks.json

hazelcast-jet

streaming

exactly-once

Sinks.jdbc

hazelcast-jet

streaming

exactly-once

Sinks.jmsQueue

hazelcast-jet

streaming

exactly-once

Sinks.list

hazelcast-jet

batch

N/A

Sinks.map

hazelcast-jet

streaming

at-least-once

Sinks.observable

hazelcast-jet

streaming

at-least-once

Sinks.reliableTopic

hazelcast-jet

streaming

at-least-once

Sinks.socket

hazelcast-jet

streaming

at-least-once