Connector Guides

Hazelcast comes with a wide range of built-in connectors for accessing data in other platforms. You can also use Hazelcast with pre-built connectors for importing data from external systems. Both types of connector are configured to read and write data in the most efficient way for their respective platform.

Built-in connectors are available as either sources or sinks. Sources allow Hazelcast to read data from the required platform either for queries or as part of a larger data pipeline. Sinks allow Hazelcast to write data, such as query results, to a target system.

You can also use pre-built Kafka Connect Source connectors with Hazelcast. These low-code connectors allow you to stream events from non-Kafka external systems into a data pipeline ready for stream processing. Kafka Connect Source connectors are available for many popular platforms, including RabbitMQ, Neo4j, Couchbase, Scylla, SAP and Redis.

Explore the available connectors and choose one for your use case.

Processing Guarantees

To ensure that all data is processed correctly, connectors that support streaming have either an at-least-once or exactly-once processing guarantee.

Connectors that do not support streaming do not have any processing guarantees. Instead, during intermittent failures, you must restart any queries or jobs that use these connectors.

Connectors for SQL

You can use the following connectors in SQL by configuring them in the CREATE MAPPING statement.

Sources

Connector	Module	Batch/Streaming	Processing guarantee
Apache Kafka	hazelcast-jet-kafka	streaming	exactly-once
Apache Pulsar	hazelcast-jet-pulsar	streaming	exactly-once
File	Depends on the file format and/or storage system.	Depends on the storage system.	Depends on the storage system.
IMap	hazelcast-sql	batch	N/A
JDBC	hazelcast-sql	batch	N/A
MongoDB	hazelcast-sql and hazelcast-jet-mongodb	batch, streaming	exactly-once

Sinks

Connector	Module	Batch/Streaming	Processing guarantee
Apache Kafka	hazelcast-jet-kafka	streaming	exactly-once
Apache Pulsar	hazelcast-jet-pulsar	streaming	exactly-once
IMap	hazelcast-sql	batch	N/A
JDBC	hazelcast-sql	batch	N/A
MongoDB	hazelcast-sql and hazelcast-jet-mongodb	batch	exactly-once

Kafka Connect Connectors

Source

Integrate the Kafka Connect Source connector for your external system with Hazelcast by calling the KafkaConnectSources.connect() method in a pipeline.

Connector	Module	Batch/Streaming	Processing Guarantee
KafkaConnectSources.connect	hazelcast-jet-kafka-connect	streaming	at-least-once

Connectors for the Jet API

The Jet API supports more connectors than SQL.

Sources

Connector	Module	Batch/Streaming	Processing guarantee
AvroSources.files	hazelcast-jet-avro	batch	N/A
DebeziumCdcSources.debezium (Legacy)	hazelcast-jet-cdc-debezium	streaming	at-least-once
MySqlCdcSources.mysql (Legacy)	hazelcast-jet-cdc-mysql	streaming	exactly-once
PostgresCdcSources.postgres (Legacy)	hazelcast-jet-cdc-postgres	streaming	exactly-once
DebeziumCdcSources.debezium (Enterprise)	hazelcast-enterprise-cdc-debezium	streaming	at-least-once
MySqlCdcSources.mysql	hazelcast-enterprise-cdc-mysql	streaming	exactly-once
PostgresCdcSources.postgres	hazelcast-enterprise-cdc-postgres	streaming	exactly-once
ElasticSources.elastic	hazelcast-jet-elasticsearch-7	batch	N/A
HadoopSources.inputFormat	hazelcast-jet-hadoop	batch	N/A
KafkaSources.kafka	hazelcast-jet-kafka	streaming	exactly-once
KinesisSources.kinesis	hazelcast-jet-kinesis	streaming	exactly-once
MongoSources.batch	hazelcast-jet-mongodb	batch	exactly-once
MongoSources.stream	hazelcast-jet-mongodb	streaming	exactly-once
PulsarSources.pulsarConsumer	hazelcast-jet-pulsar	streaming (many consumer per topic to distribute reading)	exactly-once
PulsarSources.pulsarReader	hazelcast-jet-pulsar	streaming (single reader per topic)	exactly-once
S3Sources.s3	hazelcast-jet-s3	batch	N/A
Sources.cache, Sources.remoteCache	hazelcast	batch	N/A
Sources.cacheJournal, Sources.remoteCacheJournal	hazelcast	streaming	exactly-once
Sources.files	hazelcast	batch	N/A
Sources.fileWatcher	hazelcast	streaming	none
Sources.json	hazelcast	batch	N/A
Sources.jsonWatcher	hazelcast	streaming	none
Sources.jdbc	hazelcast	batch	N/A
Sources.jmsQueue	hazelcast	streaming	exactly-once
Sources.list, Sources.remoteList	hazelcast	batch	N/A
Sources.map, Sources.remoteMap	hazelcast	batch	N/A
Sources.mapJournal, Sources.remoteMapJournal	hazelcast	streaming	exactly-once
Sources.socket	hazelcast	streaming	none
VectorSources.fvecs Enterprise Edition	hazelcast-enterprise	batch	N/A
VectorSources.ivecs Enterprise Edition	hazelcast-enterprise	batch	N/A
TestSources.items	hazelcast	batch	N/A
TestSources.itemStream	hazelcast	streaming	none

Sinks

Connector	Module	Batch/Streaming	Processing Guarantee
AvroSinks.files	hazelcast-jet-avro	batch	N/A
CdcSinks.map	hazelcast-jet-cdc-debezium (legacy, Community Edition) or hazelcast-enterprise-cdc-debezium (Enterprise Edition)	streaming	at-least-once
ElasticSinks.elastic	hazelcast-jet-elasticsearch-7	streaming	at-least-once
HadoopSinks.outputFormat	hazelcast-jet-hadoop	batch	N/A
KafkaSinks.kafka	hazelcast-jet-kafka	streaming	exactly-once
KinesisSinks.kinesis	hazelcast-jet-kinesis	streaming	at-least-once
MongoSinks.mongodb	hazelcast-jet-kinesis	batch, streaming	exactly-once
PulsarSources.pulsarSink	hazelcast-jet-pulsar	streaming	exactly-once
S3Sinks.s3	hazelcast-jet-s3	batch	N/A
Sinks.cache, Sinks.remoteCache	hazelcast	streaming	at-least-once
Sinks.files	hazelcast	streaming	exactly-once
Sinks.json	hazelcast	streaming	exactly-once
Sinks.jdbc	hazelcast	streaming	exactly-once
Sinks.jmsQueue	hazelcast	streaming	exactly-once
Sinks.list, Sinks.remoteList	hazelcast	batch	N/A
Sinks.map, Sinks.remoteMap	hazelcast	streaming	at-least-once
Sinks.observable	hazelcast	streaming	at-least-once
Sinks.reliableTopic, Sinks.remoteReliableTopic	hazelcast	streaming	at-least-once
Sinks.socket	hazelcast	streaming	at-least-once
VectorSinks.vectorCollection Enterprise Edition	hazelcast-enterprise	batch, streaming	at-least-once