Connector Guides

Hazelcast comes out of the box with many different connectors for accessing data in a variety of platforms. These connectors are configured to read and write data in the most efficient way for their respective system. Explore the available connectors and choose one for your use case.

You can use connectors as either sources or sinks. Sources allow Hazelcast to read data from them either for queries or as part of a larger data pipeline. Sinks allow Hazelcast to write data such as query results to them.

Processing Guarantees

Some connectors support streaming, in which case they will have either an at-least-once or exactly-once processing guarantee.

Connectors that do not support streaming do not have any processing guarantees. Instead, during intermittent failures, you must restart any queries or jobs that use these connectors.

Connectors for SQL

You can use the followng connectors in SQL by configuring them in the CREATE MAPPING statement.

Sources

Connector Module Batch/Streaming Processing guarantee

Apache Kafka

hazelcast-jet-kafka

streaming

exactly-once

File

Depends on the file format and/or storage system.

Depends on the storage system.

Depends on the storage system.

IMap

hazelcast-sql

batch

N/A

Sinks

Connector Module Batch/Streaming Processing guarantee

Apache Kafka

hazelcast-jet-kafka

streaming

exactly-once

IMap

hazelcast-sql

batch

N/A

Connectors for the Jet API

The Jet API supports more connectors than SQL.

Sources

Connector Module Batch/Streaming Processing guarantee

AvroSources.files

hazelcast-jet-avro

batch

N/A

DebeziumCdcSources.debezium

hazelcast-jet-cdc-debezium

streaming

at-least-once

ElasticSources.elastic

hazelcast-jet-elasticsearch-5

batch

N/A

ElasticSources.elastic

hazelcast-jet-elasticsearch-6

batch

N/A

ElasticSources.elastic

hazelcast-jet-elasticsearch-7

batch

N/A

HadoopSources.inputFormat

hazelcast-jet-hadoop

batch

N/A

KafkaSources.kafka

hazelcast-jet-kafka

streaming

exactly-once

KinesisSources.kinesis

hazelcast-jet-kinesis

streaming

exactly-once

MySqlCdcSources.mysql

hazelcast-jet-cdc-mysql

streaming

exactly-once

PostgresCdcSources.postgres

hazelcast-jet-cdc-postgres

streaming

exactly-once

PulsarSources.pulsarConsumer

hazelcast-jet-contrib-pulsar

streaming

N/A

PulsarSources.pulsarReader

hazelcast-jet-contrib-pulsar

streaming

exactly-once

S3Sources.s3

hazelcast-jet-s3

batch

N/A

Sources.cache

hazelcast-jet

batch

N/A

Sources.cacheJournal

hazelcast-jet

streaming

exactly-once

Sources.files

hazelcast-jet

batch

N/A

Sources.fileWatcher

hazelcast-jet

streaming

none

Sources.json

hazelcast-jet

batch

N/A

Sources.jsonWatcher

hazelcast-jet

streaming

none

Sources.jdbc

hazelcast-jet

batch

N/A

Sources.jmsQueue

hazelcast-jet

streaming

exactly-once

Sources.list

hazelcast-jet

batch

N/A

Sources.map

hazelcast-jet

batch

N/A

Sources.mapJournal

hazelcast-jet

streaming

exactly-once

Sources.socket

hazelcast-jet

streaming

none

TestSources.items

hazelcast-jet

batch

N/A

TestSources.itemStream

hazelcast-jet

streaming

none

Sinks

Connector Module Batch/Streaming Processing Guarantee

AvroSinks.files

hazelcast-jet-avro

batch

N/A

CdcSinks.map

hazelcast-jet-cdc-debezium

streaming

at-least-once

ElasticSinks.elastic

hazelcast-jet-elasticsearch-5

streaming

at-least-once

ElasticSinks.elastic

hazelcast-jet-elasticsearch-6

streaming

at-least-once

ElasticSinks.elastic

hazelcast-jet-elasticsearch-7

streaming

at-least-once

HadoopSinks.outputFormat

hazelcast-jet-hadoop

batch

N/A

KafkaSinks.kafka

hazelcast-jet-kafka

streaming

exactly-once

KinesisSinks.kinesis

hazelcast-jet-kinesis

streaming

at-least-once

PulsarSources.pulsarSink

hazelcast-jet-contrib-pulsar

streaming

at-least-once

S3Sinks.s3

hazelcast-jet-s3

batch

N/A

Sinks.cache

hazelcast-jet

streaming

at-least-once

Sinks.files

hazelcast-jet

streaming

exactly-once

Sinks.json

hazelcast-jet

streaming

exactly-once

Sinks.jdbc

hazelcast-jet

streaming

exactly-once

Sinks.jmsQueue

hazelcast-jet

streaming

exactly-once

Sinks.list

hazelcast-jet

batch

N/A

Sinks.map

hazelcast-jet

streaming

at-least-once

Sinks.observable

hazelcast-jet

streaming

at-least-once

Sinks.reliableTopic

hazelcast-jet

streaming

at-least-once

Sinks.socket

hazelcast-jet

streaming

at-least-once