Skip to main content
Version: 6.0-SNAPSHOT

Configuring Data Connections to External Systems

A data connection contains the metadata that Hazelcast needs to connect an external system. You can define a data connection in your members' configuration files, in the Java member API, or in SQL, and reuse the same connection details in the Pipeline API, SQL mappings, and MapStores.

Quickstart Configuration​

To configure a data connection to an external system, you must do the following:

  • Provide a unique identifier (name) for the data connection.
  • Choose the correct type of data connection for your external system.

See examples for other connection properties for each data connection type.

Example JDBC Data Connection​

This example configuration shows a data connection to a MySQL database using a JDBC connection.

xml
<hazelcast>
  <data-connection name="my-mysql-database">
    <type>JDBC</type>
    <properties>
      <property name="jdbcUrl">jdbc:mysql://mysql.example.org:3306</property> (1)
      <property name="user">my_user</property> (2)
      <property name="password">my_password</property>
    </properties>
    <shared>true</shared>
  </data-connection>
</hazelcast>
1 (Required) JDBC URL for establishing a connection to the MySQL database
2 (Optional) Separate user credentials for authentication. Alternatively, you can include the user credentials in the JDBC URL.

For shared connections, the following optional properties allow you to control how the JDBC connection pool behaves. All other properties are passed directly to the JDBC driver.

Default values are in milliseconds except for where indicated.

PropertyDefault ValueDescription
connectionTimeout30000 (30 seconds)Maximum time that Hazelcast waits for a connection from the pool when attempting to connect to the external system, typically a database. A SQL exception is thrown when the connectionTimeout is exceeded before a connection becomes available.
idleTimeout600000 (10 minutes)

Maximum time that the JDBC connection component allows a connection to sit idle in the connection pool. After the idleTimeout is exceeded, there may be a time lag of up to +30 seconds before the connection is retired. Although, the average time lag is +15 seconds.

  • Only applies when minimumIdle is less than maximumPoolSize.
  • A value of 0 means that idle connections are never removed from the pool.
  • The minimum allowed value is 1000 ms (10 seconds)
keepaliveTime0 (Disabled)How frequently the JDBC connection component attempts to keep an idle connection alive. When the keepaliveTime is reached, the idle connection is removed from the connection pool, pinged, and then returned to the pool. The minimum allowed keepaliveTime value is 30000 ms (30 seconds), but a value in the range of minutes is recommended.
maxLifetime1800000 (30 minutes)

Maximum lifetime of a connection in the pool. Only closed connections are removed. A value of 0 indicates an infinite lifetime, unless an idleTimeout value is set and reached. The minimum allowed maxLifetime value is 30000 ms (30 seconds).

  • We recommend setting this value, and it should be several seconds shorter than any database or infrastructure imposed connection time limit.
minimumIdleSame as maximumPoolSize

Minimum number of idle connections that the JDBC connection component attempts to maintain in the connection pool. When the number of idle connections dips below the minimumIdle, and the total connections are less than maximumPoolSize, the connection component attempts to add more connections.

  • For maximum performance and responsive we do not recommend setting this value, instead use a fixed size connection pool.
maximumPoolSize10 (connections)Maximum size of the connection pool, which includes both idle and active connections. The maximumPoolSize sets the maximum number of connections to the external system. When the pool reaches this size, and no more idle connections are available, calls to get a new connection are blocked for up to the connectionTimeout before timing out.

Example Kafka Data Connection​

This example shows the configuration of a data connection to a single Kafka broker.

xml
<hazelcast>
  <data-connection name="my-kafka">
    <type>Kafka</type>
    <properties>
      <property name="bootstrap.servers">127.0.0.1:9092</property> (1)
      <property name="key.deserializer">org.apache.kafka.common.serialization.IntegerDeserializer</property> (2)
      <property name="key.serializer">org.apache.kafka.common.serialization.IntegerSerializer</property>
      <property name="value.serializer">org.apache.kafka.common.serialization.StringSerializer</property>
      <property name="value.deserializer">org.apache.kafka.common.serialization.StringDeserializer</property>
      <property name="auto.offset.reset">earliest</property> (3)
    </properties>
    <shared>true</shared>
  </data-connection>
</hazelcast>
1 (Required) Address of the Kafka consumer/producer
2 (Optional) Automatic serializers/deserializers for keys and values in Kafka messages
3 (Optional) Consumer behavior if the connection is interrupted

Example MongoDB Data Connection​

This example configuration shows data connections to two MongoDB databases.

As in the example, you can supply authentication credentials to a MongoDB instance as part of the connection string, or separately.

xml
<hazelcast>
  <data-connection name="my-mongodb">
    <type>Mongo</type>
    <properties>
      <property name="connectionString">mongodb://my_user:my_password@some-host:27017</property> (1)
      <property name="database">my_database</property> (2)
    </properties>
    <shared>true</shared>
  </data-connection>
  <data-connection name="my-other-mongodb">
    <type>Mongo</type>
    <properties>
      <property name="host">some_host</property> (3)
      <property name="username">my_user</property> (4)
      <property name="password">my_password</property>
      <property name="database">my_other_database</property> (2)
    </properties>
    <shared>true</shared>
  </data-connection>
</hazelcast>
1 (Required) Connection string of the MongoDB instance, including user credentials
2 (Optional) Name of the database to connect to
3 (Optional) Host details of the MongoDB instance, excluding user credentials
4 (Optional) User credentials for the MongoDB instance
PropertyDefault valueDescriptionWhen to use
connectionStringThe connection string used to connect to given MongoDB instance. More in Mongo documentation.Can be omitted if you prefer to provide host,
databaseNameName of the database that can be accessed via this connection. If omitted, user will have access to all databases available to given Mongo user.If you want to restrict usage of this Data Connection to a particular database. Mandatory if you want to use this Data Connection with GenericMapStore.
usernameUsername used to authenticate to Mongo.If you want to avoid putting credentials in connection string, you can use this dedicated option.
passwordPassword used to authenticate to Mongo.If you want to avoid putting credentials in connection string, you can use this dedicated option.
authDbadminAuthentication database - the database that holds user's data. It is not the same as the databaseName option, both options can specify other databases - one to authenticate, other to actually read.If you want to use username and password options, the authDb option is the way to configure authentication database name (which can be contained in connectionString).
hostHost to which Hazelcast will connect. Exclusive with connectionString.If you want to use username and password options, the host option is the way to configure host name.
connectionPoolMinSize10Sets the minimum size of MongoClient's internal connection pool.If you want to control connection pool size
connectionPoolMaxSize10Sets the maximum size of MongoClient's internal connection pool.If you want to control connection pool size
enableSslEnables SSL support for Mongo client. Default value is false.
invalidHostNameAllowedAllows invalid hostnames in SSL. Default value is false. See Mongo docs for more info.
keyStoreLocation of the key store, file must be present on all members.
keyStoreTypeType of the used key store. Defaults to system default.
keyStorePasswordPassword to the key store.
trustStoreLocation of the trust store, file must be present on all members.
trustStoreTypeType of the used trust store. Defaults to system default.
trustStorePasswordPassword to the trust store.

Example Apache Pulsar Data Connection​

xml
<hazelcast>
  <data-connection name="my-pulsar">
    <type>Pulsar</type>
    <properties>
      <property name="brokerUrl">pulsar://pulsar.mydomain.com:6650</property> (1)
      <property name="httpServiceUrl">http://pulsar.mydomain.com:8080</property> (2)
    </properties>
    <shared>true</shared>
  </data-connection>
  <data-connection name="my-other-pulsar">
    <type>Pulsar</type>
    <properties>
      <property name="brokerUrl">pulsar://pulsar.mydomain.com:6650</property> (1)
      <property name="httpServiceUrl">http://pulsar.mydomain.com:8080</property> (2)
      <property name="tenant">myTenant</property> (3)
      <property name="namespace">myNamespace</property> (3)
      <property name="enableTransactions">true</property> (4)
    </properties>
    <shared>true</shared>
  </data-connection>
</hazelcast>
1 (Required) Broker URL, mandatory property.
2 (Optional) Broker HTTP Service - necessary for resource listing.
3 (Optional) Options to limit which resources are visible when querying for resource list using this DataConnection (call to listResources).
4 (Optional) If true, transactions will be enabled on the client. Default is false. Please note, that transactions must be first enabled in the broker configuration.
PropertyDefault valueDescriptionWhen to use
brokerUrlPulsar Broker URL, in a form of pulsar://myhost:port/Mandatory
httpServiceUrlURL of Pulsar's instance HTTP service, typically on port 8080. Example: http://myhost:8080/If you want to be able to list resources via Data Connections (not just treat like a connection pool).
tenantIf set, it limits which resources (topics) will be listed to only those of given tenant.Only if you want to restrict access to subset of topics.
namespaceIf set, it limits which resources (topics) will be listed to only those of given namespace.Only if you want to restrict access to subset of topics.
enableTransactionsfalseEnables transaction for returned Pulsar Clients.If you want to use transactions with clients returned by this Data Connection. Must be set to true if used with Pulsar sink / SQL "INSERT INTO" Pulsar topic with EXACTLY_ONCE processing guarantee.

Example Hazelcast Data Connection​

This example configuration shows a data connection to a remote Hazelcast cluster. You can use a Hazelcast data connection from the Pipeline API in Sources#remoteMapJournal source.

note

Currently, no SQL connector is available for Hazelcast data connections. This means that although you can create a data connection in SQL, you cannot yet use it in SQL, for example, in a mapping statement.

xml
<hazelcast>
  <data-connection name="my-remote-hazelcast">
    <type>Hz</type>
    <properties>
      <property name="client_xml"> (1)
        <![CDATA[ (2)
          <?xml version="1.0" encoding="UTF-8"?>
          <hazelcast-client xmlns="http://www.hazelcast.com/schema/client-config"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:schemaLocation="http://www.hazelcast.com/schema/client-config
              http://www.hazelcast.com/schema/client-config/hazelcast-client-config-6.0-SNAPSHOT.xsd">

            <cluster-name>dev</cluster-name>
              <network>
                <cluster-members>
                  <address>172.17.0.2:5701</address>
                </cluster-members>
              </network>
          </hazelcast-client>
        ]]>
      </property>
    </properties>
    <shared>true</shared>
  </data-connection>
</hazelcast>
1 (Required) Specify exactly one of client_xml, client_yml, client_xml_path, client_yml_path.
2 Hazelcast client configuration to connect to a remote cluster. See Configuring Java Client. You can specify an external file with the client_xml_path property instead of using an embedded configuration file.

Configuration Options for Data Connections​

Data connections have the following configuration options.

note

If you are using Java to configure the Mapstore, use the DataConnectionConfig object.

OptionDescription
DefaultExample
name (required)The unique identifier for the data connection.
type (required)The type of data connection required for your external system. The following types of connection are supported: JDBC, Kafka, Mongo, Hz (case-insensitive).
propertiesAny configuration properties that the data connection expects to receive.
sharedWhether the data connection instance is reusable in different MapStores, jobs, and SQL mappings. This behavior depends on the implementation of the specific data connection. The default value is true. See the implementation of each data connection type for full details of reusability: KafkaDataConnection, MongoDataConnection, HazelcastDataConnection.
tip

You can use the EncryptionReplacer class to hide sensitive data in configuration files.

Types of Data Connection​

The following types of data connection are available for use.

TypeDescriptionProperties
JDBCConnect to external systems that support JDBC, including MySQL and PostgreSQL.See example. If there is more than one JDBC connection used on a single member from a single job, they will share the same data store and connection pool.
KafkaConnect to a Kafka data source.See example and Create a Kafka Mapping.
MongoConnect to a MongoDB database.See example.
HzConnect to a remote Hazelcast cluster.See example.
note

If you use the slim distribution of Hazelcast with a built-in data connector, make sure that you have an appropriate driver on your cluster's classpath.

Next Steps​

Use your configured connection:

Edit this page