Configuring the Jet Engine

Data Pipelines allow you to process data stored in one location and send the result to another, such as from a data lake to an analytics database or into a payment processing system. You can also use the same source and sink so the pipeline only processes data. You can configure Hazelcast’s Jet engine using the Hazelcast Platform Operator.

Configuring the Jet Engine

The Jet engine is enabled by default and has the following configuration options. For more detailed information, see the API Reference.

Option Default Description

enabled

true

Set to false to disable the Jet engine.

resourceUploadEnabled

false

Set to true to enable the uploading of resources for Jet jobs.

instance/cooperativeThreadCount

The number of threads Jet creates in its cooperative multithreading pool.

instance/flowControlPeriodMS

100

The duration of the interval between flow-control packets.

instance/backupCount

The number of synchronous backups to configure on the IMap, which Jet needs internally to store job metadata and snapshots.

instance/scaleUpDelayMS

10000

The delay after which the auto-scaled jobs restart if a new member joins the cluster.

instance/losslessRestartEnabled

false

Whether the Lossless Cluster Restart feature is enabled.

Lossless Restart requires Persistence to be enabled. For more info about Persistence, see Persistence, Backup, and Restore.

instance/maxProcessorAccumulatedRecords

The maximum number of records that can be accumulated by any single processor instance.

edgeDefaults/queueSize

Sets the capacity of processor-to-processor concurrent queues.

edgeDefaults/packetSizeLimit

Limits the size of the packet in bytes.

edgeDefaults/receiveWindowMultiplier

Sets the scaling factor used by the adaptive receive window sizing function.

bucketConfig

JAR files from an external bucket are placed accessible for the member when the following parameter values are supplied:

secretName: Name of the Secret object which holds the credentials for your cloud provider.
bucketURI: Full path for the external bucket. For example: gs://your-bucket/path/to/jars.

configMaps

List of names of ConfigMaps. Files in each ConfigMap will be downloaded.

remoteURLs

List of URLs from where the files will be downloaded.

Example configuration

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  version: '5.4.0-slim'
  licenseKeySecretName: hazelcast-license-key
  jet:
    enabled: true
    resourceUploadEnabled: true
    instance:
      cooperativeThreadCount: 4
      flowControlPeriodMillis: 100
      backupCount: 1
      scaleUpDelayMillis: 10000
      losslessRestartEnabled: false
      maxProcessorAccumulatedRecords: 1000000000
    edgeDefaults:
      queueSize: 1024
      packetSizeLimit: 16384
      receiveWindowMultiplier: 3

Configuration Properties

You can set configuration properties as Java system properties. See Setting Arbitrary JVM Arguments for more information.

Example configuration

apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  version: '5.4.0-slim'
  licenseKeySecretName: hazelcast-license-key
  jet:
    enabled: true
  jvm:
    args:
    - "-Dhazelcast.jet.idle.cooperative.min.microseconds=50"