This is a prerelease version.

View latest

Submit Jet jobs

latest-snapshot 5.16 5.15 5.14 5.13 5.12 5.11 5.10 5.9 5.8 5.7

To execute a data pipeline it needs to be submitted to the cluster as a job. Once the job is submitted, it’s distributed automatically and executed on all members.

With Operator, you can run pipelines from existing JAR files for processing data. Pipelines, depending on the data source, can be used for stream or batch processing.

To create a pipeline using the JetJob CR, you must enable the Jet engine in the Hazelcast CR with resourceUploadEnabled set to true. See Create data pipelines with the Jet engine.

For a worked example, see the Run a data pipeline using Jet tutorial.

Configure the JetJob resource

You can use any of the following configuration options in the JetJob resource. You can find more detailed information in API reference page.

Field Description

name

Name of the Jet job to be created. If empty, the CR name will be used. It cannot be changed later.

hazelcastResourceName

Name of the Hazelcast resource.

state

Used to manage the job state. The default value is Running and its value must be Running when the JetJob object is created for the first time.

jarName

Name of a JAR file to run. Must be accessible to the member.

mainClass

Name of the main class that will be run on the job.

bucketConfig

JAR file specified in the jarName field, accessible from an external bucket.

  • secretName: Name of the Secret object that holds the credentials for your cloud provider.

  • bucketURI: Full path for the external bucket. For example: gs://your-bucket/path/to/jars.

remoteURL

URL from where the file will be downloaded.

URL parameters

The bucketURI parameter can be used for additional configuration. For example, to specify an AWS S3 URL to send requests to, you can use the endpoint parameter: s3://my-bucket?endpoint=my.endpoint.url&disableSSL=true&s3ForcePathStyle=true.

The following options are supported.

Field Description

region

The AWS region for requests.

endpoint

The endpoint URL (hostname only or fully qualified URI).

disable_ssl, disableSSL

A value of true disables SSL when sending requests.

s3_force_path_style, s3ForcePathStyle

A value of true forces the request to use path-style addressing.

dualstack

A value of true enables dual stack (IPv4 and IPv6) endpoints.

fips

A value of true enables the use of FIPS endpoints.

Provide a JAR file for the pipeline

To run the pipeline, you need to provide a JAR file that contains the pipeline. The JAR file can be downloaded before the cluster starts by configuring jet.bucketConfig, jet.remoteURLs, or jet.configMaps in the Hazelcast CR. This way, all the files in the bucket will be accessible to the member when the cluster starts.

Another option is to configure bucketConfig or remoteURL in the JetJob CR. This way, only the JAR file specified in the jarName parameter will be downloaded in the runtime before starting the pipeline.

Job state management

Once a job is created, you can use state field to manage its lifecycle. The following state values are available:

  • Running. All the jobs must be created with the Running state. Runs a newly created job or starts a Suspended job.

  • Suspended. Gracefully suspends a Running job.

  • Canceled. Gracefully stops a job.

  • Restarted. Suspends and resumes the job in one step.

Deleting the JetJob resource will forcefully cancel the job.

Example configuration

The following JetJob resource runs the Data Pipeline for the Hazelcast resources on the source Hazelcast cluster from my-data-pipeline.jar.

Example configuration
apiVersion: hazelcast.com/v1alpha1
kind: Hazelcast
metadata:
  name: hazelcast
spec:
  clusterSize: 3
  repository: 'docker.io/hazelcast/hazelcast-enterprise'
  jet:
    enabled: true
    resourceUploadEnabled: true
    bucketConfig:
      secretName: br-secret-gcp
      bucketURI: "gs://your-bucket/path/to/jars"
  licenseKeySecretName: hazelcast-license-key
---
apiVersion: hazelcast.com/v1alpha1
kind: JetJob
metadata:
  name: jet-job-sample
spec:
  name: my-test-jet-job
  hazelcastResourceName: hazelcast
  state: Running
  jarName: my-data-pipeline.jar
For more information about accessing resources on different cloud providers, see Authorization methods to access cloud storage.

Help and support