Get Started With Hazelcast Jet Job Management

In this tutorial, you’ll learn the basics of how to manage stream processing pipelines using the Hazelcast CLC with Hazelcast Cloud. You’ll see how to connect to a cluster on Cloud, and submit a sample Jet job that generates a stream of numbers. You’ll also learn how to use simple commands to monitor and cancel jobs.

Before You Begin

You need the following:

Hazelcast CLC installed on your local machine
One running Cloud cluster. You can create a development cluster using the Hazelcast CLC.
Cloud API key and secret
JRE 11 or above.
Gradle 8 or above.

Step 1. Authenticating with Cloud

To allow the Hazelcast CLC to perform cluster operations, including submitting Jet jobs, you must generate a Cloud token.

Execute the following command to retrieve the token.
```
clc cloud loginshell
```
When prompted, enter your API key and secret. If both are correct, the token is retrieved and saved.

Step 2. List Cluster Details

Next, check that the Hazelcast CLC can access your cluster by running the following command:

clc cloud list-clustersshell

The details of all clusters linked to your Cloud account are returned, including the Cluster ID, Cluster Name, Current Status, Hazelcast Version.

Step 3. Connect to Your Cluster

For Hazelcast CLC to connect to your development cluster, you need to import the cluster’s configuration, including its connection details.

Run the following command to save the cluster configuration with the name dev. Replace the $CLUSTER_NAME placeholder with the name of your cluster, shown in the top-left corner of the dashboard.

clc cloud import-config $CLUSTER_NAME --name devshell

Step 4. Build the Jet Job

Now you’re ready to build the sample Jet job.

Clone the GitHub repository.

git clone https://github.com/hazelcast/hazelcast-commandline-clientshell

Switch to the project directory.

cd hazelcast-commandline-client/examples/platform/simple-streaming-pipeline

Build the project with the following command:
```
gradle shadowJarshell
```

Step 5. Submit and Monitor the Jet Job

Submitting the sample Jet job generates a stream of numbers and creates a hashid for each number. The number and hashid pair are then saved to a Map called my-map, with the number as the key and hashid as the value.

Execute the following command to submit the Jet job:

clc -c dev job submit build/libs/simple-streaming-pipeline-1.0-SNAPSHOT-all.jar --name simple-pipeline --waitshell

The Hazelcast CLC waits until the job starts to run.

Now, list the jobs in the cluster. You should see simple-pipeline in a RUNNING state.
```
clc -c dev job listshell
```
Create a new advanced script in your favorite text editor. For example:
```
nano monitor.starshell
```

Add the following:

def main():
    for i in range(1000):
        size = map_size(name="my-map")
        print(size)
        sleep(3 * SECOND)python

Run your script.
```
clc -c dev script run monitor.starshell
```
See how the size of my-map increases as entries are written to it.
Press Ctrl+C when you’re ready to stop the script.

Step 6. Cancel the Jet job

Now, cancel the Jet job.

clc -c dev job cancel simple-pipelineshell

Run your script (from step 5) to see the size of my-map now.

Summary

In this tutorial, you learned how to do the following:

Connect to a cluster on Cloud.
Build and submit a Hazelcast Jet job to create a data pipeline.
Manage the lifecycle of a Jet job using list and cancel commands.

Learn More

Use these resources to continue learning: