Send Logs of Hazelcast Viridian Dedicated Clusters to Amazon OpenSearch

Set up a remote logging solution using Amazon OpenSearch Service. In this tutorial, you’ll set up an Amazon OpenSearch domain and configure your cluster to automatically send log files to it. You’ll also learn how to view these logs in Kibana.

Amazon OpenSearch Service is based on Elasticsearch, and allows you to search, analyze and visualize your cluster logs using Kibana.

Step 1. Set Up an Amazon OpenSearch Domain

Create an Amazon OpenSearch domain to receive log files from your Hazelcast Viridian cluster.

  1. Create an AWS account if you don’t have one already.

  2. For testing purposes, you can sign in as root user, but we recommend that you sign in with an IAM account for Production use.

    Amazon provides a managed policy for its OpenSearch service.
  3. From the list of regions on the top menu, select the region that’s closest to your Viridian cluster.

  4. Search for and select Amazon OpenSearch Service. The Dashboard opens ready for you to start configuration.

    Dashboard for Amazon OpenSearch Service

  5. Click Create domain.

Step 2. Configure Your OpenSearch Domain

To get a logging solution up and running, update the following domain settings. You can go back and adjust them later to suit your needs.

  1. In the Domain name field, enter a short name for your OpenSearch domain. For example, hazelcast-cloud-logs.

  2. In the Deployment type section:

    • For testing purposes, select Development and testing as the deployment type.

    • From the Version list, select Elasticsearch > 7.10.

    • Select Include older versions.

  3. In the Data nodes section, under Availability Zones, select 1-AZ.

  4. In the Network section, select Public access to use fine-grained access control to secure the OpenSearch domain and data. VPC Peering connections between Viridian clusters and OpenSearch instances are not currently supported.

  5. In the Fine-grained access control section, select Create Master User and enter a username and password for the account.

    Make a note of the master user credentials. You’ll need them later to access Kibana, and to make any configuration changes to your OpenSearch domain.
  6. In the Access policy section, select Only use fine-grained access control.

  7. Click Create to build your domain. You’ll need to wait until your OpenSearch environment and the following URLs are created. You need both URLs to complete your logging setup.

    • Kibana URL

    • Domain endpoint

Step 3. Configure Logging Integration in Hazelcast Viridian

Configure your Viridian Dedicated cluster to send log files to the OpenSearch domain.

You’ll need the master user credentials and domain endpoint that you created when you configured OpenSearch.

  1. Sign into the Hazelcast Viridian console and select the cluster that want to integrate with OpenSearch.

  2. Select Manage > Logging.

  3. From the Logging Technology list, select Elastic Stack (ELK).

  4. In the Username and Password fields, enter your master user credentials.

  5. In the URL field, enter the domain endpoint for your OpenSearch domain followed by port number 443.

    OpenSearch supports HTTPS over port 443.

  6. Click Create Integration. Within 30 seconds, Hazelcast starts to send log files to OpenSearch, which are made available in Kibana.

Step 4. Set Up a Kibana Dashboard

Create an index pattern to load the logs received from OpenSearch into a Kibana dashboard.

You’ll need the master user credentials and Kibana URL that you created when you configured OpenSearch.

To create log data for viewing, check for the following:

  • Your Viridian cluster is running.

  • A client is writing data to the cluster.

  1. From your domain details page in OpenSearch, click the Kibana URL.

  2. Log in to Kibana, using your master user credentials.

  3. Select Explore on my own to skip onboarding.

  4. From the Select tenant dialog box, select Private and click Confirm.

  5. From the main menu, select Kibana > Discover to start your set up.

  6. Click Create index pattern. An index pattern tells Kibana which indices contain the data that you want to view.

  7. In the Index pattern field, enter a wildcard * to find all log data received from OpenSearch.

  8. When Kibana finds the Viridian cluster logs, click Next step.

  9. From the Time field list, select updated_at so that you’ll be able to filter your log file data by time.

  10. Click Create index pattern. Your log file data is ready to view. Simply select Kibana > Discover from the main menu.

    Log data in Kibana Dashboard

If necessary, you can now return to your OpenServe domain and increase the number of nodes or availability zones to add resilience to the logging service.

Summary

In this tutorial, you learned how to do the following:

  • Create an Amazon OpenSearch domain to receive log files from your Viridian cluster.

  • Configure your Viridian Dedicated cluster to automatically send log files to the OpenSearch domain.

  • Load log files into a Kibana dashboard using an index pattern.