Shipping Logs to Amazon ElasticSearch Service
Amazon provides ElasticSearch as a service which is called ElasticSearch Service. Basically, you can create ElasticSearch services with some built-in plugins like Kibana. In this tutorial, you will see how easy it is to set up an ElasticSearch service on AWS then ship Hazelcast Cloud Cluster logs to ElasticSearch Service.
Setting Up ElasticSearch Service Instance
Go to https://us-west-2.console.aws.amazon.com/es/home and click Create a new domain
On the initial page of domain creation, you can select your deployment type. For testing purpose, you can select Development and Testing, or if it is a live environment you can select Production option, Custom otherwise. And 7.7 is recommended for Elasticsearch version.
On the next screen, you can configure your Elasticsearch instance’s specs; domain name, machine type, and EBS volume size.
Once you configured specs, you can continue with the security part which is the most critical one. It is not possible to use VPC Access if you will ship logs from Hazelcast Cloud since there should be a VPC Peering between ES instance and Hazelcast Cloud Cluster VPC, at least there is no support for now. Select Public Access and scroll down below configurations.
Since your ES instance has public access, you need to apply Fine-Grained access control to allow certain users to access. Select Create master user and provide username/password pair.
| Save Username / Password pair to a safe place to use it Hazelcast Cloud logging configuration in next steps. |
Select Domain access policy as Allow open access to the domain
The configuration part is done, and you should see Loading in your dashboard
After a while (~5-8 mins after), you should see it is Active and it will provide some endpoints to be used.
| Please store Endpoint value in a safe place to use in Hazelcast Cloud Logging Configuration step. |
Hazelcast Cloud Logging Configuration
To configure logging, you can go to Hazelcast Cluster Detail Page > Settings > Logging Configuration
Select Elastic Stack (ELK) from Logging Technology list.
In this Configuration page, provide Username, Password, and Url you saved in previous steps. In URL section, it will require port number, so the value will be something like https://<domain>:443 since Elasticsearch Service provides Elasticsearch through this port not 9200. Once you complete this step, your logging configuration will be ready in ~30 seconds.
Setting up Elasticsearch Service Kibana Dashboard
In Elasticsearch Service instance detail page, you can see the URL for Kibana, please click on it and you will see a login page. Use username/password as you defined in the creation phase.
Once you logged in, you can skip the onboarding part, and click Discover menu on the left-hand-side to define the index pattern. Please put the wildcard char in the text box to handle all the logs coming from Hazelcast Cloud Clusters.
Once you click the Next step, you can skip the Settings part then you will see some aggregated table data for summary and now you are ready to discover data of Hazelcast Cloud Clusters.
You can go to the Discover page from the left menu and see logs coming from Hazelcast Cloud Cluster.
