Meet us at Konnect - The Flagship Event of Kalaam Telecom at Bahrain on 5th June, 2024.
See you at GITTEX GLOBAL 14-18 October 2024 as we launch our AI based Xops Platform
Meet us at Konnect - The Flagship Event of Kalaam Telecom at Bahrain on 5th June, 2024.
See you at GITTEX GLOBAL 14-18 October 2024 as we launch our AI based Xops Platform

How to Stream Logs from CloudWatch to OpenSearch Using AWS Kinesis

This comprehensive guide will take you through the process of streaming logs from Amazon CloudWatch to Amazon OpenSearch via AWS Kinesis Data Streams and Firehose, with a transformation step using AWS Lambda.


Architecture Diagram.



  • Active AWS account with administrative access to ensure the creation and management of necessary resources.
  • IAM user with attached policies granting full access to Amazon CloudWatch, Kinesis Data Stream and Firehose, AWS Lambda, and Amazon OpenSearch Service.
  • Need to have an OpenSearch Service domain up and running.
  • The process is based on logs and alerts aggregated in a CloudWatch log group in the security account. This is the same account that has the OpenSearch Service domain deployed.


Step 1: Creating a CloudWatch Log Group

  1. Access CloudWatch:
  • Open the AWS Management Console.
  • Navigate to CloudWatch under the Management & Governance section.
  1. Create Log Group:
  • In the CloudWatch dashboard, find the ‘Logs’ section in the sidebar.
  • Click ‘Create log group’ and enter a descriptive name (e.g., “/aws/lambda/my-function-logs”).
  • Configure retention settings to determine how long you want to keep the logs.
  1. Log Streams:
  • In your log group, create a log stream to categorize logs further.


Step 2: Configuring Kinesis Data Stream

  1. Open Kinesis Service:
  • Go to the Kinesis service from the AWS Management Console.
  1. Create Data Stream:
  • Click ‘Create data stream’ and enter a name (e.g., “MyCloudWatchLogsStream”).
  • Decide on the number of shards. A shard provides a fixed unit of capacity. More shards mean higher throughput.


Step 3: Streaming from CloudWatch to Kinesis Data Stream

  1. Set Up Streaming:
  • Return to the CloudWatch log group you created.
  • Select your log group, and from the ‘Actions’ dropdown, choose ‘Stream to Amazon Kinesis Data Stream’.
  • In the wizard, select the data stream you created in Step 2.
  1. Configure Log Format:
  • Choose the log format for your data stream records.
  • You can stream the entire log data or select specific fields using a subscription filter pattern.


Step 4: Transforming Logs with Lambda

  1. Create Lambda Function:
  • Go to the Lambda service from the AWS Management Console.
  • Click ‘Create function,’ select ‘Author from scratch,’ and input a name for your function (e.g., “TransformLogData”).
  • Choose an execution role with permission to read from Kinesis and write to OpenSearch and Firehose.
  1. Write Transformation Code:
  • Input your transformation logic into the Lambda code editor or upload a ZIP file if your code is more complex.
  • Set environment variables if your function requires them.
  • Configure timeout and memory settings according to the expected workload.
  1. Deploy and Test Function:
  • Save and deploy your function.
  • You can test it by configuring a test event in the Lambda console that simulates a Kinesis data record.


Step 5: Setting Up Kinesis Firehose Delivery Stream

  1. Create Firehose Stream:
  • Navigate to Kinesis Firehose in the AWS Management Console.
  • Click ‘Create delivery stream,’ and select ‘Kinesis data stream’ for the source.
  • Choose the data stream from the list provided.
  1. Configure Lambda Transformation:
  • Select the Lambda function you created for data transformation.
  • Set buffer size and interval to group records before sending them to the function.
  1. Configure Destination:
  • Select ‘Amazon OpenSearch Service’ for the destination.
  • Provide the domain details and specify the index, type, and retry options.
  1. Backup Failed Data:
  • Optionally, you can configure S3 backup for records not delivered to OpenSearch.


Step 6: Establishing OpenSearch Domain

  1. Create or Choose Domain:
  • If you don’t have an OpenSearch domain, create one by navigating to the OpenSearch Service and clicking ‘Create a new domain.’
  • If you already have a domain, ensure it’s configured to accept incoming data from Firehose.
  1. Secure Your Domain:
  • Apply access policies to control who can send data to and query your OpenSearch domain.
  • Ensure that your VPC and security group settings allow traffic from Firehose.


Step 7: Monitoring and Validation

  1. Monitor Log Group:
  • Check the CloudWatch log group to see if logs are being generated and streamed to Kinesis.
  1. Check Lambda Invocation:
  • In the Lambda console, monitor the function’s invocation metrics to ensure it’s being triggered as expected.
  1. Validate Firehose Delivery:
  • In the Firehose console, review the monitoring graphs and logs to confirm that records are being transformed and sent to OpenSearch.
  1. Verify in OpenSearch:
  • Log into the OpenSearch dashboard.
  • Verify that new indices are created, and logs are indexed correctly.
  1. Set Up Alerts:
  • Configure CloudWatch alarms or OpenSearch alerts to notify you of any issues in the pipeline.


Conclusion: You have set up a pipeline to monitor and analyze log data in near real-time by completing these steps. You can now leverage the power of OpenSearch to gain insights into your application’s performance and troubleshoot any issues.


What’s New Blog

AWS Security Compliance: A Hands-On Approach with SSM Automation

This alternative succinctly introduces the main focus of the blog post, which is navigating the realm of AWS security compliance

Streamlining Storage Management: Integrating Amazon EBS Volumes with Amazon ECS Fargate

Introduction: The integration of Amazon Elastic Container Service (ECS) with Amazon Elastic Block Store (EBS) marks a significant advancement in


Revolutionizing Remote Work with Amazon Workspace

Introduction: In the landscape of remote work, organizations are continually seeking innovative solutions to enhance productivity, collaboration, and security. Amazon


Demystifying Kubernetes: Understanding CNI, CSI, and CRI

Kubernetes has revolutionized container orchestration, offering unparalleled efficiency and scalability. Central to its success are three critical interfaces: Container Network


Revolutionizing Cloud Management with Xops Resource Optimizer

Introduction Xops offers a comprehensive suite of cloud management services, including cost optimization and security enhancements. The Resource Optimizer is

Subscribe for Faster updates