How to Monitor AWS API Gateway Access Logs via Kinesis Data Firehose and Openobserve
Manas Sharma
November 30, 2024
7 min read
Introduction
Monitoring AWS API Gateway access logs is super important for keeping an eye on how your APIs are performing, spotting issues, and making sure you're using resources efficiently. By connecting AWS API Gateway directly to Amazon Kinesis Data Firehose, you can easily stream your access logs straight to OpenObserve. This setup not only makes it simple to analyze your logs but also helps you uncover trends and detect any anomalies, so you can make informed decisions to boost your API’s reliability and performance.
What Is Amazon Kinesis Data Firehose?
Amazon Kinesis Data Firehose is a fantastic, fully managed service that lets you load real-time data streams into AWS storage services like Amazon S3, Amazon Redshift, or Amazon Elasticsearch. With Firehose, capturing and streaming your log data is a breeze—no complicated setup required! This means you can focus on monitoring and improving your APIs without getting bogged down in the technical details.
So, let’s dive in and get started!
Prerequisites
Before we start, ensure you have the following:
- An AWS account
- Basic understanding of AWS services (API Gateway, Lambda, Kinesis Firehose, CloudWatch)
- OpenObserve set up and running
Step 1: Create a Lambda Function
We will be creating a sample lambda function that will accept HTTP request events from API gateway and process them.
Navigate to the AWS Lambda Console
- Go to the AWS Lambda console.
Create a New Function
- Click on Create function.
- Choose Author from scratch.
- Function name:
ApiLogFunction - Runtime:
Python 3.x(or your preferred language). - Click Create function.
Add Code to the Lambda Function
In the Lambda function's code editor, implement a demo function that will serve as the backend for our REST API
import json
def lambda_handler(event, context):
return {
'statusCode': 200,
'body': json.dumps({'message': 'Logged to Kinesis Firehose!'})
}
Set Permissions
Ensure the Lambda function has permissions to log to CloudWatch. Attach the
AWSLambdaBasicExecutionRolepolicy.
Step 2: Create a REST API in API Gateway
Navigate to the API Gateway Console
- Go to the API Gateway console.
Create a New API
- Click on Create API.
- Choose REST API and click Build.
- API name:
accesslogsdemoapi. - Click Create API.
Create a Resource
- Select your API and click on Actions > Create Resource.
- Resource Name:
log. - Click Create Resource.
Create a Method
- Select the
/logresource. - Click on Actions > Create Method.
- Choose
GETand click the checkmark. - Integration type: Lambda Function.
- Select your Lambda function (
ApiLogFunction). - Click Save and OK to grant API Gateway permission to invoke your Lambda function.
Step 3: Create an Amazon Kinesis Data Firehose
Navigate to Kinesis Console
- In the AWS services menu, search for Kinesis and select it.
Create a Delivery Stream
- Click on Create Firehose Stream.
- Choose Direct PUT as the source.
- Click Next.
Configure Destination
- Select HTTP Endpoint as the destination.
- Endpoint URL: Enter your OpenObserve Kinesis Firehose endpoint (obtain this from OpenObserve -> Ingestions -> Custom -> Amazon Kinesis Firehose.
- Authentication Type: Choose Use access key and enter your OpenObserve access key.
- Click Next.
Configure Backup Settings
- Choose whether to back up all data or only failed data. For this example, select Failed data only.
- Choose an existing S3 bucket or create a new one for backups (e.g.,
lambda-firehose-backup). - Click Next.
Finalize Firehose Stream Setup
- Click Next, configure buffer size and interval as needed (default values will suffice for this example), then click Next again.
- Review all settings and click on Create Firehose stream.
Step 4: Enable Custom Access Logs in API Gateway
Create an IAM Role for CloudWatch Logs
- Go to the IAM console.
- Click on Roles > Create role.
- Trusted entity: Select API Gateway.
- Click Next: Permissions and attach the
AWSLambdaBasicExecutionRolepolicy. - Click Next: Tags, then Next: Review.
- Name the role:
APIGatewayCloudWatchLogsRole. - Click Create role.
Copy the Role ARN
- Click on the created role and copy the Role ARN.
Set the Role ARN in API Gateway
- Go back to the API Gateway console.
- Click on Account Settings.
- Paste the Role ARN in the CloudWatch Logs role ARN field.
- Click Save Changes.
Enable Access Logs
- Select your API and navigate to the Stages section.
- Select the dev stage.
- In the Logs/Tracing tab, enable Access Logs.
- Provide the ARN of your Kinesis Firehose stream.
Specify the log format, e.g.:
RequestId: $context.requestId, SourceIP: $context.identity.sourceIp, Method: $context.httpMethod, ResourcePath: $context.resourcePath, StatusCode: $context.status, ResponseLength: $context.responseLength, RequestTime: $context.requestTimSave your chnages.
Step 5: View REST API Access Logs in OpenoObserve
Ensure OpenObserve is set up and running. Refer to the Quickstart documentation for installation and configuration.
Invoke Rest API with Postman
Use a tool like Postman or curl to make a GET request to your API endpoint:
curl -X GET https://irj0x8btg3.execute-api.us-east-1.amazonaws.com/dev/log
Once you've invoked your API a few times with different test cases, you should be able to see meaningful logs within OpenObserve:
Analyzing Logs in OpenObserve
Once the above steps are completed, logs are ingested into OpenObserve, you can visualize and analyze them using dashboards. You can download the dashboards here.
- Go to Streams → Logs or simply navigate to the Logs tab.
- Search for logs related to your stream or filter by log level (e.g.,
level=error).
Key Features of the Rest API Access Logs Dashboard
Status Code Distribution:
- Visualize the distribution of HTTP status codes to quickly assess API health and identify common errors.
Requests by HTTP Method:
- Analyze API usage by different HTTP methods (GET, POST, etc.), helping to understand user interactions and optimize performance.
Latest Access Logs Table:
- View the most recent API access logs with key details, enabling quick identification of issues and monitoring of user activity.
Requests Over Time:
- Track trends in API requests over time, providing insights into usage patterns and helping with capacity planning.
You can now run queries on these logs and dashboard for further analysis and continuous monitoring. You've successfully created a robust and scalable system to monitor logs for your REST API!
Troubleshooting and Next Steps
Troubleshooting
If you encounter issues with your setup, pay attention to the following:
- For the Access log destination ARN, ensure you enter the ARN of your Kinesis Firehose stream in the correct format:
arn:aws:firehose:{region}:{account-id}:deliverystream/amazon-apigateway-{your-stream-name}
Note: The name of your Firehose stream must be
amazon-apigateway-{your-stream-name}. This naming convention is crucial for the integration to work seamlessly.If logs are not appearing in OpenObserve, double-check that your API Gateway is configured correctly to send logs to the specified Firehose stream. Ensure that the logging settings are enabled and that the API Gateway has the necessary permissions to write to Kinesis Firehose.
Setting Up Permissions
To ensure that your API Gateway can successfully log to Kinesis Firehose, it's important to set up appropriate permissions. You can follow the guidelines provided in the AWS documentation to configure the necessary IAM roles and policies. For detailed instructions, refer to the following link:
Setting Up Kinesis Access Logging Permissions.
Next Steps
To further enhance your monitoring capabilities, consider exploring additional features and best practices. The comprehensive guide on integrating AWS CloudWatch Logs with Amazon Kinesis Firehose provides valuable insights on improving your monitoring with OpenObserve. This guide will help you streamline your logging process and gain deeper insights into your API performance:
