The Serverless Guide to AWS CloudWatch Logs Insights

Thomas Ankcorn
Sr Software Engineer at Baselime
12 March, 2023
Demistifying uncovering your systems deepest secrets with Cloudwatch Log Insights and sharing its biggest gotcha. This blog post will make you a Cloudwatch Log Insight pro in just 10 minutes.
The Essential Guide to AWS CloudWatch Logs Insights for Serverless

CloudWatch Logs Insights is the interactive log search and analysis solution within AWS. It is a powerful analytics tool that enables you to search and query your log data. It provides you with a bespoke query syntax to debug and extract insights from your applications.

Debugging in the cloud is awesome with the right tools
Cloud Engineer Finding A Bug

CloudWatch Logs Insights is one of the essential pieces of any serverless architecture, and we will outline its benefits, limitations, and pricing in human terms.


Some foundations will help when using CloudWatch Logs Insights to understand your application. Here is a rundown:

Log Event

A log event is a record containing information emitted from your systems.

CloudWatch automatically capture logs events from the standard out and standard error streams of your AWS Lambda functions. For instance, each console.log in your JavaScript Lambda functions or fmt.Println in your Golang Lambda function. You can also send log events with the AWS SDKs or CLI using the put-log-events API.

Each log must have a timestamp and message.

Log Stream

A CloudWatch Log Stream is a sequence of log events; log streams are helpful to isolate log entries that happened within the same context. When used with the AWS serverles ecosystem, each AWS Lambda instance gets a unique log stream.

This is ideal for low-traffic systems or during developement, as individual Lambda invocations or consecutive invacations will belong to the same log stream, making it easier to quickly find them. However, log streams can become difficult to manage on systems with high traffic, as consecutive invocations are likely to trigger multiple AWS Lambda instances, creating multiple Log Streams and making it difficult to quicly find logs in CloudWatch.

Log Group

A CloudWatch Log Group is a collection of log streams; It is possible to set retention policies and access control settings on logs groups, giving you more control over your infrastructure and ultimately, your cloud bill.

For AWS Lambda, each Lambda function typically gets a unique log group.

Query Syntax

CloudWatch Logs Insights uses a custom query language designed to filter and manipulate data in your CloudWatch log groups.

Queries built with this query language are unstructured and typically composable, enabling you to reuse snippets from your existing queries or any examples you find online easily.

Analyzing log data

The first step to querying logs in CloudWatch Logs Insights is selecting the log groups to query. CloudWatch Logs Insights supports querying up to 50 log groups in a single query. Moreover, query results are available for 7 days, and queries timeout after 60 minutes [source].

When ingesting data from other AWS services, CloudWatch typically add discovered fields to your logs, to facilitate querying. For instance, when ingesting logs from AWS Lambda functions, CloudWatch adds fields such as @timestamp, @logStream, @message, @requestId, @duration, @billedDuration, @type, @maxMemoryUsed, @memorySize. The full set of discovered fields is outlined here.

The Cloudwatch Log Insights Query UI
Query UI

Displaying Data: fields and display

The first step to uncovering insights from your logs is typically selecting the data to display. There are 2 operators for displaying data in CloudWatch Logs Insights: fields and display.

Displaying a field: fields

The field command is used to show specific fields in the query results.

fields @timestamp, @requestId, @message

The query above shows the @timestamp, @requestId, and @message in every log entry.

The fields command can be used to create new fields for the query or modify field values. The example below creates a new field total as the sum of count and num and shows it in the query results.

fields count + num as total

Moreover, the field total can be used in the remainder of the query as if it was part of the log events, for example, to filter the log events based on the value of total.

Displaying the results of a query: display

The display command is very similar to the fields command, as it is used to define the fields to display in a query. However, it has multiple limitations:

  • It should be used only once per query, as only the last display command in the query is applied
  • It cannot be used to create new ephemeral fields to use elsewhere in the query

as such, the display command should typically be used when you don't want to use the fields command.

Filtering Data: filter

CloudWatch Logs Insights provides powerful filtering capabilities to finding any event using the filter command.

Working with strings

To filter on string values, CloudWatch Logs Insights provides the keyword like and the regular expression operator ~=.

filter message like "Exception"

In the example above, we're filtering all log entries where the message includes the substring Exception.

It's possible to combine multiple filters using the and and or keywords, and negate specific operations with the not keyword.

filter message like "Exception" and message not like "Handled" and error ~= /40.*/ or error like /50.*/

A handy tip is to make your queries more readable is to use multiple filter commands to chain filters instead of and.

filter message like "Exception"
| filter error ~= /40.*/

Working with numbers

To filter on numbers, CloudWatch Logs Insights provides arithmetic comparison operators: =, !=, <, <=, >, >= that work exactly how you'd expect them to.

filter count > 32

Parsing JSON and strings: parse

The parse command is used to extract data from JSON blobs and strings into ephemeral fields that can be used in subsequently in the query (for display, filtering or calculations...).

The parse command intuitively matches the string or JSON blob and pulls out the data matching the position of the * character.

Considering this log entry:

12 March 2023 21:09:43,241 [ERROR] {'error': 'ValidationError', 'status': 400} User affected: 123456

The snippet below extracts level, exception and user from the log entry.

parse @message "[*] 'error': '*' User affected: *" as level, exception, user

As such, level has the value ERROR, exception is ValidationError and user is 123456.

Computing statistics: stats

The stats command enable you to compute aggregates statistics on log entry fields. To use CloudWatch Logs Insights beyond relatively trivial debugging, it is essential to get familiar with the stats command. Appropriate use of the stats command will uncover trends and deeper insights into the behaviour of your applications.

Mutltiple functions are available, from aggregate functions such as avg, count, max, or min to non-aggregate functions such as earliest or latest. A complete list with descriptions is available in the documentation.

The example below computes the average duration of AWS Lambda invocations based on the logs.

filter @type = "REPORT"
| stats avg(@duration)

The CloudWatch Logs Insights query language provides a bin function that enables us to group results by time periods. For example, we can view the average duration of Lambda function invocations every 5 minutes with the following query.

filter @type = "REPORT"
| stats avg(@duration) by bin(5m)

The stats command can be used to build relatively complex queries.

Sorting results: sort

The sort command is used to sort the results in in ascending or descending order based on a field.

The query below shows the AWS Lambda invocations with the highest billed duration, by displaying the request ID and billed duration, sort by billed duration in descending order.

filter @type = "REPORT"
| fields @requestId, @billedDuration
| sort by @billedDuration desc

Controlling the results: limit

The limit command is used to restrict the number of rows displayed in the query results. It should however be noted that the limit command does not reduce the amount of data scanned by CloudWatch Logs Insights.

filter @type = "REPORT"
| fields @requestId, @billedDuration
| sort by @billedDuration desc
| limit 10

The query above shows the 10 AWS Lambda invocations with the highest billed duration.

The query output from cloudwatch insights
Query Output

Pricing in Human Terms

CloudWatch Logs and Logs Insights have generous free tiers for small applications. They can ingest and query up to 5GB of log data for free each month.

Once your application exceeds the tier quota, CloudWatch Logs Insights can get expensive.

The following pricing outlines pricing the eu-west-1 region. Please refer to the official documentation different regions.

  • $0.50 per GB ingestion cost
  • $0.03 per GB storage cost
  • $0.005 per GB of data scanned to query
  • $0.12 per GB of data Scanned to protect and mask sensitive data

Pricing Example

Let's say you run a small application exclusively on AWS Lambda. With low traffic, your functions are invoked 100 times a minute. You're producing 50kb of log data with each invocation. We are assuming a retention period of 30 days accross your log groups:

  • Number of invocations per month: 100 * 43800 = 4,380,000 invocations
  • Data total: 219 GB of log data
  • Ingestion cost: $109.5
  • Cost to store for 1 month: $6.57
  • Cost to mask sensitive data: $26.28

Assuming you don't act on your log data, the cost of CloudWatch Logs is $142.35.

However, observability without action is just storage.

Assuming your run 100 queries daily on your log data, each scanning 20GB of log data:

  • Number of queries: 3000
  • Data Scanned: 60TB
  • Query cost: $300

Total Cost: $442.35

Cost Optimisation

In order to optimise your CloudWatch Logs and Logs Insights bill, you should aim to keep data your data within CloudWatch only for the duration the data is vital for operations, and archive it to long term storage afterwards. Additionally, it is necessary to limit the amount of data scanned during queries.

  • Short retention periods: Depending on your compliance requirements, you should expire logs from CloudWatch when they are not relevant for your operations anymore
  • Short query time ranges: CloudWatch charges per GB of data scanned when running queries. Limiting the time range of your queries will reduce your bill
A duck looking sad because he needs top optimise his cloudwatch insihts costs
Optimise Cloudwatch Insights Costs

We recommend against ingesting less data to reduce the CloudWatch bill. It's true that some logs may seem unnecessary, but this telemetry data is our window into our systems, and removing logs can be the cause of blind spots when investigating defects or during incidents.


CloudWatch Logs Insights is typically the first step in modern observability for developers working with serverless architectures. It's a powerful solution that enables you to get an understanding of the inner workings of your applications. However, it has quite a few limitations, and the pricing model disincentivises actually making use of your logs.

We're working on a better observability solution for serverless architectures: Baselime; one that enables you to ask deeper questions of your systems, ties together your logs, metrics, traces and all other telemetry data, and with a simple pricing model that encourages making use of your telemetry data.

Try it out, it's free.

More from the blog
Changelog #006
10 March, 2023

Changelog #006

Service discovery, dashboards, reliability and much more...

Boris Tane
Understand your serverless stack with Baselime Tracing
17 February, 2023

Understand your Serverless Stack with Baselime Tracing

Whether with OpenTelemetry or AWS X-Ray, get deeper insights into your serverless architecture with distributed tracing

Boris Tane
Changelog #005
02 February, 2023

Changelog #005 - We're live

You can sign up to Baselime today, no more waiting list.

Boris Tane

Reduce your MTTR for free

Stop firefighting production defects and reduce your Mean Time To Recovery today.

🍪 By browsing this website you are agreeing to our cookies policy.