AWS CloudWatch

Amazon CloudWatch

CloudWatch is where AWS puts metrics, logs and alarms. Most of it works automatically, which is convenient and also why two specific defaults quietly cost teams money and visibility for years.

Metrics

A metric is identified by a namespace (AWS/EC2), a name (CPUUtilization) and dimensions (InstanceId=i-0abc123). AWS services publish theirs automatically at no charge.

aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0abc123 \
  --start-time 2026-09-11T00:00:00Z --end-time 2026-09-12T00:00:00Z \
  --period 300 --statistics Average Maximum

The metrics EC2 does not give you

This catches nearly everyone. EC2 publishes CPU, network and disk I/O by default, but not memory usage and not disk space used. The hypervisor cannot see inside the instance.

So the two numbers you most want during an incident are absent unless you install the CloudWatch agent:

sudo dnf install -y amazon-cloudwatch-agent
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config -m ec2 -c ssm:AmazonCloudWatch-linux -s

Teams routinely discover this mid-incident, having assumed memory was being collected all along. Install the agent as part of your base image or launch template, not as a reaction.

The agent publishes these as custom metrics, which are billed per metric per month, so publish the handful you will actually alarm on rather than everything available.

Logs

Log groups hold streams; streams hold events. Lambda, ECS and many services write here automatically.

The retention default that costs the most

A new log group's retention is "Never expire." Logs accumulate forever and are billed for storage forever. On a busy account this becomes one of the larger CloudWatch line items, entirely by default and entirely invisibly.

# Find the offenders
aws logs describe-log-groups \
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' --output table

# Fix one
aws logs put-retention-policy --log-group-name /aws/lambda/my-fn --retention-in-days 30

Set retention on every log group. Thirty days suits most application logs; ninety or more for audit trails. For long-term retention, export to S3, where storage is far cheaper than CloudWatch Logs.

Logs Insights

A query language over your logs, and far more capable than scrolling a stream:

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 50

fields @timestamp, @duration, @maxMemoryUsed
| filter @type = "REPORT"
| stats avg(@duration), max(@duration), max(@maxMemoryUsed) by bin(5m)

That second query on a Lambda log group tells you whether the function is near its memory limit and how duration is trending. The two things you need to size it correctly.

Insights bills per gigabyte scanned, so narrow the time range before running broad queries across large groups.

Metric filters

Turn a log pattern into a metric you can alarm on. Useful when an application emits an error line but no metric:

aws logs put-metric-filter \
  --log-group-name /aws/lambda/payments \
  --filter-name payment-failures \
  --filter-pattern '"PAYMENT_FAILED"' \
  --metric-transformations \
      metricName=PaymentFailures,metricNamespace=Acme/Payments,metricValue=1

Alarms

An alarm watches one metric and moves between three states:

aws cloudwatch put-metric-alarm \
  --alarm-name payments-api-5xx \
  --namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count \
  --dimensions Name=LoadBalancer,Value=app/payments/abc123 \
  --statistic Sum --period 60 \
  --evaluation-periods 3 --datapoints-to-alarm 2 \
  --threshold 10 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:oncall

Three settings there deserve attention:

--treat-missing-data decides what happens when a metric stops reporting. The default, missing, leaves the alarm in its current state, so an alarm on a metric that vanishes because the service died may sit quietly in OK. For a metric that should always have data, breaching is safer; for one that is naturally sparse, notBreaching prevents false alarms.

--datapoints-to-alarm with --evaluation-periods gives an "M out of N" rule, which tolerates a single noisy data point without missing a real problem.

INSUFFICIENT_DATA is not a failure. It commonly means the resource is idle and the metric is simply not being emitted.

Composite alarms combine several alarms with boolean logic, which is the right way to stop one incident paging five times:

aws cloudwatch put-composite-alarm \
  --alarm-name payments-degraded \
  --alarm-rule "ALARM(payments-api-5xx) OR ALARM(payments-api-latency)"

Where the bill comes from

CloudWatch costs surprise people because each component is individually cheap:

The dimension point is the one that produces shocking bills. A custom metric with a user_id dimension creates one billable metric per user. Dimensions should be low-cardinality: environment, service, region, never an identifier.

Practical controls: set retention on every log group, drop debug logging in production, publish only metrics you alarm on, and export long-term logs to S3.

Beyond the basics

• Container Insights and Lambda Insights add curated dashboards and deeper metrics for ECS, EKS and Lambda, at additional cost.
• EventBridge: formerly CloudWatch Events. Routes AWS events to targets and is how you react to state changes rather than poll for them.
• CloudWatch Synthetics runs scripted canaries against your endpoints, catching outages that internal metrics miss because nothing is being asked to fail.

Go deeper in this section

Quick reference

---

Key takeaways

• EC2 does not report memory or disk usage without the CloudWatch agent. Install it in your base image.
• Log groups default to "Never expire." Set retention on every one. It is usually the biggest avoidable cost.
• --treat-missing-data decides what happens when a metric stops. The default can leave a dead service showing OK.
• Each dimension combination is a separate billable custom metric. Never put an identifier in a dimension.
• INSUFFICIENT_DATA usually means idle, not broken.
• Use datapoints-to-alarm with evaluation-periods so one noisy sample does not page anyone.
• Composite alarms stop a single incident paging five times.

---

Check your understanding

---

Frequently Asked Questions

Why is there no memory metric for my EC2 instance?

Because EC2 only publishes what the hypervisor can observe from outside the guest. CPU, network, and disk I/O. Memory usage and filesystem utilisation live inside the operating system and require the CloudWatch agent to be installed and configured on the instance. It publishes them as custom metrics, which carry a per-metric monthly charge, so configure it to send the few you will actually alarm on. Building the agent into your AMI or launch template avoids discovering the gap during an incident.

How do I stop CloudWatch Logs costing so much?

Set a retention policy on every log group. New groups default to "Never expire", so logs accumulate and are billed indefinitely. This is usually the single largest avoidable CloudWatch cost. Find them with aws logs describe-log-groups --query 'logGroups[?!retentionInDays].logGroupName' and set something sensible, typically thirty days for application logs. Also reduce ingestion by disabling debug logging in production, since ingestion is charged per gigabyte, and export anything needing long retention to S3 where storage is far cheaper.

What does INSUFFICIENT_DATA mean on an alarm?

The alarm does not have enough data points in the evaluation window to decide. Most often this means the resource is idle and simply is not emitting the metric. A load balancer with no requests, or a queue with no messages. It is not inherently a problem. If it matters that a metric should always be present, set --treat-missing-data breaching so a metric that stops reporting triggers the alarm rather than leaving it silent, which is the safer configuration for anything that should always be alive.

How do I search across CloudWatch Logs?

Use Logs Insights, which provides a purpose-built query language over one or more log groups. Filtering with filter, aggregating with stats, and bucketing over time with bin(). It is far more effective than scrolling individual streams, and it is the practical way to answer questions like how latency is trending or how close a Lambda is to its memory limit. Queries are billed per gigabyte scanned, so narrow the time range before running something broad across a large group.

What is the difference between a metric filter and Logs Insights?

A metric filter continuously matches a pattern in incoming log events and publishes a CloudWatch metric from the matches, so you can alarm on something that only appears in logs. Logs Insights is an ad-hoc query tool for investigating logs after the fact. Use a metric filter when you need alerting on a log pattern, and Insights when you are exploring. The two complement each other: Insights to discover a pattern worth watching, a metric filter to alarm on it.

Should I use CloudWatch or Prometheus and Grafana?

CloudWatch is the path of least resistance on AWS, with automatic metrics from every service, no infrastructure to run, and native integration with alarms and EventBridge. Prometheus and Grafana offer a far richer query language, cheaper high-cardinality metrics, and portability across clouds, at the cost of running and scaling them yourself. Many teams use both. CloudWatch for AWS service metrics and alarms, Prometheus for application metrics where cardinality would make CloudWatch custom metrics prohibitively expensive.