AWS Kinesis

Amazon Kinesis

Kinesis is a family of services for streaming data. Two of them matter for most work, and choosing wrongly costs a lot of unnecessary effort.

Data Streams is a durable, ordered, replayable log. Records persist for 24 hours by default and up to a year, multiple consumers read independently, and each can re-read from any point.

Data Firehose is a managed delivery pipeline. It buffers records and writes them to S3, Redshift, OpenSearch or an HTTP endpoint with no code, no shards and no consumers to run.

If your goal is "get this data into S3", use Firehose. People reach for Data Streams, write a consumer, manage shards and reimplement batching. All to do what Firehose does with a configuration block.

Firehose first

aws firehose create-delivery-stream \
  --delivery-stream-name events-to-s3 \
  --delivery-stream-type DirectPut \
  --extended-s3-destination-configuration '{
    "RoleARN": "arn:aws:iam::123456789012:role/FirehoseRole",
    "BucketARN": "arn:aws:s3:::my-events",
    "Prefix": "events/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/",
    "ErrorOutputPrefix": "errors/",
    "BufferingHints": { "SizeInMBs": 128, "IntervalInSeconds": 300 },
    "CompressionFormat": "GZIP",
    "DataFormatConversionConfiguration": { "Enabled": true }
  }'

Two details worth getting right.

The Hive-style prefix (year=…/month=…/day=…) makes the output directly partitionable by Athena and Glue, so queries scan one day rather than the whole bucket. Retrofitting partitioning later means rewriting the object layout.

Buffering hints trade latency for cost and query efficiency. 128 MB or 300 seconds produces fewer, larger objects. Cheaper to store and far faster to query. A one-minute buffer produces many small files, which is the classic mistake that makes an Athena table slow.

DataFormatConversionConfiguration converts JSON to Parquet in flight, typically cutting both storage and query cost by an order of magnitude for analytical use.

Data Streams and shards

Use Data Streams when you need multiple independent consumers, replay, ordering per key, or sub-second processing.

aws kinesis create-stream --stream-name orders --stream-mode-details StreamMode=ON_DEMAND

Start on-demand. It scales automatically and removes shard management entirely. Provisioned is cheaper at steady high volume, and requires you to get the shard count right:

aws kinesis create-stream --stream-name orders --shard-count 4

A shard is the unit of capacity: 1 MB/s or 1,000 records per second in, 2 MB/s out. Those limits are per shard, so total capacity is shards multiplied by the limit.

The partition key decides everything

Records are assigned to shards by a hash of the partition key, so a low-cardinality key concentrates traffic on one shard, a hot shard, which throttles while the others sit idle.

aws kinesis put-record --stream-name orders \
  --partition-key "customer-4821" \
  --data "$(echo -n '{"event":"placed"}' | base64)"

That last row is the real trade. Ordering is guaranteed only within a shard, so records sharing a partition key are ordered relative to each other. Using a random key gives perfect distribution and no ordering at all. Fine for independent events, wrong for a sequence of updates to the same entity.

An error occurred (ProvisionedThroughputExceededException) when calling the
PutRecord operation: Rate exceeded for shard shardId-000000000001 in stream orders

One shard named while others are fine is a hot key, not insufficient capacity.

Consumer lag is the metric that matters

aws cloudwatch get-metric-statistics --namespace AWS/Kinesis \
  --metric-name GetRecords.IteratorAgeMilliseconds \
  --dimensions Name=StreamName,Value=orders \
  --start-time 2026-09-15T00:00:00Z --end-time 2026-09-16T00:00:00Z \
  --period 300 --statistics Maximum

IteratorAgeMilliseconds is how old the records being processed are. Steady near zero means consumers keep up; rising means they do not, and if it reaches the retention period, data is lost permanently. Records expire before anyone reads them.

This is the single most important Kinesis alarm. Set it well below retention, so a few hours of climbing lag is a page rather than a postmortem.

When lag rises, the usual causes are a slow consumer, too few shards for the volume, or a hot shard where one consumer is overwhelmed while others idle.

Consumers

Lambda is the simplest:

aws lambda create-event-source-mapping \
  --function-name process-orders \
  --event-source-arn "$STREAM_ARN" \
  --starting-position LATEST \
  --batch-size 100 \
  --maximum-retry-attempts 3 \
  --maximum-record-age-in-seconds 3600 \
  --bisect-batch-on-function-error \
  --parallelization-factor 4 \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:eu-west-1:123456789012:kinesis-dlq"}}'

Four of those options prevent real problems.

--bisect-batch-on-function-error splits a failing batch to isolate the bad record. Without it, one unparseable record fails its batch repeatedly and blocks that shard until the record expires: a poison pill that stalls the stream for up to the retention period.

--maximum-retry-attempts and --maximum-record-age-in-seconds bound how long a bad record is retried.

--destination-config sends failures to a DLQ instead of discarding them.

--parallelization-factor runs several concurrent invocations per shard, preserving order per partition key while raising throughput.

By default Lambda processes one batch per shard at a time, so throughput is bounded by shard count multiplied by how fast your function runs.

Enhanced fan-out

Standard consumers share the shard's 2 MB/s read limit, so five consumers get 400 KB/s each. Enhanced fan-out gives each registered consumer its own dedicated 2 MB/s and pushes records rather than being polled:

aws kinesis register-stream-consumer \
  --stream-arn "$STREAM_ARN" --consumer-name analytics

It costs more per consumer-shard-hour and drops propagation delay to around 70 ms. Worth it beyond two or three consumers, or where latency matters.

Kinesis, SQS or MSK

Choose SQS for distributing work among workers where each message is handled once. Choose Kinesis when several systems need the same stream, or you need replay. Choose MSK when you already have Kafka expertise and tooling, or need portability off AWS. Accepting considerably more operational work.

The common mistake is using Kinesis as a work queue. If each record has exactly one consumer and you never replay, SQS is simpler, cheaper and has dead-letter handling built in.

Resharding

aws kinesis update-shard-count --stream-name orders \
  --target-shard-count 8 --scaling-type UNIFORM_SCALING

Resharding is online but not free: old shards enter a CLOSED state that consumers must drain before reading the new ones, and the KCL and Lambda handle that automatically while custom consumers often do not. Doubling or halving is the cleanest change.

This is the main reason on-demand mode is a good default. It removes resharding from your operational surface entirely.

Common errors

ProvisionedThroughputExceededException: Rate exceeded for shard shardId-…

A hot partition key if one shard is named; genuine under-provisioning if all are.

ExpiredIteratorException: Iterator expired

A shard iterator was held longer than five minutes. Fetch a fresh one.

ResourceInUseException: Stream orders is currently UPDATING

Resharding in progress; wait for ACTIVE.

Lambda stops processing one shard. A poison record blocking the batch. Enable bisect-on-error and a failure destination.

Records missing. IteratorAgeMilliseconds reached retention and they expired unread.

Quick reference

---

Key takeaways

• If the destination is S3, use Firehose. Data Streams is for multiple consumers and replay.
• Set Hive-style prefixes and large buffers in Firehose, or your Athena queries will be slow.
• Start on-demand. It removes shard sizing and resharding entirely.
• A shard is 1 MB/s in and 2 MB/s out, shared by standard consumers.
• Partition key cardinality decides throughput. One shard throttling is a hot key.
• Ordering exists only within a shard, so a random key means no ordering.
• Alarm on IteratorAgeMilliseconds: reaching retention loses data permanently.
• Enable --bisect-batch-on-function-error or one bad record blocks a shard.
• Use SQS if each record has one consumer and you never replay.

---

Check your understanding

---

Frequently Asked Questions

What is the difference between Kinesis Data Streams and Data Firehose?

Data Streams is a durable, replayable log: records persist for 24 hours up to a year, several consumers read independently, and each tracks its own position so it can re-read. Firehose is a managed delivery pipeline that buffers records and writes them to S3, Redshift, OpenSearch or an HTTP endpoint with no consumers to write or shards to manage. If the goal is getting data into S3, use Firehose. Building that on Data Streams means reimplementing batching, compression and partitioning that Firehose does with configuration.

When should I use Kinesis instead of SQS?

When several independent systems need the same records, or when you need to replay history. SQS delivers each message to one consumer and deletes it, with no replay; Kinesis keeps records for the retention period and lets every consumer read them at its own pace from any point. If each record has exactly one consumer and you never need to re-read, SQS is simpler, cheaper, and has dead-letter handling built in. Using Kinesis as a work queue is a common and avoidable mistake.

How do I choose a Kinesis partition key?

Pick something with high cardinality so records spread evenly across shards. A customer, device or session identifier. Low-cardinality keys such as an event type concentrate traffic on a few shards, which throttle while the rest idle, and total stream capacity does not help because limits are per shard. The trade-off to understand is ordering: records are ordered only within a shard, so a random key gives perfect distribution and no ordering guarantees at all. Choose the key whose ordering you actually need.

What does IteratorAgeMilliseconds mean and why alarm on it?

It is how old the records your consumers are currently processing are. Effectively consumer lag. Steady near zero means processing keeps pace with ingestion. A rising value means it does not, and if the age reaches the stream's retention period, records expire before anyone reads them and the data is gone permanently. That makes it the most important Kinesis alarm: set the threshold well below retention so a few hours of growing lag is something you act on rather than discover afterwards.

Why did my Lambda consumer stop processing one shard?

Almost certainly a poison record. Kinesis processing is ordered per shard, so Lambda will not skip past a batch that keeps failing, by default it retries until the records expire, blocking that shard for up to the retention period. Enable --bisect-batch-on-function-error so the batch is split to isolate the offending record, set --maximum-retry-attempts and --maximum-record-age-in-seconds to bound the retrying, and add a --destination-config failure destination so the bad record is captured rather than lost.

What is enhanced fan-out and do I need it?

Standard consumers share a shard's 2 MB/s read throughput, so five consumers get roughly 400 KB/s each and compete for it. Enhanced fan-out registers a consumer for its own dedicated 2 MB/s per shard and pushes records rather than being polled, cutting propagation delay to around 70 milliseconds. It costs more per consumer-shard-hour, so it earns its place beyond two or three consumers or where latency genuinely matters. With one consumer, standard polling is fine.