AWS Step Functions

Step Functions runs a workflow as a state machine defined in JSON. Each state calls a service, branches, waits, or runs work in parallel, and the service handles retries, error paths, timeouts and state between steps.

The point is what it removes. A multi-step process chained together with Lambdas invoking Lambdas needs retry logic in every function, has no record of where an execution got to, and fails in ways nobody can reconstruct afterwards. Step Functions makes the orchestration declarative and gives you a visual history of every execution.

Standard or Express

Decide this first. It cannot be changed after creation, and the cost difference is roughly two orders of magnitude.

Standard suits long-running business processes where you need auditability and exactly-once behaviour: order fulfilment, approval flows, ETL.

Express suits high-volume short-lived work: per-request processing, streaming transformations. Because it bills on duration rather than transitions, a workflow with many quick steps is dramatically cheaper. A busy Standard workflow at scale produces genuinely startling bills, because every transition is chargeable.

Express is at-least-once, so its steps must be idempotent.

A state machine

{
  "Comment": "Process an order",
  "StartAt": "ValidateOrder",
  "States": {
    "ValidateOrder": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:eu-west-1:123456789012:function:validate",
        "Payload.$": "$"
      },
      "ResultSelector": { "valid.$": "$.Payload.valid" },
      "ResultPath": "$.validation",
      "Retry": [{
        "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException", "States.TaskFailed"],
        "IntervalSeconds": 2, "MaxAttempts": 3, "BackoffRate": 2.0
      }],
      "Catch": [{
        "ErrorEquals": ["ValidationError"],
        "ResultPath": "$.error",
        "Next": "RejectOrder"
      }],
      "Next": "ChargePayment"
    },
    "ChargePayment": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:eu-west-1:123456789012:function:charge",
        "Payload.$": "$"
      },
      "TimeoutSeconds": 30,
      "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "RefundAndFail" }],
      "Next": "Fulfil"
    },
    "Fulfil": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sqs:sendMessage",
      "Parameters": {
        "QueueUrl": "https://sqs.eu-west-1.amazonaws.com/123456789012/fulfilment",
        "MessageBody.$": "$"
      },
      "End": true
    },
    "RejectOrder": { "Type": "Fail", "Error": "OrderRejected", "Cause": "Validation failed" },
    "RefundAndFail": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "arn:aws:lambda:eu-west-1:123456789012:function:refund", "Payload.$": "$" },
      "Next": "RejectOrder"
    }
  }
}

aws stepfunctions create-state-machine --name order-processing \
  --definition file://definition.json \
  --role-arn "arn:aws:iam::123456789012:role/StepFunctionsOrderRole" \
  --type STANDARD

That RefundAndFail path is the saga pattern: when a later step fails, run compensating actions for the earlier ones. Distributed transactions do not exist across AWS services, so compensation is how you get consistency, and having it declared in the workflow rather than buried in a function is much of the value.

Retries and catchers

Retry re-attempts the same state; Catch routes to a different state when retries are exhausted. Both match on error names.

"Retry": [
  { "ErrorEquals": ["States.TaskFailed"], "IntervalSeconds": 1, "MaxAttempts": 2 },
  { "ErrorEquals": ["States.ALL"], "IntervalSeconds": 5, "MaxAttempts": 3, "BackoffRate": 2.0, "MaxDelaySeconds": 60, "JitterStrategy": "FULL" }
]

Order matters. The first matching rule wins, so specific errors go before States.ALL.

JitterStrategy: FULL randomises the delay, which matters when many executions retry simultaneously against the same downstream service; without it they retry in lockstep and keep it overloaded.

Built-in error names worth knowing: States.Timeout, States.TaskFailed, States.Permissions, States.ExceedToleratedFailureThreshold. A Lambda throwing a custom exception surfaces as that exception's name, which is how ValidationError above is matched.

Always set TimeoutSeconds on a Task. Without it a Standard workflow can wait a year for a state that will never complete, and you pay for the wait.

Input and output processing

The field that causes most confusion. Four filters apply in order:

The .$ suffix means "this value is a path, not a literal", "Payload.$": "$" passes the whole input, while "Payload": "$" passes the literal string $. Forgetting the .$ is a frequent and confusing bug.

ResultPath: "$.validation" merges the result into the existing state under that key. Omitting ResultPath replaces the entire state with the task's output, which silently loses everything upstream. The usual cause of "my data disappeared halfway through".

ResultSelector is worth using on Lambda tasks to strip the SDK envelope, so downstream states see your payload rather than $.Payload.… everywhere.

Map state for parallel work

"ProcessItems": {
  "Type": "Map",
  "ItemsPath": "$.orders",
  "MaxConcurrency": 10,
  "ToleratedFailurePercentage": 5,
  "ItemProcessor": {
    "ProcessorConfig": { "Mode": "INLINE" },
    "StartAt": "ProcessOne",
    "States": {
      "ProcessOne": {
        "Type": "Task",
        "Resource": "arn:aws:states:::lambda:invoke",
        "Parameters": { "FunctionName": "…:process", "Payload.$": "$" },
        "End": true
      }
    }
  },
  "Next": "Summarise"
}

Set MaxConcurrency. The default is unlimited, so a Map over 10,000 items launches 10,000 concurrent Lambdas, which exhausts your account concurrency, throttles every other function, and often overwhelms whatever database they all connect to. This is one of the easier ways to cause a self-inflicted outage.

ToleratedFailurePercentage lets a batch job survive a few bad items rather than failing entirely.

For very large datasets, distributed map mode reads directly from S3 and runs up to 10,000 parallel child executions, escaping the 256 KB payload limit that inline mode inherits.

Direct SDK integrations

Step Functions can call most AWS APIs directly, with no Lambda:

"SaveOrder": {
  "Type": "Task",
  "Resource": "arn:aws:states:::aws-sdk:dynamodb:putItem",
  "Parameters": {
    "TableName": "Orders",
    "Item": { "orderId": {"S.$": "$.orderId"}, "status": {"S": "processed"} }
  },
  "End": true
}

Over 200 services are available this way. Every Lambda you remove is a cold start, a deployment artefact and a per-invocation charge you no longer have, and for a straightforward "write this to DynamoDB" step, the function was never doing anything else.

Waiting for something external

"AwaitApproval": {
  "Type": "Task",
  "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
  "Parameters": {
    "FunctionName": "…:request-approval",
    "Payload": { "input.$": "$", "taskToken.$": "$$.Task.Token" }
  },
  "TimeoutSeconds": 604800,
  "Next": "Fulfil"
}

The workflow pauses, costing nothing while waiting, until something calls back with the token:

aws stepfunctions send-task-success --task-token "$TOKEN" --output '{"approved":true}'

This is how human approval steps work, and a Standard workflow can wait up to a year. $$ accesses the execution context; $$.Task.Token is the callback token.

Debugging an execution

aws stepfunctions start-execution --state-machine-arn "$SM_ARN" \
  --input '{"orderId":"A-4821"}' --name "order-A-4821-$(date +%s)"

Execution names must be unique for 90 days, which is why a timestamp is appended.

aws stepfunctions get-execution-history --execution-arn "$EXEC_ARN" \
  --reverse-order --max-items 5 \
  --query 'events[].[type,taskFailedEventDetails.error,taskFailedEventDetails.cause]' \
  --output table

------------------------------------------------------------------
| ExecutionFailed  | ValidationError | {"errorMessage":"missing customerId"} |
| TaskStateExited  | None            | None                                  |
------------------------------------------------------------------

--reverse-order puts the failure first, which is almost always what you want. The cause field carries the actual error from the underlying service, that is where the answer is, not in the state machine definition.

Common errors

States.Permissions: The state machine does not have permission to invoke

The execution role lacks permission for that service. Step Functions needs its own role; the Lambda's role is separate.

States.DataLimitExceeded

Payload between states exceeded 256 KB. Pass an S3 reference instead of the data itself.

ExecutionAlreadyExists

That execution name was used within the last 90 days.

States.Timeout

The task exceeded TimeoutSeconds, or a Lambda exceeded its own timeout.

Quick reference

---

Key takeaways

• Standard bills per transition, Express per duration: roughly 100x apart at volume, and the type is permanent.
• Express is at-least-once, so its steps must be idempotent.
• Always set MaxConcurrency on a Map state, or you exhaust account Lambda concurrency.
• Omitting ResultPath replaces the whole state, silently discarding upstream data.
• The .$ suffix means "this is a path"; without it you pass a literal string.
• Set TimeoutSeconds on every Task, or a stuck state can wait a year.
• Use direct SDK integrations instead of a Lambda that only calls one API.
• get-execution-history --reverse-order and read the cause field.

---

Check your understanding

---

Frequently Asked Questions

What is the difference between Standard and Express workflows?

Standard bills per state transition, runs up to a year, keeps a full queryable execution history for 90 days, and guarantees exactly-once execution. Express bills on requests and duration, is capped at five minutes, logs only to CloudWatch, and is at-least-once, so its steps must be idempotent. At high volume Express costs on the order of 1% of Standard, because a Standard workflow with many quick steps pays for every transition. The type is fixed at creation, so choose deliberately.

When should I use Step Functions instead of chaining Lambdas?

When the process has more than two or three steps, needs retries with backoff, has error paths that differ by failure type, or when you will later be asked what happened to a specific execution. Chained Lambdas put retry logic in every function, keep no record of progress, and are very hard to reconstruct after a failure. Step Functions makes all of that declarative and gives you a visual history per execution. For a single function reacting to an event, it is unnecessary overhead.

How do retries work in Step Functions?

Each state can carry a Retry array matching on error names, with IntervalSeconds, MaxAttempts, BackoffRate and optionally MaxDelaySeconds and JitterStrategy. Rules are evaluated in order and the first match wins, so list specific errors before States.ALL. Catch is the companion: when retries are exhausted it routes to another state rather than failing the execution. Set JitterStrategy: FULL when many executions may retry at once, so they do not hammer a recovering service in lockstep.

Why did my workflow lose data between states?

Because a task without ResultPath replaces the entire state with its own output, discarding everything upstream. Set ResultPath to a key such as "$.payment" so the result is merged in rather than substituted. ResultSelector is worth adding on Lambda tasks too, to strip the invocation envelope so downstream states read your fields directly instead of reaching into $.Payload. Neither produces an error when wrong, which is what makes it confusing to diagnose.

Can Step Functions call AWS services without a Lambda?

Yes, and it should. Optimised integrations exist for common targets such as SQS, SNS, DynamoDB, ECS and Lambda itself, and the generic aws-sdk: integration covers over 200 services and thousands of API actions. A task that only writes to DynamoDB or publishes to SNS needs no function at all. Removing it eliminates a cold start, a deployment artefact and a per-invocation charge. Reserve Lambda for steps that contain real logic.

How do I debug a failed Step Functions execution?

aws stepfunctions get-execution-history --execution-arn <arn> --reverse-order puts the failure first, and the taskFailedEventDetails.cause field carries the underlying service's actual error, that is where the answer is, not in the definition. The console's graph view highlights the failed state and shows input and output at each step, which is usually faster for a one-off. States.Permissions means the state machine's execution role is missing something, which is a separate role from any Lambda's.