aws-step-functions

Files (16)

• SKILL.md
• compensation-saga-pattern.asl.json · assets/
• express-standard-handoff.asl.json · assets/
• human-in-the-loop-with-timeout-escalation.asl.json · assets/
• nested-map-parallel-structures.asl.json · assets/
• polling-loop-wait-check-choice.asl.json · assets/
• scatter-gather-with-partial-results.asl.json · assets/
• semaphore-concurrency-lock.asl.json · assets/
• architecture-patterns.md · references/
• asl-state-types.md · references/
• error-handling.md · references/
• migrating-from-jsonpath-to-jsonata.md · references/
• processing-state-inputs-and-outputs.md · references/
• service-integrations.md · references/
• transforming-data.md · references/
• validation-and-testing.md · references/

SKILL.md (first lines)

# AWS Step Functions

## Overview

AWS Step Functions uses Amazon States Language (ASL) to define state machines as JSON. With AWS Step Functions, you can create workflows, also called state machines, to build distributed applications, automate processes, orchestrate microservices, and create data and machine learning pipelines.

This skill provides comprehensive guidance for writing state machines in ASL, covering:

- ASL structure and JSONata expression syntax
- Details on the eight available workflow states
- The `$states` reserved variable
- Workflow variables with `Assign`
- Error handling
- AWS Service integration patterns
- Example code for data transformation and architecture
- Validation and testing of state machines
- How to migrate from JSONPath to JSONata

The AWS MCP server is recommended for sandboxed execution and audit logging when following this skill, but all steps use AWS CLI syntax and work without it.

## When to Load Reference Files

Load the appropriate reference file based on what the user is working on:

- **ASL structure**, **state types**, **Task**, **Pass**, **Choice**, **Wait**, **Succeed**, **Fail**, **Parallel**, **Map** → see `references/asl-state-types.md`
- **Error handling**, **troubleshooting**, **Retry**, **Catch**, **fallback**, **error codes**, **States.Timeout**, **States.ALL** → see `references/error-handling.md`
- **Service integrations**, **Lambda invoke**, **DynamoDB**, **SNS**, **SQS**, **SDK integrations**, **Resource ARN**, **sync**, **async** → see `references/service-integrations.md`
- **Migrating from JSONPath to JSONata**, **migration**, **JSONPath to JSONata**, **InputPath**, **Parameters**, **ResultSelector**, **ResultPath**, **OutputPath**, **intrinsic functions**, **Iterator**, **payload template** → see `references/migrating-from-jsonpath-to-jsonata.md`
- **Validation**, **linting**, **testing**, **TestState**, **test state**, **mock**, **mocking**, **unit test**, **inspection level**, **DEBUG**, **TRACE**, **validate state**, **test in isolation** → see `references/validation-and-testing.md`
- **Architecture patterns**, **examples**, **polling**, **saga**, **compensation**, **scatter-gather**, **semaphore**, **lock**, **human-in-the-loop**, **escalation**, **Express to Standard** → see `references/architecture-patterns.md`
- **Data transformation**, **JSONata expressions**, **filtering**, **aggregation**, **string operations**, **$reduce**, **$lookup**, **$toMillis**, **$partition**, **$parse**, **$hash**, **$uuid** → see `references/transforming-data.md`
- **State input/output**, **$states**, **Assign**, **Output**, **Arguments**, **variable scope**, **variable limits**, **evaluation order**, **passing data between states** → see `references/processing-state-inputs-and-outputs.md`

## Quick Reference

### Standard vs Express Workflows

|                                   | Standard                             | Express                                     |
| --------------------------------- | ------------------------------------ | ------------------------------------------- |
| **Max duration**                  | 1 year                               | 5 minutes                                   |
| **Execution semantics**           | Exactly-once                         | At-least-once (async) / At-most-once (sync) |
| **Execution history**             | Retained 90 days, queryable via API  | CloudWatch Logs only                        |
| **Max throughput**                | 2,000 exec/sec                       | 100,000 exec/sec                            |
| **Pricing model**                 | Per state transition                 | Per execution count + duration              |
| **`.sync` / `.waitForTaskToken`** | Supported                            | Not supported                               |
| **Best for**                      | Auditable, non-idempotent operations | High-volume, idempotent event processing    |

**Choose Standard** for: payment processing, order fulfillment, compliance workflows, anything that must never execute twice.

**Choose Express** for: IoT data ingestion, streaming transformations, mobile backends, high-throughput short-lived processing.

> **When recommending Express, the single limitation you must always state — even for fire-and-forget / high-throughput pipelines — is that Express does NOT support `.sync` or `.waitForTaskToken`** (no callbacks, no nested `.sync` waits, no human-approval or job-completion waits). Also note: 5-minute max duration, no queryable execution history (CloudWatch Logs only), and at-least-once (async) / at-most-once (sync) execution — so non-idempotent work can run twice. If any of these matter, choose Standard (exactly-once, up to 1 year, full history).

### Setting the State Machine Query Language

JSONata is the preferred way to reference and transform data in ASL. It replaces the five JSONPath I/O fields (`InputPath`, `Parameters`, `ResultSelector`, `ResultPath`, `OutputPath`) with just two: `Arguments` (inputs) and `Output`.

**Enable at the top level** to apply to all states:

```json
{ "QueryLanguage": "JSONata", "StartAt": "...", "States": {...} }
```

**Or per-state** to migrate from JSONPath incrementally:

```json
{ "Type": "Task", "QueryLanguage": "JSONata", ... }
```

**JSONPath is supported** and is the default if `QueryLanguage` is omitted — existing state machines do not need to be migrated.

**Field mapping (JSONPath → JSONata):**

| JSONPath field | JSONata equivalent |
| --- | --- |
| `Parameters` (keys use `key.$`) | `Arguments` — drop the `.$` suffix and wrap each value in `{% %}` |
| `ResultSelector` and `OutputPath` | `Output` (reference the raw result via `$states.result`) |
| `ResultPath` | `Assign` (preferred) or `Output` |
| `InputPath` | not needed — reference `$states.input` directly |

> **A state uses one query language, not both.** Never mix JSONPath fields (`InputPath`/`Parameters`/`ResultSelector`/`ResultPath`/`OutputPath`) with JSONata fields (`Arguments`/`Output`) in the same state — this is the most common migration error. See `references/migrating-from-jsonpath-to-jsonata.md` for full details.

### How Assign and Output Are Evaluated (Parallel, Not Sequential)

Within a single state, `Assign` and `Output` are evaluated **at the same time — in parallel — both reading the same data (the state input plus the task result)**. They are NOT evaluated one after the other. Because they run together, a variable you set in `Assign` is **not** visible in that same state's `Output`: there is no ordering in which `Output` could observe the just-assigned value. The assigned value becomes available only to **subsequent** states.

So if you set a variable in `Assign` and reference it in the same state's `Output`, you get the old/undefined value — not because `Output` runs "before" `Assign`, but because both evaluate concurrently from the same snapshot. To use the value immediately, reference it in the **next** state (variables persist across states); to shape the current state's output from the task result, use `$states.result` directly in `Output`.

### Unit Testing a State with TestState

Test a single state **without deploying the state machine or calling the real service** using the TestState API (`aws stepfunctions test-state`) with `--mock`. A complete answer covers all four points:

- **Mock the service response exactly** — the `--mock` `result` MUST match the target AWS service's API response schema exactly (field names are case-sensitive). For a Lambda `invoke` Task that is `StatusCode` and `Payload`: `--mock '{"result":"{\"StatusCode\":200,\"Payload\":{...}}"}'`.
- **All three inspection levels** (`--inspection-level`): `INFO` (default — `output`, `status`, `nextState`), `DEBUG` (adds data flow: `afterArguments`, `result`, `variables` — use to debug JSONata/data flow), `TRACE` (adds raw HTTP `request`/`response`, for HTTP Task).
- **`.sync` and `.waitForTaskToken` integrations still require a mock** — for `.sync`, mock the polling API (e.g. `DescribeExecution`, not the initial call); for `.waitForTaskToken`, also pass `--context '{"Task":{"Token":"..."}}'`.
- **No deployment** or real invocation is needed — the state is tested in isolation.

See `references/validation-and-testing.md` for per-service mock structures and error/retry/Map/Parallel testing.

## Best Practices

- Set `"QueryLanguage": "JSONata"` at the top level for new state machines unless the user wants to use JSONPath
- Keep `Output` minimal — only include what the state immediately after the current state needs
- Use `Assign` to store variables needed in later states instead of threading it through Output
- Use `$states.input` to reference original state input
- `Assign` and `Output` are evaluated **in parallel from the state's entry data, NOT sequentially** — a variable set in `Assign` is therefore NOT visible in the same state's `Output` (which still sees the pre-`Assign` values); the new value takes effect only in the next state.
- All JSONata expressions must produce a defined value — `$data.nonExistentField` throws `States.QueryEvaluationError`
- Use `$states.context.Execution.Input` to access the original workflow input from any state
- Save state machine definitions with `.asl.json` extension when working outside the console
- Prefer the optimized Lambda integration (`arn:aws:states:::lambda:invoke`) over the SDK integration

## Troubleshooting

### Common Errors

- `States.QueryEvaluationError` — JSONata expression failed. Check for type errors, undefined fields, or out-of-range values.
- Mixing JSONPath fields with JSONata fields in the same state.
- Using `$` or `$$` at the top level of a JSONata expression — use `$states.input` instead.
- Forgetting `{% %}` delimiters around JSONata expressions — the string will be treated as a literal.

From aws/agent-toolkit-for-aws, Apache-2.0.

More from aws/agent-toolkit-for-aws