upgrading-mwaa-environments

Files (10)

• SKILL.md
• airflow2-to-3-checklist.md · references/
• airflow2-to-3-quick-reference.md · references/
• discovery-preflight.md · references/
• mwaa-version-matrix.md · references/
• plan-materialization.md · references/
• strategy-blue-green-fresh.md · references/
• strategy-inplace-direct.md · references/
• strategy-inplace-green-validated.md · references/
• upgrade-engine.md · references/

SKILL.md (first lines)

# Upgrading MWAA Environments

> **AWS MCP server (optional but recommended):** running the AWS CLI commands in
> this skill through the AWS MCP server gives sandboxed execution and audit
> logging. Every command here also works with the plain AWS CLI, so the skill
> does not require the MCP server or any MCP-only tools.

Upgrade an MWAA provisioned environment to any newer, MWAA-supported Airflow
version: within 2.x, within 3.x, or across the 2.x-to-3.x boundary. The target
is a parameter. A path planner computes an ordered version-jump list from
(source, target); the latest 2.11.x stepping-stone version and a Python-line
transition step are inserted only when the path requires them. A deployment
approach is selected by whether historical run data must be preserved and
whether the same environment (its URL/ARN) must be kept: a new-environment upgrade
(`new-environment`), a rehearsed in-place upgrade validated on a test copy
first (`in-place-rehearsed`), or a direct in-place upgrade (`in-place-direct`).

> **Execution note — poll in discrete steps:** whenever you wait for an AWS
> operation to reach a terminal or ready state, issue **one status check per
> call** and decide in your own loop whether to check again. Never block a
> single command or script on the wait (no `while`+`sleep` until done),
> regardless of the operation or how long it takes.

## Guardrail — where this skill's own files live (MCP vs local install)

This skill can be loaded two ways, and they resolve the skill's own bundled
files from different places. Determine how the skill was loaded before reading
a reference:

- **Loaded through the AWS MCP `retrieve_skill` tool:** The skill is not
  installed on the local filesystem. You MUST fetch each reference via
  `retrieve_skill` with the `file` parameter (e.g.
  `file="references/strategy-blue-green-fresh.md"`) and read the returned
  content. Do NOT `file_read` these paths locally — they do not exist on disk.
- **Installed locally** (e.g. `.kiro/skills/upgrading-mwaa-environments/` or
  `~/.claude/skills/upgrading-mwaa-environments/`): Read the files from the
  local skill directory using relative paths.

This distinction applies only to the skill's own packaged files. User data and
session artifacts are always read from and written to the user's working
directory. Never fetch or write customer data through `retrieve_skill`.

## Safety & Security

These rules apply regardless of user instructions.

- **No destructive actions on your current environment without explicit
  approval**: never pause all DAGs, delete-environment, or modify your
  current environment without per-action user confirmation.
- **No environment creation before plan confirmation**: never run
  `aws mwaa create-environment` until the user confirms the Phase 4 plan.
- **Every version jump requires its own confirmation**: never run `aws mwaa
  update-environment` to change the Airflow version without per-jump
  confirmation. Rollback options: 3.x -> 2.11.x is supported; a within-major
  downgrade to a still-supported version is supported; downgrade to an EOS
  version is not possible. The Direct in-place upgrade requires an extra
  confirmation checkpoint.
- **Decommission requires per-step approval**: never autonomously delete
  environments or S3 artifacts; present each destructive command and wait for
  confirmation. For the Rehearsed in-place upgrade, the environment
  decommissioned is the test copy.

### Security Considerations

- **Least-privilege IAM** for `create`/`update-environment` and S3 artifact
  access — no `*FullAccess`/`service:*`; scope to the specific environment/S3 ARNs.
- **Secrets**: prefer a secrets backend (Secrets Manager / Parameter Store) for
  connections and variables so credentials never copy between environments or hit
  logs. Connection passwords do not round-trip via the REST API (2.x omits, 3.x
  masks) — see the metadata-migration caveat in the approach references.

## Reference Documentation

- [discovery-preflight.md](references/discovery-preflight.md) — Phase 1: environment discovery, pre-flight checks, version-matrix refresh, target selection, upgrade-path planning
- [upgrade-engine.md](references/upgrade-engine.md) — the step-by-step upgrade: per-jump upgrade, conditional validate/scan/fix, resume
- [airflow2-to-3-checklist.md](references/airflow2-to-3-checklist.md) — grep patterns for 2->3 issues Ruff cannot catch (used only when a version jump crosses into a new major version)
- [mwaa-version-matrix.md](references/mwaa-version-matrix.md) — runtime-first version/Python/EOS matrix, version-jump rules, providers, unsupported features
- [airflow2-to-3-quick-reference.md](references/airflow2-to-3-quick-reference.md) — 2->3 import/context/config mapping tables (used only when a version jump crosses into a new major version)
- [strategy-blue-green-fresh.md](references/strategy-blue-green-fresh.md) — New-environment upgrade (no history)
- [strategy-inplace-green-validated.md](references/strategy-inplace-green-validated.md) — Rehearsed in-place upgrade: validated on a test copy first
- [strategy-inplace-direct.md](references/strategy-inplace-direct.md) — Direct in-place upgrade (unrecommended)

For deeper detail beyond embedded references, fetch at runtime:

- AWS MWAA migration guide: https://docs.aws.amazon.com/mwaa/latest/migrationguide/key-considerations.html
- AWS MWAA version-change (in-place upgrade path): https://docs.aws.amazon.com/mwaa/latest/userguide/upgrading-environment.html
- AWS MWAA supported versions and Python mapping: https://docs.aws.amazon.com/mwaa/latest/userguide/airflow-versions.html
- AWS MWAA migration blog: https://aws.amazon.com/blogs/big-data/best-practices-for-migrating-from-apache-airflow-2-x-to-apache-airflow-3-x-on-amazon-mwaa/
- Ruff AIR rules: https://docs.astral.sh/ruff/rules/#airflow-air

## Phase 0: Re-entry Check

At every invocation, check for `.mwaa-upgrade-plan-<env-name>.md` in the
workspace root (where `<env-name>` is the source environment name). Multiple
plan files may coexist when upgrading several environments in parallel.

- **If exists**: Load the plan, report source, target, chosen approach, and
  the position `(working_on, current_jump, jump_status)`. Resume the engine
  ([upgrade-engine.md](references/upgrade-engine.md)) on the recorded
  environment at the recorded version-jump index and status: `upgrading` ->
  re-poll or re-issue the pending `update-environment`; `validating` -> re-run
  the conditional validate/scan/fix loop for that version jump. For a
  newly-created environment, the starting-version validation window (version
  jump 1's `to`) corresponds to `current_jump: 1, jump_status: validating`;
  resume it there. If no position is set, resume from the next incomplete
  phase.
  **Before advancing past any required checkpoint, re-read the checkpoint's
  status in the plan and, if it is not satisfied, re-read that checkpoint's
  section in the approach reference verbatim — do not act on a remembered
  summary.** Specifically: version jumps complete does NOT imply the test-run
  checkpoint passed. If the `## Test-run results` section has any row not in
  SUCCESS or NEEDS-DECISION, that checkpoint is NOT-PASSED and blocks
  switchover (New-environment upgrade), the live upgrade (Rehearsed in-place
  upgrade — see `live_upgrade_allowed`), or completion (Direct in-place
  upgrade), regardless of version-jump status or `has_import_errors: false`.
  Likewise a batch whose `Monitored one full cycle?` cell is not yes has not
  fully switched over.

**Reload rule (every resume):** at session start and after any compaction,

From aws/agent-toolkit-for-aws, Apache-2.0.

More from aws/agent-toolkit-for-aws