managed-airflow-migrations

Files (5)

• SKILL.md
• airflow-3.md · references/
• attribution.md · references/
• environment-inspection.md · references/
• local-development-environment.md · references/

SKILL.md (first lines)

# Managed Service for Apache Airflow (formerly Cloud Composer) Migration Guide

This skill guides you through the process of adjusting Airflow DAGs from an
existing Managed Service for Apache Airflow (formerly Cloud Composer)
environment (or available locally) to make them compatible with **Airflow
2.11.1** (MSAA Gen 2 or 3) or **Airflow 3** (MSAA Gen 3).

--------------------------------------------------------------------------------

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:managed-airflow-migrations)"`
and append `agent-skills/1.0 (skill:managed-airflow-migrations)` to `User-Agent` headers.

## Phase 1: Discovery & Download

Before making any changes, download the existing DAG files if explicitly
requested. Inspect the source environment to confirm source version only if
explicitly requested. For detailed instructions about environment inspection and
downloading files check
[references/environment-inspection.md](references/environment-inspection.md).

--------------------------------------------------------------------------------

## Phase 2: Target Version & Dependency Mapping

### 2.1 Airflow 2.11.1+ Dependency Mapping

If migrating to Airflow 2.11.1 (MSAA Gen 2) or Airflow 3, use the list below to
trace the version progression of key dependencies. The list covers changes
needed to get to Airflow 2.11.1. Take them into account when migrating from
Airflow 2 (earlier than 2.11.1) to Airflow 3.

### Composer 2.10.0 (Airflow 2.10.2)

-   **Google Provider**: `10.26.0`
-   **SSH Provider**: `3.14.0`
-   **HTTP Provider**: `4.13.3`
-   **Breaking Changes**: *Baseline for oldest fully documented source.*

### Composer 2.15.3 (Airflow 2.10.5)

-   **Google Provider**: `18.0.0`
-   **SSH Provider**: `4.1.4`
-   **HTTP Provider**: `5.3.4`
-   **Breaking Changes**:
    -   **SSH Provider 4.0.0:** Hook `timeout` removed; `get_conn()` context
        manager.
    -   **HTTP Provider 5.0.0:** `SimpleHttpOperator` -> `HttpOperator`.
    -   **Google Provider 11.0.0:** `BigQueryExecuteQueryOperator` removed.
    -   **Google Provider 12.0.0:** Legacy Data Pipeline operators removed.
    -   **Google Provider 13.0.0:** `AutoMLBatchPredictOperator` removed.
    -   **Google Provider 17.0.0:** `BigQueryCreateEmptyTableOperator` and
        `BigQueryCreateExternalTableOperator` removed; Life Sciences operators
        removed.
    -   **Google Provider 18.0.0:** Legacy DV360 operators removed.

### Composer 2.16.1 (Airflow 2.10.5)

-   **Google Provider**: `19.0.0`
-   **SSH Provider**: `4.1.6`
-   **HTTP Provider**: `5.5.0`
-   **Breaking Changes**: **Google Provider 19.0.0:** AutoML operators removed
    (use Vertex AI).

### Composer 2.17.0 (Target Airflow 2.11.1)

-   **Google Provider**: **`20.0.0`**
-   **SSH Provider**: **`5.0.0`**
-   **HTTP Provider**: **`6.0.2`**
-   **Breaking Changes**:
    -   **SSH Provider 5.0.0:** `sshtunnel` removed (native tunneling).
    -   **HTTP Provider 6.0.0:** JSON serialization.
    -   **Google Provider 20.0.0:** ADLS Gen2 migration.

### 2.2 Airflow 3 Migration

If migrating to Airflow 3 (MSAA Gen 3), note that this is a major version
upgrade with significant changes, including:

*   Decoupled Task SDK (imports change from `airflow` to `airflow.sdk`).
*   Removal of direct metadata DB access.
*   Renaming of `Dataset` to `Asset`.
*   Removal of SubDAGs and SLAs.
*   Changes to context variables availability.

Take into account all applicable changes within Airflow 2 (e.g. when migrating
from Airflow 2.10.2, apply changes needed to move to Airflow 2.11.1 and Airflow
3 migration changes on top of that).

--------------------------------------------------------------------------------

## Phase 3: Analysis & Remediation (Scanning Downloaded Files)

Run the scan commands from the root of your local workspace
(`./migration_workspace` unless indicated otherwise).

--------------------------------------------------------------------------------

### 3.1 Airflow 2.11.1 Core & Dependency checks

Use these scans if migrating to Airflow 2.11.1+ (intermediate step when
migrating to Airflow 3).

#### 3.1.1 Dataset Scheduling (Airflow 2.11.0)

*   **Change:** DAGs scheduled on datasets only trigger if events occur while
    the DAG is unpaused.
*   **Scan Command:** `grep -rn "Dataset(" ./dags`
*   **Remediation:** You MUST document that these DAGs must remain unpaused to
    catch events, or plan manual triggers for catch-up.

#### 3.1.2 HTML in Descriptions (Airflow 2.11.0)

*   **Change:** Raw HTML in DAG docs / params is escaped by default.

From google/skills, Apache-2.0.

More from google/skills