managed-airflow-dag-authoring

Files (2)

• SKILL.md
• attribution.md · references/

SKILL.md (first lines)

# GCP Managed Airflow DAG Authoring Guide

This skill guides you through authoring and validating Apache Airflow DAGs for
Managed Service for Apache Airflow (MSAA; formerly Cloud Composer) environments.

--------------------------------------------------------------------------------

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:managed-airflow-dag-authoring)"`
and append `agent-skills/1.0 (skill:managed-airflow-dag-authoring)` to `User-Agent` headers.

## Phase 1: Context Discovery

Before writing any DAG code, you MUST understand the constraints (e.g. version
of Airflow) and capabilities of your target environment if user is willing to
provide them.

### 1.1 Identify Target Environment & Access

Determine if you have direct access to the target Managed Airflow environment,
local development environment or if you are working offline (only changing local
files without validation).

*   **If environment access is available:** Use `gcloud` to inspect the
    environment (see Section 1.3).
*   **If offline:** Rely on user provided details.

### 1.2 Identify Development Environment

Determine if a local development environment is available.

*   Check if `composer-dev` CLI is installed.
*   Check if a local Python environment with `airflow` is available.

### 1.3 Inspect Target Environment (if available and requested)

Run the following commands to discover version constraints:

1.  **Get Airflow/Image Version:**

    ```bash
    gcloud composer environments describe {env_name} \
        --location {region} \
        --format="value(config.softwareConfig.imageVersion)"
    ```

2.  **Get Installed Packages (Versions):**

    ```bash
    gcloud composer environments describe {env_name} \
        --location {region} \
        --format="value(config.softwareConfig.pypiPackages)"
    ```

3.  **Get DAGs GCS Bucket:**

    ```bash
    gcloud composer environments describe {env_name} \
        --location {region} \
        --format="value(config.dagGcsPrefix)"
    ```

--------------------------------------------------------------------------------

## Phase 2: DAG Authoring Best Practices

### 2.1 General Airflow Best Practices

*   **Idempotency:** Every task SHOULD be idempotent. Running it multiple times
    with the same inputs (e.g., execution date) SHOULD produce the same result
    and not duplicate data.
*   **No Top-Level Code Execution:** Do NOT execute database queries, external
    API calls, or heavy computations at the top level of the DAG file (outside
    of tasks/operators). This code runs every few seconds during DAG parsing and
    will degrade performance.
*   **Explicit Catchup:** Always set `catchup=False` in the DAG definition
    unless historical backfilling is explicitly required.
*   **Use Airflow Variables/Connections:** Never hardcode credentials or
    environment-specific configs. Use `Variable.get()` (with
    `deserialize_json=True` if applicable) and `BaseHook.get_connection()`.
    Access variables via Jinja templates (e.g., `{{ var.value.my_var }}`) to
    avoid database calls during DAG parsing.

### 2.2 Airflow 2 vs Airflow 3 Compatibility

Use managed-airflow-migrations skill to navigate adjusting the code to
specific target Airflow version.

--------------------------------------------------------------------------------

## Phase 3: Validation Process

You MUST validate DAGs before concluding your task.

### 3.1 Local Validation (Offline/Pre-deployment)

#### 3.1.1 Static Analysis & Linting

Use `ruff` or `pylint` if available.

```bash
ruff check path/to/dag.py
```

*   If targeting Airflow 3, check with Airflow 3 rules if rulesets are
    available.

#### 3.1.2 Local Dev Environment (`composer-dev`)

If the user has `composer-dev` configured:

1.  Copy the DAG to the local directory with DAGs:

    ```bash

From google/skills, Apache-2.0.

More from google/skills