bigquery-data-transfer-service

Files (2)

• SKILL.md
• bigquery_dts.py · scripts/

SKILL.md (first lines)

# BigQuery Data Transfer Service (DTS)

## Mandatory Guidelines

> [!IMPORTANT]
>
> All new BigQuery Data Transfer Service (DTS) configurations **MUST** be
> provisioned through the **gcp pipeline resource provisioning** framework,
> which includes generating a `deployment.yaml`.
>
> -   **Do NOT** use imperative CLI commands (e.g., `bq mk` or `gcloud`) to
>     create or update configurations.
> -   CLI commands are permitted **only** for discovery (listing/showing) and
>     triggering manual runs.

This guide enables the discovery of existing ingestion resources and provides
metadata related to ingestion when needed.

## Workflow

### Step 0: Discover Environment Parameters

Before generating configurations, discover the actual values for the target
project and region.

> [!TIP]
>
> If `deployment.yaml` already exists in the repository root, prioritize
> extracting `project` and `region` from the target environment configuration
> (e.g., `dev`).

1.  **Project**: `gcloud config get project`
2.  **Region**: `gcloud config get compute/region`

> [!TIP]
>
> Use these commands to replace placeholders like `<PROJECT_ID>` with actual
> values. Always remove associated comments that start with TODO once replaced.

### Step 1: Check for Existing Transfers

Before assuming a new transfer is needed, check for existing ones in the target
region.

1.  **List Transfers**:

    ```bash
    bq ls --transfer_config \
      --transfer_location=<REGION> \
      --project_id=<PROJECT_ID>
    ```

2.  **Analyze Existing Transfers**:

    -   **Single Transfer Found**:

        -   Check if the transfer has at least one successful run: `bq ls
            --transfer_run --transfer_config=<RESOURCE_NAME>`
        -   If found: Use existing transfer config.
        -   If not found: Confirm with user if it's ok to trigger the transfer
            run.

    -   **Multiple Transfers Found**:

        -   Attempt to guess the correct one based on context.
        -   Ask user to confirm.

    -   **Disabled Transfers Found**:

        -   Ask user if they want to enable it or create a new one.
        -   To Enable: Instruct the user to update the transfer configuration
            within their `deployment.yaml` file by setting the `disabled` field
            to `false` for the specific transfer resource.

    -   **No Transfers Found**: Proceed to create new if needed.

### Step 2: Discover & Validate Parameters (New Transfers)

If creating a new transfer, discover the required parameters using the REST API
and validate them with the user.

> [!TIP]
>
> If `<DATA_SOURCE_ID>` is unknown, run the discovery script without
> `<DATA_SOURCE_ID>` argument to list available source IDs (e.g.,
> `google_cloud_storage`). It uses the derived project and location from Step 0.
>
> ```bash
> python3 scripts/bigquery_dts.py --project_id=<PROJECT_ID>
> ```

1.  **Run Discovery Script**: Use the `bigquery_dts.py` script to inspect Data
    Source parameters via the REST API.

    ```bash
    # Passes the derived project and region to the script.
    python3 scripts/bigquery_dts.py --project_id=<PROJECT_ID> <DATA_SOURCE_ID> <REGION>
    ```

    > [!IMPORTANT]
    >
    > Run this command every time a new transfer is being planned.

2.  > [!CAUTION]
    >
    > **Mandatory User Questionnaire (CRITICAL)**:

    -   **Explicitly identify ALL specific parameters** returned by the
        discovery script. **You MUST NOT generalize or vaguely summarize them.**
    -   **OAuth Authorization (Google Data Sources)**: For Google ecosystem data
        sources (Google Ads, Youtube, etc.), if the user is not using a service
        account to configure the DTS transfer config (meaning the user is using
        End User Credentials or EUC to configure the transfer config), then
        generate an OAuth URI. Ask the user to visit this URL to authorize. Once
        the user provides the versionInfo code, use the code as
        `definition.versionInfo` in `deployment.yaml` and then you can proceed.
    -   If any parameters are related to authentication, explicitly ask the user
        to provide the Secret Manager Resource ID (e.g.,
        projects/my-project/secrets/my-secret) for these parameters

From jerrylin96/dotgemini, Apache-2.0.

More from jerrylin96/dotgemini