discovering-gcp-data-assets

Files (1)

• SKILL.md

SKILL.md (first lines)

# Instructions

## Step 1: Prioritize Assets from the Conversation

If the asset was created or mentioned earlier in the same conversation, then
proceed with that asset instead of searching. Skip steps 2, 3, and 4.

## Step 2: Handle Public Datasets or Proceed to Search

Dataplex Lookup Context provides the richest metadata for data assets. You MUST
prioritize using it for all Google Cloud assets, even if you already know their
IDs.

-   **Public Datasets (Direct Inspection)**: If the requested asset belongs to
    the `bigquery-public-data` project, Dataplex Lookup Context will fail. You
    MUST skip Steps 3 and 4 and inspect the table directly using the `bq` CLI or
    BigQuery MCP tools instead.
-   **All Other Assets (Proceed to Step 3)**: For all other BigQuery, Cloud
    Storage, Spanner, BigLake Iceberg or general GCP data assets (whether their
    IDs are known or missing), you MUST proceed to **Step 3** to search the
    Dataplex catalog and obtain their full Entry Name.

## Step 3: Execute Discovery Search

You MUST use the Dataplex search command to discover assets and retrieve their
full `projects/...` entry names. This step is required even if you already know
the asset's short ID (e.g., `my_dataset.my_table`), because Step 4 strictly
requires the full entry name.

> [!IMPORTANT]
>
> The `--project` parameter MUST ALWAYS be provided. This project_id is used to
> attribute the search only and does NOT restrict the search scope. The project
> must have the dataplex API enabled and user must have the
> `dataplex.entries.get` permissions.

### A. Semantic Search (Natural Language Intent)

Use this when the user describes the **meaning** or **intent** of the data
(e.g., "Find Q4 product sales data").

Use the `search_entries` MCP tool

OR

```bash
gcloud dataplex entries search "<NATURAL_LANGUAGE_QUERY>" \
  --project="<PROJECT_ID>" \
  --semantic-search \
  --limit=50
```

### B. Keyword Search (Technical Strings)

Use this for exact keyword matches or technical strings (e.g., `name:order_v2`).

#### Search Query Rules (MANDATORY)

-   **Mode-Specific Syntax**:
    -   **Semantic Search**: Logical operators (`AND`, `OR`) MUST be
        **UPPERCASE**. Use plural `labels.` for label filters (e.g.,
        `labels.env=prod`).
    -   **Keyword Search**: Operators are case-insensitive. Use singular
        `label.` for label filters (e.g., `label.env=prod`).
-   **Abbreviated Logic**: Use `|` for OR and `,` for AND within parentheses to
    shorten queries (e.g., `projectid:(prod|staging)` or `column:(id,name)`).
-   **Exact vs. Token Match**:
    -   Use `:` for token/substring matches (e.g., `name:sales`).
    -   Use `=` for exact matches. REQUIRED for `system`, `type`, and
        `location`.
-   **Singular Keywords**: When performing keyword search, ALWAYS convert
    plurals to singular (e.g., "product" NOT "products"). Semantic search
    handles singular/plural variations and synonyms automatically.
-   **Scope Restriction**: You SHOULD restrict the search scope using a `parent`
    filter if the project or dataset is known (e.g.,
    `parent:projects/<PROJECT_ID>`).

#### Dataplex Search Syntax Reference

-   **`name:x`**: Substring/token match on resource ID.
-   **`displayname:x`**: Substring/token match on display name.
-   **`projectid:x`**: Substring/token match on GCP project ID.
-   **`parent:x`**: Substring match on hierarchical path (e.g.,
    `projects/my-proj`).
-   **`location=x`**: Exact match on location (e.g., `us-central1`, `us`).
-   **`column:x`**: Substring/token match on column names in the schema.
-   **`system=x`**: Exact match on source system. Common values: `bigquery`,
    `storage`, `biglake`, `cloud_sql`, `cloud_spanner`, `cloud_bigtable`,
    `pubsub`.
-   **`type=x`**: Exact match on entry type (e.g., `bigquery-table`,
    `storage-bucket`, `storage-folder`).
-   **`labels.key=value`**: (Semantic Mode ONLY) Exact match on a label.
-   **`label.key=value`**: (Keyword Mode ONLY) Exact match on a label.
-   **`createtime[>|<|=]x`**: Match assets created after/before date
    `YYYY-MM-DD`.
-   **`fully_qualified_name=x`**: Exact match on the FQN (e.g.,
    `bigquery:project.dataset.table`).

> [!TIP]
>
> Dataplex search results rely on metadata being ingested into the Universal
> Catalog (often via **Discovery Scans**). If an asset is missing from search,
> it may not be indexed. - **Fallback 1**: Try searching by the
> `fully_qualified_name` qualifier. - **Fallback 2**: Use native tools (e.g.,
> `bq show`, `gcloud storage`) or specific skills for that asset type if you
> already know the ID.

```bash
gcloud dataplex entries search "<KEYWORD_SEARCH_QUERY>" \
  --project="<PROJECT_ID>" \
  --limit=50
```

> [!IMPORTANT]
>
> Handling Search Results and Avoiding Loops:
>
> 1.  **No Results:** If the search returns no entries:
>     *   **Variation Rule:** You may try AT MOST 3 variations of the search

From jerrylin96/dotgemini, Apache-2.0.

More from jerrylin96/dotgemini