finding-data-lake-assets

Files (2)

• SKILL.md
• search-strategy.md · references/

SKILL.md (first lines)

# Find Data Lake Assets

## Overview

Resolves data lake asset references to concrete catalog entries. Acts as a
resolver for other skills and direct user requests. Covers Glue,
S3, S3 Tables, and Redshift. Optimized for low token usage — return the
answer fast and get out of the way.

**Constraints for parameter acquisition:**

- You MUST accept a single argument: table name, keyword, column name, or S3 path
- You MUST accept the argument as direct input or a pointer to a file containing the spec
- You MUST ask for the target AWS region if not already set
- You MUST confirm ambiguous input before searching (e.g., "Did you mean table X or bucket Y?")
- You MUST respect the user's decision to abort at any step

## Common Tasks

You MUST execute commands using AWS MCP server tools when connected — they
provide validation, sandboxed execution, and audit logging. Fall back to
AWS CLI only if MCP is unavailable. You MUST explain each step before
executing.

### 1. Verify Dependencies

Check for required tools and AWS access before searching.

**Constraints:**

- You MUST verify AWS MCP server tools (`aws___call_aws`) are available; fall back to AWS CLI if not
- You MUST confirm credentials with `aws sts get-caller-identity`
- You MUST inform the user about any missing tools and ask whether to proceed

### 2. Consult Catalog Context (experimental — suggested first lookup)

The customer may publish **context skill assets** in the Glue Data Catalog that map
their business language to the real tables — canonical names and aliases, join keys,
metrics, usage notes, descriptions — that the raw schema does not carry. When present,
this catalog is often enough to answer the request on its own.

These are the **Glue Discovery** operations (`SearchAssets` / `GetAsset` /
`ListIterableForms` / `BatchGetIterableForms`) — a distinct metadata-search surface,
NOT the legacy `glue search-tables` used in Step 5. They are **experimental** — not
available in every CLI build. Gate the lookup on two checks first:

1. **Availability.** Confirm the `GetAsset` operation exists in the caller's Glue
   CLI model (redirect output so the CLI pager cannot block a non-interactive agent):

   ```
   aws glue get-asset help > /dev/null 2>&1
   # exit 0 = available. exit 2 (with "Invalid choice" in stderr) = not in this CLI (skip).
   # any other non-zero (network/credential error) = inconclusive; treat as unavailable.
   ```

   If it is not available, skip this step and go to the normal search workflow (Steps 3-7).
2. **User opt-in.** If available, ask the user: "I can check the Glue Data Catalog
   for customer-authored context using an experimental SearchAssets/GetAsset API.
   Use it? (yes/no)". Proceed only on an explicit yes; otherwise skip to Steps 3-7.

**How this model differs:** Discovery indexes **assets** (not databases/tables). Every
asset has an `Id` that is an **ARN**, and every lookup after `SearchAssets` keys off that ARN
via the identifier — there is no `--database-name`/`--table-name`. CLI flags are kebab-case
(`--search-text`, `--max-results`, `--filter-clause`); top-level response fields are PascalCase
(`Id`, `AssetName`, `Forms`). NOTE: a `*.Content` value is itself a JSON STRING with its own
camelCase schema (e.g. `dataLocation`, `dataFormat`, `isPartitionKey`) — parse it as embedded JSON,
do not expect PascalCase inside. The operations you need:

| Operation | Input → Output |
|---|---|
| `search-assets` | `--search-text` (+ optional `--filter-clause`) → `Items[]` of `{Id, AssetName, Type, Namespace, AssetTypeId, UpdatedAt}` (NOTE: search items do NOT include a description — call `get-asset` for `Description`/`Forms`) |
| `get-asset` | `--identifier <Id, an ARN>` → one asset's `{Description, Forms, IterableForms}`. `Forms."amazon::Table".Content` is JSON `{dataLocation, dataFormat, type}`; advertises column availability via `IterableForms: {"columns": {...}}` |
| `list-iterable-forms` | `--asset-identifier <table ARN> --iterable-form-name columns` → that table's columns `Items[]` of `{ItemId, ItemName, Description}` (ItemId = `<table-ARN>#<columnName>`) |
| `batch-get-iterable-forms` | `--asset-identifier <table ARN> --iterable-form-name columns --item-identifiers <id1> <id2> ...` (space-separated) → `Items[]` of `{ItemName, Forms}` where `Forms.Column.Content` is JSON `{"type": "...", "isPartitionKey": ...}` |

```
aws glue search-assets --search-text '<user request terms>' --max-results 5
# Id is a full ARN, e.g. arn:aws:glue:us-west-2:123456789012:table/<db>/<table>
aws glue get-asset --identifier "arn:aws:glue:<region>:<account>:table/<db>/<table>"
```

`search-assets` returns only identity fields (no description), so to judge relevance you MUST
`get-asset` the top candidates (up to ~5) and read their `Description` / `Forms` — do NOT pick by
rank alone. Only pass ARNs whose `Type` is a Glue table (`amazon.glue::GlueTable`) to `list-iterable-forms`.

**Narrow with `--filter-clause`** when the request names a database or asset type
(filterable: `type`, `amazon.glue::GlueTable.databaseName`, `dataFormat`, `createdAt`):

```
aws glue search-assets --search-text 'sales' --max-results 5 \
  --filter-clause '{"AttributeFilter": {"Attribute": "amazon.glue::GlueTable.databaseName", "Operator": "equals", "Value": {"StringValue": "<database-name, e.g. sales>"}}}'
```

**Column name is search-only** — pass it as `--search-text`, not a filter. To confirm a
column on a candidate, list its columns with `list-iterable-forms` (each item is
`{ItemId, ItemName, Description}`; column item IDs have the form `<table-ARN>#<columnName>`).
For a column's `type` and `isPartitionKey`, call `batch-get-iterable-forms` and read
`Forms.Column.Content` (JSON, e.g. `{"type": "bigint", "isPartitionKey": false}`):

```
aws glue list-iterable-forms --asset-identifier "arn:aws:glue:<region>:<account>:table/<db>/<table>" --iterable-form-name columns
aws glue batch-get-iterable-forms --asset-identifier "arn:aws:glue:<region>:<account>:table/<db>/<table>" --iterable-form-name columns --item-identifiers "arn:aws:glue:<region>:<account>:table/<db>/<table>#<columnName1>" "arn:aws:glue:<region>:<account>:table/<db>/<table>#<columnName2>"
```

**Answer from the catalog if it is sufficient (short-circuit):**

Short-circuit eligibility uses **objective criteria only** (no intent judgment, so it
cannot conflict with the Step 3 classification):

- Short-circuit ONLY when **both**: (a) `SearchAssets` returned **exactly one asset whose
  `AssetName` is an exact, case-insensitive match** for a specific table name in the
  request, AND (b) that asset provides ALL of {database, table, format, location} —
  **return that answer now and STOP. Skip Steps 3-7.** Note that the answer came from
  customer-authored catalog context.
- In **all other cases, fall through** to the remaining steps (Steps 3-7), seeding the
  search with any canonical names the catalog provided. This explicitly includes:
  multi-keyword / exploratory requests (no exact table name); `SearchAssets` returns no match
  or multiple candidates; the asset only partially answers the request; a required
  column/schema detail could not be confirmed; or the call returns AccessDenied / is

From aws/agent-toolkit-for-aws, Apache-2.0.

More from aws/agent-toolkit-for-aws