querying-aws-sagemaker-catalog

Files (1)

• SKILL.md

SKILL.md (first lines)

# Query AWS SageMaker Catalog System Tables

## Overview

**Works best with** the [AWS MCP server](https://docs.aws.amazon.com/aws-mcp/) for sandboxed execution and audit logging. All commands below use the AWS CLI and work in any environment with configured AWS credentials.

Amazon SageMaker Unified Studio (whose catalog feature is referred to below as SageMaker Catalog) exports asset metadata as a daily-snapshot
Apache Iceberg table in the AWS-managed `aws-sagemaker-catalog` table bucket. This
enables SQL queries over your entire data catalog inventory — asset counts, governance
gaps, ownership audits, and historical comparisons — without building custom ETL.

Data is partitioned by `snapshot_time` and exported once daily (around midnight per
region). The table is read-only.

## Decision Tree

| User intent | Use this skill? | Alternative |
|---|---|---|
| SQL analytics on catalog state (counts, governance, trends) | **Yes** | — |
| Historical comparison ("what changed in catalog last week") | **Yes** — time travel via `snapshot_time` | — |
| Find assets without owners or descriptions | **Yes** | — |
| Find a specific table by name or concept | **No** | `finding-data-lake-assets` or Glue Discovery `search` |
| Browse/enumerate catalog interactively | **No** | `exploring-data-catalog` |
| Run a query *on* a table's data | **No** | `querying-data-lake` |
| Manage catalog metadata (add descriptions, tags) | **No** | Glue Discovery `put-form-type` / `associate-glossary-terms` |

## Common Tasks

### 1. Check If Configured

```bash
aws datazone get-data-export-configuration \
  --domain-identifier <DOMAIN_ID> \
  --region <REGION>
```

- If no domain exists: `aws datazone list-domains --region <REGION>`
- If export not enabled: guide user to enable.
- One domain per account per region.

Verify table bucket exists:

```bash
aws s3tables list-table-buckets --region <REGION> \
  --query "tableBuckets[?name=='aws-sagemaker-catalog']"
```

### 2. Enable

**With KMS encryption (recommended for production):**

```bash
aws datazone put-data-export-configuration \
  --domain-identifier <DOMAIN_ID> \
  --region <REGION> \
  --enable-export \
  --encryption-configuration kmsKeyArn=<KMS_KEY_ARN>,sseAlgorithm=aws:kms
```

> **Note**: Encryption cannot be changed after creation. Always specify KMS for sensitive catalog data.

Without encryption (for quick testing only):

```bash
aws datazone put-data-export-configuration \
  --domain-identifier <DOMAIN_ID> \
  --region <REGION> \
  --enable-export
```

First data available within 24 hours. See:
[Exporting asset metadata](https://docs.aws.amazon.com/sagemaker-unified-studio/latest/userguide/export-asset-metadata.html)

### 3. Verify Permissions for Querying

Requires:

- S3 Tables federated catalog registered in Glue (`s3tablescatalog`)
- Lake Formation SELECT + DESCRIBE grants on the table

Grant access:

```bash
aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=<ROLE_ARN> \
  --resource '{"Table": {"CatalogId": "<ACCOUNT>:s3tablescatalog/aws-sagemaker-catalog", "DatabaseName": "asset_metadata", "Name": "asset"}}' \
  --permissions DESCRIBE SELECT \
  --region <REGION>
```

### 4. Query

**Query syntax:**

```sql
"s3tablescatalog/aws-sagemaker-catalog"."asset_metadata"."asset"
```

**Constraints:**

- You MUST always filter by `snapshot_time` — without it, the query scans all historical snapshots and returns duplicates
- You MUST confirm workgroup and output location before executing
- Default to `DATE(snapshot_time) = CURRENT_DATE` for current state
- You SHOULD use the key columns documented in this skill to build queries. If you need the full schema, run `get-tables` once:

  ```
  aws glue get-tables --catalog-id "<ACCOUNT>:s3tablescatalog/aws-sagemaker-catalog" --database-name "asset_metadata" --region <REGION>
  ```

**Key columns:**

| Column | What it holds | Usage |
|--------|--------------|-------|
| `snapshot_time` | Partition key — daily snapshot timestamp | **Always filter on this** |
| `asset_id` | Unique catalog asset identifier | Primary key for lookups |
| `resource_type_enum` | GlueTable, RedshiftTable, S3Collection, etc. | Filter by asset type |
| `resource_id` | ARN or native identifier | Cross-reference with source systems |
| `asset_name` | Business-friendly name | Display, search |
| `resource_name` | Technical name (table name, prefix) | Filtering |

From aws/agent-toolkit-for-aws, Apache-2.0.

More from aws/agent-toolkit-for-aws