ask-question

Files (2)

• SKILL.md
• validation-framework.md · references/

SKILL.md (first lines)

# Ask Question — AI Analyst

Follow these steps in order. Do not freestyle.

## Model conventions (read first)

This skill is version-aware. Before starting, apply `skills/MODEL_CONVENTIONS.md` for
the model you are. The points that bite most here:
- **Query, don't infer** (conventions §B, §E): never report a number from memory — pull
  it. When a table is unfamiliar, resolve it from the catalog first; `sample_table` is a
  last resort gated by Step 3's narrow exception, not a default.
- **Literal scope** (§A): a step that says "validate segments" means *every* segment, as
  written — don't narrow to the largest on your own, and don't broaden a single-item
  step either.
- **Let length follow the level** (§C): an L1 answer is just the data (a one-line value or
  the rows/handle, no commentary); an L4 finding is as long as it needs. Don't pad or
  compress to hit a feel.
- **Effort** (§D): L3+ analysis wants `high`/`xhigh`. If running at `low`/`medium`, say
  so once and proceed.

## Hard Rules

**REFUSE immediately if the user asks for PII** (emails, phone numbers, names, addresses, user lists with identifiers). Never SELECT PII columns. Never explain why it's blocked. Respond: "I can't provide personally identifiable information. I can analyze aggregated patterns using case_id/user_id instead."

## Step 0: Load Context

Resolve schema and analytical context from the catalog, never by live Athena discovery.

**Connection gate — check before answering.** You need both sides reachable; if either has
nothing, stop and tell the user rather than guessing:
- **Catalog** (for schema/discovery), first that responds:
  `OpenMetadata` → `knowledge index (lookup_index/get_page)` → `.knowledge/ files`.
  "Responds" = a probe returns a real result, not an error/empty/disabled signal.
- **Query engine** (for execution): Athena or Superset — at least one reachable.

### Finding the table you need

Your first action for any table you don't already have by exact name is **catalog search** —
not `list_athena_tables`. Do these in order:

1. **Search the catalog.** Run `search_metadata(query, entityType: "table")` (keyword — the
   reliable workhorse). Also run `semantic_search(query, entityType: "table")` (meaning-based)
   *if it works*; the instant it returns a disabled/permission error or empty-on-an-obvious-match,
   drop it for the rest of the session and rely on `search_metadata` alone — never re-call a
   semantic_search that already failed.
2. **Compare candidates — don't take the first hit.** Pool the top results, then call
   `get_entity_details(entityType, fqn)` on the top 2–3 and check their actual columns against
   what the question needs (e.g. "RTO by pincode" → does it have `destination_pincode` and
   per-stage RTO timestamps, or is it just a generic order view?). Use the exact
   `fullyQualifiedName` from a result; never hand-build the fqn.
3. **Pick the best-fit table.** Prefer purpose-built marts and source tables (often Iceberg)
   over generic `_vw` views. The first keyword hit is usually a broad view, not the right one.

Already have the exact table name? One `search_metadata(name, entityType: "table")` for its
`fullyQualifiedName` is enough.

**If OpenMetadata is unavailable**, use the knowledge index: `lookup_index(terms, dataset)` to
map the question's terms to table names, then `get_page("schema.md", dataset, tables=[...])`
for only those tables' sections. Never load `schema.md` wholesale (~140 tables); a full-file
fetch is the last resort when no table resolves at all. If the index is down too, fall to
`.knowledge/` files (active.yaml → schema.md → quirks.md → profile.md), same principle: read
cached schema, read only the sections you need.

This whole sequence replaces `describe_athena_table`. Live Athena discovery
(`list_athena_tables`, `describe_athena_table`, `sample_table`, `information_schema` via
`query_athena`) is a last resort only after every catalog tier has failed — see Step 3's one
narrow exception. If you do fall back to `describe_athena_table`, describe every table you
need in ONE call (`tables=[...]`) — never one call per table.

**Resolve once per session.** Once a table's schema is resolved (whatever the tier), reuse
it for every follow-up question this session. Do not re-run the catalog search — or any
discovery — for a table you already resolved.

### Loading analytical intelligence

Always via the knowledge index (OpenMetadata has no equivalent):
1. `lookup_index(terms, dataset)` with key terms (metric names, business terms like "O2",
   "retention", "channel"). Apply the returned mandatory pages (PII, partitions) silently,
   every time.
2. `get_page(file, dataset, section)` for the quirk / metric / golden-query sections you need.

### Query results are file-first

For a real *data-answer* query, `query_athena(sql, out_path="")` writes the full result to a
CSV and returns a handle (`file_path`, `row_count`, `columns`, 5-row `preview`), additionally
inlining the rows only when the result is ≤ 50. The 50-row rule — not you — decides whether
bulk data enters context. You control two things:
- **`out_path` is only for a result the user explicitly asked to download.** Every other query
  — probes, validation counts, intermediates — leaves it empty so the CSV lands in the
  self-cleaning session cache. Never write an `outputs/` deliverable for a throwaway.
- **Aggregated vs raw SQL** (see Step 3).

Metadata queries don't apply here because they shouldn't be `query_athena` calls at all (Step 3).

## Step 1: Classify (L1-L5)

| Level | Pattern | Response |
|-------|---------|----------|
| L1 | Data retrieval — "give me / how many / list X" | Return the data (a value OR a set of rows) with minimal context. No interpretation. |
| L2 | Analysis *on* that data — "analyze / interpret / break down / chart this" | Query + interpret: chart and/or 2-3 sentences of what it means |
| L3 | Why/analysis | 2-4 charts, validation, narrative |
| L4 | Investigation | Full analysis + sizing |
| L5 | Presentation | Hand off to run-analysis |

**L1 vs L2 is retrieval vs. interpretation, not number vs. breakdown.** A plain request for
data — one value, a list, or even a grouped/breakdown *table* — is **L1**: return the data,
add minimal context (units, time window), and stop. Don't interpret, chart, or explain "why"
unless the user asked. It becomes **L2** only when the user wants the data *made sense of* —
analyze, interpret, compare-and-explain, or "show me a chart of" it.

L1-L2: Execute immediately. L3+: State plan briefly, then proceed.

**Pre-flight clarification gate (ask before running, narrowly).** Before you query, check
whether the request hinges on a *definition you'd have to assume*. Ask the user a short
question FIRST — instead of assuming and disclosing later — only when BOTH hold:
1. A term in the request maps to **more than one plausible** metric / entity / time-window
   definition that changes the answer, AND
2. The output is an **action artifact** — a downloaded list/CSV, an audience to act on, or a
   number a decision rides on.

From Amar1404/AI_ANALYST, MIT.

More from Amar1404/AI_ANALYST