databricks-data-discovery

Files (4)

• SKILL.md
• openai.yaml · agents/
• databricks.png · assets/
• databricks.svg · assets/

SKILL.md (first lines)

# Databricks Data Discovery

This skill **routes data work** — decide first:
- the request is about *the data* — finding it, asking questions of it, or
  generating SQL → delegate to **Genie One**:
  `databricks experimental genie ask -s <session-label> "..."` (see Routing below).
- writing files or anything else → use your own coding-agent tools.

Genie One just needs an authenticated CLI profile (the parent `databricks-core`
skill covers auth/profiles if you need it) — but route the data request to Genie
first; don't detour into manual catalog browsing.

## Routing — Genie vs your coding agent

**Route to Genie when the request is about the data:**
- **Finding or locating data** — "what tables are in wanderbricks?", "where does X
  live?", which catalog/schema holds something. Let Genie resolve a fuzzy or partial
  name across all your data instead of guessing or grepping catalogs yourself.
- **Questions about the data** — "how many…", "what's the average…", "which … the
  most…", trends, breakdowns.
- **Generating SQL from a question** — a first-cut query you can run, adapt, or drop
  into a file.
- **Understanding or profiling data** to inform a query, dashboard, or app.

**Route to your own coding agent (do NOT use Genie) for everything else:**
- **Writing or editing files** — a `.sql` file, notebook, dashboard, app, config.
  Genie finds the data and produces the SQL; *you* write the file.
- **Authoring jobs, pipelines, apps, or DABs** — use those product skills.
- **DDL or writes** — `CREATE` / `INSERT` / `UPDATE` / `DELETE`.
- **General programming, git, infra, and any non-data task.**

**Key principle:** data discovery, data questions, and query generation → Genie One.
Everything else → your coding agent.

## Why prefer Genie for data questions

Genie runs *inside the Databricks data plane* with governed, first-hand access to
the org's Unity Catalog metadata, metric views, and curated semantic context —
context you do **not** have when reverse-engineering schemas with ad-hoc SQL. For
data questions it is often higher-quality and more performant than doing the
discovery yourself, and it keeps improving as a managed Databricks capability.
Don't default to writing your own discovery SQL just because you can.

## How to ask Genie

Always pass a session label with `-s`, and prefer reusing the **same** one: a
follow-up can only continue a conversation if the first ask set the session label,
and reusing it lets later questions build on everything asked so far ("summarize all
of the above"). Use a fresh session label only to start a deliberately separate
session, or distinct session labels to run several in parallel.

```bash
# Always pass a session label, and reuse the SAME one so follow-ups build on each other
databricks experimental genie ask -s trips "How many bookings were there last week?"
databricks experimental genie ask -s trips "Break that down by destination"
databricks experimental genie ask -s trips "Summarize all of the above"

# --include-sql also prints the SQL Genie ran (use it to generate a query, too)
databricks experimental genie ask -s trips "Write SQL for the top 5 destinations by revenue" --include-sql

# --output json gives a parseable result
databricks experimental genie ask -s trips "Top 5 destinations by revenue" --output json
# → {"status":"completed","conversation_id":"…","text":"…","tool_calls":[{"name":"execute_sql","sql":"…","title":"…"}]}
```

Genie searches across all the data you can see, runs SQL, and streams a grounded
answer — rendered with the executed SQL and, where it helps, a terminal chart. It
auto-resolves a SQL warehouse (override with `--warehouse-id`); nothing to pick or
set up.

- **Streams live**: the answer, the agent's steps, and any SQL/results appear as
  they arrive. Answers usually take ~5–30s; a stalled stream (no data for ~10 min)
  fails with a clear message, and Ctrl-C or `kill` (SIGTERM) cancels cleanly.
- **Picking a session label**: any string works — a topic like `trips`, or `$$` for a
  per-shell session. Default to reusing one session label so follow-ups keep full
  context; use a fresh one only for a deliberately separate session. An expired
  session label just starts fresh on the next ask. No id to copy around.
- **Parallelism**: to run sessions at the same time, give each its own session label
  (`-s q1`, `-s q2`, …) — independent session labels don't interfere. Within a single
  session label keep calls sequential: send a follow-up after the previous turn
  returns, and never fire two asks at once on the *same* session label (they'd split
  into two conversations and only one mapping would survive).
- **Structured output**: `--output json` gives `{status, conversation_id, text,
  tool_calls[]}`, where `tool_calls` includes the SQL Genie executed; `--raw` dumps
  the raw event stream. Note `--output json` buffers and prints once at the end (no
  live streaming) — use it for parsing, the default text output for interactive use.
- **Generating a query**: ask Genie to "write SQL for …" and read the SQL from the
  response (it's in the answer text, and `--include-sql` also shows the query Genie
  ran to verify it — so the SQL is known-good). Genie resolves the schema and joins
  for you, so this beats hand-writing SQL against unfamiliar tables.
- **For exact/full rows**: Genie shows a preview inline; to pull the complete result
  set locally, copy its SQL (`--include-sql` or the JSON `tool_calls`) into the
  parent's `... aitools tools query "<SQL>"`.
- **A non-answer is a message, not an error**: if Genie refuses or "couldn't find
  relevant data," don't retry — use the manual fallback below.

> **Naming:** "Genie One" is the current name for this cross-data chat — formerly
> "Databricks One", then "OneChat" (the backend tool is still literally named
> `onechat`). All the same thing.

## If Genie One isn't available — manual fallback

Only fall back if Genie One is genuinely unavailable — first **verify** with
`databricks experimental genie ask --help`; don't assume the command is missing.
When Genie One isn't enabled, the CLI is too old to have `experimental genie ask`,
or Genie can't cover the question, do the discovery yourself with the parent skill's
commands — see **[Manual Data Exploration](../databricks-core/manual-data-exploration.md)**
(keyword search via `information_schema`, `discover-schema`, and `tools query`).
Running known SQL or profiling a known table that way is perfectly fine on its own.
Do **not** default to `databricks tables list` or raw UC REST for data-location
questions — invoke this skill and ask Genie first.

## Relationship to the Genie One MCP

Databricks also offers this capability as a managed **MCP** server — the *Genie One
MCP*. This skill delivers the **same functionality** through the Databricks CLI, with
no MCP server to configure or host. More broadly, the Databricks Agent Skills cover
the same ground as Databricks' managed MCP servers, so you don't need any MCP wired up
to use this. If you already run the Genie One MCP, use whichever you prefer — they hit

From ThomazRossito/ai-data-agents, MIT.

More from ThomazRossito/ai-data-agents