malloy-define

Files (1)

• SKILL.md

SKILL.md (first lines)

<!--
Copyright (c) Credible Data Inc.
SPDX-License-Identifier: MIT
-->

# Propose sources and definitions

This skill covers three consecutive activities when building or extending a Malloy semantic model:

> **Tool names** are written bare here - `get_context`, `execute_query`, `search_malloy_docs`. The exact prefixed name depends on the host surface; match each against the tools you actually have.

- **Propose scope**: which tables and which questions the model is for.
- **Propose sources**: the architectural blueprint (which sources, what grain).
- **Propose definitions**: the specific fields per base source (renames, dimensions, measures).

Both happen in conversation. Propose, let the user confirm or adjust, then carry the confirmed plan forward into the actual `.malloy` model. There is no separate plan-file store: keep the source plan and field proposals in the conversation, and write the model itself when the user has confirmed. See your modeling workflow for the broader picture.

Read the existing model first so you propose against what is really there. Use `get_context` with a plain-English description to inspect the current sources and fields and find the most relevant existing sources. Confirm the scope (below) before proposing the source plan.

## Propose the analytical scope

**When:** after you have read the package's sources and fields and looked at the data distributions. **Goal:** present what you found, recommend one analytical focus, and let the user pick. Query the data with `execute_query` for row counts and data-quality problems, and record the proposal and the user's decision in your modeling workflow's `modeling-notes.md`.

**Scope is "which questions", not just "which tables".** A model aimed at recommendation looks different from one aimed at catalog analysis over the same tables. A single A/B/C question about table inclusion is the source-plan question asked too early.

Present four things:

1. **A table summary** with rows, columns, role and key relationships. Roles: **Fact** (events you measure: orders, sessions), **Dimension** (entities you slice by: customers, products), **Bridge** (many-to-many links: order_items, tags), **Operational** (ETL, staging, audit; not analytical).
2. **2-3 analytical focuses**, each with the tables it covers (for example "Order Analysis: revenue, trends, product performance"), and **one recommendation with reasons**. Say what kind of model this is and what that rules out. A dataset of events supports trends over time. A dataset of entities with cumulative counters cannot do period-over-period analysis however well it is built ("release dates are 98% null, so this model cannot do calendar analysis"). The user should learn what the model will never answer before choosing.
3. **Tables to skip, with reasons**: operational tables, staging copies of a table you model, and pre-aggregated summaries (compute fresh in Malloy; a pre-aggregated source ignores your filters, see `skill:malloy-gotchas-modeling`).
4. **Options the user can answer in one word.** If your host has an `ask_user` tool, put the options in that card and nowhere else, with the evidence and your recommendation as prose. Otherwise give lettered one-line options (label, tables, key questions), mark the recommended one, and invite a mix ("A plus suppliers").

Be opinionated: recommend one option clearly. With 20+ tables, group them by domain and focus on the most relevant cluster. Show evidence (row counts, relationship density) and flag data-quality problems you saw ("`orders` has about 3 percent duplicate rows on `order_id`").

**After the user confirms,** restate the confirmed scope (connection and schema, tables in scope with row count and role, analytical focus in one line, deferred tables with reasons) and record it in `modeling-notes.md`. Then continue with the source plan.

## Propose a source plan

**Goal:** Propose the full source architecture for the tables in scope.

### Base sources

One base source per table in scope. For each, specify:

| Source | Table | Grain | Primary Key | Role |
|--------|-------|-------|-------------|------|
| orders | sales.orders | one row per order | order_id | Fact, transactions |
| customers | sales.customers | one row per customer | customer_id | Dimension, who |
| products | sales.products | one row per product | product_id | Dimension, what |

### Computed sources

Computed sources are created from queries, not physical tables. Propose them when:

1. **Grain mismatch**: the analytical scope requires a grain that no physical table provides (e.g., customer-level metrics from an order-grain table).
2. **Repeated aggregation patterns**: the same group-by plus aggregate pattern would be used in multiple places.
3. **Cross-entity aggregations**: inspecting the model and querying the data shows that an aggregate rolled up to a different entity would be reused.

For each computed source, explain:

| Source | Source Query | Grain | Rationale |
|--------|-------------|-------|-----------|
| user_order_facts | orders grouped by customer_id | one row per customer | Need customer-level order metrics (LTV, order count, recency) for customer health analysis |

### Dependencies

Show which sources depend on which:

```
customers (physical) ← user_order_facts (derived, sources from orders)
orders (physical) → user_order_facts (derived)
products (physical): independent
```

### Deferred sources

List sources considered but not included, with reasoning:

- **order_items**: bridge table, defer until line-item analysis is needed.
- **monthly_product_facts**: derived, defer until product trend analysis is requested.

### User interaction

The user will:
- **Confirm** the source plan as-is.
- **Add** missing sources (physical or derived).
- **Remove** unnecessary sources.
- **Validate** grain assignments.
- **Defer** sources to later iterations.

Once the source plan is confirmed, carry it forward into the definitions step below. Keep the confirmed map in the conversation rather than persisting it to a separate file.

## Propose definitions

**Goal:** Propose specific fields per base source with data evidence, working from the confirmed source plan.

### For each base source

Present a table of proposed fields.

**Renames (schema cleanup):**

| Raw Column | Proposed Name | Reason |
|-----------|---------------|--------|
| `Order Date` | order_date | Whitespace in column name |
| `Type` | order_type | Reserved word |
| `number` | item_number | Reserved word |

**Dimensions:**

| Field | Logic | Data Evidence | Priority |
|-------|-------|---------------|----------|
| order_status | status column | 5 distinct values: pending, processing, shipped, delivered, cancelled | must-have |
| order_month | submitted_at.month | Time trending | must-have |
| order_size | total buckets (data-driven) | Distribution: min $5, p25 $35, median $85, p75 $150, p95 $450, max $2,400. Proposed breaks at p25/p75: <$35, $35-$150, >$150 | nice-to-have |
| is_returned | returned_at is not null | 8% of orders have non-null returned_at | nice-to-have |

**Data-driven tiers:** For bucketed dimensions like `order_size`, always derive boundaries from the actual data distribution (percentiles, natural breaks, clustering). Query `min`, `max`, `p25`, `p50`, `p75`, `p95` and propose boundaries based on the distribution. Malloy has no `percentile` function; use the two-stage nearest-rank query in `skill:malloy-discover` § Example Queries. Show the evidence so the user can confirm or adjust. Never use arbitrary hardcoded thresholds unless the user explicitly provides them.

**Measures:**

From malloydata/publisher, MIT.

More from malloydata/publisher