malloy-document

Files (1)

• SKILL.md

SKILL.md (first lines)

<!--
Copyright (c) Credible Data Inc.
SPDX-License-Identifier: MIT
-->

# Documenting a Malloy Model

Add `#(doc)` tags to describe sources and fields in plain language so they are easy to find and understand:

| Tag | Purpose | Goes on |
|-----|---------|---------|
| `#(doc)` | Plain-language description for natural-language search | source, dimension, measure, view, join |
| `#(filter)` | Deprecated. Never add one; declare a `given:` instead (see `malloy-model`) | source |

`#(doc)` is a standard Malloy annotation. It documents a field or source with a human-readable description that downstream tools can surface and search against.

## #(doc) Tag

Add before any source, dimension, measure, view, or join. When multiple fields share a keyword, use it once as a block header. Tags and field names are indented under the keyword; tags go on the line(s) directly above the field they annotate.

**Tag ordering** (when a field has multiple tags): `#(doc)` → render tags (`# currency`, `# label`, etc.) → field name. Separate each field group with a blank line:

```malloy
#(doc) Customer who placed the order
join_one: users with user_id

dimension:
  #(doc) Date the order was placed (UTC)
  order_date is created_at::date

measure:
  #(doc) Total revenue from all orders in USD
  # currency
  revenue is sum(total)
```

### Writing Doc Strings for Retrieval

Doc strings power natural-language search: users type plain-English questions and the system matches against your `#(doc)` strings. Write descriptions that match how analysts would search:

- **Include business meaning**, not code mechanics: what it represents, not how it's implemented
- **Include units** (USD, count, percentage): a unit is part of what a number means. For counts, name the unit being counted and say whether it counts distinct entities or events: "total students enrolled" on a subject×term-grain measure counts enrolments, not students, and a student taking four subjects counts four times. If the model cannot answer the distinct-entity version, say so in the doc.
- **List a categorical field's values only while the list stays short** (roughly ten or fewer). A handful of values makes a description concrete; past that, say what the field captures instead, because the dump crowds out the meaning and goes stale the moment someone adds a value. Treat ten as a rule of thumb, not a hard cap.
- **Avoid Malloy jargon**: never use "filterable", "groupable", "dimension", "measure", "aggregation"

**Good examples:**
- `#(doc) Total revenue from completed orders in USD` matches "what was our revenue?"
- `#(doc) Customer signup date (UTC)` matches "when did the customer join?"
- `#(doc) Order status: pending, processing, shipped, delivered, cancelled` matches "what are the order statuses?"

**Bad examples:**
- `#(doc) Filterable dimension for order status`: no analyst searches for "filterable"
- `#(doc) Groupable by region`: "groupable" is a system concept
- `#(doc) Aggregation of total sales`: "aggregation" doesn't match natural queries

### Mark conventions as conventions

A `#(doc)` must let a reader tell a **measured fact** from a **choice someone made**. Any dimension or measure encoding a threshold, bucket boundary, or business definition that the user did not explicitly confirm must say so in its own doc string:

```malloy
// WRONG - a chosen cutoff stated as fact
#(doc) Popularity band: Hit (70+), Popular (40-69), Moderate (15-39), Obscure (<15)

// RIGHT - the choice is visible and auditable
#(doc) Popularity band: Hit (70+), Popular (40-69), Moderate (15-39), Obscure (<15).
#(doc) 70 follows the source dataset's own high-popularity cutoff; 40 and 15 are
#(doc) working boundaries for this model, not settled by the data.
```

These are governed models: a threshold nobody confirmed is an assumption, and an unlabeled assumption reads as a fact to everyone downstream, including the agents that answer questions from these docs. The hedge in the `#(doc)` is the artifact-time record; the "Flag Ambiguous Descriptions" table below is the conversation-time surface for getting them confirmed, and `modeling-notes.md`'s "Open decisions" section (see `skill:malloy-modeling`) is where they wait for a subject-matter expert.

Do not hedge measured facts: `avg_energy is avg(energy)` needs no caveat. Hedge only where a domain expert could reasonably choose differently.

## #(filter): deprecated, see `malloy-model`

`#(filter)` is deprecated. Never add a `#(filter)` annotation: every use, including `required`, `implicit`, and date/number ranges, has a `given:` form. See `skill:malloy-model` § Parameterizing sources with `given:`.

`#(filter)` is also a `#(...)`-shaped annotation, but unlike `#(doc)` it's a **runtime/modeling construct**: it shapes governance, query latency, and correctness, not discoverability. How to read and migrate an existing one lives in `malloy-model` § Legacy: reading an existing `#(filter)` model.

One rule worth knowing here: a filter the model reads lives on the source, never on the consumer. Ad-hoc reports and notebooks that import a source inherit its givens automatically, and import them by name rather than re-declaring them. A notebook or dashboard may declare a `given:` of its own for a control only its own tiles read, and nothing else.

## `internal:` and `private:`: column-level access in a source

`#(doc)` describes what's exposed. Two access modifiers control what's exposed in the first place, and both live **inside** a source's `include {}` block. They are about the source's public API and data sensitivity, not about documentation, so reach for them when curating which columns callers can pick.

| Mechanism | Layer | Why you reach for it |
|---|---|---|
| `internal:` | Inside a source (one column in `include {}`) | The column **isn't part of your model's public API**. Common reasons: data is messy (empty/garbage, raw JSON, duplicates), or a documented derived dimension already supersedes it, or the raw column exists only to be joined on / referenced internally and shouldn't appear as a dimension callers can pick. The data may be perfectly fine, it's just not what you want exposed. |
| `private:` | Inside a source (one column in `include {}`) | The **data is sensitive**: SSN, raw credit card, password. Governance / security concern; a harder block than `internal:`. |

In one sentence: **`internal:` and `private:` shape what's inside a source's public API; `#(doc)` describes the fields you do expose.**

### Example

A base source pulled from a messy raw table often uses `internal:` to drop raw fields from the public API, while documenting the curated columns with `#(doc)`.

```malloy
// orders_base.malloy
#(doc) Raw orders. Use orders.malloy as the entry point for analysis.
source: orders_base is conn.table('orders_raw')
  include {
    public: id, customer_id, order_date, total
    internal: raw_json_payload, deprecated_status_code, _temp_dedup_marker
  }
  extend {
    primary_key: id
  }
```

```malloy
// orders.malloy
import "orders_base.malloy"

#(doc) Order analysis. Use for revenue, fulfillment, and customer-order joins.
source: orders is orders_base extend {
  // joins, measures, curated dimensions
}
```

The base source stays fully queryable (`run: orders_base -> { ... }` still works); `internal:` only governs which columns appear as public dimensions callers can pick.

From malloydata/publisher, MIT.

More from malloydata/publisher