databricks-notebook

Files (1)

• SKILL.md

SKILL.md (first lines)

# Databricks Notebook

Structure and style rules for Databricks notebooks authored as `.py` files in
Databricks Python-source format. Builds on `python-style.instructions.md`; two
rules below deliberately override its general anti-patterns for this narrow
context — each says so explicitly.

Spark/Delta version pairs and portable DataFrame rules follow the
[PySpark skill](../pyspark/SKILL.md).

## Load Patterns and Solutions

Read the following files for named, ready-to-use implementations with when/replaces/gotchas:

- `../../catalogs/pyspark-patterns.md`
- `../../catalogs/pyspark-solutions.md`

---

## Header and Cell Contract

Use when authoring or reviewing any `.py` file intended to run as a Databricks notebook.

- Line 1 must be exactly `# Databricks notebook source` — no blank line before or after.
- Delimit every cell with exactly `# COMMAND ----------` — no extra dashes or spaces.
- Treat both as non-negotiable: either violation breaks round-trip sync with the
  Databricks workspace UI.

```python
# Databricks notebook source
# MAGIC %md
# MAGIC ## Notebook: <Title>
# MAGIC
# MAGIC <one-paragraph purpose>

# COMMAND ----------

<one logical step of code per cell>
```

---

## Cell Ordering

Use when structuring a new pipeline notebook or reordering cells during review.

- Order cells: doc (`%md`) -> imports (stdlib -> pyspark -> local modules) ->
  job-parameter cell -> optional `spark.conf.set(...)` -> table and path constants
  -> one-transformation-step-per-cell logic, ending in the write cell.
- Imports come first because parameter parsing needs them; constants follow
  parameters because they may depend on them (environment, run date).
- Set Spark config only for a measured need; see `shuffle-partition-tuning` in
  [pyspark-patterns.md](../../catalogs/pyspark-patterns.md).
- Do not interleave imports or config throughout the notebook.
- Do not put all logic in one mega-cell.

---

## Parameters and Data Access

Canonical owner of Databricks table and file-access rules in this pack.
Use when a notebook reads inputs, writes outputs, or depends on run context;
the [PySpark skill](../pyspark/SKILL.md#core-rules) links here for Databricks access.

- Read job or task parameters once, in the parameter cell, with empty defaults
  and explicit validation; see `dbutils-widgets-parameterized-notebook` in
  [pyspark-patterns.md](../../catalogs/pyspark-patterns.md).
- Address tables by Unity Catalog three-level name (`catalog.schema.table`) and
  files by `/Volumes/<catalog>/<schema>/<volume>/...` or a cloud URI governed by
  an external location.
- Do not use DBFS mounts, `dbfs:/` paths, or `/dbfs/` FUSE paths in new code.
- Import shared logic from workspace `.py` modules; do not chain notebooks with
  `%run` for code reuse.

---

## Living Docstring Steps List

Use when a notebook has more than ~3 transformation steps.

- The opening `%md` cell states the notebook's title, a one-paragraph purpose, and
  a numbered Steps list.
- Update the Steps list whenever a step is added, removed, or reordered — treat it
  as a required review-diff item, not optional prose.
- For business-rule-heavy notebooks, add an `## Algorithm Description` section
  explaining the rule in plain language, independent of the code.

```python
# MAGIC %md
# MAGIC ## Notebook: <Title>
# MAGIC
# MAGIC <what this notebook does, one paragraph>
# MAGIC
# MAGIC **Steps:**
# MAGIC 1. Read <input> from <source>; filter to <condition>.
# MAGIC 2. Join with <other input> on <key>.
```

---

## Fail-Fast Validation

Use after a join, dedup, or override step whose failure mode is "silently
drops or duplicates rows."

- Gate data-quality invariants with `assert <condition>, "<message>"` immediately
  after the risky step.
- Do not use `try/except` to catch-and-continue in an unattended batch job.
- Override note: this deliberately overrides `python-style.instructions.md`'s
  general "no bare `assert` for runtime validation" rule — a scheduled notebook
  with no human watching it run should fail loudly and immediately at the exact
  step where an invariant breaks, rather than produce silently wrong output.
  Do not apply this override to interactive/user-facing code paths.

```python
assert not overrides.isEmpty(), f"No overrides for partition dt={latest_partition}"
assert (
    movement.groupBy("movement_id").count().filter("count > 1").isEmpty()
), "Duplicate movement_id"

From mabrosim/aet-content-pack, MIT.

More from mabrosim/aet-content-pack