pyspark

Files (1)

• SKILL.md

SKILL.md (first lines)

# PySpark

Rules, patterns, and solutions for PySpark and Delta Lake code.

## Baseline

- Python 3.11+; target one supported pair: Apache Spark 3.5.x with Delta Lake
  3.0.x-3.3.x, or Spark 4.0.x with Delta Lake 4.0.x.
  These are this pack's supported families as of 2026-10-12, not a promise of
  compatibility across future Spark 4 releases; see the official
  [Delta Lake compatibility matrix](https://docs.delta.io/releases/).
  Validate adapted code on the selected runtime, including managed runtimes.
- Spark 4.0 enables ANSI mode by default: invalid casts and date parses, array
  index out of bounds, integer overflow, and division by zero raise errors
  instead of returning null. Code must be correct with ANSI on or off.
- Code must run on Spark Connect: no `SparkContext`, RDD API, `SQLContext`,
  `HiveContext`, or `registerTempTable`.
- Label Databricks-only features (`dbutils`, Unity Catalog, volumes, serverless
  compute, Lakehouse Federation) in text; keep OSS Spark/Delta code portable.

## Core Rules

- Import `from pyspark.sql import DataFrame, Window, functions as F`. Never
  `from pyspark.sql.functions import *`: it shadows `sum`, `min`, `max`,
  `round`, `abs`, and `filter`.
- Prefer built-in functions over Python UDFs; follow `udf-avoidance-hierarchy`.
- Add several columns with one `withColumns({...})` or `select`, not a loop of
  `withColumn` calls (each adds a projection to the plan).
- Test emptiness with `df.isEmpty()`, not `count() == 0` or `limit(1).count()`.
- Parse untrusted values with non-throwing forms (`try_cast`, `F.try_divide`,
  `F.try_element_at`, `F.try_to_timestamp(col, F.lit(fmt))`), then reject or
  quarantine rows whose raw value is non-null but parsed value is null.
  `F.try_to_date` is not available in Spark 3.5/4.0.
- Pass `how=` on every join; use `left_semi` / `left_anti` for existence checks.
- Use `unionByName`, not positional `union`; see `union-vs-unionbyname-column-order-gotcha`.
- Pass an explicit schema for production CSV/JSON reads; never `inferSchema`.
- `collect()` / `toPandas()` only on aggregated or bounded results.
- `cache()` only for a DataFrame reused by two or more actions or scanned more
  than once in one plan (see `cache-before-reused-transform`); `unpersist()`
  after the last consuming action.
- Leave shuffle sizing, skew handling, and runtime broadcast conversion to AQE
  (on by default since Spark 3.2); tune only after the query plan shows a problem.
- Delta: read by table name or `format("delta")`, never `spark.read.parquet` on a
  Delta directory. Upsert with `MERGE`. For new tables, prefer liquid clustering
  (`CLUSTER BY`, Delta Lake 3.1+ and Databricks) over partitioning plus `ZORDER`;
  partition only very large tables on a low-cardinality, always-filtered column.
- For Databricks table and file access, follow the canonical
  [Parameters and Data Access rules](../databricks-notebook/SKILL.md#parameters-and-data-access).

## Load Patterns and Solutions

Read the following files for named, ready-to-use implementations with when/replaces/gotchas:

- `../../catalogs/pyspark-patterns.md`
- `../../catalogs/pyspark-solutions.md`

From mabrosim/aet-content-pack, MIT.

More from mabrosim/aet-content-pack