AGENTS.md · dask/dask

Sections

1. Project Overview
2. Architecture
3. Configuration
4. Build & Test Setup
5. Testing Patterns
6. Code Style & Conventions
7. Key Patterns for Contributors
8. CI/CD
9. Release Procedure
10. Common Pitfalls & Gotchas
11. Key Files to Read First (by task)
12. External Dependencies (Key)
13. Documentation
14. Quick Reference: Common Code Patterns
15. Contributing
16. Releasing

First lines (shown under the repo's BSD-3-Clause license)

# AGENTS.md — Dask Project Guide for Coding Agents

## Project Overview

**Dask** is a flexible parallel computing library for dynamic task scheduling and analytics in Python. It provides high-level interfaces that mimic NumPy (arrays), Pandas (dataframes), and generic iterables (bags), with optional integration into the Distributed scheduler for cluster computing.

- **Language**: Python 3.10–3.14 (including free-threaded/3.14t)
- **Package name**: `dask`
- **License**: BSD 3-Clause
- **Repository**: https://github.com/dask/dask

---

## Architecture

### Core Concepts

1. **Task Graphs**: Dask represents computations as directed acyclic graphs (DAGs) of tasks. Each task is a callable (function + arguments), and edges represent data dependencies.

2. **High-Level Graph (HLG)**: A symbolic representation composed of *layers* (e.g., blockwise, data, callable) that can be optimized, fused, and inspected before materializing into a low-level task graph.

3. **Low-Level Graph (LLG)**: The materialized dict-of-tuples form passed to schedulers.

4. **Schedulers**:
   - `dask.threaded` — threaded scheduler (GIL-releasing for numpy/scipy)
   - `dask.multiprocessing` — process-based scheduler
   - `dask.local` — synchronous single-threaded (for testing)
   - `distributed` — remote cluster scheduler (separate repo)

5. **Expr System (dask-expr)**: A symbolic expression tree (`dask._expr.Expr`) that enables query planning and optimization. Enabled via `dataframe.query-planning` config.

### Key Modules

| Module | Purpose |
|---|---|
| `dask/base.py` | Core abstractions: `DaskMethodsMixin`, `compute()`, `persist()`, `optimize()`, `annotate()`, `visualize()` |
| `dask/core.py` | Low-level graph operations: `istask()`, `toposort()`, `get_dependencies()`, `flatten()` |
| `dask/highlevelgraph.py` | `HighLevelGraph`, `Layer` protocol, layer types (`MaterializedLayer`, `CallableLayer`) |
| `dask/blockwise.py` | `Blockwise` layer — the symbolic representation of element-wise operations (`atop`) |
| `dask/_expr.py` | Symbolic expression tree (`Expr`), `HLGExpr`, `LLGExpr`, `SingletonExpr` — the query-planning layer |
| `dask/_task_spec.py` | Modern task representation: `Task`, `DataNode`, `TaskRef`, `execute_graph()` |
| `dask/config.py` | Configuration system: YAML-based settings, `dask.config.get/set/set_options()` |
| `dask/optimization.py` | Graph optimization: `optimize()`, peephole optimizer, caching |
| `dask/delayed.py` | `delayed()` decorator for functional-style task creation |
| `dask/typing.py` | Type aliases: `Key`, `Graph`, `DaskCollection`, `SchedulerGetCallable` |
| `dask/utils.py` | Utility functions: `ensure_dict`, `funcname`, `key_split`, `normalize_url`, `apply_import` |
| `dask/rewrite.py` | Graph transformation utilities |
| `dask/order.py` | Task ordering heuristics |
| `dask/graph_manipulation.py` | `with_output_names`, `map_names`, `clone`, `with_replaced_keys` |
| `dask/dot.py` | Graphviz DOT visualization |

### Subpackages

| Subpackage | Purpose | Key Files |
|---|---|---|
| `dask/array/` | NumPy-compatible distributed arrays | `core.py` (`Array`), `slicing.py`, `reductions.py`, `blockwise.py`, `creation.py`, `linalg.py` |
| `dask/dataframe/` | Pandas-compatible dataframes | `core.py` (`DataFrame`, `Series`, `Index`), `groupby.py`, `shuffle.py`, `dispatch.py` |
| `dask/dataframe/dask_expr/` | dask-expr-based dataframe (query planning) | `_collection.py`, `_expr.py`, `_groupby.py`, `_shuffle.py`, etc. |
| `dask/bag/` | Distributed iterables (RDD-like) | `core.py` (`Bag`, `Item`) |
| `dask/array/_array_expr/` | dask-expr-based array (experimental) | `_collection.py`, `_blockwise.py`, `_reductions.py`, `_shuffle.py` |
| `dask/bytes/` | Byte-level I/O abstraction | — |
| `dask/diagnostics/` | Performance profiling | `profiler.py`, `watch.py` (requires bokeh) |
| `dask/widgets/` | Jupyter notebook widgets | `get_template.py` |

### Module Sizes (for navigation)

Largest files by line count:
- `dask/array/core.py` — 6,278 lines (Array class + methods)
- `dask/utils.py` — 2,361 lines (utilities)
- `dask/blockwise.py` — 1,569 lines (blockwise layer + atop)
- `dask/base.py` — 1,306 lines (compute/persist/annotate)
- `dask/_expr.py` — 1,420 lines (Expr tree)
- `dask/array/slicing.py` — 2,117 lines
- `dask/array/routines.py` — 2,703 lines (NumPy API compatibility)
- `dask/_task_spec.py` — 1,227 lines (modern task spec)

---

## Configuration

Configuration lives in `dask/dask.yaml` and `dask/dask-schema.yaml`. Key paths:

```yaml
dataframe:
  backend: "pandas"              # pandas, cudf, modin, etc.
  shuffle: {method: null}        # disk, tasks, p2p
  parquet: {metadata-task-size-local: 512}
  convert-string: null           # pyarrow string conversion
  query-planning: null           # enable dask-expr

array:
  backend: "numpy"               # numpy, cupy, etc.
  chunk-size: "128MiB"
  rechunk: {method: null}
  query-planning: null

optimization:
  annotations: {fuse: true}
  tune: {active: true}
  fuse: {active: null, ave-width: 1, max-width: null}

tokenize:
  ensure-deterministic: false

admin:
  async-client-fallback: null    # "sync" or "distributed"
```

Set config at runtime:
```python
import dask
with dask.config.set({"dataframe.shuffle.method": "p2p"}):
    ...
```

---

## Build & Test Setup

### Dev Environment (pixi)

The project uses **pixi** for dependency management. Install it from https://github.com/prefix-dev/pixi.

```bash
# Default environment (Python 3.14 + all optional deps)
pixi run test

# Minimum dependency tests
pixi run --environment mindeps-optional test

# Array-expr (experimental)
pixi run --environment py314 test-array-expr

# Run arbitrary Python commands
pixi run -- python -c 'print("Hello world!")'

# Linting
pixi run lint lint

# Doctests
pixi run doctest

# Specific module
pixi run test dask/tests/test_base.py
pixi run test dask/array/tests/test_array_core.py
pixi run test dask/dataframe/tests/test_dataframe.py

# Single test
pixi run test dask/tests/test_base.py::test_compute_depth

# Slow tests
pixi run test --runslow

# Full test suite
pixi run test-ci

# Test against all supported versions of NumPy, Pandas, and PyArrow
# This list may be fall out of sync; run `pixi ls -e <environment>` to double check
pixi run -e mindeps-optional ...  # NumPy 1.24, Pandas 2.0, PyArrow 16
pixi run -e py310 ...  # NumPy 1.26, Pandas 2.1, PyArrow 18