AGENTS.md · PrefectHQ/prefect

Sections

1. Quick Reference
2. Tech Stack & Tooling
3. Architecture Overview
4. Anti-patterns
5. Make sure you have a Prefect server or Prefect Cloud
6. Code Conventions
7. Testing
8. Formal Protocol Models
9. Working on Issues
10. PR Body Style
11. Code Review Rules

First lines (shown under the repo's Apache-2.0 license)

Prefect is a workflow orchestration platform that coordinates and observes data pipelines. It provides a Python SDK for building workflows, a server backend for orchestration, and a web-based UI for managing and monitoring workflows.

# Guiding Principles

Your primary responsibility is to the project and its users. Every change should serve the broader user base — not just the immediate request. Be a quality gate: prefer correct, minimal, well-tested changes over fast ones.

# Before You Start

Before making code changes, read the public contribution guide in `docs/contribute/dev-contribute.mdx` so your work follows Prefect's contribution expectations, design principles, AI-use policy, testing standards, and maintainer review criteria.
For documentation-only changes, also read `docs/contribute/docs-contribute.mdx`.
For integration changes, also read `docs/contribute/contribute-integrations.mdx`.

# Directory Structure

```
prefect/
├── benches/                         # Benchmarks (CLI, flows, tasks, imports)
├── client/                          # prefect-client build: subset of src/prefect/ published as a separate PyPI package
├── compat-tests/                    # Tests for REST API compatibility with Prefect Cloud
├── Dockerfile                       # Production container image
├── docs/                            # Mintlify documentation (see docs/AGENTS.md)
├── examples/                        # Example flows (auto-published to docs)
├── formal/                          # Executable formal models (see formal/tla/AGENTS.md)
├── integration-tests/               # End-to-end integration tests (require running server)
├── justfile                         # Task runner (just <command>)
├── load_testing/                    # Load/performance testing
├── plans/                           # Design/implementation plan documents
├── pyproject.toml                   # Root package config
├── schemas/                         # JSON schemas (prefect.yaml, settings)
├── scripts/                         # Code generation and release scripts
├── src/
│   ├── integrations/                # External service integrations (see src/integrations/AGENTS.md)
│   └── prefect/                     # Core package: SDK, server, CLI (see src/prefect/AGENTS.md)
├── tests/                           # Test suite, mirrors src/prefect/ (see tests/AGENTS.md)
├── tools/                           # Build tools
├── ui/                              # Vue UI (legacy, will be replaced by ui-v2)
├── ui-v2/                           # React UI rewrite (see ui-v2/AGENTS.md)
└── uv.lock                          # Lockfile (auto-generated, do not edit manually)
```

# Essential Commands

```bash
# Dependencies
uv sync                                # Install all dev dependencies
just install                           # Same as above, plus perf group

# Running code
uv run -s my_script.py                             # Run a script with an editable prefect install
uv run --extra aws repros/1234.py                  # Run repro needing an integration extra
uv run --project ./src/integrations/<name> <cmd>   # Run against a local integration from repo root
prefect server start                               # Start local server
prefect config view                                # Inspect current configuration

# Testing (see tests/AGENTS.md for full details)
uv run pytest tests/path.py -k name    # Run specific test
uv run pytest tests/path.py -x -n4     # Parallel, stop on first failure

# Linting & formatting
uv run ruff check --fix .              # Lint with auto-fix
uv run ruff format .                   # Format code
uv run pre-commit run --all-files      # Run all pre-commit hooks

# Docs (from repo root)
just docs                              # Start Mintlify dev server at localhost:3000
just generate-docs                     # Regenerate all doc artifacts

# UI (from repo root)
just ui-v2                             # Start React dev server at localhost:5173

# Docker
docker build -t prefect .                              # Default build (Python 3.11)
docker build --build-arg PYTHON_VERSION=3.12 -t prefect .  # Custom Python version
docker build --build-arg EXTRA_PIP_PACKAGES="prefect-aws" -t prefect .  # With extras
```

## Quick Reference

| Component | Path | Tests |
|-----------|------|-------|
| Core SDK (flows, tasks, states, deployments) | `src/prefect/` | `tests/` |
| Flow & task engines (async orchestration) | `src/prefect/flow_engine.py`, `task_engine.py` | `tests/engine/` |
| Client SDK (schemas, HTTP client) | `src/prefect/client/` | `tests/client/` |
| Server (API, database, scheduling) | `src/prefect/server/` | `tests/server/` |
| CLI | `src/prefect/cli/` | `tests/cli/` |
| Settings | `src/prefect/settings/` | `tests/settings/` |
| Integrations (`prefect-aws`, `prefect-dbt`, etc.) | `src/integrations/` | per-integration |
| React UI (replacing Vue `ui/`) | `ui-v2/` | `ui-v2/e2e/` |

## Tech Stack & Tooling

- **Python >=3.11,<3.16** with modern typing (`list[int]`, `T | None`)
- **FastAPI** for REST APIs
- **Pydantic v2** for validation
- **SQLAlchemy 2.0** async ORM
- **Alembic** for database migrations
- **PostgreSQL/SQLite** databases
- **React + TypeScript** for UI (see ui-v2/AGENTS.md)
- **Ruff** for linting and formatting
- **Pre-commit** hooks: ruff, codespell, mypy (partial), uv-lock, no-commit-to-branch

## Architecture Overview

- **Three user-facing surfaces**: Python SDK (`@flow`/`@task` decorators), CLI (`prefect` command), REST API (FastAPI server)
- **Server is the source of truth for flow state transitions** — the flow engine proposes states, and the server's orchestration layer accepts or rejects them. Task state transitions are managed locally by the task engine via `set_state` and communicated through `prefect.task_run.*` events
- **Two published packages**: `prefect` (full SDK + server) and `prefect-client` (lightweight client subset). `client/` contains the build config that selects which files from `src/prefect/` go into `prefect-client`
- **Integrations are separate PyPI packages** (`prefect-aws`, `prefect-dbt`, etc.) each with their own version, published independently from `src/integrations/`
- **Async-first execution model** with sync wrappers — the engines (`flow_engine.py`, `task_engine.py`) are async; sync `@flow`/`@task` functions are run in workers

## Anti-patterns

- **Never bypass the server for flow state transitions** — always go through the orchestration API, even in tests. Task state transitions are managed locally by the task engine by design
- **Never use `pip install` or `uv pip`** — use `uv` for all dependency management
- **Never use deferred imports** (imports inside functions) unless required to break circular imports or for optional dependencies
- **Never commit directly to `main`** — a pre-commit hook enforces this
- **Never skip pre-commit hooks** (`--no-verify`) — fix the underlying issue instead
- **Never amend commits** (`--amend`) — create new commits instead

# Development Guidelines

## Make sure you have a Prefect server or Prefect Cloud
- use `prefect config view` to check your current profile
- run `prefect server start` in the background if needed

## Code Conventions

- Private implementation details (`_private_method`)
- No public API changes without approval
- Use single backticks for inline code references in docstrings, not double backticks

## Testing

- Directory structure mirrors source code
- Run affected tests after changes: `uv run pytest tests/module/`
- Tests require deterministic behavior
- Mock external dependencies

## Formal Protocol Models

- Use TLA+ selectively for bounded concurrent or distributed protocols whose correctness depends on interleavings, retries, duplication, time, or ownership. Prefer ordinary tests for local or sequential behavior.
- Treat each `formal/tla/<protocol>/README.md` as the authority for its scope, configurations, action-to-code mapping, owner, and review triggers.
- Before changing mapped behavior, review the model and its deterministic regressions. Update them when the abstraction changes; otherwise state in the PR why it remains valid.
- A green model validates only its documented abstraction, not Prefect's implementation. Implementation evidence comes from mapped behavior and contract tests.

## Working on Issues

- Read the GitHub issue/PR and comments (`gh issue view`, `gh pr view`) before writing code
- Understand the scope and intended behavior — ask clarifying questions if unclear
- Create reproduction scripts in `repros/` directory (create it if needed; add to `.gitignore`)
- Name files by issue number: `repros/1234.py` (only create one file per issue)
- Ensure the issue is reproducible before fixing
- Add unit tests for fixes
- DO NOT delete files from `repros/` directory after reproducing the issue unless asked

## PR Body Style

- Start with "closes #1234" if resolving issue
- Brief summary: "this PR {changes}"
- Details in `<details>` tag
- Include relevant links