AGENTS.md · kornia/kornia

Sections

1. Project shape
2. Environment and commands
3. Library preferences
4. Responding to review
5. Documentation and generated examples
6. Benchmarks
7. ONNX work
8. Helping users diagnose problems

First lines (shown under the repo's Apache-2.0 license)

# Agent guide for kornia

This file gives coding agents practical context for working in this repository.

## Project shape

kornia is a differentiable computer vision library built on PyTorch. Most code lives in subpackages under `kornia/`; reusable test support lives in `testing/`, not under `tests/`.

Important areas include:

- `kornia/geometry/`: transforms, cameras, epipolar geometry, stereo, and 3D
- `kornia/filters/`: filtering and edge operators
- `kornia/augmentation/`: differentiable augmentation pipelines
- `kornia/feature/` and `kornia/models/`: feature and model implementations
- `kornia/onnx/`: ONNX loading, execution, composition, and export
- `kornia/core/`: shared tensor, module, and ONNX mixins

Import order in `kornia/__init__.py` matters: `filters` and `geometry` are loaded before modules that depend on them. Check for circular imports when changing package exports.

## Environment and commands

Use Pixi for project tasks and environment selection. The tasks call `uv`, and Python tests run in the repository `.venv` or a versioned `.venv-py*`; do not assume packages in `.pixi` are the versions under test.

```bash
pixi install
pixi run -e default install
pixi install -e py312
pixi run -e py312 install
pixi install -e py313
pixi run -e py313 install
pixi run -e cuda install
pixi run test-module tests/path/to/test_file.py
pixi run test-module tests/path/to/test_file.py::TestClass::test_method
pixi run test-module tests/path/to/ -k "expression"
KORNIA_TEST_OPTIMIZER=inductor pixi run test-module tests/path/to/test_file.py
pixi run test-quick
pixi run test-slow
pixi run lint
pixi run pre-commit-all
pixi run typecheck
pixi run toml-fmt
pixi run doctest
pixi run doctest-weights
pixi run -e default build-docs
```

The `py312` and `py313` features select isolated `.venv-py312` and `.venv-py313` project environments automatically for install and test tasks.

Select devices and dtypes with pytest options or environment variables:

```bash
pixi run test-module tests/geometry --device=cpu --dtype=float32,float64
KORNIA_TEST_DEVICE=cuda KORNIA_TEST_DTYPE=float32 pixi run test-module tests/geometry
```

Supported fixtures include CPU, CUDA, MPS, and TPU when available, and `float16`, `bfloat16`, `float32`, and `float64`. Run focused checks first and expand them in proportion to the change.

`test-module` forwards its arguments to `pytest`, so a node ID or a `-k` expression narrows a run to a single test.

Use `KORNIA_TEST_RUNSLOW=true` to include slow tests. `KORNIA_TEST_OPTIMIZER=inductor` enables the dynamo and compile tests, which are deselected when the variable is unset; the torch.compile conformance test has its own option (below). `pixi run doctest` skips docstring examples that would download an uncached model checkpoint, so it stays fast and works offline; use `pixi run doctest-weights` (or `KORNIA_DOCTEST_DOWNLOAD=1`) to run those examples for real, which costs ~838 MB on a cold cache. Pull-request CI relies on the restored weights cache and skips whatever it misses; the scheduled run on `main` (00:23 and 12:23 UTC) downloads, so the model examples are always exercised somewhere. Before presenting a code change as finished, run the full pre-commit command above together with focused tests and other relevant checks; `pixi run lint` runs only the Ruff hooks.

The stable-core operators the survey records as compiling in one graph and matching eager, less the exclusions in `testing/conformance_promise.py`, are promised (#5819): `tests/test_inductor_conformance.py` compiles each with `torch.compile(..., fullgraph=True)` and the default inductor backend and compares the result with eager. It is opt-in, `pixi run test-module tests/test_inductor_conformance.py --compile-conformance=<package>|all`; run it for the packages a change touches and the packages that import them (`git diff --name-only origin/main... | python .github/scripts/conformance_packages.py compile` prints that selection), and see [TESTING.md](TESTING.md) for its known-failure manifests.

### Comparing a branch against another revision

kornia is installed in editable mode, so `import kornia` resolves through the editable finder to
the **primary checkout** regardless of the current directory. A measurement script that lives
outside a worktree therefore imports the primary tree even when you run it from the worktree:
`python /tmp/probe.py` puts `/tmp` first on `sys.path`, nothing shadows the editable install, and
an A/B comparison silently measures the same revision twice and reports "no difference".

Run comparison snippets from the worktree root, as stdin or `-c`, never as a file stored
elsewhere. Only then does the leading empty `sys.path` entry -- the working directory -- shadow
the editable install:

```bash
git worktree add /tmp/base-wt <base-revision>
cd /tmp/base-wt && python - <<'EOF'
import kornia
print(kornia.__file__)  # must name the worktree, not the primary checkout
EOF
```

Print `kornia.__file__` inside every measurement block and check that it names the tree you
meant. A `cd` into the worktree is not by itself enough, and neither is putting the worktree on
`PYTHONPATH` while the script still lives somewhere else. The same applies to `sys.executable`
when more than one virtualenv is in play: name the interpreter explicitly rather than relying on
`PATH`.

### Precision and device details

- For focused CPU half-precision coverage, add `--dtype=float16,bfloat16` to a `test-module` run. `pixi run test-half` runs the whole CPU test suite.
- The blocking Linux CPU half jobs run one dtype at a time with `--verify-known-failures` and an explicit
  `--known-failure-profile=cpu-float16` or `cpu-bfloat16`, with no optimizer or `--runslow`. Focused file, node, and
  `-k` runs use `--xfail-known-failures` with the same profile. Profile selection happens before parametrization and
  deterministically isolates each node's RNG. Delete a manifest line after a fix, update it after a rename or
  reparametrization, and fix or deliberately document a new failure. `pixi run test-half` is unseeded; a failure seen
  only there is a test-determinism bug, not a manifest update. See [TESTING.md](TESTING.md) for guarded candidate
  recording, fresh-process replay, and manual installation.
- CUDA `float16`/`bfloat16` tests need per-test subprocess isolation; use `pixi run -e cuda test-cuda-half` or pytest's `--isolate-half-precision` option.
- MPS does not support float64 gradcheck. MPS autocast can also change the effective dtype; inspect nearby tests before changing tolerances or skips.
- The blocking MPS job runs `--device=mps --dtype=float32 --xfail-known-failures`. Its exact strict-xfail baseline is `testing/known_failure_xfails/mps_float32.txt`. A fix removes its manifest line; a rename or reparametrization updates the node ID; a new failure is fixed, or gets its own tracking issue named in a manifest comment before being added. The manifest contract requires a full-suite run, without `-k` or a partial path.
- TF32 matmul is disabled by default; `--tf32` enables it. cuDNN convolutions still use PyTorch's TF32 default. Tests marked `tf32` are xfailed at collection unless `--tf32` is passed, and that marker is non-strict, so a default run reports neither their failure nor their recovery.
- `torch.clamp` is not a portable gradient guard: its derivative **at the bound** passes the incoming gradient
  through (`1.0`) on torch 2.5.1 and 2.9.1 and returns `0.0` on 2.14.0. Wherever the bound is also the singular
  point -- `clamp(min=0).sqrt()`, `clamp(-1, 1).acos()`, `clamp(-1, 1).asin()` -- the clamp bounds the value only,
  so the unbounded derivative still reaches the backward pass as `inf` or `nan` on the older half of the supported
  range while 2.14 masks it. A floor below the dtype's smallest subnormal is no guard either: `clamp(min=1e-12)`
  underflows to `0` in `float16`. Substitute a safe argument into the expression that gets differentiated and take
  the value from a `.detach()`ed copy, or from the other arm of a `torch.where`, instead; that is
  version-independent by construction. A CPU float32 run on a recent torch does not exercise any of this -- the
  2.5.1 leg is what discriminates. The measurements and the audit of the pattern are in #4229.
- Preserve device and dtype rather than creating implicit CPU or default-dtype tensors. Use the injected `device` and `dtype` fixtures in tests.

## Library preferences

- Search for an existing `kornia` operation before building the same operation from raw PyTorch.
- Follow nearby batching, broadcasting, shape-validation, dtype, and device conventions.
- A `KORNIA_CHECK` message that formats a tensor size or an `int` argument specializes it under a dynamic-shape `torch.compile`, which then compiles a graph per value. Compare sizes with `KORNIA_CHECK_SAME_SIZE`, `KORNIA_CHECK_SIZES` or `KORNIA_CHECK_SHAPE`, or pass the message as `lambda: f"..."`. TorchScript rejects a `lambda`, so a function it compiles formats whole shapes instead, such as `{x.shape}`, which do not specialize. `tests/core/test_check_message_call_sites.py` enforces both and lists the functions TorchScript compiles.
- Use `BaseTester` from `testing.base`, injected `device`/`dtype` fixtures, and `self.assert_close()` for tensor comparisons. Its dtype-specific tolerances are intentional.
- Follow [TESTING.md](TESTING.md) for test mechanisms. Use `self.gradcheck()` for gradient checks and the injected `torch_optimizer` fixture for `torch.compile` coverage. Include smoke, exception, shape/cardinality, numerical, gradient, and compile tests when they apply; not every change needs every kind.
- Write docstrings in the style of [CONTRIBUTING.md](CONTRIBUTING.md#docstring-style): a unique task-word title line, a contract paragraph that starts from the call, `See Also:` with explanations, and shape literals instead of `:math:`. `tests/test_docstring_style.py` checks See Also, references, shape literals and catalogs for the modules listed in `testing/docstring_rollout.py`; `kornia/geometry/depth.py` is the worked example.
- Keep numerical correctness tests self-contained. For expected values produced by an optional reference library, prefer a hardcoded literal plus the small generation snippet and source. Optional-dependency integration tests, including ONNX tests, may still use `pytest.importorskip`.
- For a new algorithm, name the source that defines it, such as a paper or a reference implementation in PyTorch, OpenCV, or scikit-image.
- Do not keep or copy a bug just because the original implementation has it. When a port differs from its reference, decide which behaviour is correct before changing anything: "the reference does it" is not a justification on its own. For example, a non-maxima suppression that keeps positions equal to their window maximum (`>=`), or that re-admits positions once their stronger neighbours are suppressed, is unstable and wrong in principle; kornia keeps strict maxima. When kornia departs from a reference on purpose, say so in the docstring and give the reason.
- Public APIs need type hints, docstrings, and exports.
- `tests/api_surface.json` inventories the public names of stable-core modules; `tests/test_api_surface.py` rejects their removal until the inventory is updated in the same change. Run `tests.test_api_surface.regenerate()` to regenerate it. Keep tracked module keys (use `[]` when removing every name). Every module that declares `__all__` under a recorded package root has its own key; deleting such a module empties its key, and that empty key stays as the record. The `Import Surface` workflow also checks that each inventory removal corresponds to an actual export removal in the same diff. An inventory edit acknowledges an `__all__` removal only for the exact same module and name. For an `__all__` name of a module without its own key, such as a module outside the recorded package roots, add the exact module and name to `tests/api_surface_removals.json`, a JSON object mapping module names to lists of removed names (for example, `{"kornia.feature.dedode": ["DeDoDe"]}`). Only entries newly added relative to the merge base count, and each must match an actual `__all__` removal in that change; an unrelated source edit or a previously recorded acknowledgement does not count. Explicit acknowledgements do not replace updates to existing inventory entries, the deprecation window, or release notes.
- The codebase keeps a 120-character line length and Apache 2.0 source headers. Ruff and `ty` enforce the current style and types.
- JIT-compatible modules have stricter typing constraints. Follow nearby annotations and use `torch.Tensor` directly where TorchScript expects it.
- For non-JIT modules, use `from __future__ import annotations`.
- Avoid new runtime dependencies. If a dependency is genuinely useful, explain why the existing stack is insufficient and consider its install, build, and platform cost.

## Responding to review

- Treat a fix that a reviewer suggests, whether a maintainer, a bot or another agent, as a hypothesis about the cause. Before applying it, check whether kornia already maintains the invariant it protects. For example, `Boxes._valid` marks rows that are not boxes: every indexed read and write carries it and every export zeroes those rows, so rows that must never export as boxes do not need a separate count that each write path would have to keep in sync.
- If two review rounds in a row report a regression that the previous fix round introduced, stop patching individual cases. State the invariant, find the code responsible for keeping it, and redesign before pushing another fix, reverting earlier rounds when that is simpler.

## Documentation and generated examples

- A user-visible change adds a `changelog.d/<PR>.<type>.md` fragment (`added`, `fixed`, or `breaking`), not an edit to `CHANGELOG.md`. See [changelog.d/README.md](changelog.d/README.md) for naming before a PR exists and the release workflow. A breaking change says what used to happen as well as what happens now. Internal changes may omit a fragment with an explanation in the PR and the `no-changelog` label. Release PRs also use that label when consuming fragments. Only release PRs assemble and consume fragments.
- Add every new public class or function to the corresponding `docs/source/*.rst` page so it appears in the rendered API reference.
- A pull request description goes in the pull request body, never in a committed file such as `PR.md`. Fill in every section of the [pull request template](.github/pull_request_template.md), including "Which part was done by a human, if any?": state plainly what the user actually did in the session (for example, `"None — @<user> asked to fix #NNNN; AI did the rest"`, or the specific diagnosis, design, or edits the user provided) rather than repeating what the agent did in a faux-human `"Used <Agent> to..."` voice.
- Every pull request description, issue body, comment, and review posted to GitHub by an agent on a user's behalf must end with the line:
  `Posted on behalf of @<github-handle> by <Agent> (<actual model>).`
  Write the user's GitHub handle and the actual agent and model you are running as, never a copied literal or a generic tool footer (`Generated with ...`, `Co-Authored-By:`).
- Some modules document a known defect inline and link its tracking issue; `grep -rn "github.com/kornia/kornia/issues/" kornia/` finds them. That prose is part of the fix's blast radius: **before merging a change that closes one of those issues, run `grep -rnE "#NNNN|issues/NNNN" kornia/ tests/`.** Match the issue number rather than a name or a phrase — the wording varies across `Tracked in #NNNN`, a lowercase `tracked in` clause, a `Note:` block, a runtime error message, an `xfail` `reason=`, a plain comment and a test-name suffix (`test_wart_*_<issue>` and `test_convention_*_<issue>`, where that convention was followed), and a rendered link label can disagree with the URL beneath it. The `#`/`issues/` anchors keep the pattern off float literals. A surviving hit in `kornia/` means the change is incomplete. Hits in `tests/` are the pins that record the documented behavior; each has to be re-checked in the same change — a now-XPASSing strict `xfail` dropped, a wart assertion inverted — but a pin goes on naming its issue afterwards, so a hit there is not by itself a defect. Not every documented issue has a pin, so an empty `tests/` result is an answer rather than a failed search.
- When adding a feature detector or descriptor, update the `responses` list in `docs/generate_examples.py` and add the matching branch that renders its heatmap or score visualization. Follow the existing `DISK`, `ALIKED`, and `XFeat` examples and preserve the expected `(B, 3, H, W)` image shapes.

## Benchmarks

Follow the methodology contract and new-benchmark checklist in [benchmarks/README.md](benchmarks/README.md), using the helpers in `benchmarks/common.py`. Benchmark public kornia APIs rather than embedding a replacement implementation. For performance changes, run the same benchmark on the relevant base revision and changed branch.

## ONNX work

ONNX support is a real subsystem, not a generic export afterthought. Inspect `kornia/onnx/`, its mixins in `kornia/core/`, and existing ONNX tests before changing export behavior. Development uses the optional `onnx`, `onnxruntime`, and `onnxscript` packages. Test both graph construction/export and runtime behavior when the change affects both.

The stable-core operators the survey records as exporting are promised (#5819): `tests/onnx/test_conformance.py` exports each with the dynamo exporter and the legacy one and compares onnxruntime with eager. It is opt-in, `pixi run test-module tests/onnx/test_conformance.py --onnx-conformance=<package>|all`; run it for the packages a change touches and the packages that import them (`git diff --name-only origin/main... | python .github/scripts/conformance_packages.py onnx` prints that selection), and see [TESTING.md](TESTING.md) for its known-failure manifests.

## Helping users diagnose problems