AGENTS.md · xorbitsai/inference

Sections

1. Project Overview
2. Working Rules
3. Environment Setup
4. Formatting and Linting
5. Python Tests
6. Frontend
7. Documentation
8. Model and Runtime Conventions
9. CI Expectations
10. Git and Review Hygiene

First lines (shown under the repo's Apache-2.0 license)

# AI Agent Guidance

Guidance for AI coding agents working in this repository.

## Project Overview

Xinference is a Python model-serving project for language, embedding, rerank,
image, video, audio, and multimodal models. It exposes CLI commands, a Python
client, REST/OpenAI-compatible APIs, an xoscar-based distributed runtime, and a
React Web UI.

Primary package and entry points:

- `xinference/`: Python package.
- `xinference/deploy/cmdline.py`: CLI entry points for `xinference`,
  `xinference-local`, `xinference-supervisor`, and `xinference-worker`.
- `xinference/core/`: supervisor/worker runtime and actor orchestration.
- `xinference/model/`: model families, engines, built-in model specs, and model
  tests.
- `xinference/api/`: API server and OpenAI-compatible routes.
- `xinference/client/`: sync and async Python clients.
- `frontend/`: Next.js Web UI.
- `doc/source/`: Sphinx documentation.
- `.github/workflows/python.yaml`: main lint and test CI.

## Working Rules

- Prefer small, focused changes that match the current module's style.
- Preserve backward compatibility. If a breaking change is unavoidable, document
  the reason and add deprecation behavior where practical.
- Do not edit `xinference/thirdparty/` unless the task explicitly concerns
  vendored code.
- Avoid broad refactors while fixing a localized bug.
- Keep public API behavior, request/response schemas, model registration names,
  and CLI flags stable unless the task requires changing them.
- Add or update tests for behavior changes. For model-runtime changes, prefer
  tests close to the affected model family under `xinference/model/**/tests/`.
- Use type hints for new Python code when practical; this project encourages
  PEP 484 style annotations.
- Treat docs and examples as user-facing API. Keep command examples accurate.

## Environment Setup

Recommended local setup:

```bash
conda create --name xinf python=3.12 nodejs
conda activate xinf
pip install -e ".[dev]"
```

Notes:

- Wheel, sdist, and editable builds (including `pip install -e .`) run the
  Web UI build through the in-tree build backend (`build_backend.py` /
  `build_web.py`) unless `NO_WEB_UI=1` is set.
- For Python 3.12 and newer, CI installs `setuptools<82`; use the same pin if
  packaging or editable installs fail.
- The project supports Python 3.10 through 3.14 in CI.
- Optional model engines have extras in `pyproject.toml`, such as
  `transformers`, `vllm`, `mlx`, `embedding`, `rerank`, `image`, `video`, and
  `audio`.

## Formatting and Linting

Python formatting and checks are managed through pre-commit:

```bash
pip install pre-commit
pre-commit run --files <modified-files>
```

For a branch-wide check against upstream main:

```bash
pre-commit run --from-ref=upstream/main --to-ref=HEAD --all-files
```

Configured hooks include Black, end-of-file/trailing-whitespace checks, Ruff,
isort, mypy with missing imports ignored, and codespell. Configuration lives in
`.pre-commit-config.yaml` and `pyproject.toml`.

The `test-platform-coverage` hook also checks changed tests against the CI
platform scopes. Include tests in `PLATFORM_TEST_PATHS` or `LINUX_TEST_PATHS` in
`.github/workflows/python.yaml`, use the existing macOS MLX scope, or declare
`# CI: linux` for portable/Linux-only regressions. Local checks discover Git
changes automatically and honor `--from-ref` / `--to-ref`; CI supplies the PR's
base and head SHAs to check the same scope as the workflow tests.

## Python Tests

Run focused tests first:

```bash
pytest -vv path/to/test_file.py
```

The broad CI-style non-GPU test command is approximately:

```bash
pytest --timeout=3000 -W ignore::PendingDeprecationWarning -vv \
  --cov-config=pyproject.toml --cov-report=xml --cov=xinference \
  --ignore xinference/core/tests/test_continuous_batching.py \
  --ignore xinference/model/image/tests/test_stable_diffusion.py \
  --ignore xinference/model/image/tests/test_got_ocr2.py \
  --ignore xinference/model/audio/tests \
  --ignore xinference/model/embedding/tests/test_integrated_embedding.py \
  --ignore xinference/model/llm/transformers/tests/test_tensorizer.py \
  --ignore xinference/model/llm/tests/test_llm_model.py \
  --ignore xinference/model/llm/vllm \
  --ignore xinference/model/llm/sglang \
  --ignore xinference/client/tests/test_client.py \
  --ignore xinference/client/tests/test_async_client.py \
  --ignore xinference/model/llm/mlx \
  xinference
```

Use narrower commands for daily development. Many model tests require large
dependencies, GPU, Metal, network access, or model downloads.

## Frontend

The Web UI is under `frontend` and is a Next.js app (React, TypeScript,
Tailwind CSS), built as a static export and served by the Python backend from
`xinference/ui/web/dist`.

Common commands:

```bash
cd frontend
npm ci
npm run dev
npm run build
npx eslint .
```

Use `npm run format` only when you intentionally want Prettier writes across
the frontend tree.

## Documentation

Documentation source is in `doc/source`.

Common docs dependencies are included in the `doc` extra:

```bash
pip install -e ".[doc]"
cd doc
make html
```

When changing CLI behavior, API behavior, deployment behavior, or model support,
update the relevant documentation pages in `doc/source`.

New or changed English documentation must include corresponding gettext updates
for every existing locale under `doc/source/locale/*/LC_MESSAGES/`. Keep PO
changes scoped to the affected messages, validate every modified catalog with
`msgfmt --check --check-format`, compile them with
`python doc/build_i18n.py --all`, and build the changed page in each maintained
language to verify the rendered translations.