AGENTS.md · arc53/DocsGPT

Sections

1. Normal local development commands
2. Repository map
3. Coding rules
4. PR readiness

First lines (shown under the repo's MIT license)

# AGENTS.md

- Read `CONTRIBUTING.md` before making non-trivial changes.
- For day-to-day development and feature work, follow the development-environment workflow rather than defaulting to `setup.sh` / `setup.ps1`.
- Avoid using the setup scripts during normal feature work unless the user explicitly asks for them. Users configure `.env` usually.
- Try to follow red/green TDD

### Check existing dev prerequisites first

For feature work, do **not** assume the environment needs to be recreated.

- Check whether the user already has a Python virtual environment such as `venv/` or `.venv/`.
- Check whether Postgres is already running and reachable via `POSTGRES_URI` (the canonical user-data store).
- Check whether Redis is already running.
- Reuse what is already working. Do not stop or recreate Postgres, Redis, or the Python environment unless the task is environment setup or troubleshooting.

> MongoDB is **not** required for the default install. It is only needed if
> the user opts into the Mongo vector-store backend (`VECTOR_STORE=mongodb`)
> or is running the one-shot `scripts/db/backfill.py` to migrate existing
> user data from the legacy Mongo-based install. `pymongo` is not a
> dependency or an extra: install it separately with
> `uv pip install 'pymongo>=4.6'` (a later `uv sync` removes it again).
> The vector store needs Atlas `$vectorSearch`, not a plain `mongo` container.

## Normal local development commands

Use these commands once the dev prerequisites above are satisfied.

### Backend

```bash
uv sync                    # deps from uv.lock + the dev group (test tools, ruff) + docsgpt itself, editable
source .venv/bin/activate  # macOS/Linux
# pip instead: pip install -r docsgpt/requirements.txt -r tests/requirements.txt && pip install -e .
# Optional extras (not installed by default); name every extra you want, `uv sync` removes the others:
# uv sync --extra docling   # docling parser engine (OCR backend, structured output)
# uv sync --extra milvus    # VECTOR_STORE=milvus
# pip: pip install -r docsgpt/requirements-docling.txt (or requirements-milvus.txt); each file = core + the extra.
# `uv pip install -r docsgpt/requirements-docling.txt` needs UV_INDEX_STRATEGY=unsafe-best-match
# (the file adds the PyTorch CPU index; prefer `uv sync --extra docling`).
```

Python 3.12 (`requires-python >=3.12`; CI runs 3.12). A dev `.env` starts from
`cp .env-template .env` and needs at least `POSTGRES_URI` and `INTERNAL_KEY`
(see `docs/content/Deploying/Development-Environment.mdx`).

The backend is also an installable package (`pyproject.toml`, hatchling).
`uv sync` (or `pip install -e .`) installs it editable and puts a `docsgpt` command on PATH;
`python -m docsgpt <command>` works without that step. The quickest dev loop is
`docsgpt dev`: it runs the API and the worker (with beat) from the checkout, both
reloading on save, in one terminal; `--ui` adds the Vite dev server and
`--mock-llm` runs `scripts/mock_llm.py` so no provider key is needed.
`docsgpt doctor` checks PostgreSQL (reachable, schema current), Redis, the model
provider and the API port; run it first when something does not start.
Other commands: `docsgpt api --reload`, `docsgpt worker`, `docsgpt beat`, `docsgpt migrate`,
`docsgpt grant-admin`, `docsgpt prefetch-models`, `docsgpt verify-offline`. Runtime data (`.env`,
`inputs/`, `indexes/`) lives in the checkout by default; `DOCSGPT_HOME` moves
that data home, and `DOCSGPT_ENV_FILE` selects only the `.env` file (see
`docsgpt/core/paths.py`). `bash scripts/build_frontend.sh` builds the web UI
into `docsgpt/static` (gitignored); the API serves it when present
(`docsgpt/ui.py`, switch `SERVE_UI`), and the package workflows run the script
before `uv build` so the wheel ships it.

Dependencies are declared in `pyproject.toml` and locked in `uv.lock`; the
`docsgpt/requirements*.txt` files are exported from the lock. To add or
bump a package: edit `pyproject.toml`, run `uv lock`, then
`bash scripts/export_requirements.sh` (CI fails if the exports are stale).
Never edit the requirements files by hand.

Run the API. For local dev, prefer the ASGI entrypoint under uvicorn — it
serves the **whole** app, matches production, and hot-reloads:

```bash
uvicorn docsgpt.asgi:asgi_app --host 0.0.0.0 --port 7091 --reload
```

`flask --app docsgpt/app.py run --host=0.0.0.0 --port=7091` is a faster
inner loop (quick startup, the Werkzeug interactive debugger), but it serves
**only** the WSGI Flask app and omits the routes mounted on the ASGI shell
in `docsgpt/asgi.py`:

- the `/mcp` FastMCP endpoint,
- the chat reconnect reader `GET /api/messages/<id>/events`,
- the notification stream `GET /api/events`,
- the remote-device command stream `GET /api/devices/sessions/<id>/events`,
- artifact downloads `GET /api/artifacts/<id>/download`, and
- a workflow run's file downloads `GET /api/workflow_runs/<run_id>/artifacts/<artifact_id>/download`.

These are native-async Starlette routes because they hold a response open
for a long time; on Flask each would pin a WSGI threadpool slot. Under
`flask run` those paths 404: chat still works (`POST /stream` is a Flask
route), but live notifications, stream auto-resume, paired devices and
artifact downloads don't. Use `flask run` only when you don't need them.
`docsgpt api` and `docsgpt dev` serve the ASGI app. The user-facing copy of
this list is the "ASGI-only features" section of
`docs/content/Deploying/Development-Environment.mdx`; keep the two in step.

Production uses `gunicorn -k docsgpt.gunicorn_worker.BoundedDrainUvicornWorker`
against the same `docsgpt.asgi:asgi_app` target; see `docsgpt/Dockerfile` for
the full flag set.

Run the Celery worker, with the embedded beat scheduler (`-B`), in a separate
terminal:

```bash
celery -A docsgpt.app.celery worker -l INFO -B
```

`docsgpt worker` (or `python -m docsgpt worker`) runs the same thing: it adds
`-B` itself (`--no-beat` drops it) and picks the solo pool on macOS and Windows.

**Beat must run somewhere.** It fires scheduled agent runs, source syncs, background-job sweeps,
reconciliation, retention cleanups and the version check; without it they
silently never happen. Extra beat instances are safe (RedBeat holds a lock in
Redis). Celery rejects `-B` on Windows: drop it there and run
`celery -A docsgpt.app.celery beat -l INFO` (or `docsgpt beat`) next to the worker.

**The worker is required for retrieval, not optional.** `EMBEDDINGS_DELEGATE_TO_WORKER`
defaults on, so the API embeds each query by dispatching to the worker rather than
loading a model of its own — which keeps the API process around 285 MB instead of
about 660 MB. Without a worker consuming `EMBEDDINGS_QUEUE`, every search fails after
`EMBEDDINGS_DELEGATE_TIMEOUT`. To run the API on its own, either set
`EMBEDDINGS_DELEGATE_TO_WORKER=false` (loads the model in-process) or point
`EMBEDDINGS_BASE_URL` at an embeddings service.

On macOS, prefer the solo pool for Celery:

```bash
python -m celery -A docsgpt.app.celery worker -l INFO -B --pool=solo -X analytics
```

`-X analytics` keeps the conversation-analytics sweep, which holds a worker for
minutes, off the solo worker. With `DECISION_ANALYTICS_ENABLED` on, run
`docsgpt worker --analytics` (or `celery ... worker -Q analytics -n analytics@%h`)
next to it; `docsgpt worker` excludes the queue itself, and `docsgpt dev` starts
the analytics worker while the setting is on.

Note that `--pool=solo` costs roughly 350 ms per query embed against ~55 ms on the
default prefork pool — nearly all of it the solo worker picking the message up, not
the embedding itself. That only affects local dev; production runs prefork.

A bare worker (no `-Q`) consumes every configured queue, so one worker does the
whole job — app tasks, ingests, query embedding, and document parsing (the
`read_document` tool / workflow native-file parse) alike. Use `-Q` only to split load:
run the main worker with `-Q docsgpt`, a dedicated (e.g. GPU-enabled) parser worker
with `-Q parsing` for heavy OCR, `-Q embeddings` to keep query latency off the ingest
pool, and `-Q ingest` for whole-source jobs (`INGEST_QUEUE`: uploads, remote and
connector ingests, re-ingests, syncs, graph builds, wiki conversion). One of those can
hold a slot for hours, so on a shared worker they starve beat tasks; attachments and
single wiki-page re-embeds stay on `docsgpt`. `analytics` needs a consumer too
(`-Q analytics -c 1`, or listed in another worker's `-Q`). The ingest tasks parse
in-process; only `read_document` is routed to `parsing`. A worker started with an
explicit `-Q` list consumes only that list, so every queue must be covered by some
worker. When you split workers, keep `-B` on at least one of them.

### Frontend

Install dependencies only when needed, then run the dev server:

```bash