AGENTS.md · ArchiveBox/ArchiveBox

Sections

1. Shared Standards
2. Concurrency Contract
3. Development Setup
4. User-Facing Setup
5. Basic Usage
6. Verification

First lines (shown under the repo's MIT license)

# ArchiveBox Agent Guide

ArchiveBox is the full self-hosted web archiving app. Keep this repo on the `dev` branch.

## Shared Standards

- Use `uv` and `uv run` for Python commands. Do not use system `python`, direct `.venv/bin/python`, or `pip` commands.
- Prefer existing repo patterns, helper APIs, fixtures, scripts, and command surfaces.
- Keep edits focused and minimal. Do not add wrappers, shims, aliases, or extra abstraction layers unless the current code path requires them.
- Do not weaken assertions, skip tests, xfail tests, or accept flaky behavior.
- No mocks, monkeypatches, fakes, simulated handlers, fake binaries, fake hooks, fake buses, or direct shortcuts around user-facing flows.
- Tests and verification should use real CLI commands, REST/API calls, browser UI flows, real hooks, real installs, real subprocesses, real DB rows, real files, and existing fixtures.
- Assertions must verify real correctness: exit codes, returned values, DB state, filesystem contents, field values, rendered output, and side effects.
- Start behavior fixes with a red failing test when a test is requested or practical.
- Trace root causes from observed behavior. Do not paper over failures with retries, wider timeouts, broad fallbacks, or looser assertions.
- Read `README.md` for the full setup, CLI, Docker, API, and release surface.

## Concurrency Contract

- A collection has one orchestrator at a time. Local PID/process checks may warn about obvious same-machine duplicates, but must not claim to enforce ownership across machines or shared filesystems.
- SQLite remains supported with concurrent short writes from CLI and server processes. Keep the SQLite database on a local filesystem, never NFS/SMB, and use short autocommit/CAS updates instead of long transactions or database locks.
- Network calls, hook execution, filesystem migrations, and other long work belong in the orchestrator. Never hold a database transaction or lock across that work.
- PostgreSQL and shared data directories are the path to future multi-machine scheduling. Coordinate that work through database-backed per-crawl/per-snapshot claims; do not introduce file leases or timer-based orchestrator election.

## Development Setup

```bash
uv sync --dev --all-extras
mkdir -p data
cd data
uv run --project .. archivebox init --install
```

Run collection commands from inside an initialized data directory:

```bash
cd data
uv run --project .. archivebox status
uv run --project .. archivebox add --plugins=parse_txt_urls 'https://example.com/'
uv run --project .. archivebox run
```

## User-Facing Setup

Recommended CLI install:

```bash
uv tool install --python 3.13 --prerelease explicit --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
mkdir -p ~/archivebox/data
cd ~/archivebox/data
archivebox init --install
archivebox add --plugins=parse_txt_urls 'https://example.com/'
```

Alternative install methods:

- Docker Compose / Docker
- Homebrew
- Debian package
- pip

## Basic Usage

```bash
cd ~/archivebox/data
archivebox version
archivebox help
archivebox status
archivebox install
archivebox add --plugins=parse_txt_urls 'https://example.com/docs-basic-usage'
archivebox list --json --with-headers
archivebox search 'example'
archivebox update --filter-type=domain example.com
archivebox remove --yes --delete --filter-type=exact 'https://example.com/docs-basic-usage'
archivebox run
```

## Verification

Run the test suite with `uv run pytest archivebox/tests -x` and repository checks
with `uv run prek run --all-files`. For focused work, select the relevant tests.

Releases are published only by `.github/workflows/release.yml` after complete CI succeeds for the exact commit. Pushes to `main` prepare stable patch releases automatically; `dev` prepares release candidates. Version selection and dependency repinning are owned by CI and the monorepo coordinator. Local development and deployment commands must not publish packages, images, tags, or GitHub releases.