AGENTS.md · ggozad/haiku.rag

Sections

1. Quick Reference
2. Project Structure
3. Architecture
4. HaikuRAG Client API
5. Multiple Databases
6. Configuration
7. CLI Commands
8. MCP Server Tools
9. Testing
10. Code Conventions
11. Comments and Docstrings
12. Critical Rules
13. Common Gotchas
14. Planning and Commits
15. Common Patterns
16. Evaluations
17. haiku.rag.app (Web application)

First lines (shown under the repo's MIT license)

# haiku.rag Project Guide

Agentic RAG system built on LanceDB with hybrid search, multiple embedding providers, reranking, and a native Pydantic AI RAG capability with sandboxed code execution.

## Quick Reference

```bash
uv sync                                    # Install dependencies
pytest                                     # Run tests
ty check                                   # Type checking
ruff check && ruff format                  # Lint and format Python
cd app/frontend && biome check             # Lint and format frontend (TypeScript/JSX)
haiku-rag <command> --db /tmp/test.lancedb # CLI with temp DB (ALWAYS use temp DB for testing)
haiku-rag mcp --stdio                      # MCP server for Claude Desktop
```

## Project Structure

**Two PyPI packages:**
- `haiku.rag` - `haiku.rag-slim` with the docling, voyageai, cohere, zeroentropy, tui, cross-encoder and jina extras
- `haiku.rag-slim` - Core with optional extras

**Layout:**
```
haiku_rag_slim/haiku/rag/   # Source code
├── store/                  # LanceDB persistence
│   ├── engine.py           # Store class: connection, locks, migrations, vacuum, tags
│   ├── schema.py           # Table records (DocumentRecord, ChunkRecord, SettingsRecord), Arrow schemas, index_specs/ensure_indexes
│   ├── compression.py      # docling blob compression
│   ├── info.py             # gather_database_info, get_database_stats, DatabaseInfo
│   ├── models/             # Domain models: chunk (Chunk, SearchResult), document, document_item, citation
│   ├── repositories/       # CRUD: DocumentRepository, ChunkRepository, DocumentItemRepository, SettingsRepository
│   ├── upgrades/           # Version migrations (v0_20_0 … v0_89_0)
│   └── exceptions.py       # ReadOnlyError, MigrationRequiredError, UpgradeRequiredError, AmbiguousDatabaseError,
│                           # UnknownDatabaseError, AmbiguousCitationError, SourceUnavailableError, ConfigMismatchError, TagError
├── embeddings/             # VoyageAI, Cohere, vLLM, OpenRouter (ollama/openai via pydantic-ai)
├── reranking/              # cross-encoder, Cohere, Zero Entropy, Jina, Jina-local, vLLM, OpenRouter
├── sandbox/                # pydantic-monty sandbox used by RAGCapability and MCP execute_code
│   ├── sandbox.py          # Sandbox, SandboxResult
│   └── dependencies.py     # AnalysisContext (per-invocation filter)
├── providers/              # docling_serve.py (docling-serve client), picture_description.py
├── capabilities/           # Native Pydantic AI capabilities
│   ├── _tools.py           # Shared capability primitives
│   ├── ledger.py           # CapabilityEvidenceRecord, citation_status
│   ├── evidence.py         # discover_evidence(), DiscoveredEvidence
│   ├── compaction.py       # EvidenceCompactionCapability, build_capsule, compact_history
│   ├── policy.py           # CitationPolicyCapability, CitationPolicyState
│   ├── rag.py              # RAGCapability, RAGState, create_capability
│   └── instructions/       # Model instructions (rag.md, rag_multiple_collections.md)
├── utils/                  # Shared helpers organized by purpose
│   ├── models.py           # Model construction and provider settings
│   ├── images.py           # Image and retrieved-picture helpers
│   ├── sql.py              # SQL escaping and document ID filters
│   ├── formatting.py       # Citation and byte formatting
│   ├── concurrency.py      # Concurrent awaitable handling
│   ├── http.py             # Pooled httpx client, retry of a dropped connection
│   ├── paths.py            # Database and data directory paths
│   ├── packages.py         # Package versions and update checks
│   └── dependencies.py     # Optional dependency errors
├── chunkers/               # docling-local, docling-serve
├── converters/             # docling-local, docling-serve, pdf_split.py, text_utils.py, exceptions.py
├── config/                 # models.py (all config classes), loader.py
├── chat/                   # Chat TUI (app.py, widgets/)
├── inspector/              # Inspector TUI (app.py, widgets/)
├── client/                 # HaikuRAG high-level API
│   ├── __init__.py         # Lazy facade: HaikuRAG, RebuildMode, DatabaseScope, DocumentImport, all_found
│   ├── client.py           # HaikuRAG class, RebuildMode
│   ├── scope.py            # DatabaseScope, DatabaseRef (which databases an operation covers)
│   ├── session.py          # SingleDatabaseSession, FederatedSession
│   ├── documents.py        # create/import/update_document(s), create_document_from_source, DocumentImport
│   ├── processing.py       # convert, chunk
│   ├── titles.py           # generate_title
│   ├── search.py           # search, expand_context, visualize_chunk
│   ├── agents.py           # ask
│   ├── rebuild.py          # rebuild_database
│   ├── downloads.py        # download_models
│   └── exceptions.py
├── ingester/               # haiku-ingester service (own CLI + [ingester] extra)
│   ├── cli.py, app.py      # haiku-ingester entry point + application layer
│   ├── batch.py            # run-batch, dry-run manifests
│   ├── reconcile.py        # Startup reconciliation of LanceDB and sync_state
│   ├── metadata.py         # Metadata provider loading
│   ├── queue/              # db.py, migrations.py, repository.py (JobRepo/SyncStateRepo), models.py
│   ├── workers/            # pool.py (WorkerPool), pipeline.py (run_job), retry.py
│   ├── pollers/            # base, periodic, fs, manager, factory
│   ├── api/                # FastAPI control plane (server.py, auth.py, schemas.py)
│   │   ├── routes/         # health, sources, jobs, dlq, stats, config, database, dashboard, providers, reload
│   │   └── static/index.html # Self-contained vanilla-JS dashboard served at /
│   └── exceptions.py       # PermanentError, TransientError
├── sources/                # base, fs, http, s3, webdav adapters + registry, plugins, filter, walk_files
│                           # (used by the ingester AND one-shot client ingestion)
├── app.py                  # HaikuRAGApp CLI application layer
├── cli.py                  # Typer CLI entry point
├── context.py              # Section-bounded context expansion, build_toc
├── mcp.py                  # FastMCP server (create_mcp_server)
├── doctor.py               # Database health checks, provider probes, near-duplicate detection
├── circuit_breaker.py      # CircuitBreaker (ingester pollers and workers, docling-serve provider)
├── hooks.py                # Hook, IngestRequest, IngestEvent, DeleteEvent, SearchRequest, build_hooks, notify
├── s3.py                   # Object-store helpers
├── telemetry.py            # Logfire configuration
├── logging.py              # Logging configuration
└── uri.py                  # is_local_uri, uri_to_path
tests/                      # Mirrors source structure
├── conftest.py             # Fixtures (see Testing), capture_logs, for_path, writing
├── cassettes/              # VCR cassettes for the central suites (see Testing)
├── json_body_serializer.py # Custom VCR serializer for JSON bodies
├── docker/                 # docker-compose.yml for integration services
├── client/ chunkers/ converters/ embeddings/ interfaces/ providers/ reranking/ store/
├── capabilities/ chat/ ingester/ multi_db/ sandbox/ sources/ utils/
└── data/                   # Test data files
evaluations/                # Benchmarking workspace
├── configs/                # Reference configs per dataset
├── scripts/
├── evaluations/
│   ├── benchmark.py        # Typer CLI: run, pair, download, upload
│   ├── population.py       # populate_db
│   ├── retrieval.py        # run_retrieval_benchmark
│   ├── qa.py               # run_qa_benchmark, run_live_qa_benchmark
│   ├── results.py          # Per-case result files
│   ├── pairing.py          # evaluations pair
│   ├── artifacts.py        # download_dataset_db, upload_dataset_db, HF_REPO_ID
│   ├── experiment.py       # build_experiment_metadata
│   ├── config.py           # DatasetSpec, DocumentPayload, RetrievalSample
│   ├── capability_runner.py # native capability evaluation runner
│   ├── datasets/           # enterprise_rag, frames, hotpotqa, mtrag, open_rag_bench, t2_ragbench
│   └── evaluators/         # judge, map, citation, retrieval, number_match, refusal, answer_facts, conversation, transcript
└── tests/                  # benchmark and capability-runner tests
docs/                       # Documentation (zensical, not mkdocs; nav lives in zensical.toml)
docker/                     # Dockerfile (full), Dockerfile.slim (published)
examples/                   # Working examples (docker/, custom agents)
plugins/haiku-rag/          # Claude Code / Codex plugin (see MCP Server)
scripts/                    # bump_version.py, build-docker-images.sh, run-integration-tests.sh
app/                        # Web application: chat, ingestion, inspector, doctor (see below)
```

## Architecture

**Key Classes:**

| Class | Location | Purpose |
|-------|----------|---------|
| `HaikuRAG` | client/client.py | High-level API (main entry point) |
| `HaikuRAGApp` | app.py | CLI application layer |
| `Store` | store/engine.py | Async LanceDB connection, tables, upgrades |
| `DatabaseScope` | client/scope.py | The databases an operation covers, resolved once |
| `SingleDatabaseSession` | client/session.py | One database: store, repositories, lifecycle. Reads and writes |
| `FederatedSession` | client/session.py | Several, composed lazily. Reads only |
| `DocumentRepository` | store/repositories/document.py | Document CRUD |
| `ChunkRepository` | store/repositories/chunk.py | Chunk CRUD + search |
| `DocumentItemRepository` | store/repositories/document_item.py | Document items, pictures |
| `SettingsRepository` | store/repositories/settings.py | The version and embedder a database records (`recorded_settings`) |

**Factory Functions:**
- `get_embedder(config)` → `EmbedderWrapper` (embeddings/__init__.py)
- `get_reranker(config)` → `RerankerBase | None` (reranking/__init__.py)
- `get_converter(config)` → `DocumentConverter` (converters/__init__.py)
- `get_chunker(config)` → `DocumentChunker` (chunkers/__init__.py)
- `create_capability(db_path?, config?, *, defer_loading=False, rag=None, request_limit=30, sources=None, vision=None)` → `RAGCapability` (capabilities/rag.py). Tools `search(query, limit?, sources?)`, `execute_code`, `cite`, state under `rag`.
  - `request_limit` counts model requests per agent run. At the limit the capability's tools are removed except `cite`, which survives `CITATION_GRACE_REQUESTS` (2) further requests that call one of the capability's tools. Requests spent on other tools do not count.
  - A spent search or code budget does not withdraw the tool: it keeps failing. Withdrawing a tool the model still calls costs the agent's unknown-tool retries and aborts the run.