CLAUDE.md · topoteretes/cognee

Sections

1. Project Overview
2. Development Commands
3. Architecture Overview
4. Important Configuration
5. Extension Points
6. Branching Strategy
7. Code Style
8. Commit & PR Title Style
9. Testing Strategy
10. API Structure
11. Python SDK Entry Points
12. Security Considerations
13. Common Patterns
14. Key Concepts
15. Debugging & Troubleshooting
16. Resources

First lines (shown under the repo's Apache-2.0 license)

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

Cognee is an open-source AI memory platform that transforms raw data into persistent knowledge graphs for AI agents. It replaces traditional RAG (Retrieval-Augmented Generation) with an ECL (Extract, Cognify, Load) pipeline combining vector search, graph databases, and LLM-powered entity extraction.

**Requirements**: Python 3.10 - 3.14

## Design Principles

1. **Don't overengineer**: Simple beats complex.
2. **No fallbacks**: One correct path, no alternatives.
3. **One way**: One way to do things, not many.
4. **Throw errors**: Fail fast when the operation's preconditions aren't met. Observability (telemetry, logging) is best-effort and never changes the outcome of the operation.
5. **No backups**: Trust the primary mechanism.
6. **Separation of concerns**: Each function has a single responsibility.
7. **Be concise but informative**: No essays, but the docstrings and comments a reader needs.
8. **No invented limits**: Never add a cutoff (`max_n`, `[:k]`, "older than X") or an expiry (TTL, cooldown, eviction) the task did not ask for — both are silent data loss. If you believe one is unavoidable, stop and ask, stating what would be dropped and why; never add it first and justify it afterwards. Existing bounds (lock leases, recovery floors, batch caps) are deliberate: removing or changing one is the same question.

**Methodology**: surgical changes only; evidence-based debugging (minimal, targeted logging); fix root causes, not symptoms. Debug like a detective: form the theory of the crime, collect evidence, and change code only once the evidence proves it.

## Development Commands

### Setup
```bash
# Create virtual environment (recommended: uv)
uv venv && source .venv/bin/activate

# Install with pip or uv
uv pip install -e .

# Install with dev dependencies
uv pip install -e ".[dev]"

# Install with specific extras
uv pip install -e ".[postgres,neo4j,docs]"

# Set up pre-commit hooks
pre-commit install
```

### Available Installation Extras
- **postgres** / **postgres-binary** - PostgreSQL + PGVector support (also enables the Postgres session-cache backend, `CACHE_BACKEND=postgres`)
- **neo4j** - Neo4j graph database support
- **neptune** - AWS Neptune support
- **turso** - Turso (Rust rewrite of SQLite, `pyturso`) as relational, graph-as-tables and/or vector store; local files only
- **docs** - Document processing (unstructured library)
- **scraping** - Web scraping (Tavily, BeautifulSoup, Playwright; Keenable needs no extra — it uses the built-in httpx)
- **langchain** - LangChain integration
- **llama-index** - LlamaIndex integration
- **anthropic** - Anthropic Claude models
- **ollama** - Ollama local models
- **mistral** - Mistral AI models
- **groq** - Groq API support
- **llama-cpp** - Llama.cpp local inference
- **huggingface** - HuggingFace transformers
- **aws** - S3 storage backend
- **redis** - Redis caching
- **baml** - BAML structured output
- **dlt** - Data load tool (dlt) integration
- **docling** - Docling document processing, slim profile without torch (office/HTML/email/markdown/LaTeX formats)
- **docling-full** - Full docling install with torch-based ML models (adds PDF/image conversion through docling)
- **codegraph** - Compatibility alias; code graph extraction and Enola are included by default
- **gliner** - GLiNER demo runtime (gliner2 + torch from PyPI — the CUDA build on Linux, for GPU hosts). Optional: without it the first GLiNER cognify installs CPU-only torch plus this extra itself (see "LLM-free Graph Extraction")
- **evals** - Evaluation tools
- **deepeval** - DeepEval testing framework
- **posthog** - PostHog analytics
- **tracing** - OpenTelemetry tracing
- **dev** - All development tools (pytest, ty, ruff, etc.)
- **debug** - Debugpy for debugging

### Testing
```bash
# Run all tests
pytest

# Run with coverage
pytest --cov=cognee --cov-report=html

# Run specific test file
pytest cognee/tests/test_custom_model.py

# Run specific test function
pytest cognee/tests/test_custom_model.py::test_function_name

# Run async tests
pytest -v cognee/tests/integration/

# Run unit tests only
pytest cognee/tests/unit/

# Run integration tests only
pytest cognee/tests/integration/
```

### Code Quality
```bash
# Run ruff linter
ruff check .

# Run ruff formatter
ruff format .

# Run both linting and formatting (pre-commit)
pre-commit run --all-files

# Type checking with ty
ty check .
```

### Running Cognee
```bash
# Using Python SDK
uv run python examples/guides/simple_cognee_example.py

# Using CLI (memory API — the primary surface)
cognee-cli remember "Your text here"   # also accepts file paths / URLs
cognee-cli recall "Your question"
cognee-cli improve -d my_project       # enrich/index the graph
cognee-cli forget --all                # NOTE: no confirmation prompt

# Low level operations (still ship; what the memory commands call underneath)
cognee-cli add "Your text here" && cognee-cli cognify
cognee-cli search "Your query"
cognee-cli delete --all                # prompts before deleting

# Launch full stack with UI
cognee-cli -ui
```

## Architecture Overview

### Core Workflow: remember → recall (+ improve / forget)

As of cognee 1.x the memory API is the primary surface. All functions are async.

1. **remember()** - Store data in memory. Without `session_id` it runs `add()` + `cognify()` and then `improve()` (`self_improvement=True` by default); with `session_id` it writes to the fast session cache and bridges into the graph in the background.
2. **recall()** - Query memory. Auto-routes to a search strategy unless `query_type` is passed (`auto_route=False` falls back to `HYBRID_COMPLETION`). A `session_id` reads the session cache first and falls through to the graph.
3. **improve()** - Run the self-improvement stages over a dataset: with `session_ids`, bridge session Q&A, agent traces, distilled learnings and user preferences into the permanent graph and apply feedback weights; always, conflict review (on by default, skipped without an LLM) and global context indexing (on by default, `IMPROVE_GLOBAL_CONTEXT_INDEX`). Returns an `ImproveResult` with one `StageResult` per stage (see "IMPROVE: the orchestrator" below).
4. **forget()** - Unified deletion (`data_id` / `dataset` / `dataset_id` / `everything=True`, plus `memory_only=True` to drop graph+vectors but keep raw files, and `clean_history=True` to also delete the caller's own search/recall history; a search that finds nothing after `memory_only` does not mean the data is gone). Every mode removes records and memory, not bytes: LanceDB keeps deleted rows in its fragments until compaction, which rewrites a fragment only once enough of it is deleted, and the embedded graph file has no user-facing compaction, so a byte-level erasure guarantee needs the engine's own compaction/vacuum or a new store.
   One session on its own: `cognee.session.delete_session(session_id)`, `DELETE /api/v1/sessions/{session_id}` or `cognee-cli sessions delete <id>` remove its Q&A turns, agent traces, session context and improve() watermarks, its Q&A vectors, and its `session_records` / `session_model_usage` rows — the caller's own sessions only. Lessons already distilled from it into a dataset are documents and stay (`forget(data_id=...)`). It raises with `CACHING=false`.

#### Low level operations: add → cognify → search

These still ship and are what the memory API calls underneath. Reach for them to drive one stage in isolation (custom pipeline tasks, stage-level debugging), not for ordinary ingestion or retrieval.

1. **add()** - Ingest data (files, URLs, text) into datasets
2. **cognify()** - Extract entities/relationships and build knowledge graph
3. **search()** - Query knowledge using various retrieval strategies

Note: Using Low level operations over core is useful in the following contexts.
1) Some add()/cognify() options are not accepted by remember(): it routes kwargs through a fixed allow-list (`_ADD_ONLY` / `_COGNIFY_ONLY` / `_SHARED` in `cognee/api/v1/remember/remember.py`) and raises `TypeError: Unexpected keyword arguments` for anything else. Unreachable today: cognify's `chunk_attachment`, and `ontology_file_path` (remember still takes an ontology via `config={"ontology_config": {"ontology_resolver": RDFLibOntologyResolver(ontology_file=...)}}` or `ONTOLOGY_FILE_PATH`); add's `skip_connection_test`; the DLT option `column_value_columns`; and extra cognify `**kwargs` forwarded to the extraction tasks. (`add()` accepts but ignores the web-scraping options `tavily_config` / `soup_crawler_config` / `extraction_rules`; `extraction_rules` works from both via `preferred_loaders={"beautiful_soup_loader": {"extraction_rules": ...}}`, with the `scraping` extra installed; without it the rules are silently ignored.)
2) remember() hardcodes datasets_arg = [dataset_name]: always exactly one. Use cognify for this: cognify(datasets=["a","b","c"]) or datasets=None (every dataset the user owns.)
3) remember() always runs add() first. To rebuild a graph over data already in the DB — after forget(memory_only=True), or with a new graph_model/ontology, cognify() is the only path.
4) add() is like a staging area for cognify(). But remember automatically adds every time.
5) search() packs skills/tools/max_iter/code_query into retriever_specific_config for you. Using recall() you hand-build that dict yourself.
6) prune.prune_system(metadata=True) drops the relational DB (users, tenants, ACLs, the dataset_database registry, pipeline runs, search history.) forget() touches none of that. Full test teardown is prune's job.