CLAUDE.md · HolmesGPT/holmesgpt

Sections

1. Repository Overview
2. Development Commands
3. Architecture Overview
4. Testing Framework
5. Configuration
6. Development Guidelines
7. Debugging CLI / Rich Live Display Issues
8. Investigating Eval Regressions / Holmes Behavior Changes
9. Security Notes
10. Eval Tests (LLM Evaluations)
11. 🚨 CRITICAL: Always Test Your Changes
12. Shared Infrastructure Pattern
13. Setup Verification Best Practices
14. Running Kubernetes Evals in the Claude Code Sandbox
15. Reading CodeRabbit Review Comments
16. Documentation Lookup
17. MkDocs Navigation
18. Updating References When Docs Pages Change
19. MkDocs Formatting Notes
20. Robusta Platform/API URLs in Docs

First lines (shown under the repo's Apache-2.0 license)

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Repository Overview

HolmesGPT is an AI-powered troubleshooting agent that connects to observability platforms (Kubernetes, Prometheus, Grafana, etc.) to automatically diagnose and analyze infrastructure and application issues. It uses an agentic loop to investigate problems by calling tools to gather data from multiple sources.

## Development Commands

### Environment Setup
```bash
# Install dependencies with Poetry
poetry install
```

**Use Poetry 1.8.5 to regenerate `poetry.lock`.** The committed lockfile is
`lock-version = "2.0"`, which Poetry 1.8.x produces. Running `poetry lock` under
Poetry 2.x rewrites the whole file to `lock-version = "2.1"` and produces a huge
diff of unrelated churn. Always lock with 1.8.5, and prefer `--no-update` so only
your dependency change appears in the diff:

```bash
poetry lock --no-update   # with Poetry 1.8.5
poetry check --lock       # verify the lockfile matches pyproject.toml
```

If your system Poetry is 2.x, install 1.8.5 into a throwaway venv and use it just
for locking:

```bash
python3 -m venv /tmp/poetry185 && /tmp/poetry185/bin/pip install "poetry==1.8.5"
/tmp/poetry185/bin/poetry lock --no-update
```

### Testing

```bash
# Install test dependencies with Poetry
poetry install --with dev
```

```bash
# Run all non-LLM tests (unit and integration tests)
make test-without-llm
poetry run pytest tests -m "not llm"

# Run LLM evaluation tests (requires API keys)
make test-llm-ask-holmes          # Test single-question interactions
make test-llm-investigate         # Test AlertManager investigations
poetry run pytest tests/llm/ -n 6 -vv  # Run all LLM tests in parallel

# Run pre-commit checks (includes ruff, mypy, poetry validation)
# NOTE: Only run these when the user explicitly asks. They run in CI automatically.
make check
poetry run pre-commit run -a
```

### Code Quality (only run when explicitly asked)
```bash
# Format code with ruff
poetry run ruff format

# Check code with ruff (auto-fix issues)
poetry run ruff check --fix

# Type checking with mypy
poetry run mypy
```

## Architecture Overview

### Core Components

**CLI Entry Point** (`holmes/main.py`):
- Typer-based CLI with subcommands for `ask`, `investigate`, `toolset`
- Handles configuration loading, logging setup, and command routing

** Interactive mode for CLI** (`holmes/interactive.py`):
- Handles interactive mode for `ask` subcommand
- Implements slash commands

**Configuration System** (`holmes/config.py`):
- Loads settings from `~/.holmes/config.yaml` or via CLI options
- Manages API keys, model selection, and toolset configurations
- Factory methods for creating sources (AlertManager, Jira, PagerDuty, etc.)

**Core Investigation Engine** (`holmes/core/`):
- `tool_calling_llm.py`: Main LLM interaction with tool calling capabilities
- `investigation.py`: Orchestrates multi-step investigations with runbooks
- `toolset_manager.py`: Manages available tools and their configurations
- `tools.py`: Tool definitions and execution logic

**Plugin System** (`holmes/plugins/`):
- **Sources**: AlertManager, Jira, PagerDuty, OpsGenie integrations
- **Toolsets**: Kubernetes, Prometheus, Grafana, AWS, Docker, etc.
- **Prompts**: Jinja2 templates for different investigation scenarios
- **Destinations**: Slack integration for sending results

### Key Patterns

**Toolset Architecture**:
- Each toolset is a YAML file defining available tools and their parameters
- Tools can be Python functions or bash commands with safety validation
- Toolsets are loaded dynamically and can be customized via config files
- **Important**: All toolsets MUST return detailed error messages from underlying APIs to enable LLM self-correction
  - Include the exact query/command that was executed
  - Include time ranges, parameters, and filters used
  - Include the full API error response (status code and message)
  - For "no data" responses, specify what was searched and where

**Thin API Wrapper Pattern for Python Toolsets**:
- Reference implementation: `servicenow_tables/servicenow_tables.py`
- Use `requests` library for HTTP calls (not specialized client libraries like `opensearchpy`)
- Simple config class with Pydantic validation
- Health check in `prerequisites_callable()` method
- Each tool is a thin wrapper around a single API endpoint

**Server-Side Filtering is Critical**:
- **Never return unbounded data from APIs** - this causes token overflow
- Always include filter parameters on tools that query collections (e.g., `index` parameter for Elasticsearch _cat APIs)
- Example problem: `opensearch_list_shards` returned ALL shards → 25K+ tokens on large clusters
- Example fix: `elasticsearch_cat` tool requires `index` parameter for shards/segments endpoints
- When server-side filtering is not possible, use `JsonFilterMixin` (see `json_filter_mixin.py`) to add `max_depth` and `jq` parameters for client-side filtering

**Toolset Config Backwards Compatibility**:
When renaming config fields in a toolset, maintain backwards compatibility using Pydantic's `extra="allow"`:

```python
# ✅ DO: Use extra="allow" to accept deprecated fields without polluting schema
class MyToolsetConfig(BaseModel):
    model_config = ConfigDict(extra="allow")

    # Only define current field names in schema
    new_field_name: int = 10

    @model_validator(mode="after")
    def handle_deprecated_fields(self):
        extra = self.model_extra or {}
        deprecated = []

        # Map old names to new names
        if "old_field_name" in extra:
            self.new_field_name = extra["old_field_name"]
            deprecated.append("old_field_name -> new_field_name")

        if deprecated:
            logging.warning(f"Deprecated config names: {', '.join(deprecated)}")
        return self

# ❌ DON'T: Define deprecated fields in schema with Optional[None]
class BadConfig(BaseModel):
    new_field_name: int = 10
    old_field_name: Optional[int] = None  # Pollutes schema, shows in model_dump()
```

Benefits of `extra="allow"` approach:
- Schema only shows current field names
- `model_dump()` returns clean output without deprecated fields
- Old configs still work (backwards compatible)