AGENTS.md · yusufkaraaslan/Skill_Seekers

Sections

1. Project Overview
2. Setup
3. Build / Test / Lint
4. Code Style
5. Project Layout
6. Key Patterns
7. Web UI (Seeker HUD) — beta
8. CLI Commands
9. Testing Instructions
10. Git Workflow
11. Pre-commit Checklist
12. CI/CD
13. Deployment
14. Security Considerations
15. Additional Resources

First lines (shown under the repo's MIT license)

# AGENTS.md - Skill Seekers

Comprehensive reference for AI coding agents. Skill Seekers is a Python CLI tool (v3.6.0) that converts documentation sites, GitHub repos, PDFs, videos, notebooks, wikis, and more into AI-ready skills for 21+ LLM platforms and RAG pipelines.

## Project Overview

**Skill Seekers** is a universal preprocessing layer that transforms raw documentation and code into structured knowledge assets. It supports 17+ source types and exports to 21+ AI platforms including Claude, Gemini, OpenAI, LangChain, LlamaIndex, and various vector databases.

### Key Capabilities
- **Source Types (17):** Documentation websites, GitHub repos, PDFs, Word docs, EPUBs, videos, local codebases, Jupyter notebooks, HTML, OpenAPI specs, AsciiDoc, PowerPoint, Confluence, Notion, RSS feeds, man pages, chat exports
- **Export Targets (21):** Claude, Gemini, OpenAI, MiniMax, OpenCode, Kimi, DeepSeek, Qwen, OpenRouter, Together AI, Fireworks AI, Markdown, LangChain, LlamaIndex, Haystack, Weaviate, ChromaDB, FAISS, Qdrant, Pinecone
- **MCP Server:** FastMCP-based Model Context Protocol server for AI assistant integration

## Setup

```bash
# REQUIRED before running tests (src/ layout — tests hard-exit if package not installed)
pip install -e .

# With dev tools (pytest, ruff, mypy, coverage)
pip install -e ".[dev]"

# With specific LLM platform support
pip install -e ".[gemini]"      # Google Gemini
pip install -e ".[openai]"      # OpenAI ChatGPT
pip install -e ".[all-llms]"    # All LLM platforms

# With all optional dependencies (except video-full)
pip install -e ".[all]"

# Full video processing (heavy dependencies)
pip install -e ".[video-full]"
```

Note: `tests/conftest.py` checks that `skill_seekers` is importable and calls `sys.exit(1)` if not. Always install in editable mode first.

### Environment Variables

Create a `.env` file or export these variables:
```bash
ANTHROPIC_API_KEY      # For Claude AI enhancement
GOOGLE_API_KEY         # For Gemini support
OPENAI_API_KEY         # For OpenAI support
GITHUB_TOKEN           # For GitHub repo scraping (higher rate limits)
```

## Build / Test / Lint

```bash
# Full suite (never skip — all must pass)
pytest tests/ -v

# Fast iteration (skip slow, integration, E2E, network, MCP)
pytest tests/ -m "not slow and not integration and not e2e and not network and not serial and not mcp_only" -q

# Fast parallel (install pytest-xdist first)
pytest tests/ -n auto --dist=loadfile -m "not slow and not integration and not e2e and not network and not serial and not mcp_only" -q

# 3-phase runner script (recommended for local dev)
bash scripts/run_tests_fast.sh

# Single test
pytest tests/test_scraper_features.py::test_detect_language -v

# Skip slow/integration
pytest tests/ -v -m "not slow and not integration"

# With coverage
pytest tests/ --cov=src/skill_seekers --cov-report=term

# Lint + format check (matches CI)
ruff check src/ tests/
ruff format --check src/ tests/

# Type check (non-blocking — mypy is continue-on-error in CI)
mypy src/skill_seekers --show-error-codes --pretty
```

**Pytest config:** `asyncio_mode = "auto"`, so `@pytest.mark.asyncio` is implicit. Test markers: `slow`, `integration`, `e2e`, `venv`, `bootstrap`, `benchmark`, `asyncio`, `serial`, `network`, `mcp_only`.

**CI note:** CI pins `ruff==0.15.8` (not the `>=0.14.13` dev dep). If formatting behaves differently locally, check the CI version.

**CI test phases:** Tests are split into 3 parallel jobs:
- `test-fast` — 3386 unit tests with xdist across OS/Python matrix
- `test-serial` — 69 serial/integration/E2E/network tests
- `test-mcp` — 193 MCP tests (requires `[mcp]` extras)

## Code Style

### Formatting Rules (ruff — from pyproject.toml)
- **Line length:** 100 characters
- **Target Python:** 3.10+
- **Enabled lint rules:** E, W, F, I, B, C4, UP, ARG, SIM
- **Ignored rules:** E501 (line length handled by formatter), F541 (f-string style), ARG002 (unused method args for interface compliance), B007 (intentional unused loop vars), I001 (formatter handles imports), SIM114 (readability preference)

### Imports
- Sort with isort (via ruff); `skill_seekers` is first-party
- Standard library → third-party → first-party, separated by blank lines
- Use `from __future__ import annotations` only if needed for forward refs
- Guard optional imports with try/except ImportError (see `adaptors/__init__.py` pattern):
  ```python
  try:
      from .claude import ClaudeAdaptor
      from .minimax import MiniMaxAdaptor
  except ImportError:
      ClaudeAdaptor = None
      MiniMaxAdaptor = None
  ```

### Naming Conventions
- **Files:** `snake_case.py` (e.g., `source_detector.py`, `config_validator.py`)
- **Classes:** `PascalCase` (e.g., `SkillAdaptor`, `ClaudeAdaptor`, `SourceDetector`)
- **Functions/methods:** `snake_case` (e.g., `get_adaptor()`, `detect_language()`)
- **Constants:** `UPPER_CASE` (e.g., `ADAPTORS`, `DEFAULT_CHUNK_TOKENS`, `VALID_SOURCE_TYPES`)
- **Private:** prefix with `_` (e.g., `_read_existing_content()`, `_validate_unified()`)

### Type Hints
- Gradual typing — add hints where practical, not enforced everywhere
- Use modern syntax: `str | None` not `Optional[str]`, `list[str]` not `List[str]`
- MyPy config: `disallow_untyped_defs = false`, `check_untyped_defs = true`, `ignore_missing_imports = true`
- Tests are excluded from strict type checking (`disallow_untyped_defs = false`, `check_untyped_defs = false` for `tests.*`)

### Docstrings
- Module-level docstring on every file (triple-quoted, describes purpose)
- Google-style docstrings for public functions/classes
- Include `Args:`, `Returns:`, `Raises:` sections where useful

### Error Handling
- Use specific exceptions, never bare `except:`
- Provide helpful error messages with context
- Use `raise ValueError(...)` for invalid arguments, `raise RuntimeError(...)` for state errors
- Guard optional dependency imports with try/except and give clear install instructions on failure
- Chain exceptions with `raise ... from e` when wrapping

### Suppressing Lint Warnings
- Use inline `# noqa: XXXX` comments (e.g., `# noqa: F401` for re-exports, `# noqa: ARG001` for required but unused params)

## Project Layout

```
src/skill_seekers/           # Main package (src/ layout)
  cli/                       # CLI commands and entry points (100+ files)
    adaptors/                # Platform adaptors (Strategy pattern, inherit SkillAdaptor)
    arguments/               # CLI argument definitions (one per source type)
    parsers/                 # Subcommand parsers (one per source type)
    storage/                 # Cloud storage (inherit BaseStorageAdaptor)
    main.py                  # Unified CLI entry point (COMMAND_MODULES dict)
    source_detector.py       # Auto-detects source type from user input
    create_command.py        # Unified `create` command routing
    config_validator.py      # VALID_SOURCE_TYPES set + per-type validation
    unified_scraper.py       # Multi-source orchestrator (scraped_data + dispatch)
    unified_skill_builder.py # Pairwise synthesis + generic merge
  mcp/                       # MCP server (FastMCP + legacy)
    tools/                   # MCP tool implementations by category (10 files)
    server_fastmcp.py        # FastMCP server implementation
    server_legacy.py         # Legacy MCP server
  sync/                      # Sync monitoring (Pydantic models)
  benchmark/                 # Benchmarking framework
  embedding/                 # FastAPI embedding server
  workflows/                 # 67 YAML workflow presets