AGENTS.md · bunkerity/bunkerweb

Sections

1. Project Structure & Module Organization
2. Build, Test, and Development Commands
3. Coding Style & Naming Conventions
4. Testing Guidelines
5. Commit & Pull Request Guidelines
6. Security & Configuration Tips
7. Project Overview
8. Architecture
9. Development Commands
10. Key Files
11. Important Patterns
12. External Resources

First lines (shown under the repo's AGPL-3.0 license)

# AGENTS.md

This file provides guidance to AI coding agents working in this repository.

## Project Structure & Module Organization

Source lives in `src/`: `api/` (FastAPI service), `ui/` (admin UI), `linux/` (distribution packages and service units), `bw/` and `common/` (core WAF logic), plus packaging targets like `all-in-one/` and other platform bundles. Integration assets and manifests are under `examples/`, while reusable configuration templates live in `env/`. MkDocs content for docs.bunkerweb.io sits in `docs/`. System tests, fixtures, and helper scripts are consolidated in `tests/`.

## Build, Test, and Development Commands

Bootstrap Python deps per component, e.g. `pip install -r src/api/requirements.txt` or `pip install -r src/ui/requirements.txt`. Build a full appliance image with `docker build -f src/all-in-one/Dockerfile .`. Exercise integrations locally via `python3 tests/main.py docker`; swap `docker` for `linux`, `autoconf`, `swarm`, or `kubernetes` as needed (set the matching `TEST_DOMAIN*` env vars first). Regenerate docs with `mkdocs serve --watch` from the repo root. Run `pre-commit run --all-files` before pushing to execute the standard formatters and linters.

## Coding Style & Naming Conventions

Python uses Black (160 char lines) and Flake8 with ignores defined in `.pre-commit-config.yaml`; prefer snake_case for modules and functions, PascalCase for classes. Lua code is formatted with StyLua (see `stylua.toml`) and linted with Luacheck; follow lowercase module names and descriptive function names. Shell scripts must pass ShellCheck and stay POSIX-compatible unless a `#!/bin/bash` shebang is explicit. Front-end assets follow Prettier defaults.

## Testing Guidelines

High-level acceptance suites live in `tests/` and orchestrate Dockerized environments—verify Docker access and required `TEST_DOMAIN*` env vars before running. Add scenario files under `examples/<use-case>/tests.json` with descriptive names; tests should assert observable behavior rather than internals. For unit-style Python additions, provide lightweight checks inside the relevant module and hook them into integration flows when feasible. Capture regressions by replicating failing requests in the automated suites.

## Commit & Pull Request Guidelines

Use concise, present-tense messages; the history favors Conventional Commits (`feat:`, `fix:`, `docs:`) or `<component> - …` prefixes. Reference issue IDs when closing or relating tickets. Each PR should include a summary of changes, validation steps (commands or screenshots for UI changes), and updated docs or config when behavior shifts. Coordinate breaking changes with maintainers and flag them clearly in both commit body and PR description.

## Security & Configuration Tips

Never commit secrets—use sample files in `env/` or add new templates when introducing config. Review `.gitleaksignore` before adjusting dependencies. Docker Scout is used for container image vulnerability scanning in CI/CD—check the `container-build.yml` workflow for current scan configuration. When touching TLS, keys, or rule bundles, document rotation steps and default hardening in the accompanying docs update.

## Project Overview

BunkerWeb is an open-source Web Application Firewall (WAF) built on NGINX with a modular plugin architecture. It provides "security by default" for web services through multiple integration modes (Docker, Kubernetes, Swarm, Linux) and is fully configurable via environment variables.

## Architecture

### Core Components

- **BunkerWeb Core** (`src/bw/`, `src/common/core/`): NGINX-based reverse proxy with security modules in Lua (request-time) and Python (jobs). Entry point: `src/bw/lua/bunkerweb.lua`.
- **Scheduler** (`src/scheduler/`): Central orchestrator ("brain"). `main.py` runs the main loop; `JobScheduler.py` manages job execution with thread pools. Uses Python's `schedule` library.
- **Autoconf** (`src/autoconf/`): Listens for Docker/Swarm/Kubernetes events and dynamically reconfigures BunkerWeb.
- **API** (`src/api/`): FastAPI service with router-based architecture (`src/api/app/routers/` — auth, instances, services, configs, plugins, jobs). IP whitelist and rate limiting support.
- **Web UI** (`src/ui/`): Flask app using Blueprints for routing (`src/ui/app/routes/` — configs, plugins, jobs, logs, instances, profile, etc.). Uses Flask-Login for auth and Jinja2 templates. `dependencies.py` is the central dependency injection point providing DB, DATA, BW_CONFIG, BW_INSTANCES_UTILS.
- **Database** (`src/common/db/`): SQLAlchemy ORM with `model.py` defining all tables (Plugins, Settings, Services, Jobs, Custom_configs, Users, etc.). `Database.py` wraps high-level query methods. Supports SQLite (WAL mode), MariaDB, MySQL, PostgreSQL with QueuePool for connection pooling.

### Configuration Flow

1. Settings defined as environment variables (e.g., `USE_ANTIBOT=captcha`, `AUTO_LETS_ENCRYPT=yes`)
2. Scheduler reads settings from environment or database
3. **Configurator** (`src/common/gen/Configurator.py`) validates settings against `plugin.json` schemas with pre-compiled regex caches
4. **Templator** (`src/common/gen/Templator.py`) renders NGINX configs from Jinja2 templates (`src/common/confs/`) using ProcessPoolExecutor for parallel rendering
5. BunkerWeb instances reload with new configuration
6. In multisite mode, prefix settings with server name: `www.example.com_USE_ANTIBOT=captcha`
7. Multiple settings use numeric suffixes: `REVERSE_PROXY_URL_1=/api`, `REVERSE_PROXY_HOST_1=http://backend1`

### Plugin System

Each core module in `src/common/core/*/` contains:

- `plugin.json`: Metadata with settings schema (id, name, version, stream, settings with context/type/regex/default, jobs array with schedule/reload/async flags)
- `jobs/` folder: Python scripts for periodic tasks (e.g., downloading blocklists). Jobs specify `every` (once/minute/hour/day/week) and `reload` flag.
- Lua code for request-time processing
- `confs/` folder: NGINX configuration templates

External plugins follow the same structure.

### Lua Request Processing Pipeline

The Lua runtime (`src/bw/lua/`) processes requests through plugin hooks at NGINX phases: access, header_filter, body_filter, log. Key files:

- `plugin.lua`: Plugin loader and execution across phases
- `ctx.lua`: Per-request context management
- `datastore.lua`: Shared data persistence (shared dict backed)
- `cachestore.lua`: Request-level caching
- `clusterstore.lua`: Cluster-aware storage (Redis)

### Core Plugins (src/common/core/)

42 plugins organized by function:

- **Auth**: antibot (CAPTCHA), authbasic, mtls, crowdsec
- **Threat Detection**: modsecurity (OWASP WAF), badbehavior, dnsbl, reversescan
- **Access Control**: whitelist, blacklist, greylist, country, limit (rate limiting)
- **SSL/TLS**: ssl, letsencrypt, customcert, selfsigned
- **Proxy & Routing**: reverseproxy, realip, redirect, grpc, php
- **Performance**: gzip, brotli, clientcache, redis
- **Headers & Content**: headers, cors, inject, robotstxt, securitytxt
- **Management**: sessions, metrics, backup, templates, bunkernet, ui, db, jobs

### Shared Utilities (src/common/utils/)

- `common_utils.py`: Docker secrets handling, hashing, version info, integration detection
- `logger.py`: Logging with syslog support
- `jobs.py`: Job helpers (atomic writes, file hashing, tar operations)
- `ApiCaller.py`: HTTP client for inter-component API calls

## Development Commands

### Setup

```bash
pip install -r src/scheduler/requirements.txt
pip install -r src/ui/requirements.txt
pip install -r src/api/requirements.txt
pre-commit install
```

### Build

```bash
# Build Docker image (all-in-one)
docker build -f src/all-in-one/Dockerfile -t bunkerweb:dev .

# Build specific component
docker build -f src/scheduler/Dockerfile -t bunkerweb-scheduler:dev .
docker build -f src/ui/Dockerfile -t bunkerweb-ui:dev .
```

### Linting & Formatting

```bash
# Run all pre-commit hooks
pre-commit run --all-files

# Individual tools
black .                          # Python formatting (160 char lines)
flake8 .                         # Python linting (ignores E266,E402,E501,E722,W503)
stylua .                         # Lua formatting
luacheck src/                    # Lua linting (--std min)
shellcheck scripts/*.sh          # Shell script linting
prettier --write "**/*.{js,ts,css,html,json,yaml,md}"  # Frontend formatting
codespell                        # Spell checking
refurb                           # Python refactoring suggestions (excludes tests/)
```

### Run Development Instance

```bash
# Full stack with UI + API (recommended)
docker compose -f misc/dev/docker-compose.ui.api.yml up -d
```

Dev compose files in `misc/dev/`:

- `docker-compose.ui.api.yml` — Full stack (UI + API + core + MariaDB) — **recommended**
- `docker-compose.ui.yml` — UI only (no API)
- `docker-compose.all-in-one.yml` — Single container with all components
- `docker-compose.autoconf.yml` — Docker autoconf mode
- `docker-compose.wizard.yml` — Setup wizard

Dev credentials: UI `admin`/`P@ssw0rd`, API `admin`/`P@ssw0rd`, DB `bunkerweb`/`secret`.

The dev compose mounts `src/ui/app/` and `src/api/app/` as read-only volumes, so UI and API code changes apply without rebuilding (restart the container to pick up changes).

### Database Migrations

```bash
# Alembic migrations in src/common/db/alembic/
# Separate version directories per DB type: mariadb_versions, mysql_versions, postgresql_versions, sqlite_versions
# Migration scripts also in src/common/db/alembic/
```