AGENTS.md · livekit/agents-js

Sections

1. Project Overview
2. Monorepo Structure
3. Common Commands
4. Architecture
5. Code Conventions
6. Testing
7. Porting from Python (livekit-agents)
8. CI Requirements
9. LiveKit documentation
10. LiveKit agent skills

First lines (shown under the repo's Apache-2.0 license)

<!--
SPDX-FileCopyrightText: 2026 LiveKit, Inc.

SPDX-License-Identifier: Apache-2.0
-->

# AGENTS.md

## Project Overview

LiveKit Agents for Node.js — a TypeScript framework for building realtime, multimodal, and voice AI agents that run on servers. This is the Node.js distribution of the [LiveKit Agents framework](https://github.com/livekit/agents) (originally Python).

## Monorepo Structure

- **`agents/`** — Core framework (`@livekit/agents`). Agent orchestration, LLM/STT/TTS abstractions, voice pipeline, metrics, telemetry, IPC/process pooling, and the CLI.
- **`plugins/`** — Provider plugins (`@livekit/agents-plugin-*`). Each implements one or more of: LLM, STT, TTS, VAD, EOU (end-of-utterance), Realtime, or Avatar.
- **`examples/`** — Example agents (private, not published). Run with `pnpm build && node ./examples/src/<file>.ts dev`.
- **`tests/e2e/`** — End-to-end tests via Docker (separate from unit tests co-located in each package).

**Tooling:** pnpm 11 workspaces, Turborepo for builds, tsup for bundling (CJS + ESM), TypeScript 5.4+, Vitest for tests, Changesets for versioning.

## Common Commands

```bash
pnpm build                  # Build all packages (turbo)
pnpm build:agents           # Build only @livekit/agents and its deps
pnpm build:plugins          # Build only plugins and their deps
pnpm clean:build            # Clean dist/ dirs then rebuild
pnpm test                   # Run all tests (vitest)
pnpm test agents/src/llm            # Run tests by path
pnpm test -t "chat context"         # Run tests by name
pnpm test:watch             # Watch mode
pnpm lint                   # ESLint all packages
pnpm lint:fix               # ESLint with auto-fix
pnpm format:check           # Prettier check
pnpm format:write           # Prettier format
pnpm api:check              # API Extractor validation
pnpm api:update             # Update API declarations
```

### Running an example agent

```bash
pnpm build && node ./examples/src/basic_agent.ts dev --log-level=debug
```

Required env vars: `LIVEKIT_URL`, `LIVEKIT_API_KEY`, `LIVEKIT_API_SECRET`, plus provider keys (e.g. `OPENAI_API_KEY`).

> **Rule (must follow before handing off to the user to run any example/agent file):**
> Always run `pnpm build` first and iterate on TypeScript / dependency errors until it exits 0. Never tell the user to `node ./examples/src/<file>.ts` (or any `.ts` entrypoint) until the build succeeds — this includes after editing example files, after modifying framework code that breaks `tsc`, and after copy-pasting reproductions from external sources. If the build fails, fix the errors yourself (add missing deps, fix outdated API usage, update types) before asking the user to run anything.

### Pre-downloading plugin assets (for Docker builds)

`@livekit/agents` ships a `livekit-agents` bin with a `download-files` subcommand that discovers installed `@livekit/agents-plugin-*` packages from `node_modules` and runs each plugin's `downloadFiles()` (e.g. HuggingFace models for the turn detector) — without loading the user's agent code. Use it in Dockerfiles to separate dependency installation from app code so the download layer can be cached:

```dockerfile
COPY package.json pnpm-lock.yaml ./
RUN pnpm install --frozen-lockfile
RUN npx livekit-agents download-files
COPY src/ ./src
RUN pnpm build
```

The `download-files` subcommand on `cli.runApp()` is deprecated for this purpose since it requires loading the agent file first.

### Debugging individual plugins

Create a test file prefixed with `test_` in `examples/src/`. No `defineAgent` wrapper needed — just import the plugin directly and run:

```bash
pnpm build && node ./examples/src/test_my_plugin.ts
```

## Architecture

### Voice Pipeline (`agents/src/voice/`)

The core of the framework. Pipeline flow: **Audio In → VAD → STT → LLM → TTS → Audio Out**.

Key classes and their roles:

- **`Agent`** — Base class holding instructions, tools, and model config. Subclass to override pipeline hooks (`ttsNode`, `realtimeAudioOutputNode`, `sttNode`, `llmNode`). Use `voice.Agent.default.<hook>(this, ...)` to call the base implementation from overrides.
- **`AgentTask`** — Extends Agent for composable task workflows with isolated chat context and tool scoping.
- **`AgentSession`** — Orchestrates the full session lifecycle: connects to LiveKit room, manages turn detection, handles interruptions, collects metrics. Entry point: `session.start({ agent, room })`.
- **`AgentActivity`** (`agent_activity.ts`, ~100KB) — Complex state machine managing individual turns: VAD trigger → STT → endpointing → LLM inference → TTS generation → playout. Supports preemptive generation (starts LLM while user still speaking).
- **`SpeechHandle`** — Represents a unit of agent speech with lifecycle tracking and priority levels (`LOW=0`, `NORMAL=5`, `HIGH=10`). Sources: `'say'`, `'generate_reply'`, `'tool_response'`.
- **`AvatarSession`** — Base class for avatar plugin sessions. Registers `aclose()` as a job shutdown callback and warns when started after `AgentSession.start()` (which would replace the existing audio output). Plugin implementations extend this class and call `super.start(agentSession, room)` first.

Turn detection modes: `"stt"` | `"vad"` | `"realtime_llm"` | `"manual"` (configured in `turn_config/`).

Interruption detection: `"adaptive"` (ML-based via `inference/interruption/`) or `"vad"` mode.

Subdirectories: `room_io/` (LiveKit Room I/O), `transcription/` (word-level sync), `avatar/` (avatar integration), `recorder_io/` (recording), `testing/` (test utilities), `turn_config/` (turn/interruption/endpointing config).

### LLM (`agents/src/llm/`)

- **`ChatContext`** — Chronologically ordered conversation state. Supports action-aware history summarization via `_summarize(llm, { keepLastTurns })` to compress old messages while preserving tool execution results.
- **Tool system** — `llm.tool()` with Zod schemas for parameters. `handoff()` for multi-agent transfers.
- **`FallbackAdapter`** — Agent-side multi-provider LLM failover with availability tracking.
- **Provider format adapters** — `provider_format/openai.ts`, `provider_format/google.ts`, and `provider_format/mistralai.ts` for model-agnostic code.

### Other Core Modules

- **STT** (`stt/`): `SpeechStream` with automatic retry. `FallbackAdapter` for agent-side multi-provider failover. `StreamAdapter` converts non-streaming STT + VAD to streaming.
- **TTS** (`tts/`): `SynthesizeStream`, `ChunkedStream`. `FallbackAdapter` for agent-side multi-provider failover. `StreamAdapter` for non-streaming providers.
- **VAD** (`vad.ts`): Voice Activity Detection interface. Silero plugin is the primary implementation.
- **Inference** (`inference/`): LiveKit Inference Gateway clients (LLM, STT, TTS, realtime) with server-side fallback support. Always use full `provider/model` format (e.g., `'openai/gpt-4o-mini'`). Also includes `interruption/` for adaptive interruption detection via ML models.
- **Stream** (`stream/`): Composable Web Streams API primitives (`StreamChannel`, `DeferredStream`, `MultiInputStream`).
- **IPC** (`ipc/`): Process pool for running agents in child processes. Two-way IPC: child sends inference requests back to parent.
- **Worker** (`worker.ts`): Main process connecting to LiveKit server, receives job assignments, spawns agent processes.
- **Telemetry** (`telemetry/`): OpenTelemetry tracing with custom span attributes (TTFT, TTFB, interruption probability, speech IDs, provider request IDs). Pino transport for structured logging.
- **Metrics** (`metrics/`): `AgentMetrics` union type covering LLM, STT, TTS, VAD, EOU, Realtime, and Interruption metrics. `ModelUsageCollector` for aggregating per-provider usage.
- **Beta** (`beta/`): `TaskGroup` for multi-task orchestration with optional chat context summarization between tasks.

### Remote Sessions (`voice/remote_session.ts`)

Wire protocol for distributed agents. `SessionTransport` abstraction with `RoomSessionTransport` (LiveKit room channels) and `TcpSessionTransport` (raw TCP socket) implementations.

### Plugins (`plugins/`)

Each extends `Plugin` base class, auto-registers on import via `Plugin.registerPlugin()`. Pattern: `@livekit/agents-plugin-<provider>`.

Plugin capabilities by type:

- **LLM**: openai, google, baseten, mistralai
- **STT**: deepgram (v1+v2), openai, baseten, sarvam (v1/v2/v3), mistralai, inworld, cartesia
- **TTS**: cartesia, elevenlabs, deepgram, openai, neuphonic, resemble, rime, inworld, baseten, sarvam (v1/v2/v3), mistralai, fishaudio, hume, speechify
- **VAD**: silero (ONNX-based, local)
- **EOU/Turn Detection**: livekit (HuggingFace + ONNX)
- **Realtime**: openai (+ responses/, ws/ modules), google, xai, phonic
- **Avatar**: trugen, lemonslice, bey, anam, liveavatar, did, protoface
- **Test mocks**: test (private, for unit tests)

### AsyncLocalStorage Patterns

The framework uses Node.js `AsyncLocalStorage` for implicit context passing:

- `agentActivityStorage` — Access current `AgentActivity` in callbacks
- `functionCallStorage` — Access current `FunctionCall` in tool handlers
- `speechHandleStorage` — Access current `SpeechHandle`

## Code Conventions

- **Lifecycle changes**: Follow the [lifecycle review checklist](CONTRIBUTING.md#review-lifecycle-changes)
  for session helpers, shutdown, and cleanup. Apply it to Python ports too.
- **License header** required on every new file:
  ```
  // SPDX-FileCopyrightText: 2026 LiveKit, Inc.
  //
  // SPDX-License-Identifier: Apache-2.0
  ```
- **Prettier**: single quotes, trailing commas, 100 char width, sorted imports (`@trivago/prettier-plugin-sort-imports`).
- **ESLint**: `@typescript-eslint` with strict rules. Prefix unused vars with `_`. Use `type` imports (`consistent-type-imports`).
- **TypeScript**: strict mode, `noUncheckedIndexedAccess`, `verbatimModuleSyntax`, target ES2022, module node16.
- **Time units**: Use milliseconds for all time-based values by default. Only use seconds when the name explicitly ends with `InS`.
- **Changesets**: All packages in `agents/` and `plugins/` release together (fixed versioning). Run `pnpm changeset` to add a changeset before PRing. The examples package is ignored.
- **API Extractor**: Public API surface is tracked. Run `pnpm api:check` after changing exports and `pnpm api:update` to update declarations.

## Testing