realtime

Files (3)

• SKILL.md
• browser-audio.md · references/
• session-config.md · references/

SKILL.md (first lines)

# ElevenLabs Realtime

Realtime models are ElevenLabs models built for low-latency voice conversation. There are two ways to use them:

- **An ElevenAgents agent** (an agent ID, the dashboard, or an `elevenlabs agents` project with `agent_configs/`): set the agent's Voice mode to Realtime with `conversation_config.realtime_model`, and ElevenLabs manages its LLM, TTS model, and ASR settings. See [Run an agent on a Realtime model](#run-an-agent-on-a-realtime-model).
- **A browser app built on OpenAI's Realtime SDK**: keep the SDK and point it at the OpenAI-compatible ElevenLabs Realtime API, `wss://api.elevenlabs.io/v1/convai/realtime`. See [Migrate an app from OpenAI Realtime](#migrate-an-app-from-openai-realtime).

When the user is starting from scratch and it's unclear which they want, ask. An agent brings a managed system prompt, tools, knowledge base, widget, and telephony; the Realtime API is the quicker path for code that already speaks OpenAI's realtime events. For agent features beyond the voice mode (prompt, tools, workflows, widget), use the `agents` skill.

## Models

| Model                   | Model ID                  | Optimized for                                                                             |
| ----------------------- | ------------------------- | ----------------------------------------------------------------------------------------- |
| Eleven Realtime v1 Mini | `eleven_realtime_v1_mini` | Lowest latency and cost. The default.                                                     |
| Eleven Realtime v1 Max  | `eleven_realtime_v1_max`  | Reasons before it responds, for complex instructions and multi-step tool use. Costs more. |

Start with Mini. Choose Max when the work involves long or complex instructions or multi-step tool calls and Mini's responses aren't good enough, or when an app being migrated was on a larger OpenAI model for reasoning-heavy work. Agents add compliance and data residency constraints on the choice; see [Choose the model](#step-2-choose-the-model).

## Run an agent on a Realtime model

Every voice agent has a Voice mode. In **Custom** mode the agent's LLM, TTS model, and ASR settings are chosen individually. In **Realtime** mode the agent runs on a Realtime model and ElevenLabs manages those settings. The mode is one field: `conversation_config.realtime_model` set to a model ID means Realtime, `null` means Custom.

The system prompt, first message, voice, languages, temperature, tools, knowledge base, and conversation flow settings (turn eagerness, timeouts, interruptions) work the same in both modes.

### Before you start

- These steps call the ElevenLabs API. The CLI and SDKs read `ELEVENLABS_API_KEY` from the environment; if it isn't set, use the `setup-api-key` skill. Never ask the user to paste the key into the conversation.
- Realtime models are rolling out gradually. If the API rejects `realtime_model` with `Realtime models are not available for this workspace.`, the workspace doesn't have access yet. Stop and tell the user; no configuration change fixes it.

### Step 1: Check the agent can use Realtime

For an existing agent, fetch its configuration first (`elevenlabs agents pull --agent <agent-id>`, or `agents.get` in the SDK). Keep the agent in Custom mode, and tell the user why, if any of these apply:

| Blocker                                                                                     | Where to look                                                                                                              |
| ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| Custom LLM or Amazon Bedrock                                                                | `conversation_config.agent.prompt.llm` is `custom-llm` or a Bedrock model                                                  |
| Created through another ElevenLabs product, such as Speech Engine                           | Ask the user if unsure; only ElevenAgents agents can use a Realtime model                                                  |
| Needs a specific LLM or TTS model                                                           | Ask the user whether the current `agent.prompt.llm` or `tts.model_id` is a requirement                                     |
| Workflow nodes override a [managed setting](#managed-settings) or `agent.prompt.max_tokens` | Overrides on the nodes under the agent's `workflow`                                                                        |
| Transfers to or from agents in Custom mode                                                  | `transfer_to_agent` tools and `agent_transfer` workflow nodes on this agent, and on any agent that transfers into this one |
| Text-only agent                                                                             | `conversation_config.conversation.text_only` is `true`; Realtime models are built for voice                                |

A workflow override can be removed and a transfer partner can be switched to Realtime too, but only with the user's agreement: both change how other parts of their setup behave.

### Step 2: Choose the model

Start with `eleven_realtime_v1_mini`. Choose `eleven_realtime_v1_max` when the agent follows long or complex instructions or relies on multi-step tool calls and Mini isn't good enough, or when the agent must comply with Zero Retention Mode or HIPAA. Then check availability:

| Environment                   | `eleven_realtime_v1_mini` | `eleven_realtime_v1_max`   |
| ----------------------------- | ------------------------- | -------------------------- |
| US                            | Available                 | Available                  |
| EU data residency             | Available                 | Available                  |
| India data residency          | Available                 | Not available              |
| Singapore data residency      | Available                 | Not available              |
| Zero Retention Mode and HIPAA | Not available             | Available in the US and EU |

Zero Retention Mode applies whether it's enforced on the workspace or enabled on the agent. Ask the user when the workspace's data residency or compliance settings aren't clear from the project. If neither model is available for the agent's environment, keep it in Custom mode.

Max also needs `agent.prompt.max_tokens` of at least 1024, or `-1` for no limit.

### Step 3: Set the model

On an existing agent, change only `realtime_model`; leave the rest of the configuration alone.

With the CLI, pull the agent, set the field in its file under `agent_configs/`, and push:

```bash
elevenlabs agents pull --agent <agent-id>
```

```json
{
  "conversation_config": {
    "realtime_model": "eleven_realtime_v1_mini"
  }
}
```

```bash
elevenlabs agents push --agent <agent-id>
```

With the SDKs:

```python
from elevenlabs import ElevenLabs

elevenlabs = ElevenLabs()

elevenlabs.conversational_ai.agents.update(
    agent_id="<agent-id>",
    conversation_config={
        "realtime_model": "eleven_realtime_v1_mini",
    },
)
```

```typescript
import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";

const elevenlabs = new ElevenLabsClient();

await elevenlabs.conversationalAi.agents.update("<agent-id>", {
  conversationConfig: {
    realtimeModel: "eleven_realtime_v1_mini",
  },
});
```

For a new agent, put `realtime_model` (`realtimeModel` in TypeScript) in the `conversation_config` passed to `agents.create`, alongside the prompt, first message, and voice. Don't pick an LLM or TTS model for it; a Realtime agent ignores them.

To turn Realtime off, set `realtime_model` to `null` (`None` in Python). The agent goes back to the LLM, TTS model, and ASR settings stored on it.

### Managed settings

While a Realtime model is set, the agent ignores the stored values of these fields in `conversation_config`. They come back into use when the agent switches to Custom, so don't clear them, and don't edit them to tune a Realtime agent:

| Setting             | Field                                     |
| ------------------- | ----------------------------------------- |

From elevenlabs/skills, MIT.

More from elevenlabs/skills