Okareo

Generate a check from a natural language description. Uses AI to create the prompt template (model checks) or Python code (code checks), then saves the check. Use this when you only have a description of what to evaluate. When you already know the exact prompt template or Python code the check should use, call create_or_update_check directly instead. The generated prompt/code is returned in the response — review it and refine with create_or_update_check if needed. Args: name: Name for the generated check. description: Natural language description of what to evaluate (e.g., "c…

Retrieve the full configuration of a check by name, including its prompt template or code contents. Args: name: Name of the check to retrieve. version: Optional check version number to pin. Omit (or leave null) for the most recent version. The response always lists every available version under "available_versions".

Retrieve the full conversation transcript for a single data point. Use this after get_test_run_results to drill into a specific conversation. Provide either scenario_index (1-based, from the scores summary) or test_id (UUID) to identify the conversation. Returns the complete message transcript (model_input), final output (model_result), per-turn check scores (metric_value), and the scenario seed data. Args: test_run_id: The UUID of the test run. scenario_index: 1-based index of the conversation within the test run. Visible in get_test_run_results output. test_id: UUID of…

Retrieve a dashboard's full configuration by name. Args: name: Name of the dashboard to retrieve.

Query the Okareo documentation system for information about Okareo primitives and workflows. Use this tool when the agent or user needs to understand how Okareo concepts work — Scenarios, Checks, Targets, Drivers, Evaluations, and Simulations. Two modes are available: - 'conceptual': Detailed technical documentation for agent reasoning. Default top_k=5 (returns up to 5 documentation entries). - 'user_legible': Plain-language explanations for human users. Default top_k=3 (returns up to 3 documentation entries). If the Okareo documentation service is unavailable (e.g. air-gapped environme…

Retrieve a driver persona you've already configured. Retrieves a Driver by name, returning its full configuration including the persona prompt. Args: name: Name of the driver to retrieve.

Read detailed information about a registered generation model. Returns the generation model's target LLM configuration, tags, creation time, and any warnings (e.g., if the target LLM has been deprecated). Args: name: Name of the registered generation model.

Serve REPS agent-evaluation baseline material (scenario banks, drivers, checks, eval configs). REPS is Okareo's agent-evaluation workbench: per-pillar baseline material for evaluating AI agents across R-reasoning, E-execution, P-performance, and S-security, plus shared explore/ probes and a profile/ example. The material is published as tagged releases of the okareo-tools repo; this tool serves the latest release so reps skills need no local copy of the tree. Two modes: - Discovery (omit `path`): list what files exist in the served release — the full tree, or one area via `pillar`. File li…

Read a scenario's metadata and all data rows. Look up by name or scenario ID. Returns scenario details and all input/result data rows. Args: name: Name of the scenario to retrieve. scenario_id: ID of the scenario to retrieve. Takes precedence over name.

Check the current configuration of a test target. Retrieves a Target by name. Works for all target types (Generation, Custom Endpoint, and Voice). For **custom_endpoint** Targets, the response is a flat envelope whose keys mirror the kwargs accepted by `create_or_update_target`, so a copilot can read the result, swap in a new name + secrets, and feed it back to create to clone the Target. Fields the backend keeps secret (those listed in `sensitive_fields`) appear with the literal value `"***REDACTED***"` — these MUST be replaced with real values before calling `create_or_update_target`, whic…

Retrieve prompt templates for common Okareo patterns. Returns starter templates for building Okareo test components. These templates are served as static content from the MCP — no network calls required. Always available, even in air-gapped environments. Available templates: - basic_scenario: Template for creating a basic Okareo test scenario - boolean_check_prompt: Template for a pass/fail (boolean) check prompt - score_check_prompt: Template for a scored check prompt - check_code: Template for a code-based check (Python function) - target_validate_check_prompt: Template for validating targ…

Load the results of a specific test run, one page at a time. Look up by test run ID (UUID) or by name (returns the most recent run matching that name). Both lookups return the same shape. A test run is a container of conversations, so this reads like a list: it returns a bounded page, not the whole run. Three depths: - "summary" (default): run aggregates, provenance, and each conversation's whole scenario input and result. No per-check outcomes, no explanations, no transcripts. - "detailed": adds the row-level metrics block, which carries per-check outcomes AND their written explanati…

Retrieve a voice provider integration by id, including its status. Args: integration_id: The integration's id (from list_voice_integrations).

Get the inbound webhook endpoint for a voice provider. Paste the returned URL into the provider's console so its call traffic reaches Okareo monitoring. Args: provider: Voice platform — one of: retell, twilio, vapi, elevenlabs. public_id: The integration's public_id (from connect_voice_integration or get_voice_integration). Required for retell and twilio.

Submit completed voice conversations to Okareo for monitoring. Each conversation's turns become evaluable data points and any configured monitors run their checks automatically. Use this to feed production voice traffic (Retell, Twilio, VAPI, ElevenLabs, or a custom source) into Okareo monitoring. Conversations are validated individually: valid ones are ingested and invalid ones are returned in a "rejected" list — the batch is not all-or-nothing. Args: conversations: List of conversation objects. Each MUST include a "call_id" and at least one of: "transcript" (a list of …

Browse available LLMs from the Okareo registry. Returns all LLMs that can be used when registering a generation model for testing. Each entry has a name, display name, and provider. Use a model_name from this list when calling register_generation_model.

List available quality checks, grouped by category. Returns checks (both built-in and custom) available in your Okareo account, organized into `checks_by_category` using the platform's `__category:<Category>` tags; checks with no category appear under `uncategorized`. Select checks from the category matching your task AND modality: voice-specific categories (e.g. voice/audio quality) apply to voice simulations, while checks outside voice-specific categories are generally useful for both chat and voice. A check carrying multiple categories appears under each of them. Each check has a name, de…

List the analytics dashboards in your Okareo project. Args: limit: Maximum number of dashboards to return (default 20). Use 0 for no limit.

Discover the voices, voice profiles, and languages available for configuring voice-capable drivers. Call this before create_or_update_driver when building a voice agent simulation, so you can pass valid `voice`, `voice_profile`, and `language` values. Each entry in `voices` carries selection metadata — use it to pick the voice: - `language`: bare ISO code (e.g. "en", "es", "ja"). The driver's `language` is derived from the selected voice's language when omitted, and must not conflict with it. - `accent`: free-text accent label present on many voices (e.g. "British", "Mexican", "Parisia…

Find a driver persona in your organization. Drivers are shared across every project, so this list grows with the whole organization's history rather than one project's — it is bounded by default and filterable by name. Two depths: - "summary" (default): driver id, name, model, and creation date. - "detailed": adds temperature and, for voice drivers, their voice settings. Voice attributes are omitted entirely for text drivers rather than returned empty. Args: limit: Maximum number of drivers to return. Defaults to 20. Set to 0 to return all. offset: Number of drivers to s…

Browse all registered generation models in the project. Returns generation model names, IDs, target LLM configurations, and creation timestamps. Use this to see what generation models are available for testing.

See which Okareo projects you can work in, and which one is active. Work in Okareo is organized into projects: scenarios, targets, simulations, evaluations, and dashboards each belong to exactly one. Checks and drivers are shared across all of them. Each project reports its tags and whether it is archived. Archived projects are still listed and still work — archiving only hides a project from the project picker in the Okareo app. Use this to answer "which project am I in?", to find an archived project worth restoring, and whenever you need to ask the user to choose one. Create a new one wit…

List scenarios in the project, most recent first. Returns scenario names, IDs, tags, row counts, and creation dates. Use this to discover existing scenarios before running a test. Args: limit: Maximum number of scenarios to return (default 20). Set to 0 to return all scenarios.

Find or compare simulation runs in the project, most recent first. Two depths: - "summary" (default): identity, status, conversation count, timings, and what each run executed against (target, scenario, driver). - "detailed": adds aggregate model_metrics and the provenance identifiers, up to 20 runs at a time. **To compare simulations, use detail_level="detailed" here** rather than calling get_test_run_results for each run — one response carries the aggregates and the provenance needed to tell whether two runs are even comparable, without opening any of them. Neither depth returns per-…

Find a simulation target. Lists every simulation target (voice and custom_endpoint types) created via create_or_update_target. Does not include generation models registered via register_generation_model — use list_generation_models for those. Bounded by default and filterable by name. Two depths: - "summary" (default): target id, name, type, and creation date. - "detailed": adds tags and a short configuration summary. Use get_target for one target's complete configuration. Args: limit: Maximum number of targets to return. Defaults to 20. Set to 0 to return all. offset: Num…

List every Okareo organization you have access to in this MCP session. The currently-active organization is marked ``is_current: true``. The active organization is determined at sign-in (the token this session presents is already scoped to it). Only available on OAuth-authenticated sessions; on Bearer-API-key sessions returns ``tenant_selection_requires_oauth``.

Find a test run in the project, most recent first. Two depths, because finding a run and comparing runs are different jobs: - "summary" (default): identity, type, status, conversation count, timings, and what each run executed against (target, scenario, driver). Enough to find the run you mean. - "detailed": adds aggregate metrics, the app link, and the provenance identifiers. Use it to compare runs without opening any of them. Neither depth returns per-conversation scores — use get_test_run_results for those. Args: model_name: Optional filter — only show test runs using this mod…

List the voice provider integrations in your Okareo project. Args: limit: Maximum number of integrations to return (default 20). Use 0 for no limit.

Move a Scenario — and everything under it — to another project. Moves are per Scenario, never per simulation: the Scenario, all of its simulations, their datapoints, check results, and traces move together. Targets never move — they are shared across the organization and stay usable from every project. Monitors, dashboards, and notification settings stay in the source project. DRY-RUN-FIRST PROTOCOL: dry_run defaults to true and the first call must keep it that way. Present the returned plan to the user — the per-table counts and any blockers — and only after the user explicitly confirms cal…

Preview what will be deleted before removing a scenario. Shows the scenario details and count of related test runs that will also be deleted. Use delete_scenario to confirm deletion after reviewing. Args: name: Name of the scenario to preview deletion for. scenario_id: ID of the scenario. Takes precedence over name.

Query Okareo's product analytics to understand evaluation trends. Answers questions like "how is my evaluation quality trending" by aggregating measures across dimensions over a time window. Args: measures: Metrics to aggregate. Required. For the ``check_trend`` cube: avg_check_value, issue_rate, error_rate, datapoint_count, issue_count, error_count, test_run_count, avg_latency, sum_cost, input_token_count, output_token_count. dimensions: Optional group-by fields (e.g. ["check.name"], ["target.name"], ["provider"]). cube: Optional analytics cube na…

Re-score a completed test run against a set of checks. Re-runs checks against an already-finished test run without re-executing the original model or simulation, and without changing the original run's results. Useful after a check definition changed, or to score an existing run against additional checks. Args: test_run_id: UUID or name of a completed test run. checks: Optional list of check names (or IDs) to score against. When omitted, the run's existing checks are re-run.

Register a generation model for testing by selecting an LLM from the registry. Creates a generation model (Model Under Test) that points to a specific LLM (e.g., 'azure/gpt-4o-mini'). Use list_available_llms to see available LLMs. The registered generation model can then be used with run_test. Args: name: A human-readable name for this generation model (e.g., 'my-chatbot'). model_name: The LLM from the registry (from list_available_llms).

Set the display order of dashboards. Args: ordered_names: Dashboard names in the desired order.

Rotate a voice provider integration's secrets. Args: integration_id: The integration's id. secrets: The new provider secret values. The response returns only a secret summary, never raw secret values.

Run a multi-turn conversation evaluation of your AI agent. Combines a Target (the system under test), a Driver (the simulated user), and a Scenario (the test cases) to generate realistic multi-turn conversations and evaluate them with quality checks. Returns promptly so the call never times out on long runs. Short runs that finish within the buffer window return ``status: "finished"`` with results ready; longer runs return ``status: "running"`` with the ``test_run_id``, ``app_link``, and an ``estimated_runtime`` — the run continues to completion on its own. In both cases, poll get_test_run_r…

Submit a quality test that evaluates a model against a scenario using checks. Returns promptly so the call never times out on long runs. Short runs return ``status: "finished"`` with results ready; longer runs return ``status: "running"`` with the ``test_run_id`` and ``app_link`` — the run continues to completion on its own. In both cases, poll get_test_run_results with the returned test_run_id to retrieve scores. Args: scenario_name: Name of the scenario to evaluate against. model_name: Name of the registered model to evaluate. checks: List of check names to apply (e.g., ["coher…

Create or update an analytics dashboard by name (upsert). If a dashboard with this name already exists it is updated; otherwise a new one is created. Size each panel with a named ``size`` from the catalog below (PREFERRED — guarantees a legible layout) and omit positions entirely: panels are auto-placed in the order given (left-to-right, top-to-bottom on a 12-column grid, wrapping rows, never overlapping). Size catalog and when to use each: - ``small-square`` (3x6): single ``stat`` metrics. - ``half-rectangle`` (6x6): ``line``/``bar``/``area`` trends, two per row. - ``half-square`` (6x9): …

Save a named scenario for use in quality tests. Provide EXACTLY ONE dataset source: `content` or `rows`. Feed the scenario rows directly — the server cannot read files from your machine. Choose by size, to avoid unnecessary token cost: - Datasets UNDER 2,000 rows: read the .jsonl file yourself and pass its text as `content` (preferred), or pass `rows` for tiny datasets. - Datasets of 2,000 ROWS OR MORE: do NOT read the file into context. Instead, tell the user to save it locally and upload it directly to Okareo via the web app, SDK, or CLI. Passing a large dataset through the assis…

Choose the Okareo project to work in for this conversation. Validates the project exists, then tells you how to apply it. This server stores nothing: pass `project` on every subsequent project-scoped call, and record the user's choice so you can reuse it in later conversations. Takes effect immediately — changing project never requires signing out, reconnecting, or re-authorizing. An archived project can be selected like any other. Args: project: Project name or id, as shown by list_projects.

Explain how to change the Okareo organization this session works in. Does not switch it. The organization is chosen **during sign-in** (feature 030): when you connect the Okareo MCP you pick which organization to authorize, and the credential this session uses is scoped to it. This tool returns the steps to change it — reconnect/re-authenticate the Okareo MCP from your copilot and select a different organization when prompted — and the current organization's id. Use ``list_tenants`` to see which organizations you have and which one is active. On Bearer-API-key sessions returns ``tenant_select…

Restore an archived project to the Okareo project picker. Reverses archive_project. Nothing about the project's contents changes — archiving never touched them. Find archived projects with list_projects: each row reports is_archived. Args: project: The Project to unarchive (name or id).

Change the LLM that a registered generation model points to. Updates the generation model to use a different LLM from the registry. Use list_available_llms to see available LLMs. Args: name: Name of the registered generation model to update. model_name: The new LLM from the registry.

Rename an Okareo project, or change its tags. Only what you pass changes: send `name` to rename, send `tags` to replace the tag list, send both to do both. Tags are replaced wholesale, not merged — read the current ones from list_projects first if the user means "add one". Name the project explicitly; this tool never guesses which project to edit. If the new name is already taken by another project, the call is refused and nothing changes. This tool does not archive or unarchive — use archive_project and unarchive_project for that. Args: project: The Project to update (name or id). …

Update a voice provider integration's metadata. Args: integration_id: The integration's id. metadata: The new metadata object.