agent-platform-eval-flywheel

Files (13)

• SKILL.md
• dataset_schema.md · references/
• deployment.md · references/
• failure_patterns.md · references/
• metric_registry.md · references/
• sdk_patterns.md · references/
• compare_results.py · scripts/
• endpoint_evaluation.py · scripts/
• inspect_results.py · scripts/
• maas_evaluation.py · scripts/
• parse_adk_traces.py · scripts/
• render_html_report.py · scripts/
• validate_dataset.py · scripts/

SKILL.md (first lines)

# Agent Platform Eval Flywheel Skill

Help users evaluate and iteratively improve GenAI models and agents using the
Agent Platform GenAI Evaluation SDK (`google.genai` / `agentplatform`).

## When to use this skill

-   Evaluating GenAI agents or models with the Agent Platform GenAI Evaluation
    SDK (`client.evals.evaluate()`).
-   Creating evaluation datasets from session traces, pandas DataFrames, or
    synthetic generation.
-   Selecting, configuring, or writing custom evaluation metrics.
-   Analyzing rubric verdicts, loss patterns, and clustering failures.
-   Suggesting concrete code/prompt improvements based on eval results.
-   Evaluating a model served on an Agent Platform **endpoint** (BYOM) or a
    **Model-as-a-Service (MaaS)** model by ID — including deploying the model
    first if needed. For this case, follow
    [references/deployment.md](references/deployment.md) and use the
    `endpoint_evaluation.py` / `maas_evaluation.py` scripts.

## Safety & Confirmation Tiers (CRITICAL)

Before executing any commands or scripts on behalf of the user, you MUST adhere
to the following safety tiers based on the action requested:

1.  **Tier R**: Read-only (`inspect_results.py`, `compare_results.py`,
    `validate_dataset.py`, `parse_adk_traces.py`, `render_html_report.py`)
    *   **Rule**: No confirmation needed. You may execute these helper scripts
        immediately to inspect data, validate schemas, parse traces, or compare
        evaluation results.
2.  **Tier M: Read-only with Compute Costs (`client.evals.run_inference`,
    `client.evals.evaluate`, `client.evals.generate_conversation_scenarios`,
    `client.evals.generate_loss_clusters`)**
    *   **Rule**: These operations invoke LLMs or remote evaluation services
        that consume compute resources and incur costs. This requires
        **interactive confirmation** with 'Yes'/'No' options.
    *   **Confirmation for EVERY evaluation run**: Every evaluation,
        re-evaluation, metric update, parameter change, or synthetic scenario
        generation requires its own dry-run preview and interactive
        confirmation. Never execute a second evaluation, comparison pass, or
        modified evaluation without presenting a new confirmation preview and
        obtaining user approval.
    *   **Same-turn restriction**: Do not run the evaluation in the same turn as
        presenting the confirmation prompt. End your turn after asking and wait
        for the user's reply; only execute after explicit 'Yes' / approval.
        Printing a preview and then calling the tool before the user can answer
        does not count as obtaining confirmation.
    *   **No Pre-Execution of Remote Evaluation**: NEVER execute
        `client.evals.evaluate()`, `client.evals.run_inference()`,
        `client.evals.generate_conversation_scenarios()`, or run any script
        invoking these remote operations before user confirmation. In the
        initial turn, you may prepare local data structures and compose the
        script, but you MUST present the dry-run preview card and obtain
        explicit user confirmation before running any remote evaluation or
        scenario generation call.
    *   **Immediate Execution Upon Approval**: Once the user explicitly approves
        (e.g., 'Yes', 'Approved', 'Go ahead', 'Proceed'), proceed directly to
        executing the previewed evaluation script via `run_command` and report
        the results. Do not conclude the turn without executing the approved
        action.

## Setup

The scripts need `vertexai` (from `google-cloud-aiplatform[evaluation]`),
`google-genai`, `pandas`, and `requests`. Do **not** create a virtual
environment — it starts empty and hides packages the environment already
provides, forcing a redundant install. Probe, and install only what is missing:

```bash
python3 -c "import vertexai, google.genai, pandas, requests" \
  || pip install 'google-cloud-aiplatform[evaluation]>=1.163.0' 'google-genai>=1.0.0'
```

The version specifiers must stay quoted: unquoted, bash reads `>=1.154.0` as a
redirect and silently writes an empty file instead of constraining the install.

Need `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION`.

-   **Preserve User Project and Location**: Always prioritize the user's
    explicitly provided project and location (e.g. `project='<PROJECT_NUMBER>'`,
    `location='us-central1'`). Never change or override the user's requested
    location to 'global' unless the user explicitly requested 'global'.
-   **Missing Parameters**: If the user's request omits the project or location,
    you MUST pause in your response and ask the user for the missing
    location/project before preparing or running the evaluation.

### Correct SDK entrypoints

```python
import agentplatform
client = agentplatform.Client(project=PROJECT, location=LOCATION)

client.evals.run_inference(model=..., src=...)
client.evals.evaluate(dataset=..., metrics=...)
client.evals.generate_conversation_scenarios(...)
```

Two imports that look plausible and are not:

-   `from agentplatform.types import evals` -- `ModuleNotFoundError`. `types` is
    a module, not a package; use `from agentplatform import types`.
-   `from vertexai.evaluation import PointwiseMetric, EvalTask` -- the
    superseded SDK. Its classes take different arguments (`PointwiseMetric` has
    no `system_instruction`), so code written against it fails with `TypeError`
    rather than an import error. Use `agentplatform` throughout.

## The Quality Flywheel

Five stages, run in order on the first pass, then loop 2 → 5 until quality
targets are met.

### Shortcuts that waste time

| Shortcut                             | Why it fails                         |
| ------------------------------------ | ------------------------------------ |
| "I'll tune the metric threshold down | Hides real failures. Fix the agent,  |
: so it passes."                       : not the bar.                         :
| "This case is flaky, I'll skip it."  | Flakiness reveals non-determinism in |
:                                      : the agent. Fix with `temperature=0`  :

From google/skills, Apache-2.0.

More from google/skills