agent-visual-feedback

Files (1)

• SKILL.md

SKILL.md (first lines)

# Agent Visual Feedback (Describe this)

Use this skill when the operator wants the agent to **look at the current
viewer** and give actionable feedback — not a generic caption.

## When To Use

- UI **Describe this** button (stays on the Rerun/viewer tab; chat opens as a drawer)
- Chat turns containing `[npa-visual-feedback]` or “describe this viewer/visual”
- Interpreting held-out Rerun frames, Isaac/GR00T-style sim views, rollout video, images, or Data-pane JSON
- For Sim2Real recordings, inspect `summary/stage_progress` and
  `summary/policy_access` alongside `rollouts/outer_*/iter_*`: camera and action
  entities share the same frame timeline; policy access is a link/download
  action and must never be described as policy execution inside Rerun.
- Metadata-only feedback when a frame cannot be captured

## Model

1. Prefer a **quality-captured frame** (vision tier → `MiniMaxAI/MiniMax-M3`).
2. Wait for a non-blank canvas (skip uniform black/white/**mid-gray**; dense RGB, skeletons on dark grids, and meshes are valid). Cleared WebGL buffers often look mid-gray — never attach those. Blank detection must keep enough resolution to see **sparse** orange/cyan skeleton strokes (do not 80px-downscale them away).
3. Rerun capture must use the **MediaStream bridge** (`canvas.captureStream` → `<video>` → JPEG). Do **not** gate capture on sync `drawImage` blank checks — those false-negative on live WebGL/WebGPU.
4. If capture fails, send **metadata/text** and use the reasoning tier — never pretend pixels were seen.
5. Do **not** answer Describe-this from the grounded intent router.
6. Open the chat drawer and show **Describe this — capturing…** immediately; do not wait for capture before the user bubble appears.

## Visual kinds (generalized — no URI allowlists)

| Kind | Source | What to emphasize |
|------|--------|-------------------|
| `rerun` | Largest same-origin Rerun canvas after quality wait | Sim RGB, depth/seg, 3D mesh, tiled envs, policy strips — **not** “blank” by default |
| `video` | `<video>` current frame | Task progress, success/failure cues |
| `image` | Preview `<img>` | Scene contents, defects |
| `data` | `<pre>` / text excerpt | Report fields, success_rate, missing keys |

Domain hints are inferred from free-text metadata tokens (`artifact_key`, notes,
workflow name) such as isaac / gr00t / heldout / genesis / cosmos — never from a
hardcoded path allowlist.

## UX

- On the Viewer tab (Rerun/Video/Image/Data), chat is a **bottom-right collapsible widget** (`viewer-focus` + FAB).
- Clicking the **Chat** main tab from Viewer opens the widget (stays on the media pane); **Full chat** expands to the Chat tab.
- **Describe this** captures in-place, opens the widget, and queues the vision turn.
- Chat sends are **queued** (`enqueueChatJob`) so Describe + typed messages do not drop.
- Frame captures are JPEG-capped (`maxChars`) and nginx allows `client_max_body_size 32m` so multimodal POSTs do not 413 / `Failed to fetch`.
- Do **not** prefetch `.rrd` bodies (many runs). Soft-swap uses `add_receiver` only; warm wasm/js separately.

## Agent API flow

```http
POST /api/chat
{
  "session_id": "default",
  "visual_context": {
    "kind": "rerun",
    "run_id": "demo-workbench-ui",
    "artifact_key": "…/reports/sim2real.rrd",
    "frame_quality": "rendered",
    "capture": "frame"
  },
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "[npa-visual-feedback] Describe this rerun viewer…"},
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
      ]
    }
  ]
}
```

## Reply shape (required)

1. **What I see**  
2. **Likely meaning**  
3. **Operator feedback**  
4. **Next actions**

## Implementation pointers

- Helpers: `npa/src/npa/cli/agent_visual_feedback.py`
- UI: `describeVisual`, `waitForQualityRerunFrame`, `queueChatText`, viewer drawer
- Contracts: `AGENT_VISUAL_FEEDBACK_CONTRACT`, `AGENT_CHAT_QUEUE_CONTRACT`,
  `AGENT_VIEWER_CHAT_DRAWER_CONTRACT`

From nebius/nebius-physical-ai, Apache-2.0.

More from nebius/nebius-physical-ai