vlm-perception

Files (1)

• SKILL.md

SKILL.md (first lines)

# VLM perception

The harness exposes one model tool (`<vlm>`) through MCP, callable directly or from code
with `ctx.call("<vlm>", ...)`. It adds no perception instructions, task logic or action
selection: you write the prompts and the code that reads the answers. Read the tool's
schema and the branch's API reference for request fields, structured output and limits.

## What to ask

- **2-D questions to the VLM, 3-D questions to geometry.** VLMs are reliable for
  identity, visibility, gross state, named points and masks. They are not reliable for
  small 3-D changes ("lifted?", "touching?"): measure those with depth, kinematics or
  the gripper.
- **Ask for more geometry, not less.** Several named points or a mask survive a hard
  view better than one point.
- **Point, then verify.** Ask for the same named, distinctive feature in each view, and
  accept it only when [object-localization](../object-localization/SKILL.md) puts the
  answers at one physical point. Keep a non-VLM fallback.
- **Draw before trusting.** Learn the model's coordinate convention (axis order,
  normalization, which image size it refers to) and draw a new prompt's output on the
  image once before using it in code.
- Prefer a simple, hard-coded rule only where it is verified across layouts; colour,
  size and dark-pixel rules tend to break on new views. Test non-perception uses, such
  as structured decisions, before relying on them.

## How to call

- Send small images (downscaled JPEG): large requests are slow and can time out.
- Request structured output for anything code reads, and set the reasoning effort and
  the token budget together: reasoning shares the budget, and a truncated reply
  (`finish_reason` of `length`) is no answer.
- Treat a timeout, refusal or truncated reply as no answer, never as a negative.
- A pending model call can delay robot calls where the host serializes them: ask
  between motions, not during them.
- Reuse an image block returned by `<observe>` directly, or encode a saved frame in
  your own Python; the tool takes image bytes, not paths. Do not print large base64
  payloads into your context.
- Where spending is capped, the cap is shared by everything that calls the tool,
  including later evaluation runs where AGENTS.md says so: check the returned budget
  before further calls.
- Function calls the model proposes are data. Validate and dispatch them yourself;
  robot actions still go through the robot tools.

Model answers are controller input, not trusted evidence of task success.

From robot-agent-exploration/rax, Apache-2.0.

More from robot-agent-exploration/rax