autonomous-control

Files (2)

• SKILL.md
• checkpoints.md · references/

SKILL.md (first lines)

# Autonomous control

The deliverable is a controller that completes the task from a fresh scene: the code
does the motions and measurements, re-localizes everything it acts on, checks each
step, and throws an exception when the scene is not what it was built for, never
failing silently.
AGENTS.md says how it is judged: run alone on unseen scenes, or run by you with your
perception checking the decisive points.

## Start from your manual success

Read the saved strategy and decisive frames in `memory/`. Create `code/<project>/` and
turn one useful maneuver into a function that takes `ctx` and explicit inputs. Test it
on the current scene, inspect its final images and logs, fix it, and repeat before
composing the whole `main(ctx)`. Reproduce the decisions that made the manual run
work, not its recorded actions.

## Model the invariant motion; perceive what changes

A new layout should need new perception and parameters, not new motion code.

- **Invariant motion in the object's frame.** Write approach standoffs, directions,
  grasp offsets, lift heights and the contact strategy relative to the object or
  target, never as coordinates from one run.
- **A perception layer with a fixed output.** One function turns the current scene
  into a small description the motions consume (target pose and size, support height,
  obstacles). Geometry comes from depth, camera geometry and kinematics; semantic
  judgements (which object, is it held, did it tip) come from images: in code through
  a [VLM](../vlm-perception/SKILL.md), asked with structured prompts and clear
  references, and through you when a step throws an exception.
- **Three layers.** Perception, motion primitives (each with its own checks) and a
  short task script that chains them, in separate modules.
- **Parameters as data.** Object descriptions, tolerances and per-scene settings live
  in one config with the range each was tested on. A scene outside a tested range is
  a reason to stop and look, not to extrapolate.
- **Choose at run time.** Pick the arm, approach side and posture from the perceived
  scene: reason out a few candidates, check them with the robot's dry-run tools
  (`<preview>`), and take the first that passes. Do not grid-search.
- Prefer simple motion functions and rules; use the VLM where simple rules break on
  new views. A step acts on its answers alone only once that call is pressure-tested on
  varied views and repeated frames; a failed call or an answer that disagrees with the
  geometry is an exception.

## Steps you can read, resume and supervise

- Split the pipeline into steps, each with an observable precondition and completion
  check.
- Each step `fn(ctx, ...)` checks its preconditions from fresh observations, acts,
  verifies the result and returns `{"ok": bool, "measured": {...}, "why": str}` plus
  the frames it used. When a check fails or is ambiguous, or the scene is outside what
  it was tested on, it throws an exception instead of improvising: it leaves the robot
  safe (nothing held dropped mid-air, arm clear, nothing moving) and returns `ok: False`
  with `why`, its measurements and the frames to look at.
- Handle each exception with your own perception: look at its frames and fresh ones,
  then rerun the step with corrected inputs, recover by hand, or stop. Fold an
  exception that recurs, with its fix, into the step.
- Make the pipeline resumable at any step, and give every motion that can fail a
  fallback.
- Keep tunable values as named constants at the top of a module, with their tested
  ranges.
- `controller.py:main(ctx)` chains the steps, stops at the first failure and returns a
  report. `code/<project>/README.md` says how to run the whole controller and each
  step, with their preconditions.

Mark perception checkpoints where a step's success depends on what the scene looks
like. When the controller, or a skill made from it, runs in a new scene, the agent
running it looks there itself and adapts the code where the scene differs, until the
code's own checks have proved reliable there; see
[references/checkpoints.md](references/checkpoints.md).

## Ordinary Python

Write the controller against a supplied context. The entrypoint is the same in
development and in every test or evaluation run:

```python
# code/<project>/controller.py
import json

def main(ctx):
    reply = ctx.call("<observe>")            # {"content": [text and image blocks]}
    state = json.loads(reply["content"][0]["text"])
    # observation -> decision -> action, e.g. ctx.call("<move_pose>", **target)
    return {"ok": True, "measured": {}, "why": ""}
```

Only a development wrapper, kept outside `code/<project>/` and run from the workspace
root, opens the context:

```python
# experiments/run_controller.py
import sys
from api.runtime import Context

sys.path.insert(0, "code/<project>")
from controller import main

with Context() as ctx:
    print(main(ctx))
```

- `main(ctx)` and its helpers use the supplied context unchanged and pass it down;
  they never open a new `Context()`.
- `ctx.call(tool, **arguments)` uses the live tool schemas (`MCP_SESSION.md`) and
  returns MCP content blocks; JSON metadata is in the text block, images stay image
  blocks. Inspect the schemas instead of inventing arguments.
- A tool error raises `RuntimeError` carrying the full error. Handle refusals in code;
  never retry the same motion blindly.
- Robot calls run one at a time, but a script does not own the robot: your direct
  calls can interleave with it. Run one controller at a time and stop it before you
  act directly.
- Interrupting Python does not cancel a dispatched motion. After an interruption or an
  ambiguous error, check `<status>` and a fresh observation before continuing.
- Returning does not reset the scene: it keeps the script's actions, so you can
  inspect and continue by hand.
- Save replies, decisions and prints under `output/`; the harness keeps the robot's
  own frames and traces.

## Test end to end

From robot-agent-exploration/rax, Apache-2.0.

More from robot-agent-exploration/rax