axstream

Files (1)

• SKILL.md

SKILL.md (first lines)

# axstream — replay UI tasks instead of re-deriving them

## First decision: delegate multi-step UI work to the computer-use subagent

If your harness has a Task/Agent tool and an agent named `computer-use`
(plugin form: `axstream:computer-use`) is available, DELEGATE any multi-step
UI task to it instead of driving the UI from the main conversation. It runs
on a fast model with exactly the tools below, replays a matching macro when
one exists, drives novel tasks as batched verified actions, saves a macro on
success, and returns a compact outcome report — so screenshots and OCR dumps
never enter your context, and your own turns stay fast for the rest of the
session. It can run in the background while you continue other work.

- Delegate: "open X and do Y then Z", click-through flows, reproduce-a-bug
  in a running app, verify-something-on-screen, anything worth a macro.
- Do it yourself (tools below): a single action, or when you must reason
  about visual state mid-flow.
- Authorize explicitly: if the task includes an irreversible step (send,
  pay, delete), name that exact step in the delegation prompt — the agent
  refuses `risk:"risky"` ops it wasn't explicitly given.

## Fast primitives (axstream MCP tools, if connected)

When the `axstream` MCP server is available, prefer its primitives over
screenshot-driven computer use for mechanical steps — they run on-device in
milliseconds and return text, not images:

- `screen_text` (~200ms) — the window as text lines with coordinates.
  Use INSTEAD of a screenshot to learn what's on screen.
- `find` (~30-600ms) — locate one control/text; returns a click-ready target.
- `act` — a whole batch of actions in ONE call (open/click/type/key +
  `assert` guards). Plan the sequence once; don't take a model turn per step.
- `check` (~250ms) — verify an outcome ("results page shows my query")
  without a verification screenshot.
- `begin_capture` + `compile_capture` — wrap Codex's initialized native `sky`
  computer-use object, record successful actions plus their preceding AX
  state, then compile that real trace into a parameterized macro. This is the
  native fallback bridge; cua-driver's recorder cannot observe `sky` calls.

The compile loop (this is what makes everything fast over time): novel task
-> use ONE `act` batch when the steps are clear, OR call `begin_capture`
before a native Codex computer-use fallback -> if it succeeded,
`write_macro` the act ops or `compile_capture` the native trace ->
`verify_macro` when a live re-run is safe -> from then on it's a single
`replay_macro` call. Route smartly (benchmarked, 2026-07-30):

- VISUAL-STATE questions (which item is selected/highlighted, colors,
  rendered appearance) -> go native DIRECTLY. OCR text cannot see selection
  rings; paying screen_text first and falling back doubles the cost.
- Pure READING in AX-rich apps (Safari, system apps): if your native
  computer use already returns accessibility TEXT (not screenshots), it may
  be as fast as screen_text — use whichever you have; screen_text's edge is
  apps with poor/truncated AX (Electron, canvas, custom toolbars) and hosts
  whose computer use is screenshot-based.
- axstream wins BIGGEST on: repeat tasks (replay a verified macro — measured
  2.7x faster than warm native CU on the same task), batched mutations with
  outcome asserts, and anything you'll ever do twice.

The verify gate (never trust an unchecked macro): a macro can replay to
100% and still not do the task — so every macro should END with an assert
that only passes when the outcome is real, and `verify_macro` /
`axstream verify <name>` replays it once live to earn a `verified` stamp
(shown in `list_macros`; editing the macro makes it stale). Do NOT verify
macros whose re-run has irreversible effects (sending, paying, deleting) —
the gate refuses `risk:"risky"` ops for exactly that reason. Prefer
verified macros; treat unverified ones as drafts to dry-run first.

axstream turns a macOS UI task into a text file of actions (`.axstream`) that
replays deterministically through the cua-driver daemon: no screenshots to
reason over, no per-step model calls, pid-addressed background clicks that
never steal the user's mouse. A replayed action costs ~0.2–1.3s instead of a
full perceive→think→act loop. The more tasks you save, the faster you get.

## The loop to follow

1. **Before any macOS UI task**: `axstream list --json` — one JSON object per
   macro with `name`, `description`, `when_to_use`, `slots`. If one matches,
   replay it instead of driving the UI yourself.
2. **Verify first**: `axstream replay <name> --dry` prints the resolved
   action list without executing (missing slots self-fill from header
   examples). Sanity-check the ops match the intent.
3. **Run**: `axstream replay <name> --slots '{"slot": "value"}'`
4. **After doing a UI task manually** (the slow way): write the steps as a
   macro file (format below), or compile the native trace as described next;
   `--dry` it, replay it once to verify, and save it to
   `~/.axstream/macros/<name>.axstream`. Next time it's one command.

Saving rules — macros are also launched by VOICE from the menu bar, and the
library is shared, so:

- Add `"examples"` to the header: 2–4 short utterances phrased the way a
  person would SAY the task out loud (lowercase, no punctuation), each
  slotted example containing its slot values verbatim, e.g.
  `{"utterance": "google mumbai weather", "slots": {"query": "mumbai weather"}}`.
  The voice matcher effectively never picks a template with no examples.
- Save under a NEW distinct name; NEVER delete, rename, or overwrite another
  macro file — even one that looks similar (a sphere macro is not a cylinder
  macro). True same-task twins are deduplicated automatically by
  `write_macro`'s signature upsert, which archives rather than deletes.
  Deleting a neighbor macro silently breaks the user's voice commands.

Deriving a macro from an existing flow (the preferred construction): when a
requested task differs from an existing macro only in a literal (a typed
search term, a URL, a filename), do NOT clone the macro per variant —
generalize the existing one by promoting that literal to a `{slot}` with a
`description` + `example`, add example utterances covering the variants, and
re-verify. One `blender-add-mesh` with a `{shape}` slot replaces
sphere/cube/cylinder clones and handles shapes nobody recorded. A macro is
NOT done until `verify_macro` passes: the live gate is what catches
background-input apps (it auto-learns `"delivery": "foreground"`), wrong-key
assumptions, and drifted anchors. Never write an assert that would pass even
if the task failed (asserting always-visible chrome proves nothing).

## Capturing a Codex native computer-use fallback

Use this when native computer use is the right first-run executor but the
workflow is worth repeating:

1. Call `begin_capture` with a short name, description, and `when_to_use`.

From milind-soni/axstream, MIT.