computer-use

Files (1)

• SKILL.md

SKILL.md (first lines)

# Computer Use (universal, any-model, cross-platform)

You have a `computer_use` tool that drives the user's desktop in the
**background** — your actions do NOT move the user's cursor, steal
keyboard focus, or switch virtual desktops / Spaces. The user can keep
typing in their editor while you click around in a browser in another
window. This is the opposite of pyautogui-style automation.

Everything here works with any tool-capable model — Claude, GPT, Gemini,
or an open model on a local OpenAI-compatible endpoint. There is no
Anthropic-native schema to learn.

Hermes drives [cua-driver](https://github.com/trycua/cua) under the hood.
This wrapper skill teaches the Hermes `computer_use` workflow and action
vocabulary. Call the actions documented below instead of raw cua-driver MCP
tools. For driver internals and platform-specific behavior, follow the Cua
skill installed by `cua-driver skills install`. Hermes autodetection is a
planned cua-driver follow-up, so currently point Hermes at the resulting
`~/.cua-driver/skills/cua-driver` directory or symlink it into your skill space.

## The canonical workflow

**Step 1 — Capture first.** Almost every task starts with:

```
computer_use(action="capture", mode="som", app="<the app you're driving>")
```

Returns a screenshot with numbered overlays on every interactable
element AND an AX-tree index like:

```
#1  AXButton 'Back' @ (12, 80, 28, 28) [Chrome]
#2  AXTextField 'Address bar' @ (80, 80, 900, 32) [Chrome]
#7  Link 'Sign In' @ (900, 420, 80, 24) [Chrome]
...
```

The role names match the host platform's accessibility framework
(`AXButton` on macOS, `Button` on Windows UIA, `push button` on Linux
AT-SPI) — treat them as labels, not as strict types.

**Step 2 — Click by element index.** This is the single most important
habit:

```
computer_use(action="click", element=7)
```

Much more reliable than pixel coordinates for every model. Claude was
trained on both; other models are often only reliable with indices.

**Step 3 — Verify.** After any state-changing action, re-capture. You
can save a round-trip by asking for the post-action capture inline:

```
computer_use(action="click", element=7, capture_after=True)
```

## Capture modes

| `mode` | Returns | Best for |
|---|---|---|
| `som` (default) | Screenshot + numbered overlays + AX index | Vision models; preferred default |
| `vision` | Plain screenshot | When SOM overlay interferes with what you want to verify |
| `ax` | AX tree only, no image | Text-only models, or when you don't need to see pixels |

## Actions

```
capture           mode=som|vision|ax   app=…  (default: current app)
click             element=N     OR     coordinate=[x, y]    button=left|right|middle
double_click      element=N     OR     coordinate=[x, y]
right_click       element=N     OR     coordinate=[x, y]
middle_click      element=N     OR     coordinate=[x, y]
drag              from_element=N, to_element=M        (or from/to_coordinate)
scroll            direction=up|down|left|right   amount=3 (ticks)
type              text="…"
key               keys="<save shortcut>" | "return" | "escape" | "<modifier>+t"
wait              seconds=0.5
list_apps
focus_app         app="<app name>"   raise_window=false   (default: don't raise)
```

All actions accept optional `capture_after=True` to get a follow-up
screenshot in the same tool call. All actions that target an element
accept `modifiers=[…]` for held keys.

The input actions (`click`, `double_click`, `right_click`, `middle_click`,
`drag`, `scroll`, `type`, `key`) also accept `delivery_mode`. The optional
`bring_to_front=True` request invokes a separately approved standalone focus
tool before foreground input; it is never an input-action property.

## The verify → escalate ladder (background-first)

cua-driver delivers input in the **background** by default (no focus steal),
but that is the first rung, not the only one. Every input action returns a
structured verdict; read it and climb only when the driver tells you to.

Returned fields (present when the driver supports them):
- `effect`: `"confirmed"` (driver read the result back — done), `"unverifiable"`
  (delivered, but confirm it yourself by re-capturing), or `"suspected_noop"`
  (ran but almost certainly did nothing).
- `escalation`: `{recommended: "px" | "foreground" | "page", reason}` — present
  only when there's a next rung to try.
- `code`: a structured refusal like `"background_unavailable"` or
  `"foreground_unsupported"`.
- `verified`: `true` only on AX read-back.

Walk it in order:

1. **Element, background (default).** `click(element=N)`. If `effect:"confirmed"`,
   you're done.
2. **Fresh verification.** `effect:"unverifiable"` means inspect a fresh
   capture/state before any retry. Do this even when `escalation.recommended`
   is present; it is advisory, not proof that successful input should repeat.
3. **Pixel, background.** After `effect:"suspected_noop"` or a structured
   refusal recommends `"px"` (or a `degraded` capture has no elements), click
   by `coordinate=[x,y]` instead of `element`.

From merllinsbeard/merlin-CTO, MIT.

More from merllinsbeard/merlin-CTO