computer-use

Files (1)

• SKILL.md

SKILL.md (first lines)

---
name: computer-use
description: "Drive the desktop background-first; escalate on signal."
version: 2.1.0
author: Francesco Bonacci (f-trycua), Hermes Agent
license: MIT
platforms: [macos, windows, linux]
metadata:
  hermes:
    tags: [computer-use, desktop, automation, gui, cross-platform]
    category: desktop
    related_skills: []
---

# Computer Use (universal, any-model, cross-platform)

You have a `computer_use` tool that drives the user's desktop in the
**background** — your actions do NOT move the user's cursor, steal
keyboard focus, or switch virtual desktops / Spaces. The user can keep
typing in their editor while you click around in a browser in another
window. This is the opposite of pyautogui-style automation.

Everything here works with any tool-capable model — Claude, GPT, Gemini,
or an open model on a local OpenAI-compatible endpoint. There is no
Anthropic-native schema to learn.

Hermes drives [cua-driver](https://github.com/trycua/cua) under the hood.
This skill teaches the Hermes `computer_use` **action vocabulary**, which is
NOT the driver's raw MCP vocabulary. Call the actions documented below and
never the driver's tools by name: `capture` is a Hermes action that maps to
the driver's `get_window_state`; `element=N` is a Hermes argument that the
wrapper translates into the driver's `element_token` handle. If you see a
driver-side error mentioning `snapshot_id`, `element_token`, or "no reviewed
risk classification", you (or a stale description) called the raw driver
vocabulary — go back to the actions below.

## The canonical workflow

**Step 1 — Capture first.** Almost every task starts with:

```
computer_use(action="capture", mode="som", app="<the app you're driving>")
```

Returns a screenshot plus an indexed element list like:

```
#1  AXButton 'Back' @ (12, 80, 28, 28) [Chrome]
#2  AXTextField 'Address bar' @ (80, 80, 900, 32) [Chrome]
#7  Link 'Sign In' @ (900, 420, 80, 24) [Chrome]
...
```

The `#N` index is the ONLY element handle you use. Behind it the wrapper
keeps this snapshot's opaque per-element token and sends it with every
`element=N` action, so a click on an index from a superseded snapshot is
refused explicitly (`stale`) instead of landing on the wrong control.
Re-capture after anything that changes the screen; indices do not survive it.

The role names match the host platform's accessibility framework
(`AXButton` on macOS, `Button` on Windows UIA, `push button` on Linux
AT-SPI) — treat them as labels, not as strict types.

**Step 2 — Click by element index.** This is the single most important
habit:

```
computer_use(action="click", element=7)
```

Much more reliable than pixel coordinates for every model. Claude was
trained on both; other models are often only reliable with indices.

**Step 3 — Verify.** After any state-changing action, re-capture. You
can save a round-trip by asking for the post-action capture inline:

```
computer_use(action="click", element=7, capture_after=True)
```

## Capture modes

| `mode` | Returns | Best for |
|---|---|---|
| `som` (default) | Screenshot + indexed element list | Vision models; preferred default |
| `vision` | Plain screenshot, no elements | When you only need pixels (then click by `coordinate=`) |
| `ax` | Element list only, no image | Text-only models, or when you don't need to see pixels |

Current drivers always return the screenshot AND the tree in one call;
`mode` decides what Hermes hands back to you, not what the driver does.
There is no numbered overlay burned into the screenshot — the index list is
the map; ground on both and cross-check (the tree lies on some surfaces).

**No vision model?** If your main model can't read images (or the provider
rejects image tool results), Hermes routes the screenshot through the
auxiliary vision model and you get a text description instead of pixels.
Configure `auxiliary.vision` in `config.yaml` to pick that model, or use
`mode="ax"` and drive by element index without a screenshot at all.

## Actions

```
capture           mode=som|vision|ax   app=…  (default: current app)
click             element=N     OR     coordinate=[x, y]    button=left|right|middle
double_click      element=N     OR     coordinate=[x, y]
right_click       element=N     OR     coordinate=[x, y]
middle_click      element=N     OR     coordinate=[x, y]
drag              from_element=N, to_element=M        (or from/to_coordinate)
scroll            direction=up|down|left|right   amount=3 (ticks)
type              text="…"
key               keys="<save shortcut>" | "return" | "escape" | "<modifier>+t"
set_value         element=N  value="…"     (selects/sliders without opening the menu)
wait              seconds=0.5
list_apps
list_windows
focus_app         app="<app name>"   raise_window=false   (default: don't raise)
```

All actions accept optional `capture_after=True` to get a follow-up
screenshot in the same tool call. All actions that target an element

From Hadi-Akram-Ramadhan/hadooyy-cli, MIT.