unix_computer_use

Files (2)

• SKILL.md
• plugin.py

SKILL.md (first lines)

# Unix Computer Use

This bundled extension exposes a reviewable computer-use substrate for local
macOS/Linux desktops and explicitly configured remote targets. **All actions are
for supervised, low-risk workflows under human or benchmark observation.** It is
offered and on by default on macOS and Linux (`os`); Windows desktops use
`windows_computer_use`. A previously enabled Unix remote skill stays available
on Windows through later OFF/ON. Actor-less legacy ON is retained too: manual
and early automatic choices cannot be distinguished. This adds no local Windows backend.

Tools:

- `capabilities` reports the platform, display-session type, and detected
  screenshot/input backends plus the active connection/backend.
- `list_connections`, `add_connection`, `test_connection`, `activate_connection`,
  and `use_local` manage the local/remote connection registry in skill state.
- `screenshot` captures the desktop, downscales the image to fit WXGA
  (1280x800, configurable) and persists the exact image-to-input coordinate
  transform. Pass coordinates read off the returned image directly to the
  input tools — they are remapped automatically. The result carries the typed
  `auto_attach_image` field, so the host attaches the image to the
  conversation in the same round (v6.81.1) — no `view_image` call is needed
  for a fresh capture; the returned `path` stays `view_image`-readable for
  re-viewing later.
- `click` (left/right/middle, double/triple), the thin aliases
  `double_click`/`triple_click` (models call these names unprompted), `move`,
  `left_click_drag`, `mouse_down`/`mouse_up`, `cursor_position`, `type_text`,
  `key`, `hold_key`, `scroll`, and `wait` execute input through platform tools
  when available. Pointer coordinates pass through one normalizer that accepts
  the malformations models actually emit (the pair packed into `x` with `y`
  absent or duplicated) and fails loudly on anything ambiguous.
- `window_list` lists visible windows/processes where the platform exposes a
  lightweight backend.
- `ax_tree` returns a set-of-marks accessibility snapshot of the frontmost
  window on macOS: numbered interactive elements (role, title, center
  coordinates in input space — click them with `raw=true`). It degrades
  honestly to a process/window list when the AX walk fails or on Linux.
- `remote_exec` runs a shell command on the active remote backend only; it
  refuses on local backend. Use ordinary Ouroboros shell/file tools for local
  work.

Coordinate contract:

- Input tools accept coordinates in the LAST screenshot's image space and
  remap them through the stored transform; `raw=true` bypasses remapping.
- macOS input consumes LOGICAL points while screenshots are physical pixels;
  the stored transform already folds the Retina scale in. On multi-display
  Macs the scale is approximate (`approx` flag) — prefer `ax_tree` marks.

Input receipts:

- A successful input call returns `observed_effect: true`: the input command
  ran and reported success, on every backend. It proves dispatch, not that the
  screen changed (macOS TCC caveat below) or that the task's goal was met; the
  host's task result review reads it as work the task produced.
- An input of several steps that fails after earlier steps succeeded returns
  `ok: false` with `steps_done` and keeps the mark. A failed or timed-out first
  step, a refusal, reads, `wait` and `remote_exec` carry none.

Backend matrix (honest limitations):

- macOS: `screencapture` + `cliclick` + `osascript`; `sips` for downscaling.
  `scroll` is **unsupported** (cliclick has no scroll-wheel command — page via
  `key` with `page-down`/`page-up`); `middle` click and non-left
  `mouse_down`/`mouse_up` are unsupported via cliclick. `hold_key` holds
  PURE-MODIFIER combos only (`cmd`, `cmd+shift`, ...): cliclick `kd:`/`ku:`
  accept only modifiers and `kp:` cannot hold — non-modifier holds report an
  honest error. In `key` combos the base key is tapped while modifiers are
  held (`cmd+s` = hold cmd, tap s). TCC permission state (Screen Recording /
  Accessibility) is NOT verified: tools exit 0 even when denied
  (wallpaper-only capture / dropped input) — ensure grants.
- Linux X11: `xdotool` (input), `gnome-screenshot`/`scrot` (capture),
  `wmctrl` (windows), ImageMagick (downscaling). Function keys and
  case-sensitive keysyms are supported (`f5`→`F5`, `XF86AudioPlay` as-is).
- Linux Wayland: `ydotool` (pointer + typing; requires a running `ydotoold`)
  or `wtype` (typing only), `grim` or `gnome-screenshot` (capture). `xdotool`
  does NOT work on Wayland. `key` and `hold_key` are **unsupported** on
  Wayland (`ydotool key` takes raw keycodes only — combos would silently
  no-op, which this skill refuses to fake); use `type_text` for text.
  Pointer press/release uses ydotool button masks (`0x40`/`0x80`).
- Remote OSWorld HTTP: `screenshot` uses `GET <target>/screenshot`; input and
  `remote_exec` use `POST <target>/execute`, matching OSWorld's in-VM
  `pyautogui` channel. The runner writes an `osworld_http` connection before
  each task. Honesty guarantees on this backend: `scroll` clicks are 1:1 wheel
  detents (not multiplied); non-ASCII `type_text` is pasted via the in-VM
  clipboard (`pyperclip` + `ctrl+v`, ASCII `typewrite` fallback) so Unicode is
  not silently dropped; every action's `ok` reflects the guest process
  `returncode`/`status` (HTTP 200 alone is not success); `key` `backspace`
  presses BackSpace and `fwd-delete` presses forward Delete; screenshots are
  capped at 20 MB and PNG-validated before use.
- Remote SSH macOS: `screenshot` uses remote `screencapture` + `scp`; input uses
  a remote input helper path. The remote Mac must have Screen Recording and
  Accessibility grants configured by the owner. This skill does not install or
  store private SSH keys.

Connection registry:

- Connections live in `data/state/skills/unix_computer_use/connections.json`.
- Default active connection is `local`.
- `add_connection` can add `osworld_http` or `ssh_macos` connection metadata, but
  it never accepts private key material.
- For SSH, configure `~/.ssh/config`, `IdentityFile`, and/or `ssh-agent`
  yourself. If auth fails, `test_connection` tells you what to fix.
- Agents should add/test/activate connections only when the user or a benchmark
  runner provides an explicit target. They should not scan for arbitrary remote
  machines.
- A disabled active connection FAILS CLOSED: action tools return an error rather
  than silently falling back to the local desktop (an action aimed at a remote
  target must never land on the host).

Permissions note: the remote backends require the `net` permission. `net` needs
no owner grant (skill grants only gate `inject_chat`/`subscribe_event`), and the
launcher names this skill among the bundled seeds that start enabled anyway; the
network is used for configured connections, which the owner, a runner or the
agent can add via `add_connection`. Agents should use only explicitly provided
targets as described above. An owner who turns the skill off keeps it off across updates.

The skill intentionally does not hide OS permission requirements, and missing
backends produce explicit errors with the capability report instead of

From razzant/ouroboros, MIT.

More from razzant/ouroboros