mac-control

Files (1)

• SKILL.md

SKILL.md (first lines)

# Mac Control

Three tiers, cheapest first. Start at the lowest that fits.

| Need | Use |
|---|---|
| Local Mac, interactive Claude Code session, Pro/Max plan | **Tier 0** - the built-in `computer-use` MCP (section below). Screenshots, clicks, typing, per-app approval. Runs on the machine the session is on; not available with `-p` or in background jobs. A session started over ssh (a Herdr pane on a remote Mac) counts as local to that Mac. |
| Any Mac reachable from a shell (ssh to a headless mini, cron, background sessions) | **Tier 1** - stock macOS: `screencapture` for eyes, System Events (osascript) for hands. No install. This skill's core. |
| Heavy or repeated UI work on that machine: annotated screenshots with element IDs, semantic clicks, background delivery without stealing focus | **Tier 2** - [Peekaboo](https://github.com/steipete/Peekaboo) CLI (`brew install openclaw/tap/peekaboo`, macOS 15+, MIT). |

Browser content is never done with any of these - chrome-control (CDP) is
exact and cheaper.

## Tier 0: the built-in computer use (Claude Code)

Tools are `mcp__computer-use__*` (load them all with one ToolSearch:
query `computer-use`, max_results 30). Enable once per project via `/mcp`
→ `computer-use` → Enable. Requires an interactive session on the local Mac.

**Flow, every session:**

1. `request_access` with the list of apps you need - the user approves per
   app, per session (Finder counts as an app: desktop, Dock, Go to Folder).
   Re-request mid-task when a new app comes up. `list_granted_applications`
   tells you what you already have.
2. `screenshot` first; act; `screenshot` again to verify. Screenshots are
   auto-downscaled - never resize the display. Use `zoom` on a region when
   text is too small instead of guessing.
3. Batch a fixed sequence (`left_click`, `type`, `key`, `wait`) with
   `computer_batch` - one round-trip instead of five. Split the batch at any
   point where you must look before continuing.
4. Type with `type`; shortcuts with `key` (`cmd+s`); paste large text via
   `write_clipboard` + `key cmd+v`; `read_clipboard` to get text back out.

**App tiers** (enforced against the frontmost app, the error names the tier):
browsers → *read* (screenshots only - drive them with chrome-control);
terminals/IDEs → *click* (no typing, no right-click - shell work goes
through Bash); everything else → *full*.

**Behaviour to expect:** other apps are hidden while it works and restored
after the turn; your terminal is excluded from screenshots; one session
holds a machine-wide lock until it exits (a second session errors naming
the holder); the user can abort with Esc at any time. It refuses to
execute trades or move money - hand those to the user. Never click links
in mail/messages with it (open the URL through chrome-control instead).

**When Tier 0 is the wrong tool, even locally:**

- the target is a browser tab, a terminal, or an IDE (tier-limited);
- the action must run without hiding the user's apps or grabbing the
  session lock (a background dialog watcher, a launchd job, a cron);
- the session is `-p`/non-interactive, a subagent, or a background job;
- you need a window that is not frontmost, or a precise element by name
  rather than by pixel.

Those are Tier 1 jobs. **Mixing them in one task is normal:** Tier 0 for
the interactive look-and-click parts, a System Events one-liner (or a
spawned script like chrome-control's `cdp-allow`) for the sheet that pops
up behind it. The TCC grants differ: Tier 0's grants belong to the terminal
app running Claude Code; a Bash one-liner's grants belong to that same
terminal app locally, or to `sshd-keygen-wrapper` over ssh (table below).
Granting both once per machine avoids the "it worked from the MCP but not
from Bash" surprise.

## Prerequisites (TCC) - one human grant per machine

macOS attributes a process's permissions to its **responsible process**,
not to the binary that made the call:

| How the code was launched | Binary TCC blames |
|---|---|
| ssh (any ssh-descended shell, incl. an agent started over ssh) | `/usr/libexec/sshd-keygen-wrapper` |
| a terminal app on the machine | that terminal app (Terminal, iTerm, Ghostty...) |
| a launchd agent/daemon | the executable in `ProgramArguments` (or its .app) |
| an MCP server spawned over stdio | the client app that spawned it - adding the server binary itself does nothing |

Grant that binary, once, in System Settings → Privacy & Security:

- **Screen Recording** (Tahoe names the pane *Screen & System Audio
  Recording*) - needed by `screencapture`. Without it: `could not create
  image from display`, or a wallpaper-only/black image, no other error.
- **Accessibility** - needed for every synthetic click/keystroke and for
  reading other apps' UI trees. Without it: `-25211` / "not allowed
  assistive access".
- **Automation** - per (source app, target app) pair, granted by clicking
  Allow on the "X wants to control System Events" dialog that appears on the
  machine's display the first time. Denied → `-1743`; `AppleEvent timed out
  (-1712)` from a fresh ssh shell means that dialog is up on the display
  right now, unanswered. A prompt that times out or whose caller is killed
  is recorded as **Deny** and never shown again - flip it on afterwards in
  Privacy & Security → **Automation** → the source binary → target app
  (verify in the user TCC db: `kTCCServiceAppleEvents` rows, `auth_value` 2
  = allowed, 0 = denied). When that pane's toggles refuse to move (seen on
  26.x for path-identified clients like `sshd-keygen-wrapper`), clear the
  rows and re-prompt instead: `tccutil reset AppleEvents` (no bundle-id
  form exists for path clients; it wipes every app's Automation grants, so
  check the table first), then fire the request with a long timeout
  (`osascript -e 'with timeout of 600 seconds' -e 'tell application
  "System Events" to get name of every process' -e 'end timeout'`) so the
  dialog stays up until a human clicks Allow over Screen Sharing.

Rules that bite:

- Hidden paths (`/usr/libexec/...`, Homebrew) are added with **+** → ⌘⇧G.
  macOS 26.1-26.2 had a bug hiding CLI binaries from these panes entirely;
  dragging the binary from Finder onto the list still granted it. Fixed in
  26.3.
- A grant belongs to the binary's code identity: `brew upgrade` or a
  reinstall of an ad-hoc-signed tool changes the hash and macOS asks again.
  Stale rows can show "on" for a path that no longer exists - `tccutil
  reset Accessibility <bundle-id>` / `tccutil reset ScreenCapture
  <bundle-id>` and re-grant (omitting the bundle id wipes every app).
- A running process keeps its cached verdict: after granting, restart the
  responsible process (Screen Recording often needs a full quit).
- The ssh path works when the ssh user is the same user logged into the
  console session (auto-login on a headless machine). Some guides claim ssh
  grants never work; in practice FDA/Accessibility/Screen Recording on
  `sshd-keygen-wrapper` are what CI boxes rely on.
- The display must be awake and unlocked: `caffeinate -u -t 2` wakes it; the

From alexjmiller5/agent-config-public, MIT.

More from alexjmiller5/agent-config-public