macbeth

Files (1)

• SKILL.md

SKILL.md (first lines)

# Macbeth — Computer Use for macOS

Use this skill whenever you automate a Mac app through Macbeth MCP tools. App-specific skills (Safari, Mail, Electron, …) add quirks on top of this baseline — load those after this one when the target app has one.

Call `load_skill` with **no arguments** (or `name: "macbeth"`) to reload these instructions.

## Mental model

1. **Discover** — `list_apps` / `list_windows` / `query_tree`
2. **Act** — `click` / `fill` / `select_menu_item` / `wait_for`
3. **Verify** — `get_element` / `query_tree` / `screenshot` / `extract_text`
4. **Fall back** — menus → keyboard → screenshot/OCR when Accessibility is thin

Prefer structured Accessibility actions over screenshots and keystrokes. Prefer menus over `press_key` (menus do not steal focus).

## Output format

Tool **results** are YAML; tool **arguments** remain JSON (MCP input schemas are JSON Schema). Read fields by key path, not by line position — paths and titles never get hard-wrapped.

## Permissions

- **Accessibility** — required for almost everything. If every app is not-connectable with AX `-25211`, tell the user to grant Accessibility to the host that launched Macbeth, then restart the agent.
- **Screen Recording** — only for `screenshot` and window-based `extract_text`. OCR of supplied PNG `data` does not need it.
- Diagnose with `npx macbeth doctor` in a terminal when tools fail mysteriously.

## First moves

```jsonc
// What is running and connectable?
{ "name": "list_apps" }

// Is app X open, and what windows does it show? (no AX connect required)
{ "name": "list_windows", "arguments": {} }
{ "name": "list_windows", "arguments": { "app": "Unity" } }

// Menu-bar / LSUIElement apps (Übersicht, Raycast, iTerm2) are omitted from
// list_apps. Discover them with list_windows, then pass ownerName or ownerPid.

// Inspect UI (connects automatically — no prior connect_app)
{ "name": "query_tree", "arguments": { "app": "Finder", "maxDepth": 5 } }
```

### Do you need `connect_app`?

**No.** Every app-taking tool connects on its own from `app` (fuzzy name, PID, or `h_N` app handle).

Call `connect_app` only to:

- preflight Accessibility reachability
- see how a fuzzy name resolved (`matchKind`, aliases, bundle id, `runtime`)
- warm an Electron tree with a longer `readyTimeoutMs`

Reuse the returned app handle:

```jsonc
{ "name": "connect_app", "arguments": { "name": "Slack", "readyTimeoutMs": 5000 } }
// → "Pass app: \"h_3\" to any other tool…"
{ "name": "query_tree", "arguments": { "app": "h_3" } }
```

## Locators (`query`)

Most action tools take either:

- `query` — a chain of steps, each matching descendants of the previous match (recursive; skip intermediate containers), or
- `handleId` — a raw `h_N` from `query_tree` / `get_element` / `read_form`

Each step may set `role`, `title`, `identifier`, `titlePattern` (regex), and `index` (0-based when several match).

```jsonc
{
  "app": "TextEdit",
  "query": [
    { "role": "window" },
    { "role": "text_area" }
  ]
}
```

```jsonc
{
  "app": "Safari",
  "query": [
    { "role": "window" },
    { "role": "web_area" },
    { "role": "link", "titlePattern": "Sign.?in", "index": 0 }
  ]
}
```

Common roles: `window`, `button`, `text_field`, `text_area`, `checkbox`, `radio`, `menu`, `menu_item`, `toolbar`, `scroll_area`, `table`, `row`, `cell`, `group`, `dialog`, `link`, `heading`, `web_area`, `static_text`, `slider`, `pop_up_button`.

Always `query_tree` before inventing locators. Keep `maxDepth` modest (4–6) on huge trees; raise it only for the subtree you care about. When a `query_tree` result shows a `[truncated: …] re-query with handleId h_X` marker, pass that `handleId` back with a higher `maxNodes` to drill into the named subtree (root resets to depth 0 under the handle, so the same `maxDepth` reaches deeper).

## Actions

| Goal | Tool | Notes |
|---|---|---|
| Click a control | `click` | Auto-waits. `strategy`: `auto` (default) \| `ax` \| `mouse` |
| Set a text field | `fill` | Auto-waits. `strategy`: `auto` \| `ax` \| `keyboard` |
| Wait for UI | `wait_for` | `exists` (default), `value_equals`, `value_changes`, `enabled` |
| Read one element | `get_element` | role, title, value, enabled, focused |
| Read form controls | `read_form` | Labels + values for a panel/subtree |
| Native menu | `select_menu_item` | `menuPath: ["File", "Save"]` — no focus steal |
| Compact menus only | `list_menu_bar` | Usually unnecessary — `query_tree` already includes menus |
| Keystroke | `press_key` / `press_keys` | **Last resort** — activates the target app |

### Click / fill strategies

- Leave `strategy` unset (`auto`) unless something fails.
- Electron / React fields: `auto` fill synthesizes keystrokes so the framework sees input. If a value “sticks” in AX but the app ignores it, use `"keyboard"`. Avoid forcing `"ax"` on Electron.
- Canvas / geometry-only UIs: `click` with `"mouse"`. Optional `waitForIdleMs` (≤5000) avoids mid-keystroke focus steals.

### Keyboard outcomes

`press_key` / `press_keys` report `outcome`:

- `dispatched` — events entered the system stream (app may still ignore them)
- `attempted` — could not confirm dispatch

From wende/macbeth, MIT.

More from wende/macbeth