macos-ui-testing

Files (2)

• SKILL.md
• osascript-patterns.md · references/

SKILL.md (first lines)

# macOS UI Testing (Background-Safe)

Run the `/peekaboo` skill first to load full Peekaboo CLI reference. This skill adds background-safe patterns on top.

## Background Safety Rules

Simulated keystrokes (`peekaboo type`, `peekaboo paste`, `peekaboo hotkey`) go to the **frontmost app**, not the target app. Always use background-safe alternatives.

| Action | Background-Safe | Command |
|---|---|---|
| Read UI tree | Yes | `peekaboo see --app APP --json` |
| Screenshot | Yes | `peekaboo see --window-id WID --no-elements` |
| Click element | Yes | `peekaboo click --no-auto-focus --on ELEM --app APP` |
| Set text value | Yes* | `osascript` with `set value of` |
| Read text value | Yes | `osascript` with `get value of` |
| Click button | Yes | `osascript` with `click button` |
| Type keystrokes | **NO** | Lands in frontmost app |
| Paste (Cmd+V) | **NO** | Lands in frontmost app |
| Hotkeys | **NO** | Lands in frontmost app |

*`set value` updates the accessibility layer but **does not trigger SwiftUI `@State` bindings**. Use `keystroke` inside a single osascript block when SwiftUI binding updates are needed (see Multi-Step Interaction Sequences below).

Run the procedures below in the `/lume-vm` VM, so the user's Mac is not disturbed: they run there through `vm.sh gui` against a pushed app bundle, and Ducko's scripts through `vm.sh ui`. Run them on the user's Mac only when the user asks for that, and even then tell the user before a step that moves the pointer, brings a window to the front or switches the system appearance.

## Check for a Locked Screen

While the session screen is locked, window accessibility elements report the role `AXApplication`, controls inside windows can't be resolved, and window captures fail (`screencapture -l` reports "could not create image from window"). Read the lock state before starting the workflow:

```bash
ioreg -n Root -d1 -a | grep -A1 CGSSessionScreenIsLocked
```

`<true/>` on the line after the key means locked. No output means unlocked, because the key is absent then. An `ioreg` error, such as "can't open file" inside a sandbox, leaves the lock state unknown. When the screen is locked, record the GUI check as inconclusive instead of working around it.

## A Glitch in an App's First Frames

In the observed launch, the first accessibility answer came about half a second after the process started, so a layout glitch in an app's first frames is over before the first read. A screen recording is no substitute (see Step 4). Log the geometry from inside the app instead: put a logging probe into a scratch copy of the code and launch that. To check a fix, put the same probe into scratch copies of the unfixed and the fixed code and compare the logs.

## Workflow

### Step 1: Launch the App

```bash
# For SwiftPM-built apps:
swift run AppName &>/dev/null &
APP_PID=$!
sleep 3

# For installed apps:
open -a "AppName"
```

### Step 2: Verify the App Is Running

```bash
peekaboo app list | grep -i AppName
```

### Step 3: Read the UI Tree

```bash
peekaboo see --app AppName --json
```

Parse the JSON output to find element IDs (`elem_N`), roles, and labels.

### Step 4: Take a Screenshot

Resolve the window ID first, then capture by ID:

```bash
peekaboo window list --app AppName
peekaboo see --window-id WID --no-elements --path /tmp/screenshot.png
```

Capture by window ID only. A screen recording, or a capture of the whole screen or a region of it, holds everything else the user has open, and shows whatever is in front, which may not be the app. `screencapture -l WID` has captured a window correctly while it was on another Space.

In top-level code of a Swift script that captures with ScreenCaptureKit, annotate the result (`let image: CGImage = try await SCScreenshotManager.captureImage(...)`), or the completion-handler overload is chosen and the result is `Void`. Touch `NSApplication.shared` before the first capture, or it asserts in `CGS_REQUIRE_INIT`.

To cut a region out of a capture, read it with `CGImageSource`, crop it with `CGImage.cropping(to:)` and write it with `CGImageDestination`. The rectangle is in pixels, and one that runs past the image is clamped without an error, so check the result's size. `sips -c H W --cropOffset 0 0` crops the center rather than the top left, and sips cannot write its temporary file inside the Bash sandbox.

### Step 5: Interact with Elements

**Click an element** (accessibility API, no focus needed):

```bash
peekaboo click --no-auto-focus --on elem_4 --app AppName
```

**Set a text field value** (background-safe, but does NOT trigger SwiftUI bindings):

```bash
osascript -e 'tell application "System Events" to set value of text field 1 of group 1 of window 1 of process "AppName" to "text"'
```

**Read a text field value**:

```bash
osascript -e 'tell application "System Events" to get value of text field 1 of group 1 of window 1 of process "AppName"'
```

**Click a button by name**:

```bash
osascript -e 'tell application "System Events" to click button "Submit" of group 1 of window 1 of process "AppName"'
```

> **Synthetic coordinate clicks don't drive SwiftUI gestures.** AppleScript `click at {x,y}` and `peekaboo click --at X,Y` do **not** trigger `.onTapGesture`/`.simultaneousGesture` or `List(selection:)` selection. A tap gesture fires only for a real user click, and a `Button` also fires on an accessibility **`AXPress`** (`peekaboo click --on ELEM` from `see --json`). An AXPress on a list row, which a System Events `click` on it is, leaves the selection unchanged too: set `AXSelected` to true on the row's enclosing `AXRow` instead, which does select it (observed on macOS 27). UI relying on tap gestures **can't be confirmed by synthetic clicks** — treat such a result as a signal, not proof, and hand the app to the user when only real clicks suffice.
>
> Peekaboo targets apps by **display name** (`--app Ducko`), not the executable name (`DuckoApp`).
>
> Peekaboo 4 replaced `click --coords` with `--at X,Y`. In the background, `--at` coordinates are relative to the window of a `--snapshot` from a fresh `see --window-id WID`. `--foreground` focuses the target app instead, so use it only in the `/lume-vm` VM.

### Step 6: Verify Results

Take another screenshot or read values back to confirm the interaction succeeded.

### Step 7: Stop the App

From ducko-im/ducko, GPL-3.0.

More from ducko-im/ducko