gui-control

Files (3)

• SKILL.md
• gui.py · scripts/
• macro_store.py · scripts/

SKILL.md (first lines)

# GUI Control

Desktop automation skill. Uses a vision loop (screenshot → Claude --image → JSON action → execute)
to automate GUI tasks, and saves completed sequences as macros for instant replay.

## JSON Action Protocol

When you are operating as the GUI agent in the vision loop, you MUST respond with a **single JSON
object on stdout** — nothing else. The caller reads your full stdout and passes it to `json.loads()`.

### Supported actions

```json
{ "action": "click",        "x": 960,  "y": 540 }
{ "action": "right_click",  "x": 960,  "y": 540 }
{ "action": "double_click", "x": 960,  "y": 540 }
{ "action": "type",         "text": "hello world" }
{ "action": "key",          "keys": "ctrl+c" }
{ "action": "scroll",       "amount": 3 }
{ "action": "move",         "x": 960,  "y": 540 }
{ "action": "navigate",     "url": "https://example.com" }
{ "action": "wait",         "ms": 500 }
{ "action": "done" }
```

### Rules for the vision loop agent

1. Look at the screenshot carefully before deciding.
2. Emit **exactly one** JSON action per turn.
3. When the task is fully complete, emit `{"action": "done"}` — this saves the macro.
4. If the task is impossible or you are stuck after several attempts, emit `{"action": "done"}` with
   a note in a `"note"` field (the macro will NOT be saved on error exit).
5. Never emit explanatory prose — only the JSON object. The caller cannot parse prose.
6. Include `"x"` and `"y"` for all pointer actions. Coordinates are absolute screen pixels.
7. For `"key"`, separate modifiers and keys with `+` (e.g. `"ctrl+shift+t"`, `"alt+F4"`).

### Example turn sequence

```
[system] Task: "Open a new terminal tab"
[step 1] Screenshot → Claude sees terminal window
         → { "action": "key", "keys": "ctrl+shift+t" }
[step 2] Screenshot → Claude sees new tab opened
         → { "action": "done" }
→ Macro saved as gui-macros/default/open-a-new-terminal-tab
```

## Computer-Use Loop

```
gui run "<task>"
    │
    ├─ macro_store.find(task) → hit? → replay without screenshots (fast path)
    │
    └─ miss → vision loop:
         for step in 1..max_steps:
             screenshot → /tmp/dexter_gui_{ts}_{step}.png
             claude -p "{system_prompt + task + history}"
                    --image /tmp/dexter_gui_{ts}_{step}.png
                    --dangerously-skip-permissions
             parse JSON from stdout
             if action == done → break → save macro
             execute action via pyautogui
             with 25% probability → spot-check screenshot:
                 claude asks "did that action succeed?"
                 if no → log warning, Claude adapts next step
         if max_steps reached → print error, exit 1, do NOT save
```

## Macro Memory Layer

Completed task sequences are stored in Engram with topic key `gui-macros/default/{slug}`.
On subsequent runs of the same task, the vision loop is skipped entirely — steps replay directly.

- **Save**: automatic after `action: done`
- **List**: `gui macro list`
- **Replay**: `gui macro replay <slug>`
- **Delete**: `gui macro delete <slug>`
- **Fallback**: `~/.dexter/gui-macros.json` when Engram CLI is unavailable

## Setup Requirements

```bash
# Required: pyautogui for mouse/keyboard automation
pip install pyautogui

# Linux X11: scrot for screenshots (preferred)
sudo apt install scrot
# or: imagemagick for `import -window root` fallback
sudo apt install imagemagick

# Linux: xdotool (optional, for extra input capabilities)
sudo apt install xdotool

# macOS: screencapture is built-in (no install needed)
# Windows: PIL/Pillow for screenshots
pip install Pillow
```

## Verification

When the user says **"no se envió"**, **"no funcionó"**, **"qué pasó"**, **"verificá"**,
**"it didn't work"**, **"check the screen"** — run `gui verify` immediately:

```bash
python3 skills/gui-control/scripts/gui.py verify "did the email send?"
python3 skills/gui-control/scripts/gui.py verify "is the file saved?"
python3 skills/gui-control/scripts/gui.py verify  # general OK check
```

Claude takes a screenshot and returns `{"ok": true/false, "note": "what it sees"}`.
No action is executed — this is read-only.

During a `gui run`, random spot-checks happen automatically (~25% of steps).
Disable with `--verify-rate 0`. Increase vigilance with `--verify-rate 1.0`.

## Usage Examples

```bash

From EmilioUgarte65/Dexter, Apache-2.0.

More from EmilioUgarte65/Dexter