macro-agent

Files (93)

• SKILL.md
• 123_20260115_125057.png · data/captures/
• Abril_gaspar_hija_whatsapp_web_20260116_084957.png · data/captures/
• ask_google_gemini_input_20260116_161523.png · data/captures/
• ask_google_gemini_input_text_20260116_164737.png · data/captures/
• brave_icon_open_118_1061.png · data/captures/
• chatgpt_icon_tab_20260120_224431.png · data/captures/
• chatgpt_input_ask_anything_20260120_224747.png · data/captures/
• chatgpt_send_question_message_action_20260120_224916.png · data/captures/
• chrome_icon_239_1059.png · data/captures/
• conspiraciones_vergas_grupo_web_whatsapp_20260116_090255.png · data/captures/
• current_screen.png · data/captures/
• debug.png · data/captures/
• debug_gemini_menu.png · data/captures/
• escribir_mensaje_whatsapp_800_1001.png · data/captures/
• estado_actual.png · data/captures/
• firefox_icon_bottom_panel_deactivated_198_1059.png · data/captures/
• google_gemini_hamburguer_menu_20260116_161344.png · data/captures/
• google_gemini_send_message_icon_action_20260116_161625.png · data/captures/
• google_gemini_tab_brave_20260116_161237.png · data/captures/
• google_gemini_tab_brave_20260116_164312.png · data/captures/
• google_gemini_temporal_chat_icon_20260116_161419.png · data/captures/
• grupo_amigos_x_20260116_102049.png · data/captures/
• home_icon_button_action_20260120_225325.png · data/captures/
• home_icon_button_action_20260120_230520.png · data/captures/
• pantalla_actual.png · data/captures/
• post_message_button_publish_20260120_225158.png · data/captures/
• post_message_button_publish_20260120_232534.png · data/captures/
• search_bar_youtube_input_text_20260120_225522.png · data/captures/
• send_whatsapp_message_1879_1003.png · data/captures/
• skillrunnerinput.png · data/captures/
• test_base64.png · data/captures/
• user_screenshot.png · data/captures/
• vscode_copilot_input.png · data/captures/
• vscode_icon_active_window_159_1062.png · data/captures/
• vscode_input_copilot_20260115_153100.png · data/captures/
• vscode_input_copilot_20260115_161212.png · data/captures/
• web_whatsapp_icon_20260115_131447.png · data/captures/
• web_whatsapp_icon_with_messages_1478_20.png · data/captures/
• web_whattsapp_icon_browser_1477_20.png · data/captures/
• whatsapp_current.png · data/captures/
• whatsapp_marco_primo_218_656.png · data/captures/
• whatsapp_ross_173_353.png · data/captures/
• x_postear_big_left_bar_20260121_072340.png · data/captures/
• x_social_network_textinput_20260120_225026.png · data/captures/
• x_social_network_textinput_20260120_230825.png · data/captures/
• x_social_network_textinput_20260121_072615.png · data/captures/
• x_tab_icon_browser_20260120_224527.png · data/captures/
• x_tab_icon_browser_20260120_230441.png · data/captures/
• youtube_search_bar_icon_action_button_20260121_075524.png · data/captures/
• youtube_tab_icon_browser_20260120_224614.png · data/captures/
• elements.json · data/
• question_state.json · data/
• gemini.csv · data/questions/
• brave_chrome_brave.json · data/sequences/
• chatgpt_send_message.json · data/sequences/
• gemini_temporal_chat.json · data/sequences/
• message_marco_jarvis.json · data/sequences/
• post_to_x.json · data/sequences/
• test_message_ross.json · data/sequences/
• …and 33 more

SKILL.md (first lines)

# Macro Agent

Desktop automation and UI control skill with **image recognition**.

## 🚨 CRITICAL: How to Handle User Requests

**BEFORE doing ANY action, ALWAYS check if a sequence exists for it:**

1. **FIRST** run `seq-list` to see available sequences
2. **LOOK** for sequences that match the user's intent (e.g., `whatsapp_send_marco` for "send message to Marco")
3. **IF sequence exists**: Use `seq-run <sequence_name>` then add your custom actions (write message, press enter)
4. **IF NO sequence exists**: Then use individual commands

### Common Workflow: Send Message to Contact

When user says "send message to X" or "envía mensaje a X":

```
1. seq-list                          # Check available sequences
2. seq-run whatsapp_send_<contact>   # Run the messaging sequence  
3. write "<message>"                 # Type the message
4. press enter                       # Send it
```

**NEVER** use `hotkey super` or manual navigation when a sequence exists!

### Available Sequences (check with seq-list)

The user has pre-configured sequences for common tasks. Always check them first!
- `whatsapp_send_ross` - Opens WhatsApp and selects Ross contact
- `whatsapp_send_marco` - Opens WhatsApp and selects Marco contact
- Other sequences may exist - always run `seq-list` first!

## 🎯 How Element Detection Works

When using `click-on` or `move-to`, the agent **ALWAYS** uses image recognition:
1. Searches for element image on screen (template matching)
2. If not found → **FAILS** (no fallback to coordinates)

This ensures elements are found dynamically based on their actual position.

**Output includes `method` field:**
- `image` = Found by template matching ✅
- `not_found` = Image not visible on screen ❌

**If element not found:** You need to capture it first with `region-capture`.

## ⚠️ Important

**NO "navigate" command exists**. To navigate:
1. `find <name>` - Search for element info
2. `click-on <name>` - Click using image recognition (ALWAYS)

## Usage

```bash
python ~/.copilot/skills/macro-agent/macro_agent.py <command> [args]
```

## Commands Reference

| Action | Command | Example |
|--------|---------|---------|
| Search element | `find <name>` | `find brave` |
| Search text | `search <text>` | `search save` |
| Click element | `click-on <name>` | `click-on brave` |
| Click coords | `click X Y` | `click 500 300` |
| Move to element | `move-to <name>` | `move-to button` |
| Move to coords | `move X Y` | `move 500 300` |
| Write text | `write <text>` | `write "hello"` |
| Press key | `press <key>` | `press enter` |
| Hotkey | `hotkey <keys>` | `hotkey ctrl c` |
| Scroll | `scroll N` | `scroll -3` |
| Screenshot | `screenshot <name>` | `screenshot test` |
| Region capture | `region-capture` | `region-capture` |

## Sequence Commands

| Command | Description |
|---------|-------------|
| `seq-create <name>` | Create new sequence |
| `seq-add <name> "<action>"` | Add action to sequence |
| `seq-show <name>` | View sequence |
| `seq-run <name>` | Execute sequence |
| `seq-list` | List all sequences |
| `seq-delete <name>` | Delete sequence |

## Output

JSON with:
- `success`: true/false
- `action`: Command executed
- `target`: Element name (if applicable)
- `coordinates`: {x, y} position
- `message`: Result description

## Data Locations

- **Elements**: `~/.copilot/skills/macro-agent/data/elements.json` (elemento definitions)
- **Captures**: `~/.copilot/skills/macro-agent/data/captures/` (template images)
- **Sequences**: `~/.copilot/skills/macro-agent/data/sequences/` (action sequences)

## Examples

### Find and Click App
```bash
python ~/.copilot/skills/macro-agent/macro_agent.py find chrome
python ~/.copilot/skills/macro-agent/macro_agent.py click-on chrome
```

### Type and Submit
```bash
python ~/.copilot/skills/macro-agent/macro_agent.py write "search query"
python ~/.copilot/skills/macro-agent/macro_agent.py press enter
```

### Keyboard Shortcut
```bash
python ~/.copilot/skills/macro-agent/macro_agent.py hotkey ctrl shift s

From ignaciosua/copilot-skills-repo, MIT.

More from ignaciosua/copilot-skills-repo