excel-background-verification

Files (4)

• SKILL.md
• axdump.swift · scripts/
• read-taskpane-setting.py · scripts/
• scratch-workbook.py · scripts/

SKILL.md (first lines)

# Background Excel end-to-end tests

Follow the acceptance policy in `docs/coding-standards.md`. The model performs the work through the taskpane; the test bridge submits prompts and independently reads the workbook.

## Set up

Run from the worktree containing the code under test. You need:

- the sideloaded manifest pointing to `https://localhost:3141/src/taskpane.html`
- trusted `cert.pem` and `key.pem` files in the worktree
- an approved current model with working authentication
- a computer-use tool with Accessibility and Screen Recording permissions

Inspect existing listeners before starting servers. Reuse the correct test setup or stop only processes you own.

```bash
RUN_DIR=$(mktemp -d /tmp/pi-excel-e2e.XXXXXX)
TOKEN=$(node scripts/background-verify-bridge-server.mjs token)

PI_BACKGROUND_VERIFY_TOKEN="$TOKEN" PI_BACKGROUND_VERIFY_HOST=localhost \
  npm run background:verify:bridge > "$RUN_DIR/bridge.log" 2>&1 &
BRIDGE_PID=$!

VITE_PI_BACKGROUND_VERIFY_URL=https://localhost:3157 \
VITE_PI_BACKGROUND_VERIFY_TOKEN="$TOKEN" \
  npm run dev > "$RUN_DIR/vite.log" 2>&1 &
VITE_PID=$!

curl --fail https://localhost:3157/health
```

Wait for server readiness before probing. Keep the bridge loopback-only and tokened; do not print or commit its token. Start `npm run proxy:https` if the configured provider route needs the local CORS proxy (the `openai-codex` route does).

### Open the pane without the foreground

Excel for Mac does not put a sideloaded add-in on the ribbon until it has been activated once in the running process, and the Home-tab "Add-ins" flyout does not render for a background app. Bootstrap through the document instead: a workbook can carry an auto-open task pane part that references the sideloaded manifest (`store="developer" storeType="Registry"`).

```bash
SCRATCH=~/Library/Containers/com.microsoft.Excel/Data/Documents/pi-e2e-scratch.xlsx
python3 .agents/skills/excel-background-verification/scripts/scratch-workbook.py create "$SCRATCH"
open -g -a "Microsoft Excel" "$SCRATCH"      # launch straight into the workbook, in the background
sleep 12; curl --fail https://localhost:3157/health   # expect one client
```

Excel opens the pane on document open and the `Open Pi` ribbon button appears for the rest of the process. The auto-open fires once per Excel process: to reload the pane (for example after switching the Vite worktree) press `Close Pi for Excel` and then `Open Pi` through `computer_use`, passing `element_index` as a string. Keep the scratch workbook inside the Excel container so the sandbox never raises a "Grant File Access" sheet; `scratch-workbook.py tag <existing.xlsx>` adds the part to a workbook Excel has saved.

If `clients` is still empty, inspect Excel with `computer_use` / `sky.get_app_state`. Check the workbook, pane, manifest URL and server before changing caches. For a pane showing a network error, use a semantic accessibility `AXPress` on Try Again after starting the server.

Use background semantic accessibility actions only. Record the foreground app and window before and after the test.

### Before you call it blocked

Everything below has looked like a foreground-only step and was worked around in the background. Report a blocker only after these are exhausted, and say which one you hit.

| Looks like | Is | Do |
|---|---|---|
| No `Open Pi` button on the ribbon after Excel relaunched; manifest is intact | Sideloaded add-ins are absent from the ribbon cache until activated once per Excel process | `scratch-workbook.py create` and `open -g` into it; the pane opens from the document and the button appears |
| Home-tab *Add-ins* flyout does not open (AXPress toggles the button, no popover window) | It does not render for a background app | Not needed; use the document bootstrap above |
| `sky.get_app_state` fails with `cgWindowNotFound`, or Excel lists `AXApplication` as its only window | Seen once right after `open -g` with no document | Wait for the launch to settle and query again; otherwise quit Excel and `open -g` straight into the scratch workbook, which exposed a normal `AXStandardWindow` |
| `computer_use` click does nothing | `element_index` was passed as a number | Pass it as a string, e.g. `"63"` |
| `setTimeout is not defined` inside `computer_use` | The JS sandbox has no timers | Sequence `sky` calls with `await`; wait in the shell (`sleep`) between calls |
| AppleScript `save workbook as` returns `Parameter error (-50)` | Path built from `POSIX path of (path to home folder)` | Pass a literal absolute path string, inside the Excel container |
| Pane cannot be reloaded after switching the Vite worktree | The document auto-open fires once per process | `Close Pi for Excel` then `Open Pi` through `computer_use` |
| Persisted state cannot be checked without trusting the pane | It can | `read-taskpane-setting.py <key>` decodes the record from WebKit's IndexedDB |
| `computer_use` returns `codex app-server exited before returning a response` on every call | The tool's own server is down; Excel is fine | Read the tree with `scripts/axdump.swift` (below); pressing buttons waits for the tool to come back |

Opening a workbook does not activate Excel: both `open -g -a "Microsoft Excel" <file>` and `osascript -e 'tell application "Microsoft Excel" to open POSIX file "…"'` leave the frontmost app unchanged.

## Run the test

Create a test chat through the actual UI. The bridge's `submitPrompt` sends message text, so `/new` and `/name` can reach the model literally rather than dispatching commands.

Define a command shorthand in the same shell:

```bash
bridge() {
  PI_BACKGROUND_VERIFY_TOKEN="$TOKEN" PI_BACKGROUND_VERIFY_HOST=localhost \
    npm run background:verify:command -- "$@"
}

bridge status
bridge selectModel '{"provider":"openai-codex","modelId":"gpt-5.6-sol"}'
```

Use a currently approved model. If health lists multiple clients, pass `--clientId <id>` to target the scratch workbook on each command.

Adapt the prompt to the changed feature. This example exercises model-driven sheet creation, writes, formulas and reads:

```bash
bridge submitPrompt \
  '{"text":"Use workbook tools to create a new sheet _pi_e2e_smoke; stop if it exists. Write 2 and 3 into A1:A2 and =SUM(A1:A2) into A3. Read back the values and formula. Leave the sheet for independent inspection. Do not change other sheets.","waitForIdle":true,"timeoutMs":120000}' \
  --timeout 130000

bridge readRange '{"address":"_pi_e2e_smoke!A1:A3"}'
```

Check actual tool execution and `lastAssistant.model` / `stopReason`. An idle runtime can also mean the request failed. Independently assert that A3 is 5 and retains `=SUM(A1:A2)`; check formatting or objects when relevant.

Clean up through the model, then verify the scratch sheet is gone:

```bash
bridge submitPrompt \
  '{"text":"Delete only _pi_e2e_smoke using your workbook tool. This scratch-sheet deletion is approved. Leave all other sheets unchanged.","waitForIdle":true,"timeoutMs":120000}' \
  --timeout 130000
bridge officeProbe
```

## Diagnose failures

| Symptom | Check |
|---|---|
| No bridge client | Pane not open: bootstrap with `scratch-workbook.py` and `open -g` into the workbook, or press `Open Pi` on the ribbon if the add-in has been activated this process; Vite URL and bridge environment configured |
| Pane does not auto-open from a second tagged workbook | Expected within one Excel process; press `Open Pi` instead, or quit and relaunch Excel into the workbook |
| Pane opens as "New Office Add-in" with an add-in error or endless loading | The webextension reference does not resolve; keep `store="developer" storeType="Registry"` and the manifest `<Id>` |
| Pane loads, model returns `Load failed` | Configured proxy is running and reachable |
| Cannot set the WebKit textarea through accessibility | Submit through the bridge |
| Unsaved workbook has no workbook ID | Identify it through the client and sheet/range evidence; `documentIdentityProbe` shows both host URL sources beside the resolved context |
| Every Office.js call hangs after an AppleScript save | Excel is showing a "Grant File Access" sheet for a folder without a sandbox bookmark (`~/Documents` included); cancel it and save into `~/Library/Containers/com.microsoft.Excel/Data/Documents/` |
| Model turn fails with `Importing a module script failed.` | `npm ci` ran under the live Vite server and wiped `node_modules/.vite/deps`; restart Vite and reload the pane |

From tmustier/pi-for-excel, MIT.

More from tmustier/pi-for-excel