logging-diagnostics

Files (1)

• SKILL.md

SKILL.md (first lines)

# Logging Diagnostics

Use this skill when debugging a failure, checking whether logging is enabled, changing log levels, adding a missing diagnostic log, or cleaning up generated logs.

Check existing evidence before creating another capture:

1. Check disk, logging status and the time of the incident. For this repository, `node scripts/diagnostics-logging.js status` identifies the effective diagnostics configuration and capture directory.
2. Identify the Provider, Invocation, native session/thread and turn IDs from the Worker journal or failure record. Search the specific session or time range, keeping prompts, credentials and tool output out of user-facing summaries.
3. Use the source map below. Native session content answers what the model received or produced; bot records answer what was transported, presented or recovered. Similar names do not establish equivalent coverage.
4. Enable a narrow timed capture only for a question the existing evidence cannot answer. Reproduce once, disable the seam, and record the finding. Never leave an indefinite payload trace enabled.
5. Use the maintained retention job for owned diagnostic files. Preserve recent logs, Provider session/resume history, credentials, application recovery journals, usage metering and unrelated services. Do not vacuum shared systemd journals for a bot-only cleanup.

| Question | Existing source and correlation |
| --- | --- |
| Codex prompt, output, tool call/result, compaction or token usage | `$CODEX_HOME/sessions` and `archived_sessions` (default `~/.codex`), matched by native thread ID in filename/session metadata; narrow by turn ID, call ID and timestamp. Rollouts can contain repeated history, so file totals are not deduplicated accounting. |
| Codex engine, network or retry failure | Read-only `$CODEX_HOME/logs_2.sqlite`: inspect the schema first, then bounded `logs` rows by `thread_id`, `process_uuid`, time, level or target. This is separate from session content and does not prove successful bot delivery. |
| Claude or Pi native conversation | Claude's configured projects/session files (`CLAUDE_CONFIG_DIR` or `~/.claude/projects`); Pi session files under its configured agent directory (normally `~/.pi/agent/sessions`). Confirm the active Provider session ID and actual home, especially for sandboxed channels. |
| Adapter transport failure or unusual stderr | `$XDG_STATE_HOME/madabot/codex-acp-logs` (default `~/.local/state/madabot`): `process/<pid>.log`, `sessions/<thread>/session.log`, `sessions/<thread>/turns/<turn>.log`, including gzip rotations. Routine RPC/write/delivery detail is off unless its timed seam is active. |
| WhatsApp delivery, approval, restart or resume | App/Worker journal and targeted `journalctl --user -u <unit> --since ... --until ...`; correlate Invocation, turn and delivery IDs. These contain bot-only state that native rollouts cannot replace. Never delete recovery state as duplicate logging. |
| Historical workspace cost | Private `workspace-usage/usage.sqlite` and metering projections. These retain billing evidence after source cleanup and support deduplicated reports; they are not disposable diagnostic copies. |

See `docs/diagnostic-sources.md` and `docs/log-maintenance.md` for the verified boundaries and retention policy. The maintained skill is `skills/logging-diagnostics/SKILL.md`; `.agents/skills/logging-diagnostics` links to it.

Do not recreate the removed `harness.raw-event` capturer or enable an intermediate copy merely to inspect derived events. Retrieve native content or the earlier `acp.protocol` capture and reproduce the conversion with the relevant code and state. History projection no longer saves another raw archive. Local ACP terminal failures retain compact `[acp-terminal]` warnings with session/terminal/PID correlation and exit status; command text and output remain in their existing sources. Existing historical capture files remain subject to the same retention policy.

- Treat timestamps visible in user-provided screenshots or screen recordings as `Europe/Dublin` local time unless the user says otherwise.

Runtime tools in this repo:

```bash
node "$(git rev-parse --show-toplevel)/scripts/diagnostics-logging.js" status
node "$(git rev-parse --show-toplevel)/scripts/diagnostics-logging.js" capture codex.app-server on --minutes 5
node "$(git rev-parse --show-toplevel)/scripts/diagnostics-logging.js" capture codex.app-server off
node "$(git rev-parse --show-toplevel)/scripts/diagnostics-logging.js" capture acp.protocol on --minutes 5
node "$(git rev-parse --show-toplevel)/scripts/diagnostics-logging.js" capture acp.protocol shape --minutes 5
node "$(git rev-parse --show-toplevel)/scripts/diagnostics-logging.js" capture whatsapp.inbound on --minutes 5 --rotate-minutes 1
node "$(git rev-parse --show-toplevel)/scripts/diagnostics-logging.js" capture whatsapp.inbound full-raw --minutes 1
node "$(git rev-parse --show-toplevel)/scripts/diagnostics-logging.js" level debug
node "$(git rev-parse --show-toplevel)/scripts/diagnostics-logging.js" capture all off
```

`codex.app-server` enables existing compact adapter traffic/bookkeeping records, reloading within one second and expiring automatically. Its output stays in the scoped adapter logs; `shape`, `full-raw`, fixture rotation and queue options do not apply to this seam. Use it for the inner app-server boundary only when native evidence and routine error records are insufficient. `acp.protocol` captures the distinct adapter-to-Worker boundary. Do not enable both without needing to compare those boundaries. Neither seam is required to retrieve ordinary model/tool content already retained in native history.

The hourly maintenance job preserves at least seven days of dated adapter archives and fixture captures; longer configured fixture retention is honored. `node scripts/log-maintenance.js --plan` is a read-only adapter-retention preview. Canonical logs, unrecognized/manual archive names and open adapter files are retained. Existing archives may therefore outlive seven days; never broaden deletion to force a disk target.

If a needed diagnostic does not have a runtime toggle:

- Add a small explicit toggle instead of relying only on environment variables.
- Keep the default off if the log can grow quickly or contain sensitive data.
- Log structured records with timestamps and enough identifiers to correlate the event.
- Add focused tests for default-off, enabled, disabled, and runtime-change behavior.

Fixture capture logs:

- Treat fixture captures as raw diagnostic material for debugging and later manual test-fixture promotion.
- Keep fixture capture default-off, runtime-bounded, and scoped to explicit seams.
- Use `shape` when the investigation needs event names and payload structure but not source values. Shape capture omits scalar values, previews, source-value hashes, exact content lengths, and binary bytes. It retains only allowlisted protocol-owned object keys, collapses every other key into a redacted-field summary, and records value types, coarse size buckets, truncation reasons, and a fingerprint computed only from the generated shape.
- Shape capture may retain a specifically allowlisted protocol discriminator such as declared ACP `params.update.sessionUpdate` variants; treat those enums as event identity, not payload content. Unknown or malformed discriminator text must be reduced to a sentinel and coarse size.
- Read the CLI-reported `captureDir`; production captures use the stable writable diagnostics state directory rather than an immutable Worker release or checkout-relative path.
- If large raw values are capped, write a `fixtureCapture.meta` record in the capture file before event records.
- The meta record must clearly define the truncation marker, currently `__fixtureCaptureTruncated`, and state that capped values are replaced by metadata plus a preview.
- Capped values should be obvious objects, not strings that look complete. Include type, original size, hash, and preview fields where available.

Rules:

- Keep context small; read recent logs, not whole files.
- Treat logs as sensitive.
- Do not leave verbose diagnostics enabled after the investigation.
- Prefer structured logs and runtime toggles over broad permanent logging.

From madacol/whatsapp-llm-bot, Unlicense.

More from madacol/whatsapp-llm-bot