activity-monitor

Files (83)

• SKILL.md
• package-lock.json
• package.json
• health-engine.md · references/
• claude-pid.test.js · scripts/__tests__/
• emit-custom-inject.test.js · scripts/__tests__/
• guardian.test.js · scripts/__tests__/
• health-engine.test.js · scripts/__tests__/
• health-evidence.test.js · scripts/__tests__/
• health-file-layout.test.js · scripts/__tests__/
• health-notice-classification.test.js · scripts/__tests__/
• health-notice-match.test.js · scripts/__tests__/
• health-notice-pipeline.test.js · scripts/__tests__/
• health-quota.test.js · scripts/__tests__/
• hook-activity.test.js · scripts/__tests__/
• hook-auth-prompt.test.js · scripts/__tests__/
• hook-health.test.js · scripts/__tests__/
• memory-sync-gate.test.js · scripts/__tests__/
• memory-sync-integration.test.js · scripts/__tests__/
• message-router.test.js · scripts/__tests__/
• monitor-orchestrator.test.js · scripts/__tests__/
• router-socket-process.test.js · scripts/__tests__/
• router-socket.test.js · scripts/__tests__/
• session-foreground.test.js · scripts/__tests__/
• session-start-orchestrator.test.js · scripts/__tests__/
• shard-mode.test.js · scripts/__tests__/
• shard-sequencer.test.js · scripts/__tests__/
• status-writer.test.js · scripts/__tests__/
• task-scheduler.test.js · scripts/__tests__/
• tmux-input-state.test.js · scripts/__tests__/
• tool-event-stream.test.js · scripts/__tests__/
• tool-lifecycle.test.js · scripts/__tests__/
• tool-rules.test.js · scripts/__tests__/
• tool-watchdog.test.js · scripts/__tests__/
• usage-check-engine.test.js · scripts/__tests__/
• usage-check-init.test.js · scripts/__tests__/
• usage-codex-rollout-reader.test.js · scripts/__tests__/
• usage-monitor-file-reader.test.js · scripts/__tests__/
• usage-monitor-optional-sqlite.test.js · scripts/__tests__/
• usage-monitor.test.js · scripts/__tests__/
• activity-monitor.js · scripts/
• runtime-components.js · scripts/adapters/
• claude-pid.js · scripts/
• context-monitor.js · scripts/
• emit-custom-inject.js · scripts/
• guardian.js · scripts/
• health-controller.js · scripts/
• health-engine.js · scripts/
• health-evidence.js · scripts/
• health-notice-classification.js · scripts/
• health-notice-match.js · scripts/
• health-quota.js · scripts/
• health-store.js · scripts/
• hook-activity.js · scripts/
• hook-auth-prompt.js · scripts/
• hook-health.js · scripts/
• memory-sync-gate.js · scripts/
• message-router.js · scripts/
• monitor-orchestrator.js · scripts/
• monitor.js · scripts/
• …and 23 more

SKILL.md (first lines)

# Activity Monitor Skill

PM2 guardian service that monitors the active runtime agent's (Claude or Codex) activity state and ensures it's always running.

## When to Use

This is a **PM2 service** (not directly invoked by Claude). It runs continuously in the background.

## What It Does

1. **Activity Monitoring**: Tracks the agent's busy/idle state every second
2. **Status File**: Writes `~/zylos/activity-monitor/agent-status.json` with current state (busy/idle, idle_seconds, health)
3. **Guardian Mode**: Automatically restarts the agent if it stops or crashes
4. **Maintenance Awareness**: Waits for restart/upgrade scripts to complete before starting the agent
5. **Recovery Liveness Detection**: Sends C4 heartbeat probes during recovery and after restarts; healthy sessions have no periodic primary heartbeat
6. **Health Check**: Periodically enqueues system health checks (PM2, disk, memory) via the C4 control queue
7. **Daily Upgrade**: Enqueues a Claude Code upgrade via the C4 control queue at 5:00 AM local time daily (disabled by default; enable with `zylos config set daily_upgrade_enabled true`)
8. **Context Monitoring**: Receives context usage data after every turn; triggers early memory sync and new-session handoff at thresholds read from `config.json` (`new_session_threshold` for Claude, `codex_new_session_threshold` for Codex), each with a built-in runtime fallback if the key is unset

## Status File Format

```json
{
  "state": "idle",
  "last_activity": 1738675200,
  "last_check": 1738675210,
  "last_check_human": "2026-02-04 20:30:10",
  "idle_seconds": 5,
  "inactive_seconds": 10,
  "source": "conv_file",
  "health": "ok"
}
```

- `state`: "busy" | "idle" | "stopped" | "offline"
- `idle_seconds`: Time since entering idle state (0 when busy)
- `source`: "conv_file" (reliable) | "tmux_activity" (fallback)
- `health`: "ok" | "unavailable" | "rate_limited" | "auth_failed" — current public health; legacy `recovering` / `down` states remain readable

## PM2 Management

```bash
# Start
pm2 start ~/zylos/.claude/skills/activity-monitor/scripts/activity-monitor.js --name activity-monitor

# Restart
pm2 restart activity-monitor

# View logs
pm2 logs activity-monitor

# Check status
pm2 list
```

## Guardian Behavior

- **Detection**: Checks if the agent is running every second
- **Restart Delay**: Waits 5 seconds of continuous "not running" before restarting
- **Maintenance Wait**: Detects restart/upgrade scripts and waits for completion
- **Recovery Prompt**: Sends catch-up message via C4 after restart

## How It Works

1. **Monitor Loop** (every 1s):
   - Check if tmux session exists
   - Check if agent process is running
   - Detect activity (conversation file mtime for Claude; tmux activity for Codex)
   - Calculate idle/busy state
   - Write status to ~/zylos/activity-monitor/agent-status.json

2. **Guardian Logic**:
   - If agent not running for 5+ seconds → restart
   - Wait for maintenance scripts to complete
   - Send recovery prompt via C4 after successful restart

3. **Activity Detection**:
   - Claude primary: Conversation file modification time (reliable)
   - Codex / fallback: tmux window activity
   - Threshold: 3 seconds without activity = idle

## Heartbeat Liveness Detection

The health engine uses the C4 control queue to verify runtime responsiveness during recovery. Healthy sessions do not receive periodic `primary` probes, and there is no heartbeat enable/interval setting. The internal maintenance timer still processes pending probe results and recovery scheduling.

### Recovery Probes

| Phase | Trigger | On Success | On Failure |
|-------|---------|------------|------------|
| `recovery` | Unavailable/recovering maintenance retry or user-message recovery | Return to `ok` | Apply existing rate-limit detection or recovery handling |
| `post_restart` | Restart notification or process-running signal after its grace period | Return to `ok` | Apply existing recovery handling |
| `down-check` | Legacy `down` state retry interval | Return to `ok` | Stay `down` and retry later |

Retries in unavailable/recovering use exponential backoff (1m, 5m, 25m, then 60m). Rate-limit cooldown expiry only clears the wait; a later recovery trigger verifies responsiveness. Failed or expired pending probes still run their existing failure handling, including API-error fast scanning while waiting.

Recovery and post-restart probes require end-to-end acknowledgement. The dispatcher retains confirmed-active busy auto-ack for other heartbeat phases; idle heartbeat auto-ack has been removed.

See [Recovery and maintenance behavior](references/health-engine.md) for timing and compatibility details.

### Unhealthy Message Routing

When health is not `ok`, C4 intake still records incoming messages as delivered. The MessageRouter returns an immediate status response for the current message when replies are enabled; `--no-reply` messages are accepted silently. If the router IPC is unavailable, `c4-receive.js` falls back to the status file and follows the same delivered/current-message behavior.

## Health Check

The activity monitor periodically enqueues system health checks via the C4 control queue.

- **Enabled**: On by default. Disable with `zylos config set health_check_enabled false`
- **Interval**: Every 24 hours (86400 seconds)
- **Persisted state**: `~/zylos/activity-monitor/health-check-state.json` (survives restarts)
- **Priority**: 3 (normal)
- **Gated by health**: Only enqueued when `health === 'ok'`
- **Gated by agent**: Only enqueued when the agent process is running

The health check control message instructs Claude to:
1. Check PM2 services via the health-check skill's sanitized PM2 snapshot helper
2. Check disk space via `df -h`
3. Check memory via `free -m`
4. If issues found, notify the most recent communication channel

From zylos-ai/zylos-core, MIT.

More from zylos-ai/zylos-core