hermes-gateway-troubleshooting

Files (1)

• SKILL.md

SKILL.md (first lines)

# Hermes Gateway Troubleshooting

## When to Use

This skill covers diagnosing and fixing issues with the **Hermes Gateway** — the systemd service that connects Hermes Agent to messaging platforms (Telegram, Discord, WhatsApp, etc.).

**Triggers:**
- "Telegram no responde" / "bot is not responding"
- Gateway starts but blocks messages
- Gateway won't start / crashes immediately
- Platform integration stopped working after upgrade
- "ModuleNotFoundError" or "RuntimeError" in gateway logs
- Users are blocked as "unauthorized"

**DO NOT use for:** CLI-only issues, Hermes TUI rendering. Model/provider failures ("The model provider failed after retries") ARE covered here — see Layer 2.5.

## Diagnostic Approach — Layer by Layer

Always work from outermost to innermost layer:

1. **Is the process running?** → `systemctl --user status hermes-gateway.service`
2. **Are there error logs?** → `tail -100 ~/.hermes/logs/gateway.log`
3. **Is the service config correct?** → `systemctl --user cat hermes-gateway.service`
4. **Is the Python environment intact?** → Check uv tool site-packages
5. **Is the plugin/adapter code correct?** → Check adapter's handling of config values

---

## Layer 1 — Process Health

```bash
# Check if gateway is running
systemctl --user status hermes-gateway.service

# Check gateway PID and uptime
ps aux | grep "gateway run" | grep -v grep

# Restart if needed
systemctl --user restart hermes-gateway.service
```

## Layer 2 — Log Analysis

```bash
# Full recent log
tail -100 ~/.hermes/logs/gateway.log

# Only errors
grep -i "error\|traceback\|exception\|blocked\|unauthorized" ~/.hermes/logs/gateway.log

# Only Telegram activity
grep "\[Telegram\]" ~/.hermes/logs/gateway.log

# Follow in real time
tail -f ~/.hermes/logs/gateway.log
```

**Key log signatures:**

| Log line | Meaning | Likely Fix |
|----------|---------|------------|
| `Blocked unauthorized user X` | Allowlist check failed | Check `allow_from` format in config.yaml |
| `ModuleNotFoundError: No module named 'pydantic_core._pydantic_core'` | Python ABI mismatch | Check uv tool has correct site-packages |
| `Failed to initialize OpenAI client` | Python deps broken | Reinstall missing packages |
| `Connected to Telegram (polling mode)` | OK | — |
| `set_my_commands OK` | OK | — |
| `ImportError: cannot import name '<X>' from '<module>'` | Version skew — daemon running pre-update code against post-update files | See Layer 1.5 |

## Layer 2.4 — Version Skew: gateway process predating a code update

**Symptom:** every inbound message fails with `ImportError: cannot import name
'X' from 'Y'`, naming a symbol that plainly exists in the file on disk. Multiple
unrelated symbols fail at once, from modules that import fine interactively.

**Cause:** `hermes update` (or a git checkout of a new tag) replaces the source
tree while the long-running gateway keeps the OLD modules already imported in
memory. The process holds a mix of old code and new files: old call sites
import symbols the new layout moved or renamed. Nothing on disk is broken.

**Diagnose — compare process start time against the update, not against the file:**

```bash
# When did the running gateway start?
ps -o pid,lstart,etime -p "$(pgrep -f 'hermes_cli.main gateway run' | head -1)"

# When was the source last replaced?
stat -c '%y' ~/.hermes/hermes-agent/hermes_cli/providers.py
git -C ~/.hermes/hermes-agent log -1 --format='%ci %s'
```

If the process started BEFORE the update, version skew is the diagnosis.

**Confirm the disk side is actually healthy** — this is the step that separates
skew from a real breakage:

```bash
~/.hermes/hermes-agent/venv/bin/python -c "from hermes_cli.providers import LLAMACPP_ALIASES; print('imports OK')"
~/.hermes/hermes-agent/venv/bin/python -c "from tools.mcp_tool_common import mcp_server_enabled; print('imports OK')"
```

If those succeed, the fix is a restart, not a code change or reinstall:

```bash
systemctl --user restart hermes-gateway.service
```

**Verify by scoping errors to the new process, never by counting the whole log.**
`grep -c ImportError gateway.log` is cumulative and will show the historical
failures forever:

```bash
awk '/<restart timestamp>/,0' ~/.hermes/logs/gateway.log | grep -c ImportError   # must be 0
grep -E "telegram connected" ~/.hermes/logs/gateway.log | tail -1
```

Healthy reconnect looks like:
```
[Telegram] Telegram polling confirmed healthy: getUpdates progressing (generation 1)
gateway.run: ✓ telegram connected

From iberi22/skills, MIT.

More from iberi22/skills