telemetry

Files (1)

• SKILL.md

SKILL.md (first lines)

# Telemetry Skill

Access the CTO observability stack for logs, metrics, and dashboards.

## When to Use

- Querying pod logs via Loki
- Checking metrics via Prometheus
- Viewing dashboards in Grafana
- Debugging agent failures
- Monitoring Play workflow health

---

## Stack Overview

| Service | Port | Purpose |
|---------|------|---------|
| **Prometheus** | 9090 | Metrics collection and querying |
| **Loki** | 3100 | Log aggregation (like Prometheus for logs) |
| **Grafana** | 3000 | Dashboards and visualization |

### Port Forwards (Required for Local Access)

```bash
kubectl port-forward svc/prometheus-server -n observability 9090:80
kubectl port-forward svc/loki-gateway -n observability 3100:80
kubectl port-forward svc/grafana -n observability 3000:80
```

---

## MCP Tools Available

### Prometheus Tools

| Tool | Purpose |
|------|---------|
| `prometheus_query` | Instant query (current value) |
| `prometheus_query_range` | Range query (time series) |
| `prometheus_labels` | List all label names |
| `prometheus_series` | Find series matching labels |

### Loki Tools

| Tool | Purpose |
|------|---------|
| `loki_query` | Query logs with LogQL |
| `loki_labels` | List all label names |
| `loki_label_values` | Get values for a label |

### Grafana Tools

| Tool | Purpose |
|------|---------|
| `grafana_search_dashboards` | Find dashboards by name |
| `grafana_get_dashboard` | Get dashboard definition |
| `grafana_query_prometheus` | Query Prometheus via Grafana |
| `grafana_query_loki_logs` | Query Loki via Grafana |
| `grafana_list_alert_rules` | List configured alerts |

---

## Loki (Logs)

### LogQL Basics

```logql
# All logs from CTO namespace
{namespace="cto"}

# Filter by pod name
{namespace="cto", pod=~"coderun-.*"}

# Search for errors
{namespace="cto"} |= "error"

# JSON parsing
{namespace="cto"} | json | level="error"

# Regex filter
{namespace="cto"} |~ "tool.*mismatch"
```

### Common Queries for CTO

```logql
# All CodeRun pod logs
{namespace="cto", app="coderun"}

# Morgan intake logs
{namespace="cto", pod=~"intake-.*"}

# Play workflow logs
{namespace="cto", pod=~"play-.*"}

# Errors only
{namespace="cto"} |= "error" | json

# Tool inventory issues (A10)
{namespace="cto"} |~ "tool.*(mismatch|missing)"

# MCP initialization failures (A12)
{namespace="cto"} |~ "mcp.*failed"

# Config issues (A11)
{namespace="cto"} |~ "cto-config.*(missing|invalid)"
```

### Via MCP Tool

```
loki_query(query='{namespace="cto"} |= "error"', limit=100)
```

### Via curl

```bash
curl -G "http://localhost:3100/loki/api/v1/query_range" \

From 5DLabsInc/cto, AGPL-3.0.

More from 5DLabsInc/cto