add-pipeline

Files (8)

• SKILL.md
• pipeline-pause.md · templates/
• pipeline-recover.md · templates/
• pipeline-resume.md · templates/
• pipeline-status.md · templates/
• pipeline-tick.md · templates/
• pipeline.yaml.template · templates/
• state.json.template · templates/

SKILL.md (first lines)

# Add Pipeline

> ℹ️ **First, set expectations:** before anything else, print one short line with this skill's version and its most recent change — the top entry of `metadata.changelog` above — e.g. `add-pipeline vX.Y — recent: <summary>`. Then proceed.

Add a long-running, multi-stage **pipeline** to any Trinity-compatible agent. Implements the canonical pipeline spec: the agent owns the DAG and stage logic; Trinity owns the read surface (`~/.trinity/pipelines/*.yaml` + `~/.trinity/pipeline-state/**/*.json`); a single heartbeat skill (`pipeline-tick`) owns advancement, retry, and escalation.

**When to use a pipeline (and when not):** reach for this when the work is a *population* of items that each crawl through *multiple stages over many runs* — or a **multi-agent process where each stage is another agent's playbook** (a release propagation, an onboarding chain; one *instance* = one run, `agent:` on the stage, the tick dispatches a one-line playbook call — fleet convention `protocols/playbook-call.md`) — and you need durable per-item state, isolated retries, and an at-a-glance "what stage is each item in" — e.g. per-customer onboarding, document ingestion, batched research crawls, especially when the whole batch can't finish in one scheduled run. If it's a single recurring task, a scheduled playbook is simpler — don't reach for a pipeline. And mind the boundary: instances are **multi-instance, not multi-tenant** — their *state* is isolated but they all run inside the **same agent** (same credentials, context window, and heartbeat), so for genuinely isolated or large-scale tenants, deploy **one agent per tenant** (Trinity fan-out) rather than one pipeline with many instances — that cross-agent layer is `/add-orchestrator`'s domain: its `/orchestrate` skill does the routing, fan-out, and ephemeral roll-out/tear-down across a fleet, and its `/discover-agents` surfaces this agent's pipelines to the fleet map.

**Keep heavy CPU jobs out of the heartbeat turn.** A stage that does real compute (a FAISS/index rebuild, bulk embedding, a full bootstrap) must **not** run synchronously inside `pipeline-tick`'s LLM turn. A headless run is a single agent turn and can't host a job past the **~10-min synchronous Bash ceiling** — past it the harness auto-backgrounds the job, active waiting is blocked, and the turn ending reaps the backgrounded task (it fires `killed`, not `completed`; see `/trinity:onboard` → *Long-running jobs inside a run*). Model the heavy work as an **OS-level job** (a container cron / systemd unit or a small non-LLM sidecar writing a **done-marker**), and let the stage only **trigger it and verify the artifact moved** (mtime advanced / count > 0 — not an exit code or `business_status`). The heartbeat stays a cheap read-and-advance loop.

**What gets installed:**

| Artifact | Location | Purpose |
|---|---|---|
| `projects/<slug>/project.md` | agent repo | human-readable project description |
| `projects/<slug>/pipeline.yaml` | agent repo | canonical DAG definition (you edit this) |
| `projects/<slug>/instances/` | agent repo | one subdir per tenant/zone (populate via `/add-pipeline-instance`) |
| `.claude/skills/pipeline-tick/` | agent repo | the heartbeat — autonomous, scheduled |
| `.claude/skills/pipeline-status/` | agent repo | operator view |
| `.claude/skills/pipeline-recover/` | agent repo | operator override |
| `.claude/skills/pipeline-pause/` | agent repo | operator pause |
| `.claude/skills/pipeline-resume/` | agent repo | operator resume |
| `~/.trinity/pipelines/<slug>.yaml` | user home | write-through copy — Trinity's read surface |
| `~/.trinity/pipeline-state/<slug>/` | user home | per-instance state, read by dashboards and other agents |
| `dashboard.yaml` `Pipelines` section | agent repo (if present) | table widget (Trinity's `sections[]`→`widgets[]` schema) — rows materialized by `pipeline-tick` each pass |
| heartbeat schedule | Trinity MCP (if available) | `pipeline-<slug>-heartbeat` cron `*/15 * * * *` |
| `template.yaml` `schedules:` entry | agent repo (if present) | durable copy of the heartbeat — reconciled by `/trinity:sync`, read by fleet orchestrators (`/discover-agents`) |

---

## Process

### Step 1: Preflight

Run from inside the target agent directory (or ask for the path). Verify:

```bash
# Must be an agent root (CLAUDE.md present)
[ -f CLAUDE.md ] || ask_user_for_agent_path

# Must have a .claude/skills/ directory (create if missing)
mkdir -p .claude/skills

# Must have jq + yq available (used by templates and validators)
command -v jq >/dev/null 2>&1 || warn "jq not installed — runtime skills will fail. Install: brew install jq"
command -v yq >/dev/null 2>&1 || warn "yq not installed — runtime skills will fail. Install: brew install yq"
```

If `CLAUDE.md` is missing, ask the user to point to the correct directory or run `/create-agent` first.

### Step 2: Gather pipeline metadata

Use `AskUserQuestion` (one question at a time, or batch if you prefer):

**Q1 — Pipeline slug** (free text)
- "Short identifier for this pipeline, lowercase-kebab-case (e.g. `columns`, `ingest`, `onboarding`). Used as the directory name and the `pipeline_id` in yaml."
- Validate: matches `^[a-z][a-z0-9-]{1,40}$`. Refuse if `projects/<slug>/` already exists (suggest `/add-pipeline-stage` to extend, or pick a different slug).

**Q2 — Pipeline display name** (free text)
- "Human-readable name (e.g. 'Columns Production Pipeline')."

**Q3 — One-line description** (free text)
- "What does this pipeline do? One or two sentences."

**Q4 — Heartbeat cadence**
- `*/15 * * * *` (every 15 minutes — recommended default)
- `0 * * * *` (hourly — for slow-moving pipelines)
- `*/5 * * * *` (every 5 minutes — for time-sensitive flows)
- Custom (free text cron expression)

**Q5 — Install runtime skills?**
- `Yes — install all five` (pipeline-tick, pipeline-status, pipeline-recover, pipeline-pause, pipeline-resume) — recommended
- `Only the heartbeat` (just pipeline-tick; skip operator skills) — for headless agents
- `Skip — I'll wire my own` — only scaffold the project files

If any of the five runtime skill directories already exist under `.claude/skills/`, ask: overwrite / skip existing / cancel.

### Step 3: Scaffold project directory

```bash
SLUG="<pipeline-slug from Q1>"
mkdir -p "projects/$SLUG/instances"
mkdir -p "projects/$SLUG/skills"  # for stage-skill symlinks if the user wants them grouped
```

Write `projects/$SLUG/project.md`:

```markdown
---
name: <Pipeline Display Name>
status: active
created: <YYYY-MM-DD>
owner: <agent-name from CLAUDE.md or env user>
pipeline_id: <SLUG>
---

# <Pipeline Display Name>

<one-line description from Q3>

## Stages

Stages live in `pipeline.yaml`. Edit there or use `/add-pipeline-stage <slug>` to extend.

## Instances

Instances live in `instances/<instance-slug>/`. Add new ones via `/add-pipeline-instance <slug> <instance-slug>`.

## Operations

- `/pipeline-status <slug>` — current state of every instance
- `/pipeline-recover <slug> <instance>` — unstick a blocked instance
- `/pipeline-pause` / `/pipeline-resume` — maintenance toggles
```

Write `projects/$SLUG/pipeline.yaml` from `templates/pipeline.yaml.template`, substituting:
- `{{PIPELINE_ID}}` → slug
- `{{PIPELINE_NAME}}` → display name
- `{{PIPELINE_DESCRIPTION}}` → description

From Abilityai/trinity-skills, Apache-2.0.

More from Abilityai/trinity-skills