agent-fleet-analysis

Files (2)

• SKILL.md
• generate_report.py · scripts/

SKILL.md (first lines)

# Agent Fleet Analysis

> ℹ️ **First, set expectations:** before anything else, print one short line with this skill's version and its most recent change — the top entry of `metadata.changelog` above — e.g. `agent-fleet-analysis vX.Y — recent: <summary>`. Then proceed.

Scan a collection of agents — Claude Code agents AND agents built any other way (n8n workflows, LangChain/CrewAI/AutoGen apps, freeform-coded LLM loops) — produce a strategic fleet design recommendation (hierarchy, knowledge brain, memory, canon layer) with every improvement mapped to an installable skill from the abilities plugin marketplace, and generate two artifacts:

1. **A4 PDF report** — for humans, with a shareable download URL when running on Trinity.
2. **Markdown report** — the machine-readable twin, written alongside the PDF. This is the agent-executable version: every gap, quick win, and roadmap item is a checkbox an agent can work through. Hand it to a Claude Code session (or the fleet hub) as the work order for fixing the fleet.

The report is useful **regardless of deployment platform**. It tells you how well your agents are designed, how they should be organized, and what's missing. If you want to run the fleet autonomously in the cloud, the last section shows how Trinity handles that — but the organizational work comes first and stands on its own.

**Timescale framing (important):** everything the report recommends is agent-executable work. A single Claude Code session can fix the gaps across every agent, create Trinity-compatible copies, and deploy them — in a couple of hours, not weeks. The roadmap phases are a **sequence** (do 1 before 2), never a calendar. Write the report accordingly: no "this week / next month" language anywhere.

## Usage

```
/agent-fleet-analysis [path ...]
```

- `path` (optional, one or more): directories to scan. Defaults to `.` (current directory). Multiple paths (space- or newline-separated) are scanned in one run and merged into a single report.

The skill is fully autonomous — it runs through all steps and prints the report locations (and a shareable URL when on Trinity) at the end.

---

## Step 1: Resolve the Scan Paths

Parse `$ARGS` into a list of paths (whitespace/newline separated; ignore any leading free-text note the user included). If no paths given, use the current directory.

For each path: if it doesn't exist, report it and continue with the remaining paths. If NO valid paths remain, print an error and stop.

## Step 2: Discover Agents (any paradigm)

Discovery runs in two passes over each scan path, up to **3 levels deep** (agents often live one level down inside an `agents/` or `workflows/` folder). Every discovered agent becomes a first-class inventory entry with a `paradigm` field — nothing is demoted to a footnote.

### Pass 1 — Claude Code agents (`paradigm: claude-code`)

An agent directory is any directory containing at least one of `CLAUDE.md`, `template.yaml`, or a `.claude/` subdirectory:

```bash
# Find candidate directories (run per scan path)
find "$SCAN_PATH" -maxdepth 3 \( -name "CLAUDE.md" -o -name "template.yaml" \) \
  -not -path "*/node_modules/*" -not -path "*/.venv/*" -not -path "*/venv/*" \
  | xargs -I{} dirname {} | sort -u
find "$SCAN_PATH" -maxdepth 3 -type d -name ".claude" \
  -not -path "*/node_modules/*" | xargs -I{} dirname {} | sort -u
```

Deduplicate (same directory found via multiple markers). Nested agent directories are allowed (a sub-agent inside a parent repo) — record both, and note the nesting in the report.

### Pass 2 — other agent paradigms

For directories NOT already claimed by Pass 1, classify by evidence (first match wins):

| Paradigm | Evidence |
|----------|----------|
| `n8n` | `workflow.json`, or any `*.json` whose top level has both `"nodes"` and `"connections"` keys — each workflow (or a folder of them) is one agent entry |
| `framework` | Source files importing `langchain`, `crewai`, `autogen`, `openai` Assistants API, or similar agent frameworks (grep `*.py`/`*.js`/`*.ts`, excluding venv/node_modules) |
| `freeform-coded` | LLM API usage (anthropic/openai/gemini SDK or direct REST calls) PLUS prompt assets (a `prompts/` dir, `*_prompt*` files, `system_prompt*`) or an evident agentic loop — an agent written in plain code with no framework |
| `non-agent` | None of the above — list the path once under "Scanned, not agents" in the report and move on |

```bash
# n8n detection example
find "$SCAN_PATH" -maxdepth 3 -name "*.json" -not -path "*/node_modules/*" \
  | head -50 | xargs grep -l '"nodes"' 2>/dev/null | xargs grep -l '"connections"' 2>/dev/null
# framework / LLM-SDK detection example
grep -rlE "from langchain|import crewai|import autogen|from anthropic|import anthropic|from openai|import openai" \
  "$SCAN_PATH" --include="*.py" --include="*.js" --include="*.ts" \
  --exclude-dir=node_modules --exclude-dir=.venv --exclude-dir=venv 2>/dev/null | head -5
```

Use judgment on granularity: an n8n export folder with one orchestrator + several sub-workflows can be inventoried as one fleet-let with named members, or as individual agents if they are clearly independent.

If zero agents are found across ALL paths (all paradigms), print:
```
No agents found in <paths>. Try a directory containing CLAUDE.md files,
n8n workflow exports, or agent code (LangChain/CrewAI/LLM API loops).
```
and stop.

**Build the initial agent list** — for each discovered agent, record:
- `dir`: absolute path
- `name`: directory basename (or `name:` field from template.yaml / n8n workflow `name` if present)
- `paradigm`: `claude-code` | `n8n` | `framework` | `freeform-coded`

## Step 3: Per-Agent Inventory and Fleet Maturity Score

For each **`claude-code`** agent directory, run 3a–3f. Agents of other paradigms skip to **3g (Migration Readiness)** — their record uses `migration_readiness` instead of `maturity_score`. Build a structured record per agent either way.

### 3a: Read Core Files

```bash
AGENT_DIR="<agent_dir>"
test -f "$AGENT_DIR/CLAUDE.md"         && HAS_CLAUDE_MD=1    || HAS_CLAUDE_MD=0
test -f "$AGENT_DIR/template.yaml"     && HAS_TEMPLATE=1     || HAS_TEMPLATE=0
test -f "$AGENT_DIR/.mcp.json.template" && HAS_MCP_TEMPLATE=1 || HAS_MCP_TEMPLATE=0
test -f "$AGENT_DIR/.env.example"      && HAS_ENV_EXAMPLE=1  || HAS_ENV_EXAMPLE=0
test -f "$AGENT_DIR/.gitignore"        && HAS_GITIGNORE=1    || HAS_GITIGNORE=0
test -d "$AGENT_DIR/.claude"           && HAS_CLAUDE_DIR=1   || HAS_CLAUDE_DIR=0
test -d "$AGENT_DIR/.claude/skills"    && HAS_SKILLS=1       || HAS_SKILLS=0
test -d "$AGENT_DIR/.claude/memory"    && HAS_MEMORY=1       || HAS_MEMORY=0
test -f "$AGENT_DIR/fleet.yaml"        && HAS_FLEET=1        || HAS_FLEET=0
```

Read CLAUDE.md to extract: purpose (first non-empty paragraph), any explicit autonomy/role signals.

If template.yaml exists, read it for: `name`, `display_name`, `description`, `resources`, `credentials`, `schedules`.

If .gitignore exists, check it excludes: `.env`, `.env.*`, `.mcp.json`, `credentials.json`, `*.pem`, `*.key`, `content/`, `.claude/projects/`, and `.claude/plugins/`. Do **not** require `.claude/settings.json` — it is the agent's project settings and may be committed (Trinity ent#708); flag a committed copy only when it registers `/opt/trinity/` hook paths or carries a credential key (`env`, `apiKeyHelper`, `awsAuthRefresh`, `awsCredentialExport`, `gcpAuthRefresh`, `otelHeadersHelper`). For a Trinity-deployed agent this list is enforced platform-side on every Push, so a gap is repaired rather than silently persisted — but it churns until the repo matches.

### 3b: Credential Security Scan

```bash
# Check for hardcoded credentials (never expose results in the report — just flag pass/fail)
grep -r --include="*.yaml" --include="*.yml" --include="*.json" --include="*.md" \
  --include="*.py" --include="*.sh" --include="*.js" --include="*.ts" --include="*.env" \
  -lE "(api_key|API_KEY|secret|password|token)\s*[:=]\s*['\"][A-Za-z0-9_\-]{12,}" \
  "$AGENT_DIR" 2>/dev/null \
  | grep -vE "\.gitignore|\.env\.example|template\.yaml|node_modules|\.venv|__pycache__" \

From Abilityai/trinity-skills, Apache-2.0.

More from Abilityai/trinity-skills