comfyui-workflow-builder

Files (11)

• SKILL.md
• EVAL.md · eval/
• benchmark.yaml · eval/
• .gitignore · eval/results/
• .gitkeep · eval/results/
• run-eval.sh · eval/
• test-cases.yaml · eval/
• trigger-tests.yaml · eval/
• models.md · references/
• prompt-templates.md · references/
• workflows.md · references/

SKILL.md (first lines)

# ComfyUI Workflow Builder

> All recommendations in this skill are subordinate to `state/inventory.json`, `foundation/hardware-profile.md`, and current version constraints.

Translates natural language requests into executable ComfyUI workflow JSON. Always validates against inventory before generating.

## Workflow Generation Process

### Step 1: Understand the Request

Parse the user's intent into:
- **Output type**: Image, video, or audio
- **Source material**: Text-only, reference image(s), existing video
- **Identity method**: None, zero-shot (InstantID/PuLID), LoRA, Kontext
- **Quality level**: Draft (fast iteration) vs production (maximum quality)
- **Special requirements**: ControlNet, inpainting, upscaling, lip-sync

### Step 2: Check Inventory

Read `state/inventory.json` to determine:
- Available checkpoints → select best match for task
- Available identity models → determine which methods are possible
- Available ControlNet models → enable pose/depth control if available
- Custom nodes installed → verify all required nodes exist
- VRAM available → optimize settings accordingly

### Step 3: Select Pipeline Pattern

Based on request + inventory, choose from:

| Pattern | When | Key Nodes |
|---------|------|-----------|
| Text-to-Image | Simple generation | Checkpoint → CLIP → KSampler → VAE |
| Identity-Preserved Image | Character consistency | + InstantID/PuLID/IP-Adapter |
| LoRA Character | Trained character | + LoRA Loader |
| Image-to-Video (Wan) | High-quality video | Diffusion Model → Wan I2V → Video Combine |
| Image-to-Video (AnimateDiff) | Fast video, motion control | + AnimateDiff Loader + Motion LoRAs |
| Talking Head | Character speaks | Image → Video → Voice → Lip-Sync |
| Upscale | Enhance resolution | Image → UltimateSDUpscale → Save |
| Inpainting | Edit regions | Image + Mask → Inpaint Model → KSampler |

These rows describe graph intent only. Emit no named node until its exact `class_type`,
inputs, and output indices have been confirmed through inventory or `/object_info`.

### Step 4: Generate Workflow JSON

**ComfyUI workflow format:**

```json
{
  "{node_id}": {
    "class_type": "{NodeClassName}",
    "inputs": {
      "{param_name}": "{value}",
      "{connected_param}": ["{source_node_id}", {output_index}]
    }
  }
}
```

**Rules:**
- Node IDs are strings (typically "1", "2", "3"...)
- Connected inputs use array format: `["source_node_id", output_index]`
- Output index is 0-based integer
- Filenames must match exactly what's in inventory
- Seed values: use random large integer or fixed for reproducibility

### Step 5: Validate

Before presenting to user:

1. Every `class_type` exists in inventory's node list
2. Every model filename exists in inventory's model list
3. All required connections are present (no dangling inputs)
4. VRAM estimate doesn't exceed available VRAM
5. Resolution is compatible with the exact selected model and workflow version

### Step 6: Output

**If online mode**: Queue via `comfyui-api` skill
**If offline mode**: Save JSON to `projects/{project}/workflows/` with descriptive name

## Workflow Templates

### Basic Text-to-Image (Checkpoint-Based)

Illustrative API shape only. Replace the filename and validate every node schema before
execution.

```json
{
  "1": {
    "class_type": "CheckpointLoaderSimple",
    "inputs": {"ckpt_name": "<exact checkpoint filename from inventory>"}
  },
  "2": {
    "class_type": "CLIPTextEncode",
    "inputs": {"text": "{positive_prompt}", "clip": ["1", 1]}
  },
  "3": {
    "class_type": "CLIPTextEncode",
    "inputs": {"text": "{negative_prompt}", "clip": ["1", 1]}
  },
  "4": {
    "class_type": "EmptyLatentImage",
    "inputs": {"width": 1024, "height": 1024, "batch_size": 1}
  },
  "5": {
    "class_type": "KSampler",
    "inputs": {
      "seed": 42,
      "steps": 25,
      "cfg": 3.5,
      "sampler_name": "euler",
      "scheduler": "normal",
      "denoise": 1.0,
      "model": ["1", 0],
      "positive": ["2", 0],
      "negative": ["3", 0],

From calebmills99/VUORSE-VORTEX, MIT.

More from calebmills99/VUORSE-VORTEX