asset-pipeline

Files (1)

• SKILL.md

SKILL.md (first lines)

# Asset Pipeline

## Tools

- `python <plugin>/mcp/assets/assets.py webp <image> [quality]` — any raster →
  webp beside it (Pillow). Run on EVERY raster an agent adds to `public/`.
- `python <plugin>/mcp/assets/assets.py frames <video> [fps]` — mp4 → webp
  frame sequence (needs ffmpeg; missing → tell the user the one-line install,
  continue with other work).
- `python <plugin>/mcp/assets/assets.py cut <image> [tol=32]` — flat background →
  transparent webp (samples the corner color; raise tol for gradients/noise).
- All record `asset_manifest` rows. Serve only webp to browsers.

## Transparency & icons (no model can do this — we do)

NO image model (Gemini, Imagen, DALL·E) outputs a real alpha channel. The
recipe: generate the mark on a FLAT background (the logo prompt below already
forces this), then `assets.py cut` strips it locally — free, offline, zero
credentials. Photos / complex subjects: `pip install rembg`, then
`rembg i in.png out.png` (local U2Net model, also key-free).

## Scrollytelling recipe (the storytelling primitive)

1. Storyboard scroll beats first: entry, 2–4 transitions, resolution, and the
   content pinned beside each beat. Motion supports the story; it is not a
   decorative loop.
2. Get one continuous 5–8 second source clip (user-provided in
   `workspace/references/video/`, or generate below). Locked camera direction,
   constant motion, and no cuts produce clean bidirectional scrubbing.
3. `assets.py frames video.mp4 24` → `video_frames/frame_0001.webp …`.
   Produce desktop and mobile crops when composition cannot safely center-crop.
4. Copy frames to `public/frames/<name>/`; use `templates/ScrollScrub.tsx`.
   It loads a small window around scroll position, favors scroll direction,
   evicts distant frames, caps DPR, and draws cover-fit through one canvas.
5. Ship a meaningful poster. Reduced-motion, save-data, low-memory devices,
   failed decoding, and no JavaScript receive that poster and the complete
   narrative as ordinary semantic HTML.
6. Keep headings, copy, links, and calls to action outside canvas/WebGL so they
   remain selectable, accessible, indexable, and stable before frames arrive.
7. Design-law still applies: one cinematic sequence per page. Run web-quality
   checks for overflow, reduced motion, accessibility, metadata, and SEO.

Frame delivery is progressive. Load frame zero immediately, then a bounded
forward/backward chunk near the target. Never create every `Image` at mount for
a long sequence. Use immutable, content-hashed frame URLs with long cache
headers; do not lazy-load the poster or first frame. Decode before draw, render
at most once per animation frame, and recalculate canvas size through
`ResizeObserver`, not layout reads on every scroll event.

## AI video generation — the ladder (never exit on missing credentials)

**Omni** = `gemini-omni-flash-preview`, Google's text→video model (Veo 3.1 is the
fallback family). Don't hand-roll the REST calls — the ladder is implemented:

```
python <plugin>/mcp/assets/video.py "a slow orbit around a matte black cube"
# or: from video import generate; rung, detail = generate(subject, out="hero.mp4")
```

### Verify every generation — measure, don't eyeball

```
python <plugin>/mcp/assets/verify.py hero.mp4 "#10B981" --audio
```

Prints accent coverage % and mean audio dB, exits nonzero on failure. Run it on
every generated video before showing the user. Two real defects shipped because
nobody measured: a file with a valid AAC stream that was **silent** (mean −91 dB
— "has audio" is not "is audible"), and a Veo clip whose prompt demanded one
accent while the frame read **90% emerald** against the ≤10% budget. Rewriting
the colour rule three ways brought it to 1.8%, then 0.7% — but only measuring
proved it. Report the number, never an adjective.

### Audio — silence is a choice, not a default to inherit

Veo 3+ generates **synced audio natively** (its headline feature over Veo 2).
Two things must agree or you get silence:

```python
generate("a machine room", audio="low industrial hum, distant servos")
```

This sets `generateAudio: true` in the request **and** writes the soundscape into
the prompt. Setting only the prompt does nothing — the API param governs.

`audio=False` (the default) is correct **only** for scroll-scrubbed
scrollytelling, where the browser never plays the file. For a hero, demo, or
social video, **ask the user** — do not inherit silence from the scrollytelling
recipe below. That inheritance shipped silent Veo 3 output for exactly this reason.

`generate()` returns `(rung, detail)`. Rungs 1–2 wrote the mp4 and `detail` is the
model that served it. **Rung 3 means `detail` IS the paste-prompt** — that is a
success path, not a failure: show it to the user and keep working. An expired key
falls through to the next rung instead of raising.

| Rung | Needs | Model |
|------|-------|-------|
| 1. Public Gemini API | `GOOGLE_API_KEY` | `gemini-omni-flash-preview` (live since 2026-06-30); Veo 3.1 fallback |
| 2. Vertex AI | `GOOGLE_APPLICATION_CREDENTIALS` (service-account JSON path) + `GOOGLE_CLOUD_PROJECT` + `GOOGLE_CLOUD_LOCATION` | **`veo-3.1-generate-001`** (GA) is the correct primary target. `veo-3.1-generate-**preview**` is a *different, experimental* id that 404s on most projects — mistaking it for "3.1 unavailable" is why builds silently fell back. `veo-3.0-generate-001` was slated for discontinuation **2026-06-30**: last resort only. Omni is tried first only when `audio=False` — it is silent-only and speaks the `v1beta1 interactions` API, not `predictLongRunning`; probing it the Veo way 404s and misreads as "model missing". Always report which model served. |
| 3. **No credentials** | nothing | Emit the paste-prompt below for the user's Gemini app (Omni Flash powers it for subscribers). NOT an error — say "no token, continuing" and proceed. |
| 4. Nothing works | — | Ask the user to drop any mp4 into `workspace/references/video/`. |

Credentials live in the project `.env` (gitignored — verify), written at
bootstrap. Standard var names above (they match gcloud/Vertex conventions).
Never store keys in the SQLite store or any committed file.

## The paste-prompt (rung 3 — write it FOR the user, filled in)

> Generate a short video for a website scroll animation.
> **Audio:** none — silent scroll-scrubbed footage. *(For a hero/social video
> instead, replace this line with the soundscape you want.)*
> **Subject:** <one concrete scene — object, environment, mood>.
> **Style:** premium industrial minimalism; near-monochrome palette on a dark
> `#0A0A0B` background with a single <accent color> accent; soft studio
> lighting; no on-screen text, no logos, no people (unless specified).
> **Camera:** one continuous slow <push-in | orbit | pan>, locked horizon,
> absolutely no cuts.
> **Motion:** smooth, constant speed throughout — this will be scrubbed by
> scroll, so no speed ramps, no flicker.

From INERATE/atelier, MIT.

More from INERATE/atelier