dubbing

Files (33)

• SKILL.md
• openai.yaml · agents/
• api_adapter.mjs · lib/
• billing.mjs · lib/
• client_info.mjs · lib/
• config.mjs · lib/
• ffmpeg.mjs · lib/
• gates.mjs · lib/
• http_client.mjs · lib/
• input.mjs · lib/
• languages.mjs · lib/
• mask.mjs · lib/
• merge.mjs · lib/
• messages.mjs · lib/
• scheduler.mjs · lib/
• space.mjs · lib/
• split.mjs · lib/
• status.mjs · lib/
• telemetry.mjs · lib/
• tmp.mjs · lib/
• tts_models.mjs · lib/
• update_check.mjs · lib/
• web_attribution.mjs · lib/
• billing.mjs · scripts/
• check_deps.mjs · scripts/
• connect.mjs · scripts/
• dubbing.mjs · scripts/
• languages.mjs · scripts/
• merge.mjs · scripts/
• prepare_input.mjs · scripts/
• probe_split.mjs · scripts/
• resolve_key.mjs · scripts/
• speaker.mjs · scripts/

SKILL.md (first lines)

# /dubbing

A skill that auto-dubs videos via the Perso AI Dubbing API.

## Core rules (must follow)

- **Only the worker sees the raw key.** Never open the key with `Read`, echo it, or pass it as a command-line argument. Workers send it in the `XP-API-KEY` header only.
- **One command = the whole job.** `scripts/dubbing.mjs` handles upload · split · translate · merge · save by itself. **Never run `prepare_input.mjs` or `probe_split.mjs` before it** — they are debug tools, and `probe_split.mjs` performs a real upload (the video would upload twice).
- **Always run `dubbing.mjs` in the background.** Jobs take minutes to hours; a foreground shell timeout kills the run mid-way.
- Languages unspecified → source `auto`, target `en`.
- **Multiple languages go in one command** (`--target en,zh,ja`) — upload/split happens once and is reused per language. Never run once per language (re-uploads the source each time).
- **Relay progress faithfully.** Surface the worker's stdout `[progress]` lines to chat, verbatim or summarized — and don't announce steps that didn't appear (splitting/merging exist only for over-limit videos). The indented stderr detail logs don't need relaying.
- **Keep the user informed while it runs.** Dubbing, lip-sync, and audio separation all print a `[status]` heartbeat (stage + item counts, e.g. `ja 4/6 · en 1/6` or `separating — file 2/5`, with a `next check ~Nm` hint), and each finished item streams as a `[progress]` line. Check the background output ~3 minutes after launch, then on the hinted interval (10 min for media ≤10 min, 30 min for longer): relay any new `[progress]` line right away, otherwise post the latest `[status]` line as a one-line update. Never invent percentages, elapsed time, or ETAs — relay only what the lines say. Stops that need the user (`[space-select]`, `[split-confirm]`, `[free-limit]`, `[model-plan]`, `[model-select]`, `[resume-check]`, `[credit-check]`, credit-out, completion) end the process, so they surface without polling.
- **Name your runtime for telemetry (silent).** Add `--host <claude-code|codex|cursor|antigravity|gemini>` to the worker command, naming the agent you are running as, if you can tell. It only tags usage telemetry — never surface it to the user and never ask them about it. Omit it if unsure.
- **Voice model — mention only when asked.** Never bring up TTS voice models or `--tts-model` on your own: run with the default and say nothing about models in kickoff lines, progress, or the final report. Explain the models and let the user choose **only** when the user asks about voice models / voice engines, or the worker prints a `[model-plan]` or `[model-select]` line. When the user names a model ("use Nightingale"), don't explain or ask — state its price in one line and run it right away — see **Voice model**.
- **Unsupported formats are skipped automatically** and the rest keep processing — relay the skip notice.
- **A pause is not a failure.** A `[space-select]`, `[split-confirm]`, `[free-limit]`, `[model-plan]`, `[model-select]`, `[credit-check]`, or `[resume-check]` line means the run **stopped to ask you something** — it exits 0 (so it never looks like an error), but the job is **not** complete. Whenever one of these lines is present, never report the run as done: relay it and act on it. The lines are the signal, not the exit code.

## Setup (lazy — no upfront step)

1. **API key** — **not a step you run first.** The worker gates itself the moment a command that needs the Perso API runs: on the first such run it opens a browser page (sign in, one click — the key is stored encrypted), or prints file-based instructions if no browser can open — relay them to the user. On a headless/SSH machine the worker skips the browser flow entirely and starts with file/`--import` registration (the browser page can only deliver a key to the machine it runs on) — relay those instructions the same way. Never trigger key registration yourself: the worker spawns `connect.mjs`/`resolve_key.mjs --watch` when needed, so don't run those, and don't run `resolve_key.mjs --check` as a pre-step (only if the user explicitly asks whether a key is registered — a missing key is not an error). Never paste the key into chat. Get a key: https://developers.perso.ai/api-keys
2. **ffmpeg/ffprobe** — auto-installed only when a video exceeds the plan limit and must be split (approve if permission is requested). Manual check: `node scripts/check_deps.mjs`.

## Run

Collect the input (local path or URL — re-ask if missing) and run **in the background** — the worker performs the key gate itself on first need; just relay its printed instructions if registration starts:

- Single: `node scripts/dubbing.mjs "<file|URL>" [--source auto] [--target en] [--space "space name"] [--out result.mp4] [--lipsync] [--no-save]`
- Multi-language / multi-input (one or more inputs; URLs and files can be mixed): `node scripts/dubbing.mjs "<URL>" "<file>" --target en,ja` — one output per input × language, saved next to each source (`--out <folder>` collects them).
- Folder (batch): `node scripts/dubbing.mjs "<folder>" [--target en,zh] [--recursive] [--out output-folder]`

**Language codes** — `--target` takes a language code (`en`, `ja`) or a regional tag. Regional variants exist for **English** (`en` = US default, `en-GB` = UK), **Portuguese** (`pt` = Brazil default, `pt-PT` = Portugal) and **Spanish** (`es` = Mexico default, `es-ES` = Spain) only; `en-US`, `pt-BR` and `es-MX` are accepted spellings of those defaults. Every other language has a single variant, and `--source` only needs the language (a regional tag there is treated as the plain language).

- Region named in the request → use that exact tag. A region with **no** variant (e.g. Australian English) → tell the user only the listed options exist and let them pick before running.
- **Two-letter tokens are language codes, not countries**: `uk` is **Ukrainian** — British English is `en-GB`. Same trap for `ca` (Catalan), `be` (Belarusian), `eu` (Basque). If the user's wording could mean either, ask which they mean **before** running.
- Language named without a region → the default variant. For those three languages, name the applied default in the delivery note (e.g. "Spanish (Mexico) — use `es-ES` for Spain").

**Save location** — `--out` takes either a **folder** (an existing directory, or a path ending in `/`) which collects the results inside it, or a **file path** for a single input (the real extension is appended if the path has none). Default is next to each source file; don't ask. On the session's first job, state it in the kickoff line (e.g. "Results will be saved next to the originals — tell me now if you'd like a different folder") and start right away, without waiting for an answer. If the user names a folder — with this request or any time later in this session — run with `--out "<folder>"` and keep using that folder for every later job until they change it. If they redirect after a run already started, move the finished files there when the run completes. Exception: when every input is a URL (no local source folder), ask where to save before starting.

**Space selection** — with several workspaces the worker stops before uploading, prints `[space-select]` lines (**name | (plan) | remaining credits**) and stops. Show the user ONLY those options (no internal numbers), ask which one, and re-run with `--space "<space name>"`. One space → no question; `PERSO_SPACE_SEQ` pins it.

**Split confirmation** — if the input exceeds the length or size limit and must be auto split & merged (dubbing, dub+lip-sync, or audio separation), the worker stops and prints `[split-confirm]` lines. Relay them and ask the user: it exceeds the length/size limit, so it needs **automatic split → process → merge**, which can come out **less polished than splitting it up themselves** — proceed automatically? On a yes, re-run the **same command with `--allow-split`** (nothing is billed until this point, so re-running is free). Batch runs: `--allow-split` authorizes every split in the run.

**Don't save (server-only)** — when the user wants the video dubbed but **not** saved as a local file, add `--no-save`: the worker leaves the result in the user's Perso workspace and skips the download, printing `Kept on server, not saved: … → project <seq>` (the `[project-ref]` line is still emitted, so the dub can be lip-synced or retrieved later). **Single/unsplit videos only** — a split video's merged file needs a local download, so it is saved normally and the worker says so. Cannot be combined with `--lipsync` (the lip-synced video must be downloaded).

**Free plan (results stay on Perso)** — a Free workspace can dub, lip-sync and separate normally, but the server refuses **every** download, so the worker never saves a file there. It prints `Free plan — <name>: … ready on Perso → <url> (downloads need a paid plan)` — relay that link as **the delivered result**: the run is **complete**, nothing failed and there is nothing to resume. If the user wants actual files, that needs a paid plan — offer the upgrade path (see **Plan upgrade & credits**), never a re-run.

**Free plan `[free-limit]` pauses** — two of them, both before anything is billed:

- **30-second preview.** A dub/lip-sync input longer than 30 s stops with `[free-limit]` lines: on Free only the **first 30 seconds** would be dubbed, the rest left out. Relay the question, and on a yes re-run the **same command with `--allow-preview`** (the worker trims the first 30 s locally and submits only that). A platform link imported by Perso can't be trimmed — there the answer is a local file, a shorter clip, or an upgrade.
- **Over the plan limit.** Media past the plan's length/size limit gets **no** split offer on Free (`--allow-split` does not apply): the `[free-limit]` lines say a shorter file or an upgrade is the only way. Relay them and stop.

While it runs (for explaining the wait):

- **Split**: the whole file is uploaded first; only a plan-limit rejection installs ffmpeg and splits losslessly. External URLs (YouTube·TikTok·Drive·Vimeo) are handled server-side.
- **Queue**: all inputs × parts × languages share one pool; a full queue is re-checked every 5 minutes. An engine error on a part cancels that part's other languages; a silent part passes the original through; an idle guard prevents hanging forever.
- **Save**: parts are merged back into one file per (input × language). An unsplit output keeps the Perso filename; a merged one is `<original-name>.dubbed.<lang>.<ext>`; collisions get `_2`,`_3`….

## Voice model (only when the user asks)

`--tts-model <linnet_4|linnet_3|oriole|wren|finch|nightingale>` picks the TTS voice model for the whole run (every `--target`). Omitted → no model is sent and the server applies its default (currently `linnet_4`, or `oriole` for a language without it). Legacy names still work (`dodo`/`eleven_v3` → `linnet_3`, `eleven_v2` → `wren`, `audio_engine_v3` → `oriole`). Follow the Core rule: never offer this unless the user asks about voice models or the worker prints a model line.

When the user asks, explain with these facts only (don't describe voice character or quality — there is no source for it):

| Model | Plan | Credits (dubbing) | Lip-sync | Languages |
|---|---|---|---|---|
| `linnet_4` (server default) | any | ≈ ×1/s (4K ×3 on pro/business/enterprise) | yes | most languages |
| `oriole` | any | ≈ ×1/s (4K ×3 on pro+) | yes | most languages (the default where `linnet_4` is missing) |
| `linnet_3` | any | ≈ ×1/s (4K ×3 on pro+) | yes | fewer languages |
| `wren` | any | ≈ ×1/s (4K ×3 on pro+) | yes | fewer languages |
| `finch` | **Pro or higher** | **0.2/s** (4K ×3 on pro+) | yes | most languages |
| `nightingale` | **Pro or higher** | **3/s during the launch event** (regular 6/s; no 4K surcharge) | **not yet** (in preparation) | some languages are Nightingale-only |

Server billing is authoritative.

**The user names a model** ("use Nightingale", "나이팅게일로 해줘") → no table, no question: tell them the price in one line (Nightingale: 3 credits per second during the launch event, regular 6; Finch: 0.2 credits per second) and start the run with `--tts-model <name>` right away. The worker checks the plan and the languages before uploading — if one doesn't fit it stops with `[model-plan]` / `[model-select]` / an unsupported-language error, and you relay that instead.
 For which models a given language supports, run `node scripts/languages.mjs` (each line lists its `models`). Once the user picks, add `--tts-model <name>` and keep it on every re-run of that job (`--allow-split`, `--allow-preview`, `--force`).

- **`[model-plan]`** — the chosen model isn't available on this space's plan (Nightingale and Finch need Pro or higher). Nothing was uploaded or billed. Ask the user one question: **switch to another voice model** (the lines list the ones this plan allows) **or dub without choosing a model** (the server default). Re-run the same command with `--tts-model <model>`, or without `--tts-model`. If the user asks to upgrade instead, see **Plan upgrade & credits** and re-run the same command after payment.
- **`[model-select]`** — the server's default model can't dub a requested language; the lines list the models that can (and whether this plan allows them). Nothing was uploaded or billed. Ask the user which to use and re-run the same command with `--tts-model <model>`; never pick a pricier model for them.
- `--tts-model` applies to new dubbing only: it can't be combined with `--separate` or `--lipsync-only`, and `--resume` keeps the model the run started with.
- If the server still refuses a submission, the worker stops without retrying and prints the server's code and message (`Could not dub: … rejected by the server (<code>): …`) plus a hint — relay them.

## Lip-sync

Lip-sync (mouth matched to the dubbed audio) runs **after** dubbing, on the finished dubbing project. Video only — audio inputs are rejected. It is a long job: **run in the background and tell the user up-front it takes considerably longer than dubbing.** Credits (server billing is authoritative): dubbing ≈ seconds ×1 · lip-sync ≈ ×2 · both ≈ ×3 — dubbing now + lip-sync later costs the same as both at once. **Exception: lip-sync for `nightingale` is still in preparation** (the worker refuses `--lipsync` with it, and `--lipsync-only` on a Nightingale dub) — until it ships, lip-sync needs a dub made with another model, which bills the dubbing again. **4K+ sources: every rate ×3 on pro/business/enterprise plans** (e.g. a 1-min 4K dub+lip-sync ≈ 60×3×3 = 540) — mention this when the video is 4K.

Pick the flow by what exists:

1. **New video + lip-sync** — one command runs the whole chain (dub → lip-sync → save); warn that both stages bill:
   `node scripts/dubbing.mjs "<file|URL>" --target en --lipsync`
   If `[credit-check]` lines print and it stops, the estimate exceeds the remaining credits: show those lines (top-up URL included), and re-run with `--force` only after the user tops up or approves continuing anyway.
2. **Dubbed earlier in this session** — every finished run prints a `[project-ref] {...}` line. **Keep it; never show it to the user.** Lip-sync without re-dubbing (×2, no dubbing charge):
   `node scripts/dubbing.mjs --lipsync-only '<that [project-ref] JSON>'`
3. **No [project-ref] in this session** — ask if the user knows the project number from the Perso portal (`--lipsync-only <number>`, ×2). Otherwise the video must be dubbed again (`--lipsync`, ×3) — confirm before re-dubbing.

Rules:

- **Repeating lip-sync on the same project bills again** (no server-side dedup). If this session already lip-synced it, point at the existing file and re-run only on explicit confirmation.
- **Lip-sync for Nightingale dubs is not available yet (in preparation).** `--lipsync-only` checks which voice model made the dub (from the `[project-ref]`, or by asking the server for a bare project number) and stops with `[lipsync-unavailable]` lines before any request — nothing is billed. Relay it as "not available yet", never as a permanent limitation: lip-sync for Nightingale dubs is still being prepared. If the user wants a lip-synced video now, the way is to dub the original again with the default voice model plus lip-sync (`--lipsync`, no `--tts-model`), which bills the dubbing again — offer it, and run it only on the user's explicit OK. If the user wants lip-sync, don't suggest Nightingale for the new dub.
- **If lip-sync fails, the worker saves the dubbed video instead** and says so in the final report — relay that clearly; the dubbing credits are not wasted.
- Credits running out between dubbing and lip-sync: the dubbed videos are saved and continuing finishes only the lip-sync — relay the top-up URL, and continue via the `[resume-state]` path once paid (see **Interruption & resume**).

## Speaker (dubbing projects only)

To assign a **new speaker** to one or more sentences of an already-dubbed project, run:

`node scripts/speaker.mjs "<project URL|projectSeq>" --at "5s,22s,98s"` (comma-separated timecodes/ranges)
`node scripts/speaker.mjs "<project URL|projectSeq>" --text "first line|second line"` (pipe-separated, quoted text)
`node scripts/speaker.mjs "<project URL|projectSeq>" --sentence 1234,1288` (re-run after a `[sentence-select]` question)
`node scripts/speaker.mjs "<project URL|projectSeq>" --list` (browse the project's sentences to find the right target)

**Dubbing projects only** — the worker rejects subtitle (STT), audio-separation, and lip-sync projects, and unfinished/failed ones. It works out the workspace on its own from the project link; you don't need to ask which space.

**A pause is not a failure.** A `[sentence-select]` line means the worker couldn't pin down a single target (nothing was given, nothing matched, or more than one sentence matched) — it exits cleanly to ask you something, not because it failed. The numbered list under it (index, time label, text) is safe to relay to the user as-is. Right after that list, a `[sentence-ref]` line prints a JSON map of the same numbers to the worker's internal sentence numbers — that line is **your own lookup only**: never show it or read it aloud. Once the user picks a number, look it up in that map and re-run the same command with `--sentence` set to that value. A `[speaker-resolve]` block alongside it means some targets already resolved but **nothing was written yet** — it has its own numbered list and its own `[sentence-ref]` line; the re-run's `--sentence` list must include both those already-resolved values and the newly chosen one(s), or the resolved ones would be dropped. When a run prints more than one `[sentence-select]` group (several ambiguous targets at once), each group has its own list and its own `[sentence-ref]` right after it — the numbering restarts per group, so don't mix a number from one group with the ref line of another. A `[speaker-confirm]` line (more than 10 targets at once) works the same way — a numbered list, then its `[sentence-ref]` — and needs a plain yes/no before writing.

Relay `[progress]` lines as the worker reads the script, and `[speaker-added]` lines as each new speaker is created. On `[script-unavailable]`, relay the printed status (still generating / failed / try again) rather than treating it as an error — the project itself is fine, its script just isn't readable yet.

Pass `--host` silently, same as the other commands. If the user wants the project link, build it from whatever project reference they originally gave you — don't invent one.

From perso-ai/perso-dubbing-plugin, MIT.

More from perso-ai/perso-dubbing-plugin