video-audio-design

Files (6)

• README.md
• SKILL.md
• evals.json
• audio-mixing-patterns.md · references/
• elevenlabs-api.md · references/
• sfx-generation.md · references/

SKILL.md (first lines)

## Key principles

1. **Layered audio architecture** - Every video has three audio layers:
   narration on top (loudest), SFX in the middle (accent volume), and
   background music at the base (lowest).

2. **Narration drives timing** - Generate narration first, measure its
   duration, then set scene timing to match. Never fit narration into
   arbitrary scene lengths.

3. **Duck music during speech** - Background music must drop 50-60% when
   narration plays. Use smooth ramps (10-15 frames) to avoid jarring jumps.

4. **SFX as accents, not distractions** - Keep SFX short (under 0.5s),
   subtle in volume, and relevant to on-screen action.

5. **Test audio in context** - Always preview the full mix with all layers
   together. Listen for muddy speech, volume spikes, or dead silence.

---

## Core concepts

### 3-layer audio architecture

| Layer | Role | Base Volume | During Narration |
|---|---|---|---|
| Narration | Conveys information, drives pacing | 0.8-1.0 | N/A (top layer) |
| SFX | Accents transitions and actions | 0.3-0.5 | 0.3-0.5 (unchanged) |
| Background Music | Sets emotional tone, fills silence | 0.3-0.5 | 0.15-0.25 (ducked) |

### ElevenLabs API model

ElevenLabs provides neural TTS via a REST API. The core flow:
1. Pick a voice (pre-made or cloned) - each has a `voice_id`
2. Send text + voice settings to `/v1/text-to-speech/{voice_id}`
3. Receive raw audio bytes (mp3 by default)
4. Write to file and measure duration for scene timing

Voice settings:

| Setting | Range | Low | High | Recommended |
|---|---|---|---|---|
| stability | 0-1 | More expressive, variable | More consistent, monotone | 0.4-0.6 |
| similarity_boost | 0-1 | More creative | Closer to original voice | 0.6-0.8 |
| style | 0-1 | Neutral delivery | Exaggerated style | 0.3-0.6 |

### Audio ducking concept

Audio ducking reduces background music volume when narration starts and
restores it when narration ends. In Remotion, use `interpolate()`:

```
Music volume:  0.4 ---\              /--- 0.4
                       \            /
               0.15     \__________/
                     narration start → end
```

Ramps should take 10-15 frames (~0.3-0.5s at 30fps).

### Frame-based audio sync in Remotion

- `useCurrentFrame()` returns the current frame number
- `interpolate()` maps frame ranges to value ranges (e.g., volume)
- `<Sequence from={frame}>` places audio at a specific frame
- `<Audio volume={fn}>` accepts a static number or a per-frame function

Convert seconds to frames: `frames = seconds * fps`.

---

## Common tasks

### 1. Set up ElevenLabs API key and generate narration

```typescript
import fs from 'fs';

const ELEVENLABS_API_URL = 'https://api.elevenlabs.io/v1';

async function generateNarration(
  text: string,
  voiceId: string,
  outputPath: string
): Promise<void> {
  const response = await fetch(
    `${ELEVENLABS_API_URL}/text-to-speech/${voiceId}`,
    {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'xi-api-key': process.env.ELEVEN_LABS_API_KEY!,
      },
      body: JSON.stringify({
        text,
        model_id: 'eleven_multilingual_v2',
        voice_settings: {
          stability: 0.5,
          similarity_boost: 0.75,
          style: 0.5,
          use_speaker_boost: true,
        },
      }),
    }
  );

  if (!response.ok) {
    const error = await response.text();
    throw new Error(`ElevenLabs API error ${response.status}: ${error}`);
  }

  const buffer = Buffer.from(await response.arrayBuffer());
  fs.writeFileSync(outputPath, buffer);
}
```

### 2. Select and configure voice settings

From mkurman/zorai, MIT.

More from mkurman/zorai