av-sync-workflow

Files (4)

• SKILL.md
• beat_sync_principles.md · references/
• audio_analysis.py · scripts/
• simple_sync.py · scripts/

SKILL.md (first lines)

# AV-Sync Workflow

Transform audio into a professionally edited video synchronized to beats, mood, and scene.

## Workflow Overview

```
Audio → Analysis → Clip Matching → Beat Sync → Video Assembly → Export
```

## Step 1: Analyze Audio

Use `scripts/audio_analysis.py` to extract:
- **Beats/BPM**: Timestamp of each beat, overall tempo (BPM)
- **Sections**: Verse, chorus, bridge, outro markers
- **Emotion/Mood**: Energy level, valence (happy/sad), tempo category
- **Key moments**: High-impact points (drops, climaxes, transitions)

```bash
python3 scripts/audio_analysis.py /path/to/song.mp3 --output /tmp/analysis.json
```

Output structure:
```json
{
  "bpm": 120,
  "duration": 214,
  "beats": [0.0, 0.5, 1.0, ...],
  "sections": [
    {"type": "intro", "start": 0, "end": 15},
    {"type": "verse", "start": 15, "end": 45},
    {"type": "chorus", "start": 45, "end": 75}
  ],
  "mood": {"energy": 0.7, "valence": 0.6, "danceability": 0.8},
  "key_moments": [
    {"time": 45.0, "type": "chorus_drop", "intensity": 1.0}
  ]
}
```

## Step 2: Gather Video Clips

User provides video clips OR search for stock footage:

**Stock footage sources:**
- Pexels: `https://www.pexels.com/search/videos/{query}/`
- Pixabay: `https://pixabay.com/videos/search/{query}/`
- Coverr: `https://coverr.co/search/{query}`

**Download stock video:**
```bash
# Via yt-dlp (for pexels/pixabay)
yt-dlp -f "best[height<=1080]" -o "/tmp/clip_%(id)s.%(ext)s" "https://pexels.com/video/12345"

# Via direct URL
ffmpeg -i "https://example.com/video.mp4" -c copy /tmp/clip.mp4
```

## Step 3: Analyze Each Clip

For each clip, extract:
- Scene type (indoor/outdoor, city/nature, close-up/wide)
- Mood/style (energetic/calm, happy/sad)
- Duration and cut points
- Visual elements (faces, motion, colors)

```bash
python3 scripts/video_analysis.py /tmp/clip.mp4 --output /tmp/clip_analysis.json
```

## Step 4: Match Clips to Audio Sections

Algorithm: Map clips to audio sections based on:
1. **Emotion matching**: High-energy chorus → energetic clips
2. **Scene continuity**: Smooth transitions between scenes
3. **Beat alignment**: Cut on beats for rhythm
4. **Length fit**: Clip duration matches section duration

```bash
python3 scripts/match_clips.py \
  --audio-analysis /tmp/analysis.json \
  --clips /tmp/clip1.mp4,/tmp/clip2.mp4 \
  --clip-analyses /tmp/clip1_analysis.json,/tmp/clip2_analysis.json \
  --output /tmp/edit_plan.json
```

## Step 5: Generate Beat-Synced Video

```bash
python3 scripts/assemble_video.py \
  --edit-plan /tmp/edit_plan.json \
  --audio /path/to/song.mp3 \
  --output /tmp/final_video.mp4 \
  --format mp4 \
  --codec h264 \
  --quality high
```

## Reference Scripts

### `scripts/audio_analysis.py`

Analyzes audio file using librosa. Extracts:
- Beat timestamps (per-beat and bar-level)
- BPM
- Onset strength envelope
- Spectral features for mood
- librosa-beat-grid output option

### `scripts/video_analysis.py`

Analyzes video clip:
- Dominant colors / color mood
- Scene type classification (urban, nature, indoor, etc.)
- Motion level (static, moderate, high)
- Detected faces / people
- Suggested cut points (scene changes)

### `scripts/match_clips.py`

From aAAaqwq/AGI-Super-Team, MIT.

More from aAAaqwq/AGI-Super-Team