video-analyzer

Files (5)

• README.md
• SKILL.md
• evals.json
• ffmpeg-recipes.md · references/
• vision-analysis-prompts.md · references/

SKILL.md (first lines)

## Key principles

1. **Extract then analyze** - Always separate frame extraction (FFmpeg) from
   semantic analysis (AI vision). Trying to do both in one step leads to
   brittle pipelines. Extract frames to disk first, then analyze them.

2. **Use ffprobe before ffmpeg** - Before processing any video, inspect it
   with ffprobe to understand its properties. Blindly running FFmpeg commands
   on unknown formats leads to silent failures and corrupted output.

3. **Scene detection over fixed intervals** - When analyzing video content,
   extract frames at scene boundaries rather than fixed time intervals. Scene
   change frames capture the visual diversity of the video with far fewer
   frames than one-per-second extraction.

4. **JSON output everywhere** - Use ffprobe's JSON output format and structure
   your analysis results as JSON. This makes pipelines composable and results
   machine-readable.

5. **Disk space awareness** - Video frame extraction can generate thousands of
   large image files. Always estimate output size before extracting, use
   appropriate image formats (JPEG for analysis, PNG for pixel-perfect work),
   and clean up temporary frames after analysis.

---

## Core concepts

### FFmpeg pipeline architecture

FFmpeg processes video through a pipeline of demuxing, decoding, filtering,
encoding, and muxing. For analysis, we primarily use the decode and filter
stages:

```
Input file -> Demuxer -> Decoder -> Filter graph -> Output (frames/data)
```

Key filter concepts for analysis:
- `select` filter: choose which frames to output based on expressions
- `showinfo` filter: print frame metadata (timestamps, picture type, etc.)
- `scene` detection: pixel-level difference score between consecutive frames
- `fps` filter: reduce frame rate to extract at regular intervals

### Scene detection

Scene detection works by comparing consecutive frames using pixel difference.
FFmpeg's `scene` filter produces a score from 0.0 (identical) to 1.0
(completely different). A threshold of 0.3-0.4 catches major scene changes
while ignoring camera motion and lighting shifts.

| Threshold | Behavior |
|-----------|----------|
| 0.1-0.2 | Very sensitive - catches pans, zooms, lighting changes |
| 0.3-0.4 | Balanced - catches cuts, transitions, major changes |
| 0.5-0.7 | Conservative - only hard cuts and dramatic scene changes |
| 0.8-1.0 | Too aggressive - misses most scene changes |

### AI vision analysis workflow

The workflow for extracting structured data from video using AI vision:

1. **Probe** - Get video metadata with ffprobe (duration, resolution, fps)
2. **Extract** - Pull key frames at scene boundaries using FFmpeg
3. **Read** - Load each frame image using the Read tool (supports images)
4. **Analyze** - For each frame, identify colors, typography, layout, components
5. **Aggregate** - Find consistent patterns across frames
6. **Output** - Produce structured design system or content analysis

---

## Common tasks

### 1. Install and verify FFmpeg

Check if FFmpeg is available and inspect its version and capabilities.

```bash
# Check FFmpeg installation
ffmpeg -version

# Check ffprobe installation
ffprobe -version

# Install on macOS
brew install ffmpeg

# Install on Ubuntu/Debian
sudo apt-get update && sudo apt-get install -y ffmpeg

# Verify supported formats
ffmpeg -formats 2>/dev/null | head -20

# Verify supported codecs
ffmpeg -codecs 2>/dev/null | grep -i h264
```

### 2. Extract key frames at scene boundaries

Extract only the frames where significant visual changes occur. This is the
most efficient way to sample video content.

```bash
# Extract frames at scene changes (threshold 0.3)
mkdir -p scenes
ffmpeg -i input.mp4 \
  -vf "select='gt(scene,0.3)',showinfo" \
  -vsync vfr \
  scenes/scene_%04d.png \
  2>&1 | grep showinfo

# Extract with timestamps logged to a file
ffmpeg -i input.mp4 \
  -vf "select='gt(scene,0.3)',showinfo" \
  -vsync vfr \
  scenes/scene_%04d.png \
  2>&1 | grep "pts_time" > scenes/timestamps.txt

# Extract scene frames as JPEG (smaller files, good for analysis)

From mkurman/zorai, MIT.

More from mkurman/zorai