video-frames

Files (3)

• SKILL.md
• llm-image-specs.md · references/
• extract_frames.py · scripts/

SKILL.md (first lines)

# Video Frames

Extract frames from video files using ffmpeg, producing JPEG images optimized for LLM vision analysis. Supports multiple frame-selection strategies (fixed FPS, scene detection, target frame count), quality presets, model-aware dimension optimization, and OCR enhancements.

## Prerequisites

ffmpeg and ffprobe must be installed and on PATH:

```bash
brew install ffmpeg  # macOS
```

## Workflow

1. Receive a video file path from the user
2. Run `scripts/extract_frames.py` to extract JPEG frames
3. Parse the JSON output for frame paths, resolution, and token estimates
4. Read the extracted frames as image attachments for analysis
5. Answer the user's question about the video content
6. Clean up temp directories when done

## Quick Start

The simplest invocation -- extracts 1 frame/second at balanced quality:

```bash
python3 scripts/extract_frames.py video.mp4
```

For most use cases, use `--max-frames` to let the script auto-calculate FPS:

```bash
python3 scripts/extract_frames.py video.mp4 --max-frames 30
```

This is the **preferred approach** over manually setting `--fps`, since it adapts to any video length and keeps the frame count predictable.

## Presets

Four quality presets control resolution, JPEG quality, and image processing:

| Preset      | Max dim | Quality | Extras                              | Best for                   |
| ----------- | ------- | ------- | ----------------------------------- | -------------------------- |
| `efficient` | 768px   | 5       | --                                  | Bulk frames, long videos   |
| `balanced`  | 1024px  | 3       | --                                  | General analysis (default) |
| `detailed`  | 1568px  | 2       | --                                  | Fine detail, small objects |
| `ocr`       | 1568px  | 1       | grayscale + high contrast + sharpen | Text/document extraction   |

```bash
# Long video, keep costs low
python3 scripts/extract_frames.py long_video.mp4 --max-frames 20 --preset efficient

# Need to read text in a screencast
python3 scripts/extract_frames.py screencast.mp4 --max-frames 40 --preset ocr
```

Quality (1=best, 31=worst) and max dimension can be overridden independently:

```bash
python3 scripts/extract_frames.py video.mp4 --preset balanced --quality 1 --max-dimension 1568
```

## Scene-Change Detection

Instead of extracting at a fixed rate, detect visual scene changes and extract one frame per scene. This is ideal for videos with distinct segments (presentations, edited footage, tutorials).

```bash
python3 scripts/extract_frames.py video.mp4 --scene-threshold 0.3
```

- `--scene-threshold` (float, 0.0-1.0): Sensitivity. Lower = more sensitive, detects smaller changes. Start with `0.3` (the default when the flag is used).
- `--min-scene-interval` (float, default: 1.0): Minimum seconds between detected scenes. Prevents burst detections during rapid cuts.

**Note:** `--fps` and `--scene-threshold` are mutually exclusive. `--max-frames` can only be used with `--fps` mode, not scene detection.

```bash
# Presentation with clear slide transitions
python3 scripts/extract_frames.py presentation.mp4 --scene-threshold 0.2

# Action footage -- less sensitive, min 2s apart
python3 scripts/extract_frames.py action.mp4 --scene-threshold 0.5 --min-scene-interval 2.0
```

## Model-Aware Optimization

Use `--target-model` to resize frames to dimensions that align with a specific model's tile boundaries, minimizing wasted tokens:

| Model       | Max dim | Rationale                                      |
| ----------- | ------- | ---------------------------------------------- |
| `claude`    | 1568px  | Max native resolution before auto-resize       |
| `openai`    | 768px   | Aligned to 512px tile grid (shortest side 768) |
| `gemini`    | 768px   | Aligned to 768px tile boundaries               |
| `universal` | 768px   | Sweet spot across all models (default)         |

```bash
# Optimized for Claude -- maximum detail
python3 scripts/extract_frames.py video.mp4 --max-frames 30 --target-model claude

# Optimized for GPT-4o -- efficient tile packing
python3 scripts/extract_frames.py video.mp4 --max-frames 30 --target-model openai
```

`--target-model` sets the max dimension unless `--max-dimension` is explicitly provided (CLI override takes priority).

See `references/llm-image-specs.md` for detailed token formulas, tile calculations, and optimal dimension tables for each model.

## OCR and Grayscale Mode

For videos containing text (screencasts, presentations, documents, terminal recordings):

```bash
# Full OCR pipeline via preset
python3 scripts/extract_frames.py screencast.mp4 --preset ocr --max-frames 40

# Manual OCR flags (can combine with any preset)
python3 scripts/extract_frames.py video.mp4 --grayscale --high-contrast
```

- `--grayscale`: Converts frames to grayscale. Reduces file size ~60% with no OCR accuracy loss.

From tdragon6/weAgentSlim, MIT.

More from tdragon6/weAgentSlim