ai-multimodal

Files (15)

• .env.example
• SKILL.md
• audio-processing.md · references/
• image-generation.md · references/
• video-analysis.md · references/
• vision-understanding.md · references/
• document_converter.py · scripts/
• gemini_batch_process.py · scripts/
• media_optimizer.py · scripts/
• requirements.txt · scripts/
• .coverage · scripts/tests/
• requirements.txt · scripts/tests/
• test_document_converter.py · scripts/tests/
• test_gemini_batch_process.py · scripts/tests/
• test_media_optimizer.py · scripts/tests/

SKILL.md (first lines)

# AI Multimodal Processing Skill

Process audio, images, videos, documents, and generate images using Google Gemini's multimodal API. Unified interface for all multimedia content understanding and generation.

## Core Capabilities

### Audio Processing

- Transcription with timestamps (up to 9.5 hours)
- Audio summarization and analysis
- Speech understanding and speaker identification
- Music and environmental sound analysis
- Text-to-speech generation with controllable voice

### Image Understanding

- Image captioning and description
- Object detection with bounding boxes (2.0+)
- Pixel-level segmentation (2.5+)
- Visual question answering
- Multi-image comparison (up to 3,600 images)
- OCR and text extraction

### Video Analysis

- Scene detection and summarization
- Video Q&A with temporal understanding
- Transcription with visual descriptions
- YouTube URL support
- Long video processing (up to 6 hours)
- Frame-level analysis

### Document Extraction

- Native PDF vision processing (up to 1,000 pages)
- Table and form extraction
- Chart and diagram analysis
- Multi-page document understanding
- Structured data output (JSON schema)
- Format conversion (PDF to HTML/JSON)

### Image Generation

- Text-to-image generation
- Image editing and modification
- Multi-image composition (up to 3 images)
- Iterative refinement
- Multiple aspect ratios (1:1, 16:9, 9:16, 4:3, 3:4)
- Controllable style and quality

## Capability Matrix

| Task              | Audio | Image | Video | Document | Generation |
| ----------------- | :---: | :---: | :---: | :------: | :--------: |
| Transcription     |   ✓   |   -   |   ✓   |    -     |     -      |
| Summarization     |   ✓   |   ✓   |   ✓   |    ✓     |     -      |
| Q&A               |   ✓   |   ✓   |   ✓   |    ✓     |     -      |
| Object Detection  |   -   |   ✓   |   ✓   |    -     |     -      |
| Text Extraction   |   -   |   ✓   |   -   |    ✓     |     -      |
| Structured Output |   ✓   |   ✓   |   ✓   |    ✓     |     -      |
| Creation          |  TTS  |   -   |   -   |    -     |     ✓      |
| Timestamps        |   ✓   |   -   |   ✓   |    -     |     -      |
| Segmentation      |   -   |   ✓   |   -   |    -     |     -      |

## Model Selection Guide

### Gemini 2.5 Series (Recommended)

- **gemini-2.5-pro**: Highest quality, all features, 1M-2M context
- **gemini-2.5-flash**: Best balance, all features, 1M-2M context
- **gemini-2.5-flash-lite**: Lightweight, segmentation support
- **gemini-2.5-flash-image**: Image generation only

### Gemini 2.0 Series

- **gemini-2.0-flash**: Fast processing, object detection
- **gemini-2.0-flash-lite**: Lightweight option

### Feature Requirements

- **Segmentation**: Requires 2.5+ models
- **Object Detection**: Requires 2.0+ models
- **Multi-video**: Requires 2.5+ models
- **Image Generation**: Requires flash-image model

### Context Windows

- **2M tokens**: ~6 hours video (low-res) or ~2 hours (default)
- **1M tokens**: ~3 hours video (low-res) or ~1 hour (default)
- **Audio**: 32 tokens/second (1 min = 1,920 tokens)
- **PDF**: 258 tokens/page (fixed)
- **Image**: 258-1,548 tokens based on size

## Quick Start

### Prerequisites

**API Key Setup**: Supports both Google AI Studio and Vertex AI.

The skill checks for `GEMINI_API_KEY` in this order:

1. Process environment: `export GEMINI_API_KEY="your-key"`
2. Project root: `.env`
3. `.claude/.env`
4. `.claude/skills/.env`
5. `.claude/skills/ai-multimodal/.env`

**Get API key**: https://aistudio.google.com/apikey

**For Vertex AI**:

```bash
export GEMINI_USE_VERTEX=true
export VERTEX_PROJECT_ID=your-gcp-project-id
export VERTEX_LOCATION=us-central1  # Optional
```

**Install SDK**:

From vibery-studio/templates, MIT.

More from vibery-studio/templates