VLM

Files (3)

• LICENSE.txt
• SKILL.md
• vlm.ts · scripts/

SKILL.md (first lines)

# VLM(Vision Chat) Skill

This skill guides the implementation of vision chat functionality using the z-ai-web-dev-sdk package, enabling AI models to understand and respond to images combined with text prompts.

## Skills Path

**Skill Location**: `{project_path}/skills/VLM`

this skill is located at above path in your project.

**Reference Scripts**: Example test scripts are available in the `{Skill Location}/scripts/` directory for quick testing and reference. See `{Skill Location}/scripts/vlm.ts` for a working example.

## Overview

Vision Chat allows you to build applications that can analyze images, extract information from visual content, and answer questions about images through natural language conversation.

**IMPORTANT**: z-ai-web-dev-sdk MUST be used in backend code only. Never use it in client-side code.

## Prerequisites

The z-ai-web-dev-sdk package is already installed. Import it as shown in the examples below.

## CLI Usage (For Simple Tasks)

For simple image analysis tasks, you can use the z-ai CLI instead of writing code. This is ideal for quick image descriptions, testing vision capabilities, or simple automation.

### Basic Image Analysis

```bash
# Describe an image from URL
z-ai vision --prompt "What's in this image?" --image "https://example.com/photo.jpg"

# Using short options
z-ai vision -p "Describe this image" -i "https://example.com/image.png"
```

### Analyze Local Images

```bash
# Analyze a local image file
z-ai vision -p "What objects are in this photo?" -i "./photo.jpg"

# Save response to file
z-ai vision -p "Describe the scene" -i "./landscape.png" -o description.json
```

### Multiple Images

```bash
# Analyze multiple images at once
z-ai vision \
  -p "Compare these two images" \
  -i "./photo1.jpg" \
  -i "./photo2.jpg" \
  -o comparison.json

# Multiple images with detailed analysis
z-ai vision \
  --prompt "What are the differences between these images?" \
  --image "https://example.com/before.jpg" \
  --image "https://example.com/after.jpg"
```

### With Thinking (Chain of Thought)

```bash
# Enable thinking for complex visual reasoning
z-ai vision \
  -p "Count the number of people in this image and describe their activities" \
  -i "./crowd.jpg" \
  --thinking \
  -o analysis.json
```

### Streaming Output

```bash
# Stream the vision analysis
z-ai vision -p "Describe this image in detail" -i "./photo.jpg" --stream
```

### CLI Parameters

- `--prompt, -p <text>`: **Required** - Question or instruction about the image(s)
- `--image, -i <URL or path>`: Optional - Image URL or local file path (can be used multiple times)
- `--thinking, -t`: Optional - Enable chain-of-thought reasoning (default: disabled)
- `--output, -o <path>`: Optional - Output file path (JSON format)
- `--stream`: Optional - Stream the response in real-time

### Supported Image Formats

- PNG (.png)
- JPEG (.jpg, .jpeg)
- GIF (.gif)
- WebP (.webp)
- BMP (.bmp)

### When to Use CLI vs SDK

**Use CLI for:**
- Quick image analysis
- Testing vision model capabilities
- One-off image descriptions
- Simple automation scripts

**Use SDK for:**
- Multi-turn conversations with images
- Dynamic image analysis in applications
- Batch processing with custom logic
- Production applications with complex workflows

## Recommended Approach

For better performance and reliability, use base64 encoding to pass images to the model instead of image URLs.

## Supported Content Types

The Vision Chat API supports three types of media content:

From nordeim/health-care-clinic, MIT.