serverless-modal

Files (1)

• SKILL.md

SKILL.md (first lines)

# Modal Cloud GPU — Training & Inference

Task: $ARGUMENTS

## Overview

**Modal** is a serverless GPU cloud. Key advantages over SSH-based platforms (vast.ai, remote servers):
- **Zero config**: no SSH, no Docker, no port forwarding. Write Python → `modal run` → done.
- **Auto scale-to-zero**: billing stops the instant your code finishes. No idle instances.
- **Local-first**: run `modal run` from your laptop. Code, data, and results stay local; only the GPU function runs remotely.
- **Reproducible environments**: dependencies declared in code via `modal.Image`, not system-level packages.
  Treat the `modal.Image` chain as the RENDERED form of the declarative env spec in
  `../shared-references/compute-env-contract.md` — same spec fields (base, ordered
  pip phases, env vars, smoke probes), same `env:<name>@<specHash>` ledger entry in
  `.aris/compute/modal.md`, same three-tier validation before a long run.

**Best for**: Users without a local GPU who need to debug CUDA code, run small-scale tests, or iterate quickly on experiments. The $5 free tier (no card) is enough for code debugging; $30 (with card) covers most small-scale experiment runs.

**Trade-off**: Modal costs more per GPU-hour than vast.ai or Lightning for some GPU tiers, but eliminates setup time and idle billing, often making it cheaper for short/medium workloads. For long training runs (>4 hours), consider vast.ai for lower $/hr.

## Authentication

```bash
pip install modal
modal setup          # Opens browser login, writes token to ~/.modal.toml
# Verify:
modal run -q 'print("ok")'
```

- Sign up: https://modal.com (GitHub/Google login)
- Free (no card): **$5/month** — enough for quick tests
- Free (with card): **$30/month** — bind a payment method at https://modal.com/settings for the full free tier. Set a **workspace spending limit** to prevent accidental overcharge (Settings → Usage → Spending Limit)
- Academic: apply for $10k credits | Startups: apply for $25k credits
- Secrets: `modal secret create huggingface-secret HF_TOKEN=hf_xxxxx`

> **Recommended setup**: Bind a card to unlock $30/month, then immediately set a spending limit (e.g., $30) so you never exceed the free tier. Modal will pause your workloads when the limit is hit.
>
> **SECURITY WARNING**: Always bind your card and set spending limits directly on https://modal.com/settings in your browser. NEVER enter payment information, card numbers, or billing details through Claude Code or any CLI tool. Only the official Modal website is safe for payment operations.

## Pricing (source: modal.com/pricing, per-second billing)

| GPU | $/sec | ≈$/hr | VRAM | Bandwidth GB/s | Free budget → hours |
|---|---|---|---|---|---|
| T4 | $0.000164 | $0.59 | 16GB | 300 | ~8.5 hr ($5) / 50.8 hr ($30) |
| L4 | $0.000222 | $0.80 | 24GB | 300 | ~6.3 hr / 37.5 hr |
| A10 | $0.000306 | $1.10 | 24GB | 600 | ~4.5 hr / 27.3 hr |
| L40S | $0.000542 | $1.95 | 48GB | 864 | ~2.6 hr / 15.4 hr |
| A100-40GB | $0.000583 | $2.10 | 40GB | 1555 | ~2.4 hr / 14.3 hr |
| A100-80GB | $0.000694 | $2.50 | 80GB | 2039 | ~2.0 hr / 12.0 hr |
| H100 | $0.001097 | $3.95 | 80GB | 3352 | ~1.3 hr / 7.6 hr |
| H200 | $0.001261 | $4.54 | 141GB | 4800 | ~1.1 hr / 6.6 hr |
| B200 | $0.001736 | $6.25 | 192GB | 8000 | ~0.8 hr / 4.8 hr |

CPU: $0.047/core/hr | RAM: $0.008/GiB/hr (GPU typically 90%+ of total cost)

## !! Cost Estimation Required !!

Before EVERY run, estimate cost and show to user for confirmation.

Key insights:
- Inference bottleneck is **memory bandwidth**, not compute → high-bandwidth GPUs are often cheaper overall
- 7-8B BF16 inference needs **~22GB VRAM** (weights 15G + KV cache 1G + overhead), T4 (16GB) insufficient
- H100 is often **cheaper than L4** for benchmarks (11x faster but only 5x more expensive)

### Cost Estimation Template (required before every run)

```
Cost estimate (Modal):
  Model: [name] ([params], [precision])
  VRAM: ~[X]GB (weights + KV cache + overhead)
  GPU: [type] ([VRAM]GB, $[X]/sec = $[X]/hr, bandwidth [X] GB/s)
  Estimate: ~[N] min, ~$[X]
```

### 7-8B BF16 Benchmark Cost Comparison

| GPU | Speed tok/s | $/hr | 1000 samples x 200tok cost | Duration |
|---|---|---|---|---|
| **H100** | **224** | $3.95 | **$0.98** | **15 min** |
| A100-40GB | 104 | $2.10 | $1.12 | 32 min |
| L4 | 20 | $0.80 | $2.22 | 167 min |

## Workflow

### Step 1: Analyze Task → Estimate Cost → Choose GPU

Same analysis as any GPU skill — determine VRAM needs from model size, pick GPU, estimate hours, calculate cost. See pricing table above.

**VRAM Rules of Thumb:**
| Model Size | FP16 VRAM | Recommended GPU |
|---|---|---|
| ≤3B | ~8GB | T4, L4 |
| 7-8B | ~22GB | L4, A10, A100-40GB |
| 13B | ~30GB | L40S, A100-40GB |
| 30B | ~65GB | A100-80GB, H100 |
| 70B | ~140GB | H100:2, H200 |

### Step 2: Generate Modal Launcher

Based on the task type, generate the appropriate launcher script.

#### Pattern A: One-Shot GPU Function (training, evaluation, benchmark)

The most common pattern for `run-experiment` integration. Wraps an existing training script:

```python
import modal

app = modal.App("experiment-name")
# One .pip_install() call per SPEC PHASE (chained calls install in order, so a
# pinned torch in the first call can't be dragged by packages in the second —
# the rendered form of compute-env-contract.md's ordered pip_phases):
image = (
    modal.Image.debian_slim(python_version="3.11")
    .pip_install("torch")                                        # phase 1: pins
    .pip_install("transformers", "accelerate", "datasets", "wandb")  # phase 2
)

# Mount local project code into the container

From wanshuiyin/Auto-claude-code-research-in-sleep, MIT.

More from wanshuiyin/Auto-claude-code-research-in-sleep