diffusion-models

Files (1)

• SKILL.md

SKILL.md (first lines)

# Diffusion Models

## Overview

Diffusion models learn to reverse a gradual noising process to generate high-quality images from noise, conditioned on text prompts, images, or other signals. The Hugging Face Diffusers library provides a consistent API across Stable Diffusion, SDXL, Kandinsky, DeepFloyd, and ControlNet pipelines. Fine-tuning via LoRA (Low-Rank Adaptation) lets you adapt any base model to custom styles or subjects with minimal compute and no catastrophic forgetting.

## When to Use

- Generating product images, marketing assets, or concept art from text descriptions
- Inpainting to remove objects or replace backgrounds in existing images
- Style transfer and image-to-image transformations
- Fine-tuning a base model on proprietary brand imagery or character consistency
- ControlNet for pose/depth/edge-conditioned generation
- Building a production image generation API with GPU batching
- Creating synthetic training data for downstream computer vision models

## Step-by-Step Workflow

### 1. Basic Text-to-Image Pipeline

```python
# pip install diffusers transformers accelerate torch
from diffusers import StableDiffusionXLPipeline
import torch

def load_sdxl_pipeline(model_id: str = "stabilityai/stable-diffusion-xl-base-1.0") -> StableDiffusionXLPipeline:
    """Load SDXL with memory optimizations for consumer GPUs."""
    pipe = StableDiffusionXLPipeline.from_pretrained(
        model_id,
        torch_dtype=torch.float16,
        use_safetensors=True,
        variant="fp16",
    )
    pipe = pipe.to("cuda")

    # Memory optimizations
    pipe.enable_model_cpu_offload()  # Moves models to CPU when not in use
    pipe.enable_vae_slicing()         # Processes VAE in slices to reduce VRAM
    pipe.enable_attention_slicing()   # Slices attention computation

    return pipe

def generate_image(
    pipe: StableDiffusionXLPipeline,
    prompt: str,
    negative_prompt: str = "blurry, low quality, distorted",
    width: int = 1024,
    height: int = 1024,
    num_inference_steps: int = 30,
    guidance_scale: float = 7.5,
    seed: int | None = None,
):
    generator = torch.Generator("cuda").manual_seed(seed) if seed is not None else None
    images = pipe(
        prompt=prompt,
        negative_prompt=negative_prompt,
        width=width,
        height=height,
        num_inference_steps=num_inference_steps,
        guidance_scale=guidance_scale,
        generator=generator,
    ).images
    return images[0]

# Usage
pipe = load_sdxl_pipeline()
image = generate_image(
    pipe,
    prompt="A majestic wolf howling at the moon, dramatic lighting, photorealistic, 8k",
    seed=42,
)
image.save("output.png")
```

### 2. Image-to-Image and Inpainting

```python
from diffusers import StableDiffusionXLImg2ImgPipeline, StableDiffusionXLInpaintPipeline
from PIL import Image
import torch

# Image-to-image: transform an existing image
def img2img(
    source_image: Image.Image,
    prompt: str,
    strength: float = 0.7,  # 0=no change, 1=completely new image
    guidance_scale: float = 7.5,
    num_steps: int = 30,
) -> Image.Image:
    pipe = StableDiffusionXLImg2ImgPipeline.from_pretrained(
        "stabilityai/stable-diffusion-xl-refiner-1.0",
        torch_dtype=torch.float16,
        use_safetensors=True,
        variant="fp16",
    ).to("cuda")

    result = pipe(
        prompt=prompt,
        image=source_image.resize((1024, 1024)),
        strength=strength,
        guidance_scale=guidance_scale,
        num_inference_steps=num_steps,
    ).images[0]
    return result

# Inpainting: replace masked regions
def inpaint(
    source_image: Image.Image,
    mask_image: Image.Image,   # White=region to fill, Black=keep
    prompt: str,
    negative_prompt: str = "blurry, artifacts",
    num_steps: int = 50,
) -> Image.Image:
    pipe = StableDiffusionXLInpaintPipeline.from_pretrained(
        "diffusers/stable-diffusion-xl-1.0-inpainting-0.1",
        torch_dtype=torch.float16,
        use_safetensors=True,
    ).to("cuda")

From UltronCore/claude-skill-vault, MIT.

More from UltronCore/claude-skill-vault