llm-assisted-curation

Files (1)

• SKILL.md

SKILL.md (first lines)

# LLM-Assisted Dataset Curation

## Overview

Modern dataset curation uses LLMs as quality filters, rewriters, labelers, and synthetic data generators. This skill covers hosting models locally with vLLM/SGLang and using them for dataset work — not for interactive chat, but for batch, structured, reproducible data operations.

## When to Use

Use this skill when:
- Scoring or filtering dataset examples with an LLM quality judge.
- Rewriting noisy text (queries, answers, reasoning traces) in bulk.
- Generating synthetic examples to balance classes or fill gaps.
- Extracting structured labels from unstructured text.
- Running curriculum scoring (difficulty, complexity, educational value).
- Implementing DataRater-style learned quality scoring (2025).

Do not use for:
- Interactive chat or single-example inspection — use a UI.
- Exact-match deduplication — use hashing.
- Embedding-based dedup — use `embedding-analysis` skill.

## Prerequisites

Requires a running vLLM or SGLang server. See `vllm` and `sglang` skills for server setup.

```bash
# vLLM (high throughput)
vllm serve Qwen/Qwen2.5-7B-Instruct --port 8000 --max-model-len 8192

# SGLang (structured output)
python -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --port 30000
```

## Core Patterns

### 1. LLM-as-Judge Quality Scoring

Score each example on clarity, correctness, and usefulness.

```python
from openai import OpenAI
import json
from datasets import load_dataset

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

QUALITY_PROMPT = """Score the following example on these dimensions (1-5 each):
- clarity: Is the text well-written and understandable?
- correctness: Are the facts accurate?
- usefulness: Would this help someone learn or solve a problem?

Respond with ONLY valid JSON: {"clarity": N, "correctness": N, "usefulness": N}

Example:
{sample}
"""

def score_example(sample: dict) -> dict:
    prompt = QUALITY_PROMPT.format(sample=json.dumps(sample))
    response = client.chat.completions.create(
        model="Qwen/Qwen2.5-7B-Instruct",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0,  # deterministic
        max_tokens=128,
    )
    try:
        scores = json.loads(response.choices[0].message.content)
    except json.JSONDecodeError:
        scores = {"clarity": 0, "correctness": 0, "usefulness": 0}
    return {**sample, **scores}

# Batch scoring with datasets
dataset = load_dataset("my-dataset", split="train")
scored = dataset.map(score_example)

# Filter low-quality examples
filtered = scored.filter(lambda x: x["clarity"] >= 3 and x["correctness"] >= 3)
```

### 2. Structured Output Filtering (SGLang)

Use SGLang's constrained decoding for guaranteed JSON schema output.

```python
import sglang as sgl

@sgl.function
def classify_quality(s, text: str):
    s += sgl.system("You classify dataset examples. Output ONLY valid JSON.")
    s += sgl.user(f"Classify this example:\n\n{text}")
    s += sgl.gen("result", max_tokens=256, temperature=0.0, schema=json.dumps({
        "type": "object",
        "properties": {
            "quality": {"type": "string", "enum": ["high", "medium", "low", "noise"]},
            "language": {"type": "string", "enum": ["en", "code", "other"]},
            "topic": {"type": "string"},
            "issues": {"type": "array", "items": {"type": "string"}},
        },
        "required": ["quality", "language", "topic", "issues"],
    }))

state = classify_quality.run(text=example["text"])
result = state["result"]  # guaranteed valid JSON
```

### 3. Batch Rewriting/Refinement

Clean noisy data by rewriting through an LLM.

```python
REWRITE_PROMPT = """Rewrite the following text to be clear, grammatical, and well-structured.
Preserve all factual information. Fix typos, grammar, and awkward phrasing.

Original: {text}

Rewritten:"""

def rewrite_text(sample: dict, client, model: str) -> dict:
    prompt = REWRITE_PROMPT.format(text=sample["text"])

From mkurman/zorai, MIT.

More from mkurman/zorai