hf-datasets

Files (1)

• SKILL.md

SKILL.md (first lines)

# HuggingFace Datasets

## Overview

The `datasets` library by HuggingFace provides memory-efficient, Apache Arrow-backed access to tens of thousands of public datasets and your own local/remote data. It supports streaming for datasets that don't fit in RAM and integrates directly with `transformers` tokenizers and training loops.

## When to Use

Use this skill when:
- Loading datasets from the HuggingFace Hub or local files (Parquet, JSONL, CSV, Arrow).
- Streaming datasets too large for memory.
- Applying preprocessing, tokenization, or filtering at scale with `.map()` and `.filter()`.
- Pushing curated datasets to the Hub for sharing or versioning.
- Interleaving or concatenating multiple datasets.

Do not use for:
- Small in-memory pandas DataFrames with < 100K rows — use pandas directly.
- Model inference — use `transformers` or `vllm` skills.
- Embedding computation — use `embedding-analysis` skill.

## Installation

```bash
uv pip install datasets pyarrow huggingface_hub
```

## Quick Start

### Loading

```python
from datasets import load_dataset, load_from_disk

# From the Hub
dataset = load_dataset("imdb", split="train")

# From local files
dataset = load_dataset("parquet", data_files="data/*.parquet", split="train")

# From disk (pre-saved)
dataset = load_from_disk("./my_saved_dataset")

# Streaming (never loads full dataset into memory)
dataset = load_dataset("c4", "en", split="train", streaming=True)
```

### Inspection

```python
print(dataset)                    # Dataset info
print(dataset.shape)              # (n_rows, n_columns)
print(dataset.features)           # Schema
print(dataset[0])                 # First row
print(dataset[:5])                # First 5 rows as dict
print(dataset.column_names)       # Column list
```

## Core Operations

### Map (Apply Function)

```python
# Basic mapping
def add_length(example):
    example["text_length"] = len(example["text"])
    return example

dataset = dataset.map(add_length)

# Batched mapping (faster for large datasets)
dataset = dataset.map(add_length, batched=True, batch_size=1000)

# Tokenization with transformers
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

def tokenize(batch):
    return tokenizer(batch["text"], truncation=True, padding="max_length", max_length=512)

dataset = dataset.map(tokenize, batched=True)
```

### Filter

```python
# Quality filter: remove short or low-quality examples
dataset = dataset.filter(lambda x: len(x["text"]) > 100)
dataset = dataset.filter(lambda x: x["score"] > 0.5 if x["score"] is not None else False)

# Language filter
from langdetect import detect
dataset = dataset.filter(lambda x: detect(x["text"]) == "en")
```

### Split

```python
# Random split
train_test = dataset.train_test_split(test_size=0.2, seed=42)
train = train_test["train"]
test = train_test["test"]

# Stratified split (requires class labels)
from datasets import ClassLabel, DatasetDict

split_dataset = dataset.class_encode_column("label").train_test_split(
    test_size=0.2, stratify_by_column="label", seed=42
)
```

### Select, Shuffle, Sort

```python
# Shuffle
dataset = dataset.shuffle(seed=42)

# Select specific indices
dataset = dataset.select(range(1000))

From mkurman/zorai, MIT.

More from mkurman/zorai