dataset-versioning

Files (1)

• SKILL.md

SKILL.md (first lines)

# Dataset Versioning

## Overview

Datasets are artifacts that evolve. Versioning turns "which version did we use?" from a forensic investigation into a one-line answer. Every released dataset gets a checksum, a manifest, and a semantic version tag.

## When to Use

Use this skill when:
- Creating, cleaning, or updating a dataset that others (or future-you) depend on.
- Tracking provenance: what source data, code, and parameters produced this dataset.
- Collaborating on datasets where multiple versions coexist.
- Reproducing past experiments that depend on a specific dataset version.

Do not use for:
- One-off exploration that won't be reused.
- Interim files that are fully regenerable from versioned sources.
- Model versioning — use `mlflow` or `wandb` skills.

## Versioning Workflow

### 1. Initialize Version Tracking

```bash
# DVC (Data Version Control) — recommended for datasets > 10 MB
pip install dvc
dvc init
git commit -m "Initialize DVC"
```

Or minimal filesystem-based versioning:

```python
import hashlib
import json
from pathlib import Path
from datetime import datetime, timezone

DATASET_ROOT = Path("datasets")
VERSION_REGISTRY = DATASET_ROOT / "versions.jsonl"

def compute_checksum(filepath: Path, algo: str = "sha256") -> str:
    h = hashlib.new(algo)
    with open(filepath, "rb") as f:
        for chunk in iter(lambda: f.read(8192), b""):
            h.update(chunk)
    return h.hexdigest()
```

### 2. Create a Dataset Manifest

Every dataset release needs a `manifest.json`:

```json
{
  "name": "customer-churn-prediction",
  "version": "1.0.0",
  "created_at": "2026-05-11T17:29:51Z",
  "description": "Cleaned customer churn dataset for binary classification",
  "files": {
    "train.parquet": {
      "checksum_sha256": "a1b2c3d4...",
      "rows": 80000,
      "columns": 24,
      "size_bytes": 5242880
    },
    "val.parquet": {
      "checksum_sha256": "e5f6g7h8...",
      "rows": 10000,
      "columns": 24,
      "size_bytes": 655360
    },
    "test.parquet": {
      "checksum_sha256": "i9j0k1l2...",
      "rows": 10000,
      "columns": 24,
      "size_bytes": 655360
    },
    "schema.json": {
      "checksum_sha256": "m3n4o5p6...",
      "size_bytes": 2048
    },
    "cleaning_audit.json": {
      "checksum_sha256": "q7r8s9t0...",
      "size_bytes": 4096
    }
  },
  "provenance": {
    "source": "https://data.example.com/customers/export/2026-Q1",
    "query_timestamp": "2026-05-10T08:00:00Z",
    "cleaning_script": "scripts/clean_churn.py",
    "cleaning_script_sha256": "u1v2w3x4...",
    "split_seed": 42,
    "transformations_applied": [
      "median_imputation_age",
      "dedup_on_user_id",
      "cap_age_0_to_120",
      "one_hot_encode_region"
    ]
  },
  "schema": {
    "target_column": "churned",
    "protected_attributes": ["gender", "age_group"],
    "train_val_test_split": [0.70, 0.15, 0.15]
  },
  "license": "CC-BY-4.0",
  "limitations": [
    "Data from Q1 2026 only; seasonal patterns not captured",
    "Region X undersampled (3% vs 15% in production)"
  ]
}
```

### 3. Version with DVC

```bash
# Track dataset files with DVC
dvc add datasets/customer-churn-v1.0.0/

From mkurman/zorai, MIT.

More from mkurman/zorai