dataset-cleaning

Files (1)

• SKILL.md

SKILL.md (first lines)

# Dataset Cleaning

## Overview

Dataset cleaning transforms raw, messy data into a consistent, analysis-ready form. Every operation must be documented: what was changed, why, and how many rows/columns were affected. This is not a one-shot script — it's a reproducible pipeline.

## When to Use

Use this skill when:
- Raw data has missing values, duplicates, inconsistent types, or outliers.
- Data comes from multiple sources with conflicting formats or encodings.
- You need a documented cleaning pipeline, not silent `dropna()`.
- The dataset will be used for modeling, analysis, or sharing.

Do not use for:
- Exploratory analysis without transformation — use `exploratory-data-analysis` first.
- Splitting strategies — use `dataset-splitting`.
- Version tracking — use `dataset-versioning`.

## Cleaning Workflow

### 1. Missing Value Strategy

Choose and document ONE strategy per column:

| Strategy | When to use | Risk |
||-------------|------|
| **Drop rows** | < 5% missing, rows are independent | Loss of rare cases |
| **Drop column** | > 40% missing and not critical | Loss of signal |
| **Mean/median imputation** | Continuous, symmetric distribution | Underestimates variance |
| **Mode imputation** | Categorical, dominant class clear | Over-represents majority |
| **Constant fill** | Domain-knowledge default exists | May introduce bias |
| **Model-based imputation** | High missingness, strong predictors | Leakage if not careful |
| **Indicator flag** | Missingness itself is informative | Adds dimensionality |

```python
import pandas as pd
import numpy as np

# NEVER do this silently:
# df.dropna(inplace=True)

# Instead — audit first:
missing = df.isnull().sum()
missing_pct = df.isnull().mean() * 100
print(missing_pct[missing_pct > 0].sort_values(ascending=False))

# Documented imputation with audit trail:
audit = {}
mask = df['age'].isnull()
audit['age_imputed_count'] = mask.sum()
df.loc[mask, 'age'] = df['age'].median()
df['age_imputed'] = mask.astype(int)  # flag for downstream awareness
```

### 2. Deduplication

```python
# Define identity columns explicitly before deduping
identity_cols = ['user_id', 'timestamp']
n_before = len(df)
df = df.drop_duplicates(subset=identity_cols, keep='first')
audit['duplicates_removed'] = n_before - len(df)

# Near-duplicate detection (fuzzy):
from difflib import SequenceMatcher
# Use for text fields where exact match is too strict
```

### 3. Type Normalization

```python
# Date/time normalization
df['created_at'] = pd.to_datetime(df['created_at'], utc=True, errors='coerce')

# String normalization
df['category'] = df['category'].str.strip().str.lower().str.replace(r'\s+', '_', regex=True)

# Numeric coercion with audit
original = df['price'].copy()
df['price'] = pd.to_numeric(df['price'], errors='coerce')
audit['price_coerced_nulls'] = df['price'].isnull().sum() - original.isnull().sum()
```

### 4. Outlier Handling

```python
# Domain-based capping, not arbitrary percentiles
# Example: age cannot be < 0 or > 120
df.loc[df['age'] < 0, 'age'] = np.nan
df.loc[df['age'] > 120, 'age'] = 120  # cap, don't drop

# For statistical outliers — use IQR with domain validation:
Q1 = df['value'].quantile(0.25)
Q3 = df['value'].quantile(0.75)
IQR = Q3 - Q1
lower = Q1 - 3.0 * IQR  # wider fence for less aggressive removal
upper = Q3 + 3.0 * IQR
outliers = (df['value'] < lower) | (df['value'] > upper)
audit['outliers_flagged'] = outliers.sum()
df['outlier_flag'] = outliers.astype(int)
```

### 5. Categorical Encoding

```python
# One-hot for < 20 categories, label encoding otherwise
n_unique = df['category'].nunique()
if n_unique <= 20:
    df = pd.get_dummies(df, columns=['category'], drop_first=True)
else:
    df['category_code'] = df['category'].astype('category').cat.codes
```

## Audit Trail

Always produce a structured audit log:

```python

From mkurman/zorai, MIT.

More from mkurman/zorai