dataset-quality-audit

Files (3)

• LICENSE
• SKILL.md
• data_quality_checker.py · scripts/

SKILL.md (first lines)

# dataset-quality-audit

A data quality auditing tool that runs 12-dimension quality checks on tabular data, producing per-dimension scores (0–100), an overall grade, and actionable fix suggestions.

## Capabilities

| Dimension | Description |
|-----------|-------------|
| Missing Values | Count and percentage of null/NaN values per column |
| Duplicate Rows | Number and percentage of fully duplicated rows |
| Type Consistency | Mixed types within a single column (e.g., numbers mixed with text) |
| Value Range / Outliers | Outlier detection using the IQR method |
| Format Compliance | Consistency of date, email, phone number, and other formatted fields |
| Uniqueness Constraints | Whether ID-type columns contain duplicates |
| Whitespace Issues | Leading/trailing spaces, empty strings, whitespace-only values |
| Constant Columns | Columns with only a single unique value (zero information) |
| Distribution Skewness | Whether numeric columns have excessive skewness |
| Column Naming | Spaces, special characters, or inconsistent casing in column names |
| Cardinality Anomalies | Unusually high or low number of unique values |
| Cross-Column Consistency | Logical checks across columns (e.g., start date before end date) |

## Quick Start

```bash
# Basic quality check
python3 scripts/data_quality_checker.py data.csv

# Save report as JSON
python3 scripts/data_quality_checker.py data.csv --output report.json

# Specify ID columns (for uniqueness checks)
python3 scripts/data_quality_checker.py users.csv --id-columns "user_id,email"

# Specify date columns (for format checks)
python3 scripts/data_quality_checker.py orders.csv --date-columns "created_at,updated_at"
```

## Detailed Usage

### Basic Invocation

```bash
python3 scripts/data_quality_checker.py <data-file> [options]
```

### Parameters

| Parameter | Short | Required | Default | Description |
|-----------|-------|----------|---------|-------------|
| `input` | — | Yes | — | Path to input file (CSV/TSV/Excel/JSON) |
| `--output` | `-o` | No | stdout | Path for the JSON report output |
| `--id-columns` | `-id` | No | Auto-detect | Comma-separated column names that should be unique |
| `--date-columns` | `-dc` | No | Auto-detect | Comma-separated column names containing dates |
| `--sample` | `-s` | No | All rows | Number of rows to sample (useful for large files) |
| `--encoding` | `-e` | No | utf-8 | File encoding |

## Output Format (JSON)

```json
{
  "file": "data.csv",
  "rows": 10000,
  "columns": 15,
  "overall_score": 78.5,
  "grade": "B",
  "dimensions": {
    "missing_values": {
      "score": 85.0,
      "issues": [
        {"column": "age", "missing_count": 150, "missing_pct": 1.5, "suggestion": "Fill with median or mode"}
      ]
    },
    "duplicates": {
      "score": 95.0,
      "issues": [...]
    }
  },
  "top_suggestions": [
    "Column 'age' has 1.5% missing values — consider filling with the median",
    "Found 200 fully duplicated rows — consider deduplication"
  ]
}
```

## Grading Scale

| Grade | Score Range | Meaning |
|-------|------------|---------|
| A+ | 95–100 | Excellent quality — ready for use as-is |
| A | 90–95 | Good quality — minor issues only |
| B | 80–90 | Moderate quality — recommended to fix before use |
| C | 60–80 | Poor quality — significant cleaning required |
| D | 40–60 | Very poor quality — many issues need attention |
| F | 0–40 | Essentially unusable — requires re-collection or major cleanup |

## Dependencies

- Python 3.8+
- pandas
- numpy

```bash
pip install pandas numpy
```

From zebbern/claude-code-guide, MIT.

More from zebbern/claude-code-guide