senior-data-engineer

Files (14)

• HOW_TO_USE.md
• SKILL.md
• .gitkeep · assets/
• data-quality-checklist.md · assets/
• pipeline-design-template.md · assets/
• frameworks.md · references/
• templates.md · references/
• tools.md · references/
• data_quality_validator.py · scripts/
• etl_performance_optimizer.py · scripts/
• kafka_config_generator.py · scripts/
• pipeline_orchestrator.py · scripts/
• stream_processor.py · scripts/
• streaming_quality_validator.py · scripts/

SKILL.md (first lines)

# Senior Data Engineer

## Core Capabilities

- **Batch Pipeline Orchestration** - Design and implement production-ready ETL/ELT pipelines with Airflow, intelligent dependency resolution, retry logic, and comprehensive monitoring
- **Real-Time Streaming** - Build event-driven streaming pipelines with Kafka, Flink, Kinesis, and Spark Streaming with exactly-once semantics and sub-second latency
- **Data Quality Management** - Comprehensive batch and streaming data quality validation covering completeness, accuracy, consistency, timeliness, and validity
- **Streaming Quality Monitoring** - Track consumer lag, data freshness, schema drift, throughput, and dead letter queue rates for streaming pipelines
- **Performance Optimization** - Analyze and optimize pipeline performance with query optimization, Spark tuning, and cost analysis recommendations


## Key Workflows

### Workflow 1: Build ETL Pipeline

**Time:** 2-4 hours

**Steps:**
1. Design pipeline architecture using Lambda, Kappa, or Medallion pattern
2. Configure YAML pipeline definition with sources, transformations, targets
3. Generate Airflow DAG with `pipeline_orchestrator.py`
4. Define data quality validation rules
5. Deploy and configure monitoring/alerting

**Expected Output:** Production-ready ETL pipeline with 99%+ success rate, automated quality checks, and comprehensive monitoring

### Workflow 2: Build Real-Time Streaming Pipeline

**Time:** 3-5 days

**Steps:**
1. Select streaming architecture (Kappa vs Lambda) based on requirements
2. Configure streaming pipeline YAML (sources, processing, sinks, quality)
3. Generate Kafka configurations with `kafka_config_generator.py`
4. Generate Flink/Spark job scaffolding with `stream_processor.py`
5. Deploy and monitor with `streaming_quality_validator.py`

**Expected Output:** Streaming pipeline processing 10K+ events/sec with P99 latency < 1s, exactly-once delivery, and real-time quality monitoring


World-class data engineering for production-grade data systems, scalable pipelines, and enterprise data platforms.

## Overview

This skill provides comprehensive expertise in data engineering fundamentals through advanced production patterns. From designing medallion architectures to implementing real-time streaming pipelines, it covers the full spectrum of modern data engineering including ETL/ELT design, data quality frameworks, pipeline orchestration, and DataOps practices.

**What This Skill Provides:**
- Production-ready pipeline templates (Airflow, Spark, dbt)
- Comprehensive data quality validation framework
- Performance optimization and cost analysis tools
- Data architecture patterns (Lambda, Kappa, Medallion)
- Complete DataOps CI/CD workflows

**Best For:**
- Building scalable data pipelines for enterprise systems
- Implementing data quality and governance frameworks
- Optimizing ETL performance and cloud costs
- Designing modern data architectures (lake, warehouse, lakehouse)
- Production ML/AI data infrastructure

## Quick Start

### Pipeline Orchestration

```bash
# Generate Airflow DAG from configuration
python scripts/pipeline_orchestrator.py --config pipeline_config.yaml --output dags/

# Validate pipeline configuration
python scripts/pipeline_orchestrator.py --config pipeline_config.yaml --validate

# Use incremental load template
python scripts/pipeline_orchestrator.py --template incremental --output dags/
```

### Data Quality Validation

```bash
# Validate CSV file with quality checks
python scripts/data_quality_validator.py --input data/sales.csv --output report.html

# Validate database table with custom rules
python scripts/data_quality_validator.py \
    --connection postgresql://user:pass@host/db \
    --table sales_transactions \
    --rules rules/sales_validation.yaml \
    --threshold 0.95
```

### Performance Optimization

```bash
# Analyze pipeline performance and get recommendations
python scripts/etl_performance_optimizer.py \
    --airflow-db postgresql://host/airflow \
    --dag-id sales_etl_pipeline \
    --days 30 \
    --optimize

# Analyze Spark job performance
python scripts/etl_performance_optimizer.py \
    --spark-history-server http://spark-history:18080 \
    --app-id app-20250115-001
```

### Real-Time Streaming

```bash
# Validate streaming pipeline configuration
python scripts/stream_processor.py --config streaming_config.yaml --validate

# Generate Kafka topic and client configurations
python scripts/kafka_config_generator.py \
    --topic user-events \
    --partitions 12 \
    --replication 3 \
    --output kafka/topics/

# Generate exactly-once producer configuration
python scripts/kafka_config_generator.py \

From benchflow-ai/skillsbench, MIT.

More from benchflow-ai/skillsbench