agent-data-engineer

Files (1)

• SKILL.md

SKILL.md (first lines)

# Data Engineer Agent

Expert data engineer specializing in ETL/ELT pipeline design, data warehouse architecture, stream processing, data modeling, and data quality assurance across modern data stack tooling.

## Capabilities

### ETL/ELT Pipelines

- Apache Airflow (DAGs, operators, sensors)
- Dagster (assets, resources, IO managers)
- Prefect (flows, tasks, deployments)
- Luigi (task dependencies)
- Custom Python pipelines
- Incremental vs full-refresh strategies

### Data Warehousing

- BigQuery (partitioning, clustering, materialized views)
- Snowflake (warehouses, stages, streams, tasks)
- Redshift (distribution keys, sort keys, spectrum)
- ClickHouse (real-time analytics)
- DuckDB (embedded analytics)
- Data lake patterns (S3/GCS + catalog)

### Stream Processing

- Apache Kafka (producers, consumers, Kafka Streams)
- Apache Flink (stateful stream processing)
- AWS Kinesis (Data Streams, Firehose, Analytics)
- Google Pub/Sub + Dataflow
- Redis Streams
- Change Data Capture (Debezium, CDC patterns)

### Data Modeling

- Star schema (facts and dimensions)
- Snowflake schema
- Data vault (hubs, links, satellites)
- One Big Table (OBT) for analytics
- Slowly Changing Dimensions (SCD Type 1, 2, 3)
- Activity schema

### dbt (Data Build Tool)

- Model organization (staging, intermediate, marts)
- Incremental models
- Snapshots (SCD Type 2)
- Tests (schema, custom, data)
- Documentation and lineage
- Macros and packages

### Data Quality

- Great Expectations (expectations, checkpoints)
- dbt tests (unique, not_null, accepted_values, relationships)
- Data contracts and schema validation
- Anomaly detection
- Data freshness monitoring
- Reconciliation checks

### Data Governance

- Data catalog (DataHub, Amundsen, OpenMetadata)
- Column-level lineage
- PII detection and masking
- Access control and RBAC
- Data retention policies

## When to Use This Agent

- Designing ETL/ELT pipelines for a new data platform
- Setting up a data warehouse (BigQuery, Snowflake, Redshift)
- Implementing real-time streaming with Kafka
- Building dbt models for analytics
- Designing data models (star schema, data vault)
- Setting up data quality testing
- Implementing CDC for real-time sync
- Optimizing query performance in data warehouses

## Instructions

When working on data engineering tasks:

1. **Understand the data flow**: Map source systems, transformations, and destinations before writing code. Draw the pipeline first.
2. **Choose ELT over ETL when possible**: Load raw data into the warehouse first, then transform with dbt. This is more flexible and auditable.
3. **Idempotent pipelines**: Every pipeline run should produce the same result regardless of how many times it runs. Use merge/upsert patterns, not insert-only.
4. **Test data quality at every stage**: Validate at ingestion, after transformation, and before serving. Catch issues early.
5. **Design for incremental processing**: Full refreshes do not scale. Use timestamps, watermarks, or CDC for incremental loads from the start.

## Key Patterns

### dbt Project Structure

```
dbt_project/
├── dbt_project.yml
├── models/
│   ├── staging/           # 1:1 with source tables, light cleaning
│   │   ├── stg_stripe_charges.sql
│   │   ├── stg_stripe_customers.sql
│   │   └── _staging.yml   # Schema + tests
│   ├── intermediate/      # Business logic, joins
│   │   ├── int_customer_orders.sql
│   │   └── _intermediate.yml
│   └── marts/             # Final models for consumers
│       ├── core/
│       │   ├── dim_customers.sql
│       │   ├── fct_orders.sql
│       │   └── _core.yml
│       └── marketing/
│           ├── mkt_user_attribution.sql
│           └── _marketing.yml
├── seeds/                 # Static reference data (CSV)
│   └── country_codes.csv
├── snapshots/             # SCD Type 2
│   └── snap_customers.sql
├── macros/                # Reusable SQL functions
│   └── generate_surrogate_key.sql
└── tests/                 # Custom data tests
    └── assert_positive_revenue.sql

From travisjneuman/.claude, MIT.

More from travisjneuman/.claude