“Data pipelines”

Data pipeline patterns: ingestion, ETL, serialisation, partitioned reads/writes, file conversion, parallel batch processing, Lambda-triggered ingestion.

Primary entry point for building, managing, and orchestrating data pipelines on Google Cloud.

MANDATORY when building data processing pipelines or consuming message queues.

Create efficient data pipelines with tf.data

Designs and builds ETL/ELT data pipelines.

Design or review data pipeline correctness, lineage, quality, and replay behavior.

Unified data collection, quality management, and pipeline orchestration.

Use when running multiple data pipelines on one host: memory admission, systemd MemoryMax, OOM that only appears under coexistence, capacity ratchets…

Designs ETL/ELT data pipelines with proper extraction, transformation, and loading patterns, including orchestration, error handling, and data quality…

You are a data pipeline architecture expert specializing in scalable, reliable, and cost-effective data pipelines for batch and streaming data processing.

Organism-expressed skill from paradigm data_pipeline->asynchronous

Build data ingestion pipelines for batch and streaming data from multiple sources.

ETL/ELT patterns, batch vs streaming, idempotency, data quality framework, and pipeline orchestration

Universal data connector for CSV, Postgres, and REST APIs via DuckDB

Use when the user wants a production-grade scraping/API/browser pipeline design or implementation plan: pipeline.yaml, schemas, raw/staged/normalized outputs…

Trigger when the user wants to collect, structure, evaluate, crawl, extract, refresh, or build reusable data acquisition pipelines from websites, APIs…

Run a large, multi-stage data-analysis or research-engineering task as a structured plan-review-implement-verify-reflect loop instead of one long unstructured…

Data pipeline specialist for ETL design, data quality, CDC patterns, and batch/stream processingUse when "data pipeline, etl, cdc, data quality, batch…

Data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure.

Design and implement repeatable preprocessing pipelines for cleaning, encoding, transforming, and validating ML input data.

Specialist subagent: ETL pipelines, data warehousing, stream processing, and data infrastructure specialist.

Run all three data pipelines (crypto cycle, 13F filings, congress trading) and report freshness.

World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, real-time streaming, and data infrastructure.

Customer.io Data Pipelines.