ingestion-api

Files (1)

• SKILL.md

SKILL.md (first lines)

# Ingestion API

Design guidance for the boundary that accepts data **into** a system from external producers — webhooks, telemetry/event beacons, bulk file/CSV uploads, IoT/device streams, partner data feeds, log/metrics collectors.

## When to Use

Build a dedicated ingestion API when:
- An external or semi-trusted producer (partner system, device fleet, webhook sender, client SDK) pushes data **in**, as opposed to a client pulling data out
- Volume or burstiness makes synchronous processing risky (spiky webhook fan-in, batch file drops, high-frequency telemetry)
- The producer cannot be forced to slow down, retry correctly, or wait — the boundary must absorb load on the producer's terms, not the consumer's
- Data must be durably accepted before it is fully processed (accept now, validate/enrich/store asynchronously)
- Multiple producers with different trust levels, schemas, or versions feed the same downstream pipeline

## When NOT to Use

Do not stand up a distinct ingestion layer when:
- The write is a normal user-initiated CRUD operation from your own frontend — that's `skills/architecture/api-design/SKILL.md`, not ingestion
- Volume is low and synchronous validate-then-persist is fast enough (~200ms) and the caller is trusted — plain REST/GraphQL endpoint is sufficient
- The data originates internally between your own services with a shared schema and controlled cadence — use `skills/architecture/event-driven/SKILL.md` (pub/sub, sagas) instead
- You only need to *pull* data from a third party on a schedule — that's an outbound integration, not ingestion

Ingestion is defined by the boundary property (uncontrolled external push, must-not-drop) — not by data volume alone.

---

## Detection Signals

- Endpoint names/paths containing `ingest`, `collect`, `events`, `webhook`, `telemetry`, `beacon`, `bulk-upload`
- A queue or buffer sitting directly behind a public HTTP endpoint (SQS/Pub/Sub/Kafka topic fed by an API Gateway route)
- Webhook receivers from third parties (Stripe, GitHub, Twilio, payment/CRM providers)
- Bulk/batch file ingestion (CSV/JSON/Parquet drop, S3 event-triggered processing)
- High-cardinality client-side event/analytics collection endpoints

---

## Core Design Rules

### 1. Accept Fast, Process Later
The HTTP response confirms **receipt**, not **completion**. Validate the envelope (auth, shape, size) synchronously; hand the payload to a queue or durable store and return `202 Accepted` (or `200` with a receipt id). Full processing — enrichment, business validation, side effects — happens as a background job. See `skills/architecture/async-jobs/SKILL.md` for the job-side idempotency and retry rules once payload hands off there.

### 2. Idempotency at the Boundary
Producers retry on timeout or ambiguous response — the same event will arrive more than once.
- Require or generate a de-duplication key (`Idempotency-Key` header, or a producer-supplied event id) and reject/short-circuit duplicates within a bounded window
- For webhooks, verify the provider's signature **and** dedupe on their event id — signature verification alone does not stop replays
- Never let "accept twice, process twice" happen silently — a passed idempotency check must be visible in logs/metrics, not just swallowed

### 3. Validate the Envelope, Defer the Semantics
Two validation passes, not one:
- **Synchronous, at the edge:** authentication/signature, content-type, size limits, required envelope fields (id, timestamp, schema version). Reject immediately — the producer can act on the error now.
- **Asynchronous, in the worker:** business-rule validation, cross-record checks, enrichment. Route failures to a dead-letter queue (DLQ) — do not block the accept path on rules the producer cannot fix in real time.

### 4. Backpressure Is a Feature, Not a Bug
The ingestion boundary must be able to say "slow down" without dropping data:
- Prefer `429` with `Retry-After` over silently discarding a request when downstream is saturated
- Size the buffer (queue depth) explicitly; alert before it fills, not after
- If the producer is yours (your own SDK/agent), implement client-side backpressure/batching instead of hammering the endpoint — see `skills/architecture/rate-limiting/SKILL.md` for algorithms and headers
- Never let an unbounded queue become an unbounded memory/disk leak — cap it and define the overflow behavior (reject new, or shed oldest, per data criticality)

### 5. Schema Evolution and Versioning
Producers you don't control ship on their own schedule.
- Version the payload envelope explicitly (`schema_version` field or versioned path/media type) — never infer version from field presence
- New optional fields must not break existing producers; removing or repurposing a field is a breaking change and needs a migration window
- Reject unknown-but-critical fields loudly in a sandbox/staging producer, but **accept and store unknown fields as opaque data** in production rather than dropping the whole event — a strict schema at the boundary turns a producer's minor addition into full data loss
- Cross-reference `skills/architecture/api-versioning/SKILL.md` for the general versioning contract this specializes

### 6. Batch vs. Streaming Ingestion
| Shape | Use when | Watch for |
|---|---|---|
| Single-event HTTP push (webhook) | Low-to-moderate frequency, one producer event per request | Signature verification, per-event idempotency key |
| Bulk file/batch upload | Producer emits periodic large batches (CSV export, nightly feed) | Partial-failure reporting — never all-or-nothing on a 100k-row file; report per-row/per-chunk status |
| Streaming (Kafka/Kinesis/WebSocket ingest) | Continuous high-frequency telemetry, ordering matters within a partition key | Consumer lag as the primary health signal, not endpoint latency |

### 7. Authentication and Trust Tiering
- Machine-to-machine producers authenticate with API keys, mTLS, or provider-specific signatures (never end-user session cookies)
- Trust-tier producers explicitly (first-party service, verified partner, unauthenticated public collector) and apply validation/rate limits proportional to trust — do not apply one policy to all
- Rotate and scope ingestion credentials per producer so one compromised partner key doesn't grant write access to the whole pipeline

### 8. Observability
Minimum signals for an ingestion boundary:
- Accept rate, reject rate (by reason: auth, validation, rate-limit), and DLQ growth rate
- End-to-end lag: time from accept to fully-processed, not just endpoint response time
- Duplicate/idempotency-hit rate — a sudden spike usually means a producer-side retry storm, not a boundary bug

---

## Related Skills

| Concern | Skill |
|---|---|
| What happens to the payload after it's queued | `skills/architecture/async-jobs/SKILL.md` |
| Producer-facing throttling algorithms and headers | `skills/architecture/rate-limiting/SKILL.md` |
| General REST/GraphQL contract rules | `skills/architecture/api-design/SKILL.md` |
| Internal service-to-service messaging, sagas | `skills/architecture/event-driven/SKILL.md` |
| Payload envelope/version contract | `skills/architecture/api-versioning/SKILL.md` |
| Large file transfer into the system | `skills/architecture/multipart-upload/SKILL.md` |
| Retries/circuit breakers on downstream calls made during processing | `skills/architecture/resilience/SKILL.md` |
| Ingesting documents into a RAG corpus specifically | `skills/architecture/llm-integration/SKILL.md` § Ingestion and Chunking |

From Dev-Toolbelt/dev-team-agents, MIT.

More from Dev-Toolbelt/dev-team-agents