clickhouse-js-node-rowbinary

Files (193)

• .gitignore
• AGENTS.md
• CHANGELOG.md
• EXAMPLES.md
• README.md
• SKILL.md
• iot-rowbinary-vs-json.md · case-studies/
• ledger-rowbinary-vs-json.md · case-studies/
• logs-json-wins.md · case-studies/
• wasm-vs-js.md · case-studies/
• eval_result.md
• eval_result_composer.md
• eval_result_haiku.md
• eval_result_sonnet.md
• package-lock.json
• package.json
• reader.md
• carts.ts · src/examples/
• events.ts · src/examples/
• iot.ts · src/examples/
• ledger.ts · src/examples/
• logs.ts · src/examples/
• observability.ts · src/examples/
• orders.ts · src/examples/
• profiles.ts · src/examples/
• telemetry.ts · src/examples/
• aggregateFunction.ts · src/readers/
• bool.ts · src/readers/
• columnar.ts · src/readers/
• compile.ts · src/readers/
• composite.ts · src/readers/
• core.ts · src/readers/
• datetime.ts · src/readers/
• decimals.ts · src/readers/
• dynamic.ts · src/readers/
• enums.ts · src/readers/
• floats.ts · src/readers/
• geo.ts · src/readers/
• header.ts · src/readers/
• integers.ts · src/readers/
• interval.ts · src/readers/
• ip.ts · src/readers/
• json.ts · src/readers/
• lowCardinality.ts · src/readers/
• nested.ts · src/readers/
• nothing.ts · src/readers/
• reader.ts · src/readers/
• rowBinaryWithNamesAndTypes.ts · src/readers/
• rows.ts · src/readers/
• simpleAggregateFunction.ts · src/readers/
• stream.ts · src/readers/
• strings.ts · src/readers/
• time.ts · src/readers/
• uuid.ts · src/readers/
• varint.ts · src/readers/
• aggregateFunction.ts · src/writers/
• bool.ts · src/writers/
• composite.ts · src/writers/
• core.ts · src/writers/
• datetime.ts · src/writers/
• …and 133 more

SKILL.md (first lines)

# ClickHouse JS RowBinary Codec Generator for Node.js

This skill generates both directions of the wire format: **readers** (decode
bytes → values) and **writers** (encode values → bytes, the mirror). A given
task normally needs only one side. This file is the shared entry point — the
format gate plus the principles common to both directions; the per-direction
decisions, guidance, and the per-type reference tables live in two sibling files.

**Pick your side — read only the one you need:**

- **Decoding a `RowBinary*` response** from ClickHouse into JS values →
  **[reader.md](reader.md)**. Streaming vs whole-buffer, row-objects vs columnar,
  fixed vs runtime schema, and the per-type reader reference.
- **Encoding JS values into a `RowBinary` payload** to send to ClickHouse →
  **[writer.md](writer.md)**. The `Sink`/`writeX` building blocks, `writeRows`
  streaming, and the per-type writer reference.

The per-type code is real, split by direction under `src/readers/` and
`src/writers/`.

## First: is RowBinary even the right format?

RowBinary exists for throughput, but it is **not automatically the fastest
path** — match the format to the shape of the data before committing to a
bespoke parser.

**Prefer a `JSON*` format (e.g. `JSONEachRow`) when** the result is mostly
strings / JSON-like values that you consume wholesale — randomly accessing
essentially every field, running string/regexp methods on them, treating values
as text. V8's native `JSON.parse` is heavily optimized C++ and builds JS strings
and objects faster than a JS-level RowBinary decoder can; pair it with HTTP
response compression (`gzip` / `zstd`, which crushes JSON's repetitive keys) and
the wire cost shrinks too.

**RowBinary clearly wins when** the result is dominated by:

- **Wide numerics** — `Int128`/`Int256`/`UInt128`/`UInt256`,
  `Decimal128`/`Decimal256`.
- **Binary / fixed-width blobs** — `IPv4`, `IPv6`, `UUID`, `FixedString`.
- **High-volume fixed-width numeric columns** generally, where each value is a
  single `DataView` read.

**Prefer the `Native` format when** columnar load and client-side analytics are
the main goal (fold/scan/filter columns, feed typed arrays to a Worker or WASM).
`Native` is column-major, so it loads straight into one typed array per column
with no transpose.

For help choosing and consuming a `JSON*` format (or CSV / TSV) instead, use the
**`clickhouse-js-node-coding`** skill.

## Core guidance (both directions)

These principles apply whether you are generating a reader or a writer; the
side-specific operational guidance is in [reader.md](reader.md) /
[writer.md](writer.md).

- **Little-endian only.** RowBinary is little-endian; target x86/ARM. Read and
  write every multi-byte number with `DataView` accessors passing a **literal**
  `true` for the `littleEndian` flag.

- **Correct first, then optimize.** First emit a correct codec built from the
  plain per-type API. Only after it's correct (and tested) specialize it. Don't
  bake performance assumptions in before correctness.

- **Monomorphize generic/composite types.** Emit specialized, inlined code per
  type combination instead of passing functions as arguments where the type is
  known ahead of time.

- **Inline the leaf ops.** The per-type `readX`/`writeX` functions are the
  correct, composable reference; the generated codec should INLINE their bodies,
  not call them, so the row loop is straight-line with no per-field indirection
  (and so the fixed-width coalescing can fold the offset arithmetic together).

- **Annotate the type per column.** Inlining erases the type structure, so put a
  short comment above each column's encode/decode block naming the ClickHouse
  type it handles.

- **Shared scratch is not reentrant.** Some hot methods reuse a module-level
  scratch buffer as a write-then-read pair — correct only because the access is
  fully synchronous. An `async`/`yield` boundary between populating and reading
  it corrupts the value.

- **TypeScript by default.** Generate TypeScript code and helpers unless the user
  explicitly asks for plain JavaScript.

## Worked examples

Six end-to-end examples with real speedup are catalogued in [EXAMPLES.md](EXAMPLES.md).

## Out of scope

- **JSON / CSV / TSV / Parquet parsing** → use `clickhouse-js-node-coding`.
- **Connection errors, hangs, type mismatches** → use
  `clickhouse-js-node-troubleshooting`.
- **Browser / Web Worker / Edge** → `@clickhouse/client-web`.

## Still Stuck?

- [ClickHouse RowBinary format](https://clickhouse.com/docs/interfaces/formats#rowbinary)
- [ClickHouse data types](https://clickhouse.com/docs/sql-reference/data-types)
- [ClickHouse JS client docs](https://clickhouse.com/docs/integrations/javascript)

From ClickHouse/agent-skills, Apache-2.0.

More from ClickHouse/agent-skills