embedding-lifecycle

Files (2)

• SKILL.md
• model_registry_and_cutover.md · references/

SKILL.md (first lines)

# Embedding Lifecycle

> Vector search skills tell you how to *query* embeddings. This one is about everything that
> happens to them over time: they were produced by a specific model, at a specific dimension,
> after specific preprocessing — and the day any of those three changes, the column becomes a
> mixture of incompatible spaces that no database will ever complain about.

## Scope

- Changing embedding model, provider, or version on a corpus that already has vectors.
- Changing dimension — including "just truncating" an existing vector.
- Adding or removing a prefix, instruction, or template in the text sent to the embedder.
- Backfilling embeddings into a live table while queries are being served.
- Deciding whether a partially-populated embedding column may be trusted yet.
- Planning the cutover, and deciding what evidence justifies the swap.
- Auditing why retrieval quality silently degraded after an "unrelated" change.

## Do NOT use for

- Choosing which embedding model to adopt, or reading model benchmarks -> model provider docs and
  independent retrieval benchmarks.
- Chunking, splitting, or document-segmentation strategy -> `rag-systems`.
- ANN index mechanics and tuning (HNSW `m`/`ef_construction`/`ef_search`, IVFFlat `lists`/`probes`,
  hybrid RRF, quantization) -> `postgres-extensions-and-vector-search`.
- Retrieval pipeline architecture, rerankers, query routing -> `rag-systems`.
- General schema migration risk and DDL locking -> `postgres-schema-migrations`.
- Deciding whether identity or biometric embeddings may be *persisted at all* — retention, consent,
  erasure obligations, and the cost-versus-privacy trade of recomputing a vector per request instead
  of storing it -> `privacy-data-protection`. This skill governs a vector column that exists; that
  one governs whether it is allowed to exist.

## First principle: the three facts that make a vector meaningful

An embedding is only comparable to another embedding produced by **the same model, at the same
dimension, from the same preprocessing**. Those three facts are properties of the *column*, not of
the pipeline that happened to run last.

Nothing enforces this. A `vector(1536)` column accepts vectors from any model that outputs 1536
dimensions, and the distance operator will happily compute a number for two vectors from different
spaces. The number is meaningless, but it is a *plausible* number — it sorts, it ranks, it fills a
results page. **A mixed-model column is corrupted data with no error message and no exception in
any log.**

So pin all three where they cannot drift:

```yaml
# One authoritative declaration. Config, schema comment, or migration -- pick one and
# make everything else read from it.
embedding:
  model_id: "<exact provider model identifier>"
  dimensions: 1536
  preprocessing: "title + '\n\n' + body, normalized whitespace, no prefix"
  normalized: true         # unit-normalized at write time?
  distance: cosine
```

And enforce it at the boundary: the writer asserts that the vector it is about to store came from
the declared model/dimension/preprocessing, and refuses otherwise. An assertion here costs
microseconds and is the only thing standing between you and a silent corpus-wide corruption.

## Pillar 1 — dimension transitions and shadow columns

### Never write two dimensions into one column

During a transition, run **shadow columns per dimension**:

```sql
ALTER TABLE docs ADD COLUMN emb_3072 vector(3072);   -- new space, unindexed at first
-- existing: emb_1536 vector(1536), indexed, serving traffic
```

The old column keeps serving while the new one fills. Readers explicitly name the column they
trust; the schema makes the mistake unrepresentable rather than merely discouraged. Drop the old
column only after the cutover gate passes (below) — and after a retention window long enough to
roll back without re-embedding everything again.

### Truncation: native versus post-hoc

Lower dimensions cost less to store and search, so the temptation to shorten vectors is constant.
Whether that is legitimate depends on how the model was trained.

**Matryoshka Representation Learning** (Kusupati et al., arXiv:2205.13147, 2022) trains a single
model so that its prefixes are themselves usable embeddings — the abstract states MRL "learns
coarse-to-fine representations that are at least as accurate and rich as independently trained
low-dimensional representations." When a model is trained this way, a shorter vector is a
*supported output*, not a lossy hack.

Current embedding APIs expose this directly (both checked at the vendor's own docs, 2026-07-29):

| Provider | Parameter | Documented note |
|---|---|---|
| OpenAI embeddings (`developers.openai.com/api/docs/guides/embeddings`) | `dimensions` | Using the parameter is "the suggested approach"; if you slice manually instead, "you need to be sure to normalize the dimensions of the embedding". |
| Gemini embeddings (`ai.google.dev/gemini-api/docs/embeddings`) | `output_dimensionality` | Default output 3072. `gemini-embedding-2` "introduces automatic renormalization for non-default dimensions"; with `gemini-embedding-001` "you must manually normalize non-3072 dimensions". |

Two rules follow:

1. **Ask the API for the dimension you want** rather than slicing the returned array. The
   parameter is the supported path and it handles normalization where the model requires it.
2. **If you must slice manually, re-normalize.** Truncating an L2-normalized vector produces a
   vector that is no longer unit length, and cosine/inner-product distances computed against it are
   quietly wrong. This is the single most common truncation bug.

For a model **not** trained with prefix-usable representations, truncation is not a dimension
choice — it is discarding information from an arbitrary basis, and it must be validated by measured
recall or not done. Verify the property against the model's own documentation before assuming it.

> Because truncated output from an MRL-style model is a first-class output, a lower dimension can
> be a deliberate *design* choice — cheaper storage, faster search, smaller index — rather than a
> workaround for a dimension cap. Decide it on measured recall and cost, not on what fits.

## Pillar 2 — re-embedding is a blue-green migration

Re-embedding a populated corpus is not a backfill job. It is a migration of the substrate every
read depends on, running while those reads continue. Treat it with migration discipline:

1. **Stage without an index.** Write new vectors into the staging column with no ANN index —
   maintaining an index during a bulk write is far more expensive than building it once at the end,
   and the half-built index is not usable anyway.
2. **Drain the backlog** with a worker that is idempotent, resumable, and rate-limited to the

From rafaelob/aiLine, Apache-2.0.

More from rafaelob/aiLine