sync-guide

Files (7)

• SKILL.md
• api-pagination-patterns.md
• incremental-basic.ts · examples/
• incremental-bimodal.ts · examples/
• incremental-events.ts · examples/
• replace-paginated.ts · examples/
• replace-simple.ts · examples/

SKILL.md (first lines)

## What is a Sync?

A sync is a recurring `execute` function that returns data changes to populate a Notion database. The runtime calls `execute` in a loop:

```ts
const db = worker.database("myDb", {
  type: "managed",
  initialTitle: "My Data",
  primaryKeyProperty: "ID",
  schema: {
    properties: {
      Name: Schema.title(),
      ID: Schema.richText(),
    },
  },
})

worker.sync("mySync", {
  database: db,
  execute: async (state, { notion }) => ({
    changes: [
      {
        type: "upsert",
        key: "1",
        properties: {
          Name: Builder.title("Item 1"),
          ID: Builder.richText("1"),
        },
      },
    ],
    hasMore: false,
    nextState: undefined,
  }),
})
```

Each call returns `{ changes, hasMore, nextState }`. If `hasMore` is `true`, the runtime calls `execute` again with `nextState`. This continues until `hasMore` is `false`, completing a **cycle**. The next cycle begins at the scheduled interval with the state from the end of the previous cycle.

**Imports:**

```ts
import { Worker } from "@notionhq/workers"
import * as Builder from "@notionhq/workers/builder"
import * as Schema from "@notionhq/workers/schema"
```

## Decision Framework

### Step 1: Choose an Architecture

The deciding factor is **API capability and dataset size**. Two tiers:

| Condition                                                         | Architecture                                                       |
| ----------------------------------------------------------------- | ------------------------------------------------------------------ |
| Small source (<1k records) or API with no change tracking         | **Simple replace sync** — one sync, `mode: "replace"`              |
| Everything else (API supports `updated_at`, change feeds, events) | **Backfill + delta pair** — two syncs writing to the same database |

**Simple replace sync**: One sync returns the full dataset each cycle. After the final `hasMore: false`, any records not seen are deleted automatically. Use when the dataset is small enough to re-fetch entirely.

**Backfill + delta pair**: Two syncs share a single database. The **backfill sync** (`mode: "replace"`, `schedule: "manual"`) re-fetches everything when triggered. The **delta sync** (`mode: "incremental"`, frequent schedule) fetches only changes since the last run. This separates concerns cleanly — no bi-modal state machine, no backfill-to-delta transition bugs.

Initialize the delta cursor before starting the backfill. When both syncs can
write the same record, include `upstreamUpdatedAt` on each upsert so a stale
backfill response cannot overwrite a newer delta response.

### Step 2: Understand Your API's Pagination

Most APIs require paginating through results. Return batches of ~100 changes. Returning too many changes in one `execute` call will fail.

**Backfill pagination** (full dataset load):

1. **Opaque cursor token** — GraphQL `endCursor`, Stripe `starting_after`
2. **Page number / offset** — `?page=N&limit=100`
3. **Keyset (timestamp + id)** — `WHERE created_at > X OR (created_at = X AND id > Y)` — the gold standard for timestamp-sorted mutable data

**Delta pagination** (change-only loads, incremental mode):

1. **Timestamp cursor** — `?updated_since=<cursor>` with consistency buffer
2. **Keyset on updated_at + id** — same keyset pattern on the modification timestamp
3. **Event/changelog feed** — `GET /events?after=<eventId>`
4. **Same opaque cursor** — when the API sorts by `updated_at`, the backfill cursor works for delta too

### Step 3: Consistency Buffer (Delta Syncs)

APIs tend to be eventually consistent. A record that was just written or updated may not appear in query results immediately. Since the cursor never resets in incremental mode, if it advances past a record that hasn't been indexed yet, that record is skipped permanently. Lag the cursor 10-60 seconds behind "now":

```ts
const bufferMs = 15_000
const maxCursor = new Date(Date.now() - bufferMs).toISOString()
function minTimestamp(a: string, b: string): string {
  return Date.parse(a) <= Date.parse(b) ? a : b
}

const nextCursor =
  records.length > 0 ? minTimestamp(lastRecord.updatedAt, maxCursor) : maxCursor
```

### Step 4: Deletion Strategies

1. **Backfill sync (replace mode)**: free — unseen records are auto-deleted each cycle. This is the primary mechanism for handling deletes when the API has no delete signal.
2. **Delta sync with delete API**: emit `{ type: "delete", key }` markers. If the delete signal comes from a separate endpoint (audit log, archived filter), use the **flip-flop pattern**: run the main delta stream until caught up (`hasMore: false`), then switch to the delete stream for a cycle, then back. Both cursors persist in state independently.
3. **No delete API, large dataset**: rely on the backfill sync's replace-mode mark-and-sweep. Trigger the backfill manually or on a slow schedule to clean up stale records.

## Replace Mode

Simple: fetch everything, return it all, let the runtime handle deletes. Use as a standalone sync for small sources, or as the backfill half of a backfill+delta pair.

```ts
const db = worker.database("records", {
  type: "managed",
  initialTitle: "Records",
  primaryKeyProperty: "ID",
  schema: {
    properties: { Name: Schema.title(), ID: Schema.richText() },
  },
})

const apiPacer = worker.pacer("myApi", {
  allowedRequests: 10,

From makenotion/notion-cookbook, MIT.

More from makenotion/notion-cookbook