portolan-migrate

Files (9)

• SKILL.md
• README.md · reference/
• conformance.md · reference/
• apply_metadata.py · reference/tools/
• build_collection.py · reference/tools/
• fix_styles.py · reference/tools/
• reencode.py · reference/tools/
• sld_graduated.py · reference/tools/
• validate_with_data.py · reference/tools/

SKILL.md (first lines)

<!-- drift: depends-on: portolan-cli, rashid, portolan-spec, portolan-catalog-template -->
<!-- Defects and counts come from two migrations run in August 2026:
     pergamino-ide-catalog (183 collections, WFS/GeoServer) and
     microsoft-ml-road-detections (one collection, 235 partitions, 12 GB). -->

# Portolan Migrate

Something already exists. A catalog built by an older toolchain, a folder of GeoParquet with a README, a Source Cooperative product that predates Portolan. It has users, download counts, and URLs other people have written down. Your job is to bring it into compliance without breaking any of that.

Use `portolan-bootstrap` when there is no catalog yet and you are converting a data source. Use this skill when the artifact exists and must survive.

## Preserve What Already Exists

The data is innocent until a named requirement proves otherwise. That rule decides most of what follows.

Do not re-download upstream, reconstruct from source, filter rows, rename descriptive fields, or change CRS, geometry, or partitioning because a rewrite would be tidier. When a rule forces a change, quantify it. Report before and after row count, file count, and size. Say which rule forced it, and say whether it is reversible:

```
files       235 -> 235
rows        256,555,010 -> 256,555,010
length      54,225,233 km -> 54,225,233 km
row groups  376 -> 2,709
max rg rows 3,593,665 -> 100,000   (PORTO-FMT-009 caps at 150,000)
bytes       11,415,256,478 -> 11,999,057,592   (+5.11%)
```

That table is the deliverable for any data change. Without it you cannot tell a repair from a corruption.

## Principles

**Audit before you plan.** The plan's value is showing how little work is required. One migration ran a fourteen-point data audit and found exactly one mandatory change.

**Falsify your green run.** A validator that has never failed on this catalog has told you nothing. Plant a known violation and confirm the validator reports it. Then trust a clean pass.

**Measure before you write.** Every derived number in the documentation comes from a query you ran over the whole dataset. Do not use a sample. Do not use an estimate.

**The old catalog is live.** Its defects are served right now, and its stale objects will survive your push. Treat the remote as something to reconcile, not something that updates itself.

**Ask the publisher.** Licence, sensitivity, and exclusions are their call, not yours. Two of the three cost real rework when guessed.

**Fix upstream faults upstream.** File the issue rather than papering over it here. A workaround in this catalog becomes a defect every other catalog inherits.

## Before You Touch Anything

### Back up before you change anything

```bash
BK=~/backups/<catalog>-premigration-<YYYYMMDD>
rsync -a \
  --exclude='.git/' --exclude='.venv/' \
  --exclude='.env*' --exclude='.claude/' \
  "$SRC/" "$BK/tree/"
diff -r -q "$SRC" "$BK/tree"
chmod -R a-w "$BK/tree"
```

**A bare `rsync -a` sweeps in credentials.** One session started copying the working tree and had to kill the run, because it was pulling in `.env.local`, `.venv`, and `.git`. Exclude explicitly, then prove the result is clean:

```bash
find "$BK" \( -name '.env*' -o -name '*credential*' \
  -o -name '*secret*' \) -print
```

That must return nothing.

**Build the manifest in Python, not in a shell pipeline.** Terminal output decoration corrupted `find` and `ls` three separate times in one session, once producing an empty manifest that verified successfully. Walk the tree in Python, write sha256 for every file, and assert the file count you expect before declaring the backup good.

Do not treat git as the backup. Data files are gitignored, so a repository is not a copy of the dataset.

### Pull the Remote in Full

A diff is not enough. Fetch the remote bytes and compare them.

One migration found all 178 `collection.json` files byte-identical between local and remote, which meant every defect was live. The same pull found something a metadata diff would have missed. The remote stored a **curated** `.portolan/metadata.yaml`. It listed a real title, a Spanish description, a contact, `CC-BY-4.0`, a citation, and thirteen thematic keywords. A local re-extraction had already overwritten it with `TODO: Add value`. The good metadata existed only on the server.

The same pull also found 106 declared style assets that were never uploaded and returned live 404s, two corrupt 28,672-byte PMTiles that were SQLite intermediates, and stray `.pmtiles-journal` files. Against 20,513 downloads in the previous 28 days.

For a large partitioned dataset, join on partition key and size rather than fetching everything, then spot-check ETags against local md5. Fall back to size alone where the ETag shows a multipart upload.

Stop and ask if the two copies differ in a way that changes which is authoritative.

## Audit

### Read the Extraction Report First

If the catalog came from `portolan extract`, `.portolan/extraction-report.json` names every layer that failed and why. One catalog's report read `188 layers, 179 succeeded, 9 failed`, in two clean classes:

```
Cannot do natural order without a primary key, please add
```

WFS pagination on a table with no primary key. Five layers, three attempts each.

```
Unable to merge: Field alt-piso has incompatible types: double vs int64
```

An Arrow schema conflict between pages. Four layers.

The report also missed two failures. One layer extracted 1,098 features whose geometries were all NULL and was dropped downstream in silence. Another failed at the tiling stage yet stayed linked in `catalog.json`, so the catalog advertised a collection without tiles, a thumbnail, or a style.

Compare the child links in `catalog.json` against the directories on disk. That catalog had 188 directories and 178 links, so ten collections existed and were unreachable.

### Count Every Defect Across Every Collection

Sampling hides the shape of the problem. Count. One catalog's audit, which is a serviceable checklist for the next one:

| Field | Found | Requirement |
|---|---|---|
| `title` | 1 of 178 | MUST |
| `providers` | 0 of 178 | MUST |
| Licence other than `proprietary` | 0 of 178 | `proprietary` MUST NOT be used (PORTO-CORE-060) |
| `rel: describedby` | 0 of 178 | MUST |
| `rel: agents` | 0 of 178 | MUST |
| Portolan schema URI | 0 of 178 | MUST (PORTO-CORE-006) |
| Style carrying the `default` role | 0 of 178, though 167 have two styles | MUST where more than one |
| `AGENTS.md` on disk | 0 | MUST |
| Populated temporal extent | 0 of 178 | |
| `documentation` asset | 56 of 178, though 178 READMEs existed on disk | |

From portolan-sdi/portolan-skills, Apache-2.0.

More from portolan-sdi/portolan-skills