portolan-bootstrap

Files (3)

• SKILL.md
• catalog-navigation.md · reference/
• probe-upstream.sh · scripts/

SKILL.md (first lines)

<!-- drift: depends-on: portolan-cli, portolan-spec, portolan-browser -->

# Portolan Bootstrap

This skill takes the data-first path. You hold the files. You convert and document them. Then you publish the whole catalog to a bucket.

If the user wants catalog metadata to live in a git repository, validated by CI and open to pull requests, use the `git-backed-catalog` skill instead.

If a catalog or a published dataset already exists and the job is to bring it into compliance, use the `portolan-migrate` skill. It audits what is there and repairs it in place.

## The Goal

A finished catalog meets three standards. Someone who has never seen the data can decide in one screen whether to trust it. An agent can write a correct query on the first try. Every collection renders a readable map the moment it opens.

Validation is the floor, not the goal. A catalog can pass every check and still tell a reader nothing. That catalog is not done.

Read an exemplar before you start. [portolan-nl](https://source.coop/cholmes/portolan-nl) is a published catalog that meets the three standards. The [portolan-reference catalog](https://github.com/portolan-sdi/portolan-spec/tree/main/examples/catalog/portolan-reference) in portolan-spec is the annotated minimum.

The [spec](https://github.com/portolan-sdi/portolan-spec) is the standard. The CLI implements it and is one way to get there, not the definition of done. Where a better tool fits a step, use it:

- [gpio](https://github.com/geoparquet/geoparquet-io) for conversion and spatial sorting
- tippecanoe directly for tiling options the CLI does not expose
- DuckDB for profiling a column before you style it

Record what you ran in the collection's `AGENTS.md`.

Read [philosophy.md](https://github.com/portolan-sdi/portolan-spec/blob/main/specs/best-practices/philosophy.md) for what the standard is buying.

## Working Agreement

**Checkpoint when uncertain.** Asking costs a minute. Guessing costs a republish.

**Preview before anything expensive.** `extract` and `push` take `--dry-run`. `check --fix --dry-run` previews conversion. `add --pmtiles` is the tiling path and has no dry run, so size the job from the extraction report before you tile. Report the numbers before you spend the time.

**Warn inline, summarize at the end.** A failed layer does not stop the run. It also does not disappear from the final report.

**Front-load research and quality control.** A rendering that lies costs more to fix than to prevent. The user finds it on a screenshot after publication.

**Fix upstream faults upstream.** When the browser, the CLI, or the spec is wrong, file the issue there. A workaround here becomes a defect every other catalog inherits.

**Reuse the publisher's vocabulary.** Their topics, tags, and department names are what the data's audience already searches for.

**Publish when you can defend it.** Every collection should answer the question "where did this sentence come from, and does that map show what the legend claims?"

## Record the source of every claim

Research widely. Then make each claim traceable. A fact belongs in the catalog when it fits one of three tiers.

**Attested.** The source states it in its service metadata, a GetCapabilities document, the portal's dataset page, or a published data dictionary. Copy the wording for license, attribution, and legal statements. Record where it came from.

**Researched and cited.** The claim comes from outside the source's own metadata. Look in the publisher's data dictionary PDF, an agency program page, a standard's code list, or the ordinance that created the program. This tier is encouraged. The condition is a link. Record the URL in the collection's `processing_notes`, and as a markdown link in the description or `AGENTS.md` where a reader benefits.

**Derived.** The number comes from the published data. Examples are a row count, a value distribution, and a measure like sale price per square foot. Put the query in `AGENTS.md` so the reader can rerun it. Never state a derived number you did not compute.

Anything else is invention, including the plausible kind. Guessing that pipe-material code 109 means ductile iron is invention. Finding the utility's code list and citing it is research. Reporting that 109 is the most common value is derivation.

When research fails, say so. "The publisher has not released a code list for `MATERIAL`. The values are opaque integers" is honest and useful. A column left undescribed is a smaller failure than a column described wrongly. Record such gaps in `known_issues` in `.portolan/metadata.yaml`.

### Provenance fields

The spec derives whether a catalog is official or a mirror from `providers`. Every collection MUST list `providers` with at least one `producer` and exactly one `host`, listed last (PORTO-CORE-046, PORTO-CORE-047). The host MUST carry a `url` or an `email` (PORTO-CORE-051). When producer and host differ, the collection is a mirror. A mirror MUST carry a `via` link of type `text/html` to the original source (PORTO-CORE-053) and a top-level `updated` field set to the time of the sync (PORTO-CORE-057).

Declare the providers and the source in `.portolan/metadata.yaml`. The CLI writes the `via` link from `source_url` and sets `updated` when the providers make the collection a mirror. `portolan extract` also writes a `via` link on each collection it creates.

```yaml
providers:
  - name: "City of Example, Assessor's Office"
    roles: ["producer", "licensor"]
    url: "https://example.gov/assessor"
  - name: "Example Data Cooperative"
    roles: ["host"]
    url: "https://example.org/contact"
source_url: "https://example.gov/data/parcels"
attribution: "City of Example, Assessor's Office"
processing_notes: >
  Extracted from the ArcGIS FeatureServer on
  2026-08-12. MATERIAL codes decoded from
  https://example.gov/water/pipe-codes.pdf.
  Price per square foot derived from SALE_PRICE
  and SHAPE_Area. See AGENTS.md for the query.
```

When the source publishes its own STAC catalog, add a `canonical` link to that STAC root (PORTO-CORE-054). `portolan check` cannot know whether such a catalog exists, so this is your research finding, not a validator result.

**The test.** Before publishing, take any sentence from a description, a README, or an `AGENTS.md` and answer two questions: which tier is this, and where is its source? Do not publish a sentence with no answer.

Title, license, and contact never come from research alone. They stay checkpointed. You may propose a license you found on the publisher's terms page. The user confirms it.

Translation is checkpointed too. Do not translate a title or description unless the source publishes both languages or the user asks. The spec models a translation as a separate STAC tree per language, linked from the root with an `alternate` link (PORTO-CORE-079). A translated tree is never a `child`. Read [multilingual-catalogs.md](https://github.com/portolan-sdi/portolan-spec/blob/main/specs/best-practices/multilingual-catalogs.md) before you build one.

## Research the Data Before You Convert It

Do this before extraction. What you learn here decides which datasets are worth mirroring. It also decides which columns need decoding and which styles will say anything.

Work these targets in order.

**1. The portal dataset page and its siblings.** Read the page for each candidate dataset, plus the portal's own topic and tag structure. Publishers explain a dataset in prose on the portal that the service metadata omits.

**2. The service's own metadata.** For ArcGIS, fetch the service and layer JSON:

```bash
curl -s "<SERVICE_URL>/0?f=json" | jq '.fields'
```

Field aliases and `domains` carry the coded-value lists that decode integer columns. For WFS, read the capabilities and schema documents:

```bash
curl -s "<WFS_URL>?service=WFS\
&request=GetCapabilities"
```

`portolan extract <arcgis|wfs|carto> <URL> --dry-run` lists layers or tables without downloading anything. Use it here to size the job.

**3. Published data dictionaries and metadata files.** Attached PDFs, FGDC or ISO XML, and the `.txt` readme inside a shapefile download are the usual home of column meanings.

**4. The publishing department and the program.** Which office publishes a dataset explains update cadence, coverage gaps, and what the rows count. This is also where the `producer` provider comes from.

**5. The licensing terms page.** Find the terms that apply to this dataset, not the portal-wide default.

From portolan-sdi/portolan-skills, Apache-2.0.

More from portolan-sdi/portolan-skills