git-backed-catalog

Files (1)

• SKILL.md

SKILL.md (first lines)

<!-- drift: depends-on: portolan-catalog-template, portolan-spec, rashid, portolan-browser -->

# Git-Backed Portolan Catalogs

A git-backed catalog keeps its metadata in a repository and its data in object storage. The repository publishes the catalog. The bucket serves readers.

This pattern is useful for three reasons:

* **Rollback.** Git keeps every previous version of the catalog and the changes between them.
* **A validation loop.** CI runs `rashid` on every pull request. Edit, validate, fix, and repeat until the checks pass. Scheduled checks run documented queries against published data.
* **A contribution path.** Someone who finds a wrong license can open a pull request instead of hunting for a contact address.

See the [git-backed catalogs guidance](https://github.com/portolan-sdi/portolan-spec/blob/main/specs/best-practices/git-backed-catalogs.md) for the background and tradeoffs. The specification is ground truth. `rashid` implements it.

Pick the mode that matches what the user asked for.

## Catalog navigation

Apply the [navigation contract](../portolan-bootstrap/reference/catalog-navigation.md) in the repository's generator and checks. Replace the template's root `vcs` and `issues` placeholders.

## Mode A: create a new git-backed catalog

This path suits a catalog that takes contributions or needs rollback. If the user wants to convert a data source and publish it to a bucket, use the `portolan-bootstrap` skill instead.

Start from the template. It provides a stub catalog that passes `rashid` on the first commit, so CI is green before the catalog is populated.

```bash
gh repo create <owner>/<name> \
  --template portolan-sdi/portolan-catalog-template \
  --public --clone
cd <name>
```

Then work through `SETUP.md` in the new repository. It is authoritative. Do not improvise around it. Its ten steps are:

1. Point `catalog.publish.yaml` at your storage: `write_prefix`, `public_base`, and `region`.
2. Name the catalog in `catalog/catalog.json`: `id`, `title`, and `description`.
3. Rewrite `catalog/README.md` and `catalog/AGENTS.md`.
4. Rewrite the repository `README.md`.
5. Decide official or mirror. A mirror collection needs `providers` with the `producer` and `host` roles set correctly, a `via` link to the source (PORTO-CORE-053), and `updated` at each sync (PORTO-CORE-057).
6. Add a logo, optionally, as a root `rel: icon` link with an image media type (PORTO-CORE-075).
7. Replace the `TODO(setup)` hrefs in the `vcs` and `issues` links the template provides. Both are absolute URLs, because the repository is outside the published catalog.
8. Add collections.
9. Finish: no `TODO(setup)` remains, `python3 tests/run_all.py` passes, `python3 tools/publish.py` dry-runs clean. Then delete `tests/test_setup.py` and `SETUP.md`.
10. Outside the `portolan-sdi` organization, remove `.github/workflows/repo-checks.yml` and the `ops-sync` blocks.

These two guards stop you, and both are deliberate:

* `tests/test_setup.py` fails while the repository is half-edited. An untouched template passes, and so does a finished one.
* `tools/publish.py` refuses to upload while a sentinel value remains in `catalog.publish.yaml`. It checks before any AWS call, so this works without credentials.

### Declare the current schema version

The pinned template declares `https://schemas.portolan-sdi.org/portolan/v0.2.0/schema.json` in `catalog/catalog.json`. See PORTO-CORE-006, PORTO-CORE-009, and PORTO-CORE-081 for schema and `self` link requirements.

### Raise the validator floor in the gate

To use the pinned rashid 0.3.0, update `tests/test_conformance.py`: set `MIN_VERSION = (0, 3, 0)`, `MAX_VERSION = (0, 4, 0)`, and `SPEC = "rashid>=0.3.0,<0.4.0"`. Install the same version locally and in CI:

```bash
python3 -m venv .venv
.venv/bin/pip install 'rashid>=0.3.0,<0.4.0'
.venv/bin/rashid --version
```

### Adding a collection

Data files do not belong in git. Build them, upload them to the bucket, and write STAC that references them by public URL. The `.gitignore` already blocks common data formats.

Each collection directory contains `collection.json`, `AGENTS.md`, and `README.md` (PORTO-CORE-005). The `AGENTS.md` link is `rel: agents` and the `README.md` link is `rel: describedby`, both `type: text/markdown` (PORTO-CORE-061, PORTO-CORE-062). The root's `child` link has `type: application/json` and a `title` (PORTO-CORE-033, PORTO-CORE-039). An absolute asset href uses `https`, never `s3` (PORTO-CORE-023). Offer the `s3` form through the alternate-assets extension (PORTO-CORE-024).

Items are normative. A collection with more than one raster scene models each scene as an item in its own subdirectory (PORTO-CORE-015, PORTO-CORE-071). Commit that item JSON. A collection with many items also publishes `items.parquet`, registered as a collection asset with role `collection-mirror` (PORTO-FMT-040, PORTO-FMT-041). The mirror is derived from the items. It never replaces item JSON or the collection's `item` links (PORTO-FMT-042). Generate the mirror with `portolan stac-geoparquet`, and keep the generator, not only its output, in the repository.

The single-file rule is the exception: one GeoParquet file or one COG is a collection-level asset with no item (PORTO-CORE-017, PORTO-CORE-072).

Run the gates before committing. `tests/test_links.py` catches a `child` link added before its directory exists.

### Uploading the data

`tools/publish.py` syncs `catalog/` and nothing else. The narrow scope keeps a scratch file out of a public bucket. Do not widen it.

The template provides `tools/upload_data.py` for the data. Set `data_dir` in `catalog.publish.yaml` to the staging directory. The script admits only files under `data_dir` with a publishable suffix, imports its change detection and sentinel guard from `publish.py`, and never deletes. Run it dry first:

```bash
python3 tools/upload_data.py            # dry run
python3 tools/upload_data.py --confirm  # upload; needs AWS credentials
```

Its change detection compares a multipart object by size alone. Pass `--force` after you replace a file with one of the same size.

### CI checks links without the bytes

A fresh clone in CI has the metadata and not the data, so asset hrefs do not resolve there. The template's `ci.yml` sets `CI_LIGHT=1`, and `tests/test_links.py` then exempts asset hrefs with a data suffix. Structural links stay checked. Leave `CI_LIGHT` unset locally, so the full check runs where the bytes are.

### Test the documented queries and claims

The template does not test documented queries or claims, so add a check that reads them directly from Markdown under `catalog/`. Keep the queries in the docs rather than maintaining a second copy in the tests.

* Run each query against the published URLs and fail the check if it errors or returns no rows. If a query should return no rows or is an incomplete fragment, mark that exception before its fence.
* Beside each data claim, include a query that returns rows contradicting it. You can put the query in an HTML comment. If the query returns any rows, fail the check.
* Run the checks on pull requests, after uploads, and on a schedule that matches how often the source changes.
* If a scheduled check fails, open an issue or update the existing one, then close it when the checks pass. Grant issue write permission only to the job that manages this issue.

Choose a query engine and setup that suit the data. The [Philadelphia housing example](https://github.com/nlebovits/phl-housing-demo) runs documented SQL with DuckDB after each refresh.

For row counts and percentages, provide a query instead of a value that can go stale.

## Mode B: maintain an existing catalog

The loop is:

```bash
python3 tests/run_all.py
git add -A && git commit -m "fix: ..."
python3 tools/publish.py             # dry run
python3 tools/publish.py --confirm   # upload; needs AWS credentials
```

When maintaining a catalog:

From portolan-sdi/portolan-skills, Apache-2.0.

More from portolan-sdi/portolan-skills