sourcecoop

Files (1)

• SKILL.md

SKILL.md (first lines)

<!-- drift: depends-on: portolan-cli, rashid, portolan-spec, source-coop-cli -->

# Source Cooperative Upload Skill

You are helping a user publish a Portolan catalog to [Source Cooperative](https://source.coop), an open data commons for geospatial data. This skill covers the upload path. To build the catalog from a data source, use the `portolan-bootstrap` skill first, then return here.

The [Portolan spec](https://github.com/portolan-sdi/portolan-spec) is ground truth. The CLI and the validator implement it. Run `portolan <cmd> --help` for the flags of any command below. Do not restate the help text in your answers.

## Prerequisites

Source Cooperative serves its object storage through a data proxy at `data.source.coop`. The [`source-coop` CLI](https://github.com/source-cooperative/source-coop-cli) authenticates you against the proxy and issues temporary S3 credentials. Check for it first:

```bash
source-coop --version
```

This skill describes version 0.3.0. An older version does not refresh expired credentials. Install it with Homebrew, or with the installer script the CLI README documents:

```bash
brew install source-cooperative/tap/source-coop
```

You also need a Source Cooperative account with write access to the repository. The user without access requests it at hello@source.coop.

## Workflow Overview

1. Gather the organization and product names.
2. Log in with `source-coop`, set the remote and the endpoint in `.env`, and export the credentials.
3. Initialize the catalog with a license.
4. Add files.
5. Write metadata: contact, license, providers, `source_url`.
6. Generate READMEs and write `AGENTS.md`.
7. Run the validator locally.
8. Push, then probe the published host.

## Step 1: Gather Information

Ask the user for:

1. The Source Cooperative organization slug (required). Examples: `nlebovits`, `radiant-mlhub`, `vida`.
2. The product name (optional). Defaults to the current directory name.

The proxy addresses the account as the bucket and the repository as the key prefix. Build the remote URL from them:

```
s3://{org}/{product}/
```

The same URL form works for every storage region. Source Cooperative stores repositories in more than one region, such as `us-west-2`, `ca-central-1`, and an Azure region in West Europe. The proxy finds the region of the repository. The remote contains no region.

Do not use a regional bucket such as `s3://us-west-2.opendata.source.coop/{org}/{product}/` or `s3://ca-central-1.opendata.source.coop/{org}/{product}/`. The `source-coop` credentials are valid only at the proxy. AWS rejects them at a regional bucket with `InvalidAccessKeyId`.

## Step 2: Log in and set the remote

`source-coop login` opens a browser for the OAuth2 authorization code flow. It caches the temporary credentials and a refresh token in the OS keyring:

```bash
source-coop login
```

`login` receives the OAuth2 redirect on a local port. On a machine with no browser, forward a port over SSH and give the same port to `login`:

```bash
ssh -L 8400:127.0.0.1:8400 user@server
source-coop login --port 8400
```

Run the second command on the server. Open the URL that it prints in a local browser.

### The remote

The remote is the destination that `portolan push` writes to. The proxy addresses the account as the bucket and the repository as the key prefix. `portolan push` reads the destination from `PORTOLAN_REMOTE` when you give it no argument.

The CLI refuses to store `remote`, `profile`, `region`, and `s3_endpoint` in `.portolan/config.yaml`. That file is pushed with the catalog. Put the remote and the proxy endpoint in `.env` at the catalog root. `portolan push` excludes `.env` from the upload.

```bash
cat > .env << 'EOF'
PORTOLAN_REMOTE=s3://{org}/{product}/
PORTOLAN_S3_ENDPOINT=https://data.source.coop
EOF

portolan config list
```

`config list` shows `remote` and `s3_endpoint` with `(from env)`. Without the endpoint, the push goes to AWS and fails with `InvalidAccessKeyId`.

### The credentials

`portolan push` reads the credentials from the `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` environment variables. Export them in the shell you push from:

```bash
eval "$(source-coop creds --format env)"
```

The credentials expire, and the shell keeps the old values. `source-coop creds` refreshes expired credentials from the cached refresh token. When an upload fails with `AccessDenied` or `403 Forbidden`, run the `eval` line again.

Do not set `PORTOLAN_PROFILE`, and do not pass `--profile` to `portolan push`. portolan-cli 0.8.0 reads a named profile only from `~/.aws/credentials`. It ignores `credential_process` and `endpoint_url` in `~/.aws/config`. A profile name other than `default` also discards the credentials that the `eval` line set. The push then finds no credentials and stalls against the instance metadata service at `169.254.169.254`.

### The AWS profile for other tools

The aws CLI, boto3, rclone, and DuckDB run `credential_process` from `~/.aws/config`. The [Source Cooperative docs](https://docs.source.coop/upload-with-the-cli) give this profile:

```ini
[profile source-coop]
credential_process = source-coop creds
endpoint_url = https://data.source.coop
```

`portolan push` does not use this profile. Use it to list the prefix and to confirm what the push wrote:

```bash
aws s3 ls s3://{org}/{product}/ --profile source-coop --recursive
```

## Step 3: Initialize Catalog

`portolan init` requires a license. With `--auto` and no `--license`, it exits with `PRTLN-VAL004`. Every collection inherits the catalog license, and `portolan add` refuses a collection without one.

```bash

From portolan-sdi/portolan-skills, Apache-2.0.

More from portolan-sdi/portolan-skills