earth2studio-create-datasource

Files (19)

• BENCHMARK.md
• SKILL.md
• config.yml · evals/
• Dockerfile · evals/environment/
• bootstrap.sh · evals/environment/setup/
• evals.json · evals/
• eval_1_target.py · evals/targets/
• eval_2_target.py · evals/targets/
• eval_3_target.py · evals/targets/
• eval_4_target.py · evals/targets/
• eval_1_test_target.py · evals/targets/test/
• eval_2_test_target.py · evals/targets/test/
• eval_3_test_target.py · evals/targets/test/
• eval_4_test_target.py · evals/targets/test/
• implementation-guide.py · references/
• testing-guide.py · references/
• validation-guide.md · references/
• skill-card.md
• skill.oms.sig

SKILL.md (first lines)

# Create and Validate Data Source

## Purpose

End-to-end workflow for implementing a new Earth2Studio data source wrapper
that connects a remote data store (S3, GCS, Azure, HTTP, HuggingFace) to
Earth2Studio's async data fetching infrastructure — from analysis through
implementation, testing, validation, and PR submission.

## Prerequisites

- Earth2Studio dev environment with `uv` (`uv run python` must work)
- Git configured with fork (`origin`) and upstream (`upstream`) remotes
- Access to the target remote data store (credentials if private)
- Python 3.10+

## Workspace

Use the directory containing `pyproject.toml`. For Harbor evals, write to
`/workspace/output/` preserving paths. Never read `evals/targets/`.

## Instructions

> **Python Environment:** Always use `uv run python` or the local `.venv`.
> Never use the system Python directly.

Follow every step in order.

> **[CONFIRM] gates:** Only Step 1 (Source Type) and Step 12 (Sanity-Check
> Plots) require explicit user approval. All other `[CONFIRM]` markers are
> advisory — present decisions inline and proceed without blocking.
>
> **Deliverables first:** Write the source file and test file (Steps 6–7)
> before extended exploration, documentation, registration, CHANGELOG, or PR
> work. Skip Steps 8–14 when the user asks for implementation only.
>
> **Before you finish:** Run verification commands in the repo root so results
> appear in the session log:
>
> ```bash
> uv run pytest test/data/test_<source>.py -x
> make format && make lint
> ```
>
> **Be concise:** Avoid long architecture reports; summarize decisions in a
> few sentences and move on to file writes.
>
> **Hangs or User Feedback** If agent becomes stuck or user provides a
> correction during this skills use, conservatively review relevant part of
> the skill and improve. Be concise.
>
> **One source type per invocation.** Invoke again for companion types.

### Reference Files

Load these on demand during the relevant steps:

| File | Content | Load at |
|---|---|---|
| `references/implementation-guide.py` | Skeleton source with FILL comments | Steps 3–10 |
| `references/testing-guide.py` | Test skeleton with FILL comments | Step 11 |
| `references/validation-guide.md` | Plot templates, PR body template, Greptile handling | Steps 12–14 (optional, for templates) |

---

### Workflow Overview

```text
Step 0: Obtain reference → Step 1: Determine type → Step 2: Dependencies
→ Step 3: Add deps → Step 4: Create lexicon → Step 5: Update vocab/schema
→ Step 6: Create skeleton → Step 7: Implement source → Step 8: Register
→ Step 9: Documentation → Step 10: CHANGELOG → Step 11: Tests
→ Step 12: Validate & plots (user confirms) → Step 13: PR + sanity comment
→ Step 14: Greptile review
```

---

### Step 0 — Obtain Remote Data Store Reference

If `$ARGUMENTS` is provided, use it (URL → WebFetch; file path → read).

If empty, ask:

> Please provide a URL, API documentation link, or description of the
> remote data store. This will be used to understand storage format,
> access pattern, variable inventory, temporal/spatial resolution.

---

### Step 1 — Determine Source Type

| Protocol | Returns | Has `lead_time`? | Use |
|---|---|---|---|
| **DataSource** | `xr.DataArray` | No | Gridded analysis/reanalysis |
| **ForecastSource** | `xr.DataArray` | Yes | Gridded forecast |
| **DataFrameSource** | `pd.DataFrame` | No | Sparse/station obs |
| **ForecastFrameSource** | `pd.DataFrame` | Yes | Sparse forecast obs |

Key factors: gridded vs sparse → DataArray vs DataFrame; analysis vs forecast → Source vs ForecastSource.

#### [CONFIRM — Source Type]

Present recommended type with justification. Ask for confirmation.

---

### Step 2 — Examine Remote Store & Propose Dependencies

**Analyze:** storage backend, file format, authentication, access pattern,
temporal/spatial resolution, variable inventory.

**Prefer fsspec:**

| Backend | Preferred | Avoid |
|---|---|---|
| AWS S3 | `s3fs` (core dep) | `boto3` directly |
| GCS | `gcsfs` (core dep) | `google-cloud-storage` |
| Azure | `adlfs` | `azure-storage-blob` |

From NVIDIA/skills, Apache-2.0.

More from NVIDIA/skills