datasets

Files (27)

• SKILL.md
• api-surface.md · references/
• repo-provenance.md · references/
• repo-routing-metadata.json · references/
• troubleshooting.md · references/
• datasets_api_smoke.py · scripts/
• SKILL.md · sub-skills/features-formats/
• format-loaders.md · sub-skills/features-formats/references/
• schema-and-features.md · sub-skills/features-formats/references/
• troubleshooting.md · sub-skills/features-formats/references/
• feature_schema_smoke.py · sub-skills/features-formats/scripts/
• SKILL.md · sub-skills/loading-local-hub/
• api-reference.md · sub-skills/loading-local-hub/references/
• loading-workflows.md · sub-skills/loading-local-hub/references/
• troubleshooting.md · sub-skills/loading-local-hub/references/
• create_local_loading_fixture.py · sub-skills/loading-local-hub/scripts/
• SKILL.md · sub-skills/processing-streaming/
• api-reference.md · sub-skills/processing-streaming/references/
• processing-workflows.md · sub-skills/processing-streaming/references/
• troubleshooting.md · sub-skills/processing-streaming/references/
• processing_smoke.py · sub-skills/processing-streaming/scripts/
• SKILL.md · sub-skills/sharing-cli-cache/
• cli-cache-reference.md · sub-skills/sharing-cli-cache/references/
• sharing-and-hub.md · sub-skills/sharing-cli-cache/references/
• troubleshooting.md · sub-skills/sharing-cli-cache/references/
• cli_cache_smoke.py · sub-skills/sharing-cli-cache/scripts/
• dataset_card_minimal.md · sub-skills/sharing-cli-cache/scripts/

SKILL.md (first lines)

# Hugging Face Datasets

Use this repo skill for practical work with the `datasets` Python package. It is self-contained guidance distilled from the Datasets source, docs, tests, and verified public API surface.

## Start Here

- Install the package with `pip install datasets`; add extras only for workflows that need them, such as `datasets[audio]`, `datasets[vision]`, `datasets[pdfs]`, `datasets[nibabel]`, or ML framework packages.
- Run `python -c "from datasets import Dataset, load_dataset; print(Dataset.from_dict({'x':[1]}))"` for a minimal import/API smoke check.
- Use `load_dataset(...)` for Hub datasets, local files, local dataset repositories, and packaged format builders.
- Use `Dataset.from_dict`, `Dataset.from_list`, `Dataset.from_pandas`, or `Dataset.from_generator` when data already exists in Python.

## Route by Task

- Load datasets from the Hub, local files/directories, Python objects, generators, packaged modules, or disk snapshots with `sub-skills/loading-local-hub/SKILL.md`.
- Transform, filter, batch, split, shuffle, stream, format, combine, or convert datasets with `sub-skills/processing-streaming/SKILL.md`.
- Define `Features`, cast columns, troubleshoot schemas, and work with CSV/JSON/Parquet/Arrow/text/image/audio/video/PDF/NIfTI/mesh formats with `sub-skills/features-formats/SKILL.md`.
- Push datasets to the Hub, create dataset cards, use `datasets-cli`, manage cache/offline mode, or diagnose filesystem/storage issues with `sub-skills/sharing-cli-cache/SKILL.md`.

## Shared References and Scripts

- Read `references/api-surface.md` for the verified top-level API map and the most important signatures.
- Read `references/troubleshooting.md` for cross-cutting install, import, optional dependency, auth, cache, and workflow routing failures.
- Read `references/repo-provenance.md` when checking whether this skill is stale relative to a Datasets checkout.
- Run `scripts/datasets_api_smoke.py --help` for a safe local smoke helper that verifies import, core APIs, tiny in-memory processing, and CLI availability.

## Common Decisions

- Prefer explicit `data_files`, `split`, and `features` when automatic local file inference would be ambiguous.
- Use `streaming=True` for large or remote data that should not be fully downloaded; route downstream iterable semantics to `processing-streaming`.
- Use `decode=False` for media columns when optional decoders are unavailable or when you only need paths/bytes metadata.
- Do not put tokens in code, logs, dataset cards, or generated examples; pass tokens through normal Hugging Face auth mechanisms.
- Treat `delete_from_hub` and `push_to_hub` as state-changing operations; dry-run the plan and confirm repo IDs/config names before running them.

## Validation Pattern

1. Confirm the package imports and the relevant optional dependency is installed.
2. Build or load a tiny representative dataset first.
3. Declare `Features` when schema inference is risky.
4. Run the intended map/filter/cast/format operation on a tiny subset.
5. Save, reload, or push only after the local smoke path behaves as expected.

From VectorSpaceLab/AREX-Skill, Apache-2.0.

More from VectorSpaceLab/AREX-Skill