dask

Files (37)

• SKILL.md
• installation-and-environment.md · references/
• repo-provenance.md · references/
• repo-routing-metadata.json · references/
• troubleshooting.md · references/
• dask_package_smoke.py · scripts/
• SKILL.md · sub-skills/array-workflows/
• api-reference.md · sub-skills/array-workflows/references/
• chunking-and-performance.md · sub-skills/array-workflows/references/
• troubleshooting.md · sub-skills/array-workflows/references/
• workflows.md · sub-skills/array-workflows/references/
• array_smoke.py · sub-skills/array-workflows/scripts/
• SKILL.md · sub-skills/bag-bytes-workflows/
• api-reference.md · sub-skills/bag-bytes-workflows/references/
• io-and-data-formats.md · sub-skills/bag-bytes-workflows/references/
• troubleshooting.md · sub-skills/bag-bytes-workflows/references/
• workflows.md · sub-skills/bag-bytes-workflows/references/
• bag_text_smoke.py · sub-skills/bag-bytes-workflows/scripts/
• SKILL.md · sub-skills/configuration-diagnostics-cli/
• cli-reference.md · sub-skills/configuration-diagnostics-cli/references/
• configuration.md · sub-skills/configuration-diagnostics-cli/references/
• development-and-testing.md · sub-skills/configuration-diagnostics-cli/references/
• diagnostics-and-performance.md · sub-skills/configuration-diagnostics-cli/references/
• troubleshooting.md · sub-skills/configuration-diagnostics-cli/references/
• dask_cli_smoke.py · sub-skills/configuration-diagnostics-cli/scripts/
• SKILL.md · sub-skills/core-graphs-schedulers/
• api-reference.md · sub-skills/core-graphs-schedulers/references/
• troubleshooting.md · sub-skills/core-graphs-schedulers/references/
• workflows.md · sub-skills/core-graphs-schedulers/references/
• core_smoke.py · sub-skills/core-graphs-schedulers/scripts/
• SKILL.md · sub-skills/dataframe-workflows/
• api-reference.md · sub-skills/dataframe-workflows/references/
• io-and-data-formats.md · sub-skills/dataframe-workflows/references/
• troubleshooting.md · sub-skills/dataframe-workflows/references/
• workflows.md · sub-skills/dataframe-workflows/references/
• dataframe_demo_smoke.py · sub-skills/dataframe-workflows/scripts/
• dataframe_smoke.py · sub-skills/dataframe-workflows/scripts/

SKILL.md (first lines)

# Dask Repo Skill

Use this skill when a task involves Dask's public APIs, internal task-graph model, collection workflows, configuration, diagnostics, or contributor test/build practices. Dask provides lazy parallel collections and schedulers for Python analytics: delayed objects, arrays, dataframes, bags, low-level task graphs, and local/distributed execution integrations.

## First Checks

- Import check: `python -c "import dask; print(dask.__version__)"`.
- CLI check: `dask --help` should show `config`, `docs`, and `info` command groups.
- Minimal compute check: `python scripts/dask_package_smoke.py --scheduler synchronous`.
- Install extras: use `dask[array]` for array workflows, `dask[dataframe]` for dataframe workflows, `dask[diagnostics]` for local dashboard/profiling helpers, and `dask[distributed]` only when remote cluster APIs are needed.

## Route By Task

- **Core graphs and schedulers:** Use `sub-skills/core-graphs-schedulers/SKILL.md` for `dask.delayed`, `compute`, `persist`, `optimize`, annotations, tokenization, HighLevelGraph, low-level task specs, scheduler choice, and graph debugging.
- **Array workflows:** Use `sub-skills/array-workflows/SKILL.md` for `dask.array`, chunk planning, `from_array`, slicing, `map_blocks`, `blockwise`, reductions, overlap, rechunking, gufuncs, random arrays, linalg, FFT, stats, and array backend caveats.
- **DataFrame workflows:** Use `sub-skills/dataframe-workflows/SKILL.md` for `dask.dataframe`, pandas-like APIs, CSV/Parquet/JSON/SQL IO, partitions/divisions, joins, groupby, shuffle, repartitioning, pyarrow strings, categoricals, and `dask_expr` query planning.
- **Bag and bytes workflows:** Use `sub-skills/bag-bytes-workflows/SKILL.md` for `dask.bag`, `dask.bytes`, text and byte IO, JSON-like records, Avro, fsspec URLs, compression, `foldby`, and small-file object pipelines.
- **Configuration, diagnostics, and CLI:** Use `sub-skills/configuration-diagnostics-cli/SKILL.md` for `dask.config`, YAML config paths, environment variables, `dask config`, `dask info`, local profilers, progress bars, callbacks/cache, install extras, and contributor validation commands.

## Shared References

- Read `references/installation-and-environment.md` when choosing extras, verifying imports, or diagnosing optional dependency availability.
- Read `references/troubleshooting.md` for cross-cutting failures that affect multiple Dask collections or schedulers.
- Read `references/repo-provenance.md` before relying on this skill for a changed checkout; refresh the skill when commit, package version, or major evidence paths drift.

## Dask Working Rules

- Keep graph construction lazy. Do not call `.compute()` or `.persist()` inside collection methods or while defining a reusable graph unless the user explicitly wants materialization.
- Use metadata, divisions, chunks, dtypes, and `meta` objects to infer output shape/schema instead of computing sample data.
- Choose the smallest collection that matches the data model: delayed for arbitrary Python functions, array for NumPy-like blocked tensors, dataframe for pandas-like tabular data, bag for unordered Python records, and bytes for low-level file blocks.
- Prefer explicit scheduler and config scopes when reproducing behavior: `with dask.config.set({...}): ...` and `compute(..., scheduler="synchronous")` for deterministic local debugging.
- Treat `array.query-planning` and `dataframe.query-planning` as import-time configuration; set them before importing the relevant collection modules in a fresh process.
- For dataframe and array APIs, validate `meta`, chunks, divisions, and optional dependency requirements before changing algorithms.

## Validation Pattern

1. Run the root smoke script for package-level sanity.
2. Run the owning sub-skill smoke script for the collection or support workflow.
3. Add a tiny fixture or in-memory example that exercises the changed route.
4. For repo contribution work, run the narrowest relevant pytest selection before broader suites.
5. Record skipped native cases separately when they require network, GPU, distributed services, large data, or credentials.

## Common Pitfalls

- Importing `dask.dataframe` without the dataframe extra can fail due to missing pandas or pyarrow.
- Unknown chunks or divisions often make otherwise valid operations expensive, ambiguous, or unsupported.
- Multiprocessing schedulers require pickleable functions and guarded entry points; use threads or synchronous mode for quick debugging.
- Config files may be read from multiple paths; `dask config find <key>` is the safest way to diagnose where a value comes from.
- Optional GPU/cloud/storage backends are not part of the base package; keep those checks conditional and skip unsafe native cases unless dependencies and hardware are available.

From VectorSpaceLab/AREX-Skill, Apache-2.0.

More from VectorSpaceLab/AREX-Skill