accelerated-computing-cudf

Files (34)

• BENCHMARK.md
• SKILL.md
• evals.json · evals/
• generate_data.py · evals/files/cudf-apply-udf/code/
• udf_pipeline.py · evals/files/cudf-apply-udf/code/
• etl_pipeline.py · evals/files/cudf-csv-etl/code/
• generate_data.py · evals/files/cudf-csv-etl/code/
• generate_data.py · evals/files/cudf-groupby-agg/code/
• groupby_analysis.py · evals/files/cudf-groupby-agg/code/
• generate_data.py · evals/files/cudf-multi-join/code/
• multi_join.py · evals/files/cudf-multi-join/code/
• NOTICE.md · evals/files/cudf-native-stream-handoff-boundary/
• run_smoke.sh · evals/files/cudf-native-stream-handoff-boundary/code/
• threaded_handoff.cu · evals/files/cudf-native-stream-handoff-boundary/code/
• generate_data.py · evals/files/cudf-null-handling/code/
• null_pipeline.py · evals/files/cudf-null-handling/code/
• generate_data.py · evals/files/cudf-parquet-io/code/
• parquet_pipeline.py · evals/files/cudf-parquet-io/code/
• generate_data.py · evals/files/cudf-pivot-melt/code/
• reshape_analysis.py · evals/files/cudf-pivot-melt/code/
• clean_contacts.py · evals/files/cudf-string-ops/code/
• generate_data.py · evals/files/cudf-string-ops/code/
• generate_data.py · evals/files/cudf-timeseries-resample/code/
• timeseries_analysis.py · evals/files/cudf-timeseries-resample/code/
• generate_data.py · evals/files/cudf-window-functions/code/
• window_analysis.py · evals/files/cudf-window-functions/code/
• train.py · evals/files/negative-deep-learning-training/code/
• NOTICE.md · evals/files/source-cudf-null-fillna-semantics/
• null_cleanup.py · evals/files/source-cudf-null-fillna-semantics/code/
• api-patterns.md · references/
• cudf-pandas-accelerator.md · references/
• dask-cudf-patterns.md · references/
• skill-card.md
• skill.oms.sig

SKILL.md (first lines)

# cuDF & dask-cuDF Implementer's Guide

## Compatibility

- Development release tracked by this skill: 26.12 (`VERSION`: `26.12.00`). Use the selected installed release for deployment requirements.
- Read `dependencies.yaml` for the support matrix (`py_version` group for Python versions, `cuda_version` for CUDA versions). Match cuDF, pylibcudf, libcudf, and RMM release versions, and match pip wheel suffixes (`-cu12` / `-cu13`) to the CUDA major version.
- Requires a supported NVIDIA GPU and compatible driver. Check the [installation requirements](https://docs.nvidia.com/datascience/install/#system-req) for the selected release and CUDA version.
- For another checkout or installed release, check `VERSION`, `dependencies.yaml`, `python/cudf/pyproject.toml`, and `cudf.__version__` before choosing versions or relying on an API.

## Naming

Use NVIDIA library-first wording in user-facing answers. Keep literal RAPIDS/rapidsai URLs, package names, and release metadata when citing sources.

## Role

You are a cuDF expert helping an implementer work with GPU DataFrames. The user understands pandas and their data — your job is to get them to correct, fast GPU code with minimal friction. Choose the path from the user's intent: `cudf.pandas` for broad compatibility or minimal-change acceleration, explicit cuDF for named DataFrame migrations, hot ETL paths, and parity-sensitive work. Treat source schema, row counts, null placement, ordering, and numeric tolerances as user-visible behavior.

## Critical Rules

1. **Choose the right cuDF path.** Use `cudf.pandas` for broad compatibility or minimal-change acceleration. Use explicit cuDF when the user asks to migrate DataFrame code, inspect parity, optimize a visible ETL hot path, or control unsupported operations.
2. **Benchmark the working set.** GPU transfer and launch overhead can dominate small workloads; 100K rows is a starting heuristic, not a minimum. Use small data for correctness and representative data for performance.
3. **Keep conversions at boundaries.** Use `.to_pandas()` or `.to_numpy()` for CPU-only libraries, display, or final output boundaries. `.values` and `.to_cupy()` return GPU arrays, not NumPy arrays. Keep intermediate ETL data on GPU.
4. **Choose precision deliberately.** Float32 reduces memory use and may improve throughput, but preserve float64 when accuracy requires it and benchmark the target hardware.
5. **Validate semantics on representative slices.** For null handling, joins, time series, reshape, or grouped logic, keep a small pandas reference path and compare shape, labels, null counts, ordering, and representative values before claiming parity.
6. **For data > GPU memory**, move to dask-cuDF with `enable_cudf_spill=True`. See `references/dask-cudf-patterns.md`.

## Three Paths to GPU DataFrames

### Path 1: cudf.pandas Accelerator (Compatibility / Minimal Change)

Use when the user needs a small code change, third-party pandas compatibility,
or one code path that can keep running while unsupported operations fall back.

**Jupyter/IPython:**
```python
%load_ext cudf.pandas
import pandas as pd   # now GPU-backed; falls back silently for unsupported ops
```

**Script:**
```bash
python -m cudf.pandas my_script.py
```

**With multiprocessing:**
```python
import cudf.pandas
cudf.pandas.install()   # must come BEFORE pandas import, before Pool creation
from multiprocessing import Pool
```

Confirm acceleration with the cudf.pandas profiler before claiming speedup.
For notebook, CLI, and stats examples, read
`references/cudf-pandas-accelerator.md`. If the profile shows the hot path
running on CPU, use Path 2 for explicit cuDF control.

### Path 2: Explicit cuDF API

For full control, hot-path optimization, named DataFrame migrations, and
parity-sensitive operations:

```python
import cudf

# Read data directly to GPU
df = cudf.read_parquet("data.parquet")

# Operations mirror pandas
result = df.groupby("key")["value"].sum()
merged = df.merge(lookup, on="id", how="left")
filtered = df[df["amount"] > 1000]

# String operations
df["clean"] = df["name"].str.strip().str.lower()

# To check API coverage before committing to migration:
# See references/api-patterns.md for known gaps and workarounds
```

**Keep data on GPU end-to-end.** Only call `.to_pandas()` at the very end for display or CPU or non-GPU handoff.

Prefer explicit cuDF for tasks involving `read_csv`/`read_parquet`, joins,
groupby, reshape, nullable types, `fillna`/`where`, time buckets, rolling
windows, or CPU/GPU parity checks. Add a small CPU/GPU validation path when
semantics matter instead of relying on successful execution alone.

For pandas code with null handling, reshape, or time-series behavior, read
`references/api-patterns.md` for the relevant semantic checklist before
rewriting. A `cudf.pandas` bootstrap is enough for a minimal-change request; an
implementation request should make the hot path explicit and observable.

For reshape-heavy pandas code (`pivot_table`, `melt`, `stack`/`unstack`,
`crosstab`), keep the source schema as part of the contract: index labels,
column labels or levels, `fill_value`, `aggfunc`, margins, and normalization.
Use explicit cuDF where the equivalent is supported; use `cudf.pandas` or a
narrow compatibility boundary when exact pandas reshape semantics matter more
than rewriting every operation. Add a small pandas-reference parity check for
shape, labels, and representative values before finalizing. See
`references/api-patterns.md`.

### Path 3: dask-cuDF (Multi-GPU / Large Data)

When dataset exceeds GPU memory. See `references/dask-cudf-patterns.md` for full patterns.

```python
from dask_cuda import LocalCUDACluster
from dask.distributed import Client
import dask
import dask.dataframe as dd

dask.config.set({"dataframe.backend": "cudf"})

cluster = LocalCUDACluster(enable_cudf_spill=True)  # one worker per GPU
client = Client(cluster)

ddf = dd.read_parquet("s3://bucket/data/*.parquet")
result = ddf.groupby("key").agg({"value": "sum"}).compute()
```

From NVIDIA/skills, CC-BY-4.0 AND Apache-2.0.

More from NVIDIA/skills