dataflow

Files (40)

• SKILL.md
• api-overview.md · references/
• installation-and-backends.md · references/
• repo-provenance.md · references/
• repo-routing-metadata.json · references/
• troubleshooting.md · references/
• check_dataflow_env.py · scripts/
• inspect_dataflow_surface.py · scripts/
• SKILL.md · sub-skills/document-vision-rag/
• document-rag-workflows.md · sub-skills/document-vision-rag/references/
• pdf2model-and-vqa.md · sub-skills/document-vision-rag/references/
• serving-and-dependencies.md · sub-skills/document-vision-rag/references/
• troubleshooting.md · sub-skills/document-vision-rag/references/
• check_document_workflow_inputs.py · sub-skills/document-vision-rag/scripts/
• SKILL.md · sub-skills/pipeline-foundations/
• api-reference.md · sub-skills/pipeline-foundations/references/
• pipeline-patterns.md · sub-skills/pipeline-foundations/references/
• storage-and-data-formats.md · sub-skills/pipeline-foundations/references/
• troubleshooting.md · sub-skills/pipeline-foundations/references/
• smoke_pipeline_foundations.py · sub-skills/pipeline-foundations/scripts/
• validate_tabular_input.py · sub-skills/pipeline-foundations/scripts/
• SKILL.md · sub-skills/rayorch-acceleration/
• api-reference.md · sub-skills/rayorch-acceleration/references/
• troubleshooting.md · sub-skills/rayorch-acceleration/references/
• workflows.md · sub-skills/rayorch-acceleration/references/
• smoke_rayorch_cpu.py · sub-skills/rayorch-acceleration/scripts/
• SKILL.md · sub-skills/serving-cli/
• cli-reference.md · sub-skills/serving-cli/references/
• evaluation-and-webui.md · sub-skills/serving-cli/references/
• serving-backends.md · sub-skills/serving-cli/references/
• troubleshooting.md · sub-skills/serving-cli/references/
• inspect_cli_help.py · sub-skills/serving-cli/scripts/
• smoke_api_serving_request.py · sub-skills/serving-cli/scripts/
• SKILL.md · sub-skills/text-workflows/
• data-formats.md · sub-skills/text-workflows/references/
• operator-catalog.md · sub-skills/text-workflows/references/
• text-pipelines.md · sub-skills/text-workflows/references/
• text2model-workflow.md · sub-skills/text-workflows/references/
• troubleshooting.md · sub-skills/text-workflows/references/
• make_text_fixture.py · sub-skills/text-workflows/scripts/

SKILL.md (first lines)

# DataFlow

Use this repo skill for `open-dataflow` / `dataflow` tasks that prepare data, build pipelines, launch serving backends, run evaluations, or wrap operators with RayOrch.

## First reads

- `references/repo-provenance.md` — source commit, package version, evidence paths, and refresh baseline.
- `references/installation-and-backends.md` — install commands, verified import checks, and backend selection.
- `references/api-overview.md` — verified public surface for the most important classes, CLIs, and serving helpers.
- `references/troubleshooting.md` — cross-cutting install/import, key-mismatch, TTY, and backend failure guidance.
- `references/repo-routing-metadata.json` — routing metadata used by the managed repo-skill router.

## Fast start

1. Install the package for local inspection or use the published distribution:
   - `python -m pip install -e .`
   - or `python -m pip install open-dataflow`
2. Run a safe environment smoke check:
   - `python scripts/check_dataflow_env.py`
3. Inspect the public API surface when you need signatures or command names:
   - `python scripts/inspect_dataflow_surface.py`
4. Read the focused sub-skill that matches the task family below.

## Route map

### [pipeline-foundations](sub-skills/pipeline-foundations/SKILL.md)
Use for operator, pipeline, storage, prompt, wrapper, and compile-time key-validation work.

Typical requests:
- create or debug a `PipelineABC`, `BatchedPipelineABC`, or `StreamBatchedPipelineABC`
- choose between `FileStorage`, `LazyFileStorage`, `DummyStorage`, batch storage, or MyScale storage
- fix `input_*` / `output_*` mismatches or `Key Matching Error`
- validate `prompt_restrict`, `PromptABC`, `DIYPromptABC`, or `draw_graph`

### [serving-cli](sub-skills/serving-cli/SKILL.md)
Use for CLI routing, `dataflow init`, `chat`, `eval`, `pdf2model`, `text2model`, `webui`, and serving class setup.

Typical requests:
- inspect command groups and help output
- choose an API, local, or hosted serving backend
- understand credential, timeout, or WebUI side effects
- diagnose missing optional serving dependencies

### [text-workflows](sub-skills/text-workflows/SKILL.md)
Use for text cleaning, filtering, reasoning, code, conversation, Text2SQL, prompt-driven generation, translation, and text2model prep.

Typical requests:
- adapt CPU-safe text filters or prompt-driven generators
- map columns such as `raw_content`, `instruction`, `problem`, `generated_code`, or `golden_answer`
- prepare offline text fixtures or text2model inputs
- separate API-backed stages from pure local filtering

### [document-vision-rag](sub-skills/document-vision-rag/SKILL.md)
Use for PDF/OCR, visual QA, knowledge-base cleaning, LightRAG, Agentic RAG, speech, chemistry, and `pdf2model` planning.

Typical requests:
- validate document inputs before OCR or retrieval
- choose between KBC, PDF VQA, or document-prep flows
- reason about MinerU, FlashMinerU, DataFlex, LlamaFactory, and audio extras
- diagnose missing documents, suffixes, or hardware/credential limits

### [rayorch-acceleration](sub-skills/rayorch-acceleration/SKILL.md)
Use for RayOrch wrapping, actor cleanup, and pipeline acceleration without changing the surrounding pipeline contract.

Typical requests:
- wrap an existing operator with `RayAcceleratedOperator`
- preserve order and storage behavior through CPU or GPU execution
- decide when to use `replicas`, `num_gpus_per_replica`, or `env`
- debug shutdown or optional Ray dependency issues

## How to choose

- If the task is mainly about **how DataFlow operators and pipeline storage work**, start with `pipeline-foundations`.
- If the task is mainly about **CLI commands, serving, or launch-time dependencies**, start with `serving-cli`.
- If the task is mainly about **text or tabular dataset transformation**, start with `text-workflows`.
- If the task is mainly about **documents, PDFs, OCR, RAG, or pdf2model**, start with `document-vision-rag`.
- If the task is mainly about **distributed acceleration of an existing operator**, start with `rayorch-acceleration`.
- If the request spans multiple families, use the root references first, then hand off to the narrowest sub-skill that owns the final action.

## What this root skill does not do

- It does not execute training, downloads, or backend-heavy workflows by default.
- It does not rely on the original checkout after generation; bundled references and scripts carry the reusable guidance.
- It does not replace the focused sub-skills when the request is already narrow.

## When to revisit provenance

Read `references/repo-provenance.md` before deciding whether this skill is stale for a checkout of DataFlow or before running a refresh workflow.

From VectorSpaceLab/AREX-Skill, Apache-2.0.

More from VectorSpaceLab/AREX-Skill