gfxgraph-development

Files (1)

• SKILL.md

SKILL.md (first lines)

# gfxGRAPH Development Guide

This skill provides deep knowledge of gfxGRAPH internals for agents who need to modify,
extend, or debug the library itself.

## Repository Layout

```
gfxGRAPH/
├── python/
│   ├── gfxgraph/                    # Public package (user-facing)
│   │   ├── __init__.py              # Re-exports + auto-enable via GFXGRAPH env var
│   │   ├── __main__.py              # CLI: python -m gfxgraph <script.py>
│   │   └── _enable.py               # enable()/disable()/stats()/health_check()
│   ├── hipgraph_bridge/             # Internal implementation (the actual bridge)
│   │   ├── __init__.py              # Exports BridgedCUDAGraph, ShapeBucketPool
│   │   ├── graph_manager.py         # BridgedCUDAGraph — core drop-in replacement
│   │   ├── shape_bucketing.py       # ShapeBucketPool — multi-bucket graph pool
│   │   ├── conditional.py           # ConditionalGraph — per-branch dispatch (Gap 51)
│   │   ├── compile_backend.py       # torch.compile backend registration
│   │   ├── ops.py                   # Custom torch.library ops (conditional_select)
│   │   └── _C.py                    # ctypes bindings to native .so (optional)
│   └── pyproject.toml
├── src/                             # C/HIP native bridge source
│   └── hipgraph_bridge/
│       └── bridge.hip               # Gap bridges 52, 54 (device-side launch, nested capture)
├── include/                         # C headers
├── tests/                           # HIP C++ tests + Python integration test
├── benchmarks/                      # Performance benchmarks
├── docs/
│   ├── hipgraph-bridge-design.md    # Full design specification
│   └── torch-hip-rocm-graph.md      # CUDA Graph parity matrix (HIP/Torch/MIGraphX)
├── CMakeLists.txt                   # Native build (requires ROCm SDK)
└── .github/
    ├── workflows/                   # CI
    └── skills/                      # These skill files
```

## Architecture

### Two-Package Design

- **`gfxgraph`** — Public API package. Users `import gfxgraph` and call `gfxgraph.enable()`.
  This package re-exports from `hipgraph_bridge` and manages the monkey-patch lifecycle.
- **`hipgraph_bridge`** — Internal implementation. Contains `BridgedCUDAGraph`, `ShapeBucketPool`,
  `ConditionalGraph`, the native `.so` bindings, and all bridge logic.

This separation means `hipgraph_bridge` can be imported directly by code that needs
specific classes without triggering auto-enable behavior.

### Monkey-Patch System (`_enable.py`)

`gfxgraph.enable()` does:
1. Stashes original `torch.cuda.CUDAGraph` in `_originals["CUDAGraph"]`
2. Replaces `torch.cuda.CUDAGraph` with `BridgedCUDAGraph`
3. Optionally registers the `hipgraph_bridge` torch.compile backend
4. Sets up atexit cleanup handler
5. Optionally enables validation mode (graph-vs-eager comparison)

`gfxgraph.disable()` restores originals. Thread-safe stats via `_stats_lock`.

### BridgedCUDAGraph (`graph_manager.py`)

The core class. Key design decisions:

**Low-Level API Compatibility:**
Methods `capture_begin()`, `capture_end()`, `replay()`, `pool()` delegate directly to a
real `_OriginalCUDAGraph` instance. This makes BridgedCUDAGraph a true drop-in for SGLang's
`cuda_graph_runner.py` which uses the low-level API.

**`_OriginalCUDAGraph` Reference:**
Captured at module load time BEFORE any monkey-patching:
```python
_OriginalCUDAGraph = torch.cuda.CUDAGraph
```
This ensures BridgedCUDAGraph always creates real CUDAGraph instances internally,
even after `torch.cuda.CUDAGraph` has been patched.

**Eager Fallback:**
When graph capture fails, `_eager_fallback = True` is set, and subsequent `replay()` calls
execute `model_fn` eagerly. This is the HIGH-LEVEL API path only. The low-level API
(used by SGLang) does NOT fall back — it raises exceptions that the caller handles.

**SIGABRT Prevention (CRITICAL):**
The `_CaptureContext.__exit__` must properly end HIP stream capture BEFORE dropping the
graph reference. Without this:
1. `self.parent._graph = None` triggers Python GC
2. `_OriginalCUDAGraph.__del__` → `at::cuda::CUDAGraph::~CUDAGraph()`
3. C++ destructor calls `hipStreamEndCapture` on stream still in capture mode
4. C++ exception in destructor → `std::terminate()` → SIGABRT

**Fix pattern (apply to ALL error paths in capture):**
```python
graph = self.parent._graph
if graph is not None:
    try:
        graph.capture_end()
    except Exception:
        pass
    try:
        torch.cuda.synchronize()
    except Exception:
        pass
self.parent._graph = None  # NOW safe
```

### ShapeBucketPool (`shape_bucketing.py`)

- Uses `bisect.bisect_left` to find smallest bucket ≥ input size
- Lazy capture: only captures a bucket graph when first needed
- VRAM safety: checks `torch.cuda.mem_get_info()` vs `GFXGRAPH_VRAM_CAP` before capture
- Per-bucket failure tracking: failed buckets use eager fallback permanently

### ConditionalGraph (`conditional.py`)

Gap 51 bridge. Uses per-branch dispatch (separate CUDAGraph per branch) rather than
`hipGraphNodeSetEnabled` (unreliable on gfx1030 for child graph nodes).

### Native Bridge (`_C.py` + `src/`)

From carlosfundora/gfxGRAPH, MIT.

More from carlosfundora/gfxGRAPH