pyqcu-cuda

Files (2)

• AGENTS.md
• SKILL.md

SKILL.md (first lines)

# pyqcu.cuda

Cython bridge package for the C++ CUDA backend (`libqcu.so`).

## Files

| File | Purpose |
|------|---------|
| `__init__.py` | Makes `pyqcu.cuda` a proper Python package (added 2026-07-28 R3; was missing, causing `pip install` failures) |
| `qcu/qcu.pyx` | Cython extension source — wraps C functions from `pyqcu.h` as `applyInitQcu`, `applyWilsonDslashQcu`, etc. |
| `qcu/qcu_api.pxd` | Cython declarations — aliases C symbols, marks calls `nogil`, and must match `pyqcu.h` |
| `qcu/qcu.pyi` | Type stub (390 lines) — full type annotations, docstrings, and default values for IDE support。注意 `pyqcu/cuda/qcu.pyi` 是指向 `qcu/qcu.pyi` 的符号链接，不是第二份拷贝 |
| `define.py` | Parameter constants (`_LAT_X_`, `_SET_PLAN_`, etc.) and dtype conversion helpers (`dtype()`, `epytd()`) |
| `_runtime.py` | `QcuRuntime` — lifecycle-safe flat-ABI wrapper with private control arrays, free-slot selection, `mass`/`csw` configuration, and `operation()` contexts for getters |
| `linear_operator.py` | `CudaLinearOperator` — compact Wilson/Clover Schur apply and native whole-solve adapter |
| `krylov.py` | `CudaBiCGStabDriver` / `CudaCGDriver` — one native C++ solve per request |
| `_schur_op.py` | `CudaSchurOp` — C++ Schur odd-site operator wrapper (per-instance independent params/set_ptrs, thread-safe slot allocator; matvec uses preallocated buffer + device sync + clone to avoid deep pool/stream races) |
| `_multi_gpu.py` | **TEST-ONLY, explicit opt-in**: `MultiGpuMultigrid` — single-process multi-thread multi-GPU Legacy MG driver (one-thread-one-GPU; consistency + independent-problem modes; single-MPI-rank constraint); not the production topology and not a single-solve domain decomposition |
| `_strict_multigrid.py` | `StrictMultigrid` — fine odd-Clover Schur + persistent strict coarse V-cycle + C++ fused right-FGMRES；通过 `pyqcu.cuda.multigrid` 导出 |
| `_strict_cache.py` | strict runtime cache schema v2；逐 tensor 流式摘要、完整验签后再传入 device |
| `_strict_mpi.py` | strict MPI rank 对称 preflight 与能力门禁；当前声明 setup/full/compact/R-P halo、全局归约和分布式 fused FGMRES 均可用 |
| `_logs.py` | C++ 后端收敛日志诊断 API：`parse_mg_log(path)` → (残差列表, `PROF_SECTIONS` ms 字典, 总迭代数)；`parse_convergence_histories(path, offset)` → (histories, 新偏移) 增量收集逐次求解的 `CONVERGENCE_HISTORY`。2026-10-22 审计基线下仓内没有代码或测试导入该模块；补定向解析测试前，不应把“模块存在”写成已验证的规范入口。默认路径 `<repo>/logs/clover_multigrid.log` |

## CUDA Production Topology

All new production CUDA programs follow one invariant:

```text
one Python process = one MPI rank = one GPU/DCU/GPGPU
```

Cross-device parallelism is implemented with MPI ranks and 4D lattice domain
decomposition. Do not create multiple production device workers inside one
Python process. `MultiGpuMultigrid` is a test-only exception: importing
`pyqcu`, `pyqcu.cuda`, or `pyqcu.solver` does not construct or start it; only an
explicit import and construction enable the path. Use it only for consistency,
throughput, or diagnostic tests, never as the default driver for a new CUDA
program.

Each production process owns one rank-local `LatticeSet`/solver context.
`CUDA_VISIBLE_DEVICES` and device selection must be fixed before importing or
initializing PyTorch, the Cython bridge, or native QCU. MPI rank count should
match participating devices; oversubscribed ranks are correctness smoke only,
not an acceleration path.

Multi-rank benchmark entry points accept only positive powers of two and must
reject other counts before MPI/solver startup. Two P100 ranks require a Python
environment compatible with `sm_60` and a native library set covering
`sm_60`/`sm_70`; V100+P100 mixed-device timing is not a valid
same-architecture result.

The normative project document is
[`docs/cuda_execution_topology.md`](../../docs/cuda_execution_topology.md).
The cross-document extraction index is
[`pyqcu-docs/references/extracted-guidance.md`](../pyqcu-docs/references/extracted-guidance.md#3-cuda-生产执行拓扑).
The 2026-10-23 multi-rank and MultiGrid benchmark constraints are in
[`pyqcu-benchmark/references/qcu-multigrid-20261023.md`](../pyqcu-benchmark/references/qcu-multigrid-20261023.md).

DCU/DTK builds use the same Cython and native ABI but a separate build profile:
`PYQCU_BUILD_BACKEND=dcu` with `install.sh dcu` selects
`cpp/cuda/qcu/make-dcu.sh` and `CMakeLists-dcu.txt`. The Python runtime still
uses ROCm PyTorch's CUDA-compatible API (`backend="cuda"`, `device="cuda:N"`).
`cpp/dtk` and `pyqcu/dtk` are placeholders. See
[`docs/dcu_build_backend.md`](../../docs/dcu_build_backend.md).

`source ./env.sh` exports the current checkout as `PYQCU_ROOT` and defaults
`PYQCU_CUDA_LIB`, `PYQCU_DCU_LIB`, and `PYQCU_KUPL_LIB` to their canonical library
paths. DCU builds write `${QCU_DCU_BUILD_DIR}/libqcu.so` and copy it to
`cpp/cuda/qcu/libqcu.so` for the Cython bridge. Explicit caller values take precedence.

## Public API

跨 CUDA/KUPL/Torch 的公共 PyQCU 接口规范以
[`pyqcu-api/SKILL.md`](../pyqcu-api/SKILL.md#三后端统一接口强制规范) 为唯一权威源。
`CudaLinearOperator`、`CudaBiCGStabDriver`、CUDA `Multigrid` 等属于后端适配层；
用户可见的高层求解、MG 与 cache 调用必须使用 `pyqcu.solver` 的统一选择器。新增 CUDA
入口时保持公共参数名、`SolveResult`/`MultigridCacheLoadResult` 返回类型和生命周期一致，
不支持的组合通过 `BackendCapabilities` 或构造期错误 fail-closed。

已有 CUDA 库、Strict runtime cache 或 Legacy 33-tensor cache 在 SHA256、gauge、几何、
精度和算法身份一致时直接复用，不重复 setup。缓存命中仍需完整验签；只有 miss、损坏或
身份变化时重建。恢复增量构建时优先沿用现有 build tree，不清理可复用对象。

```python
from pyqcu.cuda import qcu      # Cython bridge to libqcu.so
from pyqcu.cuda import define   # Parameter constants, dtype helpers, pre-built params/argv/set_ptrs tensors
from pyqcu.cuda import QcuRuntime  # lifecycle-safe flat-ABI calls
```

## Cython Extension — C Functions Exposed

| Function | Purpose | Plan |
|----------|---------|------|
| `applyInitQcu` / `applyEndQcu` | Allocate / free scratch buffers | — |
| `applyWilsonDslashQcu` | Wilson dslash | 0 |
| `applyCloverDslashQcu` | Clover dslash | 2 |
| `applyWilsonBistabCgQcu` / `applyWilsonBistabCgDslashQcu` | Wilson BiStabCG solver + its dslash | 1 |
| `applyWilsonCgQcu` / `applyWilsonCgDslashQcu` | Wilson CG solver + its dslash | 1 |
| `applyCloverBistabCgQcu` / `applyCloverBistabCgDslashQcu` | Clover BiStabCG (needs clover_ee/oo + inverses) | 1 |
| `applyCloverQcu` / `applyCloversQcu` | Build Clover term (and its inverse) | 2 |
| `applyDslashQcu` | Combined Wilson+Clover dslash | 0+2 |
| `applyLaplacianQcu` | Laplacian operator | -2 |
| `applyGaussGaugeQcu` | Gaussian gauge field generation | -1 |
| `applyMultigridRestrictQcu` / `applyMultigridProLongQcu` | MG restrict/prolong with null vectors | MG |
| `applyMultigridCoarseDslashQcu` | Coarse-grid dslash (hopping + sitting) | MG |
| `applyCloverMultigridQcu` | Full Clover multigrid V-cycle solver | MG |
| `applyMultigridStrictCoarseQcu` / `applyMultigridStrictMatPCQcu` | Full-coarse `X/Yhat` operator and compact-parity MATPC | strict MG |
| `applyMultigridStrictPrepareQcu` / `applyMultigridStrictReconstructQcu` | Full↔odd-Schur preparation/reconstruction | strict MG |
| `applyMultigridStrictRestrictQcu` / `applyMultigridStrictProLongQcu` | Compact fine-parity R/P with full coarse output/input | strict MG |
| `applyMultigridStrictInitQcu` / `applyMultigridStrictVCycleQcu` / `applyMultigridStrictEndQcu` | Allocate, reuse, and release the persistent strict hierarchy/arena | strict MG |
| `applyMultigridStrictFgmresQcu` | Fused restarted right-FGMRES with MPI global reductions, including warm x0 and exact workspace accounting | strict MG |

All functions take raw pointers cast to `long long` from `tensor.contiguous().data_ptr()`.

## Parameter Protocol

Three flat tensors bridge Python ↔ C++:

- **`params`** (CPU int32[58]) — lattice dims (`_LAT_X_`…`_LAT_XYZT_`), grid sizes (`_GRID_X_`…), data types (`_DATA_TYPE_`), iteration counts (`_MAX_ITER_`), plan selection (`_SET_PLAN_`), verbosity (`_VERBOSE_`), parity (`_PARITY_`), multigrid level configs and controls. The tail is `_MG_USE_GCR_` (54), `_MG_USE_DEFLATE_` (55), `_MG_MU_PRE_` (56), and `_MG_USE_INIT_GUESS_` (57); slot 57 is a 0/1 warm-start flag and does not extend the ABI.
- Multi-rank CUDA Legacy must set `params[_MG_USE_DEFLATE_]` explicitly. Leaving it unset can produce `n_vcycles=0`, which is a fine-only result and must not be reported as coarse MultiGrid performance. Verify actual `n_vcycles`/coarse iteration and correction counters instead of inferring execution from `levels` or `restart`.

From ZhangXin8069/PyQCU, MIT.

More from ZhangXin8069/PyQCU