rtx-pro-6000

Files (39)

• .dockerignore
• .env.example
• .gitattributes
• ci.yml · .github/workflows/
• .gitignore
• CHANGELOG.md
• CONTRIBUTING.md
• LICENSE
• PUBLISHING.md
• README.md
• SECURITY.md
• SKILL.md
• VALIDATION.md
• VERSION
• openai.yaml · agents/
• compose.yaml
• Dockerfile.ds4 · docker/
• manifest.json
• architecture.md · references/
• client-integration.md · references/
• model-profiles.md · references/
• troubleshooting.md · references/
• tuning-blackwell.md · references/
• activate-profile.sh · scripts/
• bootstrap.sh · scripts/
• common.sh · scripts/
• download-ds4-model.sh · scripts/
• healthcheck.sh · scripts/
• install-runtime.sh · scripts/
• install-skill.sh · scripts/
• integrate-client.sh · scripts/
• llmctl.sh · scripts/
• merge_opencode.py · scripts/
• package.sh · scripts/
• prefetch-vllm-model.sh · scripts/
• probe-host.sh · scripts/
• README.md · templates/
• qwen36_fixed_chat_template.jinja · templates/
• test-static.sh · tests/

SKILL.md (first lines)

# RTX PRO 6000 Local LLM

Install and operate the Dockerized local inference stack in this skill. The
stack contains mutually exclusive vLLM model profiles and a CUDA-native DS4
container. It can configure Codex CLI, Claude Code, and OpenCode without
replacing unrelated user configuration.

## Non-negotiable safety rules

- Run `scripts/probe-host.sh --strict` before changing the host.
- Ask before installing system packages, downloading model weights, changing a
  client configuration, or exposing an endpoint beyond localhost.
- Never start two heavyweight profiles at the same time on one GPU. Use
  `scripts/activate-profile.sh`, which stops only this stack's running profile.
- Never run `docker compose down -v`, delete model data, prune Docker globally,
  or remove an existing client configuration.
- Keep `BIND_ADDRESS=127.0.0.1` unless the user explicitly requests remote
  access and understands that these APIs do not provide production-grade auth.
- Do not put Hugging Face, Anthropic, OpenAI, or other secrets in tracked files.
- Do not substitute `latest` image tags. Update pinned versions deliberately and
  rerun the complete smoke test.
- Treat context sizes as capacity limits, not quality targets. Start with the
  profile defaults and increase only after checking RAM, VRAM, and latency.

## Standard workflow

From the skill directory:

```bash
scripts/probe-host.sh --strict
scripts/bootstrap.sh
```

If Docker or NVIDIA Container Toolkit is missing, explain the changes and ask
before running:

```bash
scripts/install-runtime.sh --yes --allow-docker-restart
```

For DS4, ask before the approximately 81 GB download:

```bash
scripts/download-ds4-model.sh q2-imatrix
scripts/activate-profile.sh ds4
```

For vLLM, select one profile and activate it. A download-only container fetches
the pinned Hugging Face snapshot without executing model code; inference then
runs offline without receiving the token. Downloads can be large:

```bash
scripts/activate-profile.sh qwen-prisma
scripts/activate-profile.sh qwen35
scripts/activate-profile.sh qwen27
scripts/activate-profile.sh kimi
```

Inspect state and logs with:

```bash
scripts/llmctl.sh status
scripts/llmctl.sh logs ds4
scripts/healthcheck.sh ds4 --full
```

Stop the current stack profile without deleting data:

```bash
scripts/llmctl.sh stop
```

## Profile selection

Read `references/model-profiles.md` before recommending a profile.

- `ds4`: DeepSeek V4 Flash GGUF, strongest direct compatibility with Codex and
  Claude Code, serialized inference, disk-backed KV reuse.
- `qwen-prisma`: Blackwell NVFP4/BF16, long-context multimodal and tool use;
  specialized configuration with a pinned chat template.
- `qwen35`: general-purpose MoE FP8 profile.
- `qwen27`: dense multimodal FP8 profile.
- `kimi`: text-only MoE FP8 profile with linear long context.

Profiles are a tested baseline, not an evergreen claim that a model remains
the best. Before changing a model ID, image, quantization, parser, or chat
template, verify upstream compatibility and benchmark the replacement.

## Client integration

Generate or apply integration with:

```bash
scripts/integrate-client.sh --client codex --profile ds4
scripts/integrate-client.sh --client claude --profile ds4
scripts/integrate-client.sh --client opencode --profile ds4
scripts/integrate-client.sh --client all --profile ds4
```

Without `--apply`, the command only writes reviewed snippets under the runtime
configuration directory. With `--apply`, it creates timestamped backups first.

- Codex should use DS4's native Responses API. vLLM profiles may be configured
  only after `scripts/healthcheck.sh PROFILE --responses` succeeds.
- Claude Code should use DS4's native Anthropic Messages endpoint through the
  generated `claude-ds4` wrapper. Do not globally replace Anthropic variables.
- OpenCode can use every profile through an OpenAI-compatible provider. Restart
  OpenCode after applying configuration because it does not hot-reload config.

Read `references/client-integration.md` for exact formats and limitations.

## Verification requirements

Do not describe an installation as complete until all applicable checks pass:

1. `docker compose config` validates.
2. The selected container becomes healthy.
3. `GET /v1/models` returns the expected model.
4. A non-streaming chat request succeeds.

From ruota/open-sinapseed, MIT.

More from ruota/open-sinapseed