cosmos3-ray-serve

Files (2)

• SKILL.md
• openai.yaml · agents/

SKILL.md (first lines)

# Cosmos3 Native Ray Serve

Use this service for repeated or batched synthetic-data generation where loading
Cosmos3-Nano once is materially better than starting one `cosmos3 generate` job
per sample. Do not substitute `npa-cosmos3-serving`: that image serves
Cosmos3-Super through vLLM-Omni and has a different model/API/runtime contract.

> **Current release status:** Cosmos3 Ray Serve has no currently qualified image.
> The retained historical bytes are quarantined, excluded from public/default
> selection, and must not be started or described as a release. The explicit
> development-tag resolver form is for a separately rebuilt validation candidate,
> not an authorization for the retained tag. Historical tags, digests, and
> B200/RTX/H200 observations do not qualify a replacement; wait for a separately
> rebuilt and qualified candidate.

## Non-negotiable contract

- Run NVIDIA cosmos-framework at pinned commit
  `5e67049cd94acb667786f1e6dd0dab821cb90c97`.
- Bind upstream `OmniModelDeployment`; its `@ray.serve.batch` method must own
  coalescing and call `OmniInference.generate_batch`.
- Keep guardrails on unless the operator explicitly opts out.
- Fetch Cosmos3-Nano, VAE, and guardrail weights only at runtime with the
  operator's access. Use the standard NPA model-cache mount; never bake caches.
- Require `NPA_COSMOS3_RAY_TOKEN` for every API endpoint.
- Move batch inputs and outputs through S3. Never transfer data directly from a
  sibling workbench service.

## Preflight

Before provisioning or starting GPUs:

```bash
npa/.venv/bin/npa workbench health preflight --checks hf,ngc,s3 --json
npa/.venv/bin/npa workbench health access --capability cosmos3 --json
npa/.venv/bin/npa workbench golden-eval show cosmos3-ray-serve
```

Treat S3 failure, missing `Cosmos-Guardrail1` access, or an unpullable exact
image digest as a stop condition. A token's presence is not model entitlement.

## Start the service

**Do not run this procedure against the retained historical image.** It is
quarantined and has no current public/default image selection. Once a separately
rebuilt candidate has passed exact-byte security and capability qualification,
follow its explicit operator-approved image binding: mount `/outputs` and the
standard model cache, and inject `HF_TOKEN` and `NPA_COSMOS3_RAY_TOKEN` as
runtime secrets. That candidate's entrypoint starts:

```text
npa workbench cosmos3 ray-serve --world-size 1 --max-batch-size 4
```

Configuration is explicit: `--world-size` sets GPUs per replica;
`--max-batch-size` and `--batch-wait-timeout-s` are upstream batching knobs;
the service sets Ray's model-replica request admission capacity to the configured
maximum batch size so batches above Ray 2.58's default capacity can form;
`--parallelism-preset` is the Cosmos placement preset; and
`--guardrails/--no-guardrails` is the explicit safety posture.

Guarded startup reuses the pinned Cosmos tokenizer materializer before Ray/NLTK
imports. Keep its verified regular-file `NLTK_DATA` cache at runtime; never bake
it or disable NLTK path enforcement to permit Hub snapshot symlinks. Missing
entitlement or invalid cache content must refuse startup. The shared materializer
is the source of truth for the Guardrail1 revision and cache validation.

Use authenticated `GET /ready`, and verify that the exact image checks the native
Serve application and model replicas. Earlier implementations returned HTTP 200
while weights were still loading. Require the selected application to be
`RUNNING`, its model deployment `HEALTHY`, and a running replica before inference.
Retain checkpoint revision and payload verification separately; cache-resolution
events or filenames alone do not prove that every required payload is complete.
`GET /health` establishes liveness. Report configured guardrails separately from
evidence that the selected model actually applied them.

## Submit a durable batch

The input is JSON with one or more upstream `OmniSampleOverrides` objects:

```json
{"model":"Cosmos3-Nano","samples":[
  {"name":"sample-a","model_mode":"text2image","prompt":"a robot workcell","seed":17},
  {"name":"sample-b","model_mode":"text2image","prompt":"a warehouse aisle","seed":23}
]}
```

Submit it with the CPU client:

```bash
npa/.venv/bin/npa workbench cosmos3 ray-batch \
  --input-path s3://<bucket>/<prefix>/batch.json \
  --output-path s3://<bucket>/<prefix>/outputs/ \
  --endpoint http://<service>:8000
```

The client sends all samples concurrently so upstream Ray Serve can coalesce
them. It downloads each returned file, verifies bytes and SHA-256, and publishes
`request.json`, `response.json`, media under `artifacts/`, and
`provenance.json` (`npa.cosmos3.ray-serve.provenance.v1`). Use
`workflows/testing/cosmos3-ray-batch.yaml` for the workflow
client; submit only to a separately rebuilt and qualified candidate service
that is ready.

The checks in this paragraph require the updated installed client (for example,
the checkout's editable `npa/.venv` installation). The reference workflow's
retained quarantined historical service image bundles the older client and does
not enable a source overlay, so it must not be used to perform these checks.
Updating NPA on the submission host does not update that packaged client.
The retained quarantined historical digest remains wire-compatible with the
updated client, but does not gain the current source's Ray management
authentication, scoped S3 input staging, or Ray 2.58/Torch 2.13 runtime
changes. Revalidate the actual service runtime when those integration boundaries
change; historical image evidence does not exercise replacement source or
runtime dependencies.

The updated client binds the supported schema, model, request ID and every requested
sample name before downloading. An omitted request ID is generated before POST
and retained in `request.json`. Require one successful native `SampleOutputs`

From nebius/nebius-physical-ai, Apache-2.0.

More from nebius/nebius-physical-ai