nemotron-experiment-submit

Files (1)

• SKILL.md

SKILL.md (first lines)

# RL on NemotronLabsDiffusion Experiment Submission

Use this skill when submitting or debugging NeMo-RL training experiments from the current RL repo on dfw.

## Working Directory

Run from the active RL checkout:

```bash
cd ~/diffusion_RL/RL
```

Submit through the repo's Nemotron diffusion sbatch wrapper:

```bash
tools/nemotron_diffusion/submit_grpo_nemotron_ar_megatron_sbatch.sh
```

The wrapper name contains `ar`, but it is also used for JustGRPO and DiffuGRPO configs unless the repo introduces a newer algorithm-specific wrapper. The selected algorithm comes from `CONFIG=...` and any `EXTRA_CONFIG_OVERRIDES`, not from the script name.

Always pass the intended config file explicitly with `CONFIG=...`; do not rely on the script default when launching a named experiment.

## Standard Submit Command

Present submissions as one command with inline environment assignments, as below,
rather than a sequence of persistent `export` commands.

```bash
RUN_NAME=<run_name> \
WANDB_RUN_NAME=<run_name> \
JOB_NAME=<short_slurm_name> \
PARTITION=batch \
TIME=04:00:00 \
NODES=1 \
ENV_TAG=mb_3rdparty_sglagn_local_fork \
CONFIG=examples/configs/<config_file>.yaml \
NRL_FORCE_REBUILD_VENVS=false \
FORCE_REINSTALL_PACKAGES=false \
FORCE_REINSTALL_SGLANG=false \
bash tools/nemotron_diffusion/submit_grpo_nemotron_ar_megatron_sbatch.sh --sbatch
```

After submission, explicitly share the exact submission command with the user together with the Slurm job id.

Use PARTITION=batch for all sbatch submissions. The dfw batch partition has a 4-hour limit, so keep TIME at or below 04:00:00. Adjust NODES as needed, but do not submit these experiments to backfill, batch_long, batch_large, or batch_large_long.

## vLLM Diffusion Backend (async-GRPO + NeMo-Gym)

The default backend above is SGLang. There is a second, separate path that runs
the dLLM over a **custom vLLM fork** with non-colocated async-GRPO and NeMo-Gym
data routing -- this is what the v30 deterministic-agent runs use.

Configs live under `examples/configs/*_vllm_*` (and
`examples/configs/grpo_sudoku6x6_*_vllm*` for the sudoku smokes). The production
v30 config is:

- main: `CONFIG=examples/configs/nemotron_labs_diffusion_8b_vllm_block_just_grpo_async_grpo_nemogym_v30.yaml`
- toy:  `CONFIG=examples/configs/nemotron_labs_diffusion_8b_vllm_block_just_grpo_async_grpo_nemogym_v30_toy.yaml`

The v30 config bakes in its own overrides (8B checkpoint, node split, grpo
sizes, `enforce_eager`, `sequence_parallel`, etc.); read its header comment for
the current values rather than re-passing them on the command line.

### What differs from an SGLang submit

- **Entry script**: `RUN_SCRIPT=examples/nemo_gym/run_grpo_nemo_gym.py` (not the
  default `examples/run_grpo.py`).
- **uv extras**: `UV_EXTRAS="mcore nemo_gym"` (adds the `nemo_gym`
  optional-dependency group on top of `mcore`).
- **NeMo-Gym server venv**: `NEMO_GYM_VENV_DIR=/lustre/fsw/portfolios/coreai/users/snorouzi/gym_venvs`
  (pre-provisioned; the gym agent/server runs from it).
- **vLLM runtime**: `NRL_VLLM_PY_EXECUTABLE=<vllm-runtime-venv>/bin/python-compat`
  -- see below.
- **ENV_TAG**: a vLLM/nemo_gym driver-env tag, distinct from the SGLang
  `mb_3rdparty_sglagn_local_fork`, e.g. `mb_3rdparty_vllm_local_fork_fixed`. The
  driver env must carry the `nemo_gym` extra; because `--no-sync` will not
  add it, provision the tag once with `--build-env` and the intended
  `UV_EXTRAS` (see "Provisioning a New ENV_TAG" below) before normal reuse.

### Datasets requiring NeMo-Gym / NeMo-Skills tools (AR GRPO on ns_tools)

For NeMo-RL datasets routed through NeMo-Gym (`env.should_use_nemo_gym: true`),
use the Gym entrypoint, `mcore nemo_gym` extras, and the Gym/vLLM environments
shown below. The standard SGLang submission above does not configure this path.
For `ns_tools` and other tasks executing Python through the NeMo-Skills sandbox,
also enable the sandbox and its lifecycle mounts. Gym-only tasks without sandbox
tools do not require the `NEMO_SKILLS_SANDBOX*` settings.

The command below is the AR GRPO ns_tools production recipe: LR 1e-6, batch 1920,
16 nodes (8 training + 8 generation), verifier isolation and session cleanup.
Use `TIME=04:00:00` for normal production runs. The YAML sets
`checkpointing.checkpoint_must_save_by: "00:03:45:00"` (DD:HH:MM:SS), replacing
the old two-hour debug stop and leaving margin for saving and shutdown. Changing
Slurm `TIME` alone does not remove a shorter checkpoint deadline in the YAML.

Run from the active RL checkout. Select `ACCOUNT` using the fair-share procedure
below; `nvr_lpr_llm` here is an example, not a permanent account preference.

```bash
RUN_NAME=ar_grpo_ns_lr1e6 \
WANDB_RUN_NAME=ar_grpo_ns_lr1e6 \
JOB_NAME=ar_grpo_ns_lr1e6 \
ACCOUNT=nvr_lpr_llm PARTITION=batch TIME=04:00:00 NODES=16 \
ENV_TAG=mb_3rdparty_vllm_local_fork_fixed \
CONFIG=examples/configs/ar_grpo_ns.yaml \
RUN_SCRIPT=examples/nemo_gym/run_grpo_nemo_gym.py \
UV_EXTRAS="mcore nemo_gym" \
NEMO_GYM_VENV_DIR=/lustre/fsw/portfolios/coreai/users/snorouzi/gym_venvs \
NRL_VLLM_PY_EXECUTABLE=/lustre/fsw/portfolios/coreai/users/snorouzi/vllm_runtimes/nemotron_dllm_792ab07/bin/python-compat \
NEMO_SKILLS_SANDBOX=1 \
NEMO_SKILLS_SANDBOX_SERVER_FILE=/lustre/fsw/portfolios/coreai/users/snorouzi/ns_tools_lifecycle_0915/main.py \
NEMO_SKILLS_SANDBOX_LAUNCHER_FILE=/lustre/fsw/portfolios/coreai/users/snorouzi/ns_tools_lifecycle_0915/start-with-nginx-fixed.sh \
NEMO_SKILLS_SANDBOX_SESSION_TIMEOUT=3600 \
NEMO_SKILLS_SANDBOX_BLAS_THREADS=1 \
EXPANDABLE_SEGMENTS=false \
NRL_FORCE_REBUILD_VENVS=false \
FORCE_REINSTALL_PACKAGES=false \
FORCE_REINSTALL_SGLANG=false \
bash tools/nemotron_diffusion/submit_grpo_nemotron_ar_megatron_sbatch.sh --sbatch

From sajadn/diffusion_RL, Apache-2.0.

More from sajadn/diffusion_RL