slurm-job-script-generator

Files (5)

• CHANGELOG.md
• SKILL.md
• evals.json · evals/
• slurm_directives.md · references/
• slurm_script_generator.py · scripts/

SKILL.md (first lines)

# SLURM Job Script Generator

## Goal

Generate a correct, copy-pasteable SLURM job script (`.sbatch`) for running a simulation, and surface common configuration mistakes (bad walltime format, conflicting memory flags, oversubscription hints).

## Requirements

- Python 3.10+
- No external dependencies (Python standard library only)
- Works on Linux, macOS, and Windows (script generation only)

## Inputs to Gather

| Input | Description | Example |
|-------|-------------|---------|
| Job name | Short identifier for the job | `phasefield-strong-scaling` |
| Walltime | SLURM time limit | `00:30:00` |
| Partition | Cluster partition/queue (if required) | `compute` |
| Account | Project/account (if required) | `matsim` |
| Nodes | Number of nodes to allocate | `2` |
| MPI tasks | Total tasks, or tasks per node | `128` or `64` per node |
| Threads | CPUs per task (OpenMP threads) | `2` |
| Memory | `--mem` or `--mem-per-cpu` (cluster policy dependent) | `32G` |
| GPUs | GPUs per node (optional) | `4` |
| Working directory | Where the run should execute | `$SLURM_SUBMIT_DIR` |
| Modules | Environment modules to load (optional) | `gcc/12`, `openmpi/4.1` |
| Run command | The command to launch under SLURM | `./simulate --config cfg.json` |

## Decision Guidance

### MPI vs MPI+OpenMP layout

```
Does the code use OpenMP / threading?
├── NO  → Use MPI-only: cpus-per-task=1
└── YES → Use hybrid: set cpus-per-task = threads per MPI rank
          and export OMP_NUM_THREADS = cpus-per-task
```

**Rule of thumb:** if you see diminishing strong-scaling efficiency at high MPI ranks, try fewer ranks with more threads per rank (and measure).

### Memory flag selection

- Use **either** `--mem` (per node) **or** `--mem-per-cpu` (per CPU), not both.
- Follow your cluster’s documentation; some sites enforce one style.
- SLURM `--mem` units are integer MB by default, or an integer with suffix `K/M/G/T` (and `--mem=0` commonly means “all memory on node”).

### Launcher selection (avoid double-wrapping)

- By default (`--launcher srun`) the generator prepends `srun --ntasks=N --cpus-per-task=T` to your run command so it inherits SLURM task placement.
- **If your run command already starts with `srun`, `mpirun`, or `mpiexec`, pass `--launcher none`** so the generator does not double-wrap it. Wrapping `srun` around `mpirun` launches N independent copies of `mpirun` (each spawning its own MPI world); wrapping `srun` around `srun` is malformed.
- As a safety net, the generator auto-detects a leading launcher (`srun`, `mpirun`, `mpiexec`, `mpiexec.hydra`, `orterun`, `aprun`, `jsrun`) and falls back to no-wrap, emitting a warning that recommends `--launcher none`.

### GPU layout

- Ranks-per-GPU = `total_ranks / (nodes * gpus_per_node)`. When `--gpus-per-node` is set the generator reports `results.derived.total_gpus` and `results.derived.ranks_per_gpu`.
- Aim to map ranks evenly to devices. If `ntasks` is **not** divisible by the total number of GPUs, the generator emits a "task-to-GPU ratio is not an integer" warning; either adjust `ntasks`/GPUs or document intentional sharing (e.g. NVIDIA MPS).
- Consider binding options such as `--gpu-bind=closest` or `--ntasks-per-gpu` (passed via your run command / `--srun-extra`). GPU and QoS policies are site-specific — confirm with your cluster docs.

## Script Outputs (JSON Fields)

| Script | Key Outputs |
|--------|-------------|
| `scripts/slurm_script_generator.py` | `results.script`, `results.directives`, `results.derived`, `results.warnings`, `results.run_line` |

`results.derived` reports `ntasks`, `ntasks_per_node`, `cpus_total_requested`, and (when applicable) `cores_per_node`, `cpus_per_node_requested`, `total_gpus`, and `ranks_per_gpu`. `results.warnings` may include CPU oversubscription, task-to-GPU ratio, and double-launcher warnings.

## Workflow

1. Gather cluster constraints (partition/account, GPU policy, memory policy).
2. Choose a process layout (MPI-only vs hybrid MPI+OpenMP).
3. Generate the script with `slurm_script_generator.py`.
4. Inspect warnings (conflicts, suspicious layouts).
5. Save the generated script as `job.sbatch`.
6. Submit with `sbatch job.sbatch` and monitor with `squeue`.

## CLI Examples

```bash
# Preview a job script (prints to stdout)
python3 skills/hpc-deployment/slurm-job-script-generator/scripts/slurm_script_generator.py \
  --job-name phasefield \
  --time 00:10:00 \
  --partition compute \
  --nodes 1 \
  --ntasks-per-node 8 \
  --cpus-per-task 2 \
  --mem 16G \
  --module gcc/12 \
  --module openmpi/4.1 \
  -- \
  ./simulate --config config.json

# Write to a file and also emit structured JSON
python3 skills/hpc-deployment/slurm-job-script-generator/scripts/slurm_script_generator.py \
  --job-name phasefield \
  --time 00:10:00 \
  --nodes 1 \
  --ntasks 16 \
  --cpus-per-task 1 \
  --out job.sbatch \
  --json \
  -- \
  /bin/echo hello
```

## Conversational Workflow Example

**User**: I need an `sbatch` script for my MPI simulation. I want 2 nodes, 64 ranks per node, 2 OpenMP threads per rank, and 2 hours.

**Agent workflow**:
1. Confirm partition/account and whether GPUs are needed.
2. Generate a hybrid job script:
   ```bash
   python3 scripts/slurm_script_generator.py --job-name run --time 02:00:00 --nodes 2 --ntasks-per-node 64 --cpus-per-task 2 -- ./simulate
   ```
3. Explain the mapping:
   - Total ranks = 128

From HeshamFS/materials-simulation-skills, Apache-2.0.

More from HeshamFS/materials-simulation-skills