gpu-capacity-quota-plan

Files (1)

• SKILL.md

SKILL.md (first lines)

# GPU Capacity and Quota Planning

The commands, `jq` filters, and observed API behavior are in the human guide:
[Plan GPU quota and capacity](../../../docs/workbench/gpu-capacity-quota-plan.md).
Follow its sections in this order.

## When To Use

- Before creating or starting a GPU VM directly with the `nebius` CLI, or when a
  launch needs quotas or capacity advice beyond what `npa cluster up` checks.
- When a launch fails and you need to decide between raising quota, waiting,
  moving region, or changing preset.
- When planning a retry loop across regions, and after it, to clean up.

## Procedure

1. **Size the demand** across every quota a GPU VM uses (guide: "Size the
   demand"), including `vpc.allocation.count` and the public IPv4 quota.
2. **Check headroom** in the tenant rows and in any project row with a limit.
   Additional demand must fit both. For a start, do not count the stopped VM's
   existing instance, disks, or retained allocations again; check compute
   resources acquired on start and any new resources. Stopped VMs keep their
   instance and disk quota until deleted.
3. **Read capacity advice** for the exact platform and preset. `available`
   decides; 0 or missing means a launch is unlikely at any availability level.
   Use only `DATA_STATE_FRESH` rows. Check quota usage before treating
   `AVAILABILITY_LEVEL_LIMIT_REACHED` as a hardware shortage. Advice is not a
   guarantee.
4. **Pick a target** within the authorized region, project, fabric, platform,
   preset, and allocation scope when quota and fresh advice both fit. Otherwise
   propose quota (step 5) or alternatives. Obtain approval before launching
   outside that scope. Unavailable or stale advice is unknown; record that
   limitation if the authorized search proceeds without it.
5. **Quota requests** go to a tenant admin through the web console. Hand the user
   the quota name, region, current limit, and requested limit. A
   `quota-allowance create` call only sets a project allowance; it never raises
   tenant quota.
6. **Launch with a run label** and record IDs. A launch succeeds only when the
   operation has finished without an error code and `instance get` reports
   `RUNNING`.
7. **Classify failures from operation details**, never from the status code:
   `QuotaFailure` names the quota to free or raise; `NotEnoughResources` means
   capacity, so retry later or move, and do not raise quota. Auth, image,
   cloud-init, and workload errors are neither.
8. **Retry within the authorized scope**: one region at a time, reuse stopped
   VMs with start-specific quota accounting, and stop at the first confirmed
   `RUNNING` VM. Follow the operator's cadence and explicit limits; do not add
   time, cost, or attempt limits when none were requested.
9. **Clean up unused attempts by ID** under the cleanup authorization already
   given. Keep the successful VM and its resources. Verify ownership against
   the attempt inventory; ask only about IDs outside the authorized scope or
   unclear ownership. Preserve pre-existing or shared disks and allocations
   unless explicitly authorized for deletion. Continue after individual cleanup
   failures, report residue, and verify deleted resources return NotFound.

Switching from on-demand to preemptible goes through
`skills/atomic/gpu-allocation-fallback/SKILL.md` and needs the user's consent.
Clean up NPA-managed resources with `skills/atomic/teardown-and-cost/SKILL.md`.

## Gotchas

- `AVAILABILITY_LEVEL_LIMIT_REACHED` also appears when your quota is used up.
- JSON output omits zero values: no `available` field means zero VMs.
- A create that fails can leave a stopped VM behind; find it by its run label.
- A stopped VM still holds instance and disk quota, and its disks keep billing.
- Never delete by name or outside the run's authorized cleanup scope.

## Evidence Contract

Retain the snapshot time, commands, platform, preset, region, fabric,
allocation type, quota rows, advice rows, operation IDs with their detail codes,
and cleanup results. Redact project, tenant, instance, disk, allocation, IP, and
credential values before placing evidence in a repository.

## Verify

```bash
npa/.venv/bin/python -m pytest npa/tests/guardrails/test_skills_index.py \
  npa/tests/guardrails/test_documentation_examples.py -q
```

From nebius/nebius-physical-ai, Apache-2.0.

More from nebius/nebius-physical-ai