Inference AIops

[WRITE][risk=medium] Hot-load a LoRA adapter onto the running engine. Args: lora_name: Adapter name to register. lora_path: Local path or HF repo id of the adapter weights. target: Inference target name from config; omit for the default.

[WRITE][risk=high] Hot-unload a LoRA adapter (traffic on it starts failing). Pass dry_run=True to preview. Args: lora_name: Adapter name to unload (from model_list). dry_run: If True, preview without unloading. target: Inference target name from config; omit for the default.

[WRITE][risk=medium] Deploy a Serve application from an import path. The app is merged into the cluster's declarative Serve config (existing apps preserved). Replica counts come from the app's own config; set them afterward with serve scale (deployment names are known only once the app materialises). Args: application: Serve application name to create/replace. import_path: Python import path of the Serve app (e.g. 'module:app'). target: Inference target name from config; omit for the default.

[READ] One model's config: max length, root/parent, permission (best-effort). Args: model_id: Model id (from model_list). target: Inference target name from config; omit for the default.

[READ] Whether the vLLM engine is suspended in Sleep Mode. Returns isSleeping: true (suspended, serving nothing), false (serving), or null when the engine did not report it — null means UNKNOWN, not awake. Sleep Mode exists only on servers started with VLLM_SERVER_DEV_MODE=1; on any other server this reports that the route is absent rather than failing vaguely. Args: target: Inference target name from config; omit for the default.

[READ] All served vLLM models, LoRA adapters flagged. Args: target: Inference target name from config; omit for the default.

[WRITE][risk=high] Suspend the engine via Sleep Mode (it stops serving requests). Frees GPU memory between bursts. level=1 offloads the weights to CPU RAM and wakes fast; level=2 discards them, so waking reloads from disk. The engine serves nothing until model_wake. Pass dry_run=True to preview. Sleep Mode exists only on servers started with VLLM_SERVER_DEV_MODE=1; on any other server this reports that the route is absent rather than failing vaguely. Args: level: 1 to offload weights to CPU RAM, 2 to discard them (default 1). dry_run: If True, preview without suspending. target:…

[WRITE][risk=high] Tear down a whole Serve application (removes all deployments). Irreversible without the original import path — pass dry_run=True to preview. Args: application: Serve application name (from serve_deployment_list). dry_run: If True, preview without undeploying. target: Inference target name from config; omit for the default.

[WRITE][risk=medium] Resume serving after Sleep Mode (weights return to GPU). The inverse of model_sleep. Pass dry_run=True to preview. Records no undo: vLLM never reports which sleep level the engine was at, so re-sleeping would have to guess between level 1 and level 2 — call model_sleep with the level you want instead. Sleep Mode exists only on servers started with VLLM_SERVER_DEV_MODE=1; on any other server this reports that the route is absent rather than failing vaguely. Args: dry_run: If True, preview without waking. target: Inference target name from config; omit for the def…

[READ] Running vs waiting requests — the leading backpressure signal. Args: target: Inference target name from config; omit for the default.

[READ] Cluster-wide CPU/GPU capacity + headroom + pending placement groups. Args: target: Inference target name from config; omit for the default.

[READ] Serve controller health + app/deployment counts. Args: target: Inference target name from config; omit for the default.

[WRITE][risk=medium] Stop a submitted/running Ray job. Args: job_id: Job id (from ray_job_list). target: Inference target name from config; omit for the default.

[READ] Submitted Ray jobs: id, status, entrypoint, start time. Returns ``{"jobs": [...], "returned": N, "limit": L, "truncated": bool}``. When ``truncated`` is true there are more jobs than were returned — re-run with a higher limit rather than treating the result as the whole history. Args: target: Inference target name from config; omit for the default. limit: Maximum job rows to return. Default 100.

[READ] Replica states for one deployment (running/starting/draining). Args: application: Serve application name. deployment: Deployment name. target: Inference target name from config; omit for the default.

[WRITE][risk=high] Restart one Serve replica — NOT available over Ray's REST API. Ray Serve exposes no per-replica restart endpoint over REST; the controller respawns unhealthy replicas on its own. This tool refuses with a teaching error. To force a cycle, scale the deployment down then up, or redeploy. The dry_run preview reports the same unavailability rather than a false green. Args: application: Serve application name. deployment: Deployment name. replica_id: Replica id (from replica_list). dry_run: If True, report availability without attempting a restart. target: In…

[READ] vLLM TTFT / TPOT / e2e latency + generation-token totals. Args: target: Inference target name from config; omit for the default.

[WRITE][risk=medium] Switch a deployment's request-routing policy. NOTE: Ray Serve's request-routing policy is a code-level option on the @serve.deployment decorator, NOT a field settable over the Dashboard REST API — there is no REST endpoint for it. This tool therefore refuses with a teaching error rather than pretend; change the policy in the deployment's source and redeploy. Args: application: Serve application name. deployment: Deployment name. policy: Desired routing policy (prefix_aware / round_robin / session_affinity). target: Inference target name from config; omit …

[WRITE][risk=high] Lower a deployment's replica count on prod (reversible → prior). Fewer replicas can strand in-flight requests — pass dry_run=True to preview. Args: application: Serve application name. deployment: Deployment name. num_replicas: New (lower) replica count. dry_run: If True, preview without scaling. target: Inference target name from config; omit for the default.

[WRITE][risk=medium] Raise a deployment's replica count (reversible → prior). Pass dry_run=True to preview: it reads the deployment's current count, so the preview reports the real from→to rather than only the requested target — and it fails here, not mid-write, if the deployment does not exist. Args: application: Serve application name. deployment: Deployment name. num_replicas: New (higher) replica count. dry_run: If True, preview without scaling. target: Inference target name from config; omit for the default.

[WRITE][risk=high] Park a deployment at 0 replicas (reversible → prior count). Stops the cost bleed but adds cold-start latency and can strand the ingress — pass dry_run=True to preview. Args: application: Serve application name. deployment: Deployment name. dry_run: If True, preview without scaling to zero. target: Inference target name from config; omit for the default.

[READ] All Ray Serve deployments: status, replica count, target. Args: target: Inference target name from config; omit for the default.

[WRITE][risk=medium] Apply a recorded undo by dispatching its inverse tool. The inverse runs through its own governed tool, so it is audited under its own name and risk level. Pass dry_run=True to preview the inverse call without executing it. A token can only be applied once. Args: undo_id: The undoId from undo_list (or an ``_undo_id`` in a write result). dry_run: If True, preview the inverse tool + params without running it. target: Passed through to the inverse tool when it accepts a target.

[READ] List recorded, not-yet-applied undo tokens (most recent first). Each entry names the original tool, the inverse tool that ``undo_apply`` would run, and a human note. Use the ``undoId`` with ``undo_apply``. Returns {"undos": [...], "returned": N, "limit": L, "truncated": bool}. ``truncated`` is *measured* (one extra row is fetched), not guessed from a length coincidence: when it is true there are MORE tokens than shown, so re-run with a higher ``limit`` rather than reporting the list as complete. Each entry carries ``effectVerified``. False means the original write lost its response, …