aiml-gpu-training-cluster-investigation

Files (117)

• .skilleval.yaml
• CHANGELOG.md
• README.md
• SKILL.md
• benchmark.json · evals/best-practices/v10/
• best-practices-tests-results.json · evals/best-practices/v10/iteration-1/
• best-practices-tests-results.json · evals/best-practices/v10/iteration-2/
• best-practices-tests-results.json · evals/best-practices/v10/iteration-3/
• evals.json · evals/
• benchmark.json · evals/functional/v5/
• evals.json · evals/functional/v5/
• functional-tests-results.json · evals/functional/v5/iteration-1/capacity-block-expiry/with_skill/
• journal_records.json · evals/functional/v5/iteration-1/capacity-block-expiry/with_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-1/capacity-block-expiry/without_skill/
• journal_records.json · evals/functional/v5/iteration-1/capacity-block-expiry/without_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-1/control-plane-log-dead/with_skill/
• journal_records.json · evals/functional/v5/iteration-1/control-plane-log-dead/with_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-1/control-plane-log-dead/without_skill/
• journal_records.json · evals/functional/v5/iteration-1/control-plane-log-dead/without_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-1/fsx-training-slowdown-cause/with_skill/
• journal_records.json · evals/functional/v5/iteration-1/fsx-training-slowdown-cause/with_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-1/fsx-training-slowdown-cause/without_skill/
• journal_records.json · evals/functional/v5/iteration-1/fsx-training-slowdown-cause/without_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-1/gpu-log-coverage-audit/with_skill/
• journal_records.json · evals/functional/v5/iteration-1/gpu-log-coverage-audit/with_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-1/gpu-log-coverage-audit/without_skill/
• journal_records.json · evals/functional/v5/iteration-1/gpu-log-coverage-audit/without_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-1/hyperpod-application-xid-verdict/with_skill/
• journal_records.json · evals/functional/v5/iteration-1/hyperpod-application-xid-verdict/with_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-1/hyperpod-application-xid-verdict/without_skill/
• journal_records.json · evals/functional/v5/iteration-1/hyperpod-application-xid-verdict/without_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-1/negative-bedrock-throttling/with_skill/
• journal_records.json · evals/functional/v5/iteration-1/negative-bedrock-throttling/with_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-1/negative-load-balancer-choice/with_skill/
• journal_records.json · evals/functional/v5/iteration-1/negative-load-balancer-choice/with_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-1/preflight-long-run-readiness/with_skill/
• journal_records.json · evals/functional/v5/iteration-1/preflight-long-run-readiness/with_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-1/preflight-long-run-readiness/without_skill/
• journal_records.json · evals/functional/v5/iteration-1/preflight-long-run-readiness/without_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-1/xid-48-reboot-first/with_skill/
• journal_records.json · evals/functional/v5/iteration-1/xid-48-reboot-first/with_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-1/xid-48-reboot-first/without_skill/
• journal_records.json · evals/functional/v5/iteration-1/xid-48-reboot-first/without_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-2/capacity-block-expiry/with_skill/
• journal_records.json · evals/functional/v5/iteration-2/capacity-block-expiry/with_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-2/capacity-block-expiry/without_skill/
• journal_records.json · evals/functional/v5/iteration-2/capacity-block-expiry/without_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-2/control-plane-log-dead/with_skill/
• journal_records.json · evals/functional/v5/iteration-2/control-plane-log-dead/with_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-2/control-plane-log-dead/without_skill/
• journal_records.json · evals/functional/v5/iteration-2/control-plane-log-dead/without_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-2/fsx-training-slowdown-cause/with_skill/
• journal_records.json · evals/functional/v5/iteration-2/fsx-training-slowdown-cause/with_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-2/fsx-training-slowdown-cause/without_skill/
• journal_records.json · evals/functional/v5/iteration-2/fsx-training-slowdown-cause/without_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-2/gpu-log-coverage-audit/with_skill/
• journal_records.json · evals/functional/v5/iteration-2/gpu-log-coverage-audit/with_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-2/gpu-log-coverage-audit/without_skill/
• journal_records.json · evals/functional/v5/iteration-2/gpu-log-coverage-audit/without_skill/outputs/
• functional-tests-results.json · evals/functional/v5/iteration-2/hyperpod-application-xid-verdict/with_skill/
• …and 57 more

SKILL.md (first lines)

# GPU Cluster Evidence, Readiness, and Fault Verdicts

For GPU clusters on SageMaker HyperPod (Slurm or EKS), AWS ParallelCluster, and self-managed
EC2 or EKS. Generic investigation finds the obvious signals; this skill adds what it gets
wrong: silent evidence, wrong-headline hardware verdicts, and predictable failures before a
long run. **Read-only.** Never reboot, replace, update, or delete anything, and never read
training data, checkpoints, or model weights.

## Critical rules R1 to R11 (apply in every mode, in this order)

R1. **Answer in one pass, and always leave room to answer.** In chat, do not stop to ask a
   question and do not hand off to a separate investigation before answering. If an input is
   missing, use the default (impact window: last 24 hours; run length: evaluate 24, 48, and
   72 hours), state the assumption, and mark dependent checks `Needs input`. For "slow" or
   performance questions with no time given, use the last 72 hours. Offer follow-ups only
   after the answer.
   **Budget the evidence gathering so the answer always gets written.** An investigation that
   runs out of room before it reports is worth nothing to the operator, and it is worse than a
   partial answer because it looks like a failure rather than a finding. So: collect the
   mandatory evidence for the mode first (R2, R4, R5, and for Mode P the P1 to P6 core), then
   write the report. Pick up the optional checks only with what is left. If you notice you are
   deep into tool calls and have not yet produced an answer, **stop collecting and report what
   you have**, marking everything unreached as `Not checked` with the call that would close
   it. Never end a turn with evidence gathered and no verdict.
R2. **Inventory and capability profile.** HyperPod: `sagemaker.DescribeCluster` and
   `ListClusterNodes` (paginate). Read `NodeProvisioningMode` from `DescribeCluster`: if it is
   `Continuous`, also pull `sagemaker.ListClusterEvents` for the window (see rule R11), which
   is the only timeline source that survives broken log delivery. On any other value the call
   is unsupported and must be skipped, not retried.
   EC2/ParallelCluster/EKS: `ec2.DescribeInstances`. For every
   GPU instance type: `ec2.DescribeInstanceTypes` (strip HyperPod `ml.`): GPU count,
   `EfaSupported`, `MaximumEfaInterfaces`; per EC2 node, attached `efa`/`efa-only` interfaces.
   Report `<attached> of <max>`, where attached = interfaces with `InterfaceType` `efa` or
   `efa-only` (the primary ENA interface does not count unless it is `efa`). HyperPod nodes
   are not visible to `DescribeInstances`: say so. On NVSwitch types, search every log source
   found under rule R4 for `Started "Nvidia Fabric Manager"` before saying it is not confirmed.
R3. **Node identity survives replacement.** A HyperPod reboot keeps the instance ID; a replace
   gives the node a **new instance ID in the same instance group**, so the current ID will
   never appear in the replace request. Query CloudTrail **by event name, not by instance
   ID**: `cloudtrail.LookupEvents` with `LookupAttributes=[{AttributeKey: EventName,
   AttributeValue: BatchReplaceClusterNodes}]`, then again for `BatchRebootClusterNodes`,
   `BatchDeleteClusterNodes`, and `UpdateCluster`, with `StartTime` = window start minus 6
   hours and `EndTime` = now as full ISO-8601 UTC timestamps, paginating with `NextToken`.
   Keep events whose `requestParameters.clusterName` is this cluster. A `nodeIds` entry
   that is not in the current `ListClusterNodes` output was replaced; the instance group
   whose node has a `LaunchTime` just after that event is the replaced group. That operator
   or automatic call is the explanation for the node going `Pending` (Branch E), not hardware.
R4. **Find every log source by substring, not prefix.** Call `logs.DescribeLogGroups` with
   `logGroupNamePattern` (case-sensitive substring) = the cluster name, then again for
   `kernel`, `messages`, `syslog`, `journal`, and `gpu`, paginating with `nextToken`. Never
   search only `/aws/parallelcluster` or `/aws/sagemaker` prefixes: customer pipelines use
   other names (for example `/aws/<pipeline>/<cluster>/kernel`). Evaluate every source found.
R5. **Prove coverage before any "no errors".** For each node and source: find the stream that
   carries `kernel:` lines, then bin **that exact stream** by hour across the window padded by
   one hour. **Always name the evidence you used: quote the full log group name and the exact
   log stream name for every node in the coverage table, and again in the answer text.** A
   coverage claim without the group and stream it rests on is not auditable, so the operator
   cannot re-run it. Any empty hour means `Not observable` for that hour. First and last event times
   are not proof, and the time of the **last `kernel:` line** is not when logging stopped:
   a healthy kernel goes quiet. Liveness comes only from the hourly bins of all lines in
   that stream. A node is `Measured` if one source passes. HyperPod: a missing
   `SagemakerHealthMonitoringAgent/<group>/<instance-id>` stream means `No HMA detections`
   when the cluster log group is otherwise live. NCCL transport with no `NCCL INFO` lines
   anywhere is `Not observable`; never infer it from the instance type.
R5a. **Name every resource you looked at, by ID.** A finding the operator cannot re-run is
   not a finding. Whatever you analysed, put its identifier in the answer: the FSx file system
   (`fs-...`) behind any storage claim, the instance IDs (`i-...`) behind any node claim, the
   cluster name, the capacity reservation (`cr-...`) behind any capacity claim, and the log
   group and stream behind any log claim as R5 already requires. "The file system was
   saturated" or "the metrics looked fine" names nothing and cannot be checked. This applies
   to the resource you cleared as much as the one you blamed, since ruling something out is
   only useful if the reader knows what was ruled out.
R6. **Verdict per node, headline to match.** `REPLACE`, `REBOOT`, `LEAVE ALONE`, `MONITOR`, or
   `NOT OBSERVABLE`, against the evidence bar in `references/incident-branches.md` (Step 4b).
   Application-class Xids (for example 13, 31) or HMA `reason: XidUserAppError` with the node
   `Running` is `LEAVE ALONE`. Never headline "hardware error" unless the verdict is `REPLACE`
   or `REBOOT` on hardware grounds.
R7. **Label every cause** `Proven` (measured signal on the affected node, before the failure,
   nothing competing) or `Hypothesis (to validate)` with the one confirming measurement. A
   spike at the same time is correlation. FSx without a saturated metric is not a proven cause.
   Only a `Proven` cause may be called the root cause, in the headline or in a branch table.
   Otherwise write `Leading hypothesis: <cause>`, or `Root cause: Not observable` when the
   deciding evidence is missing (for example a dead control-plane log). Never write "Proven
   mechanism" for something whose trigger or removal path you did not observe.
   Utilization metrics from FSx (`NetworkThroughputUtilization`, `DiskIopsUtilization`, and
   similar) and `GPUPowerUtilization` are already percent from 0 to 100: a value of `0.9` is
   0.9 percent. Quote the raw value with a percent sign.
R8. **Recovery questions** always state three things: whether automatic node recovery is on
   (`NodeRecovery`), what it does (reboot or replace the node), and that the **job** resumes
   only with checkpoints plus the orchestrator's auto-resume (Slurm on HyperPod:
   `srun --auto-resume=1`).
R9. **Capacity Blocks** begin terminating instances 30 minutes before the end time (60 for
   UltraServers); blocks end at 11:30 UTC and termination starts at 11:00 UTC on the last day.
   For a planned run, write out: usable until = end time minus the lead time; run end = start
   plus run length; hours covered = usable until minus start. Give every value as a full UTC
   date and time, and check the latest safe start is not already in the past.
R10. **Rule out the frequent non-GPU causes** in `references/cluster-edge-cases.md` before
   blaming hardware: subnet IP or network interface exhaustion, ParallelCluster bootstrap
   failures and protected mode, EFA nodes in a public subnet, a Capacity Block not yet
   active, and the FSx maintenance window. HyperPod does not export system metrics to
   CloudWatch, so HyperPod GPU activity is `Not observable` there.
R11. **When the logs are dead, ask the control plane.** On a HyperPod cluster with
   `NodeProvisioningMode = Continuous`, `sagemaker.ListClusterEvents` gives you a node and
   cluster timeline that owes nothing to a log agent, so it keeps answering when a stream has
   gone silent or a node has disappeared. Filter the window with `EventTimeAfter` and
   `EventTimeBefore`, narrow with `NodeId` or `InstanceGroupName`, sort with
   `SortBy=EventTime`, and page through `NextToken`. Where a `Description` is not
   self-explanatory, `DescribeClusterEvent` has the detail. Note that the response has no
   severity or level field at all, so any grouping you apply is your own and should be
   described that way. If `NodeProvisioningMode` is anything other than `Continuous` the call
   is not supported; write `ListClusterEvents not supported` in the coverage table and carry
   on. What you must not do is report a dead log as "no events" without either trying this
   source or saying it was unavailable.

## Pick the mode

| The user asks | Mode | Steps to run |
|---------------|------|--------------|
| Something failed, hung, slowed, or lost nodes | **I: Incident** | Steps 1 to 7 |

From aws/tools-for-devops-agent, Apache-2.0.

More from aws/tools-for-devops-agent