quota-debug

Files (1)

• SKILL.md

SKILL.md (first lines)

# ResourceQuota & LimitRange Admission Diagnosis

When pods fail to be created due to Kubernetes native namespace-level resource constraints — ResourceQuota exceeded or LimitRange violations — follow this flow to identify the root cause.

**Scope:** This skill is for **diagnosis only**. Once you identify the root cause, report it to the user and stop. Do NOT attempt to modify ResourceQuota, LimitRange, or pod specs — that should be left to the user or cluster administrator.

**When to use:** Pods are not being created at all (not Pending, not CrashLoopBackOff — simply missing). Typical trigger: a ReplicaSet or Job shows `FailedCreate` events mentioning `exceeded quota` or `forbidden: ... LimitRange`.

**Not applicable to Volcano Queue:** If the cluster uses Volcano for gang scheduling, resource quotas are managed by Volcano Queue, not Kubernetes native ResourceQuota. Use the `volcano-queue-diagnose` skill instead. To check:

```bash
kubectl get queue 2>/dev/null
```

If this command returns results (Queue resources listed), the cluster uses Volcano — use `volcano-queue-diagnose`. If it returns nothing or an error, Volcano is not installed — continue with this skill.

## Diagnostic Flow

### 1. Identify the creation failure

If the user reports a Deployment not progressing or pods not appearing, first find the controller that owns the pods:

```bash
kubectl get rs -n <ns> --sort-by='.metadata.creationTimestamp' | grep <deployment-name>
```

Then check the ReplicaSet events for creation failures:

```bash
kubectl describe rs <new-rs> -n <ns>
```

Look for events with reason `FailedCreate`. The event message reveals whether it is a ResourceQuota or LimitRange rejection.

If the user already has a specific error message, skip to step 2.

### 2. Match the rejection type

---

#### `exceeded quota` — ResourceQuota exhausted

The namespace has a ResourceQuota and creating the pod would exceed the allowed limits.

Check current quota usage:

```bash
kubectl get resourcequota -n <ns>
```

For detailed usage breakdown:

```bash
kubectl describe resourcequota -n <ns>
```

Compare the `Used` vs `Hard` columns. Common quota dimensions:
- **requests.cpu / requests.memory** — total CPU/memory requests across all pods in the namespace
- **limits.cpu / limits.memory** — total CPU/memory limits
- **pods** — maximum number of pods allowed in the namespace
- **count/deployments.apps, count/services** — object count limits
- **requests.nvidia.com/gpu** — total GPU requests (common in GPU-scheduled clusters)
- **requests.storage** — total PVC storage requested
- **persistentvolumeclaims** — number of PVCs

Then check what the pod is requesting:

```bash
kubectl get pod -n <ns> -l app=<name> -o jsonpath='{range .items[0].spec.containers[*]}{.name}{"\t"}{.resources}{"\n"}{end}' 2>/dev/null
```

If no pods exist yet (all creation failed), check the Deployment or template spec:

```bash
kubectl get deployment <name> -n <ns> -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\t"}{.resources}{"\n"}{end}'
```

**Root cause analysis:**
- If `Used` is near `Hard` for cpu/memory: existing pods are consuming most of the quota — need to scale down other workloads or increase quota
- If `pods` count is at the limit: too many pods in namespace — clean up or increase quota
- If the pod's resource requests are very large: consider reducing requests to fit within remaining quota

---

#### `forbidden: ... minimum cpu/memory` — LimitRange minimum violation

The pod's container does not meet the minimum resource request required by the namespace's LimitRange.

```bash
kubectl get limitrange -n <ns>
```

```bash
kubectl describe limitrange -n <ns>
```

Check the `Min` column for Container type. If a container does not specify resource requests, or its requests are below the minimum, admission will be rejected.

Compare with the pod's resource spec. If the container has no resource requests at all, the LimitRange default will be applied — but if the LimitRange has `min` without `default`, admission fails.

---

#### `forbidden: ... maximum cpu/memory` — LimitRange maximum violation

The pod's container exceeds the maximum resource limit allowed by the LimitRange.

Check LimitRange as above and compare the `Max` column with the container's resource requests/limits.

---

#### `forbidden: ... maxLimitRequestRatio` — LimitRange ratio violation

The ratio between the container's resource limit and request exceeds the allowed ratio (e.g., limit is 10x the request, but ratio cap is 3x).

```bash
kubectl describe limitrange -n <ns>
```

Check the `Max Limit/Request Ratio` column. Then compare the pod's limits vs requests:

From scitix/siclaw, Apache-2.0.

More from scitix/siclaw