gke-inference

Files (2)

• SKILL.md
• attribution.md · references/

SKILL.md (first lines)

# GKE AI/ML Inference

> **Routing Note:** For migrating existing AI workloads to GKE, open `google-cloud-solution-guided-gke-ai-migration/SKILL.md`. For GKE RAG with Cloud SQL or AlloyDB (`pgvector`), open `google-cloud-solution-rag-enterprise-search-gke-sqldb/SKILL.md`.

This reference covers deploying AI/ML inference workloads on GKE using Google's
Inference Quickstart (GIQ) and best practices for LLM serving.

> **MCP Tools:** `apply_k8s_manifest`, `get_k8s_resource`, `get_k8s_logs`,
> `get_k8s_rollout_status`, `describe_k8s_resource`, `list_k8s_events`.
> **CLI-only:** `gcloud container ai profiles *`

## Attribution

**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you
MUST read [Usage Attribution](references/attribution.md). For commands executed
directly in the terminal (never in generated scripts, Terraform, or user code),
prefix `gcloud` with
`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-inference)"`
and append `agent-skills/1.0 (skill:gke-inference)` to `User-Agent` headers.

## When to Use

-   Deploy an AI model (Llama, Gemma, Mistral, etc.) to GKE
-   Generate optimized Kubernetes manifests for inference
-   Select GPU/TPU accelerators for model serving
-   Configure autoscaling for LLM inference

## Prerequisites

-   A golden path GKE Autopilot cluster (GPU workloads are supported via
    ComputeClasses and NAP)
-   `gcloud` CLI authenticated
-   Sufficient GPU/TPU quota in the target region

## Workflow

### 1. Discovery: Find Models and Hardware

```bash
# List all supported models
gcloud container ai profiles models list --quiet

# Find valid accelerator/server combinations for a model
gcloud container ai profiles list --model=<MODEL_NAME> --quiet

# Example: what can run Gemma 2 9B?
gcloud container ai profiles list --model=gemma-2-9b-it --quiet
```

### 2. Generate Manifest

```bash
gcloud container ai profiles manifests create \
  --model=<MODEL_NAME> \
  --model-server=<SERVER> \
  --accelerator-type=<ACCELERATOR> \
  --target-ntpot-milliseconds=<NTPOT> --quiet > inference.yaml
```

**Parameters:**

-   `--model`: Model ID (e.g., `gemma-2-9b-it`, `llama-3-8b`)
-   `--model-server`: Inference server (`vllm`, `tgi`, `triton`, `tensorrt-llm`)
-   `--accelerator-type`: GPU/TPU type (`nvidia-l4`, `nvidia-tesla-a100`,
    `nvidia-h100-80gb`)
-   `--target-ntpot-milliseconds`: Target Normalized Time Per Output Token
    (optional, for latency optimization)

**Example:**

```bash
gcloud container ai profiles manifests create \
  --model=gemma-2-9b-it \
  --model-server=vllm \
  --accelerator-type=nvidia-l4 \
  --target-ntpot-milliseconds=50 --quiet > inference.yaml
```

### 3. Review and Deploy

```bash
# Review for placeholders (HF tokens, PVCs)
cat inference.yaml

# Deploy
kubectl apply -f inference.yaml

# Monitor
kubectl get pods -w
kubectl logs -f <POD_NAME>
```

> Some models require Hugging Face tokens. Create a Kubernetes Secret and
> reference it in the manifest.

## GPU ComputeClass for Inference

For Autopilot clusters, create a ComputeClass to target GPU nodes:

```yaml
apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  name: l4-inference
spec:
  priorities:
  - machineFamily: g2
    gpu:
      type: nvidia-l4
      count: 1
    minCores: 4
    minMemoryGb: 16
```

## Accelerator Selection Guide

| Accelerator         | Best For                 | Memory      | Relative Cost |
| ------------------- | ------------------------ | ----------- | ------------- |
| NVIDIA T4           | Budget inference,        | 16 GB       | Lowest        |

From google/skills, Apache-2.0.

More from google/skills