Gke Inference

GKE AI/ML Inference

Routing Note: For migrating existing AI workloads to GKE, open google-cloud-solution-guided-gke-ai-migration/SKILL.md. For GKE RAG with Cloud SQL or AlloyDB (pgvector), open google-cloud-solution-rag-enterprise-search-gke-sqldb/SKILL.md.

This reference covers deploying AI/ML inference workloads on GKE using Google's Inference Quickstart (GIQ) and best practices for LLM serving.

MCP Tools: apply_k8s_manifest, get_k8s_resource, get_k8s_logs, get_k8s_rollout_status, describe_k8s_resource, list_k8s_events. CLI-only: gcloud container ai profiles *

When to Use

• Deploy an AI model (Llama, Gemma, Mistral, etc.) to GKE
• Generate optimized Kubernetes manifests for inference
• Select GPU/TPU accelerators for model serving
• Configure autoscaling for LLM inference

Prerequisites

• A golden path GKE Autopilot cluster (GPU workloads are supported via ComputeClasses and NAP)
• gcloud CLI authenticated
• Sufficient GPU/TPU quota in the target region

Workflow

1. Discovery: Find Models and Hardware

# List all supported modelsgcloud container ai profiles models list --quiet
# Find valid accelerator/server combinations for a modelgcloud container ai profiles list --model=<MODEL_NAME> --quiet
# Example: what can run Gemma 2 9B?gcloud container ai profiles list --model=gemma-2-9b-it --quiet

2. Generate Manifest

gcloud container ai profiles manifests create \  --model=<MODEL_NAME> \  --model-server=<SERVER> \  --accelerator-type=<ACCELERATOR> \  --target-ntpot-milliseconds=<NTPOT> --quiet > inference.yaml

Parameters:

• --model: Model ID (e.g., gemma-2-9b-it, llama-3-8b)
• --model-server: Inference server (vllm, tgi, triton, tensorrt-llm)
• --accelerator-type: GPU/TPU type (nvidia-l4, nvidia-tesla-a100, nvidia-h100-80gb)
• --target-ntpot-milliseconds: Target Normalized Time Per Output Token (optional, for latency optimization)

Example:

gcloud container ai profiles manifests create \  --model=gemma-2-9b-it \  --model-server=vllm \  --accelerator-type=nvidia-l4 \  --target-ntpot-milliseconds=50 --quiet > inference.yaml

3. Review and Deploy

# Review for placeholders (HF tokens, PVCs)cat inference.yaml
# Deploykubectl apply -f inference.yaml
# Monitorkubectl get pods -wkubectl logs -f <POD_NAME>

Some models require Hugging Face tokens. Create a Kubernetes Secret and reference it in the manifest.

GPU ComputeClass for Inference

For Autopilot clusters, create a ComputeClass to target GPU nodes:

apiVersion: cloud.google.com/v1kind: ComputeClassmetadata:  name: l4-inferencespec:  priorities:  - machineFamily: g2    gpu:      type: nvidia-l4      count: 1    minCores: 4    minMemoryGb: 16

Accelerator Selection Guide

Autoscaling LLM Inference

GPU-based autoscaling

Use custom metrics for GPU utilization:

apiVersion: autoscaling/v2kind: HorizontalPodAutoscalermetadata:  name: llm-hpaspec:  scaleTargetRef:    apiVersion: apps/v1    kind: Deployment    name: llm-server  minReplicas: 1  maxReplicas: 10  metrics:  - type: Pods    pods:      metric:        name: gpu_duty_cycle      target:        type: AverageValue        averageValue: "80"

Best practices for inference autoscaling

1. Use DCGM metrics: Golden path enables DCGM monitoring for GPU utilization metrics
2. Set appropriate minReplicas: At least 1 for always-on serving; 0 for batch/on-demand
3. Tune scale-down delay: LLM model loading is slow; use longer stabilization windows
4. Consider queue depth: Scale on pending requests rather than pure GPU utilization for latency-sensitive workloads

Optimization Tips

• Quantization: Use quantized models (GPTQ, AWQ) to reduce GPU memory and increase throughput
• Batching: Configure model server batch size for throughput vs latency trade-off
• Tensor parallelism: Split large models across multiple GPUs within a node
• KV cache optimization: Tune --gpu-memory-utilization in vLLM for KV cache allocation

Troubleshooting