Gke Reliability

GKE Reliability

Routing Note: To generate Kubernetes YAML manifests (Deployment, StatefulSet, Service, ConfigMap, HTTPRoute, PodDisruptionBudget), open gke-manifest-generation/SKILL.md.

This reference covers high availability and reliability configuration for GKE clusters and workloads.

MCP Tools: get_cluster, get_k8s_resource, describe_k8s_resource, apply_k8s_manifest, list_k8s_events

Golden Path Reliability Defaults

Workflows

1. Verify Cluster High Availability

# MCP (preferred)get_cluster(name="projects/<PROJECT>/locations/<REGION>/clusters/<CLUSTER>",  readMask="location,locations,nodePools.locations")
# gcloud fallbackgcloud container clusters describe <CLUSTER> --region <REGION> \  --format="json(location, locations)" \  --quiet

• If location is a region (e.g., us-central1), the control plane is regional
• If locations has multiple entries, nodes span multiple zones

2. Pod Disruption Budgets (PDBs)

PDBs ensure minimum pod availability during voluntary disruptions (node upgrades, autoscaler scale-down).

Check existing PDBs:

# MCP (preferred)get_k8s_resource(parent="...", resourceType="poddisruptionbudget")
# kubectl fallbackkubectl get pdb --all-namespaces

Create PDB:

apiVersion: policy/v1kind: PodDisruptionBudgetmetadata:  name: my-app-pdb  namespace: defaultspec:  minAvailable: 2       # Or use maxUnavailable: 1  selector:    matchLabels:      app: my-app

Every production Deployment with 2+ replicas should have a PDB.

3. Health Probes

Every production container should have liveness and readiness probes. Startup probes are recommended for slow-starting apps.

Check existing probes:

# MCP (preferred)describe_k8s_resource(parent="...", resourceType="deployment", name="<APP>", namespace="<NS>")
# kubectl fallbackkubectl get deployment <APP> -n <NS> -o yaml | grep -E "livenessProbe|readinessProbe|startupProbe"

Recommended probe configuration:

spec:  containers:  - name: app    livenessProbe:      httpGet:        path: /healthz        port: 8080      initialDelaySeconds: 15      periodSeconds: 10      timeoutSeconds: 2      failureThreshold: 3    readinessProbe:      httpGet:        path: /readyz        port: 8080      initialDelaySeconds: 5      periodSeconds: 5      timeoutSeconds: 2      failureThreshold: 3    startupProbe:             # For slow-starting apps      httpGet:        path: /healthz        port: 8080      initialDelaySeconds: 10      periodSeconds: 5      timeoutSeconds: 2      failureThreshold: 30    # 30 * 5s = 150s max startup time

• Readiness: Determines when a pod can accept traffic
• Liveness: Determines when to restart a container
• Startup: Disables liveness/readiness until the app is ready (prevents premature restarts)

4. Graceful Shutdown

Ensure applications handle SIGTERM and drain in-flight requests:

spec:  terminationGracePeriodSeconds: 30    # Default; increase for long-running requests  containers:  - name: app    lifecycle:      preStop:        exec:          command: ["/bin/sh", "-c", "sleep 5"]  # Allow LB to deregister

5. Topology Spread Constraints

Distribute pods across zones and nodes to survive failures:

spec:  topologySpreadConstraints:  - maxSkew: 1    topologyKey: topology.kubernetes.io/zone    whenUnsatisfiable: DoNotSchedule    labelSelector:      matchLabels:        app: my-app  - maxSkew: 1    topologyKey: kubernetes.io/hostname    whenUnsatisfiable: ScheduleAnyway    labelSelector:      matchLabels:        app: my-app

• Zone spread (DoNotSchedule): Hard requirement -- pods must be balanced across zones
• Node spread (ScheduleAnyway): Best-effort -- prefer distribution but don't block scheduling

6. Replicas

Best Practices & Production Guidelines

1. Regional clusters for production: Always use regional clusters to survive zone failures.
2. PDBs for everything: Every production workload with 2+ replicas needs a PodDisruptionBudget (PDB) to protect against voluntary disruptions.
3. Probes with Explicit Timeouts: Every production container must have both liveness and readiness probes defined. Always explicitly define initialDelaySeconds, periodSeconds, and timeoutSeconds for all probes. Never rely on the Kubernetes default timeout of 1 second if your application requires more, but always set a strict limit to prevent hanging connections.
4. Zone spreading: Use topology spread constraints to distribute pods across failure domains (zones and nodes).
5. Graceful shutdown: Handle SIGTERM and set appropriate terminationGracePeriodSeconds with a preStop sleep hook to allow load balancer deregistration.
6. Maintenance windows: Schedule upgrades during low-traffic periods (see the gke-upgrades skill).