Gke Ai Troubleshooting Tpu Performance Degradation
google/skills/skills/cloud/gke-ai-troubleshooting-tpu-performance-degradationby googlec6c7e67107f02e3423df8481272f296bbe61902aNo license21K starsListed Oct 9, 2026Updated Oct 9, 2026Repository updated today
Diagnose GKE Cloud TPU training throughput drops and step-time regressions (15%+ TPU duty-cycle drop) using ML Diagnostics Workload Monitoring (`gcloud alpha mldiagnostics monitored-events` / `hypercomputecluster.googleapis.com/v1alpha`) and 1-minute Cloud Monitoring system metrics (`kubernetes.io/node/accelerator/*`). Distinguishes hardware and network fabric throttling from workload resource bottlenecks (HBM capacity, host memory, or host CPU saturation). Use when TPU training throughput or duty cycle drops without crashing pods, when `PERFORMANCE_DEGRADATION` monitored events fire, or when triaging slow multi-slice training steps. Don't use for complete multi-slice XLA execution stalls with `HANG_DETECTED` logs (use gke-ai-troubleshooting-tpu-mxla-hang) or pod eviction/interruption restarts (use gke-ai-troubleshooting-jobset-interruption).
- c6c7e67107f02e3423df8481272f296bbe61902aCurrentcommit c6c7e67Published Oct 9, 2026
Source and attribution
Source:google/skillsinskills/cloud/gke-ai-troubleshooting-tpu-performance-degradationat commitc6c7e67
License: No license
Content belongs to its original authors. SourceWeft indexes it from a public repository.
More from google/skills
Sign In With Google Web
Guidance for implementing and securing Sign In With Google (GIS) across web architectures.
Gke Ai Troubleshooting Tpu Mxla Hang
Diagnoses GKE Cloud TPU multi-slice training hangs using ML Diagnostics MXLA Hang Analyzer reports and metrics.
Google Cloud Scc Remediation
Guides remediation of Google Cloud Security Command Center findings with consent-gated plans and reference playbooks.
Google Cloud Recipe Onboarding
Guides a developer through first-time Google Cloud onboarding: authentication, project setup, billing linkage and validation.
Google Cloud Recipe Auth
Guides authentication and authorization to Google Cloud services and APIs for users, service accounts, and workloads.
Gcloud
Provides safety guardrails, syntax validation, and data-reduction rules for running gcloud CLI commands on Google Cloud.
More in DevOps & Cloud
Update Screenshots
microsoft
Updates committed CI screenshot hash baselines after the Screenshots & Tests check fails on a pull request.
Playwright Devops
microsoft
DevOps workflows for Playwright: analyze GitHub Actions failures for the last commit on main and fetch failed job logs.
M5 Onboard
anthropics
Provisions M5Stack ESP32 boards by detecting them on USB, flashing UIFlow 2.0 firmware, and installing a MicroPython app bundle.
Runbook
anthropics
Creates or updates step-by-step operational runbooks for recurring tasks, including troubleshooting, rollback and escalation.
Incident Response
anthropics
Guides an incident response workflow: severity triage, status updates, mitigation tracking, and blameless postmortems.
Deploy Checklist
anthropics
Generates a pre-deployment readiness checklist covering pre-deploy, deploy, post-deploy and rollback triggers.