Gke Ai Troubleshooting Tpu Mxla Hang
google/skills/skills/cloud/gke-ai-troubleshooting-tpu-mxla-hangby googlec6c7e67107f02e3423df8481272f296bbe61902aNo license21K starsListed Oct 9, 2026Updated Oct 9, 2026Repository updated today
Diagnose GKE Cloud TPU multi-slice training hangs (Megascale `HANG_DETECTED` logs and hang monitored events) using the ML Diagnostics `Megascale XLA (MXLA) Hang Analyzer` (`gcloud alpha mldiagnostics monitored-events`) and 1-minute Cloud Monitoring multi-slice latency metrics (`kubernetes.io/container/multislice/*`). Distinguishes XLA compiler/HLO launch divergence and host data-input stalls from TPU chip, SparseCore, ICI, or network fabric faults. Use when multi-slice TPU training jobs freeze without progressing steps, emit `HANG_DETECTED`, or stall in collective operations. Don't use for gradual step-time throughput drops without hangs (use gke-ai-troubleshooting-tpu-performance-degradation) or pod preemption/eviction restarts (use gke-ai-troubleshooting-jobset-interruption).
- c6c7e67107f02e3423df8481272f296bbe61902aCurrentcommit c6c7e67Published Oct 9, 2026
Source and attribution
Source:google/skillsinskills/cloud/gke-ai-troubleshooting-tpu-mxla-hangat commitc6c7e67
License: No license
Content belongs to its original authors. SourceWeft indexes it from a public repository.
More from google/skills
Sign In With Google Web
Guidance for implementing and securing Sign In With Google (GIS) across web architectures.
Gke Ai Troubleshooting Tpu Performance Degradation
Diagnoses GKE Cloud TPU training throughput drops using ML Diagnostics events and Cloud Monitoring metrics.
Google Cloud Scc Remediation
Guides remediation of Google Cloud Security Command Center findings with consent-gated plans and reference playbooks.
Google Cloud Recipe Onboarding
Guides a developer through first-time Google Cloud onboarding: authentication, project setup, billing linkage and validation.
Google Cloud Recipe Auth
Guides authentication and authorization to Google Cloud services and APIs for users, service accounts, and workloads.
Gcloud
Provides safety guardrails, syntax validation, and data-reduction rules for running gcloud CLI commands on Google Cloud.
More in DevOps & Cloud
Update Screenshots
microsoft
Updates committed CI screenshot hash baselines after the Screenshots & Tests check fails on a pull request.
Playwright Devops
microsoft
DevOps workflows for Playwright: analyze GitHub Actions failures for the last commit on main and fetch failed job logs.
M5 Onboard
anthropics
Provisions M5Stack ESP32 boards by detecting them on USB, flashing UIFlow 2.0 firmware, and installing a MicroPython app bundle.
Runbook
anthropics
Creates or updates step-by-step operational runbooks for recurring tasks, including troubleshooting, rollback and escalation.
Incident Response
anthropics
Guides an incident response workflow: severity triage, status updates, mitigation tracking, and blameless postmortems.
Deploy Checklist
anthropics
Generates a pre-deployment readiness checklist covering pre-deploy, deploy, post-deploy and rollback triggers.