gke-ai-troubleshooting-tpu-vbar-oom

TPU Connection Failure and VBAR OOM Troubleshooting

Use this skill to systematically diagnose and prevent vbar_control_agent segfaults and Out-Of-Memory (OOM) errors on TPU v6e nodes.

Attribution

MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read Usage Attribution. For commands executed directly in the terminal (never in generated scripts, Terraform, or user code), prefix gcloud with CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-vbar-oom)" and append agent-skills/1.0 (skill:gke-ai-troubleshooting-tpu-vbar-oom) to User-Agent headers.

⚠️ Prerequisites

• Cloud Logging must be enabled for the project.
• Access to the project and cluster via gcloud or equivalent tool.

? Diagnostic Workflow

Step 0: Context Acquisition & Time Window Definition

Independently gather required context using available GCP/GKE tools or use the provided {variable} placeholders:

• {project_id}: The GCP Project ID (e.g., customer-ai-project-123).
• {cluster_name}: The GKE Cluster Name (e.g., tpu-cluster-prod).
• {node_name}: The Node Name or Instance ID (e.g., tpu-node-1).
• {workload_name}: The Workload Name / JobSet Name (e.g., my-training-job-456).
• {namespace}: The Workload Namespace.
• {issue_time}: The timestamp of the issue (e.g., 2026-04-14T20:00:00Z).

Time Handling & Execution Rules

1. Window Calculation: If an issue timestamp {issue_time} is provided, calculate the query time window as [{issue_time} - 30m] to [{issue_time} + 30m].
   • Let {start_time} = {issue_time} - 30m
   • Let {end_time} = {issue_time} + 30m
2. Informational vs. Live Execution: If the user request is informational or query-formulation (e.g. "How can I check...", "How do I determine..."), or if live GCP project resources are not actively targetable, directly output the calculated time window, log names, and Cloud Logging filter templates without attempting live log execution commands.

Step 1: Check for vbar_control_agent OOMs

Look for specific out of memory messages from vbar_control_agent in serial console logs (serialconsole.googleapis.com%2fserial_port_1_output).

• Tool to use: query_logs (for live diagnostics)
• Filter Templates:

Serial Console Logs (OOMs):

logName="projects/{project_id}/logs/serialconsole.googleapis.com%2fserial_port_1_output"
AND labels."compute.googleapis.com/resource_name"="{node_name}"
AND SEARCH(text_payload, "Memory cgroup out of memory: Killed process .* (vbar_control_ag)")
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

• Logic: Presence of Memory cgroup out of memory messages related to vbar_control_agent. Stack traces pointing to libtpu::tpunetd::VBARControlHelper::MetricsReadFromVBAR are a strong indicator.
• Automation: Proceed to next step automatically after reporting findings.
• Reference: See references/failure_signatures.md for example log patterns.

Step 2: Investigate tpu-device-plugin Metrics Fetch Failures [Low Risk]

Check if tpu-device-plugin is reporting metric fetch failures.

• Tool to use: query_logs
• Filter Template:

resource.type="k8s_container"
AND resource.labels.project_id="{project_id}"
AND resource.labels.cluster_name="{cluster_name}"
AND resource.labels.container_name="tpu-device-plugin"
AND severity=ERROR
AND textPayload:"metrics fetch failed for .* deviceID and .* device path with error: checksum didn't match with the metrics data. Corrupt data found"
AND timestamp >= "{start_time}"
AND timestamp <= "{end_time}"

• Logic: Errors indicating "metrics fetch failed" with "checksum didn't match" suggest vBAR memory corruption.
• Automation: Proceed to next step automatically after reporting findings.

Step 3: Check for Custom Metrics Collection Usage [Low Risk]

Inspect cluster configurations, workloads, or container specs to determine if custom TPU metrics collection mechanisms are deployed.

• Action: Check if custom scripts or agents (e.g., using libtpu.sdk.tpumonitoring) are deployed that frequently query GetHostMetrics from vBAR Control Agent.

• Verification Commands:

  • Kubectl Search (Inspect workload env/specs):

  kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'
  

  • Log Search Filter (query_logs):

  resource.type="k8s_container"
  AND resource.labels.project_id="{project_id}"
  AND resource.labels.cluster_name="{cluster_name}"
  AND textPayload:"libtpu.sdk.tpumonitoring"
  AND timestamp >= "{start_time}"
  AND timestamp <= "{end_time}"
  

• Logic: Confirmation of custom metrics collection helps confirm the race condition hypothesis.

?️ Resolution Workflow

Resolution 1: Temporarily Disable Custom Metrics Collection [High Risk]

If a custom metrics collection agent is identified, recommend disabling it.

• Action: Recommend disabling the custom metrics collector.
• Justification: Prevents reads from vBAR during device resets, stopping crashes and OOMs.

Resolution 2: Await vbar_control_agent Resiliency Update [Low Risk]

Advise that a permanent fix will be available in a future GKE version.

• Action: Recommend upgrading GKE when the fix is available.
• Justification: The updated agent will be resilient to memory corruption and gracefully handle reads from unbound vBARs.

? copypaste checklist

• [ ] Acquire context and compute [{start_time}, {end_time}] window.
• [ ] Check for vbar_control_agent segfaults and OOMs using query_logs.
• [ ] Investigate tpu-device-plugin failures using query_logs.
• [ ] Inspect for custom metrics collection usage.
• [ ] Advise disabling custom metrics collection if applicable.
• [ ] Advise awaiting resiliency update.