[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7D6YVV4AZVGST1G34MWTM94":3,"skill-same:01M7D6YVV4AZVGST1G34MWTM94":75},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":16,"sourceHtmlUrl":17,"categories":18,"keywords":19,"publishedAt":20,"linkOnly":21,"markdownContent":22,"markdownBody":23,"frontMatter":24,"refName":28,"commitSha":29,"skillDirectory":5,"forksCount":30,"repositoryHtmlUrl":31,"repositoryHomepageUrl":32,"ownerHtmlUrl":33,"ownerBlogUrl":34,"ownerDescription":35,"updatedAt":36,"install":37,"resources":50},"01M7D6YVV4AZVGST1G34MWTM94","gke-ai-troubleshooting-jobset-interruption",null,"Diagnoses GKE JobSet interruptions, restarts, and preemptions for AI\u002FML training workloads autonomously. Use when troubleshooting JobSet restart loops, spot VM preemptions, node readiness failures, host VM issues, or coordinator worker crashes. Don't use for general GKE cluster creation, basic workload deployment, or non-JobSet application issues.","tangyixiao\u002Ftangyixiao-skills","tangyixiao","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F115929632?v=4","01M7D582YK6A4M63CAK4CSY4EW","Tang Yixiao","01M7D582Z64DMRSYR872SR34SA",2,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fgke-ai-troubleshooting-jobset-interruption\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fgke-ai-troubleshooting-jobset-interruption\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: gke-ai-troubleshooting-jobset-interruption\nmetadata:\n  version: 1.0.1\n  category: Containers\ndescription: Diagnoses GKE JobSet interruptions, restarts, and preemptions for AI\u002FML\n  training workloads autonomously. Use when troubleshooting JobSet restart loops,\n  spot VM preemptions, node readiness failures, host VM issues, or coordinator worker\n  crashes. Don't use for general GKE cluster creation, basic workload deployment,\n  or non-JobSet application issues.\n---\n\n# GKE JobSet Interruption Troubleshooting\n\nUse this skill to systematically diagnose and resolve JobSet interruptions,\nrestarts, and preemptions on GKE clusters hosting large-scale AI\u002FML workloads.\n\n## Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read [Usage Attribution](references\u002Fattribution.md). For commands executed\ndirectly in the terminal (never in generated scripts, Terraform, or user code),\nprefix `gcloud` with\n`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION=\"agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-jobset-interruption)\"`\nand append `agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-jobset-interruption)` to `User-Agent` headers.\n\n## ⚠️ Prerequisites & Sandbox Rules\n\n-   JobSet metrics package must be enabled in `kube-state-metrics` for your\n    cluster.\n-   Cloud Logging and Cloud Monitoring enabled for the Google Cloud Project.\n-   **Autonomous & Sandbox Execution Rule**: If API queries, tools, or commands\n    encounter `403 Permission Denied`, authentication errors, or network\n    isolation, **do NOT enter authentication or credential troubleshooting\n    loops**. Populate the query templates with the acquired variables\n    (`{project_id}`, `{cluster_name}`, `{workload_name}`, `{start_time}`,\n    `{end_time}`), inspect any locally staged telemetry or mock data files if\n    available, and complete the diagnostic workflow and resolution\n    recommendations autonomously.\n\n## 🔍 Diagnostic Workflow\n\n### Step 0: Context Acquisition & Time Window Definition\n\nIndependently gather context using tools, workspace files, environment details,\nor user prompt context:\n\n-   **Project ID** (`{project_id}`)\n-   **Cluster Name** (`{cluster_name}`)\n-   **Workload Name (JobSet Name)** (`{workload_name}`)\n-   **Workload Namespace** (`{namespace}`)\n-   **Issue Time** (`{issue_time}`)\n\nIf specific variables are not explicitly provided by the user, inspect cluster\nresources or logs to determine them, or use the `{variable}` placeholders\nprovided.\n\n#### Time Handling Rules\n\n1.  **Autonomous Time Window**: If a relative time (e.g., \"X minutes ago\") or no\n    exact timestamp is provided, calculate the query window based on current\n    time or available log timestamps.\n2.  **Window Calculation**: If a timestamp `{issue_time}` is available (or\n    calculated as `T`), set `{start_time}` = `T - 30m` and `{end_time}` = `T +\n    30m`.\n\n--------------------------------------------------------------------------------\n\n### Step 1: Identify JobSet Restarts and Attempts [Low Risk]\n\nVerify if the JobSet is experiencing restart loops and determine the frequency\nof restarts.\n\n#### Visual Chart \u002F MQL Query - restarts\n\n-   **MQL Query Specification**:\n\n    ```mql\n    fetch prometheus_target\n    | metric 'prometheus.googleapis.com\u002Fkube_jobset_restarts\u002Fgauge'\n    | filter resource.cluster_name == '{cluster_name}' && metric.jobset_name == '{workload_name}'\n    | align next_older(1m)\n    | every 1m\n    | group_by [metric.jobset_name], [val: max(value)]\n    ```\n\n#### PromQL Metric Query - restarts\n\n-   **PromQL Query Specification**:\n\n    ```promql\n    kube_jobset_restarts{jobset_name=\"{workload_name}\", cluster=\"{cluster_name}\"}\n    ```\n\n-   **Diagnostic Logic**: A non-zero or increasing value for restarts indicates\n    that the JobSet is being actively restarted by the controller due to worker\n    failure or interruption.\n\n-   **Automation**: Proceed to Step 2 automatically after reporting findings.\n\n--------------------------------------------------------------------------------\n\n### Step 2: Inspect Nodepool Interruptions [Low Risk]\n\nDetermine if the JobSet restarts were triggered by physical nodepool-level\nevents (such as spot preemptions, maintenance, or host terminations).\n\n#### A. Metrics Query (Nodepool Interruption Counts)\n\n##### Visual Chart \u002F MQL Query - interruptions\n\n-   **MQL Query Specification**:\n\n    ```mql\n    fetch k8s_node_pool\n    | metric 'kubernetes.io\u002Fnode_pool\u002Finterruption_count'\n    | filter cluster_name == '{cluster_name}'\n    | align next_older(10m)\n    | every 10m\n    | group_by [metric.interruption_type, metric.interruption_reason, metadata.system.node_pool_name], [val: sum(value)]\n    ```\n\n##### PromQL Query - interruptions\n\n-   **PromQL Query Specification**:\n\n    ```promql\n    sum by (interruption_type, interruption_reason, node_pool_name, cluster_name) (\n      avg_over_time(kubernetes_io:node_pool_interruption_count{cluster_name=\"{cluster_name}\"}[10m])\n    )\n    ```\n\n#### B. Log Query (Nodepool Life Events)\n\n-   **LQL Log Filter Specification**:\n\n    ```sql\n    resource.type=\"gke_nodepool\"\n    AND resource.labels.cluster_name=\"{cluster_name}\"\n    AND timestamp >= \"{start_time}\"\n    AND timestamp \u003C= \"{end_time}\"\n    ```\n\n-   **Diagnostic Logic**:\n\n    -   **PreemptionEvent**: Spot VMs were preempted, or node was scale-down.\n    -   **MaintenanceEvent**: Node pool updated or Google scheduled maintenance.\n    -   **TerminationEvent**: Serious host failures. Check `interruption_reason`\n        or logs for host issues.\n    -   See [Failure Signatures](references\u002Ffailure_signatures.md) for examples\n        of node termination logs and preemption events.\n\n-   **Automation**: Proceed to Step 3 automatically.\n\n--------------------------------------------------------------------------------\n\n### Step 3: Inspect Nodes and Underlying Host VMs [Low Risk]\n\nCorrelate node readiness failures with physical host VMs to see if a single\nfaulty host repeatedly fails coordinator pods.\n\n#### A. Metrics Query (Node Ready Status Check)\n\n##### Visual Chart \u002F MQL Query - node status\n\n-   **MQL Query Specification**:\n\n    ```mql\n    fetch k8s_node\n    | metric 'kubernetes.io\u002Fnode\u002Fstatus_condition'\n    | filter cluster_name == '{cluster_name}' && metric.condition == 'Ready' && metric.status == 'False'\n    | align next_older(1m)\n    | every 1m\n    | group_by [node_name, metadata.user.gke_nodepool], [val: max(value)]\n    ```\n\n##### PromQL Query - node status\n\n-   **PromQL Query Specification**:\n\n    ```promql\n    sum by (status, condition, node_pool_name) (\n      kubernetes_io:node_status_condition{cluster_name=\"{cluster_name}\", condition=\"Ready\", status=\"False\"}\n    )\n    ```\n\n#### B. Metrics Query (Node-to-Host Metadata Topology Correlation)\n\n-   **MQL Query Specification**:\n\n    ```mql\n    fetch k8s_node\n    | metric 'kubernetes.io\u002Fnode\u002Fcpu\u002Ftotal_cores'\n    | filter cluster_name == '{cluster_name}'\n    | align next_older(1m)\n    | every 1m\n    | group_by [node_name, metadata.user.gce_topology_host, metadata.user.gke_nodepool], [val: max(value)]\n    ```\n\n#### C. Log Query (Node Fault Logs)\n\n-   **LQL Log Filter Specification**:\n\n    ```sql\n    resource.type=\"k8s_node\"\n    AND resource.labels.cluster_name=\"{cluster_name}\"\n    AND (textPayload:\"host error\" OR textPayload:\"kernel panic\" OR textPayload:\"hardware failure\" OR textPayload:\"NodeNotReady\")\n    AND timestamp >= \"{start_time}\"\n    AND timestamp \u003C= \"{end_time}\"\n    ```\n\n-   **Diagnostic Logic**: Identify if specific nodes are unhealthy\n    (`Ready=False` or `Unknown`) and correlate them to their GCE physical host\n    ID via `metadata.user.gce_topology_host`. Check if the same host is\n    repeatedly failing.\n\n-   **Automation**: Proceed to Step 4 automatically.\n\n--------------------------------------------------------------------------------\n\n### Step 4: Inspect Pod and Worker \u002F Container Failures [Low Risk]\n\nAnalyze pod status phases and retrieve coordinator worker logs to identify\napplication-level crashes or network deadlocks.\n\n> **Required Execution Order**: You MUST analyze pod status phases (Section A)\n> and unschedulable pod metrics (Section B) to assess overall workload health\n> before inspecting specific worker container logs (Section C).\n\n#### A. Metrics Query (Pod Lifecycle Phases)\n\n##### Visual Chart \u002F MQL Query - pod phase\n\n-   **MQL Query Specification**:\n\n    ```mql\n    fetch k8s_pod\n    | metric 'kubernetes.io\u002Fpod\u002Fstatus\u002Fphase'\n    | filter cluster_name == '{cluster_name}' && pod_name ==~ '{workload_name}.*'\n    | align next_older(10m)\n    | every 10m\n    | group_by [metric.phase], [val: count()]\n    ```\n\n##### PromQL Query - pod phase\n\n-   **PromQL Query Specification**:\n\n    ```promql\n    sum by (phase) (\n      avg_over_time(kube_pod_status_phase{cluster=\"{cluster_name}\", pod=~\"{workload_name}.*\"}[10m])\n    )\n    ```\n\n#### B. Metrics Query (Unschedulable Pod Count)\n\n-   **MQL Query Specification**:\n\n    ```mql\n    fetch k8s_pod\n    | metric 'kubernetes.io\u002Fpod\u002Fstatus\u002Funschedulable'\n    | filter cluster_name == '{cluster_name}' && pod_name ==~ '{workload_name}.*'\n    | align next_older(10m)\n    | every 10m\n    | group_by [pod_name], [val: max(value)]\n    ```\n\n#### C. Log Query (Worker Container Logs)\n\n-   **LQL Log Filter Specification**:\n\n    ```sql\n    resource.type=\"k8s_container\"\n    AND resource.labels.cluster_name=\"{cluster_name}\"\n    AND labels.\"k8s-pod\u002Fjobset_sigs_k8s_io\u002Fjobset-name\"=\"{workload_name}\"\n    AND timestamp >= \"{start_time}\"\n    AND timestamp \u003C= \"{end_time}\"\n    ```\n\n-   **Diagnostic Logic**:\n\n    1.  Check the pod timeline to spot pending or unschedulable pods.\n    2.  Use worker container logs to analyze worker 0 in slice 0 (coordinator)\n        for NCCL timeouts, collective communication issues, or MegaScale hangs.\n\n-   **Automation**: Proceed to Resolution.\n\n--------------------------------------------------------------------------------\n\n## 🛠️ Resolution Workflow\n\n### Resolution 1: Preemption & Autoscaling Optimizations [Low Risk]\n\nIf Step 2 showed high preemption counts on Spot VMs:\n\n-   **Action**: Suggest switching critical long-running training workloads to\n    **GKE Reserved\u002FOn-Demand VMs** or utilizing **Compact Placement Policies**\n    to minimize defragmentation interruptions.\n-   **Justification**: Eliminates spot-market preemptions and reduces training\n    restarts.\n\n### Resolution 2: Quarantine Faulty Host VMs [High Risk]\n\nIf Step 3 identified a specific host ID (`gce-topology-host`) that consistently\nfails or triggers restarts across multiple attempts:\n\n-   **Action**: Recommend cordoning\u002Fdraining the GKE node, deleting the\n    underlying GCE VM instance to trigger instance recreation, and opening a\n    support ticket with Google Cloud Support specifying the physical host ID.\n-   **Justification**: GKE auto-repair will recreate the VM instance on healthy\n    physical hardware, preventing infinite restart loops.\n\n--------------------------------------------------------------------------------\n\n## 📋 Copypaste Checklist\n\n-   [ ] Gather context and compute `{start_time}` (`{issue_time} - 30m`) and\n    `{end_time}` (`{issue_time} + 30m`) window.\n-   [ ] Query JobSet restart attempts.\n-   [ ] Check Nodepool interruptions (spot preemptions vs. hardware\n    terminations).\n-   [ ] Query node-to-host mapping and check node logs for physical host errors.\n-   [ ] Inspect pod timeline status and coordinator worker container logs.\n-   [ ] Recommend appropriate scheduling strategy (On-demand vs Spot) or host VM\n    quarantining.\n","# GKE JobSet Interruption Troubleshooting\n\nUse this skill to systematically diagnose and resolve JobSet interruptions,\nrestarts, and preemptions on GKE clusters hosting large-scale AI\u002FML workloads.\n\n## Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read [Usage Attribution](references\u002Fattribution.md). For commands executed\ndirectly in the terminal (never in generated scripts, Terraform, or user code),\nprefix `gcloud` with\n`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION=\"agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-jobset-interruption)\"`\nand append `agent-skills\u002F1.0 (skill:gke-ai-troubleshooting-jobset-interruption)` to `User-Agent` headers.\n\n## ⚠️ Prerequisites & Sandbox Rules\n\n-   JobSet metrics package must be enabled in `kube-state-metrics` for your\n    cluster.\n-   Cloud Logging and Cloud Monitoring enabled for the Google Cloud Project.\n-   **Autonomous & Sandbox Execution Rule**: If API queries, tools, or commands\n    encounter `403 Permission Denied`, authentication errors, or network\n    isolation, **do NOT enter authentication or credential troubleshooting\n    loops**. Populate the query templates with the acquired variables\n    (`{project_id}`, `{cluster_name}`, `{workload_name}`, `{start_time}`,\n    `{end_time}`), inspect any locally staged telemetry or mock data files if\n    available, and complete the diagnostic workflow and resolution\n    recommendations autonomously.\n\n## 🔍 Diagnostic Workflow\n\n### Step 0: Context Acquisition & Time Window Definition\n\nIndependently gather context using tools, workspace files, environment details,\nor user prompt context:\n\n-   **Project ID** (`{project_id}`)\n-   **Cluster Name** (`{cluster_name}`)\n-   **Workload Name (JobSet Name)** (`{workload_name}`)\n-   **Workload Namespace** (`{namespace}`)\n-   **Issue Time** (`{issue_time}`)\n\nIf specific variables are not explicitly provided by the user, inspect cluster\nresources or logs to determine them, or use the `{variable}` placeholders\nprovided.\n\n#### Time Handling Rules\n\n1.  **Autonomous Time Window**: If a relative time (e.g., \"X minutes ago\") or no\n    exact timestamp is provided, calculate the query window based on current\n    time or available log timestamps.\n2.  **Window Calculation**: If a timestamp `{issue_time}` is available (or\n    calculated as `T`), set `{start_time}` = `T - 30m` and `{end_time}` = `T +\n    30m`.\n\n--------------------------------------------------------------------------------\n\n### Step 1: Identify JobSet Restarts and Attempts [Low Risk]\n\nVerify if the JobSet is experiencing restart loops and determine the frequency\nof restarts.\n\n#### Visual Chart \u002F MQL Query - restarts\n\n-   **MQL Query Specification**:\n\n    ```mql\n    fetch prometheus_target\n    | metric 'prometheus.googleapis.com\u002Fkube_jobset_restarts\u002Fgauge'\n    | filter resource.cluster_name == '{cluster_name}' && metric.jobset_name == '{workload_name}'\n    | align next_older(1m)\n    | every 1m\n    | group_by [metric.jobset_name], [val: max(value)]\n    ```\n\n#### PromQL Metric Query - restarts\n\n-   **PromQL Query Specification**:\n\n    ```promql\n    kube_jobset_restarts{jobset_name=\"{workload_name}\", cluster=\"{cluster_name}\"}\n    ```\n\n-   **Diagnostic Logic**: A non-zero or increasing value for restarts indicates\n    that the JobSet is being actively restarted by the controller due to worker\n    failure or interruption.\n\n-   **Automation**: Proceed to Step 2 automatically after reporting findings.\n\n--------------------------------------------------------------------------------\n\n### Step 2: Inspect Nodepool Interruptions [Low Risk]\n\nDetermine if the JobSet restarts were triggered by physical nodepool-level\nevents (such as spot preemptions, maintenance, or host terminations).\n\n#### A. Metrics Query (Nodepool Interruption Counts)\n\n##### Visual Chart \u002F MQL Query - interruptions\n\n-   **MQL Query Specification**:\n\n    ```mql\n    fetch k8s_node_pool\n    | metric 'kubernetes.io\u002Fnode_pool\u002Finterruption_count'\n    | filter cluster_name == '{cluster_name}'\n    | align next_older(10m)\n    | every 10m\n    | group_by [metric.interruption_type, metric.interruption_reason, metadata.system.node_pool_name], [val: sum(value)]\n    ```\n\n##### PromQL Query - interruptions\n\n-   **PromQL Query Specification**:\n\n    ```promql\n    sum by (interruption_type, interruption_reason, node_pool_name, cluster_name) (\n      avg_over_time(kubernetes_io:node_pool_interruption_count{cluster_name=\"{cluster_name}\"}[10m])\n    )\n    ```\n\n#### B. Log Query (Nodepool Life Events)\n\n-   **LQL Log Filter Specification**:\n\n    ```sql\n    resource.type=\"gke_nodepool\"\n    AND resource.labels.cluster_name=\"{cluster_name}\"\n    AND timestamp >= \"{start_time}\"\n    AND timestamp \u003C= \"{end_time}\"\n    ```\n\n-   **Diagnostic Logic**:\n\n    -   **PreemptionEvent**: Spot VMs were preempted, or node was scale-down.\n    -   **MaintenanceEvent**: Node pool updated or Google scheduled maintenance.\n    -   **TerminationEvent**: Serious host failures. Check `interruption_reason`\n        or logs for host issues.\n    -   See [Failure Signatures](references\u002Ffailure_signatures.md) for examples\n        of node termination logs and preemption events.\n\n-   **Automation**: Proceed to Step 3 automatically.\n\n--------------------------------------------------------------------------------\n\n### Step 3: Inspect Nodes and Underlying Host VMs [Low Risk]\n\nCorrelate node readiness failures with physical host VMs to see if a single\nfaulty host repeatedly fails coordinator pods.\n\n#### A. Metrics Query (Node Ready Status Check)\n\n##### Visual Chart \u002F MQL Query - node status\n\n-   **MQL Query Specification**:\n\n    ```mql\n    fetch k8s_node\n    | metric 'kubernetes.io\u002Fnode\u002Fstatus_condition'\n    | filter cluster_name == '{cluster_name}' && metric.condition == 'Ready' && metric.status == 'False'\n    | align next_older(1m)\n    | every 1m\n    | group_by [node_name, metadata.user.gke_nodepool], [val: max(value)]\n    ```\n\n##### PromQL Query - node status\n\n-   **PromQL Query Specification**:\n\n    ```promql\n    sum by (status, condition, node_pool_name) (\n      kubernetes_io:node_status_condition{cluster_name=\"{cluster_name}\", condition=\"Ready\", status=\"False\"}\n    )\n    ```\n\n#### B. Metrics Query (Node-to-Host Metadata Topology Correlation)\n\n-   **MQL Query Specification**:\n\n    ```mql\n    fetch k8s_node\n    | metric 'kubernetes.io\u002Fnode\u002Fcpu\u002Ftotal_cores'\n    | filter cluster_name == '{cluster_name}'\n    | align next_older(1m)\n    | every 1m\n    | group_by [node_name, metadata.user.gce_topology_host, metadata.user.gke_nodepool], [val: max(value)]\n    ```\n\n#### C. Log Query (Node Fault Logs)\n\n-   **LQL Log Filter Specification**:\n\n    ```sql\n    resource.type=\"k8s_node\"\n    AND resource.labels.cluster_name=\"{cluster_name}\"\n    AND (textPayload:\"host error\" OR textPayload:\"kernel panic\" OR textPayload:\"hardware failure\" OR textPayload:\"NodeNotReady\")\n    AND timestamp >= \"{start_time}\"\n    AND timestamp \u003C= \"{end_time}\"\n    ```\n\n-   **Diagnostic Logic**: Identify if specific nodes are unhealthy\n    (`Ready=False` or `Unknown`) and correlate them to their GCE physical host\n    ID via `metadata.user.gce_topology_host`. Check if the same host is\n    repeatedly failing.\n\n-   **Automation**: Proceed to Step 4 automatically.\n\n--------------------------------------------------------------------------------\n\n### Step 4: Inspect Pod and Worker \u002F Container Failures [Low Risk]\n\nAnalyze pod status phases and retrieve coordinator worker logs to identify\napplication-level crashes or network deadlocks.\n\n> **Required Execution Order**: You MUST analyze pod status phases (Section A)\n> and unschedulable pod metrics (Section B) to assess overall workload health\n> before inspecting specific worker container logs (Section C).\n\n#### A. Metrics Query (Pod Lifecycle Phases)\n\n##### Visual Chart \u002F MQL Query - pod phase\n\n-   **MQL Query Specification**:\n\n    ```mql\n    fetch k8s_pod\n    | metric 'kubernetes.io\u002Fpod\u002Fstatus\u002Fphase'\n    | filter cluster_name == '{cluster_name}' && pod_name ==~ '{workload_name}.*'\n    | align next_older(10m)\n    | every 10m\n    | group_by [metric.phase], [val: count()]\n    ```\n\n##### PromQL Query - pod phase\n\n-   **PromQL Query Specification**:\n\n    ```promql\n    sum by (phase) (\n      avg_over_time(kube_pod_status_phase{cluster=\"{cluster_name}\", pod=~\"{workload_name}.*\"}[10m])\n    )\n    ```\n\n#### B. Metrics Query (Unschedulable Pod Count)\n\n-   **MQL Query Specification**:\n\n    ```mql\n    fetch k8s_pod\n    | metric 'kubernetes.io\u002Fpod\u002Fstatus\u002Funschedulable'\n    | filter cluster_name == '{cluster_name}' && pod_name ==~ '{workload_name}.*'\n    | align next_older(10m)\n    | every 10m\n    | group_by [pod_name], [val: max(value)]\n    ```\n\n#### C. Log Query (Worker Container Logs)\n\n-   **LQL Log Filter Specification**:\n\n    ```sql\n    resource.type=\"k8s_container\"\n    AND resource.labels.cluster_name=\"{cluster_name}\"\n    AND labels.\"k8s-pod\u002Fjobset_sigs_k8s_io\u002Fjobset-name\"=\"{workload_name}\"\n    AND timestamp >= \"{start_time}\"\n    AND timestamp \u003C= \"{end_time}\"\n    ```\n\n-   **Diagnostic Logic**:\n\n    1.  Check the pod timeline to spot pending or unschedulable pods.\n    2.  Use worker container logs to analyze worker 0 in slice 0 (coordinator)\n        for NCCL timeouts, collective communication issues, or MegaScale hangs.\n\n-   **Automation**: Proceed to Resolution.\n\n--------------------------------------------------------------------------------\n\n## 🛠️ Resolution Workflow\n\n### Resolution 1: Preemption & Autoscaling Optimizations [Low Risk]\n\nIf Step 2 showed high preemption counts on Spot VMs:\n\n-   **Action**: Suggest switching critical long-running training workloads to\n    **GKE Reserved\u002FOn-Demand VMs** or utilizing **Compact Placement Policies**\n    to minimize defragmentation interruptions.\n-   **Justification**: Eliminates spot-market preemptions and reduces training\n    restarts.\n\n### Resolution 2: Quarantine Faulty Host VMs [High Risk]\n\nIf Step 3 identified a specific host ID (`gce-topology-host`) that consistently\nfails or triggers restarts across multiple attempts:\n\n-   **Action**: Recommend cordoning\u002Fdraining the GKE node, deleting the\n    underlying GCE VM instance to trigger instance recreation, and opening a\n    support ticket with Google Cloud Support specifying the physical host ID.\n-   **Justification**: GKE auto-repair will recreate the VM instance on healthy\n    physical hardware, preventing infinite restart loops.\n\n--------------------------------------------------------------------------------\n\n## 📋 Copypaste Checklist\n\n-   [ ] Gather context and compute `{start_time}` (`{issue_time} - 30m`) and\n    `{end_time}` (`{issue_time} + 30m`) window.\n-   [ ] Query JobSet restart attempts.\n-   [ ] Check Nodepool interruptions (spot preemptions vs. hardware\n    terminations).\n-   [ ] Query node-to-host mapping and check node logs for physical host errors.\n-   [ ] Inspect pod timeline status and coordinator worker container logs.\n-   [ ] Recommend appropriate scheduling strategy (On-demand vs Spot) or host VM\n    quarantining.\n",{"name":5,"metadata":25,"description":7},{"version":26,"category":27},"1.0.1","Containers","main","2a870f189175a48c39f5f7e26a6d93b0be9cd13b",0,"https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills","https:\u002F\u002Ftangyixiao.github.io\u002Ftangyixiao-skills\u002F","https:\u002F\u002Fgithub.com\u002Ftangyixiao","http:\u002F\u002Fwww.sxyz.net\u002F","I am Tang Yixiao, a student in Shaoxing No.1 High School. I am eager to learn on GitHub and contribute my part to this community.","2026-11-11 15:45:59",{"source":8,"skillName":5,"commands":38,"prompt":48,"zipAvailable":49},[39,42,45],{"tool":40,"command":41},"npx","npx skills add tangyixiao\u002Ftangyixiao-skills --skill gke-ai-troubleshooting-jobset-interruption",{"tool":43,"command":44},"bunx","bunx skills add tangyixiao\u002Ftangyixiao-skills --skill gke-ai-troubleshooting-jobset-interruption",{"tool":46,"command":47},"pnpm","pnpm dlx skills add tangyixiao\u002Ftangyixiao-skills --skill gke-ai-troubleshooting-jobset-interruption","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fgke-ai-troubleshooting-jobset-interruption\u002FSKILL.md",true,[51,57,61,65,69],{"path":52,"resourceType":53,"resourceTypeDesc":54,"sizeBytes":55,"sourceHtmlUrl":56},"INSTALL-SOURCE.json","REFERENCE","参考文档",586,"https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fgke-ai-troubleshooting-jobset-interruption\u002FINSTALL-SOURCE.json",{"path":58,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":60,"sourceHtmlUrl":16},"LICENSE","许可证",11358,{"path":62,"resourceType":53,"resourceTypeDesc":54,"sizeBytes":63,"sourceHtmlUrl":64},"references\u002Fattribution.md",1966,"https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fgke-ai-troubleshooting-jobset-interruption\u002Freferences\u002Fattribution.md",{"path":66,"resourceType":53,"resourceTypeDesc":54,"sizeBytes":67,"sourceHtmlUrl":68},"references\u002Ffailure_signatures.md",2609,"https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fgke-ai-troubleshooting-jobset-interruption\u002Freferences\u002Ffailure_signatures.md",{"path":70,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":73,"sourceHtmlUrl":74},"scripts\u002Fvalidate_queries.sh","SCRIPT","脚本",3728,"https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fgke-ai-troubleshooting-jobset-interruption\u002Fscripts\u002Fvalidate_queries.sh",[76,84,93,101,109,117],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":80,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":20},"01M7D8JK8FYBFNNAJXWD89CFH7","xlsx","Use this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like \"the xlsx in my downloads\") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fxlsx\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fxlsx\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":88,"licenseFileUrl":89,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":20},"01M7D8JG8ERGGQQAMX0DWCG9GR","writing-shape","Writing, exploit: shape raw material into an article, paragraph by paragraph.","MIT","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-shape\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-shape\u002FSKILL.md",[],[],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":88,"licenseFileUrl":97,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":20},"01M7D8JE672GW3EHSE8EF185M4","writing-fragments","Writing, explore: mine raw fragments, no structure yet.","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-fragments\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-fragments\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":88,"licenseFileUrl":105,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":20},"01M7D8JBK572YDFMND8Q6CEYJW","writing-for-agents","Writing documents for agents. Use when creating or editing skills, or modifying AGENTS.md or CLAUDE.md.","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-for-agents\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-for-agents\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":88,"licenseFileUrl":113,"sourceHtmlUrl":114,"categories":115,"keywords":116,"publishedAt":20},"01M7D8J5QPWEXX61EGBKYH73R8","writing-clearly-and-concisely","Use when writing prose humans will read—documentation, commit messages, error messages, explanations, reports, or UI text. Applies Strunk's timeless rules for clearer, stronger, more professional writing.","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-clearly-and-concisely\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-clearly-and-concisely\u002FSKILL.md",[],[],{"skillNo":118,"name":119,"nameZh":6,"description":120,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":88,"licenseFileUrl":121,"sourceHtmlUrl":122,"categories":123,"keywords":124,"publishedAt":20},"01M7D8J36YWC8WZS5DK5AH03CD","writing-beats","Writing, exploit; assemble raw material into a journey of beats, grounding each term before a beat leans on it.","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-beats\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-beats\u002FSKILL.md",[],[]]