[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7D74HT448015NCT2Y5MQ2AA":3,"skill-same:01M7D74HT448015NCT2Y5MQ2AA":65},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":16,"sourceHtmlUrl":17,"categories":18,"keywords":19,"publishedAt":20,"linkOnly":21,"markdownContent":22,"markdownBody":23,"frontMatter":24,"refName":28,"commitSha":29,"skillDirectory":5,"forksCount":30,"repositoryHtmlUrl":31,"repositoryHomepageUrl":32,"ownerHtmlUrl":33,"ownerBlogUrl":34,"ownerDescription":35,"updatedAt":36,"install":37,"resources":50},"01M7D74HT448015NCT2Y5MQ2AA","gke-node-notready",null,"Diagnoses GKE nodes reporting NotReady or Unknown status by inspecting node conditions, events, kubelet\u002Fcontainerd logs, and node metrics, then proposing safe remediations. Use when nodes show NotReady, when the kubelet stops posting node status, or when workloads are evicted or stuck Pending due to node health. Don't use for pod-level application failures (use gke-workload-troubleshooting), autoscaler scale-up\u002Fscale-down decisions (use gke-cluster-autoscaler), or non-GKE compute.","tangyixiao\u002Ftangyixiao-skills","tangyixiao","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F115929632?v=4","01M7D582YK6A4M63CAK4CSY4EW","Tang Yixiao","01M7D582Z64DMRSYR872SR34SA",2,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fgke-node-notready\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fgke-node-notready\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: gke-node-notready\nmetadata:\n  version: 1.0.1\n  category: Containers\ndescription: Diagnoses GKE nodes reporting NotReady or Unknown status by inspecting\n  node conditions, events, kubelet\u002Fcontainerd logs, and node metrics, then proposing\n  safe remediations. Use when nodes show NotReady, when the kubelet stops posting\n  node status, or when workloads are evicted or stuck Pending due to node health.\n  Don't use for pod-level application failures (use gke-workload-troubleshooting),\n  autoscaler scale-up\u002Fscale-down decisions (use gke-cluster-autoscaler), or non-GKE\n  compute.\n---\n\n# GKE Node NotReady Troubleshooting Skill\n\nUse this skill to systematically diagnose why one or more GKE nodes report a\n`NotReady` (or `Ready: Unknown`) status and to propose safe remediations. A\n`NotReady` status means the node's kubelet is not reporting to the control plane\ncorrectly, so Kubernetes stops scheduling new Pods on the node, which can reduce\napplication capacity and cause downtime.\n\nThis skill operates **non-interactively** and enforces a **read-only diagnostics\nboundary**: gather evidence first, then propose a fix (a `kubectl`\u002F`gcloud`\ncommand or a GitOps manifest change) for a human to apply. **Never** mutate the\ncluster, drain, delete, or recreate nodes automatically.\n\n> [!IMPORTANT]\n> First rule out an **expected** `NotReady`: a node that is newly provisioning,\n> upgrading, being repaired, cordoned, or scaling down will transiently report\n> `NotReady`. Only treat it as a fault if it persists beyond the expected window.\n\n## Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read [Usage Attribution](references\u002Fattribution.md). For commands executed\ndirectly in the terminal (never in generated scripts, Terraform, or user code),\nprefix `gcloud` with\n`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION=\"agent-skills\u002F1.0 (skill:gke-node-notready)\"`\nand append `agent-skills\u002F1.0 (skill:gke-node-notready)` to `User-Agent` headers.\n\n## 🔍 Diagnostic Workflow\n\n### Step 0: Context discovery & time window\n\n1.  **Parameter extraction** — obtain `project_id`, `cluster_name`,\n    `cluster_location`, and `node_name` non-interactively from the user prompt,\n    active `SETTINGS.md`, or environment defaults (`kubectl config current-context`,\n    `gcloud config get-value project`).\n2.  **Credentials & fallback** — attempt\n    `gcloud container clusters get-credentials {cluster_name} --location {cluster_location} --project {project_id}`.\n    If the cluster is unreachable or commands fail (sandbox\u002Fdry-run\u002Foffline),\n    present the exact diagnostic commands for a human to run and continue the\n    analysis from the reported symptoms.\n3.  **Time window** — determine `{issue_time}` (explicit, relative, or now) and\n    center a 1-hour window around it (`start = issue_time - 30m`,\n    `end = issue_time + 30m`) for all log\u002Fmetric queries.\n\n--------------------------------------------------------------------------------\n\n### Step 1: Identify NotReady nodes and gather initial status\n\n```bash\n# List nodes and spot NotReady status, node IPs, and container-runtime version.\nkubectl get nodes -o wide\n\n# Inspect the affected node's Conditions and Events (the primary clues).\nkubectl describe node \"{node_name}\"\n```\n\nEquivalent via Cloud Logging (preferred when kubectl access is limited or for\nhistorical events). Open it as a **Logs Explorer deep link** — URL-encode the\nquery and append the project and Step 0 time window:\n`https:\u002F\u002Fconsole.cloud.google.com\u002Flogs\u002Fquery;query={URL_ENCODED_QUERY};timeRange={start}%2F{end}?project={project_id}`\n(encode `\u002F` as `%2F`, or use `;duration=PT1H` for a rolling hour):\n\n```\nresource.type=\"k8s_node\"\nlog_id(\"events\")\nresource.labels.node_name=\"{node_name}\"\nresource.labels.cluster_name=\"{cluster_name}\"\nresource.labels.location=\"{cluster_location}\"\n```\n\n**Interpret the `Conditions` table:**\n\n- `Ready: False` \u002F `Ready: Unknown` with reason `KubeletNotReady` \u002F\n  `NodeStatusUnknown` (\"Kubelet stopped posting node status\") → kubelet or\n  runtime problem; continue to Step 2.\n- `MemoryPressure: True`, `DiskPressure: True`, `PIDPressure: True` → resource\n  exhaustion; go to Step 4b.\n- `NetworkUnavailable: True` → networking\u002FCNI problem; go to Step 4d.\n\n--------------------------------------------------------------------------------\n\n### Step 2: Scan kubelet logs for error signatures\n\nOpen these kubelet logs as a **Logs Explorer deep link** using the same\n`logs\u002Fquery;query={URL_ENCODED_QUERY};timeRange=...?project=...` pattern as Step 1.\n\n```\nresource.type=\"k8s_node\"\nresource.labels.node_name=\"{node_name}\"\nresource.labels.cluster_name=\"{cluster_name}\"\nresource.labels.location=\"{cluster_location}\"\nlog_id(\"kubelet\")\nseverity>=WARNING\n```\n\nAlso review the node's serial-console logs (`log_id(\"serialconsole.googleapis.com\u002Fserial_port_1_output\")`\nor the `resource.type=\"gce_instance\"` serial logs) for kernel `TaskHung`,\nOOM-killer, or disk I\u002FO errors that correlate with the kubelet failures.\n\n--------------------------------------------------------------------------------\n\n### Step 3: Map the signature to a root cause (decision table)\n\n| Kubelet \u002F event signature | Likely root cause | Go to |\n| --- | --- | --- |\n| `runtime is down`, `Container runtime not ready`, errors on `\u002Frun\u002Fcontainerd\u002Fcontainerd.sock` (connection refused \u002F DeadlineExceeded) | Container runtime (`containerd`) down or unresponsive | Step 4a |\n| `Got sys oom event from cadvisor` \u002F kernel OOM-killer in serial logs | System (node-level) OOM killed critical processes | Step 4b |\n| `PLEG is not healthy` | PLEG stalled, usually node overload (CPU\u002Fdisk) | Step 4c |\n| `TaskHung` for `containerd`\u002F`kubelet`, high disk latency | Disk throttling \u002F I\u002FO starvation | Step 4b |\n| `failed to ensure lease`, `leases.coordination.k8s.io ... namespace kube-node-lease ... terminating` | `kube-node-lease` termination → NotReady flapping | Step 4f |\n| Kubelet cannot reach API server, TLS\u002Fdial timeouts | Kubelet ↔ control-plane connectivity | Step 4d |\n| `NetworkPluginNotReady`, `cni plugin not initialized`, `NetworkUnavailable` | CNI plugin failure | Step 4d |\n| Node-critical DaemonSet Pods (CNI, kube-proxy, metadata) blocked from admission | Admission webhook interference | Step 4e |\n| Only generic `NodeNotReady`, no other signature | Cause unclear — widen to Step 4d, then escalate | Escalation |\n\n--------------------------------------------------------------------------------\n\n### Step 4: Branch investigations\n\n#### 4a. Container runtime (`containerd`) down\n\nConfirm the kubelet cannot talk to containerd (socket errors above). Check for\n`containerd` restarts\u002Fcrashes in serial logs. **Remediation (propose, don't run):**\nrecreate\u002Frepair the node (`kubectl drain` then let the node pool recreate it, or\n`gcloud container clusters upgrade`\u002Fnode auto-repair); if it recurs across nodes,\nsuspect a node image or custom DaemonSet interfering with containerd.\n\n#### 4b. Resource pressure & OOM\n\n```bash\n# Node allocatable vs. usage.\nkubectl describe node \"{node_name}\" | sed -n '\u002FAllocated resources\u002F,\u002FEvents\u002Fp'\n```\nCloud Monitoring metrics to inspect (read-only): `kubernetes.io\u002Fnode\u002Fmemory\u002Fused_bytes`,\n`kubernetes.io\u002Fnode\u002Fcpu\u002Fcore_usage_time`, `kubernetes.io\u002Fnode\u002Fephemeral_storage\u002Fused_bytes`.\n- **DiskPressure \u002F disk throttling**: full boot disk or slow PD → increase disk\n  size \u002F use a faster PD type; reduce image\u002Flog churn.\n- **System OOM**: node memory exhausted → set\u002Fraise Pod memory `requests`\u002F`limits`,\n  reduce over-commit, or use larger machine types. Distinguish **system OOM**\n  (node-wide, kills kubelet\u002Fruntime) from **cgroup OOM** (single container).\n- **PIDPressure**: too many processes → cap Pod PIDs \u002F reduce workload density.\n\n#### 4c. PLEG is not healthy\n\n`PLEG is not healthy` almost always means the node is overloaded (CPU saturation,\ndisk latency, or too many Pods\u002Fcontainers per node) so the runtime can't relist\nin time. Correlate with 4b metrics. **Remediation:** reduce node density, add\nCPU\u002Fdisk headroom, or spread workloads.\n\n#### 4d. Networking\n\n```bash\n# Are node-critical networking Pods healthy on this node?\nkubectl get pods -n kube-system -o wide --field-selector spec.nodeName={node_name}\n```\n- **Kubelet ↔ control-plane**: dial\u002FTLS timeouts to the API server → check\n  firewall rules, Private Google Access, authorized networks, and route\u002FNAT\n  changes.\n- **CNI failure** (`NetworkPluginNotReady`): the CNI DaemonSet\n  (`netd`\u002F`calico`\u002Fdataplane) is not running on the node → inspect those Pods'\n  logs\u002Fevents.\n\n#### 4e. Admission webhook interference\n\nA misconfigured\u002Ffailing validating or mutating webhook with a broad scope can\nblock node-critical system Pods from being admitted, keeping the node NotReady.\n\n```bash\nkubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations\n```\nLook for webhooks that intercept `kube-system` \u002F node-critical objects with\n`failurePolicy: Fail`. **Remediation (propose):** scope the webhook out of\n`kube-system`\u002Fnode-critical namespaces or set an appropriate `namespaceSelector`.\n\n#### 4f. `kube-node-lease` termination flapping\n\nIf the node flaps NotReady with `leases.coordination.k8s.io ... namespace\nkube-node-lease ... is being terminated`, the `kube-node-lease` namespace was\ndeleted\u002Fterminating. **Remediation (propose):** do not delete the\n`kube-node-lease` namespace; if terminating, identify the finalizer\u002Factor holding\nit and restore the namespace.\n\n--------------------------------------------------------------------------------\n\n### Step 5: Remediation boundary & escalation\n\n- Present the **root cause + evidence** (the exact conditions, events, log lines,\n  or metrics observed). Provide **Cloud Logging deep links** (and Cloud Monitoring\n  links for the Step 4b metrics) to the supporting entries — using the deep-link\n  pattern from Steps 1-2 — so a human can open the evidence directly.\n- Propose the fix as a **command or GitOps manifest change** for a human to apply\n  — never apply, drain, or recreate nodes automatically.\n**When to escalate (do this instead of proposing more self-service diagnostics):**\n\nEscalate when either:\n\n- the relevant logs are **unavailable** — excluded by a logging filter, or older\n  than the log bucket's retention (the `_Default` bucket defaults to 30 days, so\n  incidents older than that are permanently deleted); or\n- the kubelet\u002Fevent signature is **not in the Step 3 table** and the root cause\n  remains **undetermined** after the branch investigations.\n\nIn those cases, do all three:\n\n1.  **State the limitation plainly** (for example, \"kubelet logs for that date are\n    past the 30-day `_Default` retention window and are permanently deleted\").\n2.  **Summarize the findings you did gather** (node conditions, events, metrics,\n    and any Admin Activity audit logs still in the `_Required` bucket, default\n    400-day retention).\n3.  **Route to GKE support \u002F engineering escalation with those findings.** Do\n    **not** keep proposing further self-service investigation, and do **not**\n    fabricate a diagnosis when the evidence is missing.\n\n--------------------------------------------------------------------------------\n\n## References\n\nThis skill is derived from public Google Cloud documentation:\n\n- [Troubleshoot nodes with the NotReady status](https:\u002F\u002Fcloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready)\n  — node conditions and the kubelet \u002F PLEG \u002F system-OOM \u002F containerd \u002F\n  `kube-node-lease` \u002F CNI \u002F admission-webhook signatures and their remediations.\n- [Troubleshoot node registration](https:\u002F\u002Fcloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-registration)\n  — Node Registration Checker for nodes that never finish registering.\n- [View GKE logs](https:\u002F\u002Fcloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fview-logs)\n  and [Cloud Logging routing overview](https:\u002F\u002Fcloud.google.com\u002Flogging\u002Fdocs\u002Frouting\u002Foverview)\n  — log queries (`resource.type=\"k8s_node\"`, `log_id(\"kubelet\")`) and log-bucket\n  retention (`_Default` 30 days, `_Required` 400 days).\n- [Logs Explorer interface](https:\u002F\u002Fcloud.google.com\u002Flogging\u002Fdocs\u002Fview\u002Flogs-explorer-interface)\n  — building and sharing a query by URL (the `logs\u002Fquery;query=...` deep-link\n  format used above).\n","# GKE Node NotReady Troubleshooting Skill\n\nUse this skill to systematically diagnose why one or more GKE nodes report a\n`NotReady` (or `Ready: Unknown`) status and to propose safe remediations. A\n`NotReady` status means the node's kubelet is not reporting to the control plane\ncorrectly, so Kubernetes stops scheduling new Pods on the node, which can reduce\napplication capacity and cause downtime.\n\nThis skill operates **non-interactively** and enforces a **read-only diagnostics\nboundary**: gather evidence first, then propose a fix (a `kubectl`\u002F`gcloud`\ncommand or a GitOps manifest change) for a human to apply. **Never** mutate the\ncluster, drain, delete, or recreate nodes automatically.\n\n> [!IMPORTANT]\n> First rule out an **expected** `NotReady`: a node that is newly provisioning,\n> upgrading, being repaired, cordoned, or scaling down will transiently report\n> `NotReady`. Only treat it as a fault if it persists beyond the expected window.\n\n## Attribution\n\n**MUST READ**: Before executing any `gcloud`, REST API, or MCP command, you\nMUST read [Usage Attribution](references\u002Fattribution.md). For commands executed\ndirectly in the terminal (never in generated scripts, Terraform, or user code),\nprefix `gcloud` with\n`CLOUDSDK_METRICS_REQUEST_ATTRIBUTION=\"agent-skills\u002F1.0 (skill:gke-node-notready)\"`\nand append `agent-skills\u002F1.0 (skill:gke-node-notready)` to `User-Agent` headers.\n\n## 🔍 Diagnostic Workflow\n\n### Step 0: Context discovery & time window\n\n1.  **Parameter extraction** — obtain `project_id`, `cluster_name`,\n    `cluster_location`, and `node_name` non-interactively from the user prompt,\n    active `SETTINGS.md`, or environment defaults (`kubectl config current-context`,\n    `gcloud config get-value project`).\n2.  **Credentials & fallback** — attempt\n    `gcloud container clusters get-credentials {cluster_name} --location {cluster_location} --project {project_id}`.\n    If the cluster is unreachable or commands fail (sandbox\u002Fdry-run\u002Foffline),\n    present the exact diagnostic commands for a human to run and continue the\n    analysis from the reported symptoms.\n3.  **Time window** — determine `{issue_time}` (explicit, relative, or now) and\n    center a 1-hour window around it (`start = issue_time - 30m`,\n    `end = issue_time + 30m`) for all log\u002Fmetric queries.\n\n--------------------------------------------------------------------------------\n\n### Step 1: Identify NotReady nodes and gather initial status\n\n```bash\n# List nodes and spot NotReady status, node IPs, and container-runtime version.\nkubectl get nodes -o wide\n\n# Inspect the affected node's Conditions and Events (the primary clues).\nkubectl describe node \"{node_name}\"\n```\n\nEquivalent via Cloud Logging (preferred when kubectl access is limited or for\nhistorical events). Open it as a **Logs Explorer deep link** — URL-encode the\nquery and append the project and Step 0 time window:\n`https:\u002F\u002Fconsole.cloud.google.com\u002Flogs\u002Fquery;query={URL_ENCODED_QUERY};timeRange={start}%2F{end}?project={project_id}`\n(encode `\u002F` as `%2F`, or use `;duration=PT1H` for a rolling hour):\n\n```\nresource.type=\"k8s_node\"\nlog_id(\"events\")\nresource.labels.node_name=\"{node_name}\"\nresource.labels.cluster_name=\"{cluster_name}\"\nresource.labels.location=\"{cluster_location}\"\n```\n\n**Interpret the `Conditions` table:**\n\n- `Ready: False` \u002F `Ready: Unknown` with reason `KubeletNotReady` \u002F\n  `NodeStatusUnknown` (\"Kubelet stopped posting node status\") → kubelet or\n  runtime problem; continue to Step 2.\n- `MemoryPressure: True`, `DiskPressure: True`, `PIDPressure: True` → resource\n  exhaustion; go to Step 4b.\n- `NetworkUnavailable: True` → networking\u002FCNI problem; go to Step 4d.\n\n--------------------------------------------------------------------------------\n\n### Step 2: Scan kubelet logs for error signatures\n\nOpen these kubelet logs as a **Logs Explorer deep link** using the same\n`logs\u002Fquery;query={URL_ENCODED_QUERY};timeRange=...?project=...` pattern as Step 1.\n\n```\nresource.type=\"k8s_node\"\nresource.labels.node_name=\"{node_name}\"\nresource.labels.cluster_name=\"{cluster_name}\"\nresource.labels.location=\"{cluster_location}\"\nlog_id(\"kubelet\")\nseverity>=WARNING\n```\n\nAlso review the node's serial-console logs (`log_id(\"serialconsole.googleapis.com\u002Fserial_port_1_output\")`\nor the `resource.type=\"gce_instance\"` serial logs) for kernel `TaskHung`,\nOOM-killer, or disk I\u002FO errors that correlate with the kubelet failures.\n\n--------------------------------------------------------------------------------\n\n### Step 3: Map the signature to a root cause (decision table)\n\n| Kubelet \u002F event signature | Likely root cause | Go to |\n| --- | --- | --- |\n| `runtime is down`, `Container runtime not ready`, errors on `\u002Frun\u002Fcontainerd\u002Fcontainerd.sock` (connection refused \u002F DeadlineExceeded) | Container runtime (`containerd`) down or unresponsive | Step 4a |\n| `Got sys oom event from cadvisor` \u002F kernel OOM-killer in serial logs | System (node-level) OOM killed critical processes | Step 4b |\n| `PLEG is not healthy` | PLEG stalled, usually node overload (CPU\u002Fdisk) | Step 4c |\n| `TaskHung` for `containerd`\u002F`kubelet`, high disk latency | Disk throttling \u002F I\u002FO starvation | Step 4b |\n| `failed to ensure lease`, `leases.coordination.k8s.io ... namespace kube-node-lease ... terminating` | `kube-node-lease` termination → NotReady flapping | Step 4f |\n| Kubelet cannot reach API server, TLS\u002Fdial timeouts | Kubelet ↔ control-plane connectivity | Step 4d |\n| `NetworkPluginNotReady`, `cni plugin not initialized`, `NetworkUnavailable` | CNI plugin failure | Step 4d |\n| Node-critical DaemonSet Pods (CNI, kube-proxy, metadata) blocked from admission | Admission webhook interference | Step 4e |\n| Only generic `NodeNotReady`, no other signature | Cause unclear — widen to Step 4d, then escalate | Escalation |\n\n--------------------------------------------------------------------------------\n\n### Step 4: Branch investigations\n\n#### 4a. Container runtime (`containerd`) down\n\nConfirm the kubelet cannot talk to containerd (socket errors above). Check for\n`containerd` restarts\u002Fcrashes in serial logs. **Remediation (propose, don't run):**\nrecreate\u002Frepair the node (`kubectl drain` then let the node pool recreate it, or\n`gcloud container clusters upgrade`\u002Fnode auto-repair); if it recurs across nodes,\nsuspect a node image or custom DaemonSet interfering with containerd.\n\n#### 4b. Resource pressure & OOM\n\n```bash\n# Node allocatable vs. usage.\nkubectl describe node \"{node_name}\" | sed -n '\u002FAllocated resources\u002F,\u002FEvents\u002Fp'\n```\nCloud Monitoring metrics to inspect (read-only): `kubernetes.io\u002Fnode\u002Fmemory\u002Fused_bytes`,\n`kubernetes.io\u002Fnode\u002Fcpu\u002Fcore_usage_time`, `kubernetes.io\u002Fnode\u002Fephemeral_storage\u002Fused_bytes`.\n- **DiskPressure \u002F disk throttling**: full boot disk or slow PD → increase disk\n  size \u002F use a faster PD type; reduce image\u002Flog churn.\n- **System OOM**: node memory exhausted → set\u002Fraise Pod memory `requests`\u002F`limits`,\n  reduce over-commit, or use larger machine types. Distinguish **system OOM**\n  (node-wide, kills kubelet\u002Fruntime) from **cgroup OOM** (single container).\n- **PIDPressure**: too many processes → cap Pod PIDs \u002F reduce workload density.\n\n#### 4c. PLEG is not healthy\n\n`PLEG is not healthy` almost always means the node is overloaded (CPU saturation,\ndisk latency, or too many Pods\u002Fcontainers per node) so the runtime can't relist\nin time. Correlate with 4b metrics. **Remediation:** reduce node density, add\nCPU\u002Fdisk headroom, or spread workloads.\n\n#### 4d. Networking\n\n```bash\n# Are node-critical networking Pods healthy on this node?\nkubectl get pods -n kube-system -o wide --field-selector spec.nodeName={node_name}\n```\n- **Kubelet ↔ control-plane**: dial\u002FTLS timeouts to the API server → check\n  firewall rules, Private Google Access, authorized networks, and route\u002FNAT\n  changes.\n- **CNI failure** (`NetworkPluginNotReady`): the CNI DaemonSet\n  (`netd`\u002F`calico`\u002Fdataplane) is not running on the node → inspect those Pods'\n  logs\u002Fevents.\n\n#### 4e. Admission webhook interference\n\nA misconfigured\u002Ffailing validating or mutating webhook with a broad scope can\nblock node-critical system Pods from being admitted, keeping the node NotReady.\n\n```bash\nkubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations\n```\nLook for webhooks that intercept `kube-system` \u002F node-critical objects with\n`failurePolicy: Fail`. **Remediation (propose):** scope the webhook out of\n`kube-system`\u002Fnode-critical namespaces or set an appropriate `namespaceSelector`.\n\n#### 4f. `kube-node-lease` termination flapping\n\nIf the node flaps NotReady with `leases.coordination.k8s.io ... namespace\nkube-node-lease ... is being terminated`, the `kube-node-lease` namespace was\ndeleted\u002Fterminating. **Remediation (propose):** do not delete the\n`kube-node-lease` namespace; if terminating, identify the finalizer\u002Factor holding\nit and restore the namespace.\n\n--------------------------------------------------------------------------------\n\n### Step 5: Remediation boundary & escalation\n\n- Present the **root cause + evidence** (the exact conditions, events, log lines,\n  or metrics observed). Provide **Cloud Logging deep links** (and Cloud Monitoring\n  links for the Step 4b metrics) to the supporting entries — using the deep-link\n  pattern from Steps 1-2 — so a human can open the evidence directly.\n- Propose the fix as a **command or GitOps manifest change** for a human to apply\n  — never apply, drain, or recreate nodes automatically.\n**When to escalate (do this instead of proposing more self-service diagnostics):**\n\nEscalate when either:\n\n- the relevant logs are **unavailable** — excluded by a logging filter, or older\n  than the log bucket's retention (the `_Default` bucket defaults to 30 days, so\n  incidents older than that are permanently deleted); or\n- the kubelet\u002Fevent signature is **not in the Step 3 table** and the root cause\n  remains **undetermined** after the branch investigations.\n\nIn those cases, do all three:\n\n1.  **State the limitation plainly** (for example, \"kubelet logs for that date are\n    past the 30-day `_Default` retention window and are permanently deleted\").\n2.  **Summarize the findings you did gather** (node conditions, events, metrics,\n    and any Admin Activity audit logs still in the `_Required` bucket, default\n    400-day retention).\n3.  **Route to GKE support \u002F engineering escalation with those findings.** Do\n    **not** keep proposing further self-service investigation, and do **not**\n    fabricate a diagnosis when the evidence is missing.\n\n--------------------------------------------------------------------------------\n\n## References\n\nThis skill is derived from public Google Cloud documentation:\n\n- [Troubleshoot nodes with the NotReady status](https:\u002F\u002Fcloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-notready)\n  — node conditions and the kubelet \u002F PLEG \u002F system-OOM \u002F containerd \u002F\n  `kube-node-lease` \u002F CNI \u002F admission-webhook signatures and their remediations.\n- [Troubleshoot node registration](https:\u002F\u002Fcloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Ftroubleshooting\u002Fnode-registration)\n  — Node Registration Checker for nodes that never finish registering.\n- [View GKE logs](https:\u002F\u002Fcloud.google.com\u002Fkubernetes-engine\u002Fdocs\u002Fhow-to\u002Fview-logs)\n  and [Cloud Logging routing overview](https:\u002F\u002Fcloud.google.com\u002Flogging\u002Fdocs\u002Frouting\u002Foverview)\n  — log queries (`resource.type=\"k8s_node\"`, `log_id(\"kubelet\")`) and log-bucket\n  retention (`_Default` 30 days, `_Required` 400 days).\n- [Logs Explorer interface](https:\u002F\u002Fcloud.google.com\u002Flogging\u002Fdocs\u002Fview\u002Flogs-explorer-interface)\n  — building and sharing a query by URL (the `logs\u002Fquery;query=...` deep-link\n  format used above).\n",{"name":5,"metadata":25,"description":7},{"version":26,"category":27},"1.0.1","Containers","main","2a870f189175a48c39f5f7e26a6d93b0be9cd13b",0,"https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills","https:\u002F\u002Ftangyixiao.github.io\u002Ftangyixiao-skills\u002F","https:\u002F\u002Fgithub.com\u002Ftangyixiao","http:\u002F\u002Fwww.sxyz.net\u002F","I am Tang Yixiao, a student in Shaoxing No.1 High School. I am eager to learn on GitHub and contribute my part to this community.","2026-11-11 15:45:59",{"source":8,"skillName":5,"commands":38,"prompt":48,"zipAvailable":49},[39,42,45],{"tool":40,"command":41},"npx","npx skills add tangyixiao\u002Ftangyixiao-skills --skill gke-node-notready",{"tool":43,"command":44},"bunx","bunx skills add tangyixiao\u002Ftangyixiao-skills --skill gke-node-notready",{"tool":46,"command":47},"pnpm","pnpm dlx skills add tangyixiao\u002Ftangyixiao-skills --skill gke-node-notready","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fgke-node-notready\u002FSKILL.md",true,[51,57,61],{"path":52,"resourceType":53,"resourceTypeDesc":54,"sizeBytes":55,"sourceHtmlUrl":56},"INSTALL-SOURCE.json","REFERENCE","参考文档",511,"https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fgke-node-notready\u002FINSTALL-SOURCE.json",{"path":58,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":60,"sourceHtmlUrl":16},"LICENSE","许可证",11358,{"path":62,"resourceType":53,"resourceTypeDesc":54,"sizeBytes":63,"sourceHtmlUrl":64},"references\u002Fattribution.md",1891,"https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fgke-node-notready\u002Freferences\u002Fattribution.md",[66,74,83,91,99,107],{"skillNo":67,"name":68,"nameZh":6,"description":69,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":70,"sourceHtmlUrl":71,"categories":72,"keywords":73,"publishedAt":20},"01M7D8JK8FYBFNNAJXWD89CFH7","xlsx","Use this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like \"the xlsx in my downloads\") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fxlsx\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fxlsx\u002FSKILL.md",[],[],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":78,"licenseFileUrl":79,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":20},"01M7D8JG8ERGGQQAMX0DWCG9GR","writing-shape","Writing, exploit: shape raw material into an article, paragraph by paragraph.","MIT","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-shape\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-shape\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":78,"licenseFileUrl":87,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":20},"01M7D8JE672GW3EHSE8EF185M4","writing-fragments","Writing, explore: mine raw fragments, no structure yet.","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-fragments\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-fragments\u002FSKILL.md",[],[],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":78,"licenseFileUrl":95,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":20},"01M7D8JBK572YDFMND8Q6CEYJW","writing-for-agents","Writing documents for agents. Use when creating or editing skills, or modifying AGENTS.md or CLAUDE.md.","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-for-agents\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-for-agents\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":78,"licenseFileUrl":103,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":20},"01M7D8J5QPWEXX61EGBKYH73R8","writing-clearly-and-concisely","Use when writing prose humans will read—documentation, commit messages, error messages, explanations, reports, or UI text. Applies Strunk's timeless rules for clearer, stronger, more professional writing.","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-clearly-and-concisely\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-clearly-and-concisely\u002FSKILL.md",[],[],{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":78,"licenseFileUrl":111,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":20},"01M7D8J36YWC8WZS5DK5AH03CD","writing-beats","Writing, exploit; assemble raw material into a journey of beats, grounding each term before a beat leans on it.","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-beats\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Ftangyixiao\u002Ftangyixiao-skills\u002Fblob\u002Fmain\u002Fwriting-beats\u002FSKILL.md",[],[]]