[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7A7YFNXPDYMDFWAXVZMED0P":3,"skill-same:01M7A7YFNXPDYMDFWAXVZMED0P":139},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":43,"commitSha":44,"skillDirectory":45,"forksCount":46,"repositoryHtmlUrl":47,"repositoryHomepageUrl":6,"ownerHtmlUrl":48,"ownerBlogUrl":6,"ownerDescription":49,"updatedAt":50,"install":51,"resources":64},"01M7A7YFNXPDYMDFWAXVZMED0P","eks-pod-troubleshooter",null,"Diagnoses why Kubernetes pods fail on Amazon EKS — CrashLoopBackOff, ImagePullBackOff \u002F ErrImagePull, Pending (FailedScheduling), OOMKilled, unhealthy probes (Liveness\u002FReadiness), and Init:CrashLoopBackOff — via a symptom-to-cause decision tree that combines kubectl describe pod events, container lastState (exit code, OOMKilled), kubectl logs --previous for application errors, kubectl get events timeline, ECR auth and node-role permissions, node taints and resource pressure, and probe path\u002Fport misconfiguration. Emits a deterministic verdict (ROOT_CAUSE_FOUND | NEED_MORE_INFO | ESCALATE) with the specific failure category and evidence from kubectl describe \u002F logs \u002F get-events \u002F aws eks describe-cluster. Use when a pod is stuck in Pending, crash-looping, cannot pull an image, fails liveness\u002Freadiness probes, is OOMKilled, or has a failing init container.","jhkchan\u002Faws-agent-skills","jhkchan","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1083902?v=4","01M7A2SVQYP9GQTPTDXP5JKYQ4","Jacky Chan","01M7A2SVS117YWCTZABB75HZR1",4,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: eks-pod-troubleshooter\ndescription: Diagnoses why Kubernetes pods fail on Amazon EKS — CrashLoopBackOff, ImagePullBackOff \u002F ErrImagePull, Pending (FailedScheduling), OOMKilled, unhealthy probes (Liveness\u002FReadiness), and Init:CrashLoopBackOff — via a symptom-to-cause decision tree that combines kubectl describe pod events, container lastState (exit code, OOMKilled), kubectl logs --previous for application errors, kubectl get events timeline, ECR auth and node-role permissions, node taints and resource pressure, and probe path\u002Fport misconfiguration. Emits a deterministic verdict (ROOT_CAUSE_FOUND | NEED_MORE_INFO | ESCALATE) with the specific failure category and evidence from kubectl describe \u002F logs \u002F get-events \u002F aws eks describe-cluster. Use when a pod is stuck in Pending, crash-looping, cannot pull an image, fails liveness\u002Freadiness probes, is OOMKilled, or has a failing init container.\nlicense: Apache-2.0\ncompatibility: Agent runtime that reads SKILL.md (Claude Code, Cursor, Windsurf, Codex, Gemini). Offline diagnosis works on supplied kubectl describe \u002F logs \u002F get-events output. Live-cluster diagnosis uses kubectl get pods, describe pod, logs (incl. --previous and -c \u003Ccontainer>), get events, top nodes, top pods, aws eks describe-cluster, aws ec2 describe-vpc-endpoints, and aws ecr describe-images (kubectl v1.27+, AWS CLI v2, SSO or key-based credentials, kubeconfig pointing at the EKS cluster).\nmetadata:\n  domain: aws-cloudops\n  complexity: high\n  requires_llm: 'true'\n  phase: '2'\n  supports_pipeline: 'true'\n  entry_point: 'false'\n  family: Compute\n  task_type: troubleshoot\n  skill_class: capability\n  lifecycle_status: active\n  verdict_shape: ROOT_CAUSE_FOUND | NEED_MORE_INFO | ESCALATE\n  when_to_use: Diagnosing why a Kubernetes pod on EKS is stuck Pending, crash-looping (CrashLoopBackOff), cannot pull a container image (ImagePullBackOff \u002F ErrImagePull), is repeatedly OOMKilled, fails liveness or readiness probes while Running, or has an init container stuck in Init:CrashLoopBackOff; interpreting kubectl describe pod events, container lastState.exitCode, lastState.reason, and conditions.\n  activation_triggers: EKS pod CrashLoopBackOff, EKS pod ImagePullBackOff, EKS pod ErrImagePull, EKS pod Pending, EKS pod FailedScheduling, EKS pod OOMKilled, EKS pod liveness probe failing, EKS pod readiness probe failing, EKS init container failing, EKS pod keeps restarting, kubectl pod status\n  invocation_schema: 'Input: either (a) a symptom description (pod name, namespace, observed state, restart count, any error strings from kubectl get pods), OR (b) a live-cluster scenario where the agent runs kubectl describe pod \u002F logs \u002F get events to gather evidence. Output: a deterministic INCIDENT \u002F VERDICT \u002F ROOT_CAUSE \u002F EVIDENCE \u002F REMEDIATION block where VERDICT ∈ {ROOT_CAUSE_FOUND, NEED_MORE_INFO, ESCALATE} and ROOT_CAUSE names the specific failure category (CRASH_LOOP \u002F IMAGE_PULL \u002F PENDING \u002F OOM \u002F PROBE_FAILURE \u002F INIT_FAILURE) and the offending config element.'\n  version: 0.1.0\n  author: Jacky Chan — AWS Community Builder\n  keywords: EKS, Kubernetes, pod, CrashLoopBackOff, ImagePullBackOff, ErrImagePull, Pending, FailedScheduling, OOMKilled, Liveness probe, Readiness probe, Init container, ECR, kubectl, pod lifecycle, taints, tolerations, node affinity\n  tags: eks, kubernetes, compute, troubleshoot, pod-failure, crashloop, oom, image-pull, health-check\n---\n\n# EKS Pod Troubleshooter\n\n## Activation\n\nActivate this skill when the user reports a Kubernetes pod failure on an\nEKS cluster. Trigger phrases: \"EKS pod CrashLoopBackOff\", \"EKS pod\nImagePullBackOff\", \"EKS pod Pending\", \"EKS pod FailedScheduling\", \"EKS\npod OOMKilled\", \"EKS pod liveness probe failing\", \"EKS pod readiness\nprobe failing\", \"EKS init container failing\", \"EKS pod keeps restarting\",\n\"kubectl pod status\".\n\n## Mindset\n\n**One-line takeaway:** every Kubernetes pod failure surfaces in three\nplaces — `kubectl describe pod` (Events + container lastState), `kubectl\nlogs --previous` (application error), and `kubectl get events` (timeline\nof what the kubelet and scheduler did). Read all three before declaring\na root cause.\n\nMoved verbatim to `references\u002Fadvanced-patterns.md` — see \"**Mindset — three EKS-specific facts**\" (load on demand).\nApplies when: you need the deeper reasoning behind the three-evidence-sources rule.\n\n## Quick reference — symptom to failure category\n\n| Observed state | Failure category | First probe |\n|---|---|---|\n| `STATUS=Pending`, `REASON=FailedScheduling` in events | PENDING | `kubectl describe pod` → Events: `FailedScheduling` message; check `top nodes` capacity |\n| `STATUS=ImagePullBackOff` \u002F `ErrImagePull` | IMAGE_PULL | `kubectl describe pod` Events: `Failed to pull image` message; verify ECR auth |\n| `STATUS=CrashLoopBackOff`, restart count climbing | CRASH_LOOP | `kubectl logs \u003Cpod> --previous` for the application error |\n| Container `Last State: Terminated`, `Reason: OOMKilled`, exit 137 | OOM | `kubectl describe pod` containers[].lastState; check `resources.limits.memory` |\n| `STATUS=Running` but liveness\u002Freadiness probes failing; restart count climbing | PROBE_FAILURE | `kubectl describe pod` Events: `Liveness probe failed` \u002F `Readiness probe failed`; verify probe path\u002Fport |\n| `STATUS=Init:CrashLoopBackOff` \u002F `Init:Error` | INIT_FAILURE | `kubectl logs \u003Cpod> -c \u003Cinit-container> --previous` |\n\nSee the ordered steps below for the full diagnostic walk.\n\n## Quick navigation\n\n- **Step 0** — Capture the failure signal (pod name, namespace, phase).\n- **Step 1** — Map the symptom to a category letter (A-F).\n- **Step 2** — PENDING diagnostic (FailedScheduling).\n- **Step 3** — IMAGE_PULL diagnostic (ImagePullBackOff).\n- **Step 4** — CRASH_LOOP diagnostic (CrashLoopBackOff, exit code).\n- **Step 5** — OOM diagnostic (OOMKilled).\n- **Step 6** — PROBE_FAILURE diagnostic (Liveness\u002FReadiness).\n- **Step 7** — INIT_FAILURE diagnostic (Init:CrashLoopBackOff).\n- **Step 8** — Root-cause catalog (top patterns + canonical fixes).\n- **Step 9** — Verify the fix.\n- **Step 10** — Decide VERDICT (ROOT_CAUSE_FOUND \u002F NEED_MORE_INFO \u002F ESCALATE).\n\n## Process — Diagnostic decision tree (apply in order)\n\n### Step 0: Capture the failure signal\n\nGather these three pieces. Each step below branches on which is present.\n\n| Signal | Source | Why required |\n|---|---|---|\n| **Pod name + namespace** | User-provided or `kubectl get pods -A --field-selector status.phase!=Running` | All describe\u002Flogs calls need this |\n| **Phase + container statuses + restart count** | `kubectl get pod \u003Cpod> -n \u003Cns> -o wide` | Drives the symptom category |\n| **Events + lastState** | `kubectl describe pod \u003Cpod> -n \u003Cns>` | Narrows from symptom to cause |\n\nIf the user has not provided the pod name or namespace, output:\n\n```text\nINCIDENT: \u003Cnamespace>\u002F\u003Cpod> — \u003Csymptom>\nVERDICT: NEED_MORE_INFO\nREASON: Cannot diagnose without the pod name and namespace. Identify\nthe failing pod with: kubectl get pods -A --field-selector\nstatus.phase!=Running  (or: kubectl get pods -n \u003Cns>\n--field-selector status.containerStatuses[*].restartCount>0)\nMISSING:\n  - Pod name and namespace\n  - Last observed status (Pending \u002F ContainerCreating \u002F\n    Running \u002F CrashLoopBackOff \u002F ImagePullBackOff)\n```\n\nIf the user reports \"pods keep failing\" but does not know which pod,\nask for the namespace and deployment\u002Fpod label. Then run:\n\nMoved verbatim to `references\u002Fdiagnostic-commands.md` — see \"**Step 0 — locate the failing pod(s) when name\u002Fnamespace is unknown**\" (load on demand).\nApplies when: the user cannot name the failing pod.\n\n### Step 1: Identify the symptom category\n\nMap the observed state to one of six categories. Each category has a\ndifferent diagnostic walk in Steps 2-7.\n\n| Category | Signature | Diagnostic step |\n|---|---|---|\n| **A. PENDING** | `STATUS=Pending`; Events contain `FailedScheduling` | Step 2 |\n| **B. IMAGE_PULL** | `STATUS=ImagePullBackOff` or `ErrImagePull`; Events contain `Failed to pull image` \u002F `rpc error: code = Unknown desc = Error response from daemon` | Step 3 |\n| **C. CRASH_LOOP** | `STATUS=CrashLoopBackOff`; `status.containerStatuses[].restartCount` climbing; `lastState` has exit code 1\u002F255\u002Fetc | Step 4 |\n| **D. OOM** | `status.containerStatuses[].lastState.reason: OOMKilled`; exit code 137 | Step 5 |\n| **E. PROBE_FAILURE** | `STATUS=Running` but Events show `Liveness probe failed` \u002F `Readiness probe failed`; restart count climbing or pod stuck not-ready | Step 6 |\n| **F. INIT_FAILURE** | `STATUS=Init:CrashLoopBackOff` \u002F `Init:Error` \u002F `Init:ImagePullBackOff`; `status.initContainerStatuses` populated | Step 7 |\n\n**Earliest-transition rule.** If the symptom matches more than one\ncategory, pick the EARLIEST transition failure in the pod lifecycle.\nPENDING precedes IMAGE_PULL precedes INIT_FAILURE precedes CRASH_LOOP \u002F\nOOM \u002F PROBE_FAILURE. The pod cannot be crash-looping if it never\nscheduled.\n\n### Step 2: PENDING diagnostic (FailedScheduling)\n\nA pod in `Pending` has not been bound to a node. The failure is in the\nscheduler: insufficient resources, taints without tolerations, node\naffinity\u002Fselector with no match, or a pending PersistentVolumeClaim.\n\n| Events message sub-string | Root cause | Probe |\n|---|---|---|\n| `Insufficient cpu` \u002F `Insufficient memory` | Node free resources \u003C pod `resources.requests` | `kubectl top nodes`; `kubectl describe nodes` Allocatable vs Requests |\n| `node(s) had taints that the pod didn't tolerate` | All nodes tainted; pod has no matching toleration | `kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints` |\n| `node(s) didn't match Pod's node affinity\u002Fselector` | `nodeSelector` or `nodeAffinity` excludes all nodes | `kubectl get nodes --show-labels`; compare to pod `nodeSelector` \u002F `requiredDuringScheduling` |\n| `node(s) were unschedulable` | All matching nodes are `cordoned` (Unschedulable) | `kubectl get nodes -o wide`; look for `SchedulingDisabled` \u002F `NotReady` |\n| `pod has unbound immediate PersistentVolumeClaims` | PVC is Pending — no StorageClass match, exhausted volume IDs, or EBS AZ mismatch | `kubectl get pvc -n \u003Cns>`; `kubectl get pv`; `kubectl describe pvc \u003Cpvc> -n \u003Cns>` |\n| `Insufficient \u003Cdevice>` (e.g. gpu) | Limited resource (NVIDIA GPU) — pod requests a resource no node advertutes via device plugin | `kubectl describe nodes` Capacity; verify device-plugin DaemonSet |\n| `maximum graceful delete duration` etc. (rare) | Pod stuck in termination — different walk; not Pending | Check `deletionTimestamp` |\n\nMoved verbatim to `references\u002Fdiagnostic-commands.md` — see \"**Step 2 — PENDING (FailedScheduling) commands**\" (load on demand).\nApplies when: diagnosing a Pending pod on a live cluster.\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 2 — PENDING common fix patterns**\" (load on demand).\nApplies when: remediating a FailedScheduling finding.\n\n### Step 3: IMAGE_PULL diagnostic (ImagePullBackOff \u002F ErrImagePull)\n\nThe kubelet cannot pull the container image. On EKS the cause is almost\nalways one of: image tag typo, ECR auth from the node role, missing VPC\nendpoint for ECR in a private subnet, or a cross-account ECR repo\nwithout a `Private` namespace identity policy.\n\n| Events message sub-string | Root cause | Probe |\n|---|---|---|\n| `manifest unknown` \u002F `Requested image not found` | Tag does not exist (typo, never pushed, or ECR lifecycle policy purged it) | `aws ecr describe-images --repository-name \u003Crepo> --image-ids imageTag=\u003Ctag>` |\n| `not authorized` \u002F `403 Forbidden` | ECR auth failing — node IAM role lacks `ecr:GetDownloadUrlForLayer` \u002F `ecr:BatchGetImage`, OR cross-account repo policy missing caller | `aws iam simulate-principal-policy` against the *node* instance role ARN |\n| `RequestError: send request failed` \u002F `i\u002Fo timeout` | Network: no route to ECR (private subnet, missing VPC endpoint, broken NAT) | `aws ec2 describe-vpc-endpoints`; verify route table 0.0.0.0\u002F0 via NAT or `com.amazonaws.\u003Cregion>.ecr.dkr` endpoint |\n| `no space left on device` | Node disk full (image cache) | `kubectl debug node\u002F\u003Cnode> -it --image=busybox -- df -h \u002Fvar\u002Flib\u002Fcontainerd`; drain and roll the node |\n| Pull works for other pods on same node but not this pod | Cross-account ECR — image ARN includes a different account ID | Compare `image: \u003Cacct>.dkr.ecr.\u003Cregion>.amazonaws.com\u002F...`; verify node role trust + repo policy BOTH allow caller |\n| Pull works on newer nodes but fails on older | ECR token rotation + stale `aws-ecr-credential-provider` config on old AMI | Check node AMI version; upgrade the node group |\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 3 — IMAGE_PULL diagnostic walk**\" (load on demand).\nApplies when: walking an ImagePullBackOff \u002F ErrImagePull to its cause.\n\nMoved verbatim to `references\u002Fdiagnostic-commands.md` — see \"**Step 3 — IMAGE_PULL commands**\" (load on demand).\nApplies when: diagnosing image-pull failures on a live cluster.\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 3 — IMAGE_PULL common fix patterns**\" (load on demand).\nApplies when: remediating an image-pull finding.\n\n### Step 4: CRASH_LOOP diagnostic (CrashLoopBackOff)\n\nA CrashLoopBackOff means the kubelet restarted the container repeatedly\nwith exponential backoff. The exit code and application logs narrow the\ncause; `kubectl describe pod` Events alone are too generic.\n\n| Exit code | Meaning | First probe |\n|---|---|---|\n| 0 | Clean exit — application thinks it is done | `kubectl logs --previous` — is the workload a batch? Should `restartPolicy: Never` or `OnFailure` be set instead of `Always`? |\n| 1 | General application error | `kubectl logs \u003Cpod> -c \u003Ccontainer> --previous` — exception traceback |\n| 2 \u002F 126 \u002F 127 | `command not found` \u002F `exec format error` \u002F wrong entrypoint | Check `command` \u002F `args` in pod spec; verify image architecture (ARM64 vs AMD64) |\n| 137 | SIGKILL — usually OOM killer | Step 5 (OOM) |\n| 139 | SIGSEGV — native crash, JNI \u002F cgo bug | `kubectl logs --previous` for native stack; image arch mismatch |\n| 255 | Application-defined error | `kubectl logs --previous` |\n| Empty + Events mention `CreateContainerConfigError` | kubelet could not construct the container spec — missing ConfigMap\u002FSecret referenced in `envFrom` or `env.valueFrom` | `kubectl describe pod` Events; verify referenced Secret\u002FConfigMap exists in the namespace |\n\nMoved verbatim to `references\u002Fdiagnostic-commands.md` — see \"**Step 4 — CRASH_LOOP commands**\" (load on demand).\nApplies when: diagnosing a CrashLoopBackOff on a live cluster.\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 4 — CRASH_LOOP common root causes**\" (load on demand).\nApplies when: the exit code or log pattern does not match an obvious cause.\n\n### Step 5: OOM diagnostic (OOMKilled)\n\nOut-of-memory has two flavors:\n\n### 5a. Container OOM (`OOMKilled`, exit 137)\n\nThe container exceeded its `resources.limits.memory` cgroup. The kernel\nOOM-killer (cgroup v2) or Docker OOM-killer (cgroup v1) SIGKILLs the\nprocess.\n\n| Cause | Probe | Fix |\n|---|---|---|\n| `limits.memory` too low for the workload | `kubectl describe pod` lastState; compare to application RSS | Raise `resources.limits.memory` |\n| Memory leak in application | `kubectl top pod \u003Cpod> -n \u003Cns>` over time (or Prometheus `container_memory_rss`) | Fix the leak; meanwhile add a periodic restart via Deployment rollout |\n| Java JVM heap > limit | `-Xmx` set higher than `limits.memory`, OR unset and JVM uses host hints | Set `-XX:MaxRAMPercentage=75` so heap tracks the cgroup limit |\n| Off-heap \u002F native memory growth | `kubectl top pod` shows RSS growth beyond JVM heap | Profile native allocations; raise `limits.memory` |\n| sidecar (e.g. Envoy, Istio) consuming memory | Two containers in one pod, summed over pod limit | Tune sidecar; raise pod limit; split sidecar to own pod |\n\n### 5b. Node OOM (kernel OOM-killer picks a victim)\n\nThe container did NOT exceed its own limit, but the node ran out of\nmemory. The kernel OOM-killer picks any victim (often the highest-RSS\ncontainer, sometimes the kubelet itself).\n\n| Cause | Probe | Fix |\n|---|---|---|\n| No `limits.memory` on co-located pods | `kubectl describe pod` for each pod on the node | Set `limits.memory` on every pod (and `requests.memory` so the scheduler accounts for it) |\n| Node oversubscribed (lots of `BestEffort` or `Burstable` pods) | `kubectl describe node \u003Cnode>` Allocated resources > 90% | Add `requests.memory` everywhere; lower pod density via `topologySpreadConstraints` or node taints |\n| DaemonSet (logging agent, CNI) consuming memory | `kubectl top pods -A --sort-by=memory` for DaemonSet pods | Resize the DaemonSet, or upgrade the node instance type |\n\nMoved verbatim to `references\u002Fdiagnostic-commands.md` — see \"**Step 5 — OOM commands**\" (load on demand).\nApplies when: diagnosing OOMKilled \u002F node memory pressure on a live cluster.\n\nMoved verbatim to `references\u002Fadvanced-patterns.md` — see \"**Step 5 — EKS-specific OOM notes (cgroup v2, Java)**\" (load on demand).\nApplies when: an OOM involves AL2023 cgroup v2 pod-cgroup enforcement or JVM heap sizing.\n\n### Step 6: PROBE_FAILURE diagnostic (Liveness \u002F Readiness)\n\nA pod is `Running` but the kubelet reports failing probes. Liveness\nfailures cause restarts; readiness failures cause the pod to be removed\nfrom Service endpoints (no traffic).\n\n| Sub-symptom | Root cause | Probe |\n|---|---|---|\n| `Liveness probe failed: HTTP probe failed ... 500 \u002F 404` | Probe path returns non-200 | `kubectl exec` into the pod and `curl localhost:\u003Cport>\u003Cpath>`; verify the app actually serves it |\n| `Liveness probe failed: connect: connection refused` | Probe port wrong, OR application hasn't bound yet (initialDelaySeconds too short) | Check `livenessProbe.httpGet.port` vs `containerPort`; raise `initialDelaySeconds` |\n| `Readiness probe failed` and Service endpoints exclude this pod | App not ready; readiness probe path requires a dependency | `kubectl describe endpoints \u003Csvc> -n \u003Cns>`; check if the app has a \u002Fready that depends on DB |\n| Intermittent failures | Probe timeout too short; app slow on cold cache | Raise `timeoutSeconds` \u002F `periodSeconds`; lower `successThreshold` |\n| Liveness killing the container repeatedly | Misconfigured liveness path being confused for a real crash (loop into Step 4) | Look at Events ordering — Liveness failures precede the kill |\n| gRPC probe failing | `grpcProbe` configured but app doesn't enable gRPC health checks | Use `grpc.health.v1.Health\u002FCheck` per the gRPC health protocol |\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 6 — PROBE_FAILURE diagnostic walk**\" (load on demand).\nApplies when: walking a liveness\u002Freadiness failure to its cause.\n\nMoved verbatim to `references\u002Fdiagnostic-commands.md` — see \"**Step 6 — PROBE_FAILURE commands**\" (load on demand).\nApplies when: diagnosing probe failures on a live cluster.\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 6 — PROBE_FAILURE common fix patterns**\" (load on demand).\nApplies when: remediating a probe-failure finding.\n\n### Step 7: INIT_FAILURE diagnostic (Init:CrashLoopBackOff \u002F Init:Error)\n\nInit containers run sequentially before the main containers start. A\nfailing init container blocks the main container from starting — the pod\nstays in `Init:CrashLoopBackOff` (or `Init:Error`, `Init:ImagePullBackOff`)\nindefinitely.\n\n| Sub-symptom | Root cause | Probe |\n|---|---|---|\n| `Init:CrashLoopBackOff`, init container exit 1 | Application error in init container | `kubectl logs \u003Cpod> -c \u003Cinit-container-name> --previous` |\n| `Init:ImagePullBackOff` | Init container image fails to pull (same causes as Step 3) | `kubectl describe pod` Events; verify the init container image separately from main |\n| Init container takes too long, pod stuck in `Init:i\u002FN` | Init container doing too much (loading huge dataset) — not a failure but appears stuck | `kubectl logs \u003Cpod> -c \u003Cinit-container-name> -f` to watch progress |\n| Init container runs once but pod recreated by controller | Init containers re-run on every pod start — if it expects state from previous run, it fails | Make init containers idempotent; never depend on prior-run state |\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 7 — INIT_FAILURE diagnostic walk**\" (load on demand).\nApplies when: walking an Init:CrashLoopBackOff to its cause.\n\nMoved verbatim to `references\u002Fdiagnostic-commands.md` — see \"**Step 7 — INIT_FAILURE commands**\" (load on demand).\nApplies when: diagnosing init-container failures on a live cluster.\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 7 — INIT_FAILURE common root causes**\" (load on demand).\nApplies when: the init container fails without an obvious error.\n\n### Step 8: Map to root-cause catalog\n\nAfter the walk identifies the category, cross-reference with this\ncatalog. The catalog names the top EKS pod-failure patterns and their\ncanonical fixes.\n\n| # | Root cause | Category | Fix pattern |\n|---|---|---|---|\n| 1 | Node insufficient CPU\u002Fmemory for pod `requests` | PENDING | Scale the node group (or Karpenter NodePool); right-size pod `resources.requests` |\n| 2 | Taint without toleration; all nodes tainted | PENDING | Add `tolerations` matching the taint, OR remove the taint, OR add untainted nodes |\n| 3 | PVC Pending (StorageClass AZ mismatch, gp3) | PENDING | Use `volumeBindingMode: WaitForFirstConsumer`; verify AZ has capacity |\n| 4 | Image tag typo \u002F image purged by ECR lifecycle policy | IMAGE_PULL | Pin to existing tag (or digest); relax lifecycle policy |\n| 5 | Node instance role lacks ECR read perms | IMAGE_PULL | Attach `AmazonEC2ContainerRegistryReadOnly` to the node instance role |\n| 6 | Cross-account ECR repo policy missing caller | IMAGE_PULL | Update ECR repo policy in image-owning account to allow caller's account |\n| 7 | Application crash on missing env var \u002F missing ConfigMap | CRASH_LOOP | Add the env var \u002F `envFrom` reference; verify Secret\u002FConfigMap exists in namespace |\n| 8 | Image architecture mismatch (AMD64 vs ARM64) | CRASH_LOOP | Build multi-arch image with `docker buildx --platform linux\u002Famd64,linux\u002Farm64` |\n| 9 | Container `limits.memory` too low (OOM 137) | OOM | Raise `resources.limits.memory`; for Java use `-XX:MaxRAMPercentage=75` |\n| 10 | Liveness probe path\u002Fport\u002FinitialDelaySeconds wrong | PROBE_FAILURE | Align probe with application endpoint; add `startupProbe`; raise `initialDelaySeconds` |\n| 11 | Readiness probe missing — pod never enters Service endpoints | PROBE_FAILURE | Add a `readinessProbe` that tests a real dependency |\n| 12 | Init container failing (DB migration, missing dep) | INIT_FAILURE | Fix init container script; make idempotent; verify init container image and perms |\n\n### Step 9: Verify the fix\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 9 — verify the fix**\" (load on demand).\nApplies when: validating a proposed fix before rolling it out.\n\n### Step 10: Decide — ROOT_CAUSE_FOUND vs NEED_MORE_INFO vs ESCALATE\n\n- **ROOT_CAUSE_FOUND.** The walk identified a specific failure category\n  and a specific configuration element (pod spec field, node IAM policy,\n  PVC StorageClass, probe config). Output REMEDIATION with the exact\n  change.\n- **NEED_MORE_INFO.** The walk reached a step where the operator cannot\n  supply evidence (e.g., `kubectl logs --previous` requires access to a\n  namespace the operator cannot read). Output the list of missing\n  inputs.\n- **ESCALATE.** The walk identifies a cause outside the operator's\n  scope: node IAM role owned by the platform team, ECR repo in another\n  account, Karpenter NodePool owned by another team, NetworkPolicy\n  enforced cluster-wide. Output the escalation target and the specific\n  request to make.\n\n## Output format\n\n```text\nINCIDENT: \u003Cnamespace>\u002F\u003Cpod> on node \u003Cnode> — \u003Csymptom>\nVERDICT: ROOT_CAUSE_FOUND | NEED_MORE_INFO | ESCALATE\nROOT_CAUSE: \u003Ccategory name> — \u003Cspecific failing config element>\nEVIDENCE:\n  - kubectl describe pod: \u003Cevent line>\n  - kubectl logs --previous: \u003Ckey log line>\n  - kubectl get pod (jsonpath): \u003ClastState \u002F probe \u002F resource field>\n  - aws iam\u002Fecr\u002Fec2: \u003CAWS-side signal>\nROOT_CAUSE_CATALOG: #\u003CN>\nREMEDIATION:\n  1. \u003Cspecific config change with field name>\n  2. \u003Cverification command>\n  3. \u003Cpost-apply monitoring>\n```\n\n### Worked example — CrashLoopBackOff from missing env var\n\n```text\nINCIDENT: production\u002Fapi-server-5d4b6c7d8-x9fk2 on node ip-10-0-3-42 —\nCrashLoopBackOff (restart count 14, climbing)\nVERDICT: ROOT_CAUSE_FOUND\nROOT_CAUSE: CRASH_LOOP — application reads `DB_HOST` env var but pod\nspec does not set it; container exits with code 1 in \u003C1s\nEVIDENCE:\n  - kubectl describe pod: Last State: Terminated, Reason: Error, Exit\n    Code: 1, Started\u002FFinished ~0.8s apart\n  - kubectl logs api-server-5d4b6c7d8-x9fk2 -c api --previous:\n    \"Traceback ... KeyError: 'DB_HOST'\"\n  - kubectl get pod -o jsonpath\n    '{.spec.containers[0].env}': returns the env list — DB_HOST is\n    missing\n  - kubectl get events -n production --field-selector\n    involvedObject.name=api-server-5d4b6c7d8-x9fk2: \"Back-off\n    restarting failed container\" × N\nROOT_CAUSE_CATALOG: #7 (missing env var)\nREMEDIATION:\n  1. Patch the Deployment to add the env var (recommended: sourced from\n     a ConfigMap so it is centralised):\n     kubectl set env deployment\u002Fapi-server -n production \\\n       DB_HOST=postgres.production.svc.cluster.local\n     (or kubectl edit deployment api-server -n production and add to\n     spec.template.spec.containers[0].env)\n  2. Watch the rollout:\n     kubectl rollout status deployment\u002Fapi-server -n production\n  3. Verify the new pod reaches Running with restart count 0:\n     kubectl get pods -n production -l app=api-server -w\n  4. If DB connectivity is also wrong (logs then show\n     \"Connection refused\"), check the Service name and NetworkPolicy.\n```\n\n## Expert heuristic — \"The pod troubleshooting trinity\"\n\nMoved verbatim to `references\u002Fadvanced-patterns.md` — see \"**Expert heuristic — the pod troubleshooting trinity**\" (load on demand).\nApplies when: you want the 90%-coverage three-command diagnostic order.\n\n## Anti-Patterns — NEVER\n\n- **NEVER** declare the root cause from `kubectl get pods` alone. The\n  STATUS column is a symptom (`CrashLoopBackOff`), not a cause. Always\n  run `kubectl describe pod` + `kubectl logs --previous` before\n  declaring ROOT_CAUSE_FOUND.\n\n- **NEVER** read `kubectl logs` without `--previous` for a\n  CrashLoopBackOff pod. The current container just started; its logs\n  are empty or partial. The crash cause is in the *previous* container\n  instance's logs.\n\n- **NEVER** confuse the node instance role with the pod's IRSA \u002F Pod\n  Identity role. ECR pulls use the *node* instance role unless IRSA or\n  EKS Pod Identity is configured AND the pod has\n  `serviceAccountName` set with the right annotation. Misidentifying\n  which role pulls the image leads to fixing the wrong IAM policy.\n\n- **NEVER** raise `resources.limits.memory` without checking whether\n  the OOM is container-level (`OOMKilled` in lastState) or node-level\n  (kernel OOM-killer, no per-container reason). The fixes are\n  different: container OOM raises the limit; node OOM requires\n  `requests.memory` everywhere and possibly larger nodes.\n\n- **NEVER** recommend `privileged: true` as a fix for any Kubernetes\n  failure. Privileged grants full host access; the real fix is almost\n  always a missing capability, a mount permission, or a network policy.\n  Privileged is a security anti-pattern and forbidden by Pod Security\n  Standards `restricted`.\n\n- **NEVER** declare ROOT_CAUSE_FOUND without reading the application\n  logs. A crash-looping pod has its cause in `kubectl logs --previous`,\n  not in any kubectl field. If you cannot access logs (RBAC), emit\n  NEED_MORE_INFO.\n\n- **NEVER** confuse a `livenessProbe` with a `readinessProbe`. Liveness\n  failures cause restarts; readiness failures cause the pod to be\n  removed from Service endpoints (no traffic). Mixing them up leads to\n  \"the pod is healthy but the Service is broken\" confusion.\n\n- **NEVER** lower `livenessProbe` thresholds (`failureThreshold`,\n  `periodSeconds`) as a fix. If the probe is failing, the application\n  is not healthy or the probe is wrong. Making the probe more forgiving\n  hides the real problem.\n\n- **NEVER** recommend `imagePullPolicy: Always` as a fix for\n  ImagePullBackOff. The policy affects whether the kubelet checks the\n  registry for a new digest; it does not fix auth or manifest issues.\n\n- **NEVER** assume `kubectl describe pod` Events are exhaustive. Events\n  age out after ~1 hour (default `--event-ttl`). For incidents older\n  than an hour, the Events may be gone — fall back to Control Plane\n  Logs (`kube-apiserver` \u002F `kube-scheduler`) if enabled on the EKS\n  cluster.\n\n- **NEVER** assume a Pending pod is unschedulable because of resources.\n  Taints, affinity, PVC binding, and `PriorityClass` preemption can all\n  cause Pending. Always read the Events `FailedScheduling` message\n  verbatim.\n\n- **NEVER** \"fix\" an Init:CrashLoopBackOff by deleting the init\n  container. The init container is doing a job the main container\n  depends on. Fix the init container's actual failure.\n\n- **NEVER** trust `kubectl top pod` for OOM diagnosis in isolation. Top\n  is a real-time snapshot; if the pod already crashed, top shows\n  nothing. Use it while the pod is alive, or use Prometheus\n  `container_memory_rss` historical data.\n\n- **NEVER** conclude \"the application has a memory leak\" without\n  evidence of monotonic RSS growth. A single OOMKilled is more likely\n  a too-low limit. A leak shows a sawtooth or rising trend over hours\n  or days.\n\n- **NEVER** delete a PVC to \"fix\" a Pending pod. Deleting the PVC can\n  orphan or lose data. Read `kubectl describe pvc` Events first; the\n  StorageClass, AZ, and `WaitForFirstConsumer` binding mode are the\n  usual culprits.\n\n- **NEVER** declare IMAGE_PULL without verifying the image exists in\n  ECR. The error string \"manifest unknown\" is also returned for\n  transient registry issues. Run\n  `aws ecr describe-images --repository-name \u003Crepo> --image-ids imageTag=\u003Ctag>`\n  to confirm.\n\n- **NEVER** assume the EKS API server is reachable from your pod. If\n  `kubectl exec` fails with `error: unable to upgrade connection`,\n  check the API server endpoint access (public vs private), the\n  pod's subnet route, and security group on the control plane.\n\n- **NEVER** recommend raising `livenessProbe.timeoutSeconds` past a\n  few seconds without investigation. A probe that takes 10+ seconds\n  to respond means the application is pathologically slow; raise\n  `timeoutSeconds` only after ruling out real latency.\n\n- **NEVER** \"fix\" CrashLoopBackOff by setting `restartPolicy: Never`.\n  This hides the loop by not restarting — the pod still fails, just\n  without the visible BackOff. Use `Never` only for true Jobs.\n\n## Remediation guidance\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Remediation guidance by category**\" (load on demand).\nApplies when: emitting the REMEDIATION block for a confirmed root cause.\n\n## Recent AWS features (2024-2026)\n\nMoved verbatim to `references\u002Fadvanced-patterns.md` — see \"**Recent AWS features (2024-2026)**\" (load on demand).\nApplies when: the cluster uses Pod Identity, Karpenter v1, AL2023, gp3, Auto Mode, or VPC CNI network policies.\n\n## References\n\nSee `references\u002Ffailure-decision-tree.md` for the full symptom-to-cause\nwalk with worked examples per category, and `references\u002Fdiagnostic-\ncommands.md` for the canonical command script for each failure category.\n\n## References (load on demand)\n\n- [references\u002Ffailure-decision-tree.md](references\u002Ffailure-decision-tree.md) — full symptom-to-cause walk with worked examples per category, plus the per-step diagnostic walks, fix-pattern catalogs, verify-the-fix guidance, and remediation guidance moved verbatim from this file.\n- [references\u002Fdiagnostic-commands.md](references\u002Fdiagnostic-commands.md) — universal first commands plus the per-step\u002Fper-category command scripts moved verbatim from this file.\n- [references\u002Fadvanced-patterns.md](references\u002Fadvanced-patterns.md) — mindset deep dive, EKS-specific OOM notes (cgroup v2, Java), the troubleshooting trinity, and recent AWS features (moved verbatim from this file).\n\n## Domain\n\nAWS CloudOps \u002F EKS Compute Reliability & Kubernetes Pod Operations.\n\n## AWS documentation\n\n- **Amazon EKS User Guide** — https:\u002F\u002Fdocs.aws.amazon.com\u002Feks\u002Flatest\u002Fuserguide\u002Fwhat-is-eks.html\n- **EKS troubleshooting** — https:\u002F\u002Fdocs.aws.amazon.com\u002Feks\u002Flatest\u002Fuserguide\u002Ftroubleshooting.html\n- **Kubernetes pod lifecycle** — https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Fpod-lifecycle\u002F\n- **Kubernetes probes** — https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Fpod-lifecycle\u002F#container-probes\n- **Kubernetes init containers** — https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Finit-containers\u002F\n- **Kubernetes debug a pod** — https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Fdebug\u002Fdebug-application\u002Fdebug-pods\u002F\n- **EKS Pod Identity** — https:\u002F\u002Fdocs.aws.amazon.com\u002Feks\u002Flatest\u002Fuserguide\u002Fpod-identities.html\n- **Karpenter** — https:\u002F\u002Fkarpenter.sh\u002F\n- **ECR cross-account pull for EKS** — https:\u002F\u002Frepost.aws\u002Fknowledge-center\u002Feks-ecs-cross-account-container-pull\n- **kubectl reference** — https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Freference\u002Fgenerated\u002Fkubectl\u002Fkubectl-commands\n","# EKS Pod Troubleshooter\n\n## Activation\n\nActivate this skill when the user reports a Kubernetes pod failure on an\nEKS cluster. Trigger phrases: \"EKS pod CrashLoopBackOff\", \"EKS pod\nImagePullBackOff\", \"EKS pod Pending\", \"EKS pod FailedScheduling\", \"EKS\npod OOMKilled\", \"EKS pod liveness probe failing\", \"EKS pod readiness\nprobe failing\", \"EKS init container failing\", \"EKS pod keeps restarting\",\n\"kubectl pod status\".\n\n## Mindset\n\n**One-line takeaway:** every Kubernetes pod failure surfaces in three\nplaces — `kubectl describe pod` (Events + container lastState), `kubectl\nlogs --previous` (application error), and `kubectl get events` (timeline\nof what the kubelet and scheduler did). Read all three before declaring\na root cause.\n\nMoved verbatim to `references\u002Fadvanced-patterns.md` — see \"**Mindset — three EKS-specific facts**\" (load on demand).\nApplies when: you need the deeper reasoning behind the three-evidence-sources rule.\n\n## Quick reference — symptom to failure category\n\n| Observed state | Failure category | First probe |\n|---|---|---|\n| `STATUS=Pending`, `REASON=FailedScheduling` in events | PENDING | `kubectl describe pod` → Events: `FailedScheduling` message; check `top nodes` capacity |\n| `STATUS=ImagePullBackOff` \u002F `ErrImagePull` | IMAGE_PULL | `kubectl describe pod` Events: `Failed to pull image` message; verify ECR auth |\n| `STATUS=CrashLoopBackOff`, restart count climbing | CRASH_LOOP | `kubectl logs \u003Cpod> --previous` for the application error |\n| Container `Last State: Terminated`, `Reason: OOMKilled`, exit 137 | OOM | `kubectl describe pod` containers[].lastState; check `resources.limits.memory` |\n| `STATUS=Running` but liveness\u002Freadiness probes failing; restart count climbing | PROBE_FAILURE | `kubectl describe pod` Events: `Liveness probe failed` \u002F `Readiness probe failed`; verify probe path\u002Fport |\n| `STATUS=Init:CrashLoopBackOff` \u002F `Init:Error` | INIT_FAILURE | `kubectl logs \u003Cpod> -c \u003Cinit-container> --previous` |\n\nSee the ordered steps below for the full diagnostic walk.\n\n## Quick navigation\n\n- **Step 0** — Capture the failure signal (pod name, namespace, phase).\n- **Step 1** — Map the symptom to a category letter (A-F).\n- **Step 2** — PENDING diagnostic (FailedScheduling).\n- **Step 3** — IMAGE_PULL diagnostic (ImagePullBackOff).\n- **Step 4** — CRASH_LOOP diagnostic (CrashLoopBackOff, exit code).\n- **Step 5** — OOM diagnostic (OOMKilled).\n- **Step 6** — PROBE_FAILURE diagnostic (Liveness\u002FReadiness).\n- **Step 7** — INIT_FAILURE diagnostic (Init:CrashLoopBackOff).\n- **Step 8** — Root-cause catalog (top patterns + canonical fixes).\n- **Step 9** — Verify the fix.\n- **Step 10** — Decide VERDICT (ROOT_CAUSE_FOUND \u002F NEED_MORE_INFO \u002F ESCALATE).\n\n## Process — Diagnostic decision tree (apply in order)\n\n### Step 0: Capture the failure signal\n\nGather these three pieces. Each step below branches on which is present.\n\n| Signal | Source | Why required |\n|---|---|---|\n| **Pod name + namespace** | User-provided or `kubectl get pods -A --field-selector status.phase!=Running` | All describe\u002Flogs calls need this |\n| **Phase + container statuses + restart count** | `kubectl get pod \u003Cpod> -n \u003Cns> -o wide` | Drives the symptom category |\n| **Events + lastState** | `kubectl describe pod \u003Cpod> -n \u003Cns>` | Narrows from symptom to cause |\n\nIf the user has not provided the pod name or namespace, output:\n\n```text\nINCIDENT: \u003Cnamespace>\u002F\u003Cpod> — \u003Csymptom>\nVERDICT: NEED_MORE_INFO\nREASON: Cannot diagnose without the pod name and namespace. Identify\nthe failing pod with: kubectl get pods -A --field-selector\nstatus.phase!=Running  (or: kubectl get pods -n \u003Cns>\n--field-selector status.containerStatuses[*].restartCount>0)\nMISSING:\n  - Pod name and namespace\n  - Last observed status (Pending \u002F ContainerCreating \u002F\n    Running \u002F CrashLoopBackOff \u002F ImagePullBackOff)\n```\n\nIf the user reports \"pods keep failing\" but does not know which pod,\nask for the namespace and deployment\u002Fpod label. Then run:\n\nMoved verbatim to `references\u002Fdiagnostic-commands.md` — see \"**Step 0 — locate the failing pod(s) when name\u002Fnamespace is unknown**\" (load on demand).\nApplies when: the user cannot name the failing pod.\n\n### Step 1: Identify the symptom category\n\nMap the observed state to one of six categories. Each category has a\ndifferent diagnostic walk in Steps 2-7.\n\n| Category | Signature | Diagnostic step |\n|---|---|---|\n| **A. PENDING** | `STATUS=Pending`; Events contain `FailedScheduling` | Step 2 |\n| **B. IMAGE_PULL** | `STATUS=ImagePullBackOff` or `ErrImagePull`; Events contain `Failed to pull image` \u002F `rpc error: code = Unknown desc = Error response from daemon` | Step 3 |\n| **C. CRASH_LOOP** | `STATUS=CrashLoopBackOff`; `status.containerStatuses[].restartCount` climbing; `lastState` has exit code 1\u002F255\u002Fetc | Step 4 |\n| **D. OOM** | `status.containerStatuses[].lastState.reason: OOMKilled`; exit code 137 | Step 5 |\n| **E. PROBE_FAILURE** | `STATUS=Running` but Events show `Liveness probe failed` \u002F `Readiness probe failed`; restart count climbing or pod stuck not-ready | Step 6 |\n| **F. INIT_FAILURE** | `STATUS=Init:CrashLoopBackOff` \u002F `Init:Error` \u002F `Init:ImagePullBackOff`; `status.initContainerStatuses` populated | Step 7 |\n\n**Earliest-transition rule.** If the symptom matches more than one\ncategory, pick the EARLIEST transition failure in the pod lifecycle.\nPENDING precedes IMAGE_PULL precedes INIT_FAILURE precedes CRASH_LOOP \u002F\nOOM \u002F PROBE_FAILURE. The pod cannot be crash-looping if it never\nscheduled.\n\n### Step 2: PENDING diagnostic (FailedScheduling)\n\nA pod in `Pending` has not been bound to a node. The failure is in the\nscheduler: insufficient resources, taints without tolerations, node\naffinity\u002Fselector with no match, or a pending PersistentVolumeClaim.\n\n| Events message sub-string | Root cause | Probe |\n|---|---|---|\n| `Insufficient cpu` \u002F `Insufficient memory` | Node free resources \u003C pod `resources.requests` | `kubectl top nodes`; `kubectl describe nodes` Allocatable vs Requests |\n| `node(s) had taints that the pod didn't tolerate` | All nodes tainted; pod has no matching toleration | `kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints` |\n| `node(s) didn't match Pod's node affinity\u002Fselector` | `nodeSelector` or `nodeAffinity` excludes all nodes | `kubectl get nodes --show-labels`; compare to pod `nodeSelector` \u002F `requiredDuringScheduling` |\n| `node(s) were unschedulable` | All matching nodes are `cordoned` (Unschedulable) | `kubectl get nodes -o wide`; look for `SchedulingDisabled` \u002F `NotReady` |\n| `pod has unbound immediate PersistentVolumeClaims` | PVC is Pending — no StorageClass match, exhausted volume IDs, or EBS AZ mismatch | `kubectl get pvc -n \u003Cns>`; `kubectl get pv`; `kubectl describe pvc \u003Cpvc> -n \u003Cns>` |\n| `Insufficient \u003Cdevice>` (e.g. gpu) | Limited resource (NVIDIA GPU) — pod requests a resource no node advertutes via device plugin | `kubectl describe nodes` Capacity; verify device-plugin DaemonSet |\n| `maximum graceful delete duration` etc. (rare) | Pod stuck in termination — different walk; not Pending | Check `deletionTimestamp` |\n\nMoved verbatim to `references\u002Fdiagnostic-commands.md` — see \"**Step 2 — PENDING (FailedScheduling) commands**\" (load on demand).\nApplies when: diagnosing a Pending pod on a live cluster.\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 2 — PENDING common fix patterns**\" (load on demand).\nApplies when: remediating a FailedScheduling finding.\n\n### Step 3: IMAGE_PULL diagnostic (ImagePullBackOff \u002F ErrImagePull)\n\nThe kubelet cannot pull the container image. On EKS the cause is almost\nalways one of: image tag typo, ECR auth from the node role, missing VPC\nendpoint for ECR in a private subnet, or a cross-account ECR repo\nwithout a `Private` namespace identity policy.\n\n| Events message sub-string | Root cause | Probe |\n|---|---|---|\n| `manifest unknown` \u002F `Requested image not found` | Tag does not exist (typo, never pushed, or ECR lifecycle policy purged it) | `aws ecr describe-images --repository-name \u003Crepo> --image-ids imageTag=\u003Ctag>` |\n| `not authorized` \u002F `403 Forbidden` | ECR auth failing — node IAM role lacks `ecr:GetDownloadUrlForLayer` \u002F `ecr:BatchGetImage`, OR cross-account repo policy missing caller | `aws iam simulate-principal-policy` against the *node* instance role ARN |\n| `RequestError: send request failed` \u002F `i\u002Fo timeout` | Network: no route to ECR (private subnet, missing VPC endpoint, broken NAT) | `aws ec2 describe-vpc-endpoints`; verify route table 0.0.0.0\u002F0 via NAT or `com.amazonaws.\u003Cregion>.ecr.dkr` endpoint |\n| `no space left on device` | Node disk full (image cache) | `kubectl debug node\u002F\u003Cnode> -it --image=busybox -- df -h \u002Fvar\u002Flib\u002Fcontainerd`; drain and roll the node |\n| Pull works for other pods on same node but not this pod | Cross-account ECR — image ARN includes a different account ID | Compare `image: \u003Cacct>.dkr.ecr.\u003Cregion>.amazonaws.com\u002F...`; verify node role trust + repo policy BOTH allow caller |\n| Pull works on newer nodes but fails on older | ECR token rotation + stale `aws-ecr-credential-provider` config on old AMI | Check node AMI version; upgrade the node group |\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 3 — IMAGE_PULL diagnostic walk**\" (load on demand).\nApplies when: walking an ImagePullBackOff \u002F ErrImagePull to its cause.\n\nMoved verbatim to `references\u002Fdiagnostic-commands.md` — see \"**Step 3 — IMAGE_PULL commands**\" (load on demand).\nApplies when: diagnosing image-pull failures on a live cluster.\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 3 — IMAGE_PULL common fix patterns**\" (load on demand).\nApplies when: remediating an image-pull finding.\n\n### Step 4: CRASH_LOOP diagnostic (CrashLoopBackOff)\n\nA CrashLoopBackOff means the kubelet restarted the container repeatedly\nwith exponential backoff. The exit code and application logs narrow the\ncause; `kubectl describe pod` Events alone are too generic.\n\n| Exit code | Meaning | First probe |\n|---|---|---|\n| 0 | Clean exit — application thinks it is done | `kubectl logs --previous` — is the workload a batch? Should `restartPolicy: Never` or `OnFailure` be set instead of `Always`? |\n| 1 | General application error | `kubectl logs \u003Cpod> -c \u003Ccontainer> --previous` — exception traceback |\n| 2 \u002F 126 \u002F 127 | `command not found` \u002F `exec format error` \u002F wrong entrypoint | Check `command` \u002F `args` in pod spec; verify image architecture (ARM64 vs AMD64) |\n| 137 | SIGKILL — usually OOM killer | Step 5 (OOM) |\n| 139 | SIGSEGV — native crash, JNI \u002F cgo bug | `kubectl logs --previous` for native stack; image arch mismatch |\n| 255 | Application-defined error | `kubectl logs --previous` |\n| Empty + Events mention `CreateContainerConfigError` | kubelet could not construct the container spec — missing ConfigMap\u002FSecret referenced in `envFrom` or `env.valueFrom` | `kubectl describe pod` Events; verify referenced Secret\u002FConfigMap exists in the namespace |\n\nMoved verbatim to `references\u002Fdiagnostic-commands.md` — see \"**Step 4 — CRASH_LOOP commands**\" (load on demand).\nApplies when: diagnosing a CrashLoopBackOff on a live cluster.\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 4 — CRASH_LOOP common root causes**\" (load on demand).\nApplies when: the exit code or log pattern does not match an obvious cause.\n\n### Step 5: OOM diagnostic (OOMKilled)\n\nOut-of-memory has two flavors:\n\n### 5a. Container OOM (`OOMKilled`, exit 137)\n\nThe container exceeded its `resources.limits.memory` cgroup. The kernel\nOOM-killer (cgroup v2) or Docker OOM-killer (cgroup v1) SIGKILLs the\nprocess.\n\n| Cause | Probe | Fix |\n|---|---|---|\n| `limits.memory` too low for the workload | `kubectl describe pod` lastState; compare to application RSS | Raise `resources.limits.memory` |\n| Memory leak in application | `kubectl top pod \u003Cpod> -n \u003Cns>` over time (or Prometheus `container_memory_rss`) | Fix the leak; meanwhile add a periodic restart via Deployment rollout |\n| Java JVM heap > limit | `-Xmx` set higher than `limits.memory`, OR unset and JVM uses host hints | Set `-XX:MaxRAMPercentage=75` so heap tracks the cgroup limit |\n| Off-heap \u002F native memory growth | `kubectl top pod` shows RSS growth beyond JVM heap | Profile native allocations; raise `limits.memory` |\n| sidecar (e.g. Envoy, Istio) consuming memory | Two containers in one pod, summed over pod limit | Tune sidecar; raise pod limit; split sidecar to own pod |\n\n### 5b. Node OOM (kernel OOM-killer picks a victim)\n\nThe container did NOT exceed its own limit, but the node ran out of\nmemory. The kernel OOM-killer picks any victim (often the highest-RSS\ncontainer, sometimes the kubelet itself).\n\n| Cause | Probe | Fix |\n|---|---|---|\n| No `limits.memory` on co-located pods | `kubectl describe pod` for each pod on the node | Set `limits.memory` on every pod (and `requests.memory` so the scheduler accounts for it) |\n| Node oversubscribed (lots of `BestEffort` or `Burstable` pods) | `kubectl describe node \u003Cnode>` Allocated resources > 90% | Add `requests.memory` everywhere; lower pod density via `topologySpreadConstraints` or node taints |\n| DaemonSet (logging agent, CNI) consuming memory | `kubectl top pods -A --sort-by=memory` for DaemonSet pods | Resize the DaemonSet, or upgrade the node instance type |\n\nMoved verbatim to `references\u002Fdiagnostic-commands.md` — see \"**Step 5 — OOM commands**\" (load on demand).\nApplies when: diagnosing OOMKilled \u002F node memory pressure on a live cluster.\n\nMoved verbatim to `references\u002Fadvanced-patterns.md` — see \"**Step 5 — EKS-specific OOM notes (cgroup v2, Java)**\" (load on demand).\nApplies when: an OOM involves AL2023 cgroup v2 pod-cgroup enforcement or JVM heap sizing.\n\n### Step 6: PROBE_FAILURE diagnostic (Liveness \u002F Readiness)\n\nA pod is `Running` but the kubelet reports failing probes. Liveness\nfailures cause restarts; readiness failures cause the pod to be removed\nfrom Service endpoints (no traffic).\n\n| Sub-symptom | Root cause | Probe |\n|---|---|---|\n| `Liveness probe failed: HTTP probe failed ... 500 \u002F 404` | Probe path returns non-200 | `kubectl exec` into the pod and `curl localhost:\u003Cport>\u003Cpath>`; verify the app actually serves it |\n| `Liveness probe failed: connect: connection refused` | Probe port wrong, OR application hasn't bound yet (initialDelaySeconds too short) | Check `livenessProbe.httpGet.port` vs `containerPort`; raise `initialDelaySeconds` |\n| `Readiness probe failed` and Service endpoints exclude this pod | App not ready; readiness probe path requires a dependency | `kubectl describe endpoints \u003Csvc> -n \u003Cns>`; check if the app has a \u002Fready that depends on DB |\n| Intermittent failures | Probe timeout too short; app slow on cold cache | Raise `timeoutSeconds` \u002F `periodSeconds`; lower `successThreshold` |\n| Liveness killing the container repeatedly | Misconfigured liveness path being confused for a real crash (loop into Step 4) | Look at Events ordering — Liveness failures precede the kill |\n| gRPC probe failing | `grpcProbe` configured but app doesn't enable gRPC health checks | Use `grpc.health.v1.Health\u002FCheck` per the gRPC health protocol |\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 6 — PROBE_FAILURE diagnostic walk**\" (load on demand).\nApplies when: walking a liveness\u002Freadiness failure to its cause.\n\nMoved verbatim to `references\u002Fdiagnostic-commands.md` — see \"**Step 6 — PROBE_FAILURE commands**\" (load on demand).\nApplies when: diagnosing probe failures on a live cluster.\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 6 — PROBE_FAILURE common fix patterns**\" (load on demand).\nApplies when: remediating a probe-failure finding.\n\n### Step 7: INIT_FAILURE diagnostic (Init:CrashLoopBackOff \u002F Init:Error)\n\nInit containers run sequentially before the main containers start. A\nfailing init container blocks the main container from starting — the pod\nstays in `Init:CrashLoopBackOff` (or `Init:Error`, `Init:ImagePullBackOff`)\nindefinitely.\n\n| Sub-symptom | Root cause | Probe |\n|---|---|---|\n| `Init:CrashLoopBackOff`, init container exit 1 | Application error in init container | `kubectl logs \u003Cpod> -c \u003Cinit-container-name> --previous` |\n| `Init:ImagePullBackOff` | Init container image fails to pull (same causes as Step 3) | `kubectl describe pod` Events; verify the init container image separately from main |\n| Init container takes too long, pod stuck in `Init:i\u002FN` | Init container doing too much (loading huge dataset) — not a failure but appears stuck | `kubectl logs \u003Cpod> -c \u003Cinit-container-name> -f` to watch progress |\n| Init container runs once but pod recreated by controller | Init containers re-run on every pod start — if it expects state from previous run, it fails | Make init containers idempotent; never depend on prior-run state |\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 7 — INIT_FAILURE diagnostic walk**\" (load on demand).\nApplies when: walking an Init:CrashLoopBackOff to its cause.\n\nMoved verbatim to `references\u002Fdiagnostic-commands.md` — see \"**Step 7 — INIT_FAILURE commands**\" (load on demand).\nApplies when: diagnosing init-container failures on a live cluster.\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 7 — INIT_FAILURE common root causes**\" (load on demand).\nApplies when: the init container fails without an obvious error.\n\n### Step 8: Map to root-cause catalog\n\nAfter the walk identifies the category, cross-reference with this\ncatalog. The catalog names the top EKS pod-failure patterns and their\ncanonical fixes.\n\n| # | Root cause | Category | Fix pattern |\n|---|---|---|---|\n| 1 | Node insufficient CPU\u002Fmemory for pod `requests` | PENDING | Scale the node group (or Karpenter NodePool); right-size pod `resources.requests` |\n| 2 | Taint without toleration; all nodes tainted | PENDING | Add `tolerations` matching the taint, OR remove the taint, OR add untainted nodes |\n| 3 | PVC Pending (StorageClass AZ mismatch, gp3) | PENDING | Use `volumeBindingMode: WaitForFirstConsumer`; verify AZ has capacity |\n| 4 | Image tag typo \u002F image purged by ECR lifecycle policy | IMAGE_PULL | Pin to existing tag (or digest); relax lifecycle policy |\n| 5 | Node instance role lacks ECR read perms | IMAGE_PULL | Attach `AmazonEC2ContainerRegistryReadOnly` to the node instance role |\n| 6 | Cross-account ECR repo policy missing caller | IMAGE_PULL | Update ECR repo policy in image-owning account to allow caller's account |\n| 7 | Application crash on missing env var \u002F missing ConfigMap | CRASH_LOOP | Add the env var \u002F `envFrom` reference; verify Secret\u002FConfigMap exists in namespace |\n| 8 | Image architecture mismatch (AMD64 vs ARM64) | CRASH_LOOP | Build multi-arch image with `docker buildx --platform linux\u002Famd64,linux\u002Farm64` |\n| 9 | Container `limits.memory` too low (OOM 137) | OOM | Raise `resources.limits.memory`; for Java use `-XX:MaxRAMPercentage=75` |\n| 10 | Liveness probe path\u002Fport\u002FinitialDelaySeconds wrong | PROBE_FAILURE | Align probe with application endpoint; add `startupProbe`; raise `initialDelaySeconds` |\n| 11 | Readiness probe missing — pod never enters Service endpoints | PROBE_FAILURE | Add a `readinessProbe` that tests a real dependency |\n| 12 | Init container failing (DB migration, missing dep) | INIT_FAILURE | Fix init container script; make idempotent; verify init container image and perms |\n\n### Step 9: Verify the fix\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Step 9 — verify the fix**\" (load on demand).\nApplies when: validating a proposed fix before rolling it out.\n\n### Step 10: Decide — ROOT_CAUSE_FOUND vs NEED_MORE_INFO vs ESCALATE\n\n- **ROOT_CAUSE_FOUND.** The walk identified a specific failure category\n  and a specific configuration element (pod spec field, node IAM policy,\n  PVC StorageClass, probe config). Output REMEDIATION with the exact\n  change.\n- **NEED_MORE_INFO.** The walk reached a step where the operator cannot\n  supply evidence (e.g., `kubectl logs --previous` requires access to a\n  namespace the operator cannot read). Output the list of missing\n  inputs.\n- **ESCALATE.** The walk identifies a cause outside the operator's\n  scope: node IAM role owned by the platform team, ECR repo in another\n  account, Karpenter NodePool owned by another team, NetworkPolicy\n  enforced cluster-wide. Output the escalation target and the specific\n  request to make.\n\n## Output format\n\n```text\nINCIDENT: \u003Cnamespace>\u002F\u003Cpod> on node \u003Cnode> — \u003Csymptom>\nVERDICT: ROOT_CAUSE_FOUND | NEED_MORE_INFO | ESCALATE\nROOT_CAUSE: \u003Ccategory name> — \u003Cspecific failing config element>\nEVIDENCE:\n  - kubectl describe pod: \u003Cevent line>\n  - kubectl logs --previous: \u003Ckey log line>\n  - kubectl get pod (jsonpath): \u003ClastState \u002F probe \u002F resource field>\n  - aws iam\u002Fecr\u002Fec2: \u003CAWS-side signal>\nROOT_CAUSE_CATALOG: #\u003CN>\nREMEDIATION:\n  1. \u003Cspecific config change with field name>\n  2. \u003Cverification command>\n  3. \u003Cpost-apply monitoring>\n```\n\n### Worked example — CrashLoopBackOff from missing env var\n\n```text\nINCIDENT: production\u002Fapi-server-5d4b6c7d8-x9fk2 on node ip-10-0-3-42 —\nCrashLoopBackOff (restart count 14, climbing)\nVERDICT: ROOT_CAUSE_FOUND\nROOT_CAUSE: CRASH_LOOP — application reads `DB_HOST` env var but pod\nspec does not set it; container exits with code 1 in \u003C1s\nEVIDENCE:\n  - kubectl describe pod: Last State: Terminated, Reason: Error, Exit\n    Code: 1, Started\u002FFinished ~0.8s apart\n  - kubectl logs api-server-5d4b6c7d8-x9fk2 -c api --previous:\n    \"Traceback ... KeyError: 'DB_HOST'\"\n  - kubectl get pod -o jsonpath\n    '{.spec.containers[0].env}': returns the env list — DB_HOST is\n    missing\n  - kubectl get events -n production --field-selector\n    involvedObject.name=api-server-5d4b6c7d8-x9fk2: \"Back-off\n    restarting failed container\" × N\nROOT_CAUSE_CATALOG: #7 (missing env var)\nREMEDIATION:\n  1. Patch the Deployment to add the env var (recommended: sourced from\n     a ConfigMap so it is centralised):\n     kubectl set env deployment\u002Fapi-server -n production \\\n       DB_HOST=postgres.production.svc.cluster.local\n     (or kubectl edit deployment api-server -n production and add to\n     spec.template.spec.containers[0].env)\n  2. Watch the rollout:\n     kubectl rollout status deployment\u002Fapi-server -n production\n  3. Verify the new pod reaches Running with restart count 0:\n     kubectl get pods -n production -l app=api-server -w\n  4. If DB connectivity is also wrong (logs then show\n     \"Connection refused\"), check the Service name and NetworkPolicy.\n```\n\n## Expert heuristic — \"The pod troubleshooting trinity\"\n\nMoved verbatim to `references\u002Fadvanced-patterns.md` — see \"**Expert heuristic — the pod troubleshooting trinity**\" (load on demand).\nApplies when: you want the 90%-coverage three-command diagnostic order.\n\n## Anti-Patterns — NEVER\n\n- **NEVER** declare the root cause from `kubectl get pods` alone. The\n  STATUS column is a symptom (`CrashLoopBackOff`), not a cause. Always\n  run `kubectl describe pod` + `kubectl logs --previous` before\n  declaring ROOT_CAUSE_FOUND.\n\n- **NEVER** read `kubectl logs` without `--previous` for a\n  CrashLoopBackOff pod. The current container just started; its logs\n  are empty or partial. The crash cause is in the *previous* container\n  instance's logs.\n\n- **NEVER** confuse the node instance role with the pod's IRSA \u002F Pod\n  Identity role. ECR pulls use the *node* instance role unless IRSA or\n  EKS Pod Identity is configured AND the pod has\n  `serviceAccountName` set with the right annotation. Misidentifying\n  which role pulls the image leads to fixing the wrong IAM policy.\n\n- **NEVER** raise `resources.limits.memory` without checking whether\n  the OOM is container-level (`OOMKilled` in lastState) or node-level\n  (kernel OOM-killer, no per-container reason). The fixes are\n  different: container OOM raises the limit; node OOM requires\n  `requests.memory` everywhere and possibly larger nodes.\n\n- **NEVER** recommend `privileged: true` as a fix for any Kubernetes\n  failure. Privileged grants full host access; the real fix is almost\n  always a missing capability, a mount permission, or a network policy.\n  Privileged is a security anti-pattern and forbidden by Pod Security\n  Standards `restricted`.\n\n- **NEVER** declare ROOT_CAUSE_FOUND without reading the application\n  logs. A crash-looping pod has its cause in `kubectl logs --previous`,\n  not in any kubectl field. If you cannot access logs (RBAC), emit\n  NEED_MORE_INFO.\n\n- **NEVER** confuse a `livenessProbe` with a `readinessProbe`. Liveness\n  failures cause restarts; readiness failures cause the pod to be\n  removed from Service endpoints (no traffic). Mixing them up leads to\n  \"the pod is healthy but the Service is broken\" confusion.\n\n- **NEVER** lower `livenessProbe` thresholds (`failureThreshold`,\n  `periodSeconds`) as a fix. If the probe is failing, the application\n  is not healthy or the probe is wrong. Making the probe more forgiving\n  hides the real problem.\n\n- **NEVER** recommend `imagePullPolicy: Always` as a fix for\n  ImagePullBackOff. The policy affects whether the kubelet checks the\n  registry for a new digest; it does not fix auth or manifest issues.\n\n- **NEVER** assume `kubectl describe pod` Events are exhaustive. Events\n  age out after ~1 hour (default `--event-ttl`). For incidents older\n  than an hour, the Events may be gone — fall back to Control Plane\n  Logs (`kube-apiserver` \u002F `kube-scheduler`) if enabled on the EKS\n  cluster.\n\n- **NEVER** assume a Pending pod is unschedulable because of resources.\n  Taints, affinity, PVC binding, and `PriorityClass` preemption can all\n  cause Pending. Always read the Events `FailedScheduling` message\n  verbatim.\n\n- **NEVER** \"fix\" an Init:CrashLoopBackOff by deleting the init\n  container. The init container is doing a job the main container\n  depends on. Fix the init container's actual failure.\n\n- **NEVER** trust `kubectl top pod` for OOM diagnosis in isolation. Top\n  is a real-time snapshot; if the pod already crashed, top shows\n  nothing. Use it while the pod is alive, or use Prometheus\n  `container_memory_rss` historical data.\n\n- **NEVER** conclude \"the application has a memory leak\" without\n  evidence of monotonic RSS growth. A single OOMKilled is more likely\n  a too-low limit. A leak shows a sawtooth or rising trend over hours\n  or days.\n\n- **NEVER** delete a PVC to \"fix\" a Pending pod. Deleting the PVC can\n  orphan or lose data. Read `kubectl describe pvc` Events first; the\n  StorageClass, AZ, and `WaitForFirstConsumer` binding mode are the\n  usual culprits.\n\n- **NEVER** declare IMAGE_PULL without verifying the image exists in\n  ECR. The error string \"manifest unknown\" is also returned for\n  transient registry issues. Run\n  `aws ecr describe-images --repository-name \u003Crepo> --image-ids imageTag=\u003Ctag>`\n  to confirm.\n\n- **NEVER** assume the EKS API server is reachable from your pod. If\n  `kubectl exec` fails with `error: unable to upgrade connection`,\n  check the API server endpoint access (public vs private), the\n  pod's subnet route, and security group on the control plane.\n\n- **NEVER** recommend raising `livenessProbe.timeoutSeconds` past a\n  few seconds without investigation. A probe that takes 10+ seconds\n  to respond means the application is pathologically slow; raise\n  `timeoutSeconds` only after ruling out real latency.\n\n- **NEVER** \"fix\" CrashLoopBackOff by setting `restartPolicy: Never`.\n  This hides the loop by not restarting — the pod still fails, just\n  without the visible BackOff. Use `Never` only for true Jobs.\n\n## Remediation guidance\n\nMoved verbatim to `references\u002Ffailure-decision-tree.md` — see \"**Remediation guidance by category**\" (load on demand).\nApplies when: emitting the REMEDIATION block for a confirmed root cause.\n\n## Recent AWS features (2024-2026)\n\nMoved verbatim to `references\u002Fadvanced-patterns.md` — see \"**Recent AWS features (2024-2026)**\" (load on demand).\nApplies when: the cluster uses Pod Identity, Karpenter v1, AL2023, gp3, Auto Mode, or VPC CNI network policies.\n\n## References\n\nSee `references\u002Ffailure-decision-tree.md` for the full symptom-to-cause\nwalk with worked examples per category, and `references\u002Fdiagnostic-\ncommands.md` for the canonical command script for each failure category.\n\n## References (load on demand)\n\n- [references\u002Ffailure-decision-tree.md](references\u002Ffailure-decision-tree.md) — full symptom-to-cause walk with worked examples per category, plus the per-step diagnostic walks, fix-pattern catalogs, verify-the-fix guidance, and remediation guidance moved verbatim from this file.\n- [references\u002Fdiagnostic-commands.md](references\u002Fdiagnostic-commands.md) — universal first commands plus the per-step\u002Fper-category command scripts moved verbatim from this file.\n- [references\u002Fadvanced-patterns.md](references\u002Fadvanced-patterns.md) — mindset deep dive, EKS-specific OOM notes (cgroup v2, Java), the troubleshooting trinity, and recent AWS features (moved verbatim from this file).\n\n## Domain\n\nAWS CloudOps \u002F EKS Compute Reliability & Kubernetes Pod Operations.\n\n## AWS documentation\n\n- **Amazon EKS User Guide** — https:\u002F\u002Fdocs.aws.amazon.com\u002Feks\u002Flatest\u002Fuserguide\u002Fwhat-is-eks.html\n- **EKS troubleshooting** — https:\u002F\u002Fdocs.aws.amazon.com\u002Feks\u002Flatest\u002Fuserguide\u002Ftroubleshooting.html\n- **Kubernetes pod lifecycle** — https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Fpod-lifecycle\u002F\n- **Kubernetes probes** — https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Fpod-lifecycle\u002F#container-probes\n- **Kubernetes init containers** — https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fpods\u002Finit-containers\u002F\n- **Kubernetes debug a pod** — https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Ftasks\u002Fdebug\u002Fdebug-application\u002Fdebug-pods\u002F\n- **EKS Pod Identity** — https:\u002F\u002Fdocs.aws.amazon.com\u002Feks\u002Flatest\u002Fuserguide\u002Fpod-identities.html\n- **Karpenter** — https:\u002F\u002Fkarpenter.sh\u002F\n- **ECR cross-account pull for EKS** — https:\u002F\u002Frepost.aws\u002Fknowledge-center\u002Feks-ecs-cross-account-container-pull\n- **kubectl reference** — https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Freference\u002Fgenerated\u002Fkubectl\u002Fkubectl-commands\n",{"name":5,"license":15,"metadata":24,"description":7,"compatibility":42},{"tags":25,"phase":26,"author":27,"domain":28,"family":29,"version":30,"keywords":31,"task_type":32,"complexity":33,"entry_point":34,"skill_class":35,"when_to_use":36,"requires_llm":37,"verdict_shape":38,"lifecycle_status":39,"invocation_schema":40,"supports_pipeline":37,"activation_triggers":41},"eks, kubernetes, compute, troubleshoot, pod-failure, crashloop, oom, image-pull, health-check","2","Jacky Chan — AWS Community Builder","aws-cloudops","Compute","0.1.0","EKS, Kubernetes, pod, CrashLoopBackOff, ImagePullBackOff, ErrImagePull, Pending, FailedScheduling, OOMKilled, Liveness probe, Readiness probe, Init container, ECR, kubectl, pod lifecycle, taints, tolerations, node affinity","troubleshoot","high","false","capability","Diagnosing why a Kubernetes pod on EKS is stuck Pending, crash-looping (CrashLoopBackOff), cannot pull a container image (ImagePullBackOff \u002F ErrImagePull), is repeatedly OOMKilled, fails liveness or readiness probes while Running, or has an init container stuck in Init:CrashLoopBackOff; interpreting kubectl describe pod events, container lastState.exitCode, lastState.reason, and conditions.","true","ROOT_CAUSE_FOUND | NEED_MORE_INFO | ESCALATE","active","Input: either (a) a symptom description (pod name, namespace, observed state, restart count, any error strings from kubectl get pods), OR (b) a live-cluster scenario where the agent runs kubectl describe pod \u002F logs \u002F get events to gather evidence. Output: a deterministic INCIDENT \u002F VERDICT \u002F ROOT_CAUSE \u002F EVIDENCE \u002F REMEDIATION block where VERDICT ∈ {ROOT_CAUSE_FOUND, NEED_MORE_INFO, ESCALATE} and ROOT_CAUSE names the specific failure category (CRASH_LOOP \u002F IMAGE_PULL \u002F PENDING \u002F OOM \u002F PROBE_FAILURE \u002F INIT_FAILURE) and the offending config element.","EKS pod CrashLoopBackOff, EKS pod ImagePullBackOff, EKS pod ErrImagePull, EKS pod Pending, EKS pod FailedScheduling, EKS pod OOMKilled, EKS pod liveness probe failing, EKS pod readiness probe failing, EKS init container failing, EKS pod keeps restarting, kubectl pod status","Agent runtime that reads SKILL.md (Claude Code, Cursor, Windsurf, Codex, Gemini). Offline diagnosis works on supplied kubectl describe \u002F logs \u002F get-events output. Live-cluster diagnosis uses kubectl get pods, describe pod, logs (incl. --previous and -c \u003Ccontainer>), get events, top nodes, top pods, aws eks describe-cluster, aws ec2 describe-vpc-endpoints, and aws ecr describe-images (kubectl v1.27+, AWS CLI v2, SSO or key-based credentials, kubeconfig pointing at the EKS cluster).","main","eeace51bd5dd1db2030729d83eeab988861553aa","skills\u002Feks-pod-troubleshooter",0,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills","https:\u002F\u002Fgithub.com\u002Fjhkchan","CTO & Co-Founder @Beever-AI  & @votee. CAISP. AWS Community Builder. Organizer@GDGCloudHK. MongoDB Champion. Technopreneur + Infornographer.","2026-09-24 08:54:10",{"source":8,"skillName":5,"commands":52,"prompt":62,"zipAvailable":63},[53,56,59],{"tool":54,"command":55},"npx","npx skills add jhkchan\u002Faws-agent-skills --skill eks-pod-troubleshooter",{"tool":57,"command":58},"bunx","bunx skills add jhkchan\u002Faws-agent-skills --skill eks-pod-troubleshooter",{"tool":60,"command":61},"pnpm","pnpm dlx skills add jhkchan\u002Faws-agent-skills --skill eks-pod-troubleshooter","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002FSKILL.md",true,[65,71,75,79,83,87,91,95,99,103,107,111,115,119,123,127,131,135],{"path":66,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":69,"sourceHtmlUrl":70},"eval\u002Ftest-cases.yaml","REFERENCE","参考文档",4813,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Feval\u002Ftest-cases.yaml",{"path":72,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":73,"sourceHtmlUrl":74},"evals\u002Fbaselines\u002Fcrashloop-missing-env-var.md",791,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Fevals\u002Fbaselines\u002Fcrashloop-missing-env-var.md",{"path":76,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":77,"sourceHtmlUrl":78},"evals\u002Fbaselines\u002Fimagepullbackoff-node-role-no-ecr.md",757,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Fevals\u002Fbaselines\u002Fimagepullbackoff-node-role-no-ecr.md",{"path":80,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":81,"sourceHtmlUrl":82},"evals\u002Fbaselines\u002Finsufficient-context-need-pod-name.md",344,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Fevals\u002Fbaselines\u002Finsufficient-context-need-pod-name.md",{"path":84,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":85,"sourceHtmlUrl":86},"evals\u002Fbaselines\u002Fliveness-probe-path-mismatch.md",628,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Fevals\u002Fbaselines\u002Fliveness-probe-path-mismatch.md",{"path":88,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":89,"sourceHtmlUrl":90},"evals\u002Fbaselines\u002Fpending-failedscheduling-taint.md",635,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Fevals\u002Fbaselines\u002Fpending-failedscheduling-taint.md",{"path":92,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":93,"sourceHtmlUrl":94},"evals\u002Feval_config.yaml",2004,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Fevals\u002Feval_config.yaml",{"path":96,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":97,"sourceHtmlUrl":98},"evals\u002Fevals.json",4503,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Fevals\u002Fevals.json",{"path":100,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":101,"sourceHtmlUrl":102},"evals\u002Fprompts\u002Fcrashloop-missing-env-var.md",1405,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Fevals\u002Fprompts\u002Fcrashloop-missing-env-var.md",{"path":104,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":105,"sourceHtmlUrl":106},"evals\u002Fprompts\u002Fimagepullbackoff-node-role-no-ecr.md",1680,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Fevals\u002Fprompts\u002Fimagepullbackoff-node-role-no-ecr.md",{"path":108,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":109,"sourceHtmlUrl":110},"evals\u002Fprompts\u002Finsufficient-context-need-pod-name.md",866,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Fevals\u002Fprompts\u002Finsufficient-context-need-pod-name.md",{"path":112,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":113,"sourceHtmlUrl":114},"evals\u002Fprompts\u002Fliveness-probe-path-mismatch.md",1724,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Fevals\u002Fprompts\u002Fliveness-probe-path-mismatch.md",{"path":116,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":117,"sourceHtmlUrl":118},"evals\u002Fprompts\u002Fpending-failedscheduling-taint.md",1491,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Fevals\u002Fprompts\u002Fpending-failedscheduling-taint.md",{"path":120,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":121,"sourceHtmlUrl":122},"examples\u002FREADME.md",4598,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Fexamples\u002FREADME.md",{"path":124,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":125,"sourceHtmlUrl":126},"references\u002Fadvanced-patterns.md",5934,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Freferences\u002Fadvanced-patterns.md",{"path":128,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":129,"sourceHtmlUrl":130},"references\u002Fdiagnostic-commands.md",19290,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Freferences\u002Fdiagnostic-commands.md",{"path":132,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":133,"sourceHtmlUrl":134},"references\u002Ffailure-decision-tree.md",34837,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Freferences\u002Ffailure-decision-tree.md",{"path":136,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":137,"sourceHtmlUrl":138},"skill.usf.yaml",4437,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Feks-pod-troubleshooter\u002Fskill.usf.yaml",[140,147,154,161,168,175],{"skillNo":141,"name":142,"nameZh":6,"description":143,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":144,"categories":145,"keywords":146,"publishedAt":19},"01M7ADVD60X4PCAPKSE58P9NDC","xray-tracing-deployer","Deploys AWS X-Ray distributed tracing with production-grade configuration: X-Ray daemon deployment (EC2 systemd, ECS sidecar, EKS DaemonSet, Lambda built-in), SDK instrumentation (Java, Python, Node.js, Go, .NET) with annotations and metadata, sampling rules (default reservoir + rate vs custom rules with URL \u002F method \u002F service-name predicates), service map generation, trace query and retrieval, groups and insights, CloudWatch ServiceLens integration, IAM permissions (PutTraceSegments, GetSamplingRules), and latest features (Lambda Powertools tracing, OpenTelemetry \u002F ADOT Collector support, W3C trace context). Emits a READY_TO_DEPLOY checklist with every configuration item verified. Use when enabling X-Ray on a new application, adding tracing to ECS \u002F EKS \u002F Lambda, validating a sampling configuration, or generating daemon deployment CLI and IaC templates. Triggers: X-Ray, distributed tracing, X-Ray daemon, sampling rules, service map, X-Ray SDK, OpenTelemetry, ADOT, Lambda Powertools, CloudWatch ServiceLens.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fxray-tracing-deployer\u002FSKILL.md",[],[],{"skillNo":148,"name":149,"nameZh":6,"description":150,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":151,"categories":152,"keywords":153,"publishedAt":19},"01M7ADTVJ7DCBM9CB1FY51GH7J","wellarchitected-workload-auditor","Audits AWS Well-Architected Tool workloads for review staleness, per-pillar high-risk issue counts, milestone tracking gaps, and remediation plan completeness. Emits a deterministic verdict (STALE_REVIEW | HIGH_RISK | CONFIG_GAP | OK) per workload with per-pillar risk breakdown, milestone trend analysis, and specific remediation. Use when reviewing Well-Architected workloads, checking WA review freshness, auditing high-risk issues per pillar, validating milestone coverage, or tracking remediation plans.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwellarchitected-workload-auditor\u002FSKILL.md",[],[],{"skillNo":155,"name":156,"nameZh":6,"description":157,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":158,"categories":159,"keywords":160,"publishedAt":19},"01M7ADT59G6FTTNZYHT2QRPNNP","wellarchitected-review-operator","Operates AWS Well-Architected Tool reviews end-to-end — creates workloads, runs pillar reviews (operational excellence, security, reliability, performance, cost optimization, sustainability, Prosperity), answers questions with risk-tier rationale, generates reports and improvement plans, creates milestones, and integrates with Trusted Advisor and Well-Architected Labs. Runs deterministic pre-checks (workload existence, lens availability, pillar coverage), emits the exact update-answer \u002F get-consolidated-report CLI behind a CONFIRM gate, and verifies risk-tier transitions. Emits READY | BLOCKED | COMPLETED. Use when creating a workload, answering pillar questions, generating an improvement plan, creating a milestone, or integrating with Trusted Advisor. Triggers: well-architected tool, well architected review, workload review, pillar, operational excellence, security, reliability, performance, cost optimization, sustainability, prosperity, improvement plan, milestone, trusted advisor.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwellarchitected-review-operator\u002FSKILL.md",[],[],{"skillNo":162,"name":163,"nameZh":6,"description":164,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":165,"categories":166,"keywords":167,"publishedAt":19},"01M7ADSH5JA9QHNPPHQCD7EVA7","wafv2-web-acl-deployer","Provisions AWS WAFv2 Web ACLs with correct production defaults: scope selection (CLOUDFRONT vs REGIONAL), managed rule groups (AWSManagedRulesCommonRuleSet, SQLiRuleSet, LinuxRuleSet, WindowsRuleSet, IPReputationList, BotControlRuleSet, AmazonIpReputationList, ATP), custom rules (byte match, IP set, regex pattern, geo match, size constraint, rate-based), rule priority ordering, logging destinations (Kinesis Firehose, CloudWatch Logs, S3), resource association (ALB, API Gateway, CloudFront), rate-based rules with aggregate keys (IP, forwarded IP), Challenge and CAPTCHA actions, and account takeover prevention. Emits a READY_TO_DEPLOY checklist. Use when creating a Web ACL, attaching managed rules, configuring rate limiting, enabling WAF logging, associating a Web ACL with an ALB\u002FAPI Gateway\u002FCloudFront, or configuring CAPTCHA\u002FChallenge actions. Triggers: create WAF Web ACL, WAFv2, managed rule groups, WAF rate limiting, WAF logging, WAF CAPTCHA, WAF Challenge action, bot control, ATP managed rules.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwafv2-web-acl-deployer\u002FSKILL.md",[],[],{"skillNo":169,"name":170,"nameZh":6,"description":171,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":172,"categories":173,"keywords":174,"publishedAt":19},"01M7ADRYFAT8R73FBNCHRC573X","wafv2-web-acl-auditor","Audits AWS WAFv2 Web ACL configurations to determine whether the ACL provides effective protection: default-action posture (Allow vs Block), managed-rule-group coverage gaps, rule effectiveness (BLOCK vs COUNT, shadow\u002Fbypass rules, stale exclusions), rate-based rule correctness, logging and visibility configuration, and text-transformation bypass vectors. Emits a deterministic verdict per Web ACL. Use when reviewing a WAFv2 Web ACL for security posture, checking managed-rule coverage, validating rate-limit configuration, auditing WAF logging, or hardening a WAF before production deployment. Triggers: WAF, WAFv2, Web ACL, managed rule group, AWSManagedRules, rate-based rule, CAPTCHA, Challenge, OverrideAction, Count mode, forwarded IP, Bot Control, ATP, text transformation, WAF logging, WAF visibility, OWASP, SQLi, XSS, RFI.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwafv2-web-acl-auditor\u002FSKILL.md",[],[],{"skillNo":176,"name":177,"nameZh":6,"description":178,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":179,"categories":180,"keywords":181,"publishedAt":19},"01M7ADR8W7G72JX4KFGN03QK8Q","waf-rule-deployer","Provisions AWS WAFv2 rule sets with production defaults: Web ACL creation (CloudFront scope in us-east-1 vs Regional scope), rule groups (managed vs custom) with capacity planning in WCU (Web ACL Capacity Units, 1500 default per ACL), managed rule groups (AWSManagedRulesCommonRuleSet, AWSManagedRulesSQLiRuleSet, AWSManagedRulesAmazonIpReputationList, AWSManagedRulesLinuxRuleSet, AWSManagedRulesWindowsRuleSet, AWSManagedRulesUnixRuleSet), custom rules (byte-match, regex pattern set, geo-match, rate-based), IP set and regex pattern set resources, rule priority ordering (lowest number evaluated first), action types (allow, block, count, CAPTCHA, Challenge), bot control managed rule group, account takeover. Triggers: create WAF rule, WAF managed rule group, WAF custom rule, WAF byte match, WAF geo match, WAF rate based rule, WAF IP set, WAF regex pattern set, WAF WCU budget, WAF CloudFront ACL, WAF logging Kinesis Firehose, WAF bot control, WAF ATP, WAF label matching, WAF Challenge action, WAF CAPTCHA.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwaf-rule-deployer\u002FSKILL.md",[],[]]