[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M78JCY1FQ07FN398RAV8NG3C":3,"skill-same:01M78JCY1FQ07FN398RAV8NG3C":82},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":34,"commitSha":35,"skillDirectory":36,"forksCount":13,"repositoryHtmlUrl":37,"repositoryHomepageUrl":38,"ownerHtmlUrl":39,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":40,"install":41,"resources":54},"01M78JCY1FQ07FN398RAV8NG3C","tao-run-on-kubernetes",null,"Kubernetes execution platform — submits TAO container jobs as single-pod k8s Jobs with NVIDIA GPU scheduling. Use when running on EKS \u002F GKE \u002F AKS \u002F on-prem clusters with the NVIDIA GPU Operator installed, or when integrating TAO into an existing k8s-native ML platform.","matheussluzz\u002FZeuZ-Agent","matheussluzz","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F209255758?v=4","01M78HKHH1D1ASC9KC9BG583DH","01M78HKHHP6W7XRXQM2SBCAM3R",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Ftao-run-on-kubernetes\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: tao-run-on-kubernetes\ndescription: Kubernetes execution platform — submits TAO container jobs as single-pod k8s Jobs with NVIDIA GPU scheduling.\n  Use when running on EKS \u002F GKE \u002F AKS \u002F on-prem clusters with the NVIDIA GPU Operator installed, or when integrating TAO\n  into an existing k8s-native ML platform.\nlicense: Apache-2.0\ncompatibility: Requires GPU worker nodes with NVIDIA driver branch 580, CUDA Toolkit 13.0, and NVIDIA Container Toolkit 1.19.0; the nvidia-tao-sdk Python package with the kubernetes extra (pip install 'nvidia-tao-sdk[kubernetes]'); an authenticated cluster; and the NVIDIA GPU Operator or device plugin.\nmetadata:\n  author: NVIDIA Corporation\n  version: \"0.1.0\"\nallowed-tools: Read Bash\ntags:\n- kubernetes\n- k8s\n- gpu\n- compute\n- container\n---\n\n# Kubernetes\n\nSubmits TAO container jobs as Kubernetes Jobs. Works on any cluster reachable via kubeconfig (EKS \u002F GKE \u002F AKS \u002F on-prem) or in-cluster service account (when the SDK runs inside a pod).\n\nSingle-pod by default; opt into multi-node distributed training via `num_nodes > 1` (uses Indexed Job + headless Service, see [Multi-node training](#multi-node-training-distributed) below).\n\n## Preflight\n\nFour checks: GPU host runtime ready, SDK installed, cluster reachable, GPU\nOperator\u002Fdevice plugin present.\n\n```bash\n# 0. GPU node host runtime.\n# Run this on each self-managed GPU worker node or in the node image build.\n# Set TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1 only when using managed GPU nodes whose\n# driver\u002Ftoolkit lifecycle is owned by the cloud provider or GPU Operator policy.\nif [ \"${TAO_K8S_SKIP_NODE_RUNTIME_CHECK:-0}\" != \"1\" ]; then\n  TAO_SKILL_BANK_ROOT=\"${TAO_SKILL_BANK_ROOT:-$PWD}\"\n  SETUP_SCRIPT=\"${TAO_SKILL_BANK_ROOT}\u002Fplatform\u002Ftao-setup-nvidia-gpu-host\u002Fscripts\u002Fsetup-nvidia-gpu-host.sh\"\n\n  bash \"$SETUP_SCRIPT\" --backend kubernetes --check-only || {\n    echo \"MISSING: TAO Kubernetes GPU node runtime is not ready.\"\n    echo \"For self-managed GPU nodes, run after user approval:\"\n    echo \"  bash \\\"$SETUP_SCRIPT\\\" --backend kubernetes --install --yes\"\n    echo \"For managed clusters, verify the node image\u002FGPU Operator policy installs driver 580 and toolkit 1.19.0, then set TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1.\"\n    exit 1\n  }\nfi\n\n# 1. SDK + kubernetes extra installed.\n# nvidia-tao-sdk is on public PyPI; pin lives in versions.yaml (wheels.tao_sdk_kubernetes).\nPIN=$(\"${TAO_SKILL_BANK_PATH:?}\u002Fscripts\u002Fresolve_versions_key.py\" wheels.tao_sdk_kubernetes)\npython -c \"import tao_sdk\" 2>\u002Fdev\u002Fnull || {\n  echo \"Installing missing Python requirement: $PIN\"\n  python -m pip install \"$PIN\"\n}\npython -c \"import kubernetes\" 2>\u002Fdev\u002Fnull || {\n  echo \"Installing missing Python requirement: $PIN\"\n  python -m pip install \"$PIN\"\n}\npython -c \"import tao_sdk, kubernetes\"\n\n# 2. Cluster reachable (kubeconfig OR in-cluster service account)\npython -c \"from kubernetes import config; config.load_kube_config()\" 2>\u002Fdev\u002Fnull || \\\n  python -c \"from kubernetes import config; config.load_incluster_config()\" 2>\u002Fdev\u002Fnull || {\n    echo \"MISSING: no kubeconfig at ~\u002F.kube\u002Fconfig and not running in a pod.\"\n    echo \"Configure kubectl (e.g., 'aws eks update-kubeconfig --name my-cluster') or set \\$KUBECONFIG.\"\n    exit 1\n  }\n\n# 3. NVIDIA GPU Operator present (soft check — warn if kubectl available, don't fail)\nif command -v kubectl >\u002Fdev\u002Fnull 2>&1; then\n  gpu=$(kubectl get nodes -o jsonpath='{range .items[*]}{.status.allocatable.nvidia\\.com\u002Fgpu}{\"\\n\"}{end}' 2>\u002Fdev\u002Fnull | grep -v '^$' | head -1)\n  if [ -z \"$gpu\" ] || [ \"$gpu\" = \"0\" ]; then\n    echo \"WARN: no nvidia.com\u002Fgpu allocatable on this cluster.\"\n    echo \"Install the NVIDIA GPU Operator before submitting GPU jobs:\"\n    echo \"  https:\u002F\u002Fdocs.nvidia.com\u002Fdatacenter\u002Fcloud-native\u002Fgpu-operator\u002Flatest\u002Fgetting-started.html\"\n  fi\nfi\n```\n\nThe GPU node runtime check is mandatory for self-managed nodes. For managed\nclusters where the client is not running on a GPU worker, verify the provider\nnode image or GPU Operator policy and set `TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1`\ninstead of running the installer on the client. The final GPU capacity check is\na warning rather than a hard fail — `kubectl` isn't always installed. The SDK\ndoes a hard guard inside\n`KubernetesSDK.create_job()` that uses the kubernetes Python client to verify\nGPU capacity before submitting.\n\n## Credentials & configuration\n\n- **Kubeconfig** (one of):\n  - `~\u002F.kube\u002Fconfig` — default discovery path\n  - `$KUBECONFIG` — alternate path\n  - In-cluster service account — used when running inside a pod (no kubeconfig needed)\n- **TAO_K8S_NAMESPACE** (optional): default namespace for Job submission. Defaults to `default`.\n- **TAO_K8S_CONTEXT** (optional): kubeconfig context name to switch clusters.\n- **NGC_KEY** (optional): for nvcr.io image pulls. If you've pre-created an image-pull secret in the target namespace, pass its name to `create_job` via the `image_pull_secret` argument.\n- **ACCESS_KEY \u002F SECRET_KEY \u002F S3_BUCKET_NAME \u002F S3_ENDPOINT_URL** (optional): for S3 dataset I\u002FO via the SDK's `inputs`\u002F`outputs` script_runner wrapping.\n\nDo not ask for Brev or SLURM credentials for Kubernetes runs. Ask for\nS3 credentials only when the selected workflow uses `s3:\u002F\u002F` inputs or outputs,\nand ask for model-specific credentials such as `HF_TOKEN` only when the selected\nmodel requires them. Before launch, verify the selected namespace can create\nJobs, dataset\u002Fresult paths are visible from the pod, and PVC\u002Fmounted filesystem\npaths are proven to be mounted into the job container; an agent-host local path\nis not sufficient proof.\n\n## SDK API\n\nK8s is SDK-only — there is no `kubectl`-only launch path. Read\n`tao-skill-bank:tao-run-platform` before drafting `create_job` calls; it covers\n`build_entrypoint`, the shared kwarg contract, monitoring, and `ActionWorkflow`.\n\n```python\nfrom tao_sdk.platforms.kubernetes import KubernetesSDK\n\nsdk = KubernetesSDK()  # auto-detects auth\njob = sdk.create_job(\n    image='nvcr.io\u002Fnvidia\u002Ftao\u002Ftao-toolkit:6.26.3-pyt',\n    command='dino train -e \u002Ftmp\u002Fspec.yaml',\n    gpu_count=1,\n    env_vars={'NGC_KEY': os.environ['NGC_KEY']},\n    inputs={'\u002Fdata\u002Ftrain.json': 's3:\u002F\u002Fbucket\u002Fcoco\u002Ftrain.json'},\n    outputs=['\u002Fresults\u002F'],\n    namespace='tao-jobs',                       # optional override\n    image_pull_secret='ngc-pull-secret',         # optional, pre-created\n    node_selector={'gpu-type': 'h100'},          # optional\n)\n```\n\nThe SDK constructs a `V1Job` with:\n- `spec.template.spec.containers[0]`: the requested image and `command=[\"\u002Fbin\u002Fbash\", \"-c\", \u003Ccommand>]`.\n- `resources.limits[\"nvidia.com\u002Fgpu\"]: \u003Cgpu_count>` — schedules onto GPU nodes via the NVIDIA Device Plugin \u002F GPU Operator.\n- `env_vars` flowed through, plus auto-injected S3\u002FNGC\u002FHF credentials for `script_runner`.\n- `restart_policy=Never` and `backoff_limit=0` — failures surface to the user instead of silently retrying.\n- `ttl_seconds_after_finished=3600` — Job auto-cleans 1 hour after terminal state.\n\n## Status & monitoring\n\n```python\nstatus = sdk.get_job_status(job.id)\n# status.status ∈ {\"Pending\", \"Running\", \"Complete\", \"Error\", \"Canceled\", \"Unknown\"}\n\nlogs = sdk.get_job_logs(job.id, tail=200)  # concatenates logs from all pods of the Job\n\n# For stuck-Pending jobs — replica diagnostics:\nfor r in sdk.get_job_replicas(job.id):\n    issue = r[\"status\"].get(\"readiness_issue\")\n    if issue:\n        print(issue[\"reason\"], issue[\"message\"])\n        # e.g. \"ImagePullBackOff\" \u002F \"Back-off pulling image...\"\n        # e.g. \"Pending\"           \u002F \"0\u002F3 nodes available: 3 Insufficient nvidia.com\u002Fgpu\"\n\n# On failure:\nanalysis = sdk.get_failure_analysis(job.id)\n# {\"err_class\": \"ERR_PROGRAM\" | \"ERR_INFRA\",\n#  \"suggestion\": \"Container OOM-killed. Reduce batch size...\",\n#  \"job_failure_by_node_event\": [{\"node_event_name\": \"OOMKilled\", ...}]}\n```\n\n## Cancel & cleanup\n\n```python\nsdk.cancel_job(job.id)  # delete_namespaced_job with propagation_policy=\"Foreground\"\n```\n\n`ttl_seconds_after_finished=3600` means completed Jobs auto-delete after 1h. To cancel an in-flight Job, `cancel_job` deletes it and its pods immediately.\n\n## GPU Operator dependency\n\nThe SDK refuses to submit GPU jobs to a cluster with no `nvidia.com\u002Fgpu` allocatable. For self-managed clusters, first run the `tao-setup-nvidia-gpu-host` install action on every GPU worker node or bake the same package set into the node image:\n\n```bash\nbash skills\u002Fplatform\u002Ftao-setup-nvidia-gpu-host\u002Fscripts\u002Fsetup-nvidia-gpu-host.sh --backend kubernetes --install --yes\n```\n\nThen install the NVIDIA GPU Operator or device plugin:\n\n```bash\nhelm repo add nvidia https:\u002F\u002Fhelm.ngc.nvidia.com\u002Fnvidia\nhelm repo update\nhelm install --wait gpu-operator -n gpu-operator --create-namespace nvidia\u002Fgpu-operator\n```\n\nFull guide: https:\u002F\u002Fdocs.nvidia.com\u002Fdatacenter\u002Fcloud-native\u002Fgpu-operator\u002Flatest\u002Fgetting-started.html\n\n## Multi-node training (distributed)\n\nPass `num_nodes > 1` to `create_job()` to run distributed training across N pods. The SDK provisions:\n\n1. A **headless Service** named after the Job (selector: `job-name=\u003Cjob-name>`, `clusterIP: None`, `publishNotReadyAddresses: true` so pods can rendezvous before they're all Ready).\n2. An **Indexed Job** with `parallelism = completions = num_nodes`, `completionMode: Indexed`. Each pod gets `JOB_COMPLETION_INDEX` injected by k8s automatically (= the node rank).\n3. A **command wrapper** that exports the rendezvous env vars before invoking the user command. Two naming conventions are exported simultaneously:\n\n   | Env var | Value | Read by |\n   |---|---|---|\n   | `WORLD_SIZE` | `num_nodes` | TAO PyTorch container's `nvidia_tao_pytorch\u002Fcore\u002Fentrypoint.py` (uses this to mean *node count*, even though PyTorch's own convention is *total processes*) |\n   | `NUM_GPU_PER_NODE` | `gpu_count` | TAO PyTorch container's entrypoint |\n   | `NNODES` | `num_nodes` | `torchrun` and PyTorch-standard rendezvous |\n   | `NPROC_PER_NODE` | `gpu_count` | `torchrun` |\n   | `NODE_RANK` | `$JOB_COMPLETION_INDEX` | both |\n   | `MASTER_ADDR` | `\u003Cjob-name>-0.\u003Cjob-name>` (pod-0's DNS) | both |\n   | `MASTER_PORT` | `29500` | both (TAO's default) |\n\n   Both naming conventions are set so TAO entrypoints (`dino train`, etc.) and raw `torchrun` commands work without modification.\n\n```python\njob = sdk.create_job(\n    image='nvcr.io\u002Fnvidia\u002Ftao\u002Ftao-toolkit:6.26.3-pyt',\n    command='dino train -e \u002Ftmp\u002Fspec.yaml',  # TAO entrypoint reads spec.train.num_nodes; env vars are wired by the container\n    gpu_count=8,           # GPUs per node\n    num_nodes=4,           # 4 × 8 = 32 GPUs total\n    inputs={'\u002Fdata\u002Ftrain.json': 's3:\u002F\u002Fbucket\u002Fcoco\u002Ftrain.json'},\n    outputs=['\u002Fresults\u002F'],\n)\n```\n\nFor raw `torchrun`-based commands (non-TAO containers):\n\n```python\njob = sdk.create_job(\n    image='nvcr.io\u002Fnvidia\u002Fpytorch:25.08-py3',\n    command='torchrun --nnodes=$NNODES --nproc-per-node=$NPROC_PER_NODE --node-rank=$NODE_RANK '\n            '--master-addr=$MASTER_ADDR --master-port=$MASTER_PORT train.py',\n    gpu_count=8,\n    num_nodes=4,\n)\n```\n\nThe capacity check sums across nodes: `gpu_count × num_nodes` ≤ cluster's allocatable `nvidia.com\u002Fgpu`.\n\n### Cluster requirements for multi-node\n\n- **k8s 1.28+** is required for stable pod hostnames in Indexed Jobs (the `PodIndexLabel` feature). On older clusters the `MASTER_ADDR=\u003Cjob>-0.\u003Csvc>` DNS lookup fails. Verify with `kubectl version`.\n- **Pod-to-pod networking** must be open on port 29500 (PyTorch default; configurable via `MASTER_PORT` env var). Most CNIs (Calico, Cilium, AWS VPC CNI) allow this by default; restrictive NetworkPolicies must be relaxed.\n- **NCCL** in the container talks GPU-to-GPU; if the cluster has multi-NIC nodes or RDMA, set `NCCL_SOCKET_IFNAME` \u002F `NCCL_IB_HCA` via `env_vars`.\n\n### Reference reading\n\n- Kubernetes Indexed Job: \u003Chttps:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fcontrollers\u002Fjob\u002F#completion-mode>\n- Indexed Job for batch ML: \u003Chttps:\u002F\u002Fkubernetes.io\u002Fblog\u002F2022\u002F06\u002F01\u002Findexed-jobs-mpi\u002F>\n- PyTorch distributed (env-var rendezvous): \u003Chttps:\u002F\u002Fpytorch.org\u002Fdocs\u002Fstable\u002Felastic\u002Frun.html>\n- NCCL networking tuning (NCCL_SOCKET_IFNAME, NCCL_IB_HCA): \u003Chttps:\u002F\u002Fdocs.nvidia.com\u002Fdeeplearning\u002Fnccl\u002Fuser-guide\u002Fdocs\u002Fenv.html>\n\n### Kubernetes operator alternatives\n\nFor more sophisticated topologies (gang scheduling, PyTorch elastic \u002F fault-tolerant training, MPI \u002F Horovod, RDMA setup), reach for an operator instead of plain Indexed Job:\n\n- **MPI Operator** — \u003Chttps:\u002F\u002Fgithub.com\u002Fkubeflow\u002Fmpi-operator> — for MPI \u002F Horovod workloads.\n- **Kubeflow Training Operator** (`PyTorchJob`, `TFJob`) — \u003Chttps:\u002F\u002Fwww.kubeflow.org\u002Fdocs\u002Fcomponents\u002Ftraining\u002F> — for elastic PyTorch training with built-in restart logic.\n- **Volcano** — \u003Chttps:\u002F\u002Fvolcano.sh\u002F> — gang scheduling, queues, fair-share. Useful in shared multi-tenant clusters.\n- **Kueue** — \u003Chttps:\u002F\u002Fkueue.sigs.k8s.io\u002F> — quota \u002F queue layer on top of any of the above.\n\nThe TAO SDK's Indexed Job path is intentionally simple and dependency-free; if you need elastic restart or gang scheduling, layer one of these on top and submit jobs through the operator's CRD instead.\n\n## Common error patterns\n\n**`No nvidia.com\u002Fgpu resources allocatable on the cluster`** — the GPU Operator (or NVIDIA Device Plugin) isn't installed. Install per the link above; verify with `kubectl get nodes -o jsonpath='{.items[*].status.allocatable}'`.\n\n**`ImagePullBackOff` \u002F `ErrImagePull`** — the cluster can't pull the image. For nvcr.io: pre-create an image-pull secret in the namespace and pass its name via the `image_pull_secret` argument:\n```bash\nkubectl create secret docker-registry ngc-pull-secret \\\n  --docker-server=nvcr.io \\\n  --docker-username='$oauthtoken' \\\n  --docker-password=$NGC_KEY -n tao-jobs\n```\n\n**Pod stays `Pending` forever** — `get_job_replicas(job_id)` will show the readiness_issue. Common causes: insufficient GPU capacity (`Insufficient nvidia.com\u002Fgpu`), no node matches `node_selector`, missing image-pull secret, or PVC mount failure.\n\n**`OOMKilled` (exit 137)** — container exceeded memory. Reduce batch size, lower max_length, or add a memory request\u002Flimit and target a larger node.\n\n**`CredentialError: Could not authenticate to a Kubernetes cluster`** — neither kubeconfig nor in-cluster auth worked. Run `kubectl get nodes` to verify your config, or set `$KUBECONFIG` to the right path.\n\n## What this skill does NOT support (yet)\n\n- **Elastic \u002F fault-tolerant training.** Indexed Job has `backoff_limit=0` — failures fail the whole training run. For elastic restart (e.g., resume from checkpoint after a node death), use Kubeflow's `PyTorchJob` operator instead.\n- **Gang scheduling.** Indexed Job pods are scheduled independently — no all-or-nothing. Multi-node training will *partially* start if only some pods can be scheduled (rank-0 will hang waiting for peers). For all-or-nothing scheduling on shared clusters, use Volcano or Kueue.\n- **MPI \u002F Horovod.** Use the MPI Operator. The Indexed Job path here is PyTorch-distributed-shaped (env-var rendezvous on `MASTER_ADDR:MASTER_PORT`).\n- **Persistent volumes for shared storage.** S3 only via the script_runner. PVC support is a follow-up.\n- **Auto-creating image-pull secrets from `$NGC_KEY`.** You pre-create the secret in the target namespace and pass the name. K8s namespace conventions vary widely, so we keep secret creation explicit.\n","# Kubernetes\n\nSubmits TAO container jobs as Kubernetes Jobs. Works on any cluster reachable via kubeconfig (EKS \u002F GKE \u002F AKS \u002F on-prem) or in-cluster service account (when the SDK runs inside a pod).\n\nSingle-pod by default; opt into multi-node distributed training via `num_nodes > 1` (uses Indexed Job + headless Service, see [Multi-node training](#multi-node-training-distributed) below).\n\n## Preflight\n\nFour checks: GPU host runtime ready, SDK installed, cluster reachable, GPU\nOperator\u002Fdevice plugin present.\n\n```bash\n# 0. GPU node host runtime.\n# Run this on each self-managed GPU worker node or in the node image build.\n# Set TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1 only when using managed GPU nodes whose\n# driver\u002Ftoolkit lifecycle is owned by the cloud provider or GPU Operator policy.\nif [ \"${TAO_K8S_SKIP_NODE_RUNTIME_CHECK:-0}\" != \"1\" ]; then\n  TAO_SKILL_BANK_ROOT=\"${TAO_SKILL_BANK_ROOT:-$PWD}\"\n  SETUP_SCRIPT=\"${TAO_SKILL_BANK_ROOT}\u002Fplatform\u002Ftao-setup-nvidia-gpu-host\u002Fscripts\u002Fsetup-nvidia-gpu-host.sh\"\n\n  bash \"$SETUP_SCRIPT\" --backend kubernetes --check-only || {\n    echo \"MISSING: TAO Kubernetes GPU node runtime is not ready.\"\n    echo \"For self-managed GPU nodes, run after user approval:\"\n    echo \"  bash \\\"$SETUP_SCRIPT\\\" --backend kubernetes --install --yes\"\n    echo \"For managed clusters, verify the node image\u002FGPU Operator policy installs driver 580 and toolkit 1.19.0, then set TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1.\"\n    exit 1\n  }\nfi\n\n# 1. SDK + kubernetes extra installed.\n# nvidia-tao-sdk is on public PyPI; pin lives in versions.yaml (wheels.tao_sdk_kubernetes).\nPIN=$(\"${TAO_SKILL_BANK_PATH:?}\u002Fscripts\u002Fresolve_versions_key.py\" wheels.tao_sdk_kubernetes)\npython -c \"import tao_sdk\" 2>\u002Fdev\u002Fnull || {\n  echo \"Installing missing Python requirement: $PIN\"\n  python -m pip install \"$PIN\"\n}\npython -c \"import kubernetes\" 2>\u002Fdev\u002Fnull || {\n  echo \"Installing missing Python requirement: $PIN\"\n  python -m pip install \"$PIN\"\n}\npython -c \"import tao_sdk, kubernetes\"\n\n# 2. Cluster reachable (kubeconfig OR in-cluster service account)\npython -c \"from kubernetes import config; config.load_kube_config()\" 2>\u002Fdev\u002Fnull || \\\n  python -c \"from kubernetes import config; config.load_incluster_config()\" 2>\u002Fdev\u002Fnull || {\n    echo \"MISSING: no kubeconfig at ~\u002F.kube\u002Fconfig and not running in a pod.\"\n    echo \"Configure kubectl (e.g., 'aws eks update-kubeconfig --name my-cluster') or set \\$KUBECONFIG.\"\n    exit 1\n  }\n\n# 3. NVIDIA GPU Operator present (soft check — warn if kubectl available, don't fail)\nif command -v kubectl >\u002Fdev\u002Fnull 2>&1; then\n  gpu=$(kubectl get nodes -o jsonpath='{range .items[*]}{.status.allocatable.nvidia\\.com\u002Fgpu}{\"\\n\"}{end}' 2>\u002Fdev\u002Fnull | grep -v '^$' | head -1)\n  if [ -z \"$gpu\" ] || [ \"$gpu\" = \"0\" ]; then\n    echo \"WARN: no nvidia.com\u002Fgpu allocatable on this cluster.\"\n    echo \"Install the NVIDIA GPU Operator before submitting GPU jobs:\"\n    echo \"  https:\u002F\u002Fdocs.nvidia.com\u002Fdatacenter\u002Fcloud-native\u002Fgpu-operator\u002Flatest\u002Fgetting-started.html\"\n  fi\nfi\n```\n\nThe GPU node runtime check is mandatory for self-managed nodes. For managed\nclusters where the client is not running on a GPU worker, verify the provider\nnode image or GPU Operator policy and set `TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1`\ninstead of running the installer on the client. The final GPU capacity check is\na warning rather than a hard fail — `kubectl` isn't always installed. The SDK\ndoes a hard guard inside\n`KubernetesSDK.create_job()` that uses the kubernetes Python client to verify\nGPU capacity before submitting.\n\n## Credentials & configuration\n\n- **Kubeconfig** (one of):\n  - `~\u002F.kube\u002Fconfig` — default discovery path\n  - `$KUBECONFIG` — alternate path\n  - In-cluster service account — used when running inside a pod (no kubeconfig needed)\n- **TAO_K8S_NAMESPACE** (optional): default namespace for Job submission. Defaults to `default`.\n- **TAO_K8S_CONTEXT** (optional): kubeconfig context name to switch clusters.\n- **NGC_KEY** (optional): for nvcr.io image pulls. If you've pre-created an image-pull secret in the target namespace, pass its name to `create_job` via the `image_pull_secret` argument.\n- **ACCESS_KEY \u002F SECRET_KEY \u002F S3_BUCKET_NAME \u002F S3_ENDPOINT_URL** (optional): for S3 dataset I\u002FO via the SDK's `inputs`\u002F`outputs` script_runner wrapping.\n\nDo not ask for Brev or SLURM credentials for Kubernetes runs. Ask for\nS3 credentials only when the selected workflow uses `s3:\u002F\u002F` inputs or outputs,\nand ask for model-specific credentials such as `HF_TOKEN` only when the selected\nmodel requires them. Before launch, verify the selected namespace can create\nJobs, dataset\u002Fresult paths are visible from the pod, and PVC\u002Fmounted filesystem\npaths are proven to be mounted into the job container; an agent-host local path\nis not sufficient proof.\n\n## SDK API\n\nK8s is SDK-only — there is no `kubectl`-only launch path. Read\n`tao-skill-bank:tao-run-platform` before drafting `create_job` calls; it covers\n`build_entrypoint`, the shared kwarg contract, monitoring, and `ActionWorkflow`.\n\n```python\nfrom tao_sdk.platforms.kubernetes import KubernetesSDK\n\nsdk = KubernetesSDK()  # auto-detects auth\njob = sdk.create_job(\n    image='nvcr.io\u002Fnvidia\u002Ftao\u002Ftao-toolkit:6.26.3-pyt',\n    command='dino train -e \u002Ftmp\u002Fspec.yaml',\n    gpu_count=1,\n    env_vars={'NGC_KEY': os.environ['NGC_KEY']},\n    inputs={'\u002Fdata\u002Ftrain.json': 's3:\u002F\u002Fbucket\u002Fcoco\u002Ftrain.json'},\n    outputs=['\u002Fresults\u002F'],\n    namespace='tao-jobs',                       # optional override\n    image_pull_secret='ngc-pull-secret',         # optional, pre-created\n    node_selector={'gpu-type': 'h100'},          # optional\n)\n```\n\nThe SDK constructs a `V1Job` with:\n- `spec.template.spec.containers[0]`: the requested image and `command=[\"\u002Fbin\u002Fbash\", \"-c\", \u003Ccommand>]`.\n- `resources.limits[\"nvidia.com\u002Fgpu\"]: \u003Cgpu_count>` — schedules onto GPU nodes via the NVIDIA Device Plugin \u002F GPU Operator.\n- `env_vars` flowed through, plus auto-injected S3\u002FNGC\u002FHF credentials for `script_runner`.\n- `restart_policy=Never` and `backoff_limit=0` — failures surface to the user instead of silently retrying.\n- `ttl_seconds_after_finished=3600` — Job auto-cleans 1 hour after terminal state.\n\n## Status & monitoring\n\n```python\nstatus = sdk.get_job_status(job.id)\n# status.status ∈ {\"Pending\", \"Running\", \"Complete\", \"Error\", \"Canceled\", \"Unknown\"}\n\nlogs = sdk.get_job_logs(job.id, tail=200)  # concatenates logs from all pods of the Job\n\n# For stuck-Pending jobs — replica diagnostics:\nfor r in sdk.get_job_replicas(job.id):\n    issue = r[\"status\"].get(\"readiness_issue\")\n    if issue:\n        print(issue[\"reason\"], issue[\"message\"])\n        # e.g. \"ImagePullBackOff\" \u002F \"Back-off pulling image...\"\n        # e.g. \"Pending\"           \u002F \"0\u002F3 nodes available: 3 Insufficient nvidia.com\u002Fgpu\"\n\n# On failure:\nanalysis = sdk.get_failure_analysis(job.id)\n# {\"err_class\": \"ERR_PROGRAM\" | \"ERR_INFRA\",\n#  \"suggestion\": \"Container OOM-killed. Reduce batch size...\",\n#  \"job_failure_by_node_event\": [{\"node_event_name\": \"OOMKilled\", ...}]}\n```\n\n## Cancel & cleanup\n\n```python\nsdk.cancel_job(job.id)  # delete_namespaced_job with propagation_policy=\"Foreground\"\n```\n\n`ttl_seconds_after_finished=3600` means completed Jobs auto-delete after 1h. To cancel an in-flight Job, `cancel_job` deletes it and its pods immediately.\n\n## GPU Operator dependency\n\nThe SDK refuses to submit GPU jobs to a cluster with no `nvidia.com\u002Fgpu` allocatable. For self-managed clusters, first run the `tao-setup-nvidia-gpu-host` install action on every GPU worker node or bake the same package set into the node image:\n\n```bash\nbash skills\u002Fplatform\u002Ftao-setup-nvidia-gpu-host\u002Fscripts\u002Fsetup-nvidia-gpu-host.sh --backend kubernetes --install --yes\n```\n\nThen install the NVIDIA GPU Operator or device plugin:\n\n```bash\nhelm repo add nvidia https:\u002F\u002Fhelm.ngc.nvidia.com\u002Fnvidia\nhelm repo update\nhelm install --wait gpu-operator -n gpu-operator --create-namespace nvidia\u002Fgpu-operator\n```\n\nFull guide: https:\u002F\u002Fdocs.nvidia.com\u002Fdatacenter\u002Fcloud-native\u002Fgpu-operator\u002Flatest\u002Fgetting-started.html\n\n## Multi-node training (distributed)\n\nPass `num_nodes > 1` to `create_job()` to run distributed training across N pods. The SDK provisions:\n\n1. A **headless Service** named after the Job (selector: `job-name=\u003Cjob-name>`, `clusterIP: None`, `publishNotReadyAddresses: true` so pods can rendezvous before they're all Ready).\n2. An **Indexed Job** with `parallelism = completions = num_nodes`, `completionMode: Indexed`. Each pod gets `JOB_COMPLETION_INDEX` injected by k8s automatically (= the node rank).\n3. A **command wrapper** that exports the rendezvous env vars before invoking the user command. Two naming conventions are exported simultaneously:\n\n   | Env var | Value | Read by |\n   |---|---|---|\n   | `WORLD_SIZE` | `num_nodes` | TAO PyTorch container's `nvidia_tao_pytorch\u002Fcore\u002Fentrypoint.py` (uses this to mean *node count*, even though PyTorch's own convention is *total processes*) |\n   | `NUM_GPU_PER_NODE` | `gpu_count` | TAO PyTorch container's entrypoint |\n   | `NNODES` | `num_nodes` | `torchrun` and PyTorch-standard rendezvous |\n   | `NPROC_PER_NODE` | `gpu_count` | `torchrun` |\n   | `NODE_RANK` | `$JOB_COMPLETION_INDEX` | both |\n   | `MASTER_ADDR` | `\u003Cjob-name>-0.\u003Cjob-name>` (pod-0's DNS) | both |\n   | `MASTER_PORT` | `29500` | both (TAO's default) |\n\n   Both naming conventions are set so TAO entrypoints (`dino train`, etc.) and raw `torchrun` commands work without modification.\n\n```python\njob = sdk.create_job(\n    image='nvcr.io\u002Fnvidia\u002Ftao\u002Ftao-toolkit:6.26.3-pyt',\n    command='dino train -e \u002Ftmp\u002Fspec.yaml',  # TAO entrypoint reads spec.train.num_nodes; env vars are wired by the container\n    gpu_count=8,           # GPUs per node\n    num_nodes=4,           # 4 × 8 = 32 GPUs total\n    inputs={'\u002Fdata\u002Ftrain.json': 's3:\u002F\u002Fbucket\u002Fcoco\u002Ftrain.json'},\n    outputs=['\u002Fresults\u002F'],\n)\n```\n\nFor raw `torchrun`-based commands (non-TAO containers):\n\n```python\njob = sdk.create_job(\n    image='nvcr.io\u002Fnvidia\u002Fpytorch:25.08-py3',\n    command='torchrun --nnodes=$NNODES --nproc-per-node=$NPROC_PER_NODE --node-rank=$NODE_RANK '\n            '--master-addr=$MASTER_ADDR --master-port=$MASTER_PORT train.py',\n    gpu_count=8,\n    num_nodes=4,\n)\n```\n\nThe capacity check sums across nodes: `gpu_count × num_nodes` ≤ cluster's allocatable `nvidia.com\u002Fgpu`.\n\n### Cluster requirements for multi-node\n\n- **k8s 1.28+** is required for stable pod hostnames in Indexed Jobs (the `PodIndexLabel` feature). On older clusters the `MASTER_ADDR=\u003Cjob>-0.\u003Csvc>` DNS lookup fails. Verify with `kubectl version`.\n- **Pod-to-pod networking** must be open on port 29500 (PyTorch default; configurable via `MASTER_PORT` env var). Most CNIs (Calico, Cilium, AWS VPC CNI) allow this by default; restrictive NetworkPolicies must be relaxed.\n- **NCCL** in the container talks GPU-to-GPU; if the cluster has multi-NIC nodes or RDMA, set `NCCL_SOCKET_IFNAME` \u002F `NCCL_IB_HCA` via `env_vars`.\n\n### Reference reading\n\n- Kubernetes Indexed Job: \u003Chttps:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fcontrollers\u002Fjob\u002F#completion-mode>\n- Indexed Job for batch ML: \u003Chttps:\u002F\u002Fkubernetes.io\u002Fblog\u002F2022\u002F06\u002F01\u002Findexed-jobs-mpi\u002F>\n- PyTorch distributed (env-var rendezvous): \u003Chttps:\u002F\u002Fpytorch.org\u002Fdocs\u002Fstable\u002Felastic\u002Frun.html>\n- NCCL networking tuning (NCCL_SOCKET_IFNAME, NCCL_IB_HCA): \u003Chttps:\u002F\u002Fdocs.nvidia.com\u002Fdeeplearning\u002Fnccl\u002Fuser-guide\u002Fdocs\u002Fenv.html>\n\n### Kubernetes operator alternatives\n\nFor more sophisticated topologies (gang scheduling, PyTorch elastic \u002F fault-tolerant training, MPI \u002F Horovod, RDMA setup), reach for an operator instead of plain Indexed Job:\n\n- **MPI Operator** — \u003Chttps:\u002F\u002Fgithub.com\u002Fkubeflow\u002Fmpi-operator> — for MPI \u002F Horovod workloads.\n- **Kubeflow Training Operator** (`PyTorchJob`, `TFJob`) — \u003Chttps:\u002F\u002Fwww.kubeflow.org\u002Fdocs\u002Fcomponents\u002Ftraining\u002F> — for elastic PyTorch training with built-in restart logic.\n- **Volcano** — \u003Chttps:\u002F\u002Fvolcano.sh\u002F> — gang scheduling, queues, fair-share. Useful in shared multi-tenant clusters.\n- **Kueue** — \u003Chttps:\u002F\u002Fkueue.sigs.k8s.io\u002F> — quota \u002F queue layer on top of any of the above.\n\nThe TAO SDK's Indexed Job path is intentionally simple and dependency-free; if you need elastic restart or gang scheduling, layer one of these on top and submit jobs through the operator's CRD instead.\n\n## Common error patterns\n\n**`No nvidia.com\u002Fgpu resources allocatable on the cluster`** — the GPU Operator (or NVIDIA Device Plugin) isn't installed. Install per the link above; verify with `kubectl get nodes -o jsonpath='{.items[*].status.allocatable}'`.\n\n**`ImagePullBackOff` \u002F `ErrImagePull`** — the cluster can't pull the image. For nvcr.io: pre-create an image-pull secret in the namespace and pass its name via the `image_pull_secret` argument:\n```bash\nkubectl create secret docker-registry ngc-pull-secret \\\n  --docker-server=nvcr.io \\\n  --docker-username='$oauthtoken' \\\n  --docker-password=$NGC_KEY -n tao-jobs\n```\n\n**Pod stays `Pending` forever** — `get_job_replicas(job_id)` will show the readiness_issue. Common causes: insufficient GPU capacity (`Insufficient nvidia.com\u002Fgpu`), no node matches `node_selector`, missing image-pull secret, or PVC mount failure.\n\n**`OOMKilled` (exit 137)** — container exceeded memory. Reduce batch size, lower max_length, or add a memory request\u002Flimit and target a larger node.\n\n**`CredentialError: Could not authenticate to a Kubernetes cluster`** — neither kubeconfig nor in-cluster auth worked. Run `kubectl get nodes` to verify your config, or set `$KUBECONFIG` to the right path.\n\n## What this skill does NOT support (yet)\n\n- **Elastic \u002F fault-tolerant training.** Indexed Job has `backoff_limit=0` — failures fail the whole training run. For elastic restart (e.g., resume from checkpoint after a node death), use Kubeflow's `PyTorchJob` operator instead.\n- **Gang scheduling.** Indexed Job pods are scheduled independently — no all-or-nothing. Multi-node training will *partially* start if only some pods can be scheduled (rank-0 will hang waiting for peers). For all-or-nothing scheduling on shared clusters, use Volcano or Kueue.\n- **MPI \u002F Horovod.** Use the MPI Operator. The Indexed Job path here is PyTorch-distributed-shaped (env-var rendezvous on `MASTER_ADDR:MASTER_PORT`).\n- **Persistent volumes for shared storage.** S3 only via the script_runner. PVC support is a follow-up.\n- **Auto-creating image-pull secrets from `$NGC_KEY`.** You pre-create the secret in the target namespace and pass the name. K8s namespace conventions vary widely, so we keep secret creation explicit.\n",{"name":5,"tags":23,"license":14,"metadata":29,"description":7,"allowed-tools":32,"compatibility":33},[24,25,26,27,28],"kubernetes","k8s","gpu","compute","container",{"author":30,"version":31},"NVIDIA Corporation","0.1.0","Read Bash","Requires GPU worker nodes with NVIDIA driver branch 580, CUDA Toolkit 13.0, and NVIDIA Container Toolkit 1.19.0; the nvidia-tao-sdk Python package with the kubernetes extra (pip install 'nvidia-tao-sdk[kubernetes]'); an authenticated cluster; and the NVIDIA GPU Operator or device plugin.","main","fc4f2c7759ae400c66cd38656db70ead882a0d08","catalog\u002Fbundles\u002Fnvidia\u002Ftao-run-on-kubernetes","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent#readme","https:\u002F\u002Fgithub.com\u002Fmatheussluzz","2026-08-27 05:21:51",{"source":8,"skillName":5,"commands":42,"prompt":52,"zipAvailable":53},[43,46,49],{"tool":44,"command":45},"npx","npx skills add matheussluzz\u002FZeuZ-Agent --skill tao-run-on-kubernetes",{"tool":47,"command":48},"bunx","bunx skills add matheussluzz\u002FZeuZ-Agent --skill tao-run-on-kubernetes",{"tool":50,"command":51},"pnpm","pnpm dlx skills add matheussluzz\u002FZeuZ-Agent --skill tao-run-on-kubernetes","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Ftao-run-on-kubernetes\u002FSKILL.md",true,[55,61,65,68,72,78],{"path":56,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":59,"sourceHtmlUrl":60},"BENCHMARK.md","REFERENCE","参考文档",4100,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Ftao-run-on-kubernetes\u002FBENCHMARK.md",{"path":62,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":63,"sourceHtmlUrl":64},"config.json",1238,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Ftao-run-on-kubernetes\u002Fconfig.json",{"path":66,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":63,"sourceHtmlUrl":67},"references\u002Fskill_info.yaml","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Ftao-run-on-kubernetes\u002Freferences\u002Fskill_info.yaml",{"path":69,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":70,"sourceHtmlUrl":71},"skill-card.md",4124,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Ftao-run-on-kubernetes\u002Fskill-card.md",{"path":73,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":76,"sourceHtmlUrl":77},"skill.oms.sig","OTHER","其他",5053,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Ftao-run-on-kubernetes\u002Fskill.oms.sig",{"path":79,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":80,"sourceHtmlUrl":81},"zeuz.manifest.yaml",172,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Ftao-run-on-kubernetes\u002Fzeuz.manifest.yaml",[83,91,98,105,112,119],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":87,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":18},"01M78JQ2R0QR8H2Z6662RFX804","gamma","Fixture skill gamma for discovery benchmarks.","MIT","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Fskill-registry-mini\u002Fskills\u002Fgamma\u002FSKILL.md",[],[],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":87,"licenseFileUrl":6,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":18},"01M78JQ1KCVZ15APZG3FFF29K0","beta","Fixture skill beta for discovery benchmarks.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Fskill-registry-mini\u002Fskills\u002Fbeta\u002FSKILL.md",[],[],{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":87,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":104,"publishedAt":18},"01M78JQ0FJTK4Q9JKZMX9SVSEN","alpha","Fixture skill alpha for discovery benchmarks.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Fskill-registry-mini\u002Fskills\u002Falpha\u002FSKILL.md",[],[],{"skillNo":106,"name":107,"nameZh":6,"description":108,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":87,"licenseFileUrl":6,"sourceHtmlUrl":109,"categories":110,"keywords":111,"publishedAt":18},"01M78JPYVYPY7R8QS10KADQPWE","writing-for-agents","Writing documents for agents. Use when creating or editing skills, or modifying AGENTS.md or CLAUDE.md.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fskills\u002Fwriting-for-agents\u002FSKILL.md",[],[],{"skillNo":113,"name":114,"nameZh":6,"description":115,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":87,"licenseFileUrl":6,"sourceHtmlUrl":116,"categories":117,"keywords":118,"publishedAt":18},"01M78JPVYHB88RF5ATZVGX84RR","wizard","Generate an interactive bash wizard that walks a human through steps only they can perform. Use when provisioning infrastructure, setting up credentials or CI secrets, walking an unfamiliar third-party dashboard, or running a one-off migration or cutover. Don't invoke this for steps the agent can perform itself.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fskills\u002Fwizard\u002FSKILL.md",[],[],{"skillNo":120,"name":121,"nameZh":6,"description":122,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":87,"licenseFileUrl":6,"sourceHtmlUrl":123,"categories":124,"keywords":125,"publishedAt":18},"01M78JPSSWNTZW89DWGJTX7716","wayfinder","Plan a huge chunk of work (more than one agent session can hold) as a shared map of decision tickets on your issue tracker, and resolve them one at a time until the way to the destination is clear.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fskills\u002Fwayfinder\u002FSKILL.md",[],[]]