[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6F72CMSGJ5DGWSVTKPF51TX":3,"skill-same:01M6F72CMSGJ5DGWSVTKPF51TX":79},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":22,"publishedAt":45,"linkOnly":46,"markdownContent":47,"markdownBody":48,"frontMatter":49,"refName":37,"commitSha":59,"skillDirectory":60,"forksCount":61,"repositoryHtmlUrl":62,"repositoryHomepageUrl":6,"ownerHtmlUrl":63,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":64,"install":65,"resources":78},"01M6F72CMSGJ5DGWSVTKPF51TX","gpu-kubernetes-operations",null,"Operate GPU-backed Kubernetes clusters for AI inference and training with scheduling, autoscaling, node health, MIG partitioning, and cost controls.","boisenoise\u002Fskills-collections","boisenoise","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F193180?v=4","01M6CDBDQK6BDN55ZMA6B6PKBM","01M6CDBDVSJ3JPTBYN26FZ7TZK",12,"MIT","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-gpu-kubernetes-operations\u002FSKILL.md",[17],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000011","运维与部署","SCENARIO","使用场景",[23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44],"gpu kubernetes operations","gpu","operations","kubernetes operations","kubernetes","community","2026","autoscaling","operate","relevant","scripts","critical","os","platform","main","only","node","access","devops-security","license","training","security","2026-11-10 10:12:30",false,"---\nname: gpu-kubernetes-operations\ndescription: Operate GPU-backed Kubernetes clusters for AI inference and training\n  with scheduling, autoscaling, node health, MIG partitioning, and cost controls.\ncategory: devops\nrisk: critical\nsource: https:\u002F\u002Fgithub.com\u002FBagelHole\u002FDevOps-Security-Agent-Skills\nsource_repo: BagelHole\u002FDevOps-Security-Agent-Skills\nsource_type: community\ndate_added: '2026-09-20'\nlicense: MIT\nlicense_source: https:\u002F\u002Fgithub.com\u002FBagelHole\u002FDevOps-Security-Agent-Skills\u002Fblob\u002Fmain\u002FLICENSE\ncompatibility: Requires the relevant OS\u002Fplatform tooling and privileged access where\n  noted. Docs-only; helper scripts and templates not bundled.\nmetadata:\n  author: devops-skills\n  version: '1.0'\n---\n\n# GPU Kubernetes Operations\n\nRun resilient and cost-efficient GPU clusters for production AI workloads.\n\n## When to Use This Skill\n\n- Setting up GPU node pools in Kubernetes for AI inference or training\n- Configuring NVIDIA device plugin and GPU operator\n- Implementing MIG partitioning to share GPUs across workloads\n- Building GPU-aware autoscaling policies\n- Monitoring GPU health with DCGM and Prometheus\n- Troubleshooting GPU scheduling, driver, or OOM issues\n\nAll versions, profiles and metrics below are historical examples. Verify the current GPU\u002Fdriver\u002FCUDA\u002FKubernetes\u002Foperator compatibility matrix and exact supported MIG profiles. MIG reconfiguration, drains and driver changes disrupt workloads: require authorized maintenance scope, capacity and tested recovery. Time-slicing shares execution without MIG-style memory\u002Ffault isolation and does not multiply physical capacity.\n\n## Prerequisites\n\n- Kubernetes 1.28+ cluster with GPU-capable nodes\n- NVIDIA GPUs (A10, L4, A100, H100, or similar)\n- NVIDIA drivers installed on nodes (535+ recommended)\n- Helm 3 for operator and plugin installation\n- Prometheus stack for metrics collection\n\n## NVIDIA GPU Operator Installation\n\nThe GPU Operator automates driver, toolkit, device plugin, and DCGM deployment.\n\n```bash\n# Add NVIDIA Helm repo\nhelm repo add nvidia https:\u002F\u002Fhelm.ngc.nvidia.com\u002Fnvidia\nhelm repo update\n\n# Install GPU Operator\nhelm install gpu-operator nvidia\u002Fgpu-operator \\\n  --namespace gpu-operator \\\n  --create-namespace \\\n  --set driver.enabled=true \\\n  --set toolkit.enabled=true \\\n  --set devicePlugin.enabled=true \\\n  --set dcgmExporter.enabled=true \\\n  --set migManager.enabled=true \\\n  --set nodeStatusExporter.enabled=true \\\n  --version v24.3.0\n\n# Verify installation\nkubectl get pods -n gpu-operator\nkubectl get nodes -o json | jq '.items[].status.allocatable[\"nvidia.com\u002Fgpu\"]'\n```\n\n## NVIDIA Device Plugin (Standalone)\n\nIf not using the GPU Operator, deploy the device plugin directly.\n\n```yaml\n# nvidia-device-plugin.yaml\napiVersion: apps\u002Fv1\nkind: DaemonSet\nmetadata:\n  name: nvidia-device-plugin\n  namespace: kube-system\nspec:\n  selector:\n    matchLabels:\n      name: nvidia-device-plugin\n  template:\n    metadata:\n      labels:\n        name: nvidia-device-plugin\n    spec:\n      tolerations:\n        - key: nvidia.com\u002Fgpu\n          operator: Exists\n          effect: NoSchedule\n      priorityClassName: system-node-critical\n      containers:\n        - name: nvidia-device-plugin\n          image: nvcr.io\u002Fnvidia\u002Fk8s-device-plugin:v0.15.0\n          securityContext:\n            privileged: true\n          env:\n            - name: FAIL_ON_INIT_ERROR\n              value: \"false\"\n            - name: DEVICE_SPLIT_COUNT\n              value: \"1\"\n            - name: DEVICE_LIST_STRATEGY\n              value: \"envvar\"\n          volumeMounts:\n            - name: device-plugin\n              mountPath: \u002Fvar\u002Flib\u002Fkubelet\u002Fdevice-plugins\n      volumes:\n        - name: device-plugin\n          hostPath:\n            path: \u002Fvar\u002Flib\u002Fkubelet\u002Fdevice-plugins\n```\n\n## MIG (Multi-Instance GPU) Partitioning\n\nMIG allows a single A100 or H100 to be split into isolated GPU instances.\n\n```yaml\n# mig-config.yaml - ConfigMap for MIG Manager\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: mig-parted-config\n  namespace: gpu-operator\ndata:\n  config.yaml: |\n    version: v1\n    mig-configs:\n      # 7 small instances for inference microservices\n      all-1g.10gb:\n        - devices: all\n          mig-enabled: true\n          mig-devices:\n            \"1g.10gb\": 7\n\n      # 3 medium instances for mid-size models\n      all-2g.20gb:\n        - devices: all\n          mig-enabled: true\n          mig-devices:\n            \"2g.20gb\": 3\n\n      # Mixed: 1 large + 2 small\n      mixed-inference:\n        - devices: all\n          mig-enabled: true\n          mig-devices:\n            \"3g.40gb\": 1\n            \"1g.10gb\": 4\n\n      # Full GPU for training (no partitioning)\n      all-disabled:\n        - devices: all\n          mig-enabled: false\n```\n\n```bash\n# Apply MIG profile to a node\nkubectl label nodes gpu-node-01 nvidia.com\u002Fmig.config=all-1g.10gb --overwrite\n\n# Verify MIG instances\nkubectl exec -it nvidia-device-plugin-xxxxx -n kube-system -- nvidia-smi mig -lgi\n\n# Check available MIG resources\nkubectl get nodes gpu-node-01 -o json | jq '.status.allocatable | with_entries(select(.key | startswith(\"nvidia.com\")))'\n```\n\n### Requesting MIG Slices in Pods\n\n```yaml\n# pod-with-mig.yaml\napiVersion: v1\nkind: Pod\nmetadata:\n  name: inference-small\nspec:\n  containers:\n    - name: model\n      image: registry.internal\u002Fvllm-server:latest\n      resources:\n        limits:\n          nvidia.com\u002Fmig-1g.10gb: 1\n      # For medium slice:\n      # nvidia.com\u002Fmig-2g.20gb: 1\n      # For large slice:\n      # nvidia.com\u002Fmig-3g.40gb: 1\n```\n\n## GPU Time-Slicing\n\nFor GPUs that do not support MIG (A10, L4), use time-slicing to share a GPU.\n\n```yaml\n# time-slicing-config.yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: time-slicing-config\n  namespace: gpu-operator\ndata:\n  any: |-\n    version: v1\n    flags:\n      migStrategy: none\n    sharing:\n      timeSlicing:\n        renameByDefault: false\n        failRequestsGreaterThanOne: false\n        resources:\n          - name: nvidia.com\u002Fgpu\n            replicas: 4\n```\n\n```bash\n# Apply time-slicing config\nkubectl patch clusterpolicy\u002Fcluster-policy \\\n  --type merge \\\n  -p '{\"spec\":{\"devicePlugin\":{\"config\":{\"name\":\"time-slicing-config\",\"default\":\"any\"}}}}'\n\n# After applying, each physical GPU appears as 4 virtual GPUs\nkubectl get nodes -o json | jq '.items[].status.allocatable[\"nvidia.com\u002Fgpu\"]'\n# Output: \"4\" per physical GPU\n```\n\n## DCGM Monitoring\n\n```yaml\n# dcgm-servicemonitor.yaml\napiVersion: monitoring.coreos.com\u002Fv1\nkind: ServiceMonitor\nmetadata:\n  name: dcgm-exporter\n  namespace: gpu-operator\n  labels:\n    release: prometheus\nspec:\n  selector:\n    matchLabels:\n      app: nvidia-dcgm-exporter\n  endpoints:\n    - port: gpu-metrics\n      interval: 15s\n      path: \u002Fmetrics\n```\n\n### Key DCGM Metrics and Alert Rules\n\n```yaml\n# gpu-alerts.yaml\ngroups:\n  - name: gpu-health\n    rules:\n      - alert: GPUHighTemperature\n        expr: DCGM_FI_DEV_GPU_TEMP > 85\n        for: 5m\n        labels:\n          severity: warning\n        annotations:\n          summary: \"GPU {{ $labels.gpu }} temperature above 85C on {{ $labels.node }}\"\n\n      - alert: GPUMemoryPressure\n        expr: (DCGM_FI_DEV_FB_USED \u002F (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)) > 0.90\n        for: 5m\n        labels:\n          severity: warning\n        annotations:\n          summary: \"GPU memory above 90% on {{ $labels.node }} GPU {{ $labels.gpu }}\"\n\n      - alert: GPUECCErrors\n        expr: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[1h]) > 0\n        labels:\n          severity: critical\n        annotations:\n          summary: \"Double-bit ECC errors detected on {{ $labels.node }} GPU {{ $labels.gpu }}\"\n\n      - alert: GPUXidErrors\n        expr: DCGM_FI_DEV_XID_ERRORS != 0\n        labels:\n          severity: warning\n        annotations:\n          summary: \"Xid error on {{ $labels.node }} GPU {{ $labels.gpu }}: {{ $labels.xid }}\"\n\n      - alert: GPULowUtilization\n        expr: DCGM_FI_DEV_GPU_UTIL \u003C 10 and on(pod) kube_pod_status_phase{phase=\"Running\"} == 1\n        for: 30m\n        labels:\n          severity: info\n        annotations:\n          summary: \"GPU underutilized on {{ $labels.node }} - consider rightsizing\"\n\n      - alert: GPUDriverMismatch\n        expr: count(count by (driver_version)(DCGM_FI_DRIVER_VERSION)) > 1\n        labels:\n          severity: warning\n        annotations:\n          summary: \"Multiple GPU driver versions detected across cluster\"\n```\n\n## GPU Node Pool Configuration\n\n```yaml\n# gpu-nodepool.yaml\napiVersion: v1\nkind: Node\nmetadata:\n  labels:\n    gpu-type: a100\n    gpu-memory: \"80gb\"\n    gpu-mig-capable: \"true\"\n    node-role: gpu-inference\nspec:\n  taints:\n    - key: nvidia.com\u002Fgpu\n      value: \"true\"\n      effect: NoSchedule\n---\n# Inference deployment with GPU scheduling\napiVersion: apps\u002Fv1\nkind: Deployment\nmetadata:\n  name: llm-inference\n  namespace: ai-serving\nspec:\n  replicas: 3\n  selector:\n    matchLabels:\n      app: llm-inference\n  template:\n    metadata:\n      labels:\n        app: llm-inference\n    spec:\n      tolerations:\n        - key: nvidia.com\u002Fgpu\n          operator: Exists\n          effect: NoSchedule\n      nodeSelector:\n        gpu-type: a100\n      affinity:\n        podAntiAffinity:\n          preferredDuringSchedulingIgnoredDuringExecution:\n            - weight: 100\n              podAffinityTerm:\n                labelSelector:\n                  matchLabels:\n                    app: llm-inference\n                topologyKey: kubernetes.io\u002Fhostname\n      containers:\n        - name: vllm\n          image: registry.internal\u002Fvllm-server:0.4.1\n          resources:\n            requests:\n              nvidia.com\u002Fgpu: 1\n              cpu: \"4\"\n              memory: \"32Gi\"\n            limits:\n              nvidia.com\u002Fgpu: 1\n              cpu: \"8\"\n              memory: \"64Gi\"\n          # Let the device plugin\u002Fruntime inject assigned GPU visibility.\n```\n\n## GPU Autoscaling\n\nThe HPA requires an installed custom-metrics adapter and correctly mapped per-pod metrics. The autoscaler ConfigMap is conceptual input, not a universally consumed API; wire supported flags\u002Fsettings through the installed distribution. Node snippets illustrate labels\u002Ftaints on existing named nodes, not provisionable GPU capacity. Validate every metric name\u002Flabel\u002Ftype against actual exporter output; do not infer counters from names.\n\n```yaml\n# gpu-hpa.yaml\napiVersion: autoscaling\u002Fv2\nkind: HorizontalPodAutoscaler\nmetadata:\n  name: llm-inference-hpa\n  namespace: ai-serving\nspec:\n  scaleTargetRef:\n    apiVersion: apps\u002Fv1\n    kind: Deployment\n    name: llm-inference\n  minReplicas: 2\n  maxReplicas: 8\n  metrics:\n    - type: Pods\n      pods:\n        metric:\n          name: DCGM_FI_DEV_GPU_UTIL\n        target:\n          type: AverageValue\n          averageValue: \"75\"\n    - type: Pods\n      pods:\n        metric:\n          name: inference_queue_depth\n        target:\n          type: AverageValue\n          averageValue: \"10\"\n  behavior:\n    scaleUp:\n      stabilizationWindowSeconds: 60\n      policies:\n        - type: Pods\n          value: 2\n          periodSeconds: 120\n    scaleDown:\n      stabilizationWindowSeconds: 300\n      policies:\n        - type: Pods\n          value: 1\n          periodSeconds: 300\n---\n# Cluster Autoscaler config for GPU node pools\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: cluster-autoscaler-config\n  namespace: kube-system\ndata:\n  config: |\n    expander: priority\n    scale-down-delay-after-add: 10m\n    scale-down-unneeded-time: 10m\n    skip-nodes-with-local-storage: false\n    balance-similar-node-groups: true\n    expendable-pods-priority-cutoff: -10\n    gpu-total:\n      - min: 2\n        max: 16\n        gpu: nvidia.com\u002Fgpu\n```\n\n## Scheduling Patterns\n\n- Use node affinity by GPU type (A10\u002FL4\u002FA100\u002FH100).\n- Separate latency-critical inference from batch training.\n- Pin model replicas with anti-affinity for availability.\n- Reserve headroom for failover and rolling updates.\n\n## Cost Optimization\n\n- Prefer MIG slices for smaller inference services.\n- Schedule batch jobs in off-peak windows.\n- Route low-priority traffic to cheaper model tiers.\n- Use spot\u002Fpreemptible instances for training workloads.\n- Monitor GPU utilization and rightsize deployments.\n\n## Troubleshooting\n\n| Symptom | Check | Fix |\n|---------|-------|-----|\n| Pod stuck in Pending | `kubectl describe pod` for GPU resource events | Verify node has allocatable GPUs, check taints\u002Ftolerations |\n| CUDA OOM during inference | Model too large for GPU memory | Reduce batch size\u002Fmodel footprint or request a larger supported memory allocation; a smaller MIG slice worsens memory shortage |\n| DCGM metrics missing | ServiceMonitor labels matching | Verify DCGM exporter pod is running and scrape config |\n| Driver mismatch after upgrade | `nvidia-smi` on each node | Cordon node, drain, upgrade driver, uncordon |\n| GPU not detected | Device plugin pod logs | Restart device plugin, check NVIDIA container toolkit |\n| Time-slicing not working | ConfigMap applied but no extra GPUs | Restart device plugin pods after config change |\n| ECC errors increasing | `nvidia-smi -q -d ECC` | Schedule node drain and hardware replacement |\n\n## Related Skills\n\n- llm-inference-scaling (`llm-inference-scaling`) - Autoscale inference workloads\n- model-serving-kubernetes (`model-serving-kubernetes`) - Production model serving patterns\n- gpu-server-management (`gpu-server-management`) - Host-level GPU management fundamentals\n- multi-tenant-llm-hosting (`multi-tenant-llm-hosting`) - Multi-tenant GPU sharing\n- llm-cost-optimization (`llm-cost-optimization`) - Cost optimization strategies\n\n## Limitations\n\n- Infrastructure commands can disrupt services: confirm target host\u002Fscope and have backups\u002Fsnapshots before mutating state.\n- Docs-only import: upstream scripts and templates not bundled.\n\n","# GPU Kubernetes Operations\n\nRun resilient and cost-efficient GPU clusters for production AI workloads.\n\n## When to Use This Skill\n\n- Setting up GPU node pools in Kubernetes for AI inference or training\n- Configuring NVIDIA device plugin and GPU operator\n- Implementing MIG partitioning to share GPUs across workloads\n- Building GPU-aware autoscaling policies\n- Monitoring GPU health with DCGM and Prometheus\n- Troubleshooting GPU scheduling, driver, or OOM issues\n\nAll versions, profiles and metrics below are historical examples. Verify the current GPU\u002Fdriver\u002FCUDA\u002FKubernetes\u002Foperator compatibility matrix and exact supported MIG profiles. MIG reconfiguration, drains and driver changes disrupt workloads: require authorized maintenance scope, capacity and tested recovery. Time-slicing shares execution without MIG-style memory\u002Ffault isolation and does not multiply physical capacity.\n\n## Prerequisites\n\n- Kubernetes 1.28+ cluster with GPU-capable nodes\n- NVIDIA GPUs (A10, L4, A100, H100, or similar)\n- NVIDIA drivers installed on nodes (535+ recommended)\n- Helm 3 for operator and plugin installation\n- Prometheus stack for metrics collection\n\n## NVIDIA GPU Operator Installation\n\nThe GPU Operator automates driver, toolkit, device plugin, and DCGM deployment.\n\n```bash\n# Add NVIDIA Helm repo\nhelm repo add nvidia https:\u002F\u002Fhelm.ngc.nvidia.com\u002Fnvidia\nhelm repo update\n\n# Install GPU Operator\nhelm install gpu-operator nvidia\u002Fgpu-operator \\\n  --namespace gpu-operator \\\n  --create-namespace \\\n  --set driver.enabled=true \\\n  --set toolkit.enabled=true \\\n  --set devicePlugin.enabled=true \\\n  --set dcgmExporter.enabled=true \\\n  --set migManager.enabled=true \\\n  --set nodeStatusExporter.enabled=true \\\n  --version v24.3.0\n\n# Verify installation\nkubectl get pods -n gpu-operator\nkubectl get nodes -o json | jq '.items[].status.allocatable[\"nvidia.com\u002Fgpu\"]'\n```\n\n## NVIDIA Device Plugin (Standalone)\n\nIf not using the GPU Operator, deploy the device plugin directly.\n\n```yaml\n# nvidia-device-plugin.yaml\napiVersion: apps\u002Fv1\nkind: DaemonSet\nmetadata:\n  name: nvidia-device-plugin\n  namespace: kube-system\nspec:\n  selector:\n    matchLabels:\n      name: nvidia-device-plugin\n  template:\n    metadata:\n      labels:\n        name: nvidia-device-plugin\n    spec:\n      tolerations:\n        - key: nvidia.com\u002Fgpu\n          operator: Exists\n          effect: NoSchedule\n      priorityClassName: system-node-critical\n      containers:\n        - name: nvidia-device-plugin\n          image: nvcr.io\u002Fnvidia\u002Fk8s-device-plugin:v0.15.0\n          securityContext:\n            privileged: true\n          env:\n            - name: FAIL_ON_INIT_ERROR\n              value: \"false\"\n            - name: DEVICE_SPLIT_COUNT\n              value: \"1\"\n            - name: DEVICE_LIST_STRATEGY\n              value: \"envvar\"\n          volumeMounts:\n            - name: device-plugin\n              mountPath: \u002Fvar\u002Flib\u002Fkubelet\u002Fdevice-plugins\n      volumes:\n        - name: device-plugin\n          hostPath:\n            path: \u002Fvar\u002Flib\u002Fkubelet\u002Fdevice-plugins\n```\n\n## MIG (Multi-Instance GPU) Partitioning\n\nMIG allows a single A100 or H100 to be split into isolated GPU instances.\n\n```yaml\n# mig-config.yaml - ConfigMap for MIG Manager\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: mig-parted-config\n  namespace: gpu-operator\ndata:\n  config.yaml: |\n    version: v1\n    mig-configs:\n      # 7 small instances for inference microservices\n      all-1g.10gb:\n        - devices: all\n          mig-enabled: true\n          mig-devices:\n            \"1g.10gb\": 7\n\n      # 3 medium instances for mid-size models\n      all-2g.20gb:\n        - devices: all\n          mig-enabled: true\n          mig-devices:\n            \"2g.20gb\": 3\n\n      # Mixed: 1 large + 2 small\n      mixed-inference:\n        - devices: all\n          mig-enabled: true\n          mig-devices:\n            \"3g.40gb\": 1\n            \"1g.10gb\": 4\n\n      # Full GPU for training (no partitioning)\n      all-disabled:\n        - devices: all\n          mig-enabled: false\n```\n\n```bash\n# Apply MIG profile to a node\nkubectl label nodes gpu-node-01 nvidia.com\u002Fmig.config=all-1g.10gb --overwrite\n\n# Verify MIG instances\nkubectl exec -it nvidia-device-plugin-xxxxx -n kube-system -- nvidia-smi mig -lgi\n\n# Check available MIG resources\nkubectl get nodes gpu-node-01 -o json | jq '.status.allocatable | with_entries(select(.key | startswith(\"nvidia.com\")))'\n```\n\n### Requesting MIG Slices in Pods\n\n```yaml\n# pod-with-mig.yaml\napiVersion: v1\nkind: Pod\nmetadata:\n  name: inference-small\nspec:\n  containers:\n    - name: model\n      image: registry.internal\u002Fvllm-server:latest\n      resources:\n        limits:\n          nvidia.com\u002Fmig-1g.10gb: 1\n      # For medium slice:\n      # nvidia.com\u002Fmig-2g.20gb: 1\n      # For large slice:\n      # nvidia.com\u002Fmig-3g.40gb: 1\n```\n\n## GPU Time-Slicing\n\nFor GPUs that do not support MIG (A10, L4), use time-slicing to share a GPU.\n\n```yaml\n# time-slicing-config.yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: time-slicing-config\n  namespace: gpu-operator\ndata:\n  any: |-\n    version: v1\n    flags:\n      migStrategy: none\n    sharing:\n      timeSlicing:\n        renameByDefault: false\n        failRequestsGreaterThanOne: false\n        resources:\n          - name: nvidia.com\u002Fgpu\n            replicas: 4\n```\n\n```bash\n# Apply time-slicing config\nkubectl patch clusterpolicy\u002Fcluster-policy \\\n  --type merge \\\n  -p '{\"spec\":{\"devicePlugin\":{\"config\":{\"name\":\"time-slicing-config\",\"default\":\"any\"}}}}'\n\n# After applying, each physical GPU appears as 4 virtual GPUs\nkubectl get nodes -o json | jq '.items[].status.allocatable[\"nvidia.com\u002Fgpu\"]'\n# Output: \"4\" per physical GPU\n```\n\n## DCGM Monitoring\n\n```yaml\n# dcgm-servicemonitor.yaml\napiVersion: monitoring.coreos.com\u002Fv1\nkind: ServiceMonitor\nmetadata:\n  name: dcgm-exporter\n  namespace: gpu-operator\n  labels:\n    release: prometheus\nspec:\n  selector:\n    matchLabels:\n      app: nvidia-dcgm-exporter\n  endpoints:\n    - port: gpu-metrics\n      interval: 15s\n      path: \u002Fmetrics\n```\n\n### Key DCGM Metrics and Alert Rules\n\n```yaml\n# gpu-alerts.yaml\ngroups:\n  - name: gpu-health\n    rules:\n      - alert: GPUHighTemperature\n        expr: DCGM_FI_DEV_GPU_TEMP > 85\n        for: 5m\n        labels:\n          severity: warning\n        annotations:\n          summary: \"GPU {{ $labels.gpu }} temperature above 85C on {{ $labels.node }}\"\n\n      - alert: GPUMemoryPressure\n        expr: (DCGM_FI_DEV_FB_USED \u002F (DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE)) > 0.90\n        for: 5m\n        labels:\n          severity: warning\n        annotations:\n          summary: \"GPU memory above 90% on {{ $labels.node }} GPU {{ $labels.gpu }}\"\n\n      - alert: GPUECCErrors\n        expr: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[1h]) > 0\n        labels:\n          severity: critical\n        annotations:\n          summary: \"Double-bit ECC errors detected on {{ $labels.node }} GPU {{ $labels.gpu }}\"\n\n      - alert: GPUXidErrors\n        expr: DCGM_FI_DEV_XID_ERRORS != 0\n        labels:\n          severity: warning\n        annotations:\n          summary: \"Xid error on {{ $labels.node }} GPU {{ $labels.gpu }}: {{ $labels.xid }}\"\n\n      - alert: GPULowUtilization\n        expr: DCGM_FI_DEV_GPU_UTIL \u003C 10 and on(pod) kube_pod_status_phase{phase=\"Running\"} == 1\n        for: 30m\n        labels:\n          severity: info\n        annotations:\n          summary: \"GPU underutilized on {{ $labels.node }} - consider rightsizing\"\n\n      - alert: GPUDriverMismatch\n        expr: count(count by (driver_version)(DCGM_FI_DRIVER_VERSION)) > 1\n        labels:\n          severity: warning\n        annotations:\n          summary: \"Multiple GPU driver versions detected across cluster\"\n```\n\n## GPU Node Pool Configuration\n\n```yaml\n# gpu-nodepool.yaml\napiVersion: v1\nkind: Node\nmetadata:\n  labels:\n    gpu-type: a100\n    gpu-memory: \"80gb\"\n    gpu-mig-capable: \"true\"\n    node-role: gpu-inference\nspec:\n  taints:\n    - key: nvidia.com\u002Fgpu\n      value: \"true\"\n      effect: NoSchedule\n---\n# Inference deployment with GPU scheduling\napiVersion: apps\u002Fv1\nkind: Deployment\nmetadata:\n  name: llm-inference\n  namespace: ai-serving\nspec:\n  replicas: 3\n  selector:\n    matchLabels:\n      app: llm-inference\n  template:\n    metadata:\n      labels:\n        app: llm-inference\n    spec:\n      tolerations:\n        - key: nvidia.com\u002Fgpu\n          operator: Exists\n          effect: NoSchedule\n      nodeSelector:\n        gpu-type: a100\n      affinity:\n        podAntiAffinity:\n          preferredDuringSchedulingIgnoredDuringExecution:\n            - weight: 100\n              podAffinityTerm:\n                labelSelector:\n                  matchLabels:\n                    app: llm-inference\n                topologyKey: kubernetes.io\u002Fhostname\n      containers:\n        - name: vllm\n          image: registry.internal\u002Fvllm-server:0.4.1\n          resources:\n            requests:\n              nvidia.com\u002Fgpu: 1\n              cpu: \"4\"\n              memory: \"32Gi\"\n            limits:\n              nvidia.com\u002Fgpu: 1\n              cpu: \"8\"\n              memory: \"64Gi\"\n          # Let the device plugin\u002Fruntime inject assigned GPU visibility.\n```\n\n## GPU Autoscaling\n\nThe HPA requires an installed custom-metrics adapter and correctly mapped per-pod metrics. The autoscaler ConfigMap is conceptual input, not a universally consumed API; wire supported flags\u002Fsettings through the installed distribution. Node snippets illustrate labels\u002Ftaints on existing named nodes, not provisionable GPU capacity. Validate every metric name\u002Flabel\u002Ftype against actual exporter output; do not infer counters from names.\n\n```yaml\n# gpu-hpa.yaml\napiVersion: autoscaling\u002Fv2\nkind: HorizontalPodAutoscaler\nmetadata:\n  name: llm-inference-hpa\n  namespace: ai-serving\nspec:\n  scaleTargetRef:\n    apiVersion: apps\u002Fv1\n    kind: Deployment\n    name: llm-inference\n  minReplicas: 2\n  maxReplicas: 8\n  metrics:\n    - type: Pods\n      pods:\n        metric:\n          name: DCGM_FI_DEV_GPU_UTIL\n        target:\n          type: AverageValue\n          averageValue: \"75\"\n    - type: Pods\n      pods:\n        metric:\n          name: inference_queue_depth\n        target:\n          type: AverageValue\n          averageValue: \"10\"\n  behavior:\n    scaleUp:\n      stabilizationWindowSeconds: 60\n      policies:\n        - type: Pods\n          value: 2\n          periodSeconds: 120\n    scaleDown:\n      stabilizationWindowSeconds: 300\n      policies:\n        - type: Pods\n          value: 1\n          periodSeconds: 300\n---\n# Cluster Autoscaler config for GPU node pools\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: cluster-autoscaler-config\n  namespace: kube-system\ndata:\n  config: |\n    expander: priority\n    scale-down-delay-after-add: 10m\n    scale-down-unneeded-time: 10m\n    skip-nodes-with-local-storage: false\n    balance-similar-node-groups: true\n    expendable-pods-priority-cutoff: -10\n    gpu-total:\n      - min: 2\n        max: 16\n        gpu: nvidia.com\u002Fgpu\n```\n\n## Scheduling Patterns\n\n- Use node affinity by GPU type (A10\u002FL4\u002FA100\u002FH100).\n- Separate latency-critical inference from batch training.\n- Pin model replicas with anti-affinity for availability.\n- Reserve headroom for failover and rolling updates.\n\n## Cost Optimization\n\n- Prefer MIG slices for smaller inference services.\n- Schedule batch jobs in off-peak windows.\n- Route low-priority traffic to cheaper model tiers.\n- Use spot\u002Fpreemptible instances for training workloads.\n- Monitor GPU utilization and rightsize deployments.\n\n## Troubleshooting\n\n| Symptom | Check | Fix |\n|---------|-------|-----|\n| Pod stuck in Pending | `kubectl describe pod` for GPU resource events | Verify node has allocatable GPUs, check taints\u002Ftolerations |\n| CUDA OOM during inference | Model too large for GPU memory | Reduce batch size\u002Fmodel footprint or request a larger supported memory allocation; a smaller MIG slice worsens memory shortage |\n| DCGM metrics missing | ServiceMonitor labels matching | Verify DCGM exporter pod is running and scrape config |\n| Driver mismatch after upgrade | `nvidia-smi` on each node | Cordon node, drain, upgrade driver, uncordon |\n| GPU not detected | Device plugin pod logs | Restart device plugin, check NVIDIA container toolkit |\n| Time-slicing not working | ConfigMap applied but no extra GPUs | Restart device plugin pods after config change |\n| ECC errors increasing | `nvidia-smi -q -d ECC` | Schedule node drain and hardware replacement |\n\n## Related Skills\n\n- llm-inference-scaling (`llm-inference-scaling`) - Autoscale inference workloads\n- model-serving-kubernetes (`model-serving-kubernetes`) - Production model serving patterns\n- gpu-server-management (`gpu-server-management`) - Host-level GPU management fundamentals\n- multi-tenant-llm-hosting (`multi-tenant-llm-hosting`) - Multi-tenant GPU sharing\n- llm-cost-optimization (`llm-cost-optimization`) - Cost optimization strategies\n\n## Limitations\n\n- Infrastructure commands can disrupt services: confirm target host\u002Fscope and have backups\u002Fsnapshots before mutating state.\n- Docs-only import: upstream scripts and templates not bundled.\n\n",{"name":5,"risk":34,"source":50,"license":14,"category":51,"metadata":52,"date_added":55,"description":7,"source_repo":56,"source_type":28,"compatibility":57,"license_source":58},"https:\u002F\u002Fgithub.com\u002FBagelHole\u002FDevOps-Security-Agent-Skills","devops",{"author":53,"version":54},"devops-skills","1.0","2026-09-20","BagelHole\u002FDevOps-Security-Agent-Skills","Requires the relevant OS\u002Fplatform tooling and privileged access where noted. Docs-only; helper scripts and templates not bundled.","https:\u002F\u002Fgithub.com\u002FBagelHole\u002FDevOps-Security-Agent-Skills\u002Fblob\u002Fmain\u002FLICENSE","7bf2279c8a65758319526a721d5c4b75683aec9c","skills\u002Fantigravity-gpu-kubernetes-operations",3,"https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections","https:\u002F\u002Fgithub.com\u002Fboisenoise","2026-11-11 14:20:43",{"source":8,"skillName":5,"commands":66,"prompt":76,"zipAvailable":77},[67,70,73],{"tool":68,"command":69},"npx","npx skills add boisenoise\u002Fskills-collections --skill gpu-kubernetes-operations",{"tool":71,"command":72},"bunx","bunx skills add boisenoise\u002Fskills-collections --skill gpu-kubernetes-operations",{"tool":74,"command":75},"pnpm","pnpm dlx skills add boisenoise\u002Fskills-collections --skill gpu-kubernetes-operations","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-gpu-kubernetes-operations\u002FSKILL.md",true,[],[80,105,130,156,175,198],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":84,"licenseFileUrl":85,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":45},"01M6WNH92BHVFXHGPDZCPJ0AG0","web-access","Use Treg for public web or platform data only when existing search, browser and connected tools cannot cover the task, with explicit cost and data boundaries.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-web-access\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-web-access\u002FSKILL.md",[],[89,40,90,91,92,93,94,95,96,97,98,38,99,100,36,34,101,102,103,104,29],"web","web access","official","development","browser","search","data","Tools","read","fetch","youtube","free tools","free","cover","normal","when",{"skillNo":106,"name":107,"nameZh":6,"description":108,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":84,"licenseFileUrl":109,"sourceHtmlUrl":110,"categories":111,"keywords":115,"publishedAt":45},"01M6WNF9XAYAB2CZDDJBAKNM93","treg","Use the Treg CLI or MCP catalog to discover, price and call authorized external tools with team-scoped credentials and a user-approved budget.","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-treg\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-treg\u002FSKILL.md",[112],{"categoryNo":113,"categoryName":114,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000023","Skill与智能体开发",[107,116,117,118,91,92,42,96,119,120,121,122,123,124,125,34,126,127,128,129,29],"MCP","cli","github","Budget","credentials","team","catalog","use","call","discover","external tools","with","scoped","blob",{"skillNo":131,"name":132,"nameZh":6,"description":133,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":84,"licenseFileUrl":134,"sourceHtmlUrl":135,"categories":136,"keywords":142,"publishedAt":45},"01M6WN93MEAZ4E41DYAHJRPHQT","recruit","Turn a job description into an evidence-linked candidate shortlist through Treg, with job-related ranking and optional authorized business contact enrichment.","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-recruit\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-recruit\u002FSKILL.md",[137],{"categoryNo":138,"categoryName":139,"taxonomyType":140,"taxonomyTypeDesc":141},"01JBSEEDCAT000000000000038","人力资源","PROFESSION","适用职业",[132,118,91,42,143,144,145,146,147,148,34,149,150,151,127,152,153,154,155,129,107,29],"https","email","enrichment","business","evidence","phone","ranking","post","job description","for","person","candidate","reason",{"skillNo":157,"name":158,"nameZh":6,"description":159,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":84,"licenseFileUrl":160,"sourceHtmlUrl":161,"categories":162,"keywords":163,"publishedAt":45},"01M6WN8R5RXSMHM9B4D9FAFX42","rea-growth","Analyze a company’s public growth and content through Treg, distinguish observations from hypotheses, and produce an evidence-linked playbook within an approved budget.","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-rea-growth\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-rea-growth\u002FSKILL.md",[],[164,165,166,118,91,42,167,168,143,169,119,170,171,172,147,34,173,174,129,107,29],"growth","rea","rea growth","produce","content","playbook","marketing","analyze","public","observations","company",{"skillNo":176,"name":177,"nameZh":6,"description":178,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":84,"licenseFileUrl":179,"sourceHtmlUrl":180,"categories":181,"keywords":182,"publishedAt":45},"01M6WN2EVE3RQRGWAKBT4S7YT2","make-ugc","Create original AI UGC-style product videos through Treg: research hooks, approve a licensed presenter and voice, generate clips, add captions and report costs.","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-make-ugc\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-make-ugc\u002FSKILL.md",[],[183,184,118,185,91,186,42,187,188,189,190,191,34,192,193,194,195,196,197,129],"make","make ugc","research","create","Video","Captions","ads","short","hooks","head","style","niche","trending","talking-head","approve",{"skillNo":199,"name":200,"nameZh":6,"description":201,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":84,"licenseFileUrl":202,"sourceHtmlUrl":203,"categories":204,"keywords":205,"publishedAt":45},"01M6WN10ARAA63Y8K1YNGXJ2RM","lead-signals","Find and qualify public buying signals through Treg, rank fitting business accounts and optionally enrich authorized business contact information.","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-lead-signals\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fboisenoise\u002Fskills-collections\u002Fblob\u002Fmain\u002Fskills\u002Fantigravity-lead-signals\u002FSKILL.md",[],[206,207,208,118,91,42,209,210,146,211,212,34,213,214,215,216,217,218,219,129],"signals","lead","lead signals","how","find","ready","as","how to","qualify","people","information","enrich","changes","funding"]