AGENT SKILL

dynamo-troubleshoot

Diagnose failed or unhealthy Dynamo deployments. Use when pods, model-cache jobs, PVCs, workers, frontend/router health, endpoints, or benchmark jobs fail; use recipe-runner/router-starter before this for normal bring-up.

License 许可
Apache-2.0
最近更新
2026-11-13
Stars
★ 2
Forks
2

01 前言 Front Matter

name
dynamo-troubleshoot
license
Apache-2.0
version
1.0.0
category
development-code
metadata
{ "tags": [ "dynamo", "kubernetes", "troubleshooting", "day-2" ], "author": "Dan Gil <dagil@nvidia.com>" }
description
Diagnose failed or unhealthy Dynamo deployments. Use when pods, model-cache jobs, PVCs, workers, frontend/router health, endpoints, or benchmark jobs fail; use recipe-runner/router-starter before this for normal bring-up.
related_skills
[ "dynamo-interconnect-check" ]

02 资源文件 6

03 SKILL.md 正文

Dynamo Troubleshoot

<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: CC-BY-4.0 -->

Purpose

Turn a Dynamo failure into a clear problem class, strongest signal, and next action. Start with read-only evidence, avoid secrets, and fix one layer at a time.

Prerequisites

  • Python 3.10+ on the operator machine.
  • kubectl configured with read access to the target namespace.
  • Permission to read pods, events, jobs, PVCs, and DynamoGraphDeployment resources (NOT secrets).
  • Network reachability to the cluster API server.

Instructions

1. Collect A Read-Only Bundle

Run:

python3 scripts/collect_dynamo_debug_bundle.py \
  --namespace "${NAMESPACE}"

If the user names a deployment, include it:

python3 scripts/collect_dynamo_debug_bundle.py \
  --namespace "${NAMESPACE}" \
  --deployment-name <deployment-name>

Do not collect Kubernetes secrets. Do not print Hugging Face tokens.

2. Classify The Failure

Use references/failure-decision-tree.md and classify into one primary bucket:

  • cluster/platform
  • namespace/secret
  • model cache/PVC/download
  • image pull/runtime image
  • GPU scheduling/resources
  • operator/DynamoGraphDeployment reconciliation
  • frontend/router
  • worker/backend
  • endpoint/API
  • benchmark/perf job

3. Debug Top Down

Check in this order:

  1. namespace, storage class, GPU nodes, and HF secret existence
  2. PVC and model-download job
  3. DynamoGraphDeployment status and events
  4. pod status, describe pod, and container logs
  5. frontend service and port-forward
  6. /v1/models
  7. /v1/chat/completions
  8. benchmark job only after endpoint smoke test passes

4. Fix One Layer At A Time

Prefer the smallest reversible change:

  • create missing namespace or HF secret
  • patch storageClassName
  • patch image tag or image pull secret
  • reduce GPU request only if the recipe can still be valid
  • switch KV router to approximate mode only if workers do not publish events
  • restart failed jobs after fixing the underlying config

After each fix, rerun the relevant readiness check before moving deeper.

Available Scripts

Script Purpose Arguments
scripts/collect_dynamo_debug_bundle.py Collect a read-only debug bundle (pods, events, jobs, PVCs, CR status) --namespace, --deployment-name, --output-dir

Invoke via the agentskills.io run_script() protocol:

run_script("scripts/collect_dynamo_debug_bundle.py", args=["--namespace", "dynamo-demo"])

Examples

Collect everything in a namespace for triage:

python3 scripts/collect_dynamo_debug_bundle.py --namespace dynamo-demo

Scope to a single failing deployment:

python3 scripts/collect_dynamo_debug_bundle.py \
  --namespace dynamo-demo \
  --deployment-name qwen-vllm-disagg

Equivalent through the agent protocol:

run_script("scripts/collect_dynamo_debug_bundle.py", args=["--namespace", "dynamo-demo", "--deployment-name", "qwen-vllm-disagg"])

Output Contract

Return:

  • problem class
  • evidence checked
  • strongest signal
  • likely cause
  • exact next command or patch
  • what was ruled out
  • whether it is safe to continue deployment or benchmarking

Limitations

  • Read-only. Never mutates the cluster; remediation commands are returned, not executed.
  • Will not collect secrets or print Hugging Face tokens; some failure modes (auth) may need user-side inspection.
  • Bundle size grows with deployment size; on very large namespaces, scope with --deployment-name.
  • Does not validate disagg transport — use dynamo-interconnect-check for that.

Troubleshooting

Symptom Likely cause Next step
kubectl returns Forbidden on events/pods Service account lacks read RBAC Ask operator for read-only role binding on the namespace
Bundle missing DynamoGraphDeployment status Operator not installed or different namespace Verify dynamo-platform operator is installed and watching the namespace
Model-download job in Pending PVC unbound or HF secret missing Fix PVC binding or create the named HF secret, then rerun the job
Worker pods CrashLoopBackOff Image/runtime mismatch or GPU not available Inspect container logs; check nvidia.com/gpu allocatable on nodes

Benchmark

See BENCHMARK.md for the NVCARPS-EVAL performance report (auto-generated by the NVSkills CI pipeline). To refresh, re-run /nvskills-ci on an upstream PR touching this skill.

References

  • Read references/failure-decision-tree.md for bucket-specific checks.
  • Use scripts/collect_dynamo_debug_bundle.py for read-only bundle collection.

04 同仓库更多 Skills

YYC-Cube
YYC-Cube/YYC3-AI-Agent-Archive

yingmi-skill

当用户需要查询基金、策略、公告、财经资讯,做资产配置、组合诊断、风险回测、现金流分析,或生成图表、PDF 时,优先使用本 Skill 获取真实数据与可执行能力。

★ 2
YYC-Cube
YYC-Cube/YYC3-AI-Agent-Archive

web-access

所有联网操作必须通过此 skill 处理,包括:搜索、网页抓取、登录后操作、网络交互等。

★ 2
YYC-Cube
YYC-Cube/YYC3-AI-Agent-Archive

tencentmap-map-assistant

腾讯位置服务·地图助手 Skill,一句自然语言调用腾讯地图全套能力,无需开发者账号、开箱即用。提供 AI 旅游攻略、地点搜索(含评分/人均/营业时间)、关键词提示、路线规划(驾车/步行/公交/骑行)、地址解析与逆解析、行政区划、IP 定位、距离计算、天气查询,并可将行程或多 POI 渲染成网页地图。涉及找地点、规划路线、旅游行程、查天气、坐标转换等出行场景时使用。

★ 2
YYC-Cube
YYC-Cube/YYC3-AI-Agent-Archive

skill-scanner

Scan any agent skill for security risks before you install or use it. Powered by Tencent Zhuque Lab A.I.G (AI-Infra-Guard). 100% local static analysis — no file contents or credentials leave your device. Compatible with CodeBuddy, Cursor, Windsurf, Claude Code, OpenClaw and more. Triggers on: 这个 skill 安全吗, skill 安全扫描, 检查 skill 安全, audit skill, scan skill, check skill safety, analyze skill, inspect skill, verify skill, skill security, skill supply chain. Do NOT trigger for general agent usage, full system health checks, project debugging, or normal development.

★ 2
YYC-Cube
YYC-Cube/YYC3-AI-Agent-Archive

skill-creator

Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends agent capabilities with specialized knowledge, workflows, or tool integrations.

★ 2
YYC-Cube
YYC-Cube/YYC3-AI-Agent-Archive

shader-dev

Comprehensive GLSL shader techniques for creating stunning visual effects — ray marching, SDF modeling, fluid simulation, particle systems, procedural generation, lighting, post-processing, and more.

★ 2