dynamo-troubleshoot
Diagnose failed or unhealthy Dynamo deployments. Use when pods, model-cache jobs, PVCs, workers, frontend/router health, endpoints, or benchmark jobs fail; use recipe-runner/router-starter before this for normal bring-up.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-13
- Stars
- ★ 2
- Forks
- 2
01 前言 Front Matter
- name
- dynamo-troubleshoot
- license
- Apache-2.0
- version
- 1.0.0
- category
- development-code
- metadata
- { "tags": [ "dynamo", "kubernetes", "troubleshooting", "day-2" ], "author": "Dan Gil <dagil@nvidia.com>" }
- description
- Diagnose failed or unhealthy Dynamo deployments. Use when pods, model-cache jobs, PVCs, workers, frontend/router health, endpoints, or benchmark jobs fail; use recipe-runner/router-starter before this for normal bring-up.
- related_skills
- [ "dynamo-interconnect-check" ]
02 资源文件 6
- 参考文档BENCHMARK.md3.0 KB
- 参考文档evals/evals.json3.2 KB
- 参考文档references/failure-decision-tree.md3.7 KB
- 脚本scripts/collect_dynamo_debug_bundle.py8.3 KB
- 参考文档skill-card.md2.8 KB
- 其他skill.oms.sig5.0 KB
03 SKILL.md 正文
Dynamo Troubleshoot
<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: CC-BY-4.0 -->
Purpose
Turn a Dynamo failure into a clear problem class, strongest signal, and next action. Start with read-only evidence, avoid secrets, and fix one layer at a time.
Prerequisites
- Python 3.10+ on the operator machine.
kubectlconfigured with read access to the target namespace.- Permission to read pods, events, jobs, PVCs, and
DynamoGraphDeploymentresources (NOT secrets). - Network reachability to the cluster API server.
Instructions
1. Collect A Read-Only Bundle
Run:
python3 scripts/collect_dynamo_debug_bundle.py \
--namespace "${NAMESPACE}"
If the user names a deployment, include it:
python3 scripts/collect_dynamo_debug_bundle.py \
--namespace "${NAMESPACE}" \
--deployment-name <deployment-name>
Do not collect Kubernetes secrets. Do not print Hugging Face tokens.
2. Classify The Failure
Use references/failure-decision-tree.md and classify into one primary bucket:
- cluster/platform
- namespace/secret
- model cache/PVC/download
- image pull/runtime image
- GPU scheduling/resources
- operator/DynamoGraphDeployment reconciliation
- frontend/router
- worker/backend
- endpoint/API
- benchmark/perf job
3. Debug Top Down
Check in this order:
- namespace, storage class, GPU nodes, and HF secret existence
- PVC and model-download job
DynamoGraphDeploymentstatus and events- pod status,
describe pod, and container logs - frontend service and port-forward
/v1/models/v1/chat/completions- benchmark job only after endpoint smoke test passes
4. Fix One Layer At A Time
Prefer the smallest reversible change:
- create missing namespace or HF secret
- patch
storageClassName - patch image tag or image pull secret
- reduce GPU request only if the recipe can still be valid
- switch KV router to approximate mode only if workers do not publish events
- restart failed jobs after fixing the underlying config
After each fix, rerun the relevant readiness check before moving deeper.
Available Scripts
| Script | Purpose | Arguments |
|---|---|---|
scripts/collect_dynamo_debug_bundle.py |
Collect a read-only debug bundle (pods, events, jobs, PVCs, CR status) | --namespace, --deployment-name, --output-dir |
Invoke via the agentskills.io run_script() protocol:
run_script("scripts/collect_dynamo_debug_bundle.py", args=["--namespace", "dynamo-demo"])
Examples
Collect everything in a namespace for triage:
python3 scripts/collect_dynamo_debug_bundle.py --namespace dynamo-demo
Scope to a single failing deployment:
python3 scripts/collect_dynamo_debug_bundle.py \
--namespace dynamo-demo \
--deployment-name qwen-vllm-disagg
Equivalent through the agent protocol:
run_script("scripts/collect_dynamo_debug_bundle.py", args=["--namespace", "dynamo-demo", "--deployment-name", "qwen-vllm-disagg"])
Output Contract
Return:
- problem class
- evidence checked
- strongest signal
- likely cause
- exact next command or patch
- what was ruled out
- whether it is safe to continue deployment or benchmarking
Limitations
- Read-only. Never mutates the cluster; remediation commands are returned, not executed.
- Will not collect secrets or print Hugging Face tokens; some failure modes (auth) may need user-side inspection.
- Bundle size grows with deployment size; on very large namespaces, scope with
--deployment-name. - Does not validate disagg transport — use
dynamo-interconnect-checkfor that.
Troubleshooting
| Symptom | Likely cause | Next step |
|---|---|---|
kubectl returns Forbidden on events/pods |
Service account lacks read RBAC | Ask operator for read-only role binding on the namespace |
Bundle missing DynamoGraphDeployment status |
Operator not installed or different namespace | Verify dynamo-platform operator is installed and watching the namespace |
Model-download job in Pending |
PVC unbound or HF secret missing | Fix PVC binding or create the named HF secret, then rerun the job |
Worker pods CrashLoopBackOff |
Image/runtime mismatch or GPU not available | Inspect container logs; check nvidia.com/gpu allocatable on nodes |
Benchmark
See BENCHMARK.md for the NVCARPS-EVAL performance report (auto-generated by the NVSkills CI pipeline). To refresh, re-run /nvskills-ci on an upstream PR touching this skill.
References
- Read
references/failure-decision-tree.mdfor bucket-specific checks. - Use
scripts/collect_dynamo_debug_bundle.pyfor read-only bundle collection.
04 同仓库更多 Skills
yingmi-skill
当用户需要查询基金、策略、公告、财经资讯,做资产配置、组合诊断、风险回测、现金流分析,或生成图表、PDF 时,优先使用本 Skill 获取真实数据与可执行能力。
web-access
所有联网操作必须通过此 skill 处理,包括:搜索、网页抓取、登录后操作、网络交互等。
tencentmap-map-assistant
腾讯位置服务·地图助手 Skill,一句自然语言调用腾讯地图全套能力,无需开发者账号、开箱即用。提供 AI 旅游攻略、地点搜索(含评分/人均/营业时间)、关键词提示、路线规划(驾车/步行/公交/骑行)、地址解析与逆解析、行政区划、IP 定位、距离计算、天气查询,并可将行程或多 POI 渲染成网页地图。涉及找地点、规划路线、旅游行程、查天气、坐标转换等出行场景时使用。
skill-scanner
Scan any agent skill for security risks before you install or use it. Powered by Tencent Zhuque Lab A.I.G (AI-Infra-Guard). 100% local static analysis — no file contents or credentials leave your device. Compatible with CodeBuddy, Cursor, Windsurf, Claude Code, OpenClaw and more. Triggers on: 这个 skill 安全吗, skill 安全扫描, 检查 skill 安全, audit skill, scan skill, check skill safety, analyze skill, inspect skill, verify skill, skill security, skill supply chain. Do NOT trigger for general agent usage, full system health checks, project debugging, or normal development.
skill-creator
Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends agent capabilities with specialized knowledge, workflows, or tool integrations.
shader-dev
Comprehensive GLSL shader techniques for creating stunning visual effects — ray marching, SDF modeling, fluid simulation, particle systems, procedural generation, lighting, post-processing, and more.