dynamo-interconnect-check
Validate that a Dynamo deployment's NIXL/UCX/NCCL interconnect is ready for disaggregated serving over RDMA/NVLink. Use after recipe-runner brings a deployment up (especially disagg/multi-node) to confirm the KV transport is correct; use troubleshoot for diagnosing already-failed pods.
- License 许可
- Apache-2.0
- 最近更新
- 2026-08-25
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- dynamo-interconnect-check
- license
- Apache-2.0
- metadata
- { "tags": [ "dynamo", "nixl", "rdma", "disagg", "validation" ], "author": "Dan Gil <dagil@nvidia.com>" }
- description
- Validate that a Dynamo deployment's NIXL/UCX/NCCL interconnect is ready for disaggregated serving over RDMA/NVLink. Use after recipe-runner brings a deployment up (especially disagg/multi-node) to confirm the KV transport is correct; use troubleshoot for diagnosing already-failed pods.
02 资源文件 7
- 参考文档agents/openai.yaml365 B
- 参考文档BENCHMARK.md2.9 KB
- 参考文档evals/evals.json3.1 KB
- 参考文档references/interconnect-env-vars.md3.5 KB
- 脚本scripts/check_interconnect.py12.8 KB
- 参考文档skill-card.md2.5 KB
- 其他skill.oms.sig5.0 KB
03 SKILL.md 正文
Dynamo Interconnect Check
<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: CC-BY-4.0 -->
Purpose
Confirm that the transport disaggregated serving depends on actually works. A deployment can pass an endpoint smoke test while disagg is silently wrong: if NIXL/UCX cannot reach the peer worker over RDMA or NVLink, KV transfer falls back to a slow or broken path. Catch that with read-only checks before trusting a disagg deployment or its benchmark numbers.
This skill is read-only. It never mutates the cluster and never prints secrets.
Prerequisites
- Python 3.10+ on the operator machine.
kubectl execaccess to a worker pod in the target Dynamo deployment.- Read access to the recipe directory (
recipes/<model>/<framework>/<mode>). - For node-capability checks: tools like
ibstat,nvidia-smi,lsmodavailable in the worker pod image (missing tools are reported asskipped, not failures).
When To Use
- After
dynamo-recipe-runnerdeploys a disagg or multi-node recipe. - Before reporting disagg throughput/latency, so numbers reflect the real transport.
- When agg works but disagg is slow, hangs, or returns wrong output and you suspect the fabric rather than the model.
For diagnosing pods that are already crashing or unschedulable, use
dynamo-troubleshoot first.
Instructions
1. Check Transport Env Vars On The Recipe
python3 scripts/check_interconnect.py env recipes/<model>/<framework>/<mode>
Reports which NIXL/UCX/NCCL transport variables are set and flags
disagg-critical ones (e.g. UCX_TLS, UCX_NET_DEVICES, NCCL_IB_HCA) that are
absent. Missing here is only a warning — they may be baked into the image — so
confirm with the node and NIXL checks. See
references/interconnect-env-vars.md for what each variable does.
2. Check Node Capabilities
Locally on a GPU node, or inside a running worker pod:
python3 scripts/check_interconnect.py node \
--namespace "${NAMESPACE}" --pod <worker-pod>
Probes (read-only) for: InfiniBand devices and Active links, GPUDirect RDMA
(nvidia_peermem), GDRCopy, and NVLink in the GPU topology. Missing tools are
reported as skipped, not failures.
3. Validate NIXL Reachability
python3 scripts/check_interconnect.py nixl \
--namespace "${NAMESPACE}" --pod <worker-pod>
Looks for NIXL test tooling in the pod and surfaces the exact next step to run a pairwise prefill↔decode transfer test. A full cross-pod transfer test requires two scheduled GPU pods on the fabric.
Available Scripts
| Script | Purpose | Arguments |
|---|---|---|
scripts/check_interconnect.py env |
Inspect NIXL/UCX/NCCL env vars on a recipe | positional recipe path |
scripts/check_interconnect.py node |
Probe InfiniBand, GPUDirect RDMA, GDRCopy, NVLink on a node or pod | --namespace, --pod |
scripts/check_interconnect.py nixl |
Surface NIXL transfer-test readiness for a pod | --namespace, --pod |
Invoke via the agentskills.io run_script() protocol:
run_script("scripts/check_interconnect.py", args=["env", "recipes/qwen3-coder-480b/sglang/disagg"])
run_script("scripts/check_interconnect.py", args=["node", "--namespace", "dynamo-demo", "--pod", "qwen-worker-0"])
Examples
Verify a disagg recipe's transport env shape before deploy:
python3 scripts/check_interconnect.py env recipes/qwen3-coder-480b/sglang/disagg
After deploy, validate a worker pod's fabric:
python3 scripts/check_interconnect.py node \
--namespace dynamo-demo --pod qwen-worker-0
python3 scripts/check_interconnect.py nixl \
--namespace dynamo-demo --pod qwen-worker-0
Equivalent through the agent protocol:
run_script("scripts/check_interconnect.py", args=["nixl", "--namespace", "dynamo-demo", "--pod", "qwen-worker-0"])
Output Contract
Each check returns ok / warn / fail / skipped with a one-line detail,
plus a rolled-up verdict on disagg transport readiness. Report:
- transport env vars present vs. disagg-critical ones missing
- RDMA / GPUDirect / NVLink capability status
- whether NIXL reachability was validated, and the next command if not
- a clear statement of whether disagg can be trusted, or what to fix first
Limitations
- Read-only fabric probe; does not run a full pairwise NIXL transfer (requires two scheduled GPU pods and the in-pod NIXL test tools).
skippedresults for missing tools (ibstat,nvidia-smi,lsmod) are inconclusive, not a pass.- Env-var check inspects the recipe text; values injected at runtime via initContainers or operator-applied envs are not detected.
- Single-node agg deployments do not exercise the transport — this skill is for disagg / multi-node validation.
Troubleshooting
| Symptom | Likely cause | Next step |
|---|---|---|
env reports all critical vars missing |
Vars baked into image or injected by operator | Run the node check inside the worker pod to verify actual env |
node reports no Active IB link |
Fabric down or HCA not provisioned to the node | Contact cluster admin; verify kubectl describe node shows nvidia.com/gpu and IB labels |
nvidia_peermem missing |
GPUDirect RDMA module not loaded | Ask cluster admin to load nvidia-peermem; without it, NIXL falls back to staged copies |
nixl finds no test tools |
Worker image lacks NIXL test harness | Use a NIXL-enabled image or run the standalone transfer test from a debug pod |
Benchmark
See BENCHMARK.md for the NVCARPS-EVAL performance report (auto-generated by the NVSkills CI pipeline). To refresh, re-run /nvskills-ci on an upstream PR touching this skill.
References
references/interconnect-env-vars.md— NIXL/UCX/NCCL env var catalog and IB capability checklist.- Use
scripts/check_interconnect.pyfor all read-only checks.
04 同仓库更多 Skills
syllabus
Generates a curated supplementary reading list from any course syllabus using Consensus academic search. Grill-me intake (syllabus input format + course audience + year range) plus a grouping forcing-options checkpoint before any search runs — so the reading list matches the course's level and recency need. Parses the syllabus to extract topics and learning outcomes, searches Consensus for recent peer-reviewed papers per topic, and produces a professionally formatted .docx with clickable Consensus links, plain-language summaries calibrated to audience level, and Bloom-higher-order discussion questions tied to course learning goals. Use when the user uploads a syllabus, course outline, or curriculum document and wants supplementary readings (e.g., 'create a reading list from this syllabus', 'find recent papers for my course') — even casual mentions with a syllabus attached should trigger this skill.
pulse
Multi-source recency research skill that takes the pulse of any topic across Reddit, Hacker News, the open web, and optionally X/Twitter within a configurable recent window (default 30 days). Forcing intake clarifies topic specificity, angle (trend/sentiment/problems/opportunities/comparison), time window, and platform scope before searching. Returns a synthesized briefing with citations, engagement metrics, and cross-platform pattern analysis. Use when the user requests multi-source recency intelligence on a topic (e.g., 'pulse on [topic]', 'what's happening with [topic]', 'what are people saying about [topic]', 'current conversation about [topic]', 'take the pulse of [topic]', 'trending: [topic]', 'find me info on [topic]'), and for competitor research, trend discovery, tool comparisons, and audience sentiment analysis.
patent
Patent prior-art and landscape intelligence skill — not generic patent help. Commits to one of five sub-use-cases via forcing intake (novelty search / freedom-to-operate / competitive landscape / acquisition diligence / litigation prior-art) before any search runs. Searches Google Patents, Espacenet, USPTO, and optionally Lens.org for citation-graph signals. Output is an editable Word document (.docx) with verdict, ranked closest art (claim-text extracted), CPC-class-aware landscape, family-resolved hits, geographic coverage, FTO flags where applicable, strategy recommendations, and full audit log. Use when the user asks for patent searching or analysis (e.g., 'prior art search for [invention]', 'freedom to operate analysis for [product]'). Produces search signal, not legal advice — always recommends consulting a patent attorney before filing or licensing decisions. Trademark, copyright, and trade-secret questions are out of scope.
notebooklm
Browser automation skill for controlling Google's NotebookLM. Use when the user wants anything done in NotebookLM (e.g., 'open NotebookLM', 'check my [name] notebook', 'ask my notebook about X', 'add [source] to NotebookLM', 'generate a Video Overview from my notebook', 'use NotebookLM Studio'). Handles reading and querying notebooks, adding sources (URLs, text, files, YouTube links, synthesized content), generating Studio outputs (Audio/Video Overviews, Mind Maps, Reports incl. Briefing Doc/Study Guide/FAQ, Flashcards, Quiz, slide decks, infographics — discover the exact set from the live Studio panel; the UI evolves fast), and creating new notebooks. Requires browser automation environment — fails gracefully when unavailable.
litreview
Academic literature orientation skill that searches papers via free keyless APIs (PubMed E-utilities + OpenAlex) by default — with the Consensus MCP as an optional enhancement lane when connected — builds a strategic search plan using PICO (default) or SPIDER / Decomposition / hybrid as fallbacks, and synthesizes findings into a formatted Word (.docx) research guide. Grill-me intake (research question specificity + framework hint + tentative depth) before the recon search; a second forcing checkpoint after Phase 2 confirms framework + sub-areas + depth before searches consume budget. Configurable depth (5/10/20 queries) controls coverage vs. speed. Output is a 'launching pad' — an orientation guide that lets a researcher dive in confidently, not a finished review. Use when the user starts literature-oriented research (e.g., 'litreview on [topic]', 'literature review on [topic]', 'I'm starting a literature review on X', 'I'm writing a paper on X', 'help me research X', 'I'm doing research on X', 'can you help me research X'). Do NOT use for single one-off paper searches wanting a quick list — that's a plain PubMed/OpenAlex (or Consensus) query.
grants
NIH grant research skill for clinical researchers. Grill-me intake (research idea + career stage + preliminary data + environment + submission posture + known institute targets) locks down the funding strategy before any search runs. Runs a 5-facet Consensus positioning analysis (with draft Significance/Innovation language), maps the research to the right NIH institutes and study sections via RePORTER, finds NOSIs and funded overlap, and produces an editable Word document (.docx) with budget/scope-aware mechanism recommendations, submission timelines, and a mandatory program officer recommendation. Use when the user asks about research funding or makes any grant-related request (e.g., 'grants for [topic]', 'find grants for my research idea', 'what grants match my research', 'help me find NIH funding', 'grant opportunities for my research'). NIH-only scope — non-NIH funders (PCORI, DOD CDMRP, VA, foundations) are out of scope and flagged at intake.