AGENT SKILL

dynamo-interconnect-check

Validate that a Dynamo deployment's NIXL/UCX/NCCL interconnect is ready for disaggregated serving over RDMA/NVLink. Use after recipe-runner brings a deployment up (especially disagg/multi-node) to confirm the KV transport is correct; use troubleshoot for diagnosing already-failed pods.

License 许可
Apache-2.0
最近更新
2026-07-19
Stars
★ 1
Forks
1

01 前言 Front Matter

name
dynamo-interconnect-check
license
Apache-2.0
metadata
{ "tags": [ "dynamo", "nixl", "rdma", "disagg", "validation" ], "author": "Dan Gil <dagil@nvidia.com>" }
description
Validate that a Dynamo deployment's NIXL/UCX/NCCL interconnect is ready for disaggregated serving over RDMA/NVLink. Use after recipe-runner brings a deployment up (especially disagg/multi-node) to confirm the KV transport is correct; use troubleshoot for diagnosing already-failed pods.

02 资源文件 7

03 SKILL.md 正文

Dynamo Interconnect Check

<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: CC-BY-4.0 -->

Purpose

Confirm that the transport disaggregated serving depends on actually works. A deployment can pass an endpoint smoke test while disagg is silently wrong: if NIXL/UCX cannot reach the peer worker over RDMA or NVLink, KV transfer falls back to a slow or broken path. Catch that with read-only checks before trusting a disagg deployment or its benchmark numbers.

This skill is read-only. It never mutates the cluster and never prints secrets.

Prerequisites

  • Python 3.10+ on the operator machine.
  • kubectl exec access to a worker pod in the target Dynamo deployment.
  • Read access to the recipe directory (recipes/<model>/<framework>/<mode>).
  • For node-capability checks: tools like ibstat, nvidia-smi, lsmod available in the worker pod image (missing tools are reported as skipped, not failures).

When To Use

  • After dynamo-recipe-runner deploys a disagg or multi-node recipe.
  • Before reporting disagg throughput/latency, so numbers reflect the real transport.
  • When agg works but disagg is slow, hangs, or returns wrong output and you suspect the fabric rather than the model.

For diagnosing pods that are already crashing or unschedulable, use dynamo-troubleshoot first.

Instructions

1. Check Transport Env Vars On The Recipe

python3 scripts/check_interconnect.py env recipes/<model>/<framework>/<mode>

Reports which NIXL/UCX/NCCL transport variables are set and flags disagg-critical ones (e.g. UCX_TLS, UCX_NET_DEVICES, NCCL_IB_HCA) that are absent. Missing here is only a warning — they may be baked into the image — so confirm with the node and NIXL checks. See references/interconnect-env-vars.md for what each variable does.

2. Check Node Capabilities

Locally on a GPU node, or inside a running worker pod:

python3 scripts/check_interconnect.py node \
  --namespace "${NAMESPACE}" --pod <worker-pod>

Probes (read-only) for: InfiniBand devices and Active links, GPUDirect RDMA (nvidia_peermem), GDRCopy, and NVLink in the GPU topology. Missing tools are reported as skipped, not failures.

3. Validate NIXL Reachability

python3 scripts/check_interconnect.py nixl \
  --namespace "${NAMESPACE}" --pod <worker-pod>

Looks for NIXL test tooling in the pod and surfaces the exact next step to run a pairwise prefill↔decode transfer test. A full cross-pod transfer test requires two scheduled GPU pods on the fabric.

Available Scripts

Script Purpose Arguments
scripts/check_interconnect.py env Inspect NIXL/UCX/NCCL env vars on a recipe positional recipe path
scripts/check_interconnect.py node Probe InfiniBand, GPUDirect RDMA, GDRCopy, NVLink on a node or pod --namespace, --pod
scripts/check_interconnect.py nixl Surface NIXL transfer-test readiness for a pod --namespace, --pod

Invoke via the agentskills.io run_script() protocol:

run_script("scripts/check_interconnect.py", args=["env", "recipes/qwen3-coder-480b/sglang/disagg"])
run_script("scripts/check_interconnect.py", args=["node", "--namespace", "dynamo-demo", "--pod", "qwen-worker-0"])

Examples

Verify a disagg recipe's transport env shape before deploy:

python3 scripts/check_interconnect.py env recipes/qwen3-coder-480b/sglang/disagg

After deploy, validate a worker pod's fabric:

python3 scripts/check_interconnect.py node \
  --namespace dynamo-demo --pod qwen-worker-0
python3 scripts/check_interconnect.py nixl \
  --namespace dynamo-demo --pod qwen-worker-0

Equivalent through the agent protocol:

run_script("scripts/check_interconnect.py", args=["nixl", "--namespace", "dynamo-demo", "--pod", "qwen-worker-0"])

Output Contract

Each check returns ok / warn / fail / skipped with a one-line detail, plus a rolled-up verdict on disagg transport readiness. Report:

  • transport env vars present vs. disagg-critical ones missing
  • RDMA / GPUDirect / NVLink capability status
  • whether NIXL reachability was validated, and the next command if not
  • a clear statement of whether disagg can be trusted, or what to fix first

Limitations

  • Read-only fabric probe; does not run a full pairwise NIXL transfer (requires two scheduled GPU pods and the in-pod NIXL test tools).
  • skipped results for missing tools (ibstat, nvidia-smi, lsmod) are inconclusive, not a pass.
  • Env-var check inspects the recipe text; values injected at runtime via initContainers or operator-applied envs are not detected.
  • Single-node agg deployments do not exercise the transport — this skill is for disagg / multi-node validation.

Troubleshooting

Symptom Likely cause Next step
env reports all critical vars missing Vars baked into image or injected by operator Run the node check inside the worker pod to verify actual env
node reports no Active IB link Fabric down or HCA not provisioned to the node Contact cluster admin; verify kubectl describe node shows nvidia.com/gpu and IB labels
nvidia_peermem missing GPUDirect RDMA module not loaded Ask cluster admin to load nvidia-peermem; without it, NIXL falls back to staged copies
nixl finds no test tools Worker image lacks NIXL test harness Use a NIXL-enabled image or run the standalone transfer test from a debug pod

Benchmark

See BENCHMARK.md for the NVCARPS-EVAL performance report (auto-generated by the NVSkills CI pipeline). To refresh, re-run /nvskills-ci on an upstream PR touching this skill.

References

  • references/interconnect-env-vars.md — NIXL/UCX/NCCL env var catalog and IB capability checklist.
  • Use scripts/check_interconnect.py for all read-only checks.

04 同仓库更多 Skills

zfifteen
zfifteen/skills-catalog

Zotero

Use Zotero Desktop from Codex to enable/probe the local API, search a local Zotero library, list items/collections/tags, export BibTeX, insert citation keys into LaTeX or Markdown drafts, read indexed full text when requested, and import BibTeX/RIS records into Zotero through the connector server. Use when the user mentions Zotero, citations, references.bib, BibTeX export, local Zotero API, localhost:23119, or adding citations from a Zotero library.

★ 1
zfifteen
zfifteen/skills-catalog

zoom-apps-sdk

Use when using Apps SDK.

★ 1
zfifteen
zfifteen/skills-catalog

youtube

Obtain accurate, timestamped YouTube video transcripts (preferring manual captions over auto-generated) as the primary deliverable, then derive summaries, key claims, or notes from that transcript. Use when the user provides a YouTube URL (or video ID) and wants the transcript, a faithful summary, key claims with attribution, timestamped notes, or "what does this video actually say". The skill is transcript-first: never substitute metadata, title, description, or search snippets for the spoken content. Triggers: "youtube transcript", "get the transcript of this video", "summarize this youtube video", "what did they say in https://youtube.com/...", "key claims from this talk".

★ 1
zfifteen
zfifteen/skills-catalog

yeet

Publish local changes to GitHub by confirming scope, committing intentionally, pushing the branch, and opening a draft PR through the GitHub app from this plugin, with `gh` used only as a fallback where connector coverage is insufficient.

★ 1
zfifteen
zfifteen/skills-catalog

x-post-creator

Create X-ready post threads and co-located story packages from the active conversation or referenced project material. Use when the user says turn this into an X-post, /x-post, make this into an X thread, or wants plain-language social posts with supporting images saved to docs/x-posts/[story-name]/.

★ 1
zfifteen
zfifteen/skills-catalog

x-api

Search, read, and write X posts through the local X API MCP server (the same backend used by the Codex x-api skill). Use for searching X/x.com, reading threads and replies, creating lists, staging/publishing posts for @alltheputs, finding high-visibility reply targets for research outreach, ingesting public context, or any request that mentions /x-api or "search X".

★ 1