[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7CSK6AJ0GSM134F7K7MV3YA":3,"skill-same:01M7CSK6AJ0GSM134F7K7MV3YA":72},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":28,"repositoryHtmlUrl":29,"repositoryHomepageUrl":30,"ownerHtmlUrl":31,"ownerBlogUrl":6,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M7CSK6AJ0GSM134F7K7MV3YA","nemo-mbridge-multi-node-slurm",null,"Convert single-node scripts to multi-node Slurm sbatch jobs and debug common multi-node failures. Covers srun-native vs uv run torch.distributed approaches, container setup, NCCL timeouts, OOM sizing for MoE models, and interactive allocation.","YYC-Cube\u002FYYC3-Knowledge-Base","YYC-Cube","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F235006543?v=4","01M79J2PH4SWD2X0825X6PN748","yyc3","01M79J2PHVK3F6ZBYT8K2SCZCW",1,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Skills-技能包\u002FYYC3-Skills-技能包-skills\u002Fskills\u002Fnemo-mbridge-multi-node-slurm\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: nemo-mbridge-multi-node-slurm\ndescription: Convert single-node scripts to multi-node Slurm sbatch jobs and debug common multi-node failures. Covers srun-native vs uv run torch.distributed approaches, container setup, NCCL timeouts, OOM sizing for MoE models, and interactive allocation.\nlicense: Apache-2.0\nwhen_to_use: Writing or converting Slurm sbatch scripts, scaling to multiple nodes, debugging NCCL\u002Flaunch failures, or investigating a commit that caused multi-node training failures; 'run on multiple nodes', 'sbatch script', 'NCCL timeout', 'multi-node OOM'.\n---\n\n# Multi-Node Slurm\n\nConvert single-node `uv run python -m torch.distributed.run` commands into multi-node Slurm sbatch scripts with Enroot container support, and debug common multi-node failures.\n\n## First Answer Checklist\n\nWhen converting or debugging Bridge multi-node jobs, answer in this order:\n\n1. Prefer the **srun-native** launch shape for Bridge scripts that reach\n   `initialize.py`: `#SBATCH --ntasks-per-node=8` and a direct `srun ... uv run\n   python \u003Cscript> ...` launch. Do not wrap these jobs in\n   `python -m torch.distributed.run`.\n2. State that Bridge derives `RANK`, `WORLD_SIZE`, `LOCAL_RANK`,\n   `MASTER_ADDR`, and `MASTER_PORT` from SLURM variables during\n   `initialize.py` distributed init.\n3. Require shared paths and matching container mounts for the repo, data, logs,\n   `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME`.\n4. For NCCL timeout reports, do these first-log checks before speculating:\n   - grep for real errors while filtering warning\u002Fframe noise\n   - inspect `Failures:` to find the first failed rank and node\n   - grep for `ncclUniqueId`, `timeout`, or `crash on rank 0`\n\n## Two Approaches: srun-native vs uv run torch.distributed\n\n| Approach | `ntasks-per-node` | Process spawning | Best for |\n|---|---|---|---|\n| **srun-native** (preferred) | 8 | Slurm spawns 8 tasks\u002Fnode | Conversion, inference, Bridge scripts |\n| **uv run torch.distributed** (legacy) | 1 | `uv run python -m torch.distributed.run` spawns 8 procs\u002Fnode | MLM pretrain_gpt.py |\n\n**Prefer srun-native** — simpler, avoids shell escaping issues with TRAIN_CMD. Megatron Bridge auto-derives `RANK`, `WORLD_SIZE`, `LOCAL_RANK`, `MASTER_ADDR`, `MASTER_PORT` from SLURM env vars (`SLURM_PROCID`, `SLURM_NTASKS`, `SLURM_LOCALID`, `SLURM_NODELIST`) via `common_utils.py` helpers called during `initialize.py` distributed init, so you never need to set them manually.\n\n## Cluster Environment\n\nUse a shared filesystem for the repository, data, logs, `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME`. `NEMO_HOME` must not use the container-local default (`\u002Froot\u002F.cache\u002Fnemo`) for multi-node SFT\u002FPEFT jobs, because packed-sequence data prepared on node 0 must be visible to the other nodes.\n\nKeep credentials out of sbatch templates and logs. Provide `HF_TOKEN`, `GH_TOKEN`, and `WANDB_API_KEY` through the scheduler environment or a restricted secrets file, and never hardcode token values in the script body. For copy-paste environment and sbatch templates, read `references\u002Ftemplates.md`.\n\n### Log Directory\n\n```text\n\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_\u003Csuffix>\n```\n\n## srun-native Approach (Preferred)\n\nSlurm spawns all processes directly. No `torch.distributed.run`, no TRAIN_CMD escaping.\n\n### SBATCH Headers\n\n```bash\n#SBATCH --job-name=\u003Cmodel>-\u003Ctask>\n#SBATCH --nodes=\u003CNNODES>\n#SBATCH --ntasks-per-node=8          # Slurm spawns 8 tasks per node\n#SBATCH --gpus-per-node=8\n#SBATCH --time=00:30:00\n#SBATCH --account=\u003CYOUR_ACCOUNT>\n#SBATCH --partition=batch\n#SBATCH --output=\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_%j.log\n#SBATCH --exclusive\n```\n\n### Build and Launch\n\nUse a two-phase `srun` pattern: first run a single-process `uv sync` to populate the shared cache, then launch the full multi-node job. The full copy-paste version lives in `references\u002Ftemplates.md`.\n\n### srun-native Key Points\n\n- Phase 1 runs `uv sync` once on a single node\u002Fprocess, building all wheels into the shared cache on Lustre\n- Phase 2's `uv sync` is a fast no-op (everything is cached) — safe to run on all ranks without sleep guards\n- `initialize.py` + `common_utils.py` auto-set `RANK`, `WORLD_SIZE`, `LOCAL_RANK`, `MASTER_ADDR`, `MASTER_PORT` from SLURM env vars\n- Env vars like `HF_TOKEN`, `HF_HOME`, `UV_CACHE_DIR` exported at sbatch level are inherited by srun tasks\n- Reference: `examples\u002Fmodels\u002Fglm\u002Fglm_45v\u002Fslurm_sft.sh`, `examples\u002Fmodels\u002Fminimax\u002Fminimax_m2\u002Fslurm_conversion.sh`\n\n---\n\n## uv run torch.distributed Approach (Legacy)\n\nUse when the script requires `torch.distributed.run` (e.g., MLM pretrain_gpt.py) or when Bridge's `initialize.py` is not in the call path.\n\n### 1. Add SBATCH Headers\n\n```bash\n#SBATCH --job-name=\u003Cmodel>-\u003Cframework>\n#SBATCH --nodes=\u003CNNODES>\n#SBATCH --ntasks-per-node=1          # ALWAYS 1 — torchrun handles per-node spawning\n#SBATCH --gpus-per-node=8\n#SBATCH --time=00:30:00\n#SBATCH --account=\u003CYOUR_ACCOUNT>\n#SBATCH --partition=batch\n#SBATCH --output=\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_%j.log\n#SBATCH --exclusive\n```\n\n**Critical**: `--ntasks-per-node=1`, NOT 8. `uv run python -m torch.distributed.run --nproc_per_node=8` spawns 8 processes per node. Using `ntasks-per-node=8` causes EADDRINUSE port collisions (8 tasks x 8 procs = 64 per node).\n\n### 2. Convert to Multi-Node\n\nReplace single-node:\n\n```bash\nuv run python -m torch.distributed.run --nproc_per_node=8 \\\n  \u003Cscript> \u003Cargs>\n```\n\nWith multi-node (inside `TRAIN_CMD` string):\n\n```bash\nuv run python -m torch.distributed.run \\\n  --nproc_per_node=8 \\\n  --nnodes=\\${SLURM_JOB_NUM_NODES} \\\n  --node_rank=\\${SLURM_NODEID} \\\n  \u003Cscript> \u003Cargs>\n```\n\n`MASTER_ADDR` and `MASTER_PORT` are auto-derived from SLURM env vars by `initialize.py` \u002F `common_utils.py` — no need to set them.\n\n### 3. Wrap in TRAIN_CMD + two-phase srun\n\nUse the same two-phase pattern: first a single-process srun to warm the uv cache, then the full run.\n\nSet runtime variables inside the container, but do not inject token values into a long `bash -c` string. Export credentials through the scheduler or source a restricted secrets file before the job starts. Keep `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME` on shared storage.\n\n### 4. Launch (two-phase)\n\nUse the two-phase launch template in `references\u002Ftemplates.md`, keeping `#SBATCH --ntasks-per-node=1` for this legacy approach.\n\n### 5. (Optional) Add Loss Extraction Footer\n\n```bash\necho \"======================================\"\necho \"Done. Losses:\"\necho \"======================================\"\ngrep -E \"iteration\\s+\" \"$LOGDIR\u002F\u003Cprefix>_${SLURM_JOB_ID}.log\" | grep -iE \"lm loss|reduced_train_loss\" | head -25\n```\n\n---\n\n## Interactive GPU Allocation (`salloc` + `srun`)\n\nFor ad-hoc testing (inference, conversion debugging), always follow these 3 steps:\n\n### Step 1: Allocate the node\n\n```bash\nsalloc --account \u003CYOUR_ACCOUNT> -N 1 \\\n  -J \u003CYOUR_ACCOUNT>-debug \\\n  -p interactive --gpus-per-node=8 -t 240\n```\n\n### Step 2: Launch container shell\n\n```bash\nsrun --mpi=pmix --no-kill \\\n  --container-image $CONTAINER_IMAGE \\\n  --container-mounts $CONTAINER_MOUNTS \\\n  --account \u003CYOUR_ACCOUNT> -N 1 \\\n  -J \u003CYOUR_ACCOUNT>-debug \\\n  --no-container-mount-home --gpus-per-node=8 \\\n  -p interactive --pty bash\n```\n\n### Step 3: Set up environment inside container\n\n```bash\nexport GH_TOKEN=\u003CYOUR_GITHUB_TOKEN>\nwandb login \u003CYOUR_WANDB_KEY>\nexport HF_TOKEN=\u003CYOUR_HF_TOKEN>\nexport HF_HOME=\u003CSHARED_FS>\u002FHF_HOME\nexport UV_CACHE_DIR=\"\u003CSHARED_FS>\u002Fuv_cache\"\nexport NEMO_HOME=\"\u003CSHARED_FS>\u002Fcache\u002Fnemo\"\nuv sync\n```\n\nThen run commands with `uv run` (uses the synced virtualenv):\n\n```bash\nuv run python -m torch.distributed.run --nproc_per_node=8 \\\n  examples\u002Fconversion\u002Fhf_to_megatron_generate_text.py \\\n  --hf_model_path \u003Corg>\u002F\u003Cmodel> --prompt \"What is AI?\" --max_new_tokens 50 --ep 8\n```\n\n**Pitfalls with interactive allocation:**\n\n| Error | Cause | Fix |\n|---|---|---|\n| `Cannot find GPU specification` | Missing `--gpus-per-node` | Always include `--gpus-per-node=8` in both `salloc` and `srun` |\n| `invalid partition specified: pool0` | Wrong partition name | Use `interactive` for interactive, `batch` for sbatch. Check: `sinfo --summarize` |\n| `Invalid account or account\u002Fpartition combination` | Partition not available for account | Check combos: `sacctmgr -nP show assoc where user=$USER format=account,partition` |\n| `Unable to create step for job... Requested node configuration is not available` | `-w \u003Cnode>` conflicts with allocation | Remove `-w` flag — HF cache is on shared filesystem, accessible from any node |\n| `uv: command not found` inside container | Container doesn't have `uv` pre-installed | Use a container with `uv` pre-installed, or `pip install uv` |\n| `No space left on device` during `uv` or `pip` | Container's `\u002Froot\u002F.cache\u002F` is full | Redirect: `export UV_CACHE_DIR=\u003CSHARED_FS>\u002Fuv_cache` |\n| `ModuleNotFoundError: No module named 'megatron.core.activations'` | Container's pre-installed megatron-core conflicts with local `3rdparty\u002FMegatron-LM` | Install local: `pip install -e 3rdparty\u002FMegatron-LM --no-deps --no-build-isolation` |\n\n---\n\n## Debugging Multi-Node Failures\n\n### Quick Diagnosis\n\nCheck the log for these patterns (in order):\n\n```bash\n# 1. Find the actual error (filter noise)\ngrep -a 'Error\\|OOM\\|CUDA out of memory\\|FAILED\\|Killed' job.log \\\n  | grep -v 'UserWarning\\|AllocatorConfig\\|transformer_engine\\|frame\\|srun: error'\n\n# 2. Check which rank crashed first\ngrep -a 'Failures:' -A 20 job.log | head -25\n\n# 3. Check for NCCL timeout\ngrep -a 'ncclUniqueId\\|timeout\\|crash on rank 0' job.log | head -5\n```\n\n### Debugging Checklist\n\nWhen a multi-node job fails:\n\n1. **Check exit code**: 1 = Python error, 9 = OOM killed, 143 = SIGTERM (timeout or cascade)\n2. **Find first failure**: Which task\u002Fnode crashed first? Others get SIGTERM (143) as cascade\n3. **grep the actual error**: Filter out UserWarnings, NCCL frame dumps\n4. **Check rank 0 specifically**: Most save\u002Fexport errors happen on rank 0\n5. **Verify EP sizing**: For MoE models, ensure `num_experts \u002F EP` fits in GPU memory with headroom\n6. **Try interactive first**: Use `salloc -N 2 -p interactive` to iterate faster than sbatch queue\n\n### NCCL Timeout at `dist.barrier()` — \"crash on rank 0\"\n\n**Symptom**: All ranks on node 2+ show:\n```text\n[rank8] is setting up NCCL communicator and retrieving ncclUniqueId from [0]\n... wait timeout after 600000ms\nThis may indicate a possible application crash on rank 0\n```\n\n**Root causes** (check in order):\n\n| Cause | How to verify | Fix |\n|---|---|---|\n| `save_artifacts` hangs on rank 0 | Error is in `save_hf_weights` → `dist.barrier()` | Increase timeout: `init_process_group(\"nccl\", timeout=timedelta(minutes=60))` |\n| `ImportError` in custom model code | `grep ImportError job.log` | Catch `ImportError` in `save_artifacts` (see below) |\n| Rank 0 OOM during export | `grep 'OutOfMemory' job.log` | Increase EP or nodes |\n| Network issue between nodes | Error only on cross-node ranks | Check `sinfo`, try different nodes |\n\n**The `save_artifacts` problem**: When `trust_remote_code=True`, rank 0 runs `save_artifacts()` (downloads tokenizer, config, custom modeling code) while all other ranks skip directly to `dist.barrier()`. If `save_artifacts` is slow or crashes, other ranks timeout.\n\n**Fix for ImportError in save_artifacts** (`hf_pretrained\u002Fbase.py`):\n```python\n# Change:\nexcept OSError:\n    pass\n# To:\nexcept (OSError, ImportError):\n    pass\n```\n\n### OOM for MoE Models\n\n**Symptom**: `torch.OutOfMemoryError: CUDA out of memory` during model loading or forward pass.\n\n**Key insight**: TP does NOT reduce expert memory. Only EP splits experts across GPUs.\n\n**Sizing formula**:\n```text\nexperts_per_gpu = num_experts \u002F EP\nexpert_memory_gb ≈ experts_per_gpu * expert_params * 2 \u002F 1e9  (bf16)\ntotal_per_gpu ≈ expert_memory_gb + attention_memory_gb + kv_cache_gb\n```\n\n**MiniMax-M2 example** (256 experts, ~230GB fp8 → ~460GB bf16):\n\n| Config | Nodes | GPUs | Experts\u002FGPU | Result |\n|---|---|---|---|---|\n| TP=2, EP=4 | 1 | 8 | 64 | OOM (too many experts) |\n| TP=2, EP=8 | 2 | 16 | 32 | Works for roundtrip (weight-only), OOM for inference |\n| TP=1, EP=16 | 2 | 16 | 16 | Works for inference |\n| TP=2, EP=32 | 8 | 64 | 8 | Comfortable for training |\n\n**Rules of thumb**:\n- Roundtrip (weight-only): can use more experts per GPU (~60GB model params OK)\n- Inference (forward pass + KV cache): needs headroom (~40GB model params max)\n- Training (activations + optimizer): needs even more headroom (~30GB model params max)\n\n### `ModuleNotFoundError: No module named 'megatron.core.tensor_parallel'`\n\n**Cause**: Container's pre-installed megatron-core conflicts with local `3rdparty\u002FMegatron-LM`.\n\n**Fix**: Add `uv sync` before running:\n```bash\nCMD=\"if [ \\\"\\$SLURM_LOCALID\\\" -eq 0 ]; then uv sync; else sleep 10; fi && \"\nCMD=\"${CMD}uv run --no-sync python \u003Cscript> \u003Cargs>\"\n```\n\n### FP8 Weight Mismatch in Roundtrip\n\n**Symptom**: Roundtrip completes but shows ❌ for all expert weights and raises `ValueError: Weight mismatch detected`.\n\n**Cause**: Original HF weights are FP8, Megatron stores in BF16. Exported weights are BF16. Comparison against original FP8 exceeds `atol=1e-1`.\n\n**This is expected for FP8 models.** The conversion is correct; the comparison tolerance is insufficient for the FP8→BF16 precision gap.\n\n### `WORLD_SIZE` Not Set with srun\n\n**Symptom**: Script exits with \"must be launched with torchrun\".\n\n**Cause**: Scripts check `os.environ.get(\"WORLD_SIZE\")` which torchrun sets but srun doesn't.\n\n**Fix**: Also check `SLURM_NTASKS`:\n```python\nif os.environ.get(\"WORLD_SIZE\") is None and os.environ.get(\"SLURM_NTASKS\") is None:\n    sys.exit(1)\n```\n\nBridge's `common_utils.py` helpers (called by `initialize.py`) populate env vars from SLURM:\n```python\nif \"RANK\" not in os.environ:\n    os.environ[\"RANK\"] = str(get_rank_safe())          # uses SLURM_PROCID\nif \"WORLD_SIZE\" not in os.environ:\n    os.environ[\"WORLD_SIZE\"] = str(get_world_size_safe())  # uses SLURM_NTASKS\nif \"MASTER_ADDR\" not in os.environ:\n    os.environ[\"MASTER_ADDR\"] = get_master_addr_safe()     # parses SLURM_NODELIST\nif \"MASTER_PORT\" not in os.environ:\n    os.environ[\"MASTER_PORT\"] = str(get_master_port_safe()) # derives from SLURM_JOB_ID\n```\n\n---\n\n## Key Gotchas\n\n1. **Two-phase srun for `uv sync`**: Run a single-process srun first to warm the cache, then the full multi-node srun. The second `uv sync` is a fast no-op since everything is already cached on the shared filesystem.\n\n2. **`--no-container-mount-home`** is an `srun` flag, NOT an `#SBATCH` directive.\n\n3. **Escaping inside TRAIN_CMD**: Since `TRAIN_CMD` is a double-quoted string, escape inner `$` for Slurm variables that must expand at runtime (not sbatch time):\n   - `\\${SLURM_PROCID}`, `\\${SLURM_JOB_NUM_NODES}`, `\\${SLURM_NODEID}`\n   - Host-side variables like `$GH_TOKEN`, `$LOGDIR`, `$WORKDIR` expand at sbatch time — no escaping needed.\n\n4. **Bridge `rm -rf nemo_experiments`**: Add before training to avoid stale checkpoint auto-resume.\n\n5. **MLM needs PYTHONPATH**: For pretrain_gpt.py scripts, add inside TRAIN_CMD:\n   ```bash\n   PYTHONPATH=${WORKDIR}\u002F3rdparty\u002FMegatron-LM:\\${PYTHONPATH:-} \\\n   ```\n\n6. **Node count heuristic**: Total GPUs = `NNODES * 8`. Must satisfy: `TP * PP * EP * DP >= total_GPUs` where `DP = total_GPUs \u002F (TP * PP * EP)`.\n\n7. **`NEMO_HOME` on shared filesystem for multi-node SFT**: The default nemo cache (`\u002Froot\u002F.cache\u002Fnemo`)\n   is container-local. Multi-node SFT with packed sequences prepares `.npy` files on one node\n   that are invisible to others. Set `export NEMO_HOME=\u003CSHARED_FS>\u002Fcache\u002Fnemo` so packed data\n   is shared. Without this, ranks on other nodes fail with `TypeError: 'NoneType' object is not an iterator`.\n\n## Full Templates and Command Bodies\n\nFor copyable sbatch scaffolding and Bridge\u002FMLM-specific `TRAIN_CMD` bodies, read\n[references\u002Ftemplates.md](references\u002Ftemplates.md).\n","# Multi-Node Slurm\n\nConvert single-node `uv run python -m torch.distributed.run` commands into multi-node Slurm sbatch scripts with Enroot container support, and debug common multi-node failures.\n\n## First Answer Checklist\n\nWhen converting or debugging Bridge multi-node jobs, answer in this order:\n\n1. Prefer the **srun-native** launch shape for Bridge scripts that reach\n   `initialize.py`: `#SBATCH --ntasks-per-node=8` and a direct `srun ... uv run\n   python \u003Cscript> ...` launch. Do not wrap these jobs in\n   `python -m torch.distributed.run`.\n2. State that Bridge derives `RANK`, `WORLD_SIZE`, `LOCAL_RANK`,\n   `MASTER_ADDR`, and `MASTER_PORT` from SLURM variables during\n   `initialize.py` distributed init.\n3. Require shared paths and matching container mounts for the repo, data, logs,\n   `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME`.\n4. For NCCL timeout reports, do these first-log checks before speculating:\n   - grep for real errors while filtering warning\u002Fframe noise\n   - inspect `Failures:` to find the first failed rank and node\n   - grep for `ncclUniqueId`, `timeout`, or `crash on rank 0`\n\n## Two Approaches: srun-native vs uv run torch.distributed\n\n| Approach | `ntasks-per-node` | Process spawning | Best for |\n|---|---|---|---|\n| **srun-native** (preferred) | 8 | Slurm spawns 8 tasks\u002Fnode | Conversion, inference, Bridge scripts |\n| **uv run torch.distributed** (legacy) | 1 | `uv run python -m torch.distributed.run` spawns 8 procs\u002Fnode | MLM pretrain_gpt.py |\n\n**Prefer srun-native** — simpler, avoids shell escaping issues with TRAIN_CMD. Megatron Bridge auto-derives `RANK`, `WORLD_SIZE`, `LOCAL_RANK`, `MASTER_ADDR`, `MASTER_PORT` from SLURM env vars (`SLURM_PROCID`, `SLURM_NTASKS`, `SLURM_LOCALID`, `SLURM_NODELIST`) via `common_utils.py` helpers called during `initialize.py` distributed init, so you never need to set them manually.\n\n## Cluster Environment\n\nUse a shared filesystem for the repository, data, logs, `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME`. `NEMO_HOME` must not use the container-local default (`\u002Froot\u002F.cache\u002Fnemo`) for multi-node SFT\u002FPEFT jobs, because packed-sequence data prepared on node 0 must be visible to the other nodes.\n\nKeep credentials out of sbatch templates and logs. Provide `HF_TOKEN`, `GH_TOKEN`, and `WANDB_API_KEY` through the scheduler environment or a restricted secrets file, and never hardcode token values in the script body. For copy-paste environment and sbatch templates, read `references\u002Ftemplates.md`.\n\n### Log Directory\n\n```text\n\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_\u003Csuffix>\n```\n\n## srun-native Approach (Preferred)\n\nSlurm spawns all processes directly. No `torch.distributed.run`, no TRAIN_CMD escaping.\n\n### SBATCH Headers\n\n```bash\n#SBATCH --job-name=\u003Cmodel>-\u003Ctask>\n#SBATCH --nodes=\u003CNNODES>\n#SBATCH --ntasks-per-node=8          # Slurm spawns 8 tasks per node\n#SBATCH --gpus-per-node=8\n#SBATCH --time=00:30:00\n#SBATCH --account=\u003CYOUR_ACCOUNT>\n#SBATCH --partition=batch\n#SBATCH --output=\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_%j.log\n#SBATCH --exclusive\n```\n\n### Build and Launch\n\nUse a two-phase `srun` pattern: first run a single-process `uv sync` to populate the shared cache, then launch the full multi-node job. The full copy-paste version lives in `references\u002Ftemplates.md`.\n\n### srun-native Key Points\n\n- Phase 1 runs `uv sync` once on a single node\u002Fprocess, building all wheels into the shared cache on Lustre\n- Phase 2's `uv sync` is a fast no-op (everything is cached) — safe to run on all ranks without sleep guards\n- `initialize.py` + `common_utils.py` auto-set `RANK`, `WORLD_SIZE`, `LOCAL_RANK`, `MASTER_ADDR`, `MASTER_PORT` from SLURM env vars\n- Env vars like `HF_TOKEN`, `HF_HOME`, `UV_CACHE_DIR` exported at sbatch level are inherited by srun tasks\n- Reference: `examples\u002Fmodels\u002Fglm\u002Fglm_45v\u002Fslurm_sft.sh`, `examples\u002Fmodels\u002Fminimax\u002Fminimax_m2\u002Fslurm_conversion.sh`\n\n---\n\n## uv run torch.distributed Approach (Legacy)\n\nUse when the script requires `torch.distributed.run` (e.g., MLM pretrain_gpt.py) or when Bridge's `initialize.py` is not in the call path.\n\n### 1. Add SBATCH Headers\n\n```bash\n#SBATCH --job-name=\u003Cmodel>-\u003Cframework>\n#SBATCH --nodes=\u003CNNODES>\n#SBATCH --ntasks-per-node=1          # ALWAYS 1 — torchrun handles per-node spawning\n#SBATCH --gpus-per-node=8\n#SBATCH --time=00:30:00\n#SBATCH --account=\u003CYOUR_ACCOUNT>\n#SBATCH --partition=batch\n#SBATCH --output=\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_%j.log\n#SBATCH --exclusive\n```\n\n**Critical**: `--ntasks-per-node=1`, NOT 8. `uv run python -m torch.distributed.run --nproc_per_node=8` spawns 8 processes per node. Using `ntasks-per-node=8` causes EADDRINUSE port collisions (8 tasks x 8 procs = 64 per node).\n\n### 2. Convert to Multi-Node\n\nReplace single-node:\n\n```bash\nuv run python -m torch.distributed.run --nproc_per_node=8 \\\n  \u003Cscript> \u003Cargs>\n```\n\nWith multi-node (inside `TRAIN_CMD` string):\n\n```bash\nuv run python -m torch.distributed.run \\\n  --nproc_per_node=8 \\\n  --nnodes=\\${SLURM_JOB_NUM_NODES} \\\n  --node_rank=\\${SLURM_NODEID} \\\n  \u003Cscript> \u003Cargs>\n```\n\n`MASTER_ADDR` and `MASTER_PORT` are auto-derived from SLURM env vars by `initialize.py` \u002F `common_utils.py` — no need to set them.\n\n### 3. Wrap in TRAIN_CMD + two-phase srun\n\nUse the same two-phase pattern: first a single-process srun to warm the uv cache, then the full run.\n\nSet runtime variables inside the container, but do not inject token values into a long `bash -c` string. Export credentials through the scheduler or source a restricted secrets file before the job starts. Keep `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME` on shared storage.\n\n### 4. Launch (two-phase)\n\nUse the two-phase launch template in `references\u002Ftemplates.md`, keeping `#SBATCH --ntasks-per-node=1` for this legacy approach.\n\n### 5. (Optional) Add Loss Extraction Footer\n\n```bash\necho \"======================================\"\necho \"Done. Losses:\"\necho \"======================================\"\ngrep -E \"iteration\\s+\" \"$LOGDIR\u002F\u003Cprefix>_${SLURM_JOB_ID}.log\" | grep -iE \"lm loss|reduced_train_loss\" | head -25\n```\n\n---\n\n## Interactive GPU Allocation (`salloc` + `srun`)\n\nFor ad-hoc testing (inference, conversion debugging), always follow these 3 steps:\n\n### Step 1: Allocate the node\n\n```bash\nsalloc --account \u003CYOUR_ACCOUNT> -N 1 \\\n  -J \u003CYOUR_ACCOUNT>-debug \\\n  -p interactive --gpus-per-node=8 -t 240\n```\n\n### Step 2: Launch container shell\n\n```bash\nsrun --mpi=pmix --no-kill \\\n  --container-image $CONTAINER_IMAGE \\\n  --container-mounts $CONTAINER_MOUNTS \\\n  --account \u003CYOUR_ACCOUNT> -N 1 \\\n  -J \u003CYOUR_ACCOUNT>-debug \\\n  --no-container-mount-home --gpus-per-node=8 \\\n  -p interactive --pty bash\n```\n\n### Step 3: Set up environment inside container\n\n```bash\nexport GH_TOKEN=\u003CYOUR_GITHUB_TOKEN>\nwandb login \u003CYOUR_WANDB_KEY>\nexport HF_TOKEN=\u003CYOUR_HF_TOKEN>\nexport HF_HOME=\u003CSHARED_FS>\u002FHF_HOME\nexport UV_CACHE_DIR=\"\u003CSHARED_FS>\u002Fuv_cache\"\nexport NEMO_HOME=\"\u003CSHARED_FS>\u002Fcache\u002Fnemo\"\nuv sync\n```\n\nThen run commands with `uv run` (uses the synced virtualenv):\n\n```bash\nuv run python -m torch.distributed.run --nproc_per_node=8 \\\n  examples\u002Fconversion\u002Fhf_to_megatron_generate_text.py \\\n  --hf_model_path \u003Corg>\u002F\u003Cmodel> --prompt \"What is AI?\" --max_new_tokens 50 --ep 8\n```\n\n**Pitfalls with interactive allocation:**\n\n| Error | Cause | Fix |\n|---|---|---|\n| `Cannot find GPU specification` | Missing `--gpus-per-node` | Always include `--gpus-per-node=8` in both `salloc` and `srun` |\n| `invalid partition specified: pool0` | Wrong partition name | Use `interactive` for interactive, `batch` for sbatch. Check: `sinfo --summarize` |\n| `Invalid account or account\u002Fpartition combination` | Partition not available for account | Check combos: `sacctmgr -nP show assoc where user=$USER format=account,partition` |\n| `Unable to create step for job... Requested node configuration is not available` | `-w \u003Cnode>` conflicts with allocation | Remove `-w` flag — HF cache is on shared filesystem, accessible from any node |\n| `uv: command not found` inside container | Container doesn't have `uv` pre-installed | Use a container with `uv` pre-installed, or `pip install uv` |\n| `No space left on device` during `uv` or `pip` | Container's `\u002Froot\u002F.cache\u002F` is full | Redirect: `export UV_CACHE_DIR=\u003CSHARED_FS>\u002Fuv_cache` |\n| `ModuleNotFoundError: No module named 'megatron.core.activations'` | Container's pre-installed megatron-core conflicts with local `3rdparty\u002FMegatron-LM` | Install local: `pip install -e 3rdparty\u002FMegatron-LM --no-deps --no-build-isolation` |\n\n---\n\n## Debugging Multi-Node Failures\n\n### Quick Diagnosis\n\nCheck the log for these patterns (in order):\n\n```bash\n# 1. Find the actual error (filter noise)\ngrep -a 'Error\\|OOM\\|CUDA out of memory\\|FAILED\\|Killed' job.log \\\n  | grep -v 'UserWarning\\|AllocatorConfig\\|transformer_engine\\|frame\\|srun: error'\n\n# 2. Check which rank crashed first\ngrep -a 'Failures:' -A 20 job.log | head -25\n\n# 3. Check for NCCL timeout\ngrep -a 'ncclUniqueId\\|timeout\\|crash on rank 0' job.log | head -5\n```\n\n### Debugging Checklist\n\nWhen a multi-node job fails:\n\n1. **Check exit code**: 1 = Python error, 9 = OOM killed, 143 = SIGTERM (timeout or cascade)\n2. **Find first failure**: Which task\u002Fnode crashed first? Others get SIGTERM (143) as cascade\n3. **grep the actual error**: Filter out UserWarnings, NCCL frame dumps\n4. **Check rank 0 specifically**: Most save\u002Fexport errors happen on rank 0\n5. **Verify EP sizing**: For MoE models, ensure `num_experts \u002F EP` fits in GPU memory with headroom\n6. **Try interactive first**: Use `salloc -N 2 -p interactive` to iterate faster than sbatch queue\n\n### NCCL Timeout at `dist.barrier()` — \"crash on rank 0\"\n\n**Symptom**: All ranks on node 2+ show:\n```text\n[rank8] is setting up NCCL communicator and retrieving ncclUniqueId from [0]\n... wait timeout after 600000ms\nThis may indicate a possible application crash on rank 0\n```\n\n**Root causes** (check in order):\n\n| Cause | How to verify | Fix |\n|---|---|---|\n| `save_artifacts` hangs on rank 0 | Error is in `save_hf_weights` → `dist.barrier()` | Increase timeout: `init_process_group(\"nccl\", timeout=timedelta(minutes=60))` |\n| `ImportError` in custom model code | `grep ImportError job.log` | Catch `ImportError` in `save_artifacts` (see below) |\n| Rank 0 OOM during export | `grep 'OutOfMemory' job.log` | Increase EP or nodes |\n| Network issue between nodes | Error only on cross-node ranks | Check `sinfo`, try different nodes |\n\n**The `save_artifacts` problem**: When `trust_remote_code=True`, rank 0 runs `save_artifacts()` (downloads tokenizer, config, custom modeling code) while all other ranks skip directly to `dist.barrier()`. If `save_artifacts` is slow or crashes, other ranks timeout.\n\n**Fix for ImportError in save_artifacts** (`hf_pretrained\u002Fbase.py`):\n```python\n# Change:\nexcept OSError:\n    pass\n# To:\nexcept (OSError, ImportError):\n    pass\n```\n\n### OOM for MoE Models\n\n**Symptom**: `torch.OutOfMemoryError: CUDA out of memory` during model loading or forward pass.\n\n**Key insight**: TP does NOT reduce expert memory. Only EP splits experts across GPUs.\n\n**Sizing formula**:\n```text\nexperts_per_gpu = num_experts \u002F EP\nexpert_memory_gb ≈ experts_per_gpu * expert_params * 2 \u002F 1e9  (bf16)\ntotal_per_gpu ≈ expert_memory_gb + attention_memory_gb + kv_cache_gb\n```\n\n**MiniMax-M2 example** (256 experts, ~230GB fp8 → ~460GB bf16):\n\n| Config | Nodes | GPUs | Experts\u002FGPU | Result |\n|---|---|---|---|---|\n| TP=2, EP=4 | 1 | 8 | 64 | OOM (too many experts) |\n| TP=2, EP=8 | 2 | 16 | 32 | Works for roundtrip (weight-only), OOM for inference |\n| TP=1, EP=16 | 2 | 16 | 16 | Works for inference |\n| TP=2, EP=32 | 8 | 64 | 8 | Comfortable for training |\n\n**Rules of thumb**:\n- Roundtrip (weight-only): can use more experts per GPU (~60GB model params OK)\n- Inference (forward pass + KV cache): needs headroom (~40GB model params max)\n- Training (activations + optimizer): needs even more headroom (~30GB model params max)\n\n### `ModuleNotFoundError: No module named 'megatron.core.tensor_parallel'`\n\n**Cause**: Container's pre-installed megatron-core conflicts with local `3rdparty\u002FMegatron-LM`.\n\n**Fix**: Add `uv sync` before running:\n```bash\nCMD=\"if [ \\\"\\$SLURM_LOCALID\\\" -eq 0 ]; then uv sync; else sleep 10; fi && \"\nCMD=\"${CMD}uv run --no-sync python \u003Cscript> \u003Cargs>\"\n```\n\n### FP8 Weight Mismatch in Roundtrip\n\n**Symptom**: Roundtrip completes but shows ❌ for all expert weights and raises `ValueError: Weight mismatch detected`.\n\n**Cause**: Original HF weights are FP8, Megatron stores in BF16. Exported weights are BF16. Comparison against original FP8 exceeds `atol=1e-1`.\n\n**This is expected for FP8 models.** The conversion is correct; the comparison tolerance is insufficient for the FP8→BF16 precision gap.\n\n### `WORLD_SIZE` Not Set with srun\n\n**Symptom**: Script exits with \"must be launched with torchrun\".\n\n**Cause**: Scripts check `os.environ.get(\"WORLD_SIZE\")` which torchrun sets but srun doesn't.\n\n**Fix**: Also check `SLURM_NTASKS`:\n```python\nif os.environ.get(\"WORLD_SIZE\") is None and os.environ.get(\"SLURM_NTASKS\") is None:\n    sys.exit(1)\n```\n\nBridge's `common_utils.py` helpers (called by `initialize.py`) populate env vars from SLURM:\n```python\nif \"RANK\" not in os.environ:\n    os.environ[\"RANK\"] = str(get_rank_safe())          # uses SLURM_PROCID\nif \"WORLD_SIZE\" not in os.environ:\n    os.environ[\"WORLD_SIZE\"] = str(get_world_size_safe())  # uses SLURM_NTASKS\nif \"MASTER_ADDR\" not in os.environ:\n    os.environ[\"MASTER_ADDR\"] = get_master_addr_safe()     # parses SLURM_NODELIST\nif \"MASTER_PORT\" not in os.environ:\n    os.environ[\"MASTER_PORT\"] = str(get_master_port_safe()) # derives from SLURM_JOB_ID\n```\n\n---\n\n## Key Gotchas\n\n1. **Two-phase srun for `uv sync`**: Run a single-process srun first to warm the cache, then the full multi-node srun. The second `uv sync` is a fast no-op since everything is already cached on the shared filesystem.\n\n2. **`--no-container-mount-home`** is an `srun` flag, NOT an `#SBATCH` directive.\n\n3. **Escaping inside TRAIN_CMD**: Since `TRAIN_CMD` is a double-quoted string, escape inner `$` for Slurm variables that must expand at runtime (not sbatch time):\n   - `\\${SLURM_PROCID}`, `\\${SLURM_JOB_NUM_NODES}`, `\\${SLURM_NODEID}`\n   - Host-side variables like `$GH_TOKEN`, `$LOGDIR`, `$WORKDIR` expand at sbatch time — no escaping needed.\n\n4. **Bridge `rm -rf nemo_experiments`**: Add before training to avoid stale checkpoint auto-resume.\n\n5. **MLM needs PYTHONPATH**: For pretrain_gpt.py scripts, add inside TRAIN_CMD:\n   ```bash\n   PYTHONPATH=${WORKDIR}\u002F3rdparty\u002FMegatron-LM:\\${PYTHONPATH:-} \\\n   ```\n\n6. **Node count heuristic**: Total GPUs = `NNODES * 8`. Must satisfy: `TP * PP * EP * DP >= total_GPUs` where `DP = total_GPUs \u002F (TP * PP * EP)`.\n\n7. **`NEMO_HOME` on shared filesystem for multi-node SFT**: The default nemo cache (`\u002Froot\u002F.cache\u002Fnemo`)\n   is container-local. Multi-node SFT with packed sequences prepares `.npy` files on one node\n   that are invisible to others. Set `export NEMO_HOME=\u003CSHARED_FS>\u002Fcache\u002Fnemo` so packed data\n   is shared. Without this, ranks on other nodes fail with `TypeError: 'NoneType' object is not an iterator`.\n\n## Full Templates and Command Bodies\n\nFor copyable sbatch scaffolding and Bridge\u002FMLM-specific `TRAIN_CMD` bodies, read\n[references\u002Ftemplates.md](references\u002Ftemplates.md).\n",{"name":5,"license":15,"description":7,"when_to_use":24},"Writing or converting Slurm sbatch scripts, scaling to multiple nodes, debugging NCCL\u002Flaunch failures, or investigating a commit that caused multi-node training failures; 'run on multiple nodes', 'sbatch script', 'NCCL timeout', 'multi-node OOM'.","main","87afe6ce96220accf50a345dea46784c8c9f210e","YYC3-Skills-技能包\u002FYYC3-Skills-技能包-skills\u002Fskills\u002Fnemo-mbridge-multi-node-slurm",0,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base","https:\u002F\u002Fkb.yyc3.top\u002F","https:\u002F\u002Fgithub.com\u002FYYC-Cube","万象归元于云枢 | 深栈智启新纪元","2026-11-10 10:37:32",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill nemo-mbridge-multi-node-slurm",{"tool":40,"command":41},"bunx","bunx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill nemo-mbridge-multi-node-slurm",{"tool":43,"command":44},"pnpm","pnpm dlx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill nemo-mbridge-multi-node-slurm","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Skills-技能包\u002FYYC3-Skills-技能包-skills\u002Fskills\u002Fnemo-mbridge-multi-node-slurm\u002FSKILL.md",true,[48,54,58,62,66],{"path":49,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":52,"sourceHtmlUrl":53},"BENCHMARK.md","REFERENCE","参考文档",4315,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Skills-技能包\u002FYYC3-Skills-技能包-skills\u002Fskills\u002Fnemo-mbridge-multi-node-slurm\u002FBENCHMARK.md",{"path":55,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":56,"sourceHtmlUrl":57},"evals\u002Fevals.json",1729,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Skills-技能包\u002FYYC3-Skills-技能包-skills\u002Fskills\u002Fnemo-mbridge-multi-node-slurm\u002Fevals\u002Fevals.json",{"path":59,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":60,"sourceHtmlUrl":61},"references\u002Ftemplates.md",4744,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Skills-技能包\u002FYYC3-Skills-技能包-skills\u002Fskills\u002Fnemo-mbridge-multi-node-slurm\u002Freferences\u002Ftemplates.md",{"path":63,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":64,"sourceHtmlUrl":65},"skill-card.md",3738,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Skills-技能包\u002FYYC3-Skills-技能包-skills\u002Fskills\u002Fnemo-mbridge-multi-node-slurm\u002Fskill-card.md",{"path":67,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":70,"sourceHtmlUrl":71},"skill.oms.sig","OTHER","其他",4841,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Skills-技能包\u002FYYC3-Skills-技能包-skills\u002Fskills\u002Fnemo-mbridge-multi-node-slurm\u002Fskill.oms.sig",[73,81,88,95,102,109],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":19},"01M7CTDWHERCXPQCP0K03A86AS","vercel-sandbox","Run agent-browser + Chrome inside Vercel Sandbox microVMs for browser automation from any Vercel-deployed app. Use when the user needs browser automation in a Vercel app (Next.js, SvelteKit, Nuxt, Remix, Astro, etc.), wants to run headless Chrome without binary size limits, needs persistent browser sessions across commands, or wants ephemeral isolated browser environments. Triggers include \"Vercel Sandbox browser\", \"microVM Chrome\", \"agent-browser in sandbox\", \"browser automation on Vercel\", or any task requiring Chrome in a Vercel Sandbox.","MIT","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fvercel-sandbox\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M7CTD8D2BNJY54S65M47HEJA","slack","Interact with Slack workspaces using browser automation. Use when the user needs to check unread channels, navigate Slack, send messages, extract data, find information, search conversations, or automate any Slack task. Triggers include \"check my Slack\", \"what channels have unreads\", \"send a message to\", \"search Slack for\", \"extract from Slack\", \"find who said\", or any task requiring programmatic Slack interaction.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fslack\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M7CTCSSB8ZP3Q58C3YZDZK1W","electron","Automate Electron desktop apps (VS Code, Slack, Discord, Figma, Notion, Spotify, etc.) using agent-browser via Chrome DevTools Protocol. Use when the user needs to interact with an Electron app, automate a desktop app, connect to a running app, control a native app, or test an Electron application. Triggers include \"automate Slack app\", \"control VS Code\", \"interact with Discord app\", \"test this Electron app\", \"connect to desktop app\", or any task requiring automation of a native Electron application.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Felectron\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":19},"01M7CTCBJY76688V4DCZGWFHGQ","dogfood","Systematically explore and test a web application to find bugs, UX issues, and other problems. Use when asked to \"dogfood\", \"QA\", \"exploratory test\", \"find issues\", \"bug hunt\", \"test this app\u002Fsite\u002Fplatform\", or review the quality of a web application. Produces a structured report with full reproduction evidence -- step-by-step screenshots, repro videos, and detailed repro steps for every issue -- so findings can be handed directly to the responsible teams.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fdogfood\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":19},"01M7CTC4H0FFGVEAQR9PA4GA1A","agent-browser","Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to \"open a website\", \"fill out a form\", \"click a button\", \"take a screenshot\", \"scrape data from a page\", \"test this web app\", \"login to a site\", \"automate browser actions\", or any task requiring programmatic web interaction.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fagent-browser\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":77,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":19},"01M7CTBKVFYBG78FDSABSDJTTQ","shadcn","Manages shadcn components and projects — adding, searching, fixing, debugging, styling, and composing UI, including chat interfaces. Provides project context, component docs, and usage examples. Applies when working with shadcn\u002Fui, component registries, presets, --preset codes, or any project with a components.json file. Also triggers for \"shadcn init\", \"create an app with --preset\", \"switch to --preset\", and for building a design system or applying a DESIGN.md or brand spec to shadcn\u002Fui.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-UI-界面组件\u002FYYC3-UI-界面组件-ui\u002Fskills\u002Fshadcn\u002FSKILL.md",[],[]]