[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M78J2VWDTAMPKB9M4HA329NK":3,"skill-same:01M78J2VWDTAMPKB9M4HA329NK":69},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":13,"repositoryHtmlUrl":27,"repositoryHomepageUrl":28,"ownerHtmlUrl":29,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":30,"install":31,"resources":44},"01M78J2VWDTAMPKB9M4HA329NK","nemo-mbridge-multi-node-slurm",null,"Convert single-node scripts to multi-node Slurm sbatch jobs and debug common multi-node failures. Covers srun-native vs uv run torch.distributed approaches, container setup, NCCL timeouts, OOM sizing for MoE models, and interactive allocation.","matheussluzz\u002FZeuZ-Agent","matheussluzz","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F209255758?v=4","01M78HKHH1D1ASC9KC9BG583DH","01M78HKHHP6W7XRXQM2SBCAM3R",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-multi-node-slurm\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: nemo-mbridge-multi-node-slurm\ndescription: Convert single-node scripts to multi-node Slurm sbatch jobs and debug common multi-node failures. Covers srun-native vs uv run torch.distributed approaches, container setup, NCCL timeouts, OOM sizing for MoE models, and interactive allocation.\nlicense: Apache-2.0\nwhen_to_use: Writing or converting Slurm sbatch scripts, scaling to multiple nodes, debugging NCCL\u002Flaunch failures, or investigating a commit that caused multi-node training failures; 'run on multiple nodes', 'sbatch script', 'NCCL timeout', 'multi-node OOM'.\n---\n\n# Multi-Node Slurm\n\nConvert single-node `uv run python -m torch.distributed.run` commands into multi-node Slurm sbatch scripts with Enroot container support, and debug common multi-node failures.\n\n## First Answer Checklist\n\nWhen converting or debugging Bridge multi-node jobs, answer in this order:\n\n1. Prefer the **srun-native** launch shape for Bridge scripts that reach\n   `initialize.py`: `#SBATCH --ntasks-per-node=8` and a direct `srun ... uv run\n   python \u003Cscript> ...` launch. Do not wrap these jobs in\n   `python -m torch.distributed.run`.\n2. State that Bridge derives `RANK`, `WORLD_SIZE`, `LOCAL_RANK`,\n   `MASTER_ADDR`, and `MASTER_PORT` from SLURM variables during\n   `initialize.py` distributed init.\n3. Require shared paths and matching container mounts for the repo, data, logs,\n   `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME`.\n4. For NCCL timeout reports, do these first-log checks before speculating:\n   - grep for real errors while filtering warning\u002Fframe noise\n   - inspect `Failures:` to find the first failed rank and node\n   - grep for `ncclUniqueId`, `timeout`, or `crash on rank 0`\n\n## Two Approaches: srun-native vs uv run torch.distributed\n\n| Approach | `ntasks-per-node` | Process spawning | Best for |\n|---|---|---|---|\n| **srun-native** (preferred) | 8 | Slurm spawns 8 tasks\u002Fnode | Conversion, inference, Bridge scripts |\n| **uv run torch.distributed** (legacy) | 1 | `uv run python -m torch.distributed.run` spawns 8 procs\u002Fnode | MLM pretrain_gpt.py |\n\n**Prefer srun-native** — simpler, avoids shell escaping issues with TRAIN_CMD. Megatron Bridge auto-derives `RANK`, `WORLD_SIZE`, `LOCAL_RANK`, `MASTER_ADDR`, `MASTER_PORT` from SLURM env vars (`SLURM_PROCID`, `SLURM_NTASKS`, `SLURM_LOCALID`, `SLURM_NODELIST`) via `common_utils.py` helpers called during `initialize.py` distributed init, so you never need to set them manually.\n\n## Cluster Environment\n\nUse a shared filesystem for the repository, data, logs, `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME`. `NEMO_HOME` must not use the container-local default (`\u002Froot\u002F.cache\u002Fnemo`) for multi-node SFT\u002FPEFT jobs, because packed-sequence data prepared on node 0 must be visible to the other nodes.\n\nKeep credentials out of sbatch templates and logs. Provide `HF_TOKEN`, `GH_TOKEN`, and `WANDB_API_KEY` through the scheduler environment or a restricted secrets file, and never hardcode token values in the script body. For copy-paste environment and sbatch templates, read `references\u002Ftemplates.md`.\n\n### Log Directory\n\n```text\n\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_\u003Csuffix>\n```\n\n## srun-native Approach (Preferred)\n\nSlurm spawns all processes directly. No `torch.distributed.run`, no TRAIN_CMD escaping.\n\n### SBATCH Headers\n\n```bash\n#SBATCH --job-name=\u003Cmodel>-\u003Ctask>\n#SBATCH --nodes=\u003CNNODES>\n#SBATCH --ntasks-per-node=8          # Slurm spawns 8 tasks per node\n#SBATCH --gpus-per-node=8\n#SBATCH --time=00:30:00\n#SBATCH --account=\u003CYOUR_ACCOUNT>\n#SBATCH --partition=batch\n#SBATCH --output=\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_%j.log\n#SBATCH --exclusive\n```\n\n### Build and Launch\n\nUse a two-phase `srun` pattern: first run a single-process `uv sync` to populate the shared cache, then launch the full multi-node job. The full copy-paste version lives in `references\u002Ftemplates.md`.\n\n### srun-native Key Points\n\n- Phase 1 runs `uv sync` once on a single node\u002Fprocess, building all wheels into the shared cache on Lustre\n- Phase 2's `uv sync` is a fast no-op (everything is cached) — safe to run on all ranks without sleep guards\n- `initialize.py` + `common_utils.py` auto-set `RANK`, `WORLD_SIZE`, `LOCAL_RANK`, `MASTER_ADDR`, `MASTER_PORT` from SLURM env vars\n- Env vars like `HF_TOKEN`, `HF_HOME`, `UV_CACHE_DIR` exported at sbatch level are inherited by srun tasks\n- Reference: `examples\u002Fmodels\u002Fglm\u002Fglm_45v\u002Fslurm_sft.sh`, `examples\u002Fmodels\u002Fminimax\u002Fminimax_m2\u002Fslurm_conversion.sh`\n\n---\n\n## uv run torch.distributed Approach (Legacy)\n\nUse when the script requires `torch.distributed.run` (e.g., MLM pretrain_gpt.py) or when Bridge's `initialize.py` is not in the call path.\n\n### 1. Add SBATCH Headers\n\n```bash\n#SBATCH --job-name=\u003Cmodel>-\u003Cframework>\n#SBATCH --nodes=\u003CNNODES>\n#SBATCH --ntasks-per-node=1          # ALWAYS 1 — torchrun handles per-node spawning\n#SBATCH --gpus-per-node=8\n#SBATCH --time=00:30:00\n#SBATCH --account=\u003CYOUR_ACCOUNT>\n#SBATCH --partition=batch\n#SBATCH --output=\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_%j.log\n#SBATCH --exclusive\n```\n\n**Critical**: `--ntasks-per-node=1`, NOT 8. `uv run python -m torch.distributed.run --nproc_per_node=8` spawns 8 processes per node. Using `ntasks-per-node=8` causes EADDRINUSE port collisions (8 tasks x 8 procs = 64 per node).\n\n### 2. Convert to Multi-Node\n\nReplace single-node:\n\n```bash\nuv run python -m torch.distributed.run --nproc_per_node=8 \\\n  \u003Cscript> \u003Cargs>\n```\n\nWith multi-node (inside `TRAIN_CMD` string):\n\n```bash\nuv run python -m torch.distributed.run \\\n  --nproc_per_node=8 \\\n  --nnodes=\\${SLURM_JOB_NUM_NODES} \\\n  --node_rank=\\${SLURM_NODEID} \\\n  \u003Cscript> \u003Cargs>\n```\n\n`MASTER_ADDR` and `MASTER_PORT` are auto-derived from SLURM env vars by `initialize.py` \u002F `common_utils.py` — no need to set them.\n\n### 3. Wrap in TRAIN_CMD + two-phase srun\n\nUse the same two-phase pattern: first a single-process srun to warm the uv cache, then the full run.\n\nSet runtime variables inside the container, but do not inject token values into a long `bash -c` string. Export credentials through the scheduler or source a restricted secrets file before the job starts. Keep `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME` on shared storage.\n\n### 4. Launch (two-phase)\n\nUse the two-phase launch template in `references\u002Ftemplates.md`, keeping `#SBATCH --ntasks-per-node=1` for this legacy approach.\n\n### 5. (Optional) Add Loss Extraction Footer\n\n```bash\necho \"======================================\"\necho \"Done. Losses:\"\necho \"======================================\"\ngrep -E \"iteration\\s+\" \"$LOGDIR\u002F\u003Cprefix>_${SLURM_JOB_ID}.log\" | grep -iE \"lm loss|reduced_train_loss\" | head -25\n```\n\n---\n\n## Interactive GPU Allocation (`salloc` + `srun`)\n\nFor ad-hoc testing (inference, conversion debugging), always follow these 3 steps:\n\n### Step 1: Allocate the node\n\n```bash\nsalloc --account \u003CYOUR_ACCOUNT> -N 1 \\\n  -J \u003CYOUR_ACCOUNT>-debug \\\n  -p interactive --gpus-per-node=8 -t 240\n```\n\n### Step 2: Launch container shell\n\n```bash\nsrun --mpi=pmix --no-kill \\\n  --container-image $CONTAINER_IMAGE \\\n  --container-mounts $CONTAINER_MOUNTS \\\n  --account \u003CYOUR_ACCOUNT> -N 1 \\\n  -J \u003CYOUR_ACCOUNT>-debug \\\n  --no-container-mount-home --gpus-per-node=8 \\\n  -p interactive --pty bash\n```\n\n### Step 3: Set up environment inside container\n\n```bash\nexport GH_TOKEN=\u003CYOUR_GITHUB_TOKEN>\nwandb login \u003CYOUR_WANDB_KEY>\nexport HF_TOKEN=\u003CYOUR_HF_TOKEN>\nexport HF_HOME=\u003CSHARED_FS>\u002FHF_HOME\nexport UV_CACHE_DIR=\"\u003CSHARED_FS>\u002Fuv_cache\"\nexport NEMO_HOME=\"\u003CSHARED_FS>\u002Fcache\u002Fnemo\"\nuv sync\n```\n\nThen run commands with `uv run` (uses the synced virtualenv):\n\n```bash\nuv run python -m torch.distributed.run --nproc_per_node=8 \\\n  examples\u002Fconversion\u002Fhf_to_megatron_generate_text.py \\\n  --hf_model_path \u003Corg>\u002F\u003Cmodel> --prompt \"What is AI?\" --max_new_tokens 50 --ep 8\n```\n\n**Pitfalls with interactive allocation:**\n\n| Error | Cause | Fix |\n|---|---|---|\n| `Cannot find GPU specification` | Missing `--gpus-per-node` | Always include `--gpus-per-node=8` in both `salloc` and `srun` |\n| `invalid partition specified: pool0` | Wrong partition name | Use `interactive` for interactive, `batch` for sbatch. Check: `sinfo --summarize` |\n| `Invalid account or account\u002Fpartition combination` | Partition not available for account | Check combos: `sacctmgr -nP show assoc where user=$USER format=account,partition` |\n| `Unable to create step for job... Requested node configuration is not available` | `-w \u003Cnode>` conflicts with allocation | Remove `-w` flag — HF cache is on shared filesystem, accessible from any node |\n| `uv: command not found` inside container | Container doesn't have `uv` pre-installed | Use a container with `uv` pre-installed, or `pip install uv` |\n| `No space left on device` during `uv` or `pip` | Container's `\u002Froot\u002F.cache\u002F` is full | Redirect: `export UV_CACHE_DIR=\u003CSHARED_FS>\u002Fuv_cache` |\n| `ModuleNotFoundError: No module named 'megatron.core.activations'` | Container's pre-installed megatron-core conflicts with local `3rdparty\u002FMegatron-LM` | Install local: `pip install -e 3rdparty\u002FMegatron-LM --no-deps --no-build-isolation` |\n\n---\n\n## Debugging Multi-Node Failures\n\n### Quick Diagnosis\n\nCheck the log for these patterns (in order):\n\n```bash\n# 1. Find the actual error (filter noise)\ngrep -a 'Error\\|OOM\\|CUDA out of memory\\|FAILED\\|Killed' job.log \\\n  | grep -v 'UserWarning\\|AllocatorConfig\\|transformer_engine\\|frame\\|srun: error'\n\n# 2. Check which rank crashed first\ngrep -a 'Failures:' -A 20 job.log | head -25\n\n# 3. Check for NCCL timeout\ngrep -a 'ncclUniqueId\\|timeout\\|crash on rank 0' job.log | head -5\n```\n\n### Debugging Checklist\n\nWhen a multi-node job fails:\n\n1. **Check exit code**: 1 = Python error, 9 = OOM killed, 143 = SIGTERM (timeout or cascade)\n2. **Find first failure**: Which task\u002Fnode crashed first? Others get SIGTERM (143) as cascade\n3. **grep the actual error**: Filter out UserWarnings, NCCL frame dumps\n4. **Check rank 0 specifically**: Most save\u002Fexport errors happen on rank 0\n5. **Verify EP sizing**: For MoE models, ensure `num_experts \u002F EP` fits in GPU memory with headroom\n6. **Try interactive first**: Use `salloc -N 2 -p interactive` to iterate faster than sbatch queue\n\n### NCCL Timeout at `dist.barrier()` — \"crash on rank 0\"\n\n**Symptom**: All ranks on node 2+ show:\n```text\n[rank8] is setting up NCCL communicator and retrieving ncclUniqueId from [0]\n... wait timeout after 600000ms\nThis may indicate a possible application crash on rank 0\n```\n\n**Root causes** (check in order):\n\n| Cause | How to verify | Fix |\n|---|---|---|\n| `save_artifacts` hangs on rank 0 | Error is in `save_hf_weights` → `dist.barrier()` | Increase timeout: `init_process_group(\"nccl\", timeout=timedelta(minutes=60))` |\n| `ImportError` in custom model code | `grep ImportError job.log` | Catch `ImportError` in `save_artifacts` (see below) |\n| Rank 0 OOM during export | `grep 'OutOfMemory' job.log` | Increase EP or nodes |\n| Network issue between nodes | Error only on cross-node ranks | Check `sinfo`, try different nodes |\n\n**The `save_artifacts` problem**: When `trust_remote_code=True`, rank 0 runs `save_artifacts()` (downloads tokenizer, config, custom modeling code) while all other ranks skip directly to `dist.barrier()`. If `save_artifacts` is slow or crashes, other ranks timeout.\n\n**Fix for ImportError in save_artifacts** (`hf_pretrained\u002Fbase.py`):\n```python\n# Change:\nexcept OSError:\n    pass\n# To:\nexcept (OSError, ImportError):\n    pass\n```\n\n### OOM for MoE Models\n\n**Symptom**: `torch.OutOfMemoryError: CUDA out of memory` during model loading or forward pass.\n\n**Key insight**: TP does NOT reduce expert memory. Only EP splits experts across GPUs.\n\n**Sizing formula**:\n```text\nexperts_per_gpu = num_experts \u002F EP\nexpert_memory_gb ≈ experts_per_gpu * expert_params * 2 \u002F 1e9  (bf16)\ntotal_per_gpu ≈ expert_memory_gb + attention_memory_gb + kv_cache_gb\n```\n\n**MiniMax-M2 example** (256 experts, ~230GB fp8 → ~460GB bf16):\n\n| Config | Nodes | GPUs | Experts\u002FGPU | Result |\n|---|---|---|---|---|\n| TP=2, EP=4 | 1 | 8 | 64 | OOM (too many experts) |\n| TP=2, EP=8 | 2 | 16 | 32 | Works for roundtrip (weight-only), OOM for inference |\n| TP=1, EP=16 | 2 | 16 | 16 | Works for inference |\n| TP=2, EP=32 | 8 | 64 | 8 | Comfortable for training |\n\n**Rules of thumb**:\n- Roundtrip (weight-only): can use more experts per GPU (~60GB model params OK)\n- Inference (forward pass + KV cache): needs headroom (~40GB model params max)\n- Training (activations + optimizer): needs even more headroom (~30GB model params max)\n\n### `ModuleNotFoundError: No module named 'megatron.core.tensor_parallel'`\n\n**Cause**: Container's pre-installed megatron-core conflicts with local `3rdparty\u002FMegatron-LM`.\n\n**Fix**: Add `uv sync` before running:\n```bash\nCMD=\"if [ \\\"\\$SLURM_LOCALID\\\" -eq 0 ]; then uv sync; else sleep 10; fi && \"\nCMD=\"${CMD}uv run --no-sync python \u003Cscript> \u003Cargs>\"\n```\n\n### FP8 Weight Mismatch in Roundtrip\n\n**Symptom**: Roundtrip completes but shows ❌ for all expert weights and raises `ValueError: Weight mismatch detected`.\n\n**Cause**: Original HF weights are FP8, Megatron stores in BF16. Exported weights are BF16. Comparison against original FP8 exceeds `atol=1e-1`.\n\n**This is expected for FP8 models.** The conversion is correct; the comparison tolerance is insufficient for the FP8→BF16 precision gap.\n\n### `WORLD_SIZE` Not Set with srun\n\n**Symptom**: Script exits with \"must be launched with torchrun\".\n\n**Cause**: Scripts check `os.environ.get(\"WORLD_SIZE\")` which torchrun sets but srun doesn't.\n\n**Fix**: Also check `SLURM_NTASKS`:\n```python\nif os.environ.get(\"WORLD_SIZE\") is None and os.environ.get(\"SLURM_NTASKS\") is None:\n    sys.exit(1)\n```\n\nBridge's `common_utils.py` helpers (called by `initialize.py`) populate env vars from SLURM:\n```python\nif \"RANK\" not in os.environ:\n    os.environ[\"RANK\"] = str(get_rank_safe())          # uses SLURM_PROCID\nif \"WORLD_SIZE\" not in os.environ:\n    os.environ[\"WORLD_SIZE\"] = str(get_world_size_safe())  # uses SLURM_NTASKS\nif \"MASTER_ADDR\" not in os.environ:\n    os.environ[\"MASTER_ADDR\"] = get_master_addr_safe()     # parses SLURM_NODELIST\nif \"MASTER_PORT\" not in os.environ:\n    os.environ[\"MASTER_PORT\"] = str(get_master_port_safe()) # derives from SLURM_JOB_ID\n```\n\n---\n\n## Key Gotchas\n\n1. **Two-phase srun for `uv sync`**: Run a single-process srun first to warm the cache, then the full multi-node srun. The second `uv sync` is a fast no-op since everything is already cached on the shared filesystem.\n\n2. **`--no-container-mount-home`** is an `srun` flag, NOT an `#SBATCH` directive.\n\n3. **Escaping inside TRAIN_CMD**: Since `TRAIN_CMD` is a double-quoted string, escape inner `$` for Slurm variables that must expand at runtime (not sbatch time):\n   - `\\${SLURM_PROCID}`, `\\${SLURM_JOB_NUM_NODES}`, `\\${SLURM_NODEID}`\n   - Host-side variables like `$GH_TOKEN`, `$LOGDIR`, `$WORKDIR` expand at sbatch time — no escaping needed.\n\n4. **Bridge `rm -rf nemo_experiments`**: Add before training to avoid stale checkpoint auto-resume.\n\n5. **MLM needs PYTHONPATH**: For pretrain_gpt.py scripts, add inside TRAIN_CMD:\n   ```bash\n   PYTHONPATH=${WORKDIR}\u002F3rdparty\u002FMegatron-LM:\\${PYTHONPATH:-} \\\n   ```\n\n6. **Node count heuristic**: Total GPUs = `NNODES * 8`. Must satisfy: `TP * PP * EP * DP >= total_GPUs` where `DP = total_GPUs \u002F (TP * PP * EP)`.\n\n7. **`NEMO_HOME` on shared filesystem for multi-node SFT**: The default nemo cache (`\u002Froot\u002F.cache\u002Fnemo`)\n   is container-local. Multi-node SFT with packed sequences prepares `.npy` files on one node\n   that are invisible to others. Set `export NEMO_HOME=\u003CSHARED_FS>\u002Fcache\u002Fnemo` so packed data\n   is shared. Without this, ranks on other nodes fail with `TypeError: 'NoneType' object is not an iterator`.\n\n## Full Templates and Command Bodies\n\nFor copyable sbatch scaffolding and Bridge\u002FMLM-specific `TRAIN_CMD` bodies, read\n[references\u002Ftemplates.md](references\u002Ftemplates.md).\n","# Multi-Node Slurm\n\nConvert single-node `uv run python -m torch.distributed.run` commands into multi-node Slurm sbatch scripts with Enroot container support, and debug common multi-node failures.\n\n## First Answer Checklist\n\nWhen converting or debugging Bridge multi-node jobs, answer in this order:\n\n1. Prefer the **srun-native** launch shape for Bridge scripts that reach\n   `initialize.py`: `#SBATCH --ntasks-per-node=8` and a direct `srun ... uv run\n   python \u003Cscript> ...` launch. Do not wrap these jobs in\n   `python -m torch.distributed.run`.\n2. State that Bridge derives `RANK`, `WORLD_SIZE`, `LOCAL_RANK`,\n   `MASTER_ADDR`, and `MASTER_PORT` from SLURM variables during\n   `initialize.py` distributed init.\n3. Require shared paths and matching container mounts for the repo, data, logs,\n   `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME`.\n4. For NCCL timeout reports, do these first-log checks before speculating:\n   - grep for real errors while filtering warning\u002Fframe noise\n   - inspect `Failures:` to find the first failed rank and node\n   - grep for `ncclUniqueId`, `timeout`, or `crash on rank 0`\n\n## Two Approaches: srun-native vs uv run torch.distributed\n\n| Approach | `ntasks-per-node` | Process spawning | Best for |\n|---|---|---|---|\n| **srun-native** (preferred) | 8 | Slurm spawns 8 tasks\u002Fnode | Conversion, inference, Bridge scripts |\n| **uv run torch.distributed** (legacy) | 1 | `uv run python -m torch.distributed.run` spawns 8 procs\u002Fnode | MLM pretrain_gpt.py |\n\n**Prefer srun-native** — simpler, avoids shell escaping issues with TRAIN_CMD. Megatron Bridge auto-derives `RANK`, `WORLD_SIZE`, `LOCAL_RANK`, `MASTER_ADDR`, `MASTER_PORT` from SLURM env vars (`SLURM_PROCID`, `SLURM_NTASKS`, `SLURM_LOCALID`, `SLURM_NODELIST`) via `common_utils.py` helpers called during `initialize.py` distributed init, so you never need to set them manually.\n\n## Cluster Environment\n\nUse a shared filesystem for the repository, data, logs, `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME`. `NEMO_HOME` must not use the container-local default (`\u002Froot\u002F.cache\u002Fnemo`) for multi-node SFT\u002FPEFT jobs, because packed-sequence data prepared on node 0 must be visible to the other nodes.\n\nKeep credentials out of sbatch templates and logs. Provide `HF_TOKEN`, `GH_TOKEN`, and `WANDB_API_KEY` through the scheduler environment or a restricted secrets file, and never hardcode token values in the script body. For copy-paste environment and sbatch templates, read `references\u002Ftemplates.md`.\n\n### Log Directory\n\n```text\n\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_\u003Csuffix>\n```\n\n## srun-native Approach (Preferred)\n\nSlurm spawns all processes directly. No `torch.distributed.run`, no TRAIN_CMD escaping.\n\n### SBATCH Headers\n\n```bash\n#SBATCH --job-name=\u003Cmodel>-\u003Ctask>\n#SBATCH --nodes=\u003CNNODES>\n#SBATCH --ntasks-per-node=8          # Slurm spawns 8 tasks per node\n#SBATCH --gpus-per-node=8\n#SBATCH --time=00:30:00\n#SBATCH --account=\u003CYOUR_ACCOUNT>\n#SBATCH --partition=batch\n#SBATCH --output=\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_%j.log\n#SBATCH --exclusive\n```\n\n### Build and Launch\n\nUse a two-phase `srun` pattern: first run a single-process `uv sync` to populate the shared cache, then launch the full multi-node job. The full copy-paste version lives in `references\u002Ftemplates.md`.\n\n### srun-native Key Points\n\n- Phase 1 runs `uv sync` once on a single node\u002Fprocess, building all wheels into the shared cache on Lustre\n- Phase 2's `uv sync` is a fast no-op (everything is cached) — safe to run on all ranks without sleep guards\n- `initialize.py` + `common_utils.py` auto-set `RANK`, `WORLD_SIZE`, `LOCAL_RANK`, `MASTER_ADDR`, `MASTER_PORT` from SLURM env vars\n- Env vars like `HF_TOKEN`, `HF_HOME`, `UV_CACHE_DIR` exported at sbatch level are inherited by srun tasks\n- Reference: `examples\u002Fmodels\u002Fglm\u002Fglm_45v\u002Fslurm_sft.sh`, `examples\u002Fmodels\u002Fminimax\u002Fminimax_m2\u002Fslurm_conversion.sh`\n\n---\n\n## uv run torch.distributed Approach (Legacy)\n\nUse when the script requires `torch.distributed.run` (e.g., MLM pretrain_gpt.py) or when Bridge's `initialize.py` is not in the call path.\n\n### 1. Add SBATCH Headers\n\n```bash\n#SBATCH --job-name=\u003Cmodel>-\u003Cframework>\n#SBATCH --nodes=\u003CNNODES>\n#SBATCH --ntasks-per-node=1          # ALWAYS 1 — torchrun handles per-node spawning\n#SBATCH --gpus-per-node=8\n#SBATCH --time=00:30:00\n#SBATCH --account=\u003CYOUR_ACCOUNT>\n#SBATCH --partition=batch\n#SBATCH --output=\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_%j.log\n#SBATCH --exclusive\n```\n\n**Critical**: `--ntasks-per-node=1`, NOT 8. `uv run python -m torch.distributed.run --nproc_per_node=8` spawns 8 processes per node. Using `ntasks-per-node=8` causes EADDRINUSE port collisions (8 tasks x 8 procs = 64 per node).\n\n### 2. Convert to Multi-Node\n\nReplace single-node:\n\n```bash\nuv run python -m torch.distributed.run --nproc_per_node=8 \\\n  \u003Cscript> \u003Cargs>\n```\n\nWith multi-node (inside `TRAIN_CMD` string):\n\n```bash\nuv run python -m torch.distributed.run \\\n  --nproc_per_node=8 \\\n  --nnodes=\\${SLURM_JOB_NUM_NODES} \\\n  --node_rank=\\${SLURM_NODEID} \\\n  \u003Cscript> \u003Cargs>\n```\n\n`MASTER_ADDR` and `MASTER_PORT` are auto-derived from SLURM env vars by `initialize.py` \u002F `common_utils.py` — no need to set them.\n\n### 3. Wrap in TRAIN_CMD + two-phase srun\n\nUse the same two-phase pattern: first a single-process srun to warm the uv cache, then the full run.\n\nSet runtime variables inside the container, but do not inject token values into a long `bash -c` string. Export credentials through the scheduler or source a restricted secrets file before the job starts. Keep `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME` on shared storage.\n\n### 4. Launch (two-phase)\n\nUse the two-phase launch template in `references\u002Ftemplates.md`, keeping `#SBATCH --ntasks-per-node=1` for this legacy approach.\n\n### 5. (Optional) Add Loss Extraction Footer\n\n```bash\necho \"======================================\"\necho \"Done. Losses:\"\necho \"======================================\"\ngrep -E \"iteration\\s+\" \"$LOGDIR\u002F\u003Cprefix>_${SLURM_JOB_ID}.log\" | grep -iE \"lm loss|reduced_train_loss\" | head -25\n```\n\n---\n\n## Interactive GPU Allocation (`salloc` + `srun`)\n\nFor ad-hoc testing (inference, conversion debugging), always follow these 3 steps:\n\n### Step 1: Allocate the node\n\n```bash\nsalloc --account \u003CYOUR_ACCOUNT> -N 1 \\\n  -J \u003CYOUR_ACCOUNT>-debug \\\n  -p interactive --gpus-per-node=8 -t 240\n```\n\n### Step 2: Launch container shell\n\n```bash\nsrun --mpi=pmix --no-kill \\\n  --container-image $CONTAINER_IMAGE \\\n  --container-mounts $CONTAINER_MOUNTS \\\n  --account \u003CYOUR_ACCOUNT> -N 1 \\\n  -J \u003CYOUR_ACCOUNT>-debug \\\n  --no-container-mount-home --gpus-per-node=8 \\\n  -p interactive --pty bash\n```\n\n### Step 3: Set up environment inside container\n\n```bash\nexport GH_TOKEN=\u003CYOUR_GITHUB_TOKEN>\nwandb login \u003CYOUR_WANDB_KEY>\nexport HF_TOKEN=\u003CYOUR_HF_TOKEN>\nexport HF_HOME=\u003CSHARED_FS>\u002FHF_HOME\nexport UV_CACHE_DIR=\"\u003CSHARED_FS>\u002Fuv_cache\"\nexport NEMO_HOME=\"\u003CSHARED_FS>\u002Fcache\u002Fnemo\"\nuv sync\n```\n\nThen run commands with `uv run` (uses the synced virtualenv):\n\n```bash\nuv run python -m torch.distributed.run --nproc_per_node=8 \\\n  examples\u002Fconversion\u002Fhf_to_megatron_generate_text.py \\\n  --hf_model_path \u003Corg>\u002F\u003Cmodel> --prompt \"What is AI?\" --max_new_tokens 50 --ep 8\n```\n\n**Pitfalls with interactive allocation:**\n\n| Error | Cause | Fix |\n|---|---|---|\n| `Cannot find GPU specification` | Missing `--gpus-per-node` | Always include `--gpus-per-node=8` in both `salloc` and `srun` |\n| `invalid partition specified: pool0` | Wrong partition name | Use `interactive` for interactive, `batch` for sbatch. Check: `sinfo --summarize` |\n| `Invalid account or account\u002Fpartition combination` | Partition not available for account | Check combos: `sacctmgr -nP show assoc where user=$USER format=account,partition` |\n| `Unable to create step for job... Requested node configuration is not available` | `-w \u003Cnode>` conflicts with allocation | Remove `-w` flag — HF cache is on shared filesystem, accessible from any node |\n| `uv: command not found` inside container | Container doesn't have `uv` pre-installed | Use a container with `uv` pre-installed, or `pip install uv` |\n| `No space left on device` during `uv` or `pip` | Container's `\u002Froot\u002F.cache\u002F` is full | Redirect: `export UV_CACHE_DIR=\u003CSHARED_FS>\u002Fuv_cache` |\n| `ModuleNotFoundError: No module named 'megatron.core.activations'` | Container's pre-installed megatron-core conflicts with local `3rdparty\u002FMegatron-LM` | Install local: `pip install -e 3rdparty\u002FMegatron-LM --no-deps --no-build-isolation` |\n\n---\n\n## Debugging Multi-Node Failures\n\n### Quick Diagnosis\n\nCheck the log for these patterns (in order):\n\n```bash\n# 1. Find the actual error (filter noise)\ngrep -a 'Error\\|OOM\\|CUDA out of memory\\|FAILED\\|Killed' job.log \\\n  | grep -v 'UserWarning\\|AllocatorConfig\\|transformer_engine\\|frame\\|srun: error'\n\n# 2. Check which rank crashed first\ngrep -a 'Failures:' -A 20 job.log | head -25\n\n# 3. Check for NCCL timeout\ngrep -a 'ncclUniqueId\\|timeout\\|crash on rank 0' job.log | head -5\n```\n\n### Debugging Checklist\n\nWhen a multi-node job fails:\n\n1. **Check exit code**: 1 = Python error, 9 = OOM killed, 143 = SIGTERM (timeout or cascade)\n2. **Find first failure**: Which task\u002Fnode crashed first? Others get SIGTERM (143) as cascade\n3. **grep the actual error**: Filter out UserWarnings, NCCL frame dumps\n4. **Check rank 0 specifically**: Most save\u002Fexport errors happen on rank 0\n5. **Verify EP sizing**: For MoE models, ensure `num_experts \u002F EP` fits in GPU memory with headroom\n6. **Try interactive first**: Use `salloc -N 2 -p interactive` to iterate faster than sbatch queue\n\n### NCCL Timeout at `dist.barrier()` — \"crash on rank 0\"\n\n**Symptom**: All ranks on node 2+ show:\n```text\n[rank8] is setting up NCCL communicator and retrieving ncclUniqueId from [0]\n... wait timeout after 600000ms\nThis may indicate a possible application crash on rank 0\n```\n\n**Root causes** (check in order):\n\n| Cause | How to verify | Fix |\n|---|---|---|\n| `save_artifacts` hangs on rank 0 | Error is in `save_hf_weights` → `dist.barrier()` | Increase timeout: `init_process_group(\"nccl\", timeout=timedelta(minutes=60))` |\n| `ImportError` in custom model code | `grep ImportError job.log` | Catch `ImportError` in `save_artifacts` (see below) |\n| Rank 0 OOM during export | `grep 'OutOfMemory' job.log` | Increase EP or nodes |\n| Network issue between nodes | Error only on cross-node ranks | Check `sinfo`, try different nodes |\n\n**The `save_artifacts` problem**: When `trust_remote_code=True`, rank 0 runs `save_artifacts()` (downloads tokenizer, config, custom modeling code) while all other ranks skip directly to `dist.barrier()`. If `save_artifacts` is slow or crashes, other ranks timeout.\n\n**Fix for ImportError in save_artifacts** (`hf_pretrained\u002Fbase.py`):\n```python\n# Change:\nexcept OSError:\n    pass\n# To:\nexcept (OSError, ImportError):\n    pass\n```\n\n### OOM for MoE Models\n\n**Symptom**: `torch.OutOfMemoryError: CUDA out of memory` during model loading or forward pass.\n\n**Key insight**: TP does NOT reduce expert memory. Only EP splits experts across GPUs.\n\n**Sizing formula**:\n```text\nexperts_per_gpu = num_experts \u002F EP\nexpert_memory_gb ≈ experts_per_gpu * expert_params * 2 \u002F 1e9  (bf16)\ntotal_per_gpu ≈ expert_memory_gb + attention_memory_gb + kv_cache_gb\n```\n\n**MiniMax-M2 example** (256 experts, ~230GB fp8 → ~460GB bf16):\n\n| Config | Nodes | GPUs | Experts\u002FGPU | Result |\n|---|---|---|---|---|\n| TP=2, EP=4 | 1 | 8 | 64 | OOM (too many experts) |\n| TP=2, EP=8 | 2 | 16 | 32 | Works for roundtrip (weight-only), OOM for inference |\n| TP=1, EP=16 | 2 | 16 | 16 | Works for inference |\n| TP=2, EP=32 | 8 | 64 | 8 | Comfortable for training |\n\n**Rules of thumb**:\n- Roundtrip (weight-only): can use more experts per GPU (~60GB model params OK)\n- Inference (forward pass + KV cache): needs headroom (~40GB model params max)\n- Training (activations + optimizer): needs even more headroom (~30GB model params max)\n\n### `ModuleNotFoundError: No module named 'megatron.core.tensor_parallel'`\n\n**Cause**: Container's pre-installed megatron-core conflicts with local `3rdparty\u002FMegatron-LM`.\n\n**Fix**: Add `uv sync` before running:\n```bash\nCMD=\"if [ \\\"\\$SLURM_LOCALID\\\" -eq 0 ]; then uv sync; else sleep 10; fi && \"\nCMD=\"${CMD}uv run --no-sync python \u003Cscript> \u003Cargs>\"\n```\n\n### FP8 Weight Mismatch in Roundtrip\n\n**Symptom**: Roundtrip completes but shows ❌ for all expert weights and raises `ValueError: Weight mismatch detected`.\n\n**Cause**: Original HF weights are FP8, Megatron stores in BF16. Exported weights are BF16. Comparison against original FP8 exceeds `atol=1e-1`.\n\n**This is expected for FP8 models.** The conversion is correct; the comparison tolerance is insufficient for the FP8→BF16 precision gap.\n\n### `WORLD_SIZE` Not Set with srun\n\n**Symptom**: Script exits with \"must be launched with torchrun\".\n\n**Cause**: Scripts check `os.environ.get(\"WORLD_SIZE\")` which torchrun sets but srun doesn't.\n\n**Fix**: Also check `SLURM_NTASKS`:\n```python\nif os.environ.get(\"WORLD_SIZE\") is None and os.environ.get(\"SLURM_NTASKS\") is None:\n    sys.exit(1)\n```\n\nBridge's `common_utils.py` helpers (called by `initialize.py`) populate env vars from SLURM:\n```python\nif \"RANK\" not in os.environ:\n    os.environ[\"RANK\"] = str(get_rank_safe())          # uses SLURM_PROCID\nif \"WORLD_SIZE\" not in os.environ:\n    os.environ[\"WORLD_SIZE\"] = str(get_world_size_safe())  # uses SLURM_NTASKS\nif \"MASTER_ADDR\" not in os.environ:\n    os.environ[\"MASTER_ADDR\"] = get_master_addr_safe()     # parses SLURM_NODELIST\nif \"MASTER_PORT\" not in os.environ:\n    os.environ[\"MASTER_PORT\"] = str(get_master_port_safe()) # derives from SLURM_JOB_ID\n```\n\n---\n\n## Key Gotchas\n\n1. **Two-phase srun for `uv sync`**: Run a single-process srun first to warm the cache, then the full multi-node srun. The second `uv sync` is a fast no-op since everything is already cached on the shared filesystem.\n\n2. **`--no-container-mount-home`** is an `srun` flag, NOT an `#SBATCH` directive.\n\n3. **Escaping inside TRAIN_CMD**: Since `TRAIN_CMD` is a double-quoted string, escape inner `$` for Slurm variables that must expand at runtime (not sbatch time):\n   - `\\${SLURM_PROCID}`, `\\${SLURM_JOB_NUM_NODES}`, `\\${SLURM_NODEID}`\n   - Host-side variables like `$GH_TOKEN`, `$LOGDIR`, `$WORKDIR` expand at sbatch time — no escaping needed.\n\n4. **Bridge `rm -rf nemo_experiments`**: Add before training to avoid stale checkpoint auto-resume.\n\n5. **MLM needs PYTHONPATH**: For pretrain_gpt.py scripts, add inside TRAIN_CMD:\n   ```bash\n   PYTHONPATH=${WORKDIR}\u002F3rdparty\u002FMegatron-LM:\\${PYTHONPATH:-} \\\n   ```\n\n6. **Node count heuristic**: Total GPUs = `NNODES * 8`. Must satisfy: `TP * PP * EP * DP >= total_GPUs` where `DP = total_GPUs \u002F (TP * PP * EP)`.\n\n7. **`NEMO_HOME` on shared filesystem for multi-node SFT**: The default nemo cache (`\u002Froot\u002F.cache\u002Fnemo`)\n   is container-local. Multi-node SFT with packed sequences prepares `.npy` files on one node\n   that are invisible to others. Set `export NEMO_HOME=\u003CSHARED_FS>\u002Fcache\u002Fnemo` so packed data\n   is shared. Without this, ranks on other nodes fail with `TypeError: 'NoneType' object is not an iterator`.\n\n## Full Templates and Command Bodies\n\nFor copyable sbatch scaffolding and Bridge\u002FMLM-specific `TRAIN_CMD` bodies, read\n[references\u002Ftemplates.md](references\u002Ftemplates.md).\n",{"name":5,"license":14,"description":7,"when_to_use":23},"Writing or converting Slurm sbatch scripts, scaling to multiple nodes, debugging NCCL\u002Flaunch failures, or investigating a commit that caused multi-node training failures; 'run on multiple nodes', 'sbatch script', 'NCCL timeout', 'multi-node OOM'.","main","fc4f2c7759ae400c66cd38656db70ead882a0d08","catalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-multi-node-slurm","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent#readme","https:\u002F\u002Fgithub.com\u002Fmatheussluzz","2026-08-27 05:21:51",{"source":8,"skillName":5,"commands":32,"prompt":42,"zipAvailable":43},[33,36,39],{"tool":34,"command":35},"npx","npx skills add matheussluzz\u002FZeuZ-Agent --skill nemo-mbridge-multi-node-slurm",{"tool":37,"command":38},"bunx","bunx skills add matheussluzz\u002FZeuZ-Agent --skill nemo-mbridge-multi-node-slurm",{"tool":40,"command":41},"pnpm","pnpm dlx skills add matheussluzz\u002FZeuZ-Agent --skill nemo-mbridge-multi-node-slurm","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-multi-node-slurm\u002FSKILL.md",true,[45,51,55,59,65],{"path":46,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":49,"sourceHtmlUrl":50},"BENCHMARK.md","REFERENCE","参考文档",4315,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-multi-node-slurm\u002FBENCHMARK.md",{"path":52,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":53,"sourceHtmlUrl":54},"references\u002Ftemplates.md",4744,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-multi-node-slurm\u002Freferences\u002Ftemplates.md",{"path":56,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":57,"sourceHtmlUrl":58},"skill-card.md",3738,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-multi-node-slurm\u002Fskill-card.md",{"path":60,"resourceType":61,"resourceTypeDesc":62,"sizeBytes":63,"sourceHtmlUrl":64},"skill.oms.sig","OTHER","其他",4841,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-multi-node-slurm\u002Fskill.oms.sig",{"path":66,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":67,"sourceHtmlUrl":68},"zeuz.manifest.yaml",172,"https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fcatalog\u002Fbundles\u002Fnvidia\u002Fnemo-mbridge-multi-node-slurm\u002Fzeuz.manifest.yaml",[70,78,85,92,99,106],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":77,"publishedAt":18},"01M78JQ2R0QR8H2Z6662RFX804","gamma","Fixture skill gamma for discovery benchmarks.","MIT","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Fskill-registry-mini\u002Fskills\u002Fgamma\u002FSKILL.md",[],[],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":18},"01M78JQ1KCVZ15APZG3FFF29K0","beta","Fixture skill beta for discovery benchmarks.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Fskill-registry-mini\u002Fskills\u002Fbeta\u002FSKILL.md",[],[],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":18},"01M78JQ0FJTK4Q9JKZMX9SVSEN","alpha","Fixture skill alpha for discovery benchmarks.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Ftest\u002Ffixtures\u002Fskill-registry-mini\u002Fskills\u002Falpha\u002FSKILL.md",[],[],{"skillNo":93,"name":94,"nameZh":6,"description":95,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":18},"01M78JPYVYPY7R8QS10KADQPWE","writing-for-agents","Writing documents for agents. Use when creating or editing skills, or modifying AGENTS.md or CLAUDE.md.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fskills\u002Fwriting-for-agents\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":18},"01M78JPVYHB88RF5ATZVGX84RR","wizard","Generate an interactive bash wizard that walks a human through steps only they can perform. Use when provisioning infrastructure, setting up credentials or CI secrets, walking an unfamiliar third-party dashboard, or running a one-off migration or cutover. Don't invoke this for steps the agent can perform itself.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fskills\u002Fwizard\u002FSKILL.md",[],[],{"skillNo":107,"name":108,"nameZh":6,"description":109,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":74,"licenseFileUrl":6,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":18},"01M78JPSSWNTZW89DWGJTX7716","wayfinder","Plan a huge chunk of work (more than one agent session can hold) as a shared map of decision tickets on your issue tracker, and resolve them one at a time until the way to the destination is clear.","https:\u002F\u002Fgithub.com\u002Fmatheussluzz\u002FZeuZ-Agent\u002Fblob\u002Fmain\u002Fskills\u002Fwayfinder\u002FSKILL.md",[],[]]