[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7FCT5WTY774NY6PFFXYV1TA":3,"skill-same:01M7FCT5WTY774NY6PFFXYV1TA":79},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":43,"commitSha":44,"skillDirectory":45,"forksCount":46,"repositoryHtmlUrl":47,"repositoryHomepageUrl":6,"ownerHtmlUrl":48,"ownerBlogUrl":6,"ownerDescription":49,"updatedAt":50,"install":51,"resources":64},"01M7FCT5WTY774NY6PFFXYV1TA","moe-training",null,"Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.","CZX-LBJ\u002Fdeepseek-science","CZX-LBJ","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F204594689?v=4","01M7FC59WQ9PFXT04MKGJDCFFT","czx","01M7FC59XB2GW0BKWN9JTDBQS7",1,"MIT","https:\u002F\u002Fgithub.com\u002FCZX-LBJ\u002Fdeepseek-science\u002Fblob\u002Fmain\u002Fpackages\u002Fskill\u002Fskill-science\u002Fassets\u002Fsources\u002Fai-research-skills\u002F19-emerging-techniques\u002Fmoe-training\u002FSKILL.md",[],[],"2026-11-16 10:05:34",false,"---\nname: moe-training\ndescription: Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Emerging Techniques, MoE, Mixture Of Experts, Sparse Models, DeepSpeed, Expert Parallelism, Mixtral, DeepSeek, Routing, Load Balancing, Efficient Training]\ndependencies: [deepspeed, transformers, torch, accelerate]\n---\n\n# MoE Training: Mixture of Experts\n\n## When to Use This Skill\n\nUse MoE Training when you need to:\n- **Train larger models** with limited compute (5× cost reduction vs dense models)\n- **Scale model capacity** without proportional compute increase\n- **Achieve better performance** per compute budget than dense models\n- **Specialize experts** for different domains\u002Ftasks\u002Flanguages\n- **Reduce inference latency** with sparse activation (only 13B\u002F47B params active in Mixtral)\n- **Implement SOTA models** like Mixtral 8x7B, DeepSeek-V3, Switch Transformers\n\n**Notable MoE Models**: Mixtral 8x7B (Mistral AI), DeepSeek-V3, Switch Transformers (Google), GLaM (Google), NLLB-MoE (Meta)\n\n## Installation\n\n```bash\n# DeepSpeed with MoE support\npip install deepspeed>=0.6.0\n\n# Megatron-DeepSpeed for large-scale training\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FMegatron-DeepSpeed\ncd Megatron-DeepSpeed\npip install -r requirements.txt\n\n# Alternative: HuggingFace Transformers\npip install transformers accelerate\n```\n\n## Quick Start\n\n### Basic MoE Architecture\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass MoELayer(nn.Module):\n    \"\"\"Sparse Mixture of Experts layer.\"\"\"\n\n    def __init__(self, hidden_size, num_experts=8, top_k=2):\n        super().__init__()\n        self.num_experts = num_experts\n        self.top_k = top_k\n\n        # Expert networks (FFN)\n        self.experts = nn.ModuleList([\n            nn.Sequential(\n                nn.Linear(hidden_size, 4 * hidden_size),\n                nn.GELU(),\n                nn.Linear(4 * hidden_size, hidden_size)\n            )\n            for _ in range(num_experts)\n        ])\n\n        # Gating network (router)\n        self.gate = nn.Linear(hidden_size, num_experts)\n\n    def forward(self, x):\n        # x shape: (batch_size, seq_len, hidden_size)\n        batch_size, seq_len, hidden_size = x.shape\n\n        # Flatten for routing\n        x_flat = x.view(-1, hidden_size)  # (batch_size * seq_len, hidden_size)\n\n        # Compute gate scores\n        gate_logits = self.gate(x_flat)  # (batch_size * seq_len, num_experts)\n\n        # Top-k routing\n        gate_scores = torch.softmax(gate_logits, dim=-1)\n        topk_scores, topk_indices = torch.topk(gate_scores, self.top_k, dim=-1)\n\n        # Normalize top-k scores\n        topk_scores = topk_scores \u002F topk_scores.sum(dim=-1, keepdim=True)\n\n        # Dispatch and combine expert outputs\n        output = torch.zeros_like(x_flat)\n\n        for i in range(self.top_k):\n            expert_idx = topk_indices[:, i]\n            expert_scores = topk_scores[:, i].unsqueeze(-1)\n\n            # Route tokens to experts\n            for expert_id in range(self.num_experts):\n                mask = (expert_idx == expert_id)\n                if mask.any():\n                    expert_input = x_flat[mask]\n                    expert_output = self.experts[expert_id](expert_input)\n                    output[mask] += expert_scores[mask] * expert_output\n\n        # Reshape back\n        return output.view(batch_size, seq_len, hidden_size)\n```\n\n### DeepSpeed MoE Training\n\n```bash\n# Training script with MoE\ndeepspeed pretrain_gpt_moe.py \\\n  --num-layers 24 \\\n  --hidden-size 1024 \\\n  --num-attention-heads 16 \\\n  --seq-length 2048 \\\n  --max-position-embeddings 2048 \\\n  --micro-batch-size 4 \\\n  --global-batch-size 256 \\\n  --train-iters 500000 \\\n  --lr 0.0001 \\\n  --min-lr 0.00001 \\\n  --lr-decay-style cosine \\\n  --num-experts 128 \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --moe-train-capacity-factor 1.25 \\\n  --moe-eval-capacity-factor 2.0 \\\n  --fp16 \\\n  --deepspeed_config ds_config.json\n```\n\n## Core Concepts\n\n### 1. MoE Architecture\n\n**Key Components:**\n- **Experts**: Multiple specialized FFN networks (typically 8-128)\n- **Router\u002FGate**: Learned network that selects which experts to use\n- **Top-k Routing**: Activate only k experts per token (k=1 or k=2)\n- **Load Balancing**: Ensure even expert utilization\n\n```\nInput Token\n    ↓\nRouter (Gate Network)\n    ↓\nTop-k Expert Selection (e.g., 2 out of 8)\n    ↓\nExpert 1 (weight: 0.6) + Expert 5 (weight: 0.4)\n    ↓\nWeighted Combination\n    ↓\nOutput\n```\n\n### 2. Routing Mechanisms\n\n**Top-1 Routing (Switch Transformer):**\n```python\n# Simplest routing: one expert per token\ngate_logits = router(x)  # (batch, seq_len, num_experts)\nexpert_idx = torch.argmax(gate_logits, dim=-1)  # Hard routing\n```\n\n**Top-2 Routing (Mixtral):**\n```python\n# Top-2: two experts per token\ngate_scores = torch.softmax(router(x), dim=-1)\ntop2_scores, top2_indices = torch.topk(gate_scores, k=2, dim=-1)\n\n# Normalize scores\ntop2_scores = top2_scores \u002F top2_scores.sum(dim=-1, keepdim=True)\n\n# Combine expert outputs\noutput = (top2_scores[:, :, 0:1] * expert_outputs[top2_indices[:, :, 0]] +\n          top2_scores[:, :, 1:2] * expert_outputs[top2_indices[:, :, 1]])\n```\n\n**Expert Choice Routing:**\n```python\n# Experts choose top-k tokens (instead of tokens choosing experts)\n# Guarantees perfect load balancing\nexpert_scores = router(x).transpose(-1, -2)  # (batch, num_experts, seq_len)\ntopk_tokens = torch.topk(expert_scores, k=capacity_per_expert, dim=-1)\n```\n\n### 3. Load Balancing\n\n**Auxiliary Loss:**\n```python\ndef load_balancing_loss(gate_logits, expert_indices, num_experts):\n    \"\"\"Encourage uniform expert usage.\"\"\"\n    # Fraction of tokens routed to each expert\n    expert_counts = torch.bincount(expert_indices.flatten(), minlength=num_experts)\n    expert_fraction = expert_counts.float() \u002F expert_indices.numel()\n\n    # Gate probability for each expert (average across tokens)\n    gate_probs = torch.softmax(gate_logits, dim=-1).mean(dim=0)\n\n    # Auxiliary loss: encourage alignment\n    aux_loss = num_experts * (expert_fraction * gate_probs).sum()\n\n    return aux_loss\n\n# Add to main loss\ntotal_loss = language_model_loss + 0.01 * load_balancing_loss(...)\n```\n\n**Router Z-Loss (Stability):**\n```python\ndef router_z_loss(logits):\n    \"\"\"Encourage router to have lower entropy (more decisive).\"\"\"\n    z_loss = torch.logsumexp(logits, dim=-1).pow(2).mean()\n    return z_loss\n\ntotal_loss = lm_loss + 0.01 * aux_loss + 0.001 * router_z_loss(gate_logits)\n```\n\n### 4. Expert Parallelism\n\n```python\n# DeepSpeed configuration\n{\n  \"train_batch_size\": 256,\n  \"fp16\": {\"enabled\": true},\n  \"moe\": {\n    \"enabled\": true,\n    \"num_experts\": 128,\n    \"expert_parallel_size\": 8,  # Distribute 128 experts across 8 GPUs\n    \"capacity_factor\": 1.25,    # Expert capacity = tokens_per_batch * capacity_factor \u002F num_experts\n    \"drop_tokens\": true,        # Drop tokens exceeding capacity\n    \"use_residual\": false\n  }\n}\n```\n\n## Training Configuration\n\n### DeepSpeed MoE Config\n\n```json\n{\n  \"train_batch_size\": 256,\n  \"gradient_accumulation_steps\": 1,\n  \"optimizer\": {\n    \"type\": \"Adam\",\n    \"params\": {\n      \"lr\": 0.0001,\n      \"betas\": [0.9, 0.999],\n      \"eps\": 1e-8\n    }\n  },\n  \"fp16\": {\n    \"enabled\": true,\n    \"loss_scale\": 0,\n    \"initial_scale_power\": 16\n  },\n  \"moe\": {\n    \"enabled\": true,\n    \"num_experts\": 128,\n    \"expert_parallel_size\": 8,\n    \"moe_loss_coeff\": 0.01,\n    \"train_capacity_factor\": 1.25,\n    \"eval_capacity_factor\": 2.0,\n    \"min_capacity\": 4,\n    \"drop_tokens\": true,\n    \"use_residual\": false,\n    \"use_tutel\": false\n  },\n  \"zero_optimization\": {\n    \"stage\": 1\n  }\n}\n```\n\n### Training Script\n\n```bash\n#!\u002Fbin\u002Fbash\n\n# Mixtral-style MoE training\ndeepspeed --num_gpus 8 pretrain_moe.py \\\n  --model-parallel-size 1 \\\n  --num-layers 32 \\\n  --hidden-size 4096 \\\n  --num-attention-heads 32 \\\n  --seq-length 2048 \\\n  --max-position-embeddings 4096 \\\n  --micro-batch-size 2 \\\n  --global-batch-size 256 \\\n  --train-iters 500000 \\\n  --save-interval 5000 \\\n  --eval-interval 1000 \\\n  --eval-iters 100 \\\n  --lr 0.0001 \\\n  --min-lr 0.00001 \\\n  --lr-decay-style cosine \\\n  --lr-warmup-iters 2000 \\\n  --clip-grad 1.0 \\\n  --weight-decay 0.1 \\\n  --num-experts 8 \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --moe-train-capacity-factor 1.25 \\\n  --moe-eval-capacity-factor 2.0 \\\n  --disable-moe-token-dropping \\\n  --fp16 \\\n  --deepspeed \\\n  --deepspeed_config ds_config_moe.json \\\n  --data-path \u002Fpath\u002Fto\u002Fdata \\\n  --vocab-file \u002Fpath\u002Fto\u002Fvocab.json \\\n  --merge-file \u002Fpath\u002Fto\u002Fmerges.txt\n```\n\n## Advanced Patterns\n\n### Mixtral 8x7B Architecture\n\n```python\nclass MixtralMoEBlock(nn.Module):\n    \"\"\"Mixtral-style MoE block with 8 experts, top-2 routing.\"\"\"\n\n    def __init__(self, config):\n        super().__init__()\n        self.hidden_dim = config.hidden_size\n        self.ffn_dim = config.intermediate_size\n        self.num_experts = config.num_local_experts  # 8\n        self.top_k = config.num_experts_per_tok       # 2\n\n        # 8 expert FFNs\n        self.experts = nn.ModuleList([\n            nn.Sequential(\n                nn.Linear(self.hidden_dim, self.ffn_dim, bias=False),\n                nn.SiLU(),\n                nn.Linear(self.ffn_dim, self.hidden_dim, bias=False)\n            )\n            for _ in range(self.num_experts)\n        ])\n\n        # Router\n        self.gate = nn.Linear(self.hidden_dim, self.num_experts, bias=False)\n\n    def forward(self, hidden_states):\n        batch_size, sequence_length, hidden_dim = hidden_states.shape\n\n        # Flatten\n        hidden_states = hidden_states.view(-1, hidden_dim)\n\n        # Router logits\n        router_logits = self.gate(hidden_states)  # (batch * seq_len, num_experts)\n\n        # Softmax and top-2\n        routing_weights = torch.softmax(router_logits, dim=1)\n        routing_weights, selected_experts = torch.topk(routing_weights, self.top_k, dim=-1)\n\n        # Normalize routing weights\n        routing_weights \u002F= routing_weights.sum(dim=-1, keepdim=True)\n\n        # Initialize output\n        final_hidden_states = torch.zeros_like(hidden_states)\n\n        # Route to experts\n        for expert_idx in range(self.num_experts):\n            expert_layer = self.experts[expert_idx]\n            idx, top_x = torch.where(selected_experts == expert_idx)\n\n            if idx.shape[0] == 0:\n                continue\n\n            # Current expert tokens\n            current_hidden_states = hidden_states[idx]\n\n            # Expert forward\n            current_hidden_states = expert_layer(current_hidden_states)\n\n            # Weighted by routing scores\n            current_hidden_states *= routing_weights[idx, top_x, None]\n\n            # Accumulate\n            final_hidden_states.index_add_(0, idx, current_hidden_states)\n\n        # Reshape\n        return final_hidden_states.view(batch_size, sequence_length, hidden_dim)\n```\n\n### PR-MoE (Pyramid-Residual-MoE)\n\n```bash\n# DeepSpeed PR-MoE: 3x better parameter efficiency\ndeepspeed pretrain_gpt_moe.py \\\n  --num-layers 24 \\\n  --hidden-size 1024 \\\n  --num-attention-heads 16 \\\n  --num-experts \"[128, 64, 32, 16]\" \\\n  --mlp-type residual \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --fp16\n```\n\n## Best Practices\n\n### 1. Expert Count Selection\n\n```python\n# Rule of thumb: More experts = more capacity, but diminishing returns\n# Typical configurations:\n# - Small models (1B-7B): 8-16 experts\n# - Medium models (7B-30B): 8-64 experts\n# - Large models (30B+): 64-256 experts\n\n# Example: Mixtral 8x7B\n# Total params: 47B (8 experts × 7B each)\n# Active params: 13B (2 experts × 7B, top-2 routing)\n# Efficiency: 47B capacity with 13B compute\n```\n\n### 2. Capacity Factor Tuning\n\n```python\n# Capacity = (tokens_per_batch \u002F num_experts) * capacity_factor\n\n# Training: Lower capacity (faster, drops some tokens)\ntrain_capacity_factor = 1.25  # 25% buffer\n\n# Evaluation: Higher capacity (no dropping)\neval_capacity_factor = 2.0    # 100% buffer\n\n# Formula:\nexpert_capacity = int((seq_len * batch_size \u002F num_experts) * capacity_factor)\n```\n\n### 3. Learning Rate Guidelines\n\n```python\n# MoE models need lower LR than dense models\n# - Dense model: lr = 6e-4\n# - MoE model: lr = 1e-4 (3-6× lower)\n\n# Also extend decay schedule\ndense_lr_decay_iters = 300000\nmoe_lr_decay_iters = 500000  # 1.5-2× longer\n```\n\n### 4. Loss Coefficient Tuning\n\n```python\n# Start with standard values\nmoe_loss_coeff = 0.01    # Auxiliary loss (load balancing)\nrouter_z_loss_coeff = 0.001  # Router entropy (stability)\n\n# If load imbalance persists, increase aux loss\nif max_expert_usage \u002F min_expert_usage > 2.0:\n    moe_loss_coeff = 0.1  # Stronger load balancing\n\n# If training unstable, increase z-loss\nif grad_norm > 10.0:\n    router_z_loss_coeff = 0.01\n```\n\n### 5. Avoid Common Pitfalls\n\n```python\n# ❌ Bad: Using same LR as dense model\noptimizer = Adam(model.parameters(), lr=6e-4)\n\n# ✅ Good: Lower LR for MoE\noptimizer = Adam([\n    {'params': model.non_moe_params, 'lr': 6e-4},\n    {'params': model.moe_params, 'lr': 1e-4}\n])\n\n# ❌ Bad: No load balancing\nloss = lm_loss\n\n# ✅ Good: Add auxiliary loss\nloss = lm_loss + 0.01 * aux_loss + 0.001 * z_loss\n\n# ❌ Bad: Too many experts for small dataset\nnum_experts = 128  # Overfitting risk\n\n# ✅ Good: Match experts to data diversity\nnum_experts = 8  # Better for small datasets\n```\n\n## Inference Optimization\n\n### Sparse Inference\n\n```python\n# Only activate top-k experts (huge memory savings)\n@torch.no_grad()\ndef moe_inference(x, model, top_k=2):\n    \"\"\"Sparse MoE inference: only load k experts.\"\"\"\n    # Router\n    gate_logits = model.gate(x)\n    topk_scores, topk_indices = torch.topk(\n        torch.softmax(gate_logits, dim=-1),\n        k=top_k,\n        dim=-1\n    )\n\n    # Load and run only top-k experts\n    output = torch.zeros_like(x)\n    for i in range(top_k):\n        expert_idx = topk_indices[:, i]\n        # Load expert from disk\u002Foffload if needed\n        expert = model.load_expert(expert_idx)\n        output += topk_scores[:, i:i+1] * expert(x)\n\n    return output\n```\n\n## Resources\n\n- **DeepSpeed MoE Tutorial**: https:\u002F\u002Fwww.deepspeed.ai\u002Ftutorials\u002Fmixture-of-experts-nlg\u002F\n- **Mixtral Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.04088\n- **Switch Transformers**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2101.03961\n- **HuggingFace MoE Guide**: https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fmoe\n- **NVIDIA MoE Blog**: https:\u002F\u002Fdeveloper.nvidia.com\u002Fblog\u002Fapplying-mixture-of-experts-in-llm-architectures\u002F\n\n## See Also\n\n- `references\u002Farchitectures.md` - MoE model architectures (Mixtral, Switch, DeepSeek-V3)\n- `references\u002Ftraining.md` - Advanced training techniques and optimization\n- `references\u002Finference.md` - Production deployment and serving patterns\n\n\n","# MoE Training: Mixture of Experts\n\n## When to Use This Skill\n\nUse MoE Training when you need to:\n- **Train larger models** with limited compute (5× cost reduction vs dense models)\n- **Scale model capacity** without proportional compute increase\n- **Achieve better performance** per compute budget than dense models\n- **Specialize experts** for different domains\u002Ftasks\u002Flanguages\n- **Reduce inference latency** with sparse activation (only 13B\u002F47B params active in Mixtral)\n- **Implement SOTA models** like Mixtral 8x7B, DeepSeek-V3, Switch Transformers\n\n**Notable MoE Models**: Mixtral 8x7B (Mistral AI), DeepSeek-V3, Switch Transformers (Google), GLaM (Google), NLLB-MoE (Meta)\n\n## Installation\n\n```bash\n# DeepSpeed with MoE support\npip install deepspeed>=0.6.0\n\n# Megatron-DeepSpeed for large-scale training\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FMegatron-DeepSpeed\ncd Megatron-DeepSpeed\npip install -r requirements.txt\n\n# Alternative: HuggingFace Transformers\npip install transformers accelerate\n```\n\n## Quick Start\n\n### Basic MoE Architecture\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass MoELayer(nn.Module):\n    \"\"\"Sparse Mixture of Experts layer.\"\"\"\n\n    def __init__(self, hidden_size, num_experts=8, top_k=2):\n        super().__init__()\n        self.num_experts = num_experts\n        self.top_k = top_k\n\n        # Expert networks (FFN)\n        self.experts = nn.ModuleList([\n            nn.Sequential(\n                nn.Linear(hidden_size, 4 * hidden_size),\n                nn.GELU(),\n                nn.Linear(4 * hidden_size, hidden_size)\n            )\n            for _ in range(num_experts)\n        ])\n\n        # Gating network (router)\n        self.gate = nn.Linear(hidden_size, num_experts)\n\n    def forward(self, x):\n        # x shape: (batch_size, seq_len, hidden_size)\n        batch_size, seq_len, hidden_size = x.shape\n\n        # Flatten for routing\n        x_flat = x.view(-1, hidden_size)  # (batch_size * seq_len, hidden_size)\n\n        # Compute gate scores\n        gate_logits = self.gate(x_flat)  # (batch_size * seq_len, num_experts)\n\n        # Top-k routing\n        gate_scores = torch.softmax(gate_logits, dim=-1)\n        topk_scores, topk_indices = torch.topk(gate_scores, self.top_k, dim=-1)\n\n        # Normalize top-k scores\n        topk_scores = topk_scores \u002F topk_scores.sum(dim=-1, keepdim=True)\n\n        # Dispatch and combine expert outputs\n        output = torch.zeros_like(x_flat)\n\n        for i in range(self.top_k):\n            expert_idx = topk_indices[:, i]\n            expert_scores = topk_scores[:, i].unsqueeze(-1)\n\n            # Route tokens to experts\n            for expert_id in range(self.num_experts):\n                mask = (expert_idx == expert_id)\n                if mask.any():\n                    expert_input = x_flat[mask]\n                    expert_output = self.experts[expert_id](expert_input)\n                    output[mask] += expert_scores[mask] * expert_output\n\n        # Reshape back\n        return output.view(batch_size, seq_len, hidden_size)\n```\n\n### DeepSpeed MoE Training\n\n```bash\n# Training script with MoE\ndeepspeed pretrain_gpt_moe.py \\\n  --num-layers 24 \\\n  --hidden-size 1024 \\\n  --num-attention-heads 16 \\\n  --seq-length 2048 \\\n  --max-position-embeddings 2048 \\\n  --micro-batch-size 4 \\\n  --global-batch-size 256 \\\n  --train-iters 500000 \\\n  --lr 0.0001 \\\n  --min-lr 0.00001 \\\n  --lr-decay-style cosine \\\n  --num-experts 128 \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --moe-train-capacity-factor 1.25 \\\n  --moe-eval-capacity-factor 2.0 \\\n  --fp16 \\\n  --deepspeed_config ds_config.json\n```\n\n## Core Concepts\n\n### 1. MoE Architecture\n\n**Key Components:**\n- **Experts**: Multiple specialized FFN networks (typically 8-128)\n- **Router\u002FGate**: Learned network that selects which experts to use\n- **Top-k Routing**: Activate only k experts per token (k=1 or k=2)\n- **Load Balancing**: Ensure even expert utilization\n\n```\nInput Token\n    ↓\nRouter (Gate Network)\n    ↓\nTop-k Expert Selection (e.g., 2 out of 8)\n    ↓\nExpert 1 (weight: 0.6) + Expert 5 (weight: 0.4)\n    ↓\nWeighted Combination\n    ↓\nOutput\n```\n\n### 2. Routing Mechanisms\n\n**Top-1 Routing (Switch Transformer):**\n```python\n# Simplest routing: one expert per token\ngate_logits = router(x)  # (batch, seq_len, num_experts)\nexpert_idx = torch.argmax(gate_logits, dim=-1)  # Hard routing\n```\n\n**Top-2 Routing (Mixtral):**\n```python\n# Top-2: two experts per token\ngate_scores = torch.softmax(router(x), dim=-1)\ntop2_scores, top2_indices = torch.topk(gate_scores, k=2, dim=-1)\n\n# Normalize scores\ntop2_scores = top2_scores \u002F top2_scores.sum(dim=-1, keepdim=True)\n\n# Combine expert outputs\noutput = (top2_scores[:, :, 0:1] * expert_outputs[top2_indices[:, :, 0]] +\n          top2_scores[:, :, 1:2] * expert_outputs[top2_indices[:, :, 1]])\n```\n\n**Expert Choice Routing:**\n```python\n# Experts choose top-k tokens (instead of tokens choosing experts)\n# Guarantees perfect load balancing\nexpert_scores = router(x).transpose(-1, -2)  # (batch, num_experts, seq_len)\ntopk_tokens = torch.topk(expert_scores, k=capacity_per_expert, dim=-1)\n```\n\n### 3. Load Balancing\n\n**Auxiliary Loss:**\n```python\ndef load_balancing_loss(gate_logits, expert_indices, num_experts):\n    \"\"\"Encourage uniform expert usage.\"\"\"\n    # Fraction of tokens routed to each expert\n    expert_counts = torch.bincount(expert_indices.flatten(), minlength=num_experts)\n    expert_fraction = expert_counts.float() \u002F expert_indices.numel()\n\n    # Gate probability for each expert (average across tokens)\n    gate_probs = torch.softmax(gate_logits, dim=-1).mean(dim=0)\n\n    # Auxiliary loss: encourage alignment\n    aux_loss = num_experts * (expert_fraction * gate_probs).sum()\n\n    return aux_loss\n\n# Add to main loss\ntotal_loss = language_model_loss + 0.01 * load_balancing_loss(...)\n```\n\n**Router Z-Loss (Stability):**\n```python\ndef router_z_loss(logits):\n    \"\"\"Encourage router to have lower entropy (more decisive).\"\"\"\n    z_loss = torch.logsumexp(logits, dim=-1).pow(2).mean()\n    return z_loss\n\ntotal_loss = lm_loss + 0.01 * aux_loss + 0.001 * router_z_loss(gate_logits)\n```\n\n### 4. Expert Parallelism\n\n```python\n# DeepSpeed configuration\n{\n  \"train_batch_size\": 256,\n  \"fp16\": {\"enabled\": true},\n  \"moe\": {\n    \"enabled\": true,\n    \"num_experts\": 128,\n    \"expert_parallel_size\": 8,  # Distribute 128 experts across 8 GPUs\n    \"capacity_factor\": 1.25,    # Expert capacity = tokens_per_batch * capacity_factor \u002F num_experts\n    \"drop_tokens\": true,        # Drop tokens exceeding capacity\n    \"use_residual\": false\n  }\n}\n```\n\n## Training Configuration\n\n### DeepSpeed MoE Config\n\n```json\n{\n  \"train_batch_size\": 256,\n  \"gradient_accumulation_steps\": 1,\n  \"optimizer\": {\n    \"type\": \"Adam\",\n    \"params\": {\n      \"lr\": 0.0001,\n      \"betas\": [0.9, 0.999],\n      \"eps\": 1e-8\n    }\n  },\n  \"fp16\": {\n    \"enabled\": true,\n    \"loss_scale\": 0,\n    \"initial_scale_power\": 16\n  },\n  \"moe\": {\n    \"enabled\": true,\n    \"num_experts\": 128,\n    \"expert_parallel_size\": 8,\n    \"moe_loss_coeff\": 0.01,\n    \"train_capacity_factor\": 1.25,\n    \"eval_capacity_factor\": 2.0,\n    \"min_capacity\": 4,\n    \"drop_tokens\": true,\n    \"use_residual\": false,\n    \"use_tutel\": false\n  },\n  \"zero_optimization\": {\n    \"stage\": 1\n  }\n}\n```\n\n### Training Script\n\n```bash\n#!\u002Fbin\u002Fbash\n\n# Mixtral-style MoE training\ndeepspeed --num_gpus 8 pretrain_moe.py \\\n  --model-parallel-size 1 \\\n  --num-layers 32 \\\n  --hidden-size 4096 \\\n  --num-attention-heads 32 \\\n  --seq-length 2048 \\\n  --max-position-embeddings 4096 \\\n  --micro-batch-size 2 \\\n  --global-batch-size 256 \\\n  --train-iters 500000 \\\n  --save-interval 5000 \\\n  --eval-interval 1000 \\\n  --eval-iters 100 \\\n  --lr 0.0001 \\\n  --min-lr 0.00001 \\\n  --lr-decay-style cosine \\\n  --lr-warmup-iters 2000 \\\n  --clip-grad 1.0 \\\n  --weight-decay 0.1 \\\n  --num-experts 8 \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --moe-train-capacity-factor 1.25 \\\n  --moe-eval-capacity-factor 2.0 \\\n  --disable-moe-token-dropping \\\n  --fp16 \\\n  --deepspeed \\\n  --deepspeed_config ds_config_moe.json \\\n  --data-path \u002Fpath\u002Fto\u002Fdata \\\n  --vocab-file \u002Fpath\u002Fto\u002Fvocab.json \\\n  --merge-file \u002Fpath\u002Fto\u002Fmerges.txt\n```\n\n## Advanced Patterns\n\n### Mixtral 8x7B Architecture\n\n```python\nclass MixtralMoEBlock(nn.Module):\n    \"\"\"Mixtral-style MoE block with 8 experts, top-2 routing.\"\"\"\n\n    def __init__(self, config):\n        super().__init__()\n        self.hidden_dim = config.hidden_size\n        self.ffn_dim = config.intermediate_size\n        self.num_experts = config.num_local_experts  # 8\n        self.top_k = config.num_experts_per_tok       # 2\n\n        # 8 expert FFNs\n        self.experts = nn.ModuleList([\n            nn.Sequential(\n                nn.Linear(self.hidden_dim, self.ffn_dim, bias=False),\n                nn.SiLU(),\n                nn.Linear(self.ffn_dim, self.hidden_dim, bias=False)\n            )\n            for _ in range(self.num_experts)\n        ])\n\n        # Router\n        self.gate = nn.Linear(self.hidden_dim, self.num_experts, bias=False)\n\n    def forward(self, hidden_states):\n        batch_size, sequence_length, hidden_dim = hidden_states.shape\n\n        # Flatten\n        hidden_states = hidden_states.view(-1, hidden_dim)\n\n        # Router logits\n        router_logits = self.gate(hidden_states)  # (batch * seq_len, num_experts)\n\n        # Softmax and top-2\n        routing_weights = torch.softmax(router_logits, dim=1)\n        routing_weights, selected_experts = torch.topk(routing_weights, self.top_k, dim=-1)\n\n        # Normalize routing weights\n        routing_weights \u002F= routing_weights.sum(dim=-1, keepdim=True)\n\n        # Initialize output\n        final_hidden_states = torch.zeros_like(hidden_states)\n\n        # Route to experts\n        for expert_idx in range(self.num_experts):\n            expert_layer = self.experts[expert_idx]\n            idx, top_x = torch.where(selected_experts == expert_idx)\n\n            if idx.shape[0] == 0:\n                continue\n\n            # Current expert tokens\n            current_hidden_states = hidden_states[idx]\n\n            # Expert forward\n            current_hidden_states = expert_layer(current_hidden_states)\n\n            # Weighted by routing scores\n            current_hidden_states *= routing_weights[idx, top_x, None]\n\n            # Accumulate\n            final_hidden_states.index_add_(0, idx, current_hidden_states)\n\n        # Reshape\n        return final_hidden_states.view(batch_size, sequence_length, hidden_dim)\n```\n\n### PR-MoE (Pyramid-Residual-MoE)\n\n```bash\n# DeepSpeed PR-MoE: 3x better parameter efficiency\ndeepspeed pretrain_gpt_moe.py \\\n  --num-layers 24 \\\n  --hidden-size 1024 \\\n  --num-attention-heads 16 \\\n  --num-experts \"[128, 64, 32, 16]\" \\\n  --mlp-type residual \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --fp16\n```\n\n## Best Practices\n\n### 1. Expert Count Selection\n\n```python\n# Rule of thumb: More experts = more capacity, but diminishing returns\n# Typical configurations:\n# - Small models (1B-7B): 8-16 experts\n# - Medium models (7B-30B): 8-64 experts\n# - Large models (30B+): 64-256 experts\n\n# Example: Mixtral 8x7B\n# Total params: 47B (8 experts × 7B each)\n# Active params: 13B (2 experts × 7B, top-2 routing)\n# Efficiency: 47B capacity with 13B compute\n```\n\n### 2. Capacity Factor Tuning\n\n```python\n# Capacity = (tokens_per_batch \u002F num_experts) * capacity_factor\n\n# Training: Lower capacity (faster, drops some tokens)\ntrain_capacity_factor = 1.25  # 25% buffer\n\n# Evaluation: Higher capacity (no dropping)\neval_capacity_factor = 2.0    # 100% buffer\n\n# Formula:\nexpert_capacity = int((seq_len * batch_size \u002F num_experts) * capacity_factor)\n```\n\n### 3. Learning Rate Guidelines\n\n```python\n# MoE models need lower LR than dense models\n# - Dense model: lr = 6e-4\n# - MoE model: lr = 1e-4 (3-6× lower)\n\n# Also extend decay schedule\ndense_lr_decay_iters = 300000\nmoe_lr_decay_iters = 500000  # 1.5-2× longer\n```\n\n### 4. Loss Coefficient Tuning\n\n```python\n# Start with standard values\nmoe_loss_coeff = 0.01    # Auxiliary loss (load balancing)\nrouter_z_loss_coeff = 0.001  # Router entropy (stability)\n\n# If load imbalance persists, increase aux loss\nif max_expert_usage \u002F min_expert_usage > 2.0:\n    moe_loss_coeff = 0.1  # Stronger load balancing\n\n# If training unstable, increase z-loss\nif grad_norm > 10.0:\n    router_z_loss_coeff = 0.01\n```\n\n### 5. Avoid Common Pitfalls\n\n```python\n# ❌ Bad: Using same LR as dense model\noptimizer = Adam(model.parameters(), lr=6e-4)\n\n# ✅ Good: Lower LR for MoE\noptimizer = Adam([\n    {'params': model.non_moe_params, 'lr': 6e-4},\n    {'params': model.moe_params, 'lr': 1e-4}\n])\n\n# ❌ Bad: No load balancing\nloss = lm_loss\n\n# ✅ Good: Add auxiliary loss\nloss = lm_loss + 0.01 * aux_loss + 0.001 * z_loss\n\n# ❌ Bad: Too many experts for small dataset\nnum_experts = 128  # Overfitting risk\n\n# ✅ Good: Match experts to data diversity\nnum_experts = 8  # Better for small datasets\n```\n\n## Inference Optimization\n\n### Sparse Inference\n\n```python\n# Only activate top-k experts (huge memory savings)\n@torch.no_grad()\ndef moe_inference(x, model, top_k=2):\n    \"\"\"Sparse MoE inference: only load k experts.\"\"\"\n    # Router\n    gate_logits = model.gate(x)\n    topk_scores, topk_indices = torch.topk(\n        torch.softmax(gate_logits, dim=-1),\n        k=top_k,\n        dim=-1\n    )\n\n    # Load and run only top-k experts\n    output = torch.zeros_like(x)\n    for i in range(top_k):\n        expert_idx = topk_indices[:, i]\n        # Load expert from disk\u002Foffload if needed\n        expert = model.load_expert(expert_idx)\n        output += topk_scores[:, i:i+1] * expert(x)\n\n    return output\n```\n\n## Resources\n\n- **DeepSpeed MoE Tutorial**: https:\u002F\u002Fwww.deepspeed.ai\u002Ftutorials\u002Fmixture-of-experts-nlg\u002F\n- **Mixtral Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.04088\n- **Switch Transformers**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2101.03961\n- **HuggingFace MoE Guide**: https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fmoe\n- **NVIDIA MoE Blog**: https:\u002F\u002Fdeveloper.nvidia.com\u002Fblog\u002Fapplying-mixture-of-experts-in-llm-architectures\u002F\n\n## See Also\n\n- `references\u002Farchitectures.md` - MoE model architectures (Mixtral, Switch, DeepSeek-V3)\n- `references\u002Ftraining.md` - Advanced training techniques and optimization\n- `references\u002Finference.md` - Production deployment and serving patterns\n\n\n",{"name":5,"tags":24,"author":36,"license":15,"version":37,"description":7,"dependencies":38},[25,26,27,28,29,30,31,32,33,34,35],"Emerging Techniques","MoE","Mixture Of Experts","Sparse Models","DeepSpeed","Expert Parallelism","Mixtral","DeepSeek","Routing","Load Balancing","Efficient Training","Orchestra Research","1.0.0",[39,40,41,42],"deepspeed","transformers","torch","accelerate","main","18193d71ea3441d7c033dbf195b8a7c75ac96f75","packages\u002Fskill\u002Fskill-science\u002Fassets\u002Fsources\u002Fai-research-skills\u002F19-emerging-techniques\u002Fmoe-training",0,"https:\u002F\u002Fgithub.com\u002FCZX-LBJ\u002Fdeepseek-science","https:\u002F\u002Fgithub.com\u002FCZX-LBJ","一个没事喜欢随便写点代码的小菜鸡\r\n","2026-10-22 09:48:04",{"source":8,"skillName":5,"commands":52,"prompt":62,"zipAvailable":63},[53,56,59],{"tool":54,"command":55},"npx","npx skills add CZX-LBJ\u002Fdeepseek-science --skill moe-training",{"tool":57,"command":58},"bunx","bunx skills add CZX-LBJ\u002Fdeepseek-science --skill moe-training",{"tool":60,"command":61},"pnpm","pnpm dlx skills add CZX-LBJ\u002Fdeepseek-science --skill moe-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FCZX-LBJ\u002Fdeepseek-science\u002Fblob\u002Fmain\u002Fpackages\u002Fskill\u002Fskill-science\u002Fassets\u002Fsources\u002Fai-research-skills\u002F19-emerging-techniques\u002Fmoe-training\u002FSKILL.md",true,[65,71,75],{"path":66,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":69,"sourceHtmlUrl":70},"references\u002Farchitectures.md","REFERENCE","参考文档",14044,"https:\u002F\u002Fgithub.com\u002FCZX-LBJ\u002Fdeepseek-science\u002Fblob\u002Fmain\u002Fpackages\u002Fskill\u002Fskill-science\u002Fassets\u002Fsources\u002Fai-research-skills\u002F19-emerging-techniques\u002Fmoe-training\u002Freferences\u002Farchitectures.md",{"path":72,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":73,"sourceHtmlUrl":74},"references\u002Finference.md",8239,"https:\u002F\u002Fgithub.com\u002FCZX-LBJ\u002Fdeepseek-science\u002Fblob\u002Fmain\u002Fpackages\u002Fskill\u002Fskill-science\u002Fassets\u002Fsources\u002Fai-research-skills\u002F19-emerging-techniques\u002Fmoe-training\u002Freferences\u002Finference.md",{"path":76,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":77,"sourceHtmlUrl":78},"references\u002Ftraining.md",9707,"https:\u002F\u002Fgithub.com\u002FCZX-LBJ\u002Fdeepseek-science\u002Fblob\u002Fmain\u002Fpackages\u002Fskill\u002Fskill-science\u002Fassets\u002Fsources\u002Fai-research-skills\u002F19-emerging-techniques\u002Fmoe-training\u002Freferences\u002Ftraining.md",[80,87,94,101,108,115],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":19},"01M7FEEKSNMRMKW6JEK93PFDFY","bio-free-energy-calculations","Performs alchemical free-energy calculations including relative binding free energy (RBFE \u002F FEP+) and absolute binding free energy (ABFE) via OpenFE, FEP+, GROMACS, AMBER pmemd, and OpenMM with explicit lambda scheduling, soft-core potentials, MBAR\u002FBAR analysis, cycle-closure validation, and protocol-appropriate enhanced sampling. Compares ML alternatives (Boltz-2 affinity, DeepDock). Use when ranking analogs by binding affinity beyond docking accuracy, performing prospective lead optimization, or validating SAR predictions.","https:\u002F\u002Fgithub.com\u002FCZX-LBJ\u002Fdeepseek-science\u002Fblob\u002Fmain\u002Fpackages\u002Fskill\u002Fskill-science\u002Fassets\u002Fsources\u002Fbioskills\u002Fchemoinformatics\u002Ffree-energy-calculations\u002FSKILL.md",[],[],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":19},"01M7FEE5CTE5QK55CEMXNNVWY9","bio-conformer-generation","Generates 3D conformer ensembles using RDKit ETKDGv3 with knowledge-enhanced distance geometry, MMFF94\u002FUFF force-field optimization, CREST + GFN2-xTB semi-empirical refinement, and macrocycle-aware torsion preferences. Provides explicit decision rules for single vs ensemble conformer use, RMSD pruning, energy windows, conformer count, and force-field choice. Use when preparing 3D ligands for docking, generating descriptor input for 3D QSAR, or sampling macrocycle\u002Fpeptide conformational ensembles.","https:\u002F\u002Fgithub.com\u002FCZX-LBJ\u002Fdeepseek-science\u002Fblob\u002Fmain\u002Fpackages\u002Fskill\u002Fskill-science\u002Fassets\u002Fsources\u002Fbioskills\u002Fchemoinformatics\u002Fconformer-generation\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":19},"01M7FEDNGFEMFWD1MSEDR5H7CY","bio-admet-prediction","Predicts ADMET properties using ADMETlab 3.0 (119 platform features, including 77 prediction models with modeled-endpoint uncertainty), ADMET-AI, DeepChem MolNet, and chemprop D-MPNN with explicit handling of OECD QSAR principles, applicability domain assessment, calibration, hERG\u002FCYP\u002FAMES endpoints, and PAINS \u002F Lipinski \u002F Ro5 \u002F Veber \u002F BBB druglikeness filters. Use when filtering compounds for drug-likeness, prioritizing leads by predicted safety, or building an in-house ADMET QSAR model.","https:\u002F\u002Fgithub.com\u002FCZX-LBJ\u002Fdeepseek-science\u002Fblob\u002Fmain\u002Fpackages\u002Fskill\u002Fskill-science\u002Fassets\u002Fsources\u002Fbioskills\u002Fchemoinformatics\u002Fadmet-prediction\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":19},"01M7FED69GFB54FRF7AV5066DN","writing-systems-papers","Paragraph-level structural blueprint for 10-12 page systems papers targeting OSDI, SOSP, ASPLOS, NSDI, and EuroSys. Provides page allocation, paragraph templates, and writing patterns. Use when user says \"写系统论文\", \"systems paper structure\", \"OSDI paper\", \"SOSP paper\", or wants fine-grained structural guidance for a systems conference submission.","https:\u002F\u002Fgithub.com\u002FCZX-LBJ\u002Fdeepseek-science\u002Fblob\u002Fmain\u002Fpackages\u002Fskill\u002Fskill-science\u002Fassets\u002Fsources\u002Faris\u002Fskills\u002Fwriting-systems-papers\u002FSKILL.md",[],[],{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":19},"01M7FECVVDWZ0X0VEDTCE4B4DZ","wiki-enrich","Fill in the per-paper TODO sections of research-wiki\u002Fpapers\u002F\u003Cslug>.md pages that literature-ingest skills leave as bare scaffolds. Use when user says 'enrich wiki', 'fill paper TODOs', 'wiki body 補完', '把 paper 摘要寫進 wiki', 'research-wiki 自動填', or after a batch ingest that left papers\u002F as TODO scaffolds.","https:\u002F\u002Fgithub.com\u002FCZX-LBJ\u002Fdeepseek-science\u002Fblob\u002Fmain\u002Fpackages\u002Fskill\u002Fskill-science\u002Fassets\u002Fsources\u002Faris\u002Fskills\u002Fwiki-enrich\u002FSKILL.md",[],[],{"skillNo":116,"name":117,"nameZh":6,"description":118,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":119,"categories":120,"keywords":121,"publishedAt":19},"01M7FECD4BCJ2MTSBDDSE3ZAJ5","web-debug-search","Search GitHub, Stack Exchange, Chinese technical communities, official documentation, and general developer web sources for software errors, compatibility problems, API usage questions, and real-world workarounds. Use for debugging and discovery only; results are not paper-citation evidence.","https:\u002F\u002Fgithub.com\u002FCZX-LBJ\u002Fdeepseek-science\u002Fblob\u002Fmain\u002Fpackages\u002Fskill\u002Fskill-science\u002Fassets\u002Fsources\u002Faris\u002Fskills\u002Fweb-debug-search\u002FSKILL.md",[],[]]