[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M73AHTFCCEXYA4XVEERD5ZT0":3,"skill-same:01M73AHTFCCEXYA4XVEERD5ZT0":79},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":43,"commitSha":44,"skillDirectory":45,"forksCount":46,"repositoryHtmlUrl":47,"repositoryHomepageUrl":6,"ownerHtmlUrl":48,"ownerBlogUrl":49,"ownerDescription":6,"updatedAt":50,"install":51,"resources":64},"01M73AHTFCCEXYA4XVEERD5ZT0","moe-training",null,"Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.","qhjqhj00\u002Fresearch-skills-pool","qhjqhj00","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F29684092?v=4","01M72AE0ZGXVM5C57N512DGQG8","Tommy Chien","01M72AE107ESJMHT7QC5SXX144",3,"MIT","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--19-emerging-techniques__moe-training\u002FSKILL.md",[],[],"2026-11-11 15:11:14",false,"---\nname: moe-training\ndescription: Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Emerging Techniques, MoE, Mixture Of Experts, Sparse Models, DeepSpeed, Expert Parallelism, Mixtral, DeepSeek, Routing, Load Balancing, Efficient Training]\ndependencies: [deepspeed, transformers, torch, accelerate]\n---\n\n# MoE Training: Mixture of Experts\n\n## When to Use This Skill\n\nUse MoE Training when you need to:\n- **Train larger models** with limited compute (5× cost reduction vs dense models)\n- **Scale model capacity** without proportional compute increase\n- **Achieve better performance** per compute budget than dense models\n- **Specialize experts** for different domains\u002Ftasks\u002Flanguages\n- **Reduce inference latency** with sparse activation (only 13B\u002F47B params active in Mixtral)\n- **Implement SOTA models** like Mixtral 8x7B, DeepSeek-V3, Switch Transformers\n\n**Notable MoE Models**: Mixtral 8x7B (Mistral AI), DeepSeek-V3, Switch Transformers (Google), GLaM (Google), NLLB-MoE (Meta)\n\n## Installation\n\n```bash\n# DeepSpeed with MoE support\npip install deepspeed>=0.6.0\n\n# Megatron-DeepSpeed for large-scale training\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FMegatron-DeepSpeed\ncd Megatron-DeepSpeed\npip install -r requirements.txt\n\n# Alternative: HuggingFace Transformers\npip install transformers accelerate\n```\n\n## Quick Start\n\n### Basic MoE Architecture\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass MoELayer(nn.Module):\n    \"\"\"Sparse Mixture of Experts layer.\"\"\"\n\n    def __init__(self, hidden_size, num_experts=8, top_k=2):\n        super().__init__()\n        self.num_experts = num_experts\n        self.top_k = top_k\n\n        # Expert networks (FFN)\n        self.experts = nn.ModuleList([\n            nn.Sequential(\n                nn.Linear(hidden_size, 4 * hidden_size),\n                nn.GELU(),\n                nn.Linear(4 * hidden_size, hidden_size)\n            )\n            for _ in range(num_experts)\n        ])\n\n        # Gating network (router)\n        self.gate = nn.Linear(hidden_size, num_experts)\n\n    def forward(self, x):\n        # x shape: (batch_size, seq_len, hidden_size)\n        batch_size, seq_len, hidden_size = x.shape\n\n        # Flatten for routing\n        x_flat = x.view(-1, hidden_size)  # (batch_size * seq_len, hidden_size)\n\n        # Compute gate scores\n        gate_logits = self.gate(x_flat)  # (batch_size * seq_len, num_experts)\n\n        # Top-k routing\n        gate_scores = torch.softmax(gate_logits, dim=-1)\n        topk_scores, topk_indices = torch.topk(gate_scores, self.top_k, dim=-1)\n\n        # Normalize top-k scores\n        topk_scores = topk_scores \u002F topk_scores.sum(dim=-1, keepdim=True)\n\n        # Dispatch and combine expert outputs\n        output = torch.zeros_like(x_flat)\n\n        for i in range(self.top_k):\n            expert_idx = topk_indices[:, i]\n            expert_scores = topk_scores[:, i].unsqueeze(-1)\n\n            # Route tokens to experts\n            for expert_id in range(self.num_experts):\n                mask = (expert_idx == expert_id)\n                if mask.any():\n                    expert_input = x_flat[mask]\n                    expert_output = self.experts[expert_id](expert_input)\n                    output[mask] += expert_scores[mask] * expert_output\n\n        # Reshape back\n        return output.view(batch_size, seq_len, hidden_size)\n```\n\n### DeepSpeed MoE Training\n\n```bash\n# Training script with MoE\ndeepspeed pretrain_gpt_moe.py \\\n  --num-layers 24 \\\n  --hidden-size 1024 \\\n  --num-attention-heads 16 \\\n  --seq-length 2048 \\\n  --max-position-embeddings 2048 \\\n  --micro-batch-size 4 \\\n  --global-batch-size 256 \\\n  --train-iters 500000 \\\n  --lr 0.0001 \\\n  --min-lr 0.00001 \\\n  --lr-decay-style cosine \\\n  --num-experts 128 \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --moe-train-capacity-factor 1.25 \\\n  --moe-eval-capacity-factor 2.0 \\\n  --fp16 \\\n  --deepspeed_config ds_config.json\n```\n\n## Core Concepts\n\n### 1. MoE Architecture\n\n**Key Components:**\n- **Experts**: Multiple specialized FFN networks (typically 8-128)\n- **Router\u002FGate**: Learned network that selects which experts to use\n- **Top-k Routing**: Activate only k experts per token (k=1 or k=2)\n- **Load Balancing**: Ensure even expert utilization\n\n```\nInput Token\n    ↓\nRouter (Gate Network)\n    ↓\nTop-k Expert Selection (e.g., 2 out of 8)\n    ↓\nExpert 1 (weight: 0.6) + Expert 5 (weight: 0.4)\n    ↓\nWeighted Combination\n    ↓\nOutput\n```\n\n### 2. Routing Mechanisms\n\n**Top-1 Routing (Switch Transformer):**\n```python\n# Simplest routing: one expert per token\ngate_logits = router(x)  # (batch, seq_len, num_experts)\nexpert_idx = torch.argmax(gate_logits, dim=-1)  # Hard routing\n```\n\n**Top-2 Routing (Mixtral):**\n```python\n# Top-2: two experts per token\ngate_scores = torch.softmax(router(x), dim=-1)\ntop2_scores, top2_indices = torch.topk(gate_scores, k=2, dim=-1)\n\n# Normalize scores\ntop2_scores = top2_scores \u002F top2_scores.sum(dim=-1, keepdim=True)\n\n# Combine expert outputs\noutput = (top2_scores[:, :, 0:1] * expert_outputs[top2_indices[:, :, 0]] +\n          top2_scores[:, :, 1:2] * expert_outputs[top2_indices[:, :, 1]])\n```\n\n**Expert Choice Routing:**\n```python\n# Experts choose top-k tokens (instead of tokens choosing experts)\n# Guarantees perfect load balancing\nexpert_scores = router(x).transpose(-1, -2)  # (batch, num_experts, seq_len)\ntopk_tokens = torch.topk(expert_scores, k=capacity_per_expert, dim=-1)\n```\n\n### 3. Load Balancing\n\n**Auxiliary Loss:**\n```python\ndef load_balancing_loss(gate_logits, expert_indices, num_experts):\n    \"\"\"Encourage uniform expert usage.\"\"\"\n    # Fraction of tokens routed to each expert\n    expert_counts = torch.bincount(expert_indices.flatten(), minlength=num_experts)\n    expert_fraction = expert_counts.float() \u002F expert_indices.numel()\n\n    # Gate probability for each expert (average across tokens)\n    gate_probs = torch.softmax(gate_logits, dim=-1).mean(dim=0)\n\n    # Auxiliary loss: encourage alignment\n    aux_loss = num_experts * (expert_fraction * gate_probs).sum()\n\n    return aux_loss\n\n# Add to main loss\ntotal_loss = language_model_loss + 0.01 * load_balancing_loss(...)\n```\n\n**Router Z-Loss (Stability):**\n```python\ndef router_z_loss(logits):\n    \"\"\"Encourage router to have lower entropy (more decisive).\"\"\"\n    z_loss = torch.logsumexp(logits, dim=-1).pow(2).mean()\n    return z_loss\n\ntotal_loss = lm_loss + 0.01 * aux_loss + 0.001 * router_z_loss(gate_logits)\n```\n\n### 4. Expert Parallelism\n\n```python\n# DeepSpeed configuration\n{\n  \"train_batch_size\": 256,\n  \"fp16\": {\"enabled\": true},\n  \"moe\": {\n    \"enabled\": true,\n    \"num_experts\": 128,\n    \"expert_parallel_size\": 8,  # Distribute 128 experts across 8 GPUs\n    \"capacity_factor\": 1.25,    # Expert capacity = tokens_per_batch * capacity_factor \u002F num_experts\n    \"drop_tokens\": true,        # Drop tokens exceeding capacity\n    \"use_residual\": false\n  }\n}\n```\n\n## Training Configuration\n\n### DeepSpeed MoE Config\n\n```json\n{\n  \"train_batch_size\": 256,\n  \"gradient_accumulation_steps\": 1,\n  \"optimizer\": {\n    \"type\": \"Adam\",\n    \"params\": {\n      \"lr\": 0.0001,\n      \"betas\": [0.9, 0.999],\n      \"eps\": 1e-8\n    }\n  },\n  \"fp16\": {\n    \"enabled\": true,\n    \"loss_scale\": 0,\n    \"initial_scale_power\": 16\n  },\n  \"moe\": {\n    \"enabled\": true,\n    \"num_experts\": 128,\n    \"expert_parallel_size\": 8,\n    \"moe_loss_coeff\": 0.01,\n    \"train_capacity_factor\": 1.25,\n    \"eval_capacity_factor\": 2.0,\n    \"min_capacity\": 4,\n    \"drop_tokens\": true,\n    \"use_residual\": false,\n    \"use_tutel\": false\n  },\n  \"zero_optimization\": {\n    \"stage\": 1\n  }\n}\n```\n\n### Training Script\n\n```bash\n#!\u002Fbin\u002Fbash\n\n# Mixtral-style MoE training\ndeepspeed --num_gpus 8 pretrain_moe.py \\\n  --model-parallel-size 1 \\\n  --num-layers 32 \\\n  --hidden-size 4096 \\\n  --num-attention-heads 32 \\\n  --seq-length 2048 \\\n  --max-position-embeddings 4096 \\\n  --micro-batch-size 2 \\\n  --global-batch-size 256 \\\n  --train-iters 500000 \\\n  --save-interval 5000 \\\n  --eval-interval 1000 \\\n  --eval-iters 100 \\\n  --lr 0.0001 \\\n  --min-lr 0.00001 \\\n  --lr-decay-style cosine \\\n  --lr-warmup-iters 2000 \\\n  --clip-grad 1.0 \\\n  --weight-decay 0.1 \\\n  --num-experts 8 \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --moe-train-capacity-factor 1.25 \\\n  --moe-eval-capacity-factor 2.0 \\\n  --disable-moe-token-dropping \\\n  --fp16 \\\n  --deepspeed \\\n  --deepspeed_config ds_config_moe.json \\\n  --data-path \u002Fpath\u002Fto\u002Fdata \\\n  --vocab-file \u002Fpath\u002Fto\u002Fvocab.json \\\n  --merge-file \u002Fpath\u002Fto\u002Fmerges.txt\n```\n\n## Advanced Patterns\n\n### Mixtral 8x7B Architecture\n\n```python\nclass MixtralMoEBlock(nn.Module):\n    \"\"\"Mixtral-style MoE block with 8 experts, top-2 routing.\"\"\"\n\n    def __init__(self, config):\n        super().__init__()\n        self.hidden_dim = config.hidden_size\n        self.ffn_dim = config.intermediate_size\n        self.num_experts = config.num_local_experts  # 8\n        self.top_k = config.num_experts_per_tok       # 2\n\n        # 8 expert FFNs\n        self.experts = nn.ModuleList([\n            nn.Sequential(\n                nn.Linear(self.hidden_dim, self.ffn_dim, bias=False),\n                nn.SiLU(),\n                nn.Linear(self.ffn_dim, self.hidden_dim, bias=False)\n            )\n            for _ in range(self.num_experts)\n        ])\n\n        # Router\n        self.gate = nn.Linear(self.hidden_dim, self.num_experts, bias=False)\n\n    def forward(self, hidden_states):\n        batch_size, sequence_length, hidden_dim = hidden_states.shape\n\n        # Flatten\n        hidden_states = hidden_states.view(-1, hidden_dim)\n\n        # Router logits\n        router_logits = self.gate(hidden_states)  # (batch * seq_len, num_experts)\n\n        # Softmax and top-2\n        routing_weights = torch.softmax(router_logits, dim=1)\n        routing_weights, selected_experts = torch.topk(routing_weights, self.top_k, dim=-1)\n\n        # Normalize routing weights\n        routing_weights \u002F= routing_weights.sum(dim=-1, keepdim=True)\n\n        # Initialize output\n        final_hidden_states = torch.zeros_like(hidden_states)\n\n        # Route to experts\n        for expert_idx in range(self.num_experts):\n            expert_layer = self.experts[expert_idx]\n            idx, top_x = torch.where(selected_experts == expert_idx)\n\n            if idx.shape[0] == 0:\n                continue\n\n            # Current expert tokens\n            current_hidden_states = hidden_states[idx]\n\n            # Expert forward\n            current_hidden_states = expert_layer(current_hidden_states)\n\n            # Weighted by routing scores\n            current_hidden_states *= routing_weights[idx, top_x, None]\n\n            # Accumulate\n            final_hidden_states.index_add_(0, idx, current_hidden_states)\n\n        # Reshape\n        return final_hidden_states.view(batch_size, sequence_length, hidden_dim)\n```\n\n### PR-MoE (Pyramid-Residual-MoE)\n\n```bash\n# DeepSpeed PR-MoE: 3x better parameter efficiency\ndeepspeed pretrain_gpt_moe.py \\\n  --num-layers 24 \\\n  --hidden-size 1024 \\\n  --num-attention-heads 16 \\\n  --num-experts \"[128, 64, 32, 16]\" \\\n  --mlp-type residual \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --fp16\n```\n\n## Best Practices\n\n### 1. Expert Count Selection\n\n```python\n# Rule of thumb: More experts = more capacity, but diminishing returns\n# Typical configurations:\n# - Small models (1B-7B): 8-16 experts\n# - Medium models (7B-30B): 8-64 experts\n# - Large models (30B+): 64-256 experts\n\n# Example: Mixtral 8x7B\n# Total params: 47B (8 experts × 7B each)\n# Active params: 13B (2 experts × 7B, top-2 routing)\n# Efficiency: 47B capacity with 13B compute\n```\n\n### 2. Capacity Factor Tuning\n\n```python\n# Capacity = (tokens_per_batch \u002F num_experts) * capacity_factor\n\n# Training: Lower capacity (faster, drops some tokens)\ntrain_capacity_factor = 1.25  # 25% buffer\n\n# Evaluation: Higher capacity (no dropping)\neval_capacity_factor = 2.0    # 100% buffer\n\n# Formula:\nexpert_capacity = int((seq_len * batch_size \u002F num_experts) * capacity_factor)\n```\n\n### 3. Learning Rate Guidelines\n\n```python\n# MoE models need lower LR than dense models\n# - Dense model: lr = 6e-4\n# - MoE model: lr = 1e-4 (3-6× lower)\n\n# Also extend decay schedule\ndense_lr_decay_iters = 300000\nmoe_lr_decay_iters = 500000  # 1.5-2× longer\n```\n\n### 4. Loss Coefficient Tuning\n\n```python\n# Start with standard values\nmoe_loss_coeff = 0.01    # Auxiliary loss (load balancing)\nrouter_z_loss_coeff = 0.001  # Router entropy (stability)\n\n# If load imbalance persists, increase aux loss\nif max_expert_usage \u002F min_expert_usage > 2.0:\n    moe_loss_coeff = 0.1  # Stronger load balancing\n\n# If training unstable, increase z-loss\nif grad_norm > 10.0:\n    router_z_loss_coeff = 0.01\n```\n\n### 5. Avoid Common Pitfalls\n\n```python\n# ❌ Bad: Using same LR as dense model\noptimizer = Adam(model.parameters(), lr=6e-4)\n\n# ✅ Good: Lower LR for MoE\noptimizer = Adam([\n    {'params': model.non_moe_params, 'lr': 6e-4},\n    {'params': model.moe_params, 'lr': 1e-4}\n])\n\n# ❌ Bad: No load balancing\nloss = lm_loss\n\n# ✅ Good: Add auxiliary loss\nloss = lm_loss + 0.01 * aux_loss + 0.001 * z_loss\n\n# ❌ Bad: Too many experts for small dataset\nnum_experts = 128  # Overfitting risk\n\n# ✅ Good: Match experts to data diversity\nnum_experts = 8  # Better for small datasets\n```\n\n## Inference Optimization\n\n### Sparse Inference\n\n```python\n# Only activate top-k experts (huge memory savings)\n@torch.no_grad()\ndef moe_inference(x, model, top_k=2):\n    \"\"\"Sparse MoE inference: only load k experts.\"\"\"\n    # Router\n    gate_logits = model.gate(x)\n    topk_scores, topk_indices = torch.topk(\n        torch.softmax(gate_logits, dim=-1),\n        k=top_k,\n        dim=-1\n    )\n\n    # Load and run only top-k experts\n    output = torch.zeros_like(x)\n    for i in range(top_k):\n        expert_idx = topk_indices[:, i]\n        # Load expert from disk\u002Foffload if needed\n        expert = model.load_expert(expert_idx)\n        output += topk_scores[:, i:i+1] * expert(x)\n\n    return output\n```\n\n## Resources\n\n- **DeepSpeed MoE Tutorial**: https:\u002F\u002Fwww.deepspeed.ai\u002Ftutorials\u002Fmixture-of-experts-nlg\u002F\n- **Mixtral Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.04088\n- **Switch Transformers**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2101.03961\n- **HuggingFace MoE Guide**: https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fmoe\n- **NVIDIA MoE Blog**: https:\u002F\u002Fdeveloper.nvidia.com\u002Fblog\u002Fapplying-mixture-of-experts-in-llm-architectures\u002F\n\n## See Also\n\n- `references\u002Farchitectures.md` - MoE model architectures (Mixtral, Switch, DeepSeek-V3)\n- `references\u002Ftraining.md` - Advanced training techniques and optimization\n- `references\u002Finference.md` - Production deployment and serving patterns\n\n\n","# MoE Training: Mixture of Experts\n\n## When to Use This Skill\n\nUse MoE Training when you need to:\n- **Train larger models** with limited compute (5× cost reduction vs dense models)\n- **Scale model capacity** without proportional compute increase\n- **Achieve better performance** per compute budget than dense models\n- **Specialize experts** for different domains\u002Ftasks\u002Flanguages\n- **Reduce inference latency** with sparse activation (only 13B\u002F47B params active in Mixtral)\n- **Implement SOTA models** like Mixtral 8x7B, DeepSeek-V3, Switch Transformers\n\n**Notable MoE Models**: Mixtral 8x7B (Mistral AI), DeepSeek-V3, Switch Transformers (Google), GLaM (Google), NLLB-MoE (Meta)\n\n## Installation\n\n```bash\n# DeepSpeed with MoE support\npip install deepspeed>=0.6.0\n\n# Megatron-DeepSpeed for large-scale training\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FMegatron-DeepSpeed\ncd Megatron-DeepSpeed\npip install -r requirements.txt\n\n# Alternative: HuggingFace Transformers\npip install transformers accelerate\n```\n\n## Quick Start\n\n### Basic MoE Architecture\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass MoELayer(nn.Module):\n    \"\"\"Sparse Mixture of Experts layer.\"\"\"\n\n    def __init__(self, hidden_size, num_experts=8, top_k=2):\n        super().__init__()\n        self.num_experts = num_experts\n        self.top_k = top_k\n\n        # Expert networks (FFN)\n        self.experts = nn.ModuleList([\n            nn.Sequential(\n                nn.Linear(hidden_size, 4 * hidden_size),\n                nn.GELU(),\n                nn.Linear(4 * hidden_size, hidden_size)\n            )\n            for _ in range(num_experts)\n        ])\n\n        # Gating network (router)\n        self.gate = nn.Linear(hidden_size, num_experts)\n\n    def forward(self, x):\n        # x shape: (batch_size, seq_len, hidden_size)\n        batch_size, seq_len, hidden_size = x.shape\n\n        # Flatten for routing\n        x_flat = x.view(-1, hidden_size)  # (batch_size * seq_len, hidden_size)\n\n        # Compute gate scores\n        gate_logits = self.gate(x_flat)  # (batch_size * seq_len, num_experts)\n\n        # Top-k routing\n        gate_scores = torch.softmax(gate_logits, dim=-1)\n        topk_scores, topk_indices = torch.topk(gate_scores, self.top_k, dim=-1)\n\n        # Normalize top-k scores\n        topk_scores = topk_scores \u002F topk_scores.sum(dim=-1, keepdim=True)\n\n        # Dispatch and combine expert outputs\n        output = torch.zeros_like(x_flat)\n\n        for i in range(self.top_k):\n            expert_idx = topk_indices[:, i]\n            expert_scores = topk_scores[:, i].unsqueeze(-1)\n\n            # Route tokens to experts\n            for expert_id in range(self.num_experts):\n                mask = (expert_idx == expert_id)\n                if mask.any():\n                    expert_input = x_flat[mask]\n                    expert_output = self.experts[expert_id](expert_input)\n                    output[mask] += expert_scores[mask] * expert_output\n\n        # Reshape back\n        return output.view(batch_size, seq_len, hidden_size)\n```\n\n### DeepSpeed MoE Training\n\n```bash\n# Training script with MoE\ndeepspeed pretrain_gpt_moe.py \\\n  --num-layers 24 \\\n  --hidden-size 1024 \\\n  --num-attention-heads 16 \\\n  --seq-length 2048 \\\n  --max-position-embeddings 2048 \\\n  --micro-batch-size 4 \\\n  --global-batch-size 256 \\\n  --train-iters 500000 \\\n  --lr 0.0001 \\\n  --min-lr 0.00001 \\\n  --lr-decay-style cosine \\\n  --num-experts 128 \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --moe-train-capacity-factor 1.25 \\\n  --moe-eval-capacity-factor 2.0 \\\n  --fp16 \\\n  --deepspeed_config ds_config.json\n```\n\n## Core Concepts\n\n### 1. MoE Architecture\n\n**Key Components:**\n- **Experts**: Multiple specialized FFN networks (typically 8-128)\n- **Router\u002FGate**: Learned network that selects which experts to use\n- **Top-k Routing**: Activate only k experts per token (k=1 or k=2)\n- **Load Balancing**: Ensure even expert utilization\n\n```\nInput Token\n    ↓\nRouter (Gate Network)\n    ↓\nTop-k Expert Selection (e.g., 2 out of 8)\n    ↓\nExpert 1 (weight: 0.6) + Expert 5 (weight: 0.4)\n    ↓\nWeighted Combination\n    ↓\nOutput\n```\n\n### 2. Routing Mechanisms\n\n**Top-1 Routing (Switch Transformer):**\n```python\n# Simplest routing: one expert per token\ngate_logits = router(x)  # (batch, seq_len, num_experts)\nexpert_idx = torch.argmax(gate_logits, dim=-1)  # Hard routing\n```\n\n**Top-2 Routing (Mixtral):**\n```python\n# Top-2: two experts per token\ngate_scores = torch.softmax(router(x), dim=-1)\ntop2_scores, top2_indices = torch.topk(gate_scores, k=2, dim=-1)\n\n# Normalize scores\ntop2_scores = top2_scores \u002F top2_scores.sum(dim=-1, keepdim=True)\n\n# Combine expert outputs\noutput = (top2_scores[:, :, 0:1] * expert_outputs[top2_indices[:, :, 0]] +\n          top2_scores[:, :, 1:2] * expert_outputs[top2_indices[:, :, 1]])\n```\n\n**Expert Choice Routing:**\n```python\n# Experts choose top-k tokens (instead of tokens choosing experts)\n# Guarantees perfect load balancing\nexpert_scores = router(x).transpose(-1, -2)  # (batch, num_experts, seq_len)\ntopk_tokens = torch.topk(expert_scores, k=capacity_per_expert, dim=-1)\n```\n\n### 3. Load Balancing\n\n**Auxiliary Loss:**\n```python\ndef load_balancing_loss(gate_logits, expert_indices, num_experts):\n    \"\"\"Encourage uniform expert usage.\"\"\"\n    # Fraction of tokens routed to each expert\n    expert_counts = torch.bincount(expert_indices.flatten(), minlength=num_experts)\n    expert_fraction = expert_counts.float() \u002F expert_indices.numel()\n\n    # Gate probability for each expert (average across tokens)\n    gate_probs = torch.softmax(gate_logits, dim=-1).mean(dim=0)\n\n    # Auxiliary loss: encourage alignment\n    aux_loss = num_experts * (expert_fraction * gate_probs).sum()\n\n    return aux_loss\n\n# Add to main loss\ntotal_loss = language_model_loss + 0.01 * load_balancing_loss(...)\n```\n\n**Router Z-Loss (Stability):**\n```python\ndef router_z_loss(logits):\n    \"\"\"Encourage router to have lower entropy (more decisive).\"\"\"\n    z_loss = torch.logsumexp(logits, dim=-1).pow(2).mean()\n    return z_loss\n\ntotal_loss = lm_loss + 0.01 * aux_loss + 0.001 * router_z_loss(gate_logits)\n```\n\n### 4. Expert Parallelism\n\n```python\n# DeepSpeed configuration\n{\n  \"train_batch_size\": 256,\n  \"fp16\": {\"enabled\": true},\n  \"moe\": {\n    \"enabled\": true,\n    \"num_experts\": 128,\n    \"expert_parallel_size\": 8,  # Distribute 128 experts across 8 GPUs\n    \"capacity_factor\": 1.25,    # Expert capacity = tokens_per_batch * capacity_factor \u002F num_experts\n    \"drop_tokens\": true,        # Drop tokens exceeding capacity\n    \"use_residual\": false\n  }\n}\n```\n\n## Training Configuration\n\n### DeepSpeed MoE Config\n\n```json\n{\n  \"train_batch_size\": 256,\n  \"gradient_accumulation_steps\": 1,\n  \"optimizer\": {\n    \"type\": \"Adam\",\n    \"params\": {\n      \"lr\": 0.0001,\n      \"betas\": [0.9, 0.999],\n      \"eps\": 1e-8\n    }\n  },\n  \"fp16\": {\n    \"enabled\": true,\n    \"loss_scale\": 0,\n    \"initial_scale_power\": 16\n  },\n  \"moe\": {\n    \"enabled\": true,\n    \"num_experts\": 128,\n    \"expert_parallel_size\": 8,\n    \"moe_loss_coeff\": 0.01,\n    \"train_capacity_factor\": 1.25,\n    \"eval_capacity_factor\": 2.0,\n    \"min_capacity\": 4,\n    \"drop_tokens\": true,\n    \"use_residual\": false,\n    \"use_tutel\": false\n  },\n  \"zero_optimization\": {\n    \"stage\": 1\n  }\n}\n```\n\n### Training Script\n\n```bash\n#!\u002Fbin\u002Fbash\n\n# Mixtral-style MoE training\ndeepspeed --num_gpus 8 pretrain_moe.py \\\n  --model-parallel-size 1 \\\n  --num-layers 32 \\\n  --hidden-size 4096 \\\n  --num-attention-heads 32 \\\n  --seq-length 2048 \\\n  --max-position-embeddings 4096 \\\n  --micro-batch-size 2 \\\n  --global-batch-size 256 \\\n  --train-iters 500000 \\\n  --save-interval 5000 \\\n  --eval-interval 1000 \\\n  --eval-iters 100 \\\n  --lr 0.0001 \\\n  --min-lr 0.00001 \\\n  --lr-decay-style cosine \\\n  --lr-warmup-iters 2000 \\\n  --clip-grad 1.0 \\\n  --weight-decay 0.1 \\\n  --num-experts 8 \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --moe-train-capacity-factor 1.25 \\\n  --moe-eval-capacity-factor 2.0 \\\n  --disable-moe-token-dropping \\\n  --fp16 \\\n  --deepspeed \\\n  --deepspeed_config ds_config_moe.json \\\n  --data-path \u002Fpath\u002Fto\u002Fdata \\\n  --vocab-file \u002Fpath\u002Fto\u002Fvocab.json \\\n  --merge-file \u002Fpath\u002Fto\u002Fmerges.txt\n```\n\n## Advanced Patterns\n\n### Mixtral 8x7B Architecture\n\n```python\nclass MixtralMoEBlock(nn.Module):\n    \"\"\"Mixtral-style MoE block with 8 experts, top-2 routing.\"\"\"\n\n    def __init__(self, config):\n        super().__init__()\n        self.hidden_dim = config.hidden_size\n        self.ffn_dim = config.intermediate_size\n        self.num_experts = config.num_local_experts  # 8\n        self.top_k = config.num_experts_per_tok       # 2\n\n        # 8 expert FFNs\n        self.experts = nn.ModuleList([\n            nn.Sequential(\n                nn.Linear(self.hidden_dim, self.ffn_dim, bias=False),\n                nn.SiLU(),\n                nn.Linear(self.ffn_dim, self.hidden_dim, bias=False)\n            )\n            for _ in range(self.num_experts)\n        ])\n\n        # Router\n        self.gate = nn.Linear(self.hidden_dim, self.num_experts, bias=False)\n\n    def forward(self, hidden_states):\n        batch_size, sequence_length, hidden_dim = hidden_states.shape\n\n        # Flatten\n        hidden_states = hidden_states.view(-1, hidden_dim)\n\n        # Router logits\n        router_logits = self.gate(hidden_states)  # (batch * seq_len, num_experts)\n\n        # Softmax and top-2\n        routing_weights = torch.softmax(router_logits, dim=1)\n        routing_weights, selected_experts = torch.topk(routing_weights, self.top_k, dim=-1)\n\n        # Normalize routing weights\n        routing_weights \u002F= routing_weights.sum(dim=-1, keepdim=True)\n\n        # Initialize output\n        final_hidden_states = torch.zeros_like(hidden_states)\n\n        # Route to experts\n        for expert_idx in range(self.num_experts):\n            expert_layer = self.experts[expert_idx]\n            idx, top_x = torch.where(selected_experts == expert_idx)\n\n            if idx.shape[0] == 0:\n                continue\n\n            # Current expert tokens\n            current_hidden_states = hidden_states[idx]\n\n            # Expert forward\n            current_hidden_states = expert_layer(current_hidden_states)\n\n            # Weighted by routing scores\n            current_hidden_states *= routing_weights[idx, top_x, None]\n\n            # Accumulate\n            final_hidden_states.index_add_(0, idx, current_hidden_states)\n\n        # Reshape\n        return final_hidden_states.view(batch_size, sequence_length, hidden_dim)\n```\n\n### PR-MoE (Pyramid-Residual-MoE)\n\n```bash\n# DeepSpeed PR-MoE: 3x better parameter efficiency\ndeepspeed pretrain_gpt_moe.py \\\n  --num-layers 24 \\\n  --hidden-size 1024 \\\n  --num-attention-heads 16 \\\n  --num-experts \"[128, 64, 32, 16]\" \\\n  --mlp-type residual \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --fp16\n```\n\n## Best Practices\n\n### 1. Expert Count Selection\n\n```python\n# Rule of thumb: More experts = more capacity, but diminishing returns\n# Typical configurations:\n# - Small models (1B-7B): 8-16 experts\n# - Medium models (7B-30B): 8-64 experts\n# - Large models (30B+): 64-256 experts\n\n# Example: Mixtral 8x7B\n# Total params: 47B (8 experts × 7B each)\n# Active params: 13B (2 experts × 7B, top-2 routing)\n# Efficiency: 47B capacity with 13B compute\n```\n\n### 2. Capacity Factor Tuning\n\n```python\n# Capacity = (tokens_per_batch \u002F num_experts) * capacity_factor\n\n# Training: Lower capacity (faster, drops some tokens)\ntrain_capacity_factor = 1.25  # 25% buffer\n\n# Evaluation: Higher capacity (no dropping)\neval_capacity_factor = 2.0    # 100% buffer\n\n# Formula:\nexpert_capacity = int((seq_len * batch_size \u002F num_experts) * capacity_factor)\n```\n\n### 3. Learning Rate Guidelines\n\n```python\n# MoE models need lower LR than dense models\n# - Dense model: lr = 6e-4\n# - MoE model: lr = 1e-4 (3-6× lower)\n\n# Also extend decay schedule\ndense_lr_decay_iters = 300000\nmoe_lr_decay_iters = 500000  # 1.5-2× longer\n```\n\n### 4. Loss Coefficient Tuning\n\n```python\n# Start with standard values\nmoe_loss_coeff = 0.01    # Auxiliary loss (load balancing)\nrouter_z_loss_coeff = 0.001  # Router entropy (stability)\n\n# If load imbalance persists, increase aux loss\nif max_expert_usage \u002F min_expert_usage > 2.0:\n    moe_loss_coeff = 0.1  # Stronger load balancing\n\n# If training unstable, increase z-loss\nif grad_norm > 10.0:\n    router_z_loss_coeff = 0.01\n```\n\n### 5. Avoid Common Pitfalls\n\n```python\n# ❌ Bad: Using same LR as dense model\noptimizer = Adam(model.parameters(), lr=6e-4)\n\n# ✅ Good: Lower LR for MoE\noptimizer = Adam([\n    {'params': model.non_moe_params, 'lr': 6e-4},\n    {'params': model.moe_params, 'lr': 1e-4}\n])\n\n# ❌ Bad: No load balancing\nloss = lm_loss\n\n# ✅ Good: Add auxiliary loss\nloss = lm_loss + 0.01 * aux_loss + 0.001 * z_loss\n\n# ❌ Bad: Too many experts for small dataset\nnum_experts = 128  # Overfitting risk\n\n# ✅ Good: Match experts to data diversity\nnum_experts = 8  # Better for small datasets\n```\n\n## Inference Optimization\n\n### Sparse Inference\n\n```python\n# Only activate top-k experts (huge memory savings)\n@torch.no_grad()\ndef moe_inference(x, model, top_k=2):\n    \"\"\"Sparse MoE inference: only load k experts.\"\"\"\n    # Router\n    gate_logits = model.gate(x)\n    topk_scores, topk_indices = torch.topk(\n        torch.softmax(gate_logits, dim=-1),\n        k=top_k,\n        dim=-1\n    )\n\n    # Load and run only top-k experts\n    output = torch.zeros_like(x)\n    for i in range(top_k):\n        expert_idx = topk_indices[:, i]\n        # Load expert from disk\u002Foffload if needed\n        expert = model.load_expert(expert_idx)\n        output += topk_scores[:, i:i+1] * expert(x)\n\n    return output\n```\n\n## Resources\n\n- **DeepSpeed MoE Tutorial**: https:\u002F\u002Fwww.deepspeed.ai\u002Ftutorials\u002Fmixture-of-experts-nlg\u002F\n- **Mixtral Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.04088\n- **Switch Transformers**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2101.03961\n- **HuggingFace MoE Guide**: https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fmoe\n- **NVIDIA MoE Blog**: https:\u002F\u002Fdeveloper.nvidia.com\u002Fblog\u002Fapplying-mixture-of-experts-in-llm-architectures\u002F\n\n## See Also\n\n- `references\u002Farchitectures.md` - MoE model architectures (Mixtral, Switch, DeepSeek-V3)\n- `references\u002Ftraining.md` - Advanced training techniques and optimization\n- `references\u002Finference.md` - Production deployment and serving patterns\n\n\n",{"name":5,"tags":24,"author":36,"license":15,"version":37,"description":7,"dependencies":38},[25,26,27,28,29,30,31,32,33,34,35],"Emerging Techniques","MoE","Mixture Of Experts","Sparse Models","DeepSpeed","Expert Parallelism","Mixtral","DeepSeek","Routing","Load Balancing","Efficient Training","Orchestra Research","1.0.0",[39,40,41,42],"deepspeed","transformers","torch","accelerate","master","c247302c836de6438d746b45670990eb71691166","skills_collected\u002FOrchestra-Research--AI-research-SKILLs--19-emerging-techniques__moe-training",1,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool","https:\u002F\u002Fgithub.com\u002Fqhjqhj00","chien.io","2026-05-11 15:07:59",{"source":8,"skillName":5,"commands":52,"prompt":62,"zipAvailable":63},[53,56,59],{"tool":54,"command":55},"npx","npx skills add qhjqhj00\u002Fresearch-skills-pool --skill moe-training",{"tool":57,"command":58},"bunx","bunx skills add qhjqhj00\u002Fresearch-skills-pool --skill moe-training",{"tool":60,"command":61},"pnpm","pnpm dlx skills add qhjqhj00\u002Fresearch-skills-pool --skill moe-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--19-emerging-techniques__moe-training\u002FSKILL.md",true,[65,71,75],{"path":66,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":69,"sourceHtmlUrl":70},"references\u002Farchitectures.md","REFERENCE","参考文档",14044,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--19-emerging-techniques__moe-training\u002Freferences\u002Farchitectures.md",{"path":72,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":73,"sourceHtmlUrl":74},"references\u002Finference.md",8239,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--19-emerging-techniques__moe-training\u002Freferences\u002Finference.md",{"path":76,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":77,"sourceHtmlUrl":78},"references\u002Ftraining.md",9707,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--19-emerging-techniques__moe-training\u002Freferences\u002Ftraining.md",[80,87,94,101,108,115],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":19},"01M73JJB6EQPFPPVQK2Y9N67Q0","transformer-lens-interpretability","Provides guidance for mechanistic interpretability research using TransformerLens to inspect and manipulate transformer internals via HookPoints and activation caching. Use when reverse-engineering model algorithms, studying attention patterns, or performing activation patching experiments.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F130-transformer-lens\u002FSKILL.md",[],[],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":19},"01M73JHTANRRWEE9HP98TPEHAY","wiki-vitepress","Packages generated wiki Markdown into a VitePress static site with dark theme, dark-mode Mermaid diagrams with click-to-zoom, and production build output. Use when the user wants to create a browsable website from generated wiki pages.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F126-wiki-vitepress\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":19},"01M73JHJBMVFW46F1Y8K3SBF43","phoenix-observability","Open-source AI observability platform for LLM tracing, evaluation, and monitoring. Use when debugging LLM applications with detailed traces, running evaluations on datasets, or monitoring production AI systems with real-time insights.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F124-phoenix\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":19},"01M73JHFPA3A2806PMP42ZN86Q","frontend-ui-dark-ts","Build dark-themed React applications using Tailwind CSS with custom theming, glassmorphism effects, and Framer Motion animations. Use when creating dashboards, admin panels, or data-rich interfaces with a refined dark aesthetic.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F123-frontend-ui-dark-ts\u002FSKILL.md",[],[],{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":19},"01M73JFVBVK57NRJS49TT2VDCT","tensorboard","Visualize training metrics, debug models with histograms, compare experiments, visualize model graphs, and profile performance with TensorBoard - Google's ML visualization toolkit","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F113-tensorboard\u002FSKILL.md",[],[],{"skillNo":116,"name":117,"nameZh":6,"description":118,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":119,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":19},"01M73JFF6PK6ZCN41C89J7HS2B","markdown-mermaid-writing","Comprehensive markdown and Mermaid diagram writing skill. Use when creating any scientific document, report, analysis, or visualization. Establishes text-based diagrams as the default documentation standard with full style guides (markdown + mermaid), 24 diagram type references, and 9 document templates.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F110-markdown-mermaid-writing\u002FSKILL.md",[],[]]