[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M79TP5GM0YR0NHJ2D5PVCJ40":3,"skill-same:01M79TP5GM0YR0NHJ2D5PVCJ40":76},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":42,"commitSha":43,"skillDirectory":44,"forksCount":13,"repositoryHtmlUrl":45,"repositoryHomepageUrl":6,"ownerHtmlUrl":46,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":47,"install":48,"resources":61},"01M79TP5GM0YR0NHJ2D5PVCJ40","moe-training",null,"Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.","KrismithReddy12\u002Fclaude-code-templates","KrismithReddy12","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F182916557?v=4","01M79T8DNGJRHG48G3H3Q02MK8","01M79TA8ZWV2PP7R4ARWVYWZFE",0,"MIT","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-moe-training\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: moe-training\ndescription: Train Mixture of Experts (MoE) models using DeepSpeed or HuggingFace. Use when training large-scale models with limited compute (5× cost reduction vs dense models), implementing sparse architectures like Mixtral 8x7B or DeepSeek-V3, or scaling model capacity without proportional compute increase. Covers MoE architectures, routing mechanisms, load balancing, expert parallelism, and inference optimization.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Emerging Techniques, MoE, Mixture Of Experts, Sparse Models, DeepSpeed, Expert Parallelism, Mixtral, DeepSeek, Routing, Load Balancing, Efficient Training]\ndependencies: [deepspeed, transformers, torch, accelerate]\n---\n\n# MoE Training: Mixture of Experts\n\n## When to Use This Skill\n\nUse MoE Training when you need to:\n- **Train larger models** with limited compute (5× cost reduction vs dense models)\n- **Scale model capacity** without proportional compute increase\n- **Achieve better performance** per compute budget than dense models\n- **Specialize experts** for different domains\u002Ftasks\u002Flanguages\n- **Reduce inference latency** with sparse activation (only 13B\u002F47B params active in Mixtral)\n- **Implement SOTA models** like Mixtral 8x7B, DeepSeek-V3, Switch Transformers\n\n**Notable MoE Models**: Mixtral 8x7B (Mistral AI), DeepSeek-V3, Switch Transformers (Google), GLaM (Google), NLLB-MoE (Meta)\n\n## Installation\n\n```bash\n# DeepSpeed with MoE support\npip install deepspeed>=0.6.0\n\n# Megatron-DeepSpeed for large-scale training\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FMegatron-DeepSpeed\ncd Megatron-DeepSpeed\npip install -r requirements.txt\n\n# Alternative: HuggingFace Transformers\npip install transformers accelerate\n```\n\n## Quick Start\n\n### Basic MoE Architecture\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass MoELayer(nn.Module):\n    \"\"\"Sparse Mixture of Experts layer.\"\"\"\n\n    def __init__(self, hidden_size, num_experts=8, top_k=2):\n        super().__init__()\n        self.num_experts = num_experts\n        self.top_k = top_k\n\n        # Expert networks (FFN)\n        self.experts = nn.ModuleList([\n            nn.Sequential(\n                nn.Linear(hidden_size, 4 * hidden_size),\n                nn.GELU(),\n                nn.Linear(4 * hidden_size, hidden_size)\n            )\n            for _ in range(num_experts)\n        ])\n\n        # Gating network (router)\n        self.gate = nn.Linear(hidden_size, num_experts)\n\n    def forward(self, x):\n        # x shape: (batch_size, seq_len, hidden_size)\n        batch_size, seq_len, hidden_size = x.shape\n\n        # Flatten for routing\n        x_flat = x.view(-1, hidden_size)  # (batch_size * seq_len, hidden_size)\n\n        # Compute gate scores\n        gate_logits = self.gate(x_flat)  # (batch_size * seq_len, num_experts)\n\n        # Top-k routing\n        gate_scores = torch.softmax(gate_logits, dim=-1)\n        topk_scores, topk_indices = torch.topk(gate_scores, self.top_k, dim=-1)\n\n        # Normalize top-k scores\n        topk_scores = topk_scores \u002F topk_scores.sum(dim=-1, keepdim=True)\n\n        # Dispatch and combine expert outputs\n        output = torch.zeros_like(x_flat)\n\n        for i in range(self.top_k):\n            expert_idx = topk_indices[:, i]\n            expert_scores = topk_scores[:, i].unsqueeze(-1)\n\n            # Route tokens to experts\n            for expert_id in range(self.num_experts):\n                mask = (expert_idx == expert_id)\n                if mask.any():\n                    expert_input = x_flat[mask]\n                    expert_output = self.experts[expert_id](expert_input)\n                    output[mask] += expert_scores[mask] * expert_output\n\n        # Reshape back\n        return output.view(batch_size, seq_len, hidden_size)\n```\n\n### DeepSpeed MoE Training\n\n```bash\n# Training script with MoE\ndeepspeed pretrain_gpt_moe.py \\\n  --num-layers 24 \\\n  --hidden-size 1024 \\\n  --num-attention-heads 16 \\\n  --seq-length 2048 \\\n  --max-position-embeddings 2048 \\\n  --micro-batch-size 4 \\\n  --global-batch-size 256 \\\n  --train-iters 500000 \\\n  --lr 0.0001 \\\n  --min-lr 0.00001 \\\n  --lr-decay-style cosine \\\n  --num-experts 128 \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --moe-train-capacity-factor 1.25 \\\n  --moe-eval-capacity-factor 2.0 \\\n  --fp16 \\\n  --deepspeed_config ds_config.json\n```\n\n## Core Concepts\n\n### 1. MoE Architecture\n\n**Key Components:**\n- **Experts**: Multiple specialized FFN networks (typically 8-128)\n- **Router\u002FGate**: Learned network that selects which experts to use\n- **Top-k Routing**: Activate only k experts per token (k=1 or k=2)\n- **Load Balancing**: Ensure even expert utilization\n\n```\nInput Token\n    ↓\nRouter (Gate Network)\n    ↓\nTop-k Expert Selection (e.g., 2 out of 8)\n    ↓\nExpert 1 (weight: 0.6) + Expert 5 (weight: 0.4)\n    ↓\nWeighted Combination\n    ↓\nOutput\n```\n\n### 2. Routing Mechanisms\n\n**Top-1 Routing (Switch Transformer):**\n```python\n# Simplest routing: one expert per token\ngate_logits = router(x)  # (batch, seq_len, num_experts)\nexpert_idx = torch.argmax(gate_logits, dim=-1)  # Hard routing\n```\n\n**Top-2 Routing (Mixtral):**\n```python\n# Top-2: two experts per token\ngate_scores = torch.softmax(router(x), dim=-1)\ntop2_scores, top2_indices = torch.topk(gate_scores, k=2, dim=-1)\n\n# Normalize scores\ntop2_scores = top2_scores \u002F top2_scores.sum(dim=-1, keepdim=True)\n\n# Combine expert outputs\noutput = (top2_scores[:, :, 0:1] * expert_outputs[top2_indices[:, :, 0]] +\n          top2_scores[:, :, 1:2] * expert_outputs[top2_indices[:, :, 1]])\n```\n\n**Expert Choice Routing:**\n```python\n# Experts choose top-k tokens (instead of tokens choosing experts)\n# Guarantees perfect load balancing\nexpert_scores = router(x).transpose(-1, -2)  # (batch, num_experts, seq_len)\ntopk_tokens = torch.topk(expert_scores, k=capacity_per_expert, dim=-1)\n```\n\n### 3. Load Balancing\n\n**Auxiliary Loss:**\n```python\ndef load_balancing_loss(gate_logits, expert_indices, num_experts):\n    \"\"\"Encourage uniform expert usage.\"\"\"\n    # Fraction of tokens routed to each expert\n    expert_counts = torch.bincount(expert_indices.flatten(), minlength=num_experts)\n    expert_fraction = expert_counts.float() \u002F expert_indices.numel()\n\n    # Gate probability for each expert (average across tokens)\n    gate_probs = torch.softmax(gate_logits, dim=-1).mean(dim=0)\n\n    # Auxiliary loss: encourage alignment\n    aux_loss = num_experts * (expert_fraction * gate_probs).sum()\n\n    return aux_loss\n\n# Add to main loss\ntotal_loss = language_model_loss + 0.01 * load_balancing_loss(...)\n```\n\n**Router Z-Loss (Stability):**\n```python\ndef router_z_loss(logits):\n    \"\"\"Encourage router to have lower entropy (more decisive).\"\"\"\n    z_loss = torch.logsumexp(logits, dim=-1).pow(2).mean()\n    return z_loss\n\ntotal_loss = lm_loss + 0.01 * aux_loss + 0.001 * router_z_loss(gate_logits)\n```\n\n### 4. Expert Parallelism\n\n```python\n# DeepSpeed configuration\n{\n  \"train_batch_size\": 256,\n  \"fp16\": {\"enabled\": true},\n  \"moe\": {\n    \"enabled\": true,\n    \"num_experts\": 128,\n    \"expert_parallel_size\": 8,  # Distribute 128 experts across 8 GPUs\n    \"capacity_factor\": 1.25,    # Expert capacity = tokens_per_batch * capacity_factor \u002F num_experts\n    \"drop_tokens\": true,        # Drop tokens exceeding capacity\n    \"use_residual\": false\n  }\n}\n```\n\n## Training Configuration\n\n### DeepSpeed MoE Config\n\n```json\n{\n  \"train_batch_size\": 256,\n  \"gradient_accumulation_steps\": 1,\n  \"optimizer\": {\n    \"type\": \"Adam\",\n    \"params\": {\n      \"lr\": 0.0001,\n      \"betas\": [0.9, 0.999],\n      \"eps\": 1e-8\n    }\n  },\n  \"fp16\": {\n    \"enabled\": true,\n    \"loss_scale\": 0,\n    \"initial_scale_power\": 16\n  },\n  \"moe\": {\n    \"enabled\": true,\n    \"num_experts\": 128,\n    \"expert_parallel_size\": 8,\n    \"moe_loss_coeff\": 0.01,\n    \"train_capacity_factor\": 1.25,\n    \"eval_capacity_factor\": 2.0,\n    \"min_capacity\": 4,\n    \"drop_tokens\": true,\n    \"use_residual\": false,\n    \"use_tutel\": false\n  },\n  \"zero_optimization\": {\n    \"stage\": 1\n  }\n}\n```\n\n### Training Script\n\n```bash\n#!\u002Fbin\u002Fbash\n\n# Mixtral-style MoE training\ndeepspeed --num_gpus 8 pretrain_moe.py \\\n  --model-parallel-size 1 \\\n  --num-layers 32 \\\n  --hidden-size 4096 \\\n  --num-attention-heads 32 \\\n  --seq-length 2048 \\\n  --max-position-embeddings 4096 \\\n  --micro-batch-size 2 \\\n  --global-batch-size 256 \\\n  --train-iters 500000 \\\n  --save-interval 5000 \\\n  --eval-interval 1000 \\\n  --eval-iters 100 \\\n  --lr 0.0001 \\\n  --min-lr 0.00001 \\\n  --lr-decay-style cosine \\\n  --lr-warmup-iters 2000 \\\n  --clip-grad 1.0 \\\n  --weight-decay 0.1 \\\n  --num-experts 8 \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --moe-train-capacity-factor 1.25 \\\n  --moe-eval-capacity-factor 2.0 \\\n  --disable-moe-token-dropping \\\n  --fp16 \\\n  --deepspeed \\\n  --deepspeed_config ds_config_moe.json \\\n  --data-path \u002Fpath\u002Fto\u002Fdata \\\n  --vocab-file \u002Fpath\u002Fto\u002Fvocab.json \\\n  --merge-file \u002Fpath\u002Fto\u002Fmerges.txt\n```\n\n## Advanced Patterns\n\n### Mixtral 8x7B Architecture\n\n```python\nclass MixtralMoEBlock(nn.Module):\n    \"\"\"Mixtral-style MoE block with 8 experts, top-2 routing.\"\"\"\n\n    def __init__(self, config):\n        super().__init__()\n        self.hidden_dim = config.hidden_size\n        self.ffn_dim = config.intermediate_size\n        self.num_experts = config.num_local_experts  # 8\n        self.top_k = config.num_experts_per_tok       # 2\n\n        # 8 expert FFNs\n        self.experts = nn.ModuleList([\n            nn.Sequential(\n                nn.Linear(self.hidden_dim, self.ffn_dim, bias=False),\n                nn.SiLU(),\n                nn.Linear(self.ffn_dim, self.hidden_dim, bias=False)\n            )\n            for _ in range(self.num_experts)\n        ])\n\n        # Router\n        self.gate = nn.Linear(self.hidden_dim, self.num_experts, bias=False)\n\n    def forward(self, hidden_states):\n        batch_size, sequence_length, hidden_dim = hidden_states.shape\n\n        # Flatten\n        hidden_states = hidden_states.view(-1, hidden_dim)\n\n        # Router logits\n        router_logits = self.gate(hidden_states)  # (batch * seq_len, num_experts)\n\n        # Softmax and top-2\n        routing_weights = torch.softmax(router_logits, dim=1)\n        routing_weights, selected_experts = torch.topk(routing_weights, self.top_k, dim=-1)\n\n        # Normalize routing weights\n        routing_weights \u002F= routing_weights.sum(dim=-1, keepdim=True)\n\n        # Initialize output\n        final_hidden_states = torch.zeros_like(hidden_states)\n\n        # Route to experts\n        for expert_idx in range(self.num_experts):\n            expert_layer = self.experts[expert_idx]\n            idx, top_x = torch.where(selected_experts == expert_idx)\n\n            if idx.shape[0] == 0:\n                continue\n\n            # Current expert tokens\n            current_hidden_states = hidden_states[idx]\n\n            # Expert forward\n            current_hidden_states = expert_layer(current_hidden_states)\n\n            # Weighted by routing scores\n            current_hidden_states *= routing_weights[idx, top_x, None]\n\n            # Accumulate\n            final_hidden_states.index_add_(0, idx, current_hidden_states)\n\n        # Reshape\n        return final_hidden_states.view(batch_size, sequence_length, hidden_dim)\n```\n\n### PR-MoE (Pyramid-Residual-MoE)\n\n```bash\n# DeepSpeed PR-MoE: 3x better parameter efficiency\ndeepspeed pretrain_gpt_moe.py \\\n  --num-layers 24 \\\n  --hidden-size 1024 \\\n  --num-attention-heads 16 \\\n  --num-experts \"[128, 64, 32, 16]\" \\\n  --mlp-type residual \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --fp16\n```\n\n## Best Practices\n\n### 1. Expert Count Selection\n\n```python\n# Rule of thumb: More experts = more capacity, but diminishing returns\n# Typical configurations:\n# - Small models (1B-7B): 8-16 experts\n# - Medium models (7B-30B): 8-64 experts\n# - Large models (30B+): 64-256 experts\n\n# Example: Mixtral 8x7B\n# Total params: 47B (8 experts × 7B each)\n# Active params: 13B (2 experts × 7B, top-2 routing)\n# Efficiency: 47B capacity with 13B compute\n```\n\n### 2. Capacity Factor Tuning\n\n```python\n# Capacity = (tokens_per_batch \u002F num_experts) * capacity_factor\n\n# Training: Lower capacity (faster, drops some tokens)\ntrain_capacity_factor = 1.25  # 25% buffer\n\n# Evaluation: Higher capacity (no dropping)\neval_capacity_factor = 2.0    # 100% buffer\n\n# Formula:\nexpert_capacity = int((seq_len * batch_size \u002F num_experts) * capacity_factor)\n```\n\n### 3. Learning Rate Guidelines\n\n```python\n# MoE models need lower LR than dense models\n# - Dense model: lr = 6e-4\n# - MoE model: lr = 1e-4 (3-6× lower)\n\n# Also extend decay schedule\ndense_lr_decay_iters = 300000\nmoe_lr_decay_iters = 500000  # 1.5-2× longer\n```\n\n### 4. Loss Coefficient Tuning\n\n```python\n# Start with standard values\nmoe_loss_coeff = 0.01    # Auxiliary loss (load balancing)\nrouter_z_loss_coeff = 0.001  # Router entropy (stability)\n\n# If load imbalance persists, increase aux loss\nif max_expert_usage \u002F min_expert_usage > 2.0:\n    moe_loss_coeff = 0.1  # Stronger load balancing\n\n# If training unstable, increase z-loss\nif grad_norm > 10.0:\n    router_z_loss_coeff = 0.01\n```\n\n### 5. Avoid Common Pitfalls\n\n```python\n# ❌ Bad: Using same LR as dense model\noptimizer = Adam(model.parameters(), lr=6e-4)\n\n# ✅ Good: Lower LR for MoE\noptimizer = Adam([\n    {'params': model.non_moe_params, 'lr': 6e-4},\n    {'params': model.moe_params, 'lr': 1e-4}\n])\n\n# ❌ Bad: No load balancing\nloss = lm_loss\n\n# ✅ Good: Add auxiliary loss\nloss = lm_loss + 0.01 * aux_loss + 0.001 * z_loss\n\n# ❌ Bad: Too many experts for small dataset\nnum_experts = 128  # Overfitting risk\n\n# ✅ Good: Match experts to data diversity\nnum_experts = 8  # Better for small datasets\n```\n\n## Inference Optimization\n\n### Sparse Inference\n\n```python\n# Only activate top-k experts (huge memory savings)\n@torch.no_grad()\ndef moe_inference(x, model, top_k=2):\n    \"\"\"Sparse MoE inference: only load k experts.\"\"\"\n    # Router\n    gate_logits = model.gate(x)\n    topk_scores, topk_indices = torch.topk(\n        torch.softmax(gate_logits, dim=-1),\n        k=top_k,\n        dim=-1\n    )\n\n    # Load and run only top-k experts\n    output = torch.zeros_like(x)\n    for i in range(top_k):\n        expert_idx = topk_indices[:, i]\n        # Load expert from disk\u002Foffload if needed\n        expert = model.load_expert(expert_idx)\n        output += topk_scores[:, i:i+1] * expert(x)\n\n    return output\n```\n\n## Resources\n\n- **DeepSpeed MoE Tutorial**: https:\u002F\u002Fwww.deepspeed.ai\u002Ftutorials\u002Fmixture-of-experts-nlg\u002F\n- **Mixtral Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.04088\n- **Switch Transformers**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2101.03961\n- **HuggingFace MoE Guide**: https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fmoe\n- **NVIDIA MoE Blog**: https:\u002F\u002Fdeveloper.nvidia.com\u002Fblog\u002Fapplying-mixture-of-experts-in-llm-architectures\u002F\n\n## See Also\n\n- `references\u002Farchitectures.md` - MoE model architectures (Mixtral, Switch, DeepSeek-V3)\n- `references\u002Ftraining.md` - Advanced training techniques and optimization\n- `references\u002Finference.md` - Production deployment and serving patterns\n\n\n","# MoE Training: Mixture of Experts\n\n## When to Use This Skill\n\nUse MoE Training when you need to:\n- **Train larger models** with limited compute (5× cost reduction vs dense models)\n- **Scale model capacity** without proportional compute increase\n- **Achieve better performance** per compute budget than dense models\n- **Specialize experts** for different domains\u002Ftasks\u002Flanguages\n- **Reduce inference latency** with sparse activation (only 13B\u002F47B params active in Mixtral)\n- **Implement SOTA models** like Mixtral 8x7B, DeepSeek-V3, Switch Transformers\n\n**Notable MoE Models**: Mixtral 8x7B (Mistral AI), DeepSeek-V3, Switch Transformers (Google), GLaM (Google), NLLB-MoE (Meta)\n\n## Installation\n\n```bash\n# DeepSpeed with MoE support\npip install deepspeed>=0.6.0\n\n# Megatron-DeepSpeed for large-scale training\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FMegatron-DeepSpeed\ncd Megatron-DeepSpeed\npip install -r requirements.txt\n\n# Alternative: HuggingFace Transformers\npip install transformers accelerate\n```\n\n## Quick Start\n\n### Basic MoE Architecture\n\n```python\nimport torch\nimport torch.nn as nn\n\nclass MoELayer(nn.Module):\n    \"\"\"Sparse Mixture of Experts layer.\"\"\"\n\n    def __init__(self, hidden_size, num_experts=8, top_k=2):\n        super().__init__()\n        self.num_experts = num_experts\n        self.top_k = top_k\n\n        # Expert networks (FFN)\n        self.experts = nn.ModuleList([\n            nn.Sequential(\n                nn.Linear(hidden_size, 4 * hidden_size),\n                nn.GELU(),\n                nn.Linear(4 * hidden_size, hidden_size)\n            )\n            for _ in range(num_experts)\n        ])\n\n        # Gating network (router)\n        self.gate = nn.Linear(hidden_size, num_experts)\n\n    def forward(self, x):\n        # x shape: (batch_size, seq_len, hidden_size)\n        batch_size, seq_len, hidden_size = x.shape\n\n        # Flatten for routing\n        x_flat = x.view(-1, hidden_size)  # (batch_size * seq_len, hidden_size)\n\n        # Compute gate scores\n        gate_logits = self.gate(x_flat)  # (batch_size * seq_len, num_experts)\n\n        # Top-k routing\n        gate_scores = torch.softmax(gate_logits, dim=-1)\n        topk_scores, topk_indices = torch.topk(gate_scores, self.top_k, dim=-1)\n\n        # Normalize top-k scores\n        topk_scores = topk_scores \u002F topk_scores.sum(dim=-1, keepdim=True)\n\n        # Dispatch and combine expert outputs\n        output = torch.zeros_like(x_flat)\n\n        for i in range(self.top_k):\n            expert_idx = topk_indices[:, i]\n            expert_scores = topk_scores[:, i].unsqueeze(-1)\n\n            # Route tokens to experts\n            for expert_id in range(self.num_experts):\n                mask = (expert_idx == expert_id)\n                if mask.any():\n                    expert_input = x_flat[mask]\n                    expert_output = self.experts[expert_id](expert_input)\n                    output[mask] += expert_scores[mask] * expert_output\n\n        # Reshape back\n        return output.view(batch_size, seq_len, hidden_size)\n```\n\n### DeepSpeed MoE Training\n\n```bash\n# Training script with MoE\ndeepspeed pretrain_gpt_moe.py \\\n  --num-layers 24 \\\n  --hidden-size 1024 \\\n  --num-attention-heads 16 \\\n  --seq-length 2048 \\\n  --max-position-embeddings 2048 \\\n  --micro-batch-size 4 \\\n  --global-batch-size 256 \\\n  --train-iters 500000 \\\n  --lr 0.0001 \\\n  --min-lr 0.00001 \\\n  --lr-decay-style cosine \\\n  --num-experts 128 \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --moe-train-capacity-factor 1.25 \\\n  --moe-eval-capacity-factor 2.0 \\\n  --fp16 \\\n  --deepspeed_config ds_config.json\n```\n\n## Core Concepts\n\n### 1. MoE Architecture\n\n**Key Components:**\n- **Experts**: Multiple specialized FFN networks (typically 8-128)\n- **Router\u002FGate**: Learned network that selects which experts to use\n- **Top-k Routing**: Activate only k experts per token (k=1 or k=2)\n- **Load Balancing**: Ensure even expert utilization\n\n```\nInput Token\n    ↓\nRouter (Gate Network)\n    ↓\nTop-k Expert Selection (e.g., 2 out of 8)\n    ↓\nExpert 1 (weight: 0.6) + Expert 5 (weight: 0.4)\n    ↓\nWeighted Combination\n    ↓\nOutput\n```\n\n### 2. Routing Mechanisms\n\n**Top-1 Routing (Switch Transformer):**\n```python\n# Simplest routing: one expert per token\ngate_logits = router(x)  # (batch, seq_len, num_experts)\nexpert_idx = torch.argmax(gate_logits, dim=-1)  # Hard routing\n```\n\n**Top-2 Routing (Mixtral):**\n```python\n# Top-2: two experts per token\ngate_scores = torch.softmax(router(x), dim=-1)\ntop2_scores, top2_indices = torch.topk(gate_scores, k=2, dim=-1)\n\n# Normalize scores\ntop2_scores = top2_scores \u002F top2_scores.sum(dim=-1, keepdim=True)\n\n# Combine expert outputs\noutput = (top2_scores[:, :, 0:1] * expert_outputs[top2_indices[:, :, 0]] +\n          top2_scores[:, :, 1:2] * expert_outputs[top2_indices[:, :, 1]])\n```\n\n**Expert Choice Routing:**\n```python\n# Experts choose top-k tokens (instead of tokens choosing experts)\n# Guarantees perfect load balancing\nexpert_scores = router(x).transpose(-1, -2)  # (batch, num_experts, seq_len)\ntopk_tokens = torch.topk(expert_scores, k=capacity_per_expert, dim=-1)\n```\n\n### 3. Load Balancing\n\n**Auxiliary Loss:**\n```python\ndef load_balancing_loss(gate_logits, expert_indices, num_experts):\n    \"\"\"Encourage uniform expert usage.\"\"\"\n    # Fraction of tokens routed to each expert\n    expert_counts = torch.bincount(expert_indices.flatten(), minlength=num_experts)\n    expert_fraction = expert_counts.float() \u002F expert_indices.numel()\n\n    # Gate probability for each expert (average across tokens)\n    gate_probs = torch.softmax(gate_logits, dim=-1).mean(dim=0)\n\n    # Auxiliary loss: encourage alignment\n    aux_loss = num_experts * (expert_fraction * gate_probs).sum()\n\n    return aux_loss\n\n# Add to main loss\ntotal_loss = language_model_loss + 0.01 * load_balancing_loss(...)\n```\n\n**Router Z-Loss (Stability):**\n```python\ndef router_z_loss(logits):\n    \"\"\"Encourage router to have lower entropy (more decisive).\"\"\"\n    z_loss = torch.logsumexp(logits, dim=-1).pow(2).mean()\n    return z_loss\n\ntotal_loss = lm_loss + 0.01 * aux_loss + 0.001 * router_z_loss(gate_logits)\n```\n\n### 4. Expert Parallelism\n\n```python\n# DeepSpeed configuration\n{\n  \"train_batch_size\": 256,\n  \"fp16\": {\"enabled\": true},\n  \"moe\": {\n    \"enabled\": true,\n    \"num_experts\": 128,\n    \"expert_parallel_size\": 8,  # Distribute 128 experts across 8 GPUs\n    \"capacity_factor\": 1.25,    # Expert capacity = tokens_per_batch * capacity_factor \u002F num_experts\n    \"drop_tokens\": true,        # Drop tokens exceeding capacity\n    \"use_residual\": false\n  }\n}\n```\n\n## Training Configuration\n\n### DeepSpeed MoE Config\n\n```json\n{\n  \"train_batch_size\": 256,\n  \"gradient_accumulation_steps\": 1,\n  \"optimizer\": {\n    \"type\": \"Adam\",\n    \"params\": {\n      \"lr\": 0.0001,\n      \"betas\": [0.9, 0.999],\n      \"eps\": 1e-8\n    }\n  },\n  \"fp16\": {\n    \"enabled\": true,\n    \"loss_scale\": 0,\n    \"initial_scale_power\": 16\n  },\n  \"moe\": {\n    \"enabled\": true,\n    \"num_experts\": 128,\n    \"expert_parallel_size\": 8,\n    \"moe_loss_coeff\": 0.01,\n    \"train_capacity_factor\": 1.25,\n    \"eval_capacity_factor\": 2.0,\n    \"min_capacity\": 4,\n    \"drop_tokens\": true,\n    \"use_residual\": false,\n    \"use_tutel\": false\n  },\n  \"zero_optimization\": {\n    \"stage\": 1\n  }\n}\n```\n\n### Training Script\n\n```bash\n#!\u002Fbin\u002Fbash\n\n# Mixtral-style MoE training\ndeepspeed --num_gpus 8 pretrain_moe.py \\\n  --model-parallel-size 1 \\\n  --num-layers 32 \\\n  --hidden-size 4096 \\\n  --num-attention-heads 32 \\\n  --seq-length 2048 \\\n  --max-position-embeddings 4096 \\\n  --micro-batch-size 2 \\\n  --global-batch-size 256 \\\n  --train-iters 500000 \\\n  --save-interval 5000 \\\n  --eval-interval 1000 \\\n  --eval-iters 100 \\\n  --lr 0.0001 \\\n  --min-lr 0.00001 \\\n  --lr-decay-style cosine \\\n  --lr-warmup-iters 2000 \\\n  --clip-grad 1.0 \\\n  --weight-decay 0.1 \\\n  --num-experts 8 \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --moe-train-capacity-factor 1.25 \\\n  --moe-eval-capacity-factor 2.0 \\\n  --disable-moe-token-dropping \\\n  --fp16 \\\n  --deepspeed \\\n  --deepspeed_config ds_config_moe.json \\\n  --data-path \u002Fpath\u002Fto\u002Fdata \\\n  --vocab-file \u002Fpath\u002Fto\u002Fvocab.json \\\n  --merge-file \u002Fpath\u002Fto\u002Fmerges.txt\n```\n\n## Advanced Patterns\n\n### Mixtral 8x7B Architecture\n\n```python\nclass MixtralMoEBlock(nn.Module):\n    \"\"\"Mixtral-style MoE block with 8 experts, top-2 routing.\"\"\"\n\n    def __init__(self, config):\n        super().__init__()\n        self.hidden_dim = config.hidden_size\n        self.ffn_dim = config.intermediate_size\n        self.num_experts = config.num_local_experts  # 8\n        self.top_k = config.num_experts_per_tok       # 2\n\n        # 8 expert FFNs\n        self.experts = nn.ModuleList([\n            nn.Sequential(\n                nn.Linear(self.hidden_dim, self.ffn_dim, bias=False),\n                nn.SiLU(),\n                nn.Linear(self.ffn_dim, self.hidden_dim, bias=False)\n            )\n            for _ in range(self.num_experts)\n        ])\n\n        # Router\n        self.gate = nn.Linear(self.hidden_dim, self.num_experts, bias=False)\n\n    def forward(self, hidden_states):\n        batch_size, sequence_length, hidden_dim = hidden_states.shape\n\n        # Flatten\n        hidden_states = hidden_states.view(-1, hidden_dim)\n\n        # Router logits\n        router_logits = self.gate(hidden_states)  # (batch * seq_len, num_experts)\n\n        # Softmax and top-2\n        routing_weights = torch.softmax(router_logits, dim=1)\n        routing_weights, selected_experts = torch.topk(routing_weights, self.top_k, dim=-1)\n\n        # Normalize routing weights\n        routing_weights \u002F= routing_weights.sum(dim=-1, keepdim=True)\n\n        # Initialize output\n        final_hidden_states = torch.zeros_like(hidden_states)\n\n        # Route to experts\n        for expert_idx in range(self.num_experts):\n            expert_layer = self.experts[expert_idx]\n            idx, top_x = torch.where(selected_experts == expert_idx)\n\n            if idx.shape[0] == 0:\n                continue\n\n            # Current expert tokens\n            current_hidden_states = hidden_states[idx]\n\n            # Expert forward\n            current_hidden_states = expert_layer(current_hidden_states)\n\n            # Weighted by routing scores\n            current_hidden_states *= routing_weights[idx, top_x, None]\n\n            # Accumulate\n            final_hidden_states.index_add_(0, idx, current_hidden_states)\n\n        # Reshape\n        return final_hidden_states.view(batch_size, sequence_length, hidden_dim)\n```\n\n### PR-MoE (Pyramid-Residual-MoE)\n\n```bash\n# DeepSpeed PR-MoE: 3x better parameter efficiency\ndeepspeed pretrain_gpt_moe.py \\\n  --num-layers 24 \\\n  --hidden-size 1024 \\\n  --num-attention-heads 16 \\\n  --num-experts \"[128, 64, 32, 16]\" \\\n  --mlp-type residual \\\n  --moe-expert-parallel-size 4 \\\n  --moe-loss-coeff 0.01 \\\n  --fp16\n```\n\n## Best Practices\n\n### 1. Expert Count Selection\n\n```python\n# Rule of thumb: More experts = more capacity, but diminishing returns\n# Typical configurations:\n# - Small models (1B-7B): 8-16 experts\n# - Medium models (7B-30B): 8-64 experts\n# - Large models (30B+): 64-256 experts\n\n# Example: Mixtral 8x7B\n# Total params: 47B (8 experts × 7B each)\n# Active params: 13B (2 experts × 7B, top-2 routing)\n# Efficiency: 47B capacity with 13B compute\n```\n\n### 2. Capacity Factor Tuning\n\n```python\n# Capacity = (tokens_per_batch \u002F num_experts) * capacity_factor\n\n# Training: Lower capacity (faster, drops some tokens)\ntrain_capacity_factor = 1.25  # 25% buffer\n\n# Evaluation: Higher capacity (no dropping)\neval_capacity_factor = 2.0    # 100% buffer\n\n# Formula:\nexpert_capacity = int((seq_len * batch_size \u002F num_experts) * capacity_factor)\n```\n\n### 3. Learning Rate Guidelines\n\n```python\n# MoE models need lower LR than dense models\n# - Dense model: lr = 6e-4\n# - MoE model: lr = 1e-4 (3-6× lower)\n\n# Also extend decay schedule\ndense_lr_decay_iters = 300000\nmoe_lr_decay_iters = 500000  # 1.5-2× longer\n```\n\n### 4. Loss Coefficient Tuning\n\n```python\n# Start with standard values\nmoe_loss_coeff = 0.01    # Auxiliary loss (load balancing)\nrouter_z_loss_coeff = 0.001  # Router entropy (stability)\n\n# If load imbalance persists, increase aux loss\nif max_expert_usage \u002F min_expert_usage > 2.0:\n    moe_loss_coeff = 0.1  # Stronger load balancing\n\n# If training unstable, increase z-loss\nif grad_norm > 10.0:\n    router_z_loss_coeff = 0.01\n```\n\n### 5. Avoid Common Pitfalls\n\n```python\n# ❌ Bad: Using same LR as dense model\noptimizer = Adam(model.parameters(), lr=6e-4)\n\n# ✅ Good: Lower LR for MoE\noptimizer = Adam([\n    {'params': model.non_moe_params, 'lr': 6e-4},\n    {'params': model.moe_params, 'lr': 1e-4}\n])\n\n# ❌ Bad: No load balancing\nloss = lm_loss\n\n# ✅ Good: Add auxiliary loss\nloss = lm_loss + 0.01 * aux_loss + 0.001 * z_loss\n\n# ❌ Bad: Too many experts for small dataset\nnum_experts = 128  # Overfitting risk\n\n# ✅ Good: Match experts to data diversity\nnum_experts = 8  # Better for small datasets\n```\n\n## Inference Optimization\n\n### Sparse Inference\n\n```python\n# Only activate top-k experts (huge memory savings)\n@torch.no_grad()\ndef moe_inference(x, model, top_k=2):\n    \"\"\"Sparse MoE inference: only load k experts.\"\"\"\n    # Router\n    gate_logits = model.gate(x)\n    topk_scores, topk_indices = torch.topk(\n        torch.softmax(gate_logits, dim=-1),\n        k=top_k,\n        dim=-1\n    )\n\n    # Load and run only top-k experts\n    output = torch.zeros_like(x)\n    for i in range(top_k):\n        expert_idx = topk_indices[:, i]\n        # Load expert from disk\u002Foffload if needed\n        expert = model.load_expert(expert_idx)\n        output += topk_scores[:, i:i+1] * expert(x)\n\n    return output\n```\n\n## Resources\n\n- **DeepSpeed MoE Tutorial**: https:\u002F\u002Fwww.deepspeed.ai\u002Ftutorials\u002Fmixture-of-experts-nlg\u002F\n- **Mixtral Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.04088\n- **Switch Transformers**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2101.03961\n- **HuggingFace MoE Guide**: https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fmoe\n- **NVIDIA MoE Blog**: https:\u002F\u002Fdeveloper.nvidia.com\u002Fblog\u002Fapplying-mixture-of-experts-in-llm-architectures\u002F\n\n## See Also\n\n- `references\u002Farchitectures.md` - MoE model architectures (Mixtral, Switch, DeepSeek-V3)\n- `references\u002Ftraining.md` - Advanced training techniques and optimization\n- `references\u002Finference.md` - Production deployment and serving patterns\n\n\n",{"name":5,"tags":23,"author":35,"license":14,"version":36,"description":7,"dependencies":37},[24,25,26,27,28,29,30,31,32,33,34],"Emerging Techniques","MoE","Mixture Of Experts","Sparse Models","DeepSpeed","Expert Parallelism","Mixtral","DeepSeek","Routing","Load Balancing","Efficient Training","Orchestra Research","1.0.0",[38,39,40,41],"deepspeed","transformers","torch","accelerate","main","e3744adb7654df443f12927f3e001252b3a26b6d","cli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-moe-training","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates","https:\u002F\u002Fgithub.com\u002FKrismithReddy12","2026-07-29 08:15:10",{"source":8,"skillName":5,"commands":49,"prompt":59,"zipAvailable":60},[50,53,56],{"tool":51,"command":52},"npx","npx skills add KrismithReddy12\u002Fclaude-code-templates --skill moe-training",{"tool":54,"command":55},"bunx","bunx skills add KrismithReddy12\u002Fclaude-code-templates --skill moe-training",{"tool":57,"command":58},"pnpm","pnpm dlx skills add KrismithReddy12\u002Fclaude-code-templates --skill moe-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-moe-training\u002FSKILL.md",true,[62,68,72],{"path":63,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":66,"sourceHtmlUrl":67},"references\u002Farchitectures.md","REFERENCE","参考文档",14044,"https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-moe-training\u002Freferences\u002Farchitectures.md",{"path":69,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":70,"sourceHtmlUrl":71},"references\u002Finference.md",8239,"https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-moe-training\u002Freferences\u002Finference.md",{"path":73,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Ftraining.md",9707,"https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-moe-training\u002Freferences\u002Ftraining.md",[77,84,93,100,107,114],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":18},"01M79YMXJ07S2BQVB7JAM3657Q","zapier-make-patterns","No-code automation democratizes workflow building. Zapier and Make (formerly Integromat) let non-developers automate business processes without writing code. But no-code doesn't mean no-complexity - these platforms have their own patterns, pitfalls, and breaking points.  This skill covers when to use which platform, how to build reliable automations, and when to graduate to code-based solutions. Key insight: Zapier optimizes for simplicity and integrations (7000+ apps), Make optimizes for power","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fzapier-make-patterns\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":88,"licenseFileUrl":89,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":18},"01M79YMVPMK3Y4JDP2KDJKQ2ZD","yeet","Use only when the user explicitly asks to stage, commit, push, and open a GitHub pull request in one flow using the GitHub CLI (`gh`).","Apache-2.0","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fyeet\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fyeet\u002FSKILL.md",[],[],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":18},"01M79YMMQ4XXMPJHDQHY8FR7ME","workflow-automation","Workflow automation is the infrastructure that makes AI agents reliable. Without durable execution, a network hiccup during a 10-step payment flow means lost money and angry customers. With it, workflows resume exactly where they left off.  This skill covers the platforms (n8n, Temporal, Inngest) and patterns (sequential, parallel, orchestrator-worker) that turn brittle scripts into production-grade automation.  Key insight: The platforms make different tradeoffs. n8n optimizes for accessibility","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fworkflow-automation\u002FSKILL.md",[],[],{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":18},"01M79YMHQY3CP1XXMQ00TGQFAW","trigger-dev","Trigger.dev expert for background jobs, AI workflows, and reliable async execution with excellent developer experience and TypeScript-first design. Use when: trigger.dev, trigger dev, background task, ai background job, long running task.","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Ftrigger-dev\u002FSKILL.md",[],[],{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":111,"categories":112,"keywords":113,"publishedAt":18},"01M79YME4Z3DRW0MT6BNYW1D7P","slack-automation","Automate Slack workspace operations including messaging, search, channel management, and reaction workflows through Composio's Slack toolkit.","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fslack-automation\u002FSKILL.md",[],[],{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":18},"01M79YMAX7NFYDFXFZH2TJS7SS","planning-with-files","Transforms workflow to use Manus-style persistent markdown files for planning, progress tracking, and knowledge storage. Use when starting complex tasks, multi-step projects, research tasks, or when the user mentions planning, organizing work, tracking progress, or wants structured output.","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fplanning-with-files\u002FSKILL.md",[],[]]