[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EDR754KT0E9GG9T4SFTRF9":3,"skill-same:01M6EDR754KT0E9GG9T4SFTRF9":45},{"skillNo":4,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":20,"frontMatter":21,"refName":22,"commitSha":23,"skillDirectory":24,"forksCount":25,"repositoryHtmlUrl":26,"repositoryHomepageUrl":27,"ownerHtmlUrl":28,"ownerBlogUrl":6,"ownerDescription":29,"updatedAt":30,"install":31,"resources":44},"01M6EDR754KT0E9GG9T4SFTRF9","miles",null,"ComeOnOliver\u002Fskillshub","ComeOnOliver","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F43215937?v=4","01M6E5JNFECJAAVFE0W7SMWWD2","Haoran Shu","01M6E5JNNDQS7AJZDYFQ8TPZCK",67,"MIT","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002FOrchestra-Research\u002FAI-Research-SKILLs\u002Fmiles\u002FSKILL.md",[],[5],"2026-11-10 10:12:30",false,"# miles: Enterprise-Grade RL for Large-Scale Model Training\n\nmiles is a high-performance, enterprise-ready RL framework optimized for large-scale model post-training. Built as a production fork of slime, it addresses critical challenges in MoE training stability, low-precision training, and train-inference alignment.\n\n## When to Use miles\n\n**Choose miles when you need:**\n- Training 1TB+ MoE models (DeepSeek V3, Qwen3-MoE)\n- FP8 or INT4 quantization-aware training\n- Bit-wise identical train-inference alignment\n- Speculative RL for maximum throughput\n- Production stability with enterprise support\n\n**Consider alternatives when:**\n- You want the research-grade original → use **slime**\n- You need flexible backend swapping → use **verl**\n- You want PyTorch-native abstractions → use **torchforge**\n\n## Key Features\n\n### Low-Precision Training\n- **Unified FP8**: End-to-end FP8 for both inference and training\n- **INT4 QAT**: 1TB models on single-machine VRAM (H200)\n- **Rollout Routing Replay (R3)**: Bit-wise expert alignment for MoE\n\n### Performance Optimizations\n- **Speculative RL**: 25%+ rollout speedup with online SFT draft models\n- **Zero-Copy Weight Sync**: CUDA IPC zero-copy mapping\n- **Partial Rollout**: Recycle half-finished trajectories\n\n### Train-Inference Alignment\n- **TIS\u002FMIS**: Truncated\u002FMasked Importance Sampling for off-policy correction\n- **Kernel-level optimization**: FlashAttention-3, DeepGEMM integration\n\n## Installation\n\n```bash\n# Recommended: Docker\ndocker pull radixark\u002Fmiles:latest\ndocker run --rm --gpus all --ipc=host --shm-size=16g \\\n  -it radixark\u002Fmiles:latest \u002Fbin\u002Fbash\n\n# From source\ngit clone https:\u002F\u002Fgithub.com\u002Fradixark\u002Fmiles.git\ncd miles\npip install -r requirements.txt\npip install -e .\n```\n\n## Quick Start\n\nmiles inherits slime's configuration system. Basic training:\n\n```bash\npython train.py \\\n    --advantage-estimator grpo \\\n    --model-name qwen3-30b-a3b \\\n    --hf-checkpoint \u002Fpath\u002Fto\u002Fqwen3-30b-a3b-hf \\\n    --rollout-batch-size 512 \\\n    --n-samples-per-prompt 8\n```\n\n---\n\n## Workflow 1: Large MoE Training\n\nUse this workflow for training large MoE models like DeepSeek V3 or Qwen3-MoE.\n\n### Prerequisites Checklist\n- [ ] H100\u002FH200 GPUs with FP8 support\n- [ ] MoE model (DeepSeek V3, Qwen3-MoE)\n- [ ] Docker environment with miles\n\n### Step 1: Environment Setup\n\n```bash\n# FP8 block scaling (recommended for stability)\nexport NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1\nexport CUDA_DEVICE_MAX_CONNECTIONS=1\n```\n\n### Step 2: Configure Training\n\n```bash\npython train.py \\\n    --actor-num-gpus-per-node 8 \\\n    --rollout-num-gpus 8 \\\n    --hf-checkpoint \u002Fpath\u002Fto\u002Fdeepseek-v3 \\\n    --advantage-estimator grpo \\\n    --tensor-model-parallel-size 8 \\\n    --expert-model-parallel-size 4 \\\n    --prompt-data \u002Fpath\u002Fto\u002Fdata.jsonl \\\n    --num-rollout 3000\n```\n\n### Verification Checklist\n- [ ] Model loads without errors\n- [ ] Routing decisions are consistent\n- [ ] No NaN\u002FInf in loss values\n\n---\n\n## Workflow 2: Speculative RL Training\n\nUse this workflow for maximum rollout throughput with EAGLE speculative decoding.\n\n### How Speculative RL Works\n\n1. Small draft model generates candidate tokens\n2. Target model verifies in parallel\n3. Draft model updated via online SFT to track policy\n\n### Step 1: Enable Speculative Decoding\n\nmiles supports EAGLE speculative decoding via SGLang:\n\n```bash\npython train.py \\\n    --actor-num-gpus-per-node 8 \\\n    --hf-checkpoint \u002Fpath\u002Fto\u002Ftarget-model \\\n    --sglang-speculative-algorithm EAGLE \\\n    --sglang-speculative-num-steps 3 \\\n    --sglang-speculative-eagle-topk 1 \\\n    --sglang-speculative-num-draft-tokens 4 \\\n    --sglang-speculative-draft-model-path \u002Fpath\u002Fto\u002Fdraft-model \\\n    --advantage-estimator grpo \\\n    --prompt-data \u002Fpath\u002Fto\u002Fdata.jsonl\n```\n\n### Step 2: Enable Online MTP Training (Optional)\n\nFor online SFT of draft model during training:\n\n```bash\n--mtp-num-layers 1 \\\n--enable-mtp-training \\\n--mtp-loss-scaling-factor 0.2\n```\n\n**Note**: Online MTP training requires a torch dist checkpoint with MTP weights. Add `--mtp-num-layers 1` during checkpoint conversion from HuggingFace.\n\n### Expected Speedup\n\n- **Standard rollout**: Baseline\n- **Speculative RL**: 25-40% faster rollout\n- **With partial rollout**: Additional 10-15% throughput\n\n---\n\n## Configuration Reference\n\nmiles inherits all slime arguments. See [slime API Reference](..\u002Fslime\u002Freferences\u002Fapi-reference.md) for the complete list.\n\n### Cluster Resources (from slime)\n\n```bash\n--actor-num-nodes 1\n--actor-num-gpus-per-node 8\n--rollout-num-gpus 8\n--rollout-num-gpus-per-engine 2\n--colocate\n```\n\n### Megatron Parallelism (from slime)\n\n```bash\n--tensor-model-parallel-size 8\n--pipeline-model-parallel-size 2\n--expert-model-parallel-size 4    # MoE expert parallelism\n```\n\n### Speculative Decoding (miles-specific)\n\n```bash\n--sglang-speculative-algorithm EAGLE\n--sglang-speculative-num-steps 3\n--sglang-speculative-eagle-topk 1\n--sglang-speculative-num-draft-tokens 4\n--sglang-enable-draft-weights-cpu-backup\n--sglang-speculative-draft-model-path \u002Fyour\u002Fdraft\u002Fmodel\u002Fpath\n```\n\n### Online MTP Training (miles-specific)\n\n```bash\n--mtp-num-layers 1\n--enable-mtp-training\n--mtp-loss-scaling-factor 0.2\n```\n\n---\n\n## Key Features (Conceptual)\n\nThe following features are documented in miles but specific CLI flags may vary. Consult the miles repository for latest configuration.\n\n### Unified FP8 Pipeline\n\nEnd-to-end FP8 sampling and training that eliminates quantization-induced discrepancy causing RL collapse in MoE models.\n\n### Rollout Routing Replay (R3)\n\nRecords expert routing decisions during SGLang inference and replays them during Megatron training for bit-wise expert alignment.\n\n**How R3 Works**:\n1. During SGLang inference, expert routing decisions are recorded\n2. Routing decisions stored in `sample.rollout_routed_experts`\n3. During Megatron training, routing is replayed instead of recomputed\n4. Ensures identical expert selection between train and inference\n\n### INT4 Quantization-Aware Training\n\nEnables single-machine deployment of 1TB+ models (e.g., on H200).\n\n**Memory Savings with INT4**:\n\n| Model Size | BF16 VRAM | INT4 VRAM | Reduction |\n|------------|-----------|-----------|-----------|\n| 70B | 140GB | 45GB | 3.1x |\n| 235B | 470GB | 150GB | 3.1x |\n| 671B | 1.3TB | 420GB | 3.1x |\n\n### Train-Inference Alignment\n\nmiles achieves \"exactly 0 KL divergence\" between training and inference through:\n- Flash Attention 3\n- DeepGEMM\n- Batch-invariant kernels from Thinking Machines Lab\n- `torch.compile` integration\n\n---\n\n## Sample Data Structure\n\nmiles uses the same `Sample` dataclass as slime with the `rollout_routed_experts` field for MoE routing replay:\n\n```python\n@dataclass\nclass Sample:\n    prompt: str | list[dict]\n    tokens: list[int]\n    response: str\n    reward: float | dict\n    loss_mask: list[int]\n    status: Status\n    metadata: dict\n    rollout_log_probs: list[float]\n    rollout_routed_experts: list[list[int]]  # MoE routing for R3\n```\n\nSee [slime API Reference](..\u002Fslime\u002Freferences\u002Fapi-reference.md) for the complete Sample definition.\n\n---\n\n## Common Issues and Solutions\n\n### Issue: FP8 Training Collapse\n\n**Symptoms**: Loss explodes, NaN values\n\n**Solutions**:\n- Use block scaling: `export NVTE_FP8_BLOCK_SCALING_FP32_SCALES=1`\n- Reduce learning rate: `--lr 5e-7`\n- Ensure MoE routing is consistent between train\u002Finference\n\n### Issue: Speculative Draft Drift\n\n**Symptoms**: Low acceptance rate over time\n\n**Solutions**:\n- Enable online MTP training to keep draft model aligned\n- Reduce speculative steps: `--sglang-speculative-num-steps 2`\n- Use CPU backup: `--sglang-enable-draft-weights-cpu-backup`\n\n### Issue: Train-Inference Mismatch\n\n**Symptoms**: Policy divergence, reward collapse\n\n**Solutions**:\n- Use TIS for off-policy correction: `--use-tis --tis-threshold 0.9`\n- Verify log probs match between SGLang and Megatron\n- Enable R3 for MoE models\n\n---\n\n## Supported Models\n\n| Family | Models | MoE Support |\n|--------|--------|-------------|\n| DeepSeek | R1, V3, V3.2 | Full |\n| Qwen | 2, 2.5, 3 (including MoE) | Full |\n| Llama | 3, 3.1, 3.3, 4 | Dense only |\n| Gemma | 2, 3, 3N | Dense only |\n| GLM | 4.5, 4.6, 4.7 | Dense only |\n| MiniMax | M2, M2.1 | Full |\n\n---\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fradixark\u002Fmiles\n- **Introduction Blog**: https:\u002F\u002Flmsys.org\u002Fblog\u002F2025-11-19-miles\u002F\n- **Slime (upstream)**: https:\u002F\u002Fgithub.com\u002FTHUDM\u002Fslime\n- **SGLang**: https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang\n",{},"main","def8531e65114c0fca8fb8551c1871ee0eed705c","skills\u002FOrchestra-Research\u002FAI-Research-SKILLs\u002Fmiles",24,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub","https:\u002F\u002Fskillshub.wtf","https:\u002F\u002Fgithub.com\u002FComeOnOliver","AI & Blockchain Engineer\r\n","2026-06-24 22:07:19",{"source":7,"skillName":5,"commands":32,"prompt":42,"zipAvailable":43},[33,36,39],{"tool":34,"command":35},"npx","npx skills add ComeOnOliver\u002Fskillshub --skill miles",{"tool":37,"command":38},"bunx","bunx skills add ComeOnOliver\u002Fskillshub --skill miles",{"tool":40,"command":41},"pnpm","pnpm dlx skills add ComeOnOliver\u002Fskillshub --skill miles","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002FOrchestra-Research\u002FAI-Research-SKILLs\u002Fmiles\u002FSKILL.md",true,[],[46,60,70,79,85,92],{"skillNo":47,"name":48,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":49,"categories":50,"keywords":56,"publishedAt":18},"01M6FKJN13GMG4KY57J3QD8GY5","model-hierarchy-skill","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzscole\u002Fmodel-hierarchy-skill\u002Fmodel-hierarchy-skill\u002FSKILL.md",[51],{"categoryNo":52,"categoryName":53,"taxonomyType":54,"taxonomyTypeDesc":55},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[57,58,59],"model","model hierarchy","model hierarchy skill",{"skillNo":61,"name":62,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":63,"categories":64,"keywords":66,"publishedAt":18},"01M6FKJJJ6GCAMR4PPPRD12M5X","zoom-mcp","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-mcp\u002FSKILL.md",[65],{"categoryNo":52,"categoryName":53,"taxonomyType":54,"taxonomyTypeDesc":55},[67,68,69],"MCP","zoom","zoom mcp",{"skillNo":71,"name":72,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":18},"01M6FKJE3KYT3KPEZMEEHYKA6K","zoom-apps-sdk","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-apps-sdk\u002FSKILL.md",[],[76,77,78,68],"sdk","zoom apps sdk","apps",{"skillNo":80,"name":81,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":18},"01M6FKJ9EDH9Q5VWDF8HAJRSP0","windows","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwindows\u002FSKILL.md",[],[81],{"skillNo":86,"name":87,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":91,"publishedAt":18},"01M6FKJ666BEEFC3AE9GJ72Q93","whiteboard","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwhiteboard\u002FSKILL.md",[90],{"categoryNo":52,"categoryName":53,"taxonomyType":54,"taxonomyTypeDesc":55},[87],{"skillNo":93,"name":94,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":18},"01M6FKJ21WFCDGTJQ2Q91T9DVW","websockets","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwebsockets\u002FSKILL.md",[],[94]]