[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6GC8VYYNW8N3E7FTDNXBBFS":3,"skill-same:01M6GC8VYYNW8N3E7FTDNXBBFS":78},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":56,"commitSha":57,"skillDirectory":58,"forksCount":59,"repositoryHtmlUrl":60,"repositoryHomepageUrl":6,"ownerHtmlUrl":61,"ownerBlogUrl":6,"ownerDescription":62,"updatedAt":63,"install":64,"resources":77},"01M6GC8VYYNW8N3E7FTDNXBBFS","slime-rl-training",null,"Provides guidance for LLM post-training with RL using slime, a Megatron+SGLang framework. Use when training GLM models, implementing custom data generation workflows, or needing tight Megatron-LM integration for RL scaling.","zapabob\u002FSkills","zapabob","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F130830376?v=4","01M6DBMYZAGRTFQDSQYW3W764P","峯岸 亮","01M6DBMZ3GKDSJ01NZ9XTZD7JD",2,"MIT","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Farchive\u002Fskill-md-reviewed\u002Fsha256\u002Fa6\u002Fa68cadec4f202cf951f1c0e2cc12b044900c38cd2accb5c3eb5d9e777b741193\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"training","slime","slime rl training","rl","with","workflows","Megatron-LM","megatron","glm","scaling","router","use","learning","Transformers","Ray","Post-Training","data","SGLang","GRPO","framework","2026-11-10 10:12:30",false,"---\r\nname: slime-rl-training\r\ndescription: Provides guidance for LLM post-training with RL using slime, a Megatron+SGLang framework. Use when training GLM models, implementing custom data generation workflows, or needing tight Megatron-LM integration for RL scaling.\r\nversion: 1.0.0\r\nauthor: Orchestra Research\r\nlicense: MIT\r\ndependencies: [sglang-router>=0.2.3, ray, torch>=2.0.0, transformers>=4.40.0]\r\nmetadata:\r\n  hermes:\r\n    tags: [Reinforcement Learning, Megatron-LM, SGLang, GRPO, Post-Training, GLM]\r\n\r\n---\r\n\r\n# slime: LLM Post-Training Framework for RL Scaling\r\n\r\nslime is an LLM post-training framework from Tsinghua's THUDM team, powering GLM-4.5, GLM-4.6, and GLM-4.7. It connects Megatron-LM for training with SGLang for high-throughput rollout generation.\r\n\r\n## When to Use slime\r\n\r\n**Choose slime when you need:**\r\n- Megatron-LM native training with SGLang inference\r\n- Custom data generation workflows with flexible data buffers\r\n- Training GLM, Qwen3, DeepSeek V3, or Llama 3 models\r\n- Research-grade framework with production backing (Z.ai)\r\n\r\n**Consider alternatives when:**\r\n- You need enterprise-grade stability features → use **miles**\r\n- You want flexible backend swapping → use **verl**\r\n- You need PyTorch-native abstractions → use **torchforge**\r\n\r\n## Key Features\r\n\r\n- **Training**: Megatron-LM with full parallelism support (TP, PP, DP, SP)\r\n- **Rollout**: SGLang-based high-throughput generation with router\r\n- **Data Buffer**: Flexible prompt management and sample storage\r\n- **Models**: GLM-4.x, Qwen3, DeepSeek V3\u002FR1, Llama 3\r\n\r\n## Architecture Overview\r\n\r\n```\r\n┌─────────────────────────────────────────────────────────┐\r\n│                    Data Buffer                          │\r\n│ - Prompt initialization and management                  │\r\n│ - Custom data generation and filtering                  │\r\n│ - Rollout sample storage                                │\r\n└─────────────┬───────────────────────────┬───────────────┘\r\n              │                           │\r\n┌─────────────▼───────────┐ ┌─────────────▼───────────────┐\r\n│ Training (Megatron-LM)  │ │ Rollout (SGLang + Router)   │\r\n│ - Actor model training  │ │ - Response generation       │\r\n│ - Critic (optional)     │ │ - Reward\u002Fverifier output    │\r\n│ - Weight sync to rollout│ │ - Multi-turn support        │\r\n└─────────────────────────┘ └─────────────────────────────┘\r\n```\r\n\r\n## Installation\r\n\r\n```bash\r\n# Recommended: Docker\r\ndocker pull slimerl\u002Fslime:latest\r\ndocker run --rm --gpus all --ipc=host --shm-size=16g \\\r\n  -it slimerl\u002Fslime:latest \u002Fbin\u002Fbash\r\n\r\n# Inside container\r\ncd \u002Froot\u002Fslime && pip install -e . --no-deps\r\n```\r\n\r\n### From Source\r\n\r\n```bash\r\ngit clone https:\u002F\u002Fgithub.com\u002FTHUDM\u002Fslime.git\r\ncd slime\r\npip install -r requirements.txt\r\npip install -e .\r\n```\r\n\r\n## Quick Start: GRPO Training\r\n\r\n```bash\r\n# Source model configuration\r\nsource scripts\u002Fmodels\u002Fqwen3-4B.sh\r\n\r\n# Launch training\r\npython train.py \\\r\n    --actor-num-nodes 1 \\\r\n    --actor-num-gpus-per-node 4 \\\r\n    --rollout-num-gpus 4 \\\r\n    --advantage-estimator grpo \\\r\n    --use-kl-loss --kl-loss-coef 0.001 \\\r\n    --rollout-batch-size 32 \\\r\n    --n-samples-per-prompt 8 \\\r\n    --global-batch-size 256 \\\r\n    --num-rollout 3000 \\\r\n    --prompt-data \u002Fpath\u002Fto\u002Fdata.jsonl \\\r\n    ${MODEL_ARGS[@]} ${CKPT_ARGS[@]}\r\n```\r\n\r\n---\r\n\r\n## Workflow 1: Standard GRPO Training\r\n\r\nUse this workflow for training reasoning models with group-relative advantages.\r\n\r\n### Prerequisites Checklist\r\n- [ ] Docker environment or Megatron-LM + SGLang installed\r\n- [ ] Model checkpoint (HuggingFace or Megatron format)\r\n- [ ] Training data in JSONL format\r\n\r\n### Step 1: Prepare Data\r\n\r\n```python\r\n# data.jsonl format\r\n{\"prompt\": \"What is 2 + 2?\", \"label\": \"4\"}\r\n{\"prompt\": \"Solve: 3x = 12\", \"label\": \"x = 4\"}\r\n```\r\n\r\nOr with chat format:\r\n```python\r\n{\r\n    \"prompt\": [\r\n        {\"role\": \"system\", \"content\": \"You are a math tutor.\"},\r\n        {\"role\": \"user\", \"content\": \"What is 15 + 27?\"}\r\n    ],\r\n    \"label\": \"42\"\r\n}\r\n```\r\n\r\n### Step 2: Configure Model\r\n\r\nChoose a pre-configured model script:\r\n\r\n```bash\r\n# List available models\r\nls scripts\u002Fmodels\u002F\r\n# glm4-9B.sh, qwen3-4B.sh, qwen3-30B-A3B.sh, deepseek-v3.sh, llama3-8B.sh, ...\r\n\r\n# Source your model\r\nsource scripts\u002Fmodels\u002Fqwen3-4B.sh\r\n```\r\n\r\n### Step 3: Launch Training\r\n\r\n```bash\r\npython train.py \\\r\n    --actor-num-nodes 1 \\\r\n    --actor-num-gpus-per-node 8 \\\r\n    --rollout-num-gpus 8 \\\r\n    --advantage-estimator grpo \\\r\n    --use-kl-loss \\\r\n    --kl-loss-coef 0.001 \\\r\n    --prompt-data \u002Fpath\u002Fto\u002Ftrain.jsonl \\\r\n    --input-key prompt \\\r\n    --label-key label \\\r\n    --apply-chat-template \\\r\n    --rollout-batch-size 32 \\\r\n    --n-samples-per-prompt 8 \\\r\n    --global-batch-size 256 \\\r\n    --num-rollout 3000 \\\r\n    --save-interval 100 \\\r\n    --eval-interval 50 \\\r\n    ${MODEL_ARGS[@]}\r\n```\r\n\r\n### Step 4: Monitor Training\r\n- [ ] Check TensorBoard: `tensorboard --logdir outputs\u002F`\r\n- [ ] Verify reward curves are increasing\r\n- [ ] Monitor GPU utilization across nodes\r\n\r\n---\r\n\r\n## Workflow 2: Asynchronous Training\r\n\r\nUse async mode for higher throughput by overlapping rollout and training.\r\n\r\n### When to Use Async\r\n- Large models with long generation times\r\n- High GPU idle time in synchronous mode\r\n- Sufficient memory for buffering\r\n\r\n### Launch Async Training\r\n\r\n```bash\r\npython train_async.py \\\r\n    --actor-num-nodes 1 \\\r\n    --actor-num-gpus-per-node 8 \\\r\n    --rollout-num-gpus 8 \\\r\n    --advantage-estimator grpo \\\r\n    --async-buffer-size 4 \\\r\n    --prompt-data \u002Fpath\u002Fto\u002Ftrain.jsonl \\\r\n    ${MODEL_ARGS[@]}\r\n```\r\n\r\n### Async-Specific Parameters\r\n\r\n```bash\r\n--async-buffer-size 4        # Number of rollouts to buffer\r\n--update-weights-interval 2  # Sync weights every N rollouts\r\n```\r\n\r\n---\r\n\r\n## Workflow 3: Multi-Turn Agentic Training\r\n\r\nUse this workflow for training agents with tool use or multi-step reasoning.\r\n\r\n### Prerequisites\r\n- [ ] Custom generate function for multi-turn logic\r\n- [ ] Tool\u002Fenvironment interface\r\n\r\n### Step 1: Define Custom Generate Function\r\n\r\n```python\r\n# custom_generate.py\r\nasync def custom_generate(args, samples, evaluation=False):\r\n    \"\"\"Multi-turn generation with tool calling.\"\"\"\r\n    for sample in samples:\r\n        conversation = sample.prompt\r\n\r\n        for turn in range(args.max_turns):\r\n            # Generate response\r\n            response = await generate_single(conversation)\r\n\r\n            # Check for tool call\r\n            tool_call = extract_tool_call(response)\r\n            if tool_call:\r\n                tool_result = execute_tool(tool_call)\r\n                conversation.append({\"role\": \"assistant\", \"content\": response})\r\n                conversation.append({\"role\": \"tool\", \"content\": tool_result})\r\n            else:\r\n                break\r\n\r\n        sample.response = response\r\n        sample.reward = compute_reward(sample)\r\n\r\n    return samples\r\n```\r\n\r\n### Step 2: Launch with Custom Function\r\n\r\n```bash\r\npython train.py \\\r\n    --custom-generate-function-path custom_generate.py \\\r\n    --max-turns 5 \\\r\n    --prompt-data \u002Fpath\u002Fto\u002Fagent_data.jsonl \\\r\n    ${MODEL_ARGS[@]}\r\n```\r\n\r\nSee `examples\u002Fsearch-r1\u002F` for a complete multi-turn search example.\r\n\r\n---\r\n\r\n## Configuration Reference\r\n\r\n### Three Argument Categories\r\n\r\nslime uses three types of arguments:\r\n\r\n**1. Megatron Arguments** (passed directly):\r\n```bash\r\n--tensor-model-parallel-size 2\r\n--pipeline-model-parallel-size 1\r\n--num-layers 32\r\n--hidden-size 4096\r\n```\r\n\r\n**2. SGLang Arguments** (prefixed with `--sglang-`):\r\n```bash\r\n--sglang-mem-fraction-static 0.8\r\n--sglang-context-length 8192\r\n--sglang-log-level INFO\r\n```\r\n\r\n**3. slime Arguments**:\r\n```bash\r\n# Resource allocation\r\n--actor-num-nodes 1\r\n--actor-num-gpus-per-node 8\r\n--rollout-num-gpus 8\r\n--colocate  # Share GPUs between training\u002Finference\r\n\r\n# Data\r\n--prompt-data \u002Fpath\u002Fto\u002Fdata.jsonl\r\n--input-key prompt\r\n--label-key label\r\n\r\n# Training loop\r\n--num-rollout 3000\r\n--rollout-batch-size 32\r\n--n-samples-per-prompt 8\r\n--global-batch-size 256\r\n\r\n# Algorithm\r\n--advantage-estimator grpo  # or: gspo, ppo, reinforce_plus_plus\r\n--use-kl-loss\r\n--kl-loss-coef 0.001\r\n```\r\n\r\n### Key Constraints\r\n\r\n```\r\nrollout_batch_size × n_samples_per_prompt = global_batch_size × num_steps_per_rollout\r\n```\r\n\r\nExample: 32 × 8 = 256 × 1\r\n\r\n---\r\n\r\n## Data Buffer System\r\n\r\nslime's data buffer enables flexible data management:\r\n\r\n### Basic Data Source\r\n\r\n```python\r\nclass RolloutDataSource:\r\n    def get_samples(self, num_samples):\r\n        \"\"\"Fetch prompts from dataset.\"\"\"\r\n        return self.dataset.sample(num_samples)\r\n\r\n    def add_samples(self, samples):\r\n        \"\"\"Called after generation (no-op by default).\"\"\"\r\n        pass\r\n```\r\n\r\n### Buffered Data Source (Off-Policy)\r\n\r\n```python\r\nclass RolloutDataSourceWithBuffer(RolloutDataSource):\r\n    def __init__(self):\r\n        self.buffer = []\r\n\r\n    def add_samples(self, samples):\r\n        \"\"\"Store generated samples for reuse.\"\"\"\r\n        self.buffer.extend(samples)\r\n\r\n    def buffer_filter(self, args, buffer, num_samples):\r\n        \"\"\"Custom selection logic (prioritized, stratified, etc.).\"\"\"\r\n        return select_best(buffer, num_samples)\r\n```\r\n\r\n---\r\n\r\n## Common Issues and Solutions\r\n\r\n### Issue: SGLang Engine Crash\r\n\r\n**Symptoms**: Inference engine dies mid-training\r\n\r\n**Solutions**:\r\n```bash\r\n# Enable fault tolerance\r\n--use-fault-tolerance\r\n\r\n# Increase memory allocation\r\n--sglang-mem-fraction-static 0.85\r\n\r\n# Reduce batch size\r\n--rollout-batch-size 16\r\n```\r\n\r\n### Issue: Weight Sync Timeout\r\n\r\n**Symptoms**: Training hangs after rollout\r\n\r\n**Solutions**:\r\n```bash\r\n# Increase sync interval\r\n--update-weights-interval 5\r\n\r\n# Use colocated mode (no network transfer)\r\n--colocate\r\n```\r\n\r\n### Issue: OOM During Training\r\n\r\n**Symptoms**: CUDA OOM in backward pass\r\n\r\n**Solutions**:\r\n```bash\r\n# Enable gradient checkpointing\r\n--recompute-activations\r\n\r\n# Reduce micro-batch size\r\n--micro-batch-size 1\r\n\r\n# Enable sequence parallelism\r\n--sequence-parallel\r\n```\r\n\r\n### Issue: Slow Data Loading\r\n\r\n**Symptoms**: GPU idle during data fetch\r\n\r\n**Solutions**:\r\n```bash\r\n# Increase data workers\r\n--num-data-workers 4\r\n\r\n# Use streaming dataset\r\n--streaming-data\r\n```\r\n\r\n---\r\n\r\n## Supported Models\r\n\r\n| Model Family | Configurations |\r\n|--------------|----------------|\r\n| GLM | GLM-4.5, GLM-4.6, GLM-4.7, GLM-Z1-9B |\r\n| Qwen | Qwen3 (4B, 8B, 30B-A3B), Qwen3-MoE, Qwen2.5 |\r\n| DeepSeek | V3, V3.1, R1 |\r\n| Llama | Llama 3 (8B, 70B) |\r\n| Others | Kimi K2, Moonlight-16B |\r\n\r\nEach model has pre-configured scripts in `scripts\u002Fmodels\u002F`.\r\n\r\n---\r\n\r\n## Advanced Topics\r\n\r\n### Co-location Mode\r\n\r\nShare GPUs between training and inference to reduce memory:\r\n\r\n```bash\r\npython train.py \\\r\n    --colocate \\\r\n    --actor-num-gpus-per-node 8 \\\r\n    --sglang-mem-fraction-static 0.4 \\\r\n    ${MODEL_ARGS[@]}\r\n```\r\n\r\n### Custom Reward Model\r\n\r\n```python\r\n# custom_rm.py\r\nclass CustomRewardModel:\r\n    def __init__(self, model_path):\r\n        self.model = load_model(model_path)\r\n\r\n    def compute_reward(self, prompts, responses):\r\n        inputs = self.tokenize(prompts, responses)\r\n        scores = self.model(inputs)\r\n        return scores.tolist()\r\n```\r\n\r\n```bash\r\n--custom-rm-path custom_rm.py\r\n```\r\n\r\n### Evaluation Multi-Task\r\n\r\n```bash\r\n--eval-prompt-data aime \u002Fpath\u002Fto\u002Faime.jsonl \\\r\n--eval-prompt-data gsm8k \u002Fpath\u002Fto\u002Fgsm8k.jsonl \\\r\n--n-samples-per-eval-prompt 16\r\n```\r\n\r\n---\r\n\r\n## Resources\r\n\r\n- **Documentation**: https:\u002F\u002Fthudm.github.io\u002Fslime\u002F\r\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FTHUDM\u002Fslime\r\n- **Blog**: https:\u002F\u002Flmsys.org\u002Fblog\u002F2025-07-09-slime\u002F\r\n- **Examples**: See `examples\u002F` directory for 14+ worked examples\r\n","# slime: LLM Post-Training Framework for RL Scaling\r\n\r\nslime is an LLM post-training framework from Tsinghua's THUDM team, powering GLM-4.5, GLM-4.6, and GLM-4.7. It connects Megatron-LM for training with SGLang for high-throughput rollout generation.\r\n\r\n## When to Use slime\r\n\r\n**Choose slime when you need:**\r\n- Megatron-LM native training with SGLang inference\r\n- Custom data generation workflows with flexible data buffers\r\n- Training GLM, Qwen3, DeepSeek V3, or Llama 3 models\r\n- Research-grade framework with production backing (Z.ai)\r\n\r\n**Consider alternatives when:**\r\n- You need enterprise-grade stability features → use **miles**\r\n- You want flexible backend swapping → use **verl**\r\n- You need PyTorch-native abstractions → use **torchforge**\r\n\r\n## Key Features\r\n\r\n- **Training**: Megatron-LM with full parallelism support (TP, PP, DP, SP)\r\n- **Rollout**: SGLang-based high-throughput generation with router\r\n- **Data Buffer**: Flexible prompt management and sample storage\r\n- **Models**: GLM-4.x, Qwen3, DeepSeek V3\u002FR1, Llama 3\r\n\r\n## Architecture Overview\r\n\r\n```\r\n┌─────────────────────────────────────────────────────────┐\r\n│                    Data Buffer                          │\r\n│ - Prompt initialization and management                  │\r\n│ - Custom data generation and filtering                  │\r\n│ - Rollout sample storage                                │\r\n└─────────────┬───────────────────────────┬───────────────┘\r\n              │                           │\r\n┌─────────────▼───────────┐ ┌─────────────▼───────────────┐\r\n│ Training (Megatron-LM)  │ │ Rollout (SGLang + Router)   │\r\n│ - Actor model training  │ │ - Response generation       │\r\n│ - Critic (optional)     │ │ - Reward\u002Fverifier output    │\r\n│ - Weight sync to rollout│ │ - Multi-turn support        │\r\n└─────────────────────────┘ └─────────────────────────────┘\r\n```\r\n\r\n## Installation\r\n\r\n```bash\r\n# Recommended: Docker\r\ndocker pull slimerl\u002Fslime:latest\r\ndocker run --rm --gpus all --ipc=host --shm-size=16g \\\r\n  -it slimerl\u002Fslime:latest \u002Fbin\u002Fbash\r\n\r\n# Inside container\r\ncd \u002Froot\u002Fslime && pip install -e . --no-deps\r\n```\r\n\r\n### From Source\r\n\r\n```bash\r\ngit clone https:\u002F\u002Fgithub.com\u002FTHUDM\u002Fslime.git\r\ncd slime\r\npip install -r requirements.txt\r\npip install -e .\r\n```\r\n\r\n## Quick Start: GRPO Training\r\n\r\n```bash\r\n# Source model configuration\r\nsource scripts\u002Fmodels\u002Fqwen3-4B.sh\r\n\r\n# Launch training\r\npython train.py \\\r\n    --actor-num-nodes 1 \\\r\n    --actor-num-gpus-per-node 4 \\\r\n    --rollout-num-gpus 4 \\\r\n    --advantage-estimator grpo \\\r\n    --use-kl-loss --kl-loss-coef 0.001 \\\r\n    --rollout-batch-size 32 \\\r\n    --n-samples-per-prompt 8 \\\r\n    --global-batch-size 256 \\\r\n    --num-rollout 3000 \\\r\n    --prompt-data \u002Fpath\u002Fto\u002Fdata.jsonl \\\r\n    ${MODEL_ARGS[@]} ${CKPT_ARGS[@]}\r\n```\r\n\r\n---\r\n\r\n## Workflow 1: Standard GRPO Training\r\n\r\nUse this workflow for training reasoning models with group-relative advantages.\r\n\r\n### Prerequisites Checklist\r\n- [ ] Docker environment or Megatron-LM + SGLang installed\r\n- [ ] Model checkpoint (HuggingFace or Megatron format)\r\n- [ ] Training data in JSONL format\r\n\r\n### Step 1: Prepare Data\r\n\r\n```python\r\n# data.jsonl format\r\n{\"prompt\": \"What is 2 + 2?\", \"label\": \"4\"}\r\n{\"prompt\": \"Solve: 3x = 12\", \"label\": \"x = 4\"}\r\n```\r\n\r\nOr with chat format:\r\n```python\r\n{\r\n    \"prompt\": [\r\n        {\"role\": \"system\", \"content\": \"You are a math tutor.\"},\r\n        {\"role\": \"user\", \"content\": \"What is 15 + 27?\"}\r\n    ],\r\n    \"label\": \"42\"\r\n}\r\n```\r\n\r\n### Step 2: Configure Model\r\n\r\nChoose a pre-configured model script:\r\n\r\n```bash\r\n# List available models\r\nls scripts\u002Fmodels\u002F\r\n# glm4-9B.sh, qwen3-4B.sh, qwen3-30B-A3B.sh, deepseek-v3.sh, llama3-8B.sh, ...\r\n\r\n# Source your model\r\nsource scripts\u002Fmodels\u002Fqwen3-4B.sh\r\n```\r\n\r\n### Step 3: Launch Training\r\n\r\n```bash\r\npython train.py \\\r\n    --actor-num-nodes 1 \\\r\n    --actor-num-gpus-per-node 8 \\\r\n    --rollout-num-gpus 8 \\\r\n    --advantage-estimator grpo \\\r\n    --use-kl-loss \\\r\n    --kl-loss-coef 0.001 \\\r\n    --prompt-data \u002Fpath\u002Fto\u002Ftrain.jsonl \\\r\n    --input-key prompt \\\r\n    --label-key label \\\r\n    --apply-chat-template \\\r\n    --rollout-batch-size 32 \\\r\n    --n-samples-per-prompt 8 \\\r\n    --global-batch-size 256 \\\r\n    --num-rollout 3000 \\\r\n    --save-interval 100 \\\r\n    --eval-interval 50 \\\r\n    ${MODEL_ARGS[@]}\r\n```\r\n\r\n### Step 4: Monitor Training\r\n- [ ] Check TensorBoard: `tensorboard --logdir outputs\u002F`\r\n- [ ] Verify reward curves are increasing\r\n- [ ] Monitor GPU utilization across nodes\r\n\r\n---\r\n\r\n## Workflow 2: Asynchronous Training\r\n\r\nUse async mode for higher throughput by overlapping rollout and training.\r\n\r\n### When to Use Async\r\n- Large models with long generation times\r\n- High GPU idle time in synchronous mode\r\n- Sufficient memory for buffering\r\n\r\n### Launch Async Training\r\n\r\n```bash\r\npython train_async.py \\\r\n    --actor-num-nodes 1 \\\r\n    --actor-num-gpus-per-node 8 \\\r\n    --rollout-num-gpus 8 \\\r\n    --advantage-estimator grpo \\\r\n    --async-buffer-size 4 \\\r\n    --prompt-data \u002Fpath\u002Fto\u002Ftrain.jsonl \\\r\n    ${MODEL_ARGS[@]}\r\n```\r\n\r\n### Async-Specific Parameters\r\n\r\n```bash\r\n--async-buffer-size 4        # Number of rollouts to buffer\r\n--update-weights-interval 2  # Sync weights every N rollouts\r\n```\r\n\r\n---\r\n\r\n## Workflow 3: Multi-Turn Agentic Training\r\n\r\nUse this workflow for training agents with tool use or multi-step reasoning.\r\n\r\n### Prerequisites\r\n- [ ] Custom generate function for multi-turn logic\r\n- [ ] Tool\u002Fenvironment interface\r\n\r\n### Step 1: Define Custom Generate Function\r\n\r\n```python\r\n# custom_generate.py\r\nasync def custom_generate(args, samples, evaluation=False):\r\n    \"\"\"Multi-turn generation with tool calling.\"\"\"\r\n    for sample in samples:\r\n        conversation = sample.prompt\r\n\r\n        for turn in range(args.max_turns):\r\n            # Generate response\r\n            response = await generate_single(conversation)\r\n\r\n            # Check for tool call\r\n            tool_call = extract_tool_call(response)\r\n            if tool_call:\r\n                tool_result = execute_tool(tool_call)\r\n                conversation.append({\"role\": \"assistant\", \"content\": response})\r\n                conversation.append({\"role\": \"tool\", \"content\": tool_result})\r\n            else:\r\n                break\r\n\r\n        sample.response = response\r\n        sample.reward = compute_reward(sample)\r\n\r\n    return samples\r\n```\r\n\r\n### Step 2: Launch with Custom Function\r\n\r\n```bash\r\npython train.py \\\r\n    --custom-generate-function-path custom_generate.py \\\r\n    --max-turns 5 \\\r\n    --prompt-data \u002Fpath\u002Fto\u002Fagent_data.jsonl \\\r\n    ${MODEL_ARGS[@]}\r\n```\r\n\r\nSee `examples\u002Fsearch-r1\u002F` for a complete multi-turn search example.\r\n\r\n---\r\n\r\n## Configuration Reference\r\n\r\n### Three Argument Categories\r\n\r\nslime uses three types of arguments:\r\n\r\n**1. Megatron Arguments** (passed directly):\r\n```bash\r\n--tensor-model-parallel-size 2\r\n--pipeline-model-parallel-size 1\r\n--num-layers 32\r\n--hidden-size 4096\r\n```\r\n\r\n**2. SGLang Arguments** (prefixed with `--sglang-`):\r\n```bash\r\n--sglang-mem-fraction-static 0.8\r\n--sglang-context-length 8192\r\n--sglang-log-level INFO\r\n```\r\n\r\n**3. slime Arguments**:\r\n```bash\r\n# Resource allocation\r\n--actor-num-nodes 1\r\n--actor-num-gpus-per-node 8\r\n--rollout-num-gpus 8\r\n--colocate  # Share GPUs between training\u002Finference\r\n\r\n# Data\r\n--prompt-data \u002Fpath\u002Fto\u002Fdata.jsonl\r\n--input-key prompt\r\n--label-key label\r\n\r\n# Training loop\r\n--num-rollout 3000\r\n--rollout-batch-size 32\r\n--n-samples-per-prompt 8\r\n--global-batch-size 256\r\n\r\n# Algorithm\r\n--advantage-estimator grpo  # or: gspo, ppo, reinforce_plus_plus\r\n--use-kl-loss\r\n--kl-loss-coef 0.001\r\n```\r\n\r\n### Key Constraints\r\n\r\n```\r\nrollout_batch_size × n_samples_per_prompt = global_batch_size × num_steps_per_rollout\r\n```\r\n\r\nExample: 32 × 8 = 256 × 1\r\n\r\n---\r\n\r\n## Data Buffer System\r\n\r\nslime's data buffer enables flexible data management:\r\n\r\n### Basic Data Source\r\n\r\n```python\r\nclass RolloutDataSource:\r\n    def get_samples(self, num_samples):\r\n        \"\"\"Fetch prompts from dataset.\"\"\"\r\n        return self.dataset.sample(num_samples)\r\n\r\n    def add_samples(self, samples):\r\n        \"\"\"Called after generation (no-op by default).\"\"\"\r\n        pass\r\n```\r\n\r\n### Buffered Data Source (Off-Policy)\r\n\r\n```python\r\nclass RolloutDataSourceWithBuffer(RolloutDataSource):\r\n    def __init__(self):\r\n        self.buffer = []\r\n\r\n    def add_samples(self, samples):\r\n        \"\"\"Store generated samples for reuse.\"\"\"\r\n        self.buffer.extend(samples)\r\n\r\n    def buffer_filter(self, args, buffer, num_samples):\r\n        \"\"\"Custom selection logic (prioritized, stratified, etc.).\"\"\"\r\n        return select_best(buffer, num_samples)\r\n```\r\n\r\n---\r\n\r\n## Common Issues and Solutions\r\n\r\n### Issue: SGLang Engine Crash\r\n\r\n**Symptoms**: Inference engine dies mid-training\r\n\r\n**Solutions**:\r\n```bash\r\n# Enable fault tolerance\r\n--use-fault-tolerance\r\n\r\n# Increase memory allocation\r\n--sglang-mem-fraction-static 0.85\r\n\r\n# Reduce batch size\r\n--rollout-batch-size 16\r\n```\r\n\r\n### Issue: Weight Sync Timeout\r\n\r\n**Symptoms**: Training hangs after rollout\r\n\r\n**Solutions**:\r\n```bash\r\n# Increase sync interval\r\n--update-weights-interval 5\r\n\r\n# Use colocated mode (no network transfer)\r\n--colocate\r\n```\r\n\r\n### Issue: OOM During Training\r\n\r\n**Symptoms**: CUDA OOM in backward pass\r\n\r\n**Solutions**:\r\n```bash\r\n# Enable gradient checkpointing\r\n--recompute-activations\r\n\r\n# Reduce micro-batch size\r\n--micro-batch-size 1\r\n\r\n# Enable sequence parallelism\r\n--sequence-parallel\r\n```\r\n\r\n### Issue: Slow Data Loading\r\n\r\n**Symptoms**: GPU idle during data fetch\r\n\r\n**Solutions**:\r\n```bash\r\n# Increase data workers\r\n--num-data-workers 4\r\n\r\n# Use streaming dataset\r\n--streaming-data\r\n```\r\n\r\n---\r\n\r\n## Supported Models\r\n\r\n| Model Family | Configurations |\r\n|--------------|----------------|\r\n| GLM | GLM-4.5, GLM-4.6, GLM-4.7, GLM-Z1-9B |\r\n| Qwen | Qwen3 (4B, 8B, 30B-A3B), Qwen3-MoE, Qwen2.5 |\r\n| DeepSeek | V3, V3.1, R1 |\r\n| Llama | Llama 3 (8B, 70B) |\r\n| Others | Kimi K2, Moonlight-16B |\r\n\r\nEach model has pre-configured scripts in `scripts\u002Fmodels\u002F`.\r\n\r\n---\r\n\r\n## Advanced Topics\r\n\r\n### Co-location Mode\r\n\r\nShare GPUs between training and inference to reduce memory:\r\n\r\n```bash\r\npython train.py \\\r\n    --colocate \\\r\n    --actor-num-gpus-per-node 8 \\\r\n    --sglang-mem-fraction-static 0.4 \\\r\n    ${MODEL_ARGS[@]}\r\n```\r\n\r\n### Custom Reward Model\r\n\r\n```python\r\n# custom_rm.py\r\nclass CustomRewardModel:\r\n    def __init__(self, model_path):\r\n        self.model = load_model(model_path)\r\n\r\n    def compute_reward(self, prompts, responses):\r\n        inputs = self.tokenize(prompts, responses)\r\n        scores = self.model(inputs)\r\n        return scores.tolist()\r\n```\r\n\r\n```bash\r\n--custom-rm-path custom_rm.py\r\n```\r\n\r\n### Evaluation Multi-Task\r\n\r\n```bash\r\n--eval-prompt-data aime \u002Fpath\u002Fto\u002Faime.jsonl \\\r\n--eval-prompt-data gsm8k \u002Fpath\u002Fto\u002Fgsm8k.jsonl \\\r\n--n-samples-per-eval-prompt 16\r\n```\r\n\r\n---\r\n\r\n## Resources\r\n\r\n- **Documentation**: https:\u002F\u002Fthudm.github.io\u002Fslime\u002F\r\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FTHUDM\u002Fslime\r\n- **Blog**: https:\u002F\u002Flmsys.org\u002Fblog\u002F2025-07-09-slime\u002F\r\n- **Examples**: See `examples\u002F` directory for 14+ worked examples\r\n",{"name":5,"author":44,"license":15,"version":45,"metadata":46,"description":7,"dependencies":51},"Orchestra Research","1.0.0",{"hermes":47},{"tags":48},[49,25,36,37,34,50],"Reinforcement Learning","GLM",[52,53,54,55],"sglang-router>=0.2.3","ray","torch>=2.0.0","transformers>=4.40.0","main","0d1434f89f4bc6b07e9d272accd8c1013768820f","archive\u002Fskill-md-reviewed\u002Fsha256\u002Fa6\u002Fa68cadec4f202cf951f1c0e2cc12b044900c38cd2accb5c3eb5d9e777b741193",0,"https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills","https:\u002F\u002Fgithub.com\u002Fzapabob","The Open University of JAPAN　2019-now\r\n\r\nTokyo University of Science　2007-2010","2026-10-26 16:29:01",{"source":8,"skillName":5,"commands":65,"prompt":75,"zipAvailable":76},[66,69,72],{"tool":67,"command":68},"npx","npx skills add zapabob\u002FSkills --skill slime-rl-training",{"tool":70,"command":71},"bunx","bunx skills add zapabob\u002FSkills --skill slime-rl-training",{"tool":73,"command":74},"pnpm","pnpm dlx skills add zapabob\u002FSkills --skill slime-rl-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Farchive\u002Fskill-md-reviewed\u002Fsha256\u002Fa6\u002Fa68cadec4f202cf951f1c0e2cc12b044900c38cd2accb5c3eb5d9e777b741193\u002FSKILL.md",true,[],[79,117,152,177,204,220],{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":98,"publishedAt":39},"01M6GJTE1GESGGF9K028Y7753E","xurl","X\u002FTwitter via xurl CLI: post, search, DM, media, v2 API.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fxurl\u002Fvariants\u002Fhermes-agent\u002FSKILL.md",[85,90,93],{"categoryNo":86,"categoryName":87,"taxonomyType":88,"taxonomyTypeDesc":89},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",{"categoryNo":91,"categoryName":92,"taxonomyType":88,"taxonomyTypeDesc":89},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":94,"categoryName":95,"taxonomyType":96,"taxonomyTypeDesc":97},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[81,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,56,114,115,116],"openclaw","macos","cli","github","linux","windows","official","search","https","social-media","api","Media","hermes","hermes agent","twitter","social","via","post",{"skillNo":118,"name":119,"nameZh":6,"description":120,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":121,"categories":122,"keywords":134,"publishedAt":39},"01M6GJT8S1PXQ8WPXT1HSNBWH2","writing-plans","Write implementation plans: bite-sized tasks, paths, code.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fwriting-plans\u002Fvariants\u002Fhermes-agent\u002FSKILL.md",[123,126,129,130,131],{"categoryNo":124,"categoryName":125,"taxonomyType":88,"taxonomyTypeDesc":89},"01JBSEEDCAT000000000000015","视觉设计",{"categoryNo":127,"categoryName":128,"taxonomyType":88,"taxonomyTypeDesc":89},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":91,"categoryName":92,"taxonomyType":88,"taxonomyTypeDesc":89},{"categoryNo":94,"categoryName":95,"taxonomyType":96,"taxonomyTypeDesc":97},{"categoryNo":132,"categoryName":133,"taxonomyType":96,"taxonomyTypeDesc":97},"01JBSEEDCAT000000000000036","内容创作者",[135,136,137,138,100,139,103,140,104,141,142,143,144,145,146,147,148,149,150,151],"writing","writing plans","plans","design","review","workflow","documentation","tasks","development","code","planning","code-review","subagent","write","superpowers","paths","test-driven-development",{"skillNo":153,"name":154,"nameZh":6,"description":155,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":156,"categories":157,"keywords":161,"publishedAt":39},"01M6GJST0H5X7WHBJ3BZK0DVKH","whisper","OpenAI's general-purpose speech recognition model. Supports 99 languages, transcription, translation to English, and language identification. Six model sizes from tiny (39M params) to large (1550M params). Use for speech-to-text, podcast transcription, or multilingual audio processing. Best for robust, multilingual ASR.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fwhisper\u002Fvariants\u002Fhermes-agent-optional\u002FSKILL.md",[158],{"categoryNo":159,"categoryName":160,"taxonomyType":88,"taxonomyTypeDesc":89},"01JBSEEDCAT000000000000016","图像与多媒体",[154,100,103,162,163,164,32,165,166,167,168,169,30,170,171,172,173,174,175,176],"openai","speech","multimodal","ASR","Audio","text","model","speech recognition","general","multilingual","translation","english","identification","processing","Audio Processing",{"skillNo":178,"name":179,"nameZh":6,"description":180,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":181,"categories":182,"keywords":186,"publishedAt":39},"01M6GJSMRQ8EETZTYMG72P70EA","weights-and-biases","W&B: log ML experiments, sweeps, model registry, dashboards.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fweights-and-biases\u002Fvariants\u002Fhermes-agent\u002FSKILL.md",[183],{"categoryNo":184,"categoryName":185,"taxonomyType":88,"taxonomyTypeDesc":89},"01JBSEEDCAT000000000000013","分析与可视化",[187,100,188,189,104,190,191,192,193,194,195,168,196,197,198,199,200,201,202,203],"weights and biases","tensorflow","experiments","pytorch","visualization","registry","real-time","tuning","dashboards","experiment","log","time","Experiment Tracking","Model Registry","Hyperparameter Tuning","WandB","Real-Time Visualization",{"skillNo":205,"name":206,"nameZh":6,"description":207,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":208,"categories":209,"keywords":211,"publishedAt":39},"01M6GJS19KDPRY112JTR326S06","watchers","Poll RSS, JSON APIs, and GitHub with watermark dedup.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fwatchers\u002Fvariants\u002Fhermes-agent-optional\u002FSKILL.md",[210],{"categoryNo":91,"categoryName":92,"taxonomyType":88,"taxonomyTypeDesc":89},[206,100,212,102,103,213,214,215,216,217,218,111,112,219,23],"terminal","automation","http","monitoring","cron","json","rss","devops",{"skillNo":221,"name":222,"nameZh":6,"description":223,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":224,"categories":225,"keywords":228,"publishedAt":39},"01M6GJRHJD2A6Q830SWF9W48BC","serving-llms-vllm","vLLM: high-throughput LLM serving, OpenAI API, quantization.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fvllm\u002Fvariants\u002Fhermes-agent\u002FSKILL.md",[226,227],{"categoryNo":86,"categoryName":87,"taxonomyType":88,"taxonomyTypeDesc":89},{"categoryNo":94,"categoryName":95,"taxonomyType":96,"taxonomyTypeDesc":97},[229,230,231,103,232,162,233,234,235,32,236,109,237,238,239,240,241,242,243,244],"llms","vLLM","serving llms vllm","research","openai api","tensor","Inference Serving","Production","llm serving","High Throughput","PagedAttention","Continuous Batching","Quantization","Tensor Parallelism","high-throughput","throughput"]