[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EE5J9EPS7CCMAZX7JTP8ZG":3,"skill-same:01M6EE5J9EPS7CCMAZX7JTP8ZG":86},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":54,"commitSha":55,"skillDirectory":56,"forksCount":57,"repositoryHtmlUrl":58,"repositoryHomepageUrl":6,"ownerHtmlUrl":59,"ownerBlogUrl":6,"ownerDescription":60,"updatedAt":61,"install":62,"resources":75},"01M6EE5J9EPS7CCMAZX7JTP8ZG","verl-rl-training",null,"Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.","luokai0\u002Fai-agent-skills-by-luo-kai","luokai0","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F266954973?v=4","01M6E16VNT42JB8B3X3254QWJ7","luo kai","01M6E16VTFVP6S5XJF1V1MPV90",14,"MIT","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fverl\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"verl rl training","verl","training","rl","vLLM","volcano","algorithms","use","engine","learning","Transformers","llms","Ray","PPO","RLHF","Post-Training","GRPO","Distributed Training","infrastructure","distributed","2026-11-10 10:12:30",false,"---\nname: verl-rl-training\ndescription: Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.\nversion: 1.0.0\nauthor: luo-kai\nlicense: MIT\ntags: [Reinforcement Learning, RLHF, GRPO, PPO, Post-Training, Distributed Training]\ndependencies: [verl>=0.3.0, torch>=2.0.0, ray>=2.41.0, vllm>=0.8.2, transformers>=4.40.0]\n---\n\n# verl: Volcano Engine Reinforcement Learning for LLMs\n\nverl is a flexible, efficient, and production-ready RL training library for large language models from ByteDance's Seed team. It implements the HybridFlow framework (EuroSys 2025) and powers models like Doubao-1.5-pro achieving O1-level performance on math benchmarks.\n\n## When to Use verl\n\n**Choose verl when you need:**\n- Production-ready RL training at scale (tested up to 671B parameters)\n- Flexibility to swap backends (FSDP ↔ Megatron-LM ↔ vLLM ↔ SGLang)\n- Support for multiple RL algorithms (PPO, GRPO, RLOO, REINFORCE++, DAPO)\n- Multi-turn rollout with tool calling for agentic workflows\n- Vision-language model RL training\n\n**Consider alternatives when:**\n- You need Megatron-native training → use **slime** or **miles**\n- You want PyTorch-native abstractions with Monarch → use **torchforge**\n- You only need simple SFT\u002FDPO → use **TRL** or **Axolotl**\n\n## Key Features\n\n- **Training backends**: FSDP, FSDP2, Megatron-LM\n- **Rollout engines**: vLLM, SGLang, HuggingFace Transformers\n- **Algorithms**: PPO, GRPO, DAPO, RLOO, ReMax, REINFORCE++, SPIN, SPPO\n- **Models**: Qwen-3, Llama-3.1, DeepSeek, Gemma-2 (0.5B to 671B)\n- **Advanced**: LoRA RL, sequence parallelism, expert parallelism, multi-turn tools\n\n## Installation\n\n```bash\n# Option 1: pip install\npip install verl[vllm]  # or verl[sglang] for SGLang backend\n\n# Option 2: Docker (recommended for production)\ndocker pull verlai\u002Fverl:vllm011.latest\n\n# Option 3: From source\ngit clone https:\u002F\u002Fgithub.com\u002Fvolcengine\u002Fverl.git\ncd verl && pip install -e .[vllm,math]\n```\n\n## Quick Start: GRPO Training\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    algorithm.adv_estimator=grpo \\\n    data.train_files=~\u002Fdata\u002Fgsm8k\u002Ftrain.parquet \\\n    actor_rollout_ref.model.path=Qwen\u002FQwen2.5-7B \\\n    actor_rollout_ref.rollout.n=8 \\\n    actor_rollout_ref.actor.use_kl_loss=True \\\n    trainer.n_gpus_per_node=8\n```\n\n## Core Architecture\n\nverl uses a **HybridFlow** programming model separating control flow from computation:\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ Single-Process Controller (Ray)                         │\n│ - Orchestrates: rollout → reward → train → sync        │\n└─────────────────────┬───────────────────────────────────┘\n                      │\n┌─────────────────────▼───────────────────────────────────┐\n│ Multi-Process Workers                                   │\n│ ├── ActorRolloutRefWorker (policy + generation)        │\n│ ├── CriticWorker (value estimation, PPO only)          │\n│ └── RewardManager (model-based or rule-based rewards)  │\n└─────────────────────────────────────────────────────────┘\n```\n\n---\n\n## Workflow 1: Math Reasoning with GRPO\n\nUse this workflow for training reasoning models on math tasks like GSM8K or MATH.\n\n### Prerequisites Checklist\n- [ ] GPU cluster with 8+ GPUs (H100 recommended)\n- [ ] Dataset in parquet format with `prompt` and `reward_model` columns\n- [ ] Base model from HuggingFace Hub\n\n### Step 1: Prepare Dataset\n\n```python\nimport pandas as pd\n\ndata = [\n    {\n        \"prompt\": [{\"role\": \"user\", \"content\": \"What is 15 + 27?\"}],\n        \"reward_model\": {\"ground_truth\": \"42\"}\n    },\n    # ... more examples\n]\ndf = pd.DataFrame(data)\ndf.to_parquet(\"train.parquet\")\n```\n\n### Step 2: Define Reward Function\n\n```python\n# reward_function.py\nimport re\n\ndef compute_reward(responses, ground_truths):\n    rewards = []\n    for response, gt in zip(responses, ground_truths):\n        # Extract answer from response\n        match = re.search(r'\\\\boxed{([^}]+)}', response)\n        if match and match.group(1).strip() == gt.strip():\n            rewards.append(1.0)\n        else:\n            rewards.append(0.0)\n    return rewards\n```\n\n### Step 3: Create Training Config\n\n```yaml\n# config\u002Fgrpo_math.yaml\nalgorithm:\n  adv_estimator: grpo\n  gamma: 1.0\n  lam: 1.0\n\ndata:\n  train_files: \u002Fpath\u002Fto\u002Ftrain.parquet\n  val_files: \u002Fpath\u002Fto\u002Fval.parquet\n  train_batch_size: 256\n  max_prompt_length: 512\n  max_response_length: 2048\n\nactor_rollout_ref:\n  model:\n    path: Qwen\u002FQwen2.5-7B-Instruct\n  actor:\n    use_kl_loss: true\n    kl_loss_coef: 0.001\n    ppo_mini_batch_size: 64\n  rollout:\n    name: vllm\n    n: 8  # samples per prompt\n    temperature: 0.7\n    top_p: 0.95\n\ntrainer:\n  total_epochs: 3\n  n_gpus_per_node: 8\n  save_freq: 100\n```\n\n### Step 4: Launch Training\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    --config-path config \\\n    --config-name grpo_math \\\n    trainer.experiment_name=grpo_math_qwen7b\n```\n\n### Step 5: Monitor and Validate\n- [ ] Check WandB\u002FTensorBoard for loss curves\n- [ ] Verify reward is increasing over steps\n- [ ] Run evaluation on held-out test set\n\n---\n\n## Workflow 2: PPO with Critic Model\n\nUse this workflow when you need value-based advantage estimation (GAE).\n\n### Key Differences from GRPO\n- Requires separate critic model\n- Uses Generalized Advantage Estimation (GAE)\n- Better for tasks with dense rewards\n\n### Configuration\n\n```yaml\nalgorithm:\n  adv_estimator: gae  # Use GAE instead of GRPO\n  gamma: 0.99\n  lam: 0.95\n\ncritic:\n  model:\n    path: Qwen\u002FQwen2.5-7B-Instruct  # Can be same or different from actor\n  ppo_mini_batch_size: 64\n\nactor_rollout_ref:\n  actor:\n    use_kl_loss: true\n    kl_loss_coef: 0.02\n    clip_ratio: 0.2  # PPO clipping\n```\n\n### Launch with Critic\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    algorithm.adv_estimator=gae \\\n    critic.model.path=Qwen\u002FQwen2.5-7B-Instruct \\\n    trainer.n_gpus_per_node=8\n```\n\n---\n\n## Workflow 3: Large-Scale Training with Megatron\n\nUse this workflow for models >70B parameters or when you need expert parallelism.\n\n### Prerequisites\n- [ ] Install Megatron-LM bridge: `pip install mbridge`\n- [ ] Convert model to Megatron format\n- [ ] Multi-node cluster with NVLink\u002FInfiniBand\n\n### Configuration for 70B+ Models\n\n```yaml\nactor_rollout_ref:\n  model:\n    path: \u002Fpath\u002Fto\u002Fmegatron\u002Fcheckpoint\n    backend: megatron\n  actor:\n    strategy: megatron\n    tensor_model_parallel_size: 8\n    pipeline_model_parallel_size: 2\n  rollout:\n    name: vllm\n    tensor_parallel_size: 8\n```\n\n### Launch Multi-Node\n\n```bash\n# On head node\nray start --head --port=6379\n\n# On worker nodes\nray start --address='head_ip:6379'\n\n# Launch training\npython3 -m verl.trainer.main_ppo \\\n    trainer.nnodes=4 \\\n    trainer.n_gpus_per_node=8\n```\n\n---\n\n## Configuration Reference\n\n### Algorithm Selection\n\n| Algorithm | `adv_estimator` | Use Case |\n|-----------|-----------------|----------|\n| GRPO | `grpo` | Critic-free, math\u002Freasoning |\n| PPO\u002FGAE | `gae` | Dense rewards, value estimation |\n| REINFORCE++ | `reinforce_plus_plus` | Variance reduction |\n| RLOO | `rloo` | Leave-one-out baseline |\n| ReMax | `remax` | Maximum reward baseline |\n| OPO | `opo` | Optimal policy optimization |\n\n### Key Parameters\n\n```yaml\n# Rollout parameters\nactor_rollout_ref.rollout.n: 8              # Samples per prompt\nactor_rollout_ref.rollout.temperature: 0.7  # Sampling temperature\nactor_rollout_ref.rollout.top_p: 0.95       # Nucleus sampling\n\n# Training parameters\nactor_rollout_ref.actor.lr: 1e-6            # Learning rate\nactor_rollout_ref.actor.ppo_mini_batch_size: 64\nactor_rollout_ref.actor.clip_ratio: 0.2     # PPO clip range\n\n# KL control\nactor_rollout_ref.actor.use_kl_loss: true\nactor_rollout_ref.actor.kl_loss_coef: 0.001\nalgorithm.kl_ctrl.target_kl: 0.1            # For adaptive KL control\n```\n\n---\n\n## Common Issues and Solutions\n\n### Issue: OOM During Rollout\n\n**Symptoms**: CUDA out of memory during generation phase\n\n**Solutions**:\n```yaml\n# Reduce batch size\nactor_rollout_ref.rollout.log_prob_micro_batch_size: 4\n\n# Enable gradient checkpointing\nactor_rollout_ref.model.enable_gradient_checkpointing: true\n\n# Use FSDP2 with CPU offloading\nactor_rollout_ref.actor.strategy: fsdp2\nactor_rollout_ref.actor.fsdp_config.offload_policy: true\n```\n\n### Issue: Training Instability\n\n**Symptoms**: Loss spikes, reward collapse\n\n**Solutions**:\n```yaml\n# Reduce learning rate\nactor_rollout_ref.actor.lr: 5e-7\n\n# Increase KL penalty\nactor_rollout_ref.actor.kl_loss_coef: 0.01\n\n# Enable gradient clipping\nactor_rollout_ref.actor.max_grad_norm: 1.0\n```\n\n### Issue: Slow Weight Sync\n\n**Symptoms**: Long pauses between rollout and training\n\n**Solutions**:\n```bash\n# Use FSDP2 for faster resharding\nactor_rollout_ref.actor.strategy=fsdp2\n\n# Enable async weight transfer\ntrainer.async_weight_update=true\n```\n\n### Issue: vLLM Version Mismatch\n\n**Symptoms**: Import errors or generation failures\n\n**Solution**: Use compatible versions:\n```bash\npip install vllm>=0.8.5,\u003C=0.12.0\n# Avoid vLLM 0.7.x (known bugs)\n```\n\n---\n\n## Advanced Topics\n\n### Multi-Turn Tool Calling\n\nSee [references\u002Fmulti-turn.md](references\u002Fmulti-turn.md) for agentic workflows with tool use.\n\n### Vision-Language Models\n\n```yaml\nactor_rollout_ref:\n  model:\n    path: Qwen\u002FQwen2.5-VL-7B-Instruct\n  rollout:\n    name: vllm\n    enable_vision: true\n```\n\n### LoRA Training\n\n```yaml\nactor_rollout_ref:\n  actor:\n    lora:\n      enabled: true\n      r: 16\n      alpha: 32\n      target_modules: [\"q_proj\", \"v_proj\"]\n```\n\n---\n\n## Resources\n\n- **Documentation**: https:\u002F\u002Fverl.readthedocs.io\u002F\n- **Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2409.19256\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fvolcengine\u002Fverl\n- **Recipes**: https:\u002F\u002Fgithub.com\u002Fverl-project\u002Fverl-recipe (DAPO, GSPO, etc.)\n- **Community**: Slack at verl-project\n\n","# verl: Volcano Engine Reinforcement Learning for LLMs\n\nverl is a flexible, efficient, and production-ready RL training library for large language models from ByteDance's Seed team. It implements the HybridFlow framework (EuroSys 2025) and powers models like Doubao-1.5-pro achieving O1-level performance on math benchmarks.\n\n## When to Use verl\n\n**Choose verl when you need:**\n- Production-ready RL training at scale (tested up to 671B parameters)\n- Flexibility to swap backends (FSDP ↔ Megatron-LM ↔ vLLM ↔ SGLang)\n- Support for multiple RL algorithms (PPO, GRPO, RLOO, REINFORCE++, DAPO)\n- Multi-turn rollout with tool calling for agentic workflows\n- Vision-language model RL training\n\n**Consider alternatives when:**\n- You need Megatron-native training → use **slime** or **miles**\n- You want PyTorch-native abstractions with Monarch → use **torchforge**\n- You only need simple SFT\u002FDPO → use **TRL** or **Axolotl**\n\n## Key Features\n\n- **Training backends**: FSDP, FSDP2, Megatron-LM\n- **Rollout engines**: vLLM, SGLang, HuggingFace Transformers\n- **Algorithms**: PPO, GRPO, DAPO, RLOO, ReMax, REINFORCE++, SPIN, SPPO\n- **Models**: Qwen-3, Llama-3.1, DeepSeek, Gemma-2 (0.5B to 671B)\n- **Advanced**: LoRA RL, sequence parallelism, expert parallelism, multi-turn tools\n\n## Installation\n\n```bash\n# Option 1: pip install\npip install verl[vllm]  # or verl[sglang] for SGLang backend\n\n# Option 2: Docker (recommended for production)\ndocker pull verlai\u002Fverl:vllm011.latest\n\n# Option 3: From source\ngit clone https:\u002F\u002Fgithub.com\u002Fvolcengine\u002Fverl.git\ncd verl && pip install -e .[vllm,math]\n```\n\n## Quick Start: GRPO Training\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    algorithm.adv_estimator=grpo \\\n    data.train_files=~\u002Fdata\u002Fgsm8k\u002Ftrain.parquet \\\n    actor_rollout_ref.model.path=Qwen\u002FQwen2.5-7B \\\n    actor_rollout_ref.rollout.n=8 \\\n    actor_rollout_ref.actor.use_kl_loss=True \\\n    trainer.n_gpus_per_node=8\n```\n\n## Core Architecture\n\nverl uses a **HybridFlow** programming model separating control flow from computation:\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ Single-Process Controller (Ray)                         │\n│ - Orchestrates: rollout → reward → train → sync        │\n└─────────────────────┬───────────────────────────────────┘\n                      │\n┌─────────────────────▼───────────────────────────────────┐\n│ Multi-Process Workers                                   │\n│ ├── ActorRolloutRefWorker (policy + generation)        │\n│ ├── CriticWorker (value estimation, PPO only)          │\n│ └── RewardManager (model-based or rule-based rewards)  │\n└─────────────────────────────────────────────────────────┘\n```\n\n---\n\n## Workflow 1: Math Reasoning with GRPO\n\nUse this workflow for training reasoning models on math tasks like GSM8K or MATH.\n\n### Prerequisites Checklist\n- [ ] GPU cluster with 8+ GPUs (H100 recommended)\n- [ ] Dataset in parquet format with `prompt` and `reward_model` columns\n- [ ] Base model from HuggingFace Hub\n\n### Step 1: Prepare Dataset\n\n```python\nimport pandas as pd\n\ndata = [\n    {\n        \"prompt\": [{\"role\": \"user\", \"content\": \"What is 15 + 27?\"}],\n        \"reward_model\": {\"ground_truth\": \"42\"}\n    },\n    # ... more examples\n]\ndf = pd.DataFrame(data)\ndf.to_parquet(\"train.parquet\")\n```\n\n### Step 2: Define Reward Function\n\n```python\n# reward_function.py\nimport re\n\ndef compute_reward(responses, ground_truths):\n    rewards = []\n    for response, gt in zip(responses, ground_truths):\n        # Extract answer from response\n        match = re.search(r'\\\\boxed{([^}]+)}', response)\n        if match and match.group(1).strip() == gt.strip():\n            rewards.append(1.0)\n        else:\n            rewards.append(0.0)\n    return rewards\n```\n\n### Step 3: Create Training Config\n\n```yaml\n# config\u002Fgrpo_math.yaml\nalgorithm:\n  adv_estimator: grpo\n  gamma: 1.0\n  lam: 1.0\n\ndata:\n  train_files: \u002Fpath\u002Fto\u002Ftrain.parquet\n  val_files: \u002Fpath\u002Fto\u002Fval.parquet\n  train_batch_size: 256\n  max_prompt_length: 512\n  max_response_length: 2048\n\nactor_rollout_ref:\n  model:\n    path: Qwen\u002FQwen2.5-7B-Instruct\n  actor:\n    use_kl_loss: true\n    kl_loss_coef: 0.001\n    ppo_mini_batch_size: 64\n  rollout:\n    name: vllm\n    n: 8  # samples per prompt\n    temperature: 0.7\n    top_p: 0.95\n\ntrainer:\n  total_epochs: 3\n  n_gpus_per_node: 8\n  save_freq: 100\n```\n\n### Step 4: Launch Training\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    --config-path config \\\n    --config-name grpo_math \\\n    trainer.experiment_name=grpo_math_qwen7b\n```\n\n### Step 5: Monitor and Validate\n- [ ] Check WandB\u002FTensorBoard for loss curves\n- [ ] Verify reward is increasing over steps\n- [ ] Run evaluation on held-out test set\n\n---\n\n## Workflow 2: PPO with Critic Model\n\nUse this workflow when you need value-based advantage estimation (GAE).\n\n### Key Differences from GRPO\n- Requires separate critic model\n- Uses Generalized Advantage Estimation (GAE)\n- Better for tasks with dense rewards\n\n### Configuration\n\n```yaml\nalgorithm:\n  adv_estimator: gae  # Use GAE instead of GRPO\n  gamma: 0.99\n  lam: 0.95\n\ncritic:\n  model:\n    path: Qwen\u002FQwen2.5-7B-Instruct  # Can be same or different from actor\n  ppo_mini_batch_size: 64\n\nactor_rollout_ref:\n  actor:\n    use_kl_loss: true\n    kl_loss_coef: 0.02\n    clip_ratio: 0.2  # PPO clipping\n```\n\n### Launch with Critic\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    algorithm.adv_estimator=gae \\\n    critic.model.path=Qwen\u002FQwen2.5-7B-Instruct \\\n    trainer.n_gpus_per_node=8\n```\n\n---\n\n## Workflow 3: Large-Scale Training with Megatron\n\nUse this workflow for models >70B parameters or when you need expert parallelism.\n\n### Prerequisites\n- [ ] Install Megatron-LM bridge: `pip install mbridge`\n- [ ] Convert model to Megatron format\n- [ ] Multi-node cluster with NVLink\u002FInfiniBand\n\n### Configuration for 70B+ Models\n\n```yaml\nactor_rollout_ref:\n  model:\n    path: \u002Fpath\u002Fto\u002Fmegatron\u002Fcheckpoint\n    backend: megatron\n  actor:\n    strategy: megatron\n    tensor_model_parallel_size: 8\n    pipeline_model_parallel_size: 2\n  rollout:\n    name: vllm\n    tensor_parallel_size: 8\n```\n\n### Launch Multi-Node\n\n```bash\n# On head node\nray start --head --port=6379\n\n# On worker nodes\nray start --address='head_ip:6379'\n\n# Launch training\npython3 -m verl.trainer.main_ppo \\\n    trainer.nnodes=4 \\\n    trainer.n_gpus_per_node=8\n```\n\n---\n\n## Configuration Reference\n\n### Algorithm Selection\n\n| Algorithm | `adv_estimator` | Use Case |\n|-----------|-----------------|----------|\n| GRPO | `grpo` | Critic-free, math\u002Freasoning |\n| PPO\u002FGAE | `gae` | Dense rewards, value estimation |\n| REINFORCE++ | `reinforce_plus_plus` | Variance reduction |\n| RLOO | `rloo` | Leave-one-out baseline |\n| ReMax | `remax` | Maximum reward baseline |\n| OPO | `opo` | Optimal policy optimization |\n\n### Key Parameters\n\n```yaml\n# Rollout parameters\nactor_rollout_ref.rollout.n: 8              # Samples per prompt\nactor_rollout_ref.rollout.temperature: 0.7  # Sampling temperature\nactor_rollout_ref.rollout.top_p: 0.95       # Nucleus sampling\n\n# Training parameters\nactor_rollout_ref.actor.lr: 1e-6            # Learning rate\nactor_rollout_ref.actor.ppo_mini_batch_size: 64\nactor_rollout_ref.actor.clip_ratio: 0.2     # PPO clip range\n\n# KL control\nactor_rollout_ref.actor.use_kl_loss: true\nactor_rollout_ref.actor.kl_loss_coef: 0.001\nalgorithm.kl_ctrl.target_kl: 0.1            # For adaptive KL control\n```\n\n---\n\n## Common Issues and Solutions\n\n### Issue: OOM During Rollout\n\n**Symptoms**: CUDA out of memory during generation phase\n\n**Solutions**:\n```yaml\n# Reduce batch size\nactor_rollout_ref.rollout.log_prob_micro_batch_size: 4\n\n# Enable gradient checkpointing\nactor_rollout_ref.model.enable_gradient_checkpointing: true\n\n# Use FSDP2 with CPU offloading\nactor_rollout_ref.actor.strategy: fsdp2\nactor_rollout_ref.actor.fsdp_config.offload_policy: true\n```\n\n### Issue: Training Instability\n\n**Symptoms**: Loss spikes, reward collapse\n\n**Solutions**:\n```yaml\n# Reduce learning rate\nactor_rollout_ref.actor.lr: 5e-7\n\n# Increase KL penalty\nactor_rollout_ref.actor.kl_loss_coef: 0.01\n\n# Enable gradient clipping\nactor_rollout_ref.actor.max_grad_norm: 1.0\n```\n\n### Issue: Slow Weight Sync\n\n**Symptoms**: Long pauses between rollout and training\n\n**Solutions**:\n```bash\n# Use FSDP2 for faster resharding\nactor_rollout_ref.actor.strategy=fsdp2\n\n# Enable async weight transfer\ntrainer.async_weight_update=true\n```\n\n### Issue: vLLM Version Mismatch\n\n**Symptoms**: Import errors or generation failures\n\n**Solution**: Use compatible versions:\n```bash\npip install vllm>=0.8.5,\u003C=0.12.0\n# Avoid vLLM 0.7.x (known bugs)\n```\n\n---\n\n## Advanced Topics\n\n### Multi-Turn Tool Calling\n\nSee [references\u002Fmulti-turn.md](references\u002Fmulti-turn.md) for agentic workflows with tool use.\n\n### Vision-Language Models\n\n```yaml\nactor_rollout_ref:\n  model:\n    path: Qwen\u002FQwen2.5-VL-7B-Instruct\n  rollout:\n    name: vllm\n    enable_vision: true\n```\n\n### LoRA Training\n\n```yaml\nactor_rollout_ref:\n  actor:\n    lora:\n      enabled: true\n      r: 16\n      alpha: 32\n      target_modules: [\"q_proj\", \"v_proj\"]\n```\n\n---\n\n## Resources\n\n- **Documentation**: https:\u002F\u002Fverl.readthedocs.io\u002F\n- **Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2409.19256\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fvolcengine\u002Fverl\n- **Recipes**: https:\u002F\u002Fgithub.com\u002Fverl-project\u002Fverl-recipe (DAPO, GSPO, etc.)\n- **Community**: Slack at verl-project\n\n",{"name":5,"tags":44,"author":46,"license":15,"version":47,"description":7,"dependencies":48},[45,33,35,32,34,36],"Reinforcement Learning","luo-kai","1.0.0",[49,50,51,52,53],"verl>=0.3.0","torch>=2.0.0","ray>=2.41.0","vllm>=0.8.2","transformers>=4.40.0","main","4d2fb89cdfb75f3f6fd3c6c5ec9ad90ea129eb3c","ai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fverl",2,"https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai","https:\u002F\u002Fgithub.com\u002Fluokai0","a solo builder.","2026-05-06 04:29:04",{"source":8,"skillName":5,"commands":63,"prompt":73,"zipAvailable":74},[64,67,70],{"tool":65,"command":66},"npx","npx skills add luokai0\u002Fai-agent-skills-by-luo-kai --skill verl-rl-training",{"tool":68,"command":69},"bunx","bunx skills add luokai0\u002Fai-agent-skills-by-luo-kai --skill verl-rl-training",{"tool":71,"command":72},"pnpm","pnpm dlx skills add luokai0\u002Fai-agent-skills-by-luo-kai --skill verl-rl-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fverl\u002FSKILL.md",true,[76,82],{"path":77,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Fapi-reference.md","REFERENCE","参考文档",6941,"https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fverl\u002Freferences\u002Fapi-reference.md",{"path":83,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":84,"sourceHtmlUrl":85},"references\u002Ftroubleshooting.md",6792,"https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fverl\u002Freferences\u002Ftroubleshooting.md",[87,114,131,161,193,230],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":39},"01M6F2SKTE208174C7S7JJCZC7","write-contracts","Generates secure Aptos Move V2 smart contracts with Object model, Digital Asset integration, security patterns, and storage type guidance. Includes comprehensive storage decision framework for optimal data structure selection. Triggers on: 'write contract', 'create NFT collection', 'build marketplace', 'implement minting', 'generate Move module', 'create token contract', 'build DAO', 'implement staking'. Ask storage questions when: 'store', 'track', 'registry', 'mapping', 'list', 'collection'.","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fwrite-contracts\u002FSKILL.md",[],[94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113],"write","contracts","write contracts","storage","secure","create","generate","collection","model","decision","ask","smart-contracts","selection","objects","structure","nft","token","aptos","critical","move",{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":39},"01M6F2RFXJ6T9MKW0A7N7GM45B","web-search","通用网络搜索技能，支持多引擎搜索（百度、必应、DuckDuckGo），无需API密钥即可获取实时信息","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fweb-search-ex-skill\u002FSKILL.md",[],[121,122,123,124,125,126,127,128,129,130],"web","search","web search","搜索","baidu","bing","duckduckgo","百度","多引擎","internet",{"skillNo":132,"name":133,"nameZh":6,"description":134,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":135,"categories":136,"keywords":142,"publishedAt":39},"01M6F2K968G1FZEZFPW2E6B222","solo-build","Execute implementation plan tasks with TDD workflow, auto-commit, and phase gates. Use when user says \"build it\", \"start building\", \"execute plan\", \"implement tasks\", \"ship it\", or references a track ID. Do NOT use for planning (use \u002Fplan) or scaffolding (use \u002Fscaffold).","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fsolo-build\u002FSKILL.md",[137],{"categoryNo":138,"categoryName":139,"taxonomyType":140,"taxonomyTypeDesc":141},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[143,144,145,146,147,148,149,150,122,151,152,153,94,154,155,156,157,158,159,160],"build","solo","solo build","MCP","workflow","commit","tasks","code","Bash","plan","read","auto","references","context7","query","scaffolding","ship it","project",{"skillNo":162,"name":163,"nameZh":6,"description":164,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":165,"categories":166,"keywords":173,"publishedAt":39},"01M6F2GKN39SGHSPWYB3662YC8","react-email","Use when building HTML email templates with React components, adding a visual email editor to an application using the React Email visual editor, rendering emails to HTML, or sending emails with Resend. Covers welcome emails, password resets, notifications, order confirmations, newsletters, transactional emails, and the embeddable email editor component.","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Freact-email-skills\u002FSKILL.md",[167,170],{"categoryNo":168,"categoryName":169,"taxonomyType":140,"taxonomyTypeDesc":141},"01JBSEEDCAT000000000000005","邮件与沟通",{"categoryNo":171,"categoryName":172,"taxonomyType":140,"taxonomyTypeDesc":141},"01JBSEEDCAT000000000000009","前端开发",[174,175,176,177,178,179,180,181,182,183,184,185,186,26,187,188,189,190,191,192],"react","email","react email","github","editor","components","react components","component","welcome","https","notifications","password","node","html","resend","visual","emails","rendering","with",{"skillNo":194,"name":195,"nameZh":6,"description":196,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":197,"categories":198,"keywords":211,"publishedAt":39},"01M6F2FBVGVDFSG8SSFJGXA40C","APITester Agent-Driven API Testing","Test API endpoints and document responses. Define tests in plain English, run them, get formatted results. Agent-driven Postman alternative.","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fopenclaw-api-tester\u002FSKILL.md",[199,202,205,206],{"categoryNo":200,"categoryName":201,"taxonomyType":140,"taxonomyTypeDesc":141},"01JBSEEDCAT000000000000008","测试与质量",{"categoryNo":203,"categoryName":204,"taxonomyType":140,"taxonomyTypeDesc":141},"01JBSEEDCAT000000000000010","后端与接口",{"categoryNo":138,"categoryName":139,"taxonomyType":140,"taxonomyTypeDesc":141},{"categoryNo":207,"categoryName":208,"taxonomyType":209,"taxonomyTypeDesc":210},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229],"testing","api","api testing","driven","APITester Agent Driven API Testing","test","run","postman","results","plain english","tester","endpoints","document","get","define","tests","english","api-tester",{"skillNo":231,"name":232,"nameZh":6,"description":233,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":234,"categories":235,"keywords":236,"publishedAt":39},"01M6F273Y3NGAPVJZ608BC0G9G","hubspot-implementation-plan","Generate a phased implementation plan from a HubSpot audit report. Creates prioritized, sequenced cleanup processes with effort estimates, dependencies, and automation feasibility. Use after running \u002Fhubspot-audit.","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fhubspot-implementation-plan\u002FSKILL.md",[],[152,237,238,239,240,241,242,100,243,244,245,246,247,26,248,249,192,250],"implementation","hubspot","hubspot implementation plan","automation","feasibility","cleanup","planning","report","audit","audit report","phased implementation","dependencies","processes","hubspot audit"]