[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6QM76B6STT3SBS0ABKF3B06":3,"skill-same:01M6QM76B6STT3SBS0ABKF3B06":89},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":56,"commitSha":57,"skillDirectory":58,"forksCount":14,"repositoryHtmlUrl":59,"repositoryHomepageUrl":60,"ownerHtmlUrl":61,"ownerBlogUrl":62,"ownerDescription":63,"updatedAt":64,"install":65,"resources":78},"01M6QM76B6STT3SBS0ABKF3B06","verl-rl-training",null,"Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.","Tashima-Tarsh\u002FDisha6.6","Tashima-Tarsh","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F269906498?v=4","01M6PBH98X03XERH0HFT4PFB1P","NITISH KUMAR","01M6PBH9DA0M6KM3Z12P848F0K",0,"MIT","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-verl\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"training","verl rl training","verl","rl","engine","for","implementing","at","post","algorithms","use","distributed","learning","vLLM","Ray","PPO","RLHF","Post-Training","GRPO","infrastructure","2026-11-10 10:12:30",false,"---\nname: verl-rl-training\ndescription: Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Reinforcement Learning, RLHF, GRPO, PPO, Post-Training, Distributed Training]\ndependencies: [verl>=0.3.0, torch>=2.0.0, ray>=2.41.0, vllm>=0.8.2, transformers>=4.40.0]\ncategory: \"ai-research\"\n---\n\n# verl: Volcano Engine Reinforcement Learning for LLMs\n\nverl is a flexible, efficient, and production-ready RL training library for large language models from ByteDance's Seed team. It implements the HybridFlow framework (EuroSys 2025) and powers models like Doubao-1.5-pro achieving O1-level performance on math benchmarks.\n\n## When to Use verl\n\n**Choose verl when you need:**\n- Production-ready RL training at scale (tested up to 671B parameters)\n- Flexibility to swap backends (FSDP ↔ Megatron-LM ↔ vLLM ↔ SGLang)\n- Support for multiple RL algorithms (PPO, GRPO, RLOO, REINFORCE++, DAPO)\n- Multi-turn rollout with tool calling for agentic workflows\n- Vision-language model RL training\n\n**Consider alternatives when:**\n- You need Megatron-native training → use **slime** or **miles**\n- You want PyTorch-native abstractions with Monarch → use **torchforge**\n- You only need simple SFT\u002FDPO → use **TRL** or **Axolotl**\n\n## Key Features\n\n- **Training backends**: FSDP, FSDP2, Megatron-LM\n- **Rollout engines**: vLLM, SGLang, HuggingFace Transformers\n- **Algorithms**: PPO, GRPO, DAPO, RLOO, ReMax, REINFORCE++, SPIN, SPPO\n- **Models**: Qwen-3, Llama-3.1, DeepSeek, Gemma-2 (0.5B to 671B)\n- **Advanced**: LoRA RL, sequence parallelism, expert parallelism, multi-turn tools\n\n## Installation\n\n```bash\n# Option 1: pip install\npip install verl[vllm]  # or verl[sglang] for SGLang backend\n\n# Option 2: Docker (recommended for production)\ndocker pull verlai\u002Fverl:vllm011.latest\n\n# Option 3: From source\ngit clone https:\u002F\u002Fgithub.com\u002Fvolcengine\u002Fverl.git\ncd verl && pip install -e .[vllm,math]\n```\n\n## Quick Start: GRPO Training\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    algorithm.adv_estimator=grpo \\\n    data.train_files=~\u002Fdata\u002Fgsm8k\u002Ftrain.parquet \\\n    actor_rollout_ref.model.path=Qwen\u002FQwen2.5-7B \\\n    actor_rollout_ref.rollout.n=8 \\\n    actor_rollout_ref.actor.use_kl_loss=True \\\n    trainer.n_gpus_per_node=8\n```\n\n## Core Architecture\n\nverl uses a **HybridFlow** programming model separating control flow from computation:\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ Single-Process Controller (Ray)                         │\n│ - Orchestrates: rollout → reward → train → sync        │\n└─────────────────────┬───────────────────────────────────┘\n                      │\n┌─────────────────────▼───────────────────────────────────┐\n│ Multi-Process Workers                                   │\n│ ├── ActorRolloutRefWorker (policy + generation)        │\n│ ├── CriticWorker (value estimation, PPO only)          │\n│ └── RewardManager (model-based or rule-based rewards)  │\n└─────────────────────────────────────────────────────────┘\n```\n\n---\n\n## Workflow 1: Math Reasoning with GRPO\n\nUse this workflow for training reasoning models on math tasks like GSM8K or MATH.\n\n### Prerequisites Checklist\n- [ ] GPU cluster with 8+ GPUs (H100 recommended)\n- [ ] Dataset in parquet format with `prompt` and `reward_model` columns\n- [ ] Base model from HuggingFace Hub\n\n### Step 1: Prepare Dataset\n\n```python\nimport pandas as pd\n\ndata = [\n    {\n        \"prompt\": [{\"role\": \"user\", \"content\": \"What is 15 + 27?\"}],\n        \"reward_model\": {\"ground_truth\": \"42\"}\n    },\n    # ... more examples\n]\ndf = pd.DataFrame(data)\ndf.to_parquet(\"train.parquet\")\n```\n\n### Step 2: Define Reward Function\n\n```python\n# reward_function.py\nimport re\n\ndef compute_reward(responses, ground_truths):\n    rewards = []\n    for response, gt in zip(responses, ground_truths):\n        # Extract answer from response\n        match = re.search(r'\\\\boxed{([^}]+)}', response)\n        if match and match.group(1).strip() == gt.strip():\n            rewards.append(1.0)\n        else:\n            rewards.append(0.0)\n    return rewards\n```\n\n### Step 3: Create Training Config\n\n```yaml\n# config\u002Fgrpo_math.yaml\nalgorithm:\n  adv_estimator: grpo\n  gamma: 1.0\n  lam: 1.0\n\ndata:\n  train_files: \u002Fpath\u002Fto\u002Ftrain.parquet\n  val_files: \u002Fpath\u002Fto\u002Fval.parquet\n  train_batch_size: 256\n  max_prompt_length: 512\n  max_response_length: 2048\n\nactor_rollout_ref:\n  model:\n    path: Qwen\u002FQwen2.5-7B-Instruct\n  actor:\n    use_kl_loss: true\n    kl_loss_coef: 0.001\n    ppo_mini_batch_size: 64\n  rollout:\n    name: vllm\n    n: 8  # samples per prompt\n    temperature: 0.7\n    top_p: 0.95\n\ntrainer:\n  total_epochs: 3\n  n_gpus_per_node: 8\n  save_freq: 100\n```\n\n### Step 4: Launch Training\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    --config-path config \\\n    --config-name grpo_math \\\n    trainer.experiment_name=grpo_math_qwen7b\n```\n\n### Step 5: Monitor and Validate\n- [ ] Check WandB\u002FTensorBoard for loss curves\n- [ ] Verify reward is increasing over steps\n- [ ] Run evaluation on held-out test set\n\n---\n\n## Workflow 2: PPO with Critic Model\n\nUse this workflow when you need value-based advantage estimation (GAE).\n\n### Key Differences from GRPO\n- Requires separate critic model\n- Uses Generalized Advantage Estimation (GAE)\n- Better for tasks with dense rewards\n\n### Configuration\n\n```yaml\nalgorithm:\n  adv_estimator: gae  # Use GAE instead of GRPO\n  gamma: 0.99\n  lam: 0.95\n\ncritic:\n  model:\n    path: Qwen\u002FQwen2.5-7B-Instruct  # Can be same or different from actor\n  ppo_mini_batch_size: 64\n\nactor_rollout_ref:\n  actor:\n    use_kl_loss: true\n    kl_loss_coef: 0.02\n    clip_ratio: 0.2  # PPO clipping\n```\n\n### Launch with Critic\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    algorithm.adv_estimator=gae \\\n    critic.model.path=Qwen\u002FQwen2.5-7B-Instruct \\\n    trainer.n_gpus_per_node=8\n```\n\n---\n\n## Workflow 3: Large-Scale Training with Megatron\n\nUse this workflow for models >70B parameters or when you need expert parallelism.\n\n### Prerequisites\n- [ ] Install Megatron-LM bridge: `pip install mbridge`\n- [ ] Convert model to Megatron format\n- [ ] Multi-node cluster with NVLink\u002FInfiniBand\n\n### Configuration for 70B+ Models\n\n```yaml\nactor_rollout_ref:\n  model:\n    path: \u002Fpath\u002Fto\u002Fmegatron\u002Fcheckpoint\n    backend: megatron\n  actor:\n    strategy: megatron\n    tensor_model_parallel_size: 8\n    pipeline_model_parallel_size: 2\n  rollout:\n    name: vllm\n    tensor_parallel_size: 8\n```\n\n### Launch Multi-Node\n\n```bash\n# On head node\nray start --head --port=6379\n\n# On worker nodes\nray start --address='head_ip:6379'\n\n# Launch training\npython3 -m verl.trainer.main_ppo \\\n    trainer.nnodes=4 \\\n    trainer.n_gpus_per_node=8\n```\n\n---\n\n## Configuration Reference\n\n### Algorithm Selection\n\n| Algorithm | `adv_estimator` | Use Case |\n|-----------|-----------------|----------|\n| GRPO | `grpo` | Critic-free, math\u002Freasoning |\n| PPO\u002FGAE | `gae` | Dense rewards, value estimation |\n| REINFORCE++ | `reinforce_plus_plus` | Variance reduction |\n| RLOO | `rloo` | Leave-one-out baseline |\n| ReMax | `remax` | Maximum reward baseline |\n| OPO | `opo` | Optimal policy optimization |\n\n### Key Parameters\n\n```yaml\n# Rollout parameters\nactor_rollout_ref.rollout.n: 8              # Samples per prompt\nactor_rollout_ref.rollout.temperature: 0.7  # Sampling temperature\nactor_rollout_ref.rollout.top_p: 0.95       # Nucleus sampling\n\n# Training parameters\nactor_rollout_ref.actor.lr: 1e-6            # Learning rate\nactor_rollout_ref.actor.ppo_mini_batch_size: 64\nactor_rollout_ref.actor.clip_ratio: 0.2     # PPO clip range\n\n# KL control\nactor_rollout_ref.actor.use_kl_loss: true\nactor_rollout_ref.actor.kl_loss_coef: 0.001\nalgorithm.kl_ctrl.target_kl: 0.1            # For adaptive KL control\n```\n\n---\n\n## Common Issues and Solutions\n\n### Issue: OOM During Rollout\n\n**Symptoms**: CUDA out of memory during generation phase\n\n**Solutions**:\n```yaml\n# Reduce batch size\nactor_rollout_ref.rollout.log_prob_micro_batch_size: 4\n\n# Enable gradient checkpointing\nactor_rollout_ref.model.enable_gradient_checkpointing: true\n\n# Use FSDP2 with CPU offloading\nactor_rollout_ref.actor.strategy: fsdp2\nactor_rollout_ref.actor.fsdp_config.offload_policy: true\n```\n\n### Issue: Training Instability\n\n**Symptoms**: Loss spikes, reward collapse\n\n**Solutions**:\n```yaml\n# Reduce learning rate\nactor_rollout_ref.actor.lr: 5e-7\n\n# Increase KL penalty\nactor_rollout_ref.actor.kl_loss_coef: 0.01\n\n# Enable gradient clipping\nactor_rollout_ref.actor.max_grad_norm: 1.0\n```\n\n### Issue: Slow Weight Sync\n\n**Symptoms**: Long pauses between rollout and training\n\n**Solutions**:\n```bash\n# Use FSDP2 for faster resharding\nactor_rollout_ref.actor.strategy=fsdp2\n\n# Enable async weight transfer\ntrainer.async_weight_update=true\n```\n\n### Issue: vLLM Version Mismatch\n\n**Symptoms**: Import errors or generation failures\n\n**Solution**: Use compatible versions:\n```bash\npip install vllm>=0.8.5,\u003C=0.12.0\n# Avoid vLLM 0.7.x (known bugs)\n```\n\n---\n\n## Advanced Topics\n\n### Multi-Turn Tool Calling\n\nSee [references\u002Fmulti-turn.md](references\u002Fmulti-turn.md) for agentic workflows with tool use.\n\n### Vision-Language Models\n\n```yaml\nactor_rollout_ref:\n  model:\n    path: Qwen\u002FQwen2.5-VL-7B-Instruct\n  rollout:\n    name: vllm\n    enable_vision: true\n```\n\n### LoRA Training\n\n```yaml\nactor_rollout_ref:\n  actor:\n    lora:\n      enabled: true\n      r: 16\n      alpha: 32\n      target_modules: [\"q_proj\", \"v_proj\"]\n```\n\n---\n\n## Resources\n\n- **Documentation**: https:\u002F\u002Fverl.readthedocs.io\u002F\n- **Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2409.19256\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fvolcengine\u002Fverl\n- **Recipes**: https:\u002F\u002Fgithub.com\u002Fverl-project\u002Fverl-recipe (DAPO, GSPO, etc.)\n- **Community**: Slack at verl-project\n\n","# verl: Volcano Engine Reinforcement Learning for LLMs\n\nverl is a flexible, efficient, and production-ready RL training library for large language models from ByteDance's Seed team. It implements the HybridFlow framework (EuroSys 2025) and powers models like Doubao-1.5-pro achieving O1-level performance on math benchmarks.\n\n## When to Use verl\n\n**Choose verl when you need:**\n- Production-ready RL training at scale (tested up to 671B parameters)\n- Flexibility to swap backends (FSDP ↔ Megatron-LM ↔ vLLM ↔ SGLang)\n- Support for multiple RL algorithms (PPO, GRPO, RLOO, REINFORCE++, DAPO)\n- Multi-turn rollout with tool calling for agentic workflows\n- Vision-language model RL training\n\n**Consider alternatives when:**\n- You need Megatron-native training → use **slime** or **miles**\n- You want PyTorch-native abstractions with Monarch → use **torchforge**\n- You only need simple SFT\u002FDPO → use **TRL** or **Axolotl**\n\n## Key Features\n\n- **Training backends**: FSDP, FSDP2, Megatron-LM\n- **Rollout engines**: vLLM, SGLang, HuggingFace Transformers\n- **Algorithms**: PPO, GRPO, DAPO, RLOO, ReMax, REINFORCE++, SPIN, SPPO\n- **Models**: Qwen-3, Llama-3.1, DeepSeek, Gemma-2 (0.5B to 671B)\n- **Advanced**: LoRA RL, sequence parallelism, expert parallelism, multi-turn tools\n\n## Installation\n\n```bash\n# Option 1: pip install\npip install verl[vllm]  # or verl[sglang] for SGLang backend\n\n# Option 2: Docker (recommended for production)\ndocker pull verlai\u002Fverl:vllm011.latest\n\n# Option 3: From source\ngit clone https:\u002F\u002Fgithub.com\u002Fvolcengine\u002Fverl.git\ncd verl && pip install -e .[vllm,math]\n```\n\n## Quick Start: GRPO Training\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    algorithm.adv_estimator=grpo \\\n    data.train_files=~\u002Fdata\u002Fgsm8k\u002Ftrain.parquet \\\n    actor_rollout_ref.model.path=Qwen\u002FQwen2.5-7B \\\n    actor_rollout_ref.rollout.n=8 \\\n    actor_rollout_ref.actor.use_kl_loss=True \\\n    trainer.n_gpus_per_node=8\n```\n\n## Core Architecture\n\nverl uses a **HybridFlow** programming model separating control flow from computation:\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ Single-Process Controller (Ray)                         │\n│ - Orchestrates: rollout → reward → train → sync        │\n└─────────────────────┬───────────────────────────────────┘\n                      │\n┌─────────────────────▼───────────────────────────────────┐\n│ Multi-Process Workers                                   │\n│ ├── ActorRolloutRefWorker (policy + generation)        │\n│ ├── CriticWorker (value estimation, PPO only)          │\n│ └── RewardManager (model-based or rule-based rewards)  │\n└─────────────────────────────────────────────────────────┘\n```\n\n---\n\n## Workflow 1: Math Reasoning with GRPO\n\nUse this workflow for training reasoning models on math tasks like GSM8K or MATH.\n\n### Prerequisites Checklist\n- [ ] GPU cluster with 8+ GPUs (H100 recommended)\n- [ ] Dataset in parquet format with `prompt` and `reward_model` columns\n- [ ] Base model from HuggingFace Hub\n\n### Step 1: Prepare Dataset\n\n```python\nimport pandas as pd\n\ndata = [\n    {\n        \"prompt\": [{\"role\": \"user\", \"content\": \"What is 15 + 27?\"}],\n        \"reward_model\": {\"ground_truth\": \"42\"}\n    },\n    # ... more examples\n]\ndf = pd.DataFrame(data)\ndf.to_parquet(\"train.parquet\")\n```\n\n### Step 2: Define Reward Function\n\n```python\n# reward_function.py\nimport re\n\ndef compute_reward(responses, ground_truths):\n    rewards = []\n    for response, gt in zip(responses, ground_truths):\n        # Extract answer from response\n        match = re.search(r'\\\\boxed{([^}]+)}', response)\n        if match and match.group(1).strip() == gt.strip():\n            rewards.append(1.0)\n        else:\n            rewards.append(0.0)\n    return rewards\n```\n\n### Step 3: Create Training Config\n\n```yaml\n# config\u002Fgrpo_math.yaml\nalgorithm:\n  adv_estimator: grpo\n  gamma: 1.0\n  lam: 1.0\n\ndata:\n  train_files: \u002Fpath\u002Fto\u002Ftrain.parquet\n  val_files: \u002Fpath\u002Fto\u002Fval.parquet\n  train_batch_size: 256\n  max_prompt_length: 512\n  max_response_length: 2048\n\nactor_rollout_ref:\n  model:\n    path: Qwen\u002FQwen2.5-7B-Instruct\n  actor:\n    use_kl_loss: true\n    kl_loss_coef: 0.001\n    ppo_mini_batch_size: 64\n  rollout:\n    name: vllm\n    n: 8  # samples per prompt\n    temperature: 0.7\n    top_p: 0.95\n\ntrainer:\n  total_epochs: 3\n  n_gpus_per_node: 8\n  save_freq: 100\n```\n\n### Step 4: Launch Training\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    --config-path config \\\n    --config-name grpo_math \\\n    trainer.experiment_name=grpo_math_qwen7b\n```\n\n### Step 5: Monitor and Validate\n- [ ] Check WandB\u002FTensorBoard for loss curves\n- [ ] Verify reward is increasing over steps\n- [ ] Run evaluation on held-out test set\n\n---\n\n## Workflow 2: PPO with Critic Model\n\nUse this workflow when you need value-based advantage estimation (GAE).\n\n### Key Differences from GRPO\n- Requires separate critic model\n- Uses Generalized Advantage Estimation (GAE)\n- Better for tasks with dense rewards\n\n### Configuration\n\n```yaml\nalgorithm:\n  adv_estimator: gae  # Use GAE instead of GRPO\n  gamma: 0.99\n  lam: 0.95\n\ncritic:\n  model:\n    path: Qwen\u002FQwen2.5-7B-Instruct  # Can be same or different from actor\n  ppo_mini_batch_size: 64\n\nactor_rollout_ref:\n  actor:\n    use_kl_loss: true\n    kl_loss_coef: 0.02\n    clip_ratio: 0.2  # PPO clipping\n```\n\n### Launch with Critic\n\n```bash\npython3 -m verl.trainer.main_ppo \\\n    algorithm.adv_estimator=gae \\\n    critic.model.path=Qwen\u002FQwen2.5-7B-Instruct \\\n    trainer.n_gpus_per_node=8\n```\n\n---\n\n## Workflow 3: Large-Scale Training with Megatron\n\nUse this workflow for models >70B parameters or when you need expert parallelism.\n\n### Prerequisites\n- [ ] Install Megatron-LM bridge: `pip install mbridge`\n- [ ] Convert model to Megatron format\n- [ ] Multi-node cluster with NVLink\u002FInfiniBand\n\n### Configuration for 70B+ Models\n\n```yaml\nactor_rollout_ref:\n  model:\n    path: \u002Fpath\u002Fto\u002Fmegatron\u002Fcheckpoint\n    backend: megatron\n  actor:\n    strategy: megatron\n    tensor_model_parallel_size: 8\n    pipeline_model_parallel_size: 2\n  rollout:\n    name: vllm\n    tensor_parallel_size: 8\n```\n\n### Launch Multi-Node\n\n```bash\n# On head node\nray start --head --port=6379\n\n# On worker nodes\nray start --address='head_ip:6379'\n\n# Launch training\npython3 -m verl.trainer.main_ppo \\\n    trainer.nnodes=4 \\\n    trainer.n_gpus_per_node=8\n```\n\n---\n\n## Configuration Reference\n\n### Algorithm Selection\n\n| Algorithm | `adv_estimator` | Use Case |\n|-----------|-----------------|----------|\n| GRPO | `grpo` | Critic-free, math\u002Freasoning |\n| PPO\u002FGAE | `gae` | Dense rewards, value estimation |\n| REINFORCE++ | `reinforce_plus_plus` | Variance reduction |\n| RLOO | `rloo` | Leave-one-out baseline |\n| ReMax | `remax` | Maximum reward baseline |\n| OPO | `opo` | Optimal policy optimization |\n\n### Key Parameters\n\n```yaml\n# Rollout parameters\nactor_rollout_ref.rollout.n: 8              # Samples per prompt\nactor_rollout_ref.rollout.temperature: 0.7  # Sampling temperature\nactor_rollout_ref.rollout.top_p: 0.95       # Nucleus sampling\n\n# Training parameters\nactor_rollout_ref.actor.lr: 1e-6            # Learning rate\nactor_rollout_ref.actor.ppo_mini_batch_size: 64\nactor_rollout_ref.actor.clip_ratio: 0.2     # PPO clip range\n\n# KL control\nactor_rollout_ref.actor.use_kl_loss: true\nactor_rollout_ref.actor.kl_loss_coef: 0.001\nalgorithm.kl_ctrl.target_kl: 0.1            # For adaptive KL control\n```\n\n---\n\n## Common Issues and Solutions\n\n### Issue: OOM During Rollout\n\n**Symptoms**: CUDA out of memory during generation phase\n\n**Solutions**:\n```yaml\n# Reduce batch size\nactor_rollout_ref.rollout.log_prob_micro_batch_size: 4\n\n# Enable gradient checkpointing\nactor_rollout_ref.model.enable_gradient_checkpointing: true\n\n# Use FSDP2 with CPU offloading\nactor_rollout_ref.actor.strategy: fsdp2\nactor_rollout_ref.actor.fsdp_config.offload_policy: true\n```\n\n### Issue: Training Instability\n\n**Symptoms**: Loss spikes, reward collapse\n\n**Solutions**:\n```yaml\n# Reduce learning rate\nactor_rollout_ref.actor.lr: 5e-7\n\n# Increase KL penalty\nactor_rollout_ref.actor.kl_loss_coef: 0.01\n\n# Enable gradient clipping\nactor_rollout_ref.actor.max_grad_norm: 1.0\n```\n\n### Issue: Slow Weight Sync\n\n**Symptoms**: Long pauses between rollout and training\n\n**Solutions**:\n```bash\n# Use FSDP2 for faster resharding\nactor_rollout_ref.actor.strategy=fsdp2\n\n# Enable async weight transfer\ntrainer.async_weight_update=true\n```\n\n### Issue: vLLM Version Mismatch\n\n**Symptoms**: Import errors or generation failures\n\n**Solution**: Use compatible versions:\n```bash\npip install vllm>=0.8.5,\u003C=0.12.0\n# Avoid vLLM 0.7.x (known bugs)\n```\n\n---\n\n## Advanced Topics\n\n### Multi-Turn Tool Calling\n\nSee [references\u002Fmulti-turn.md](references\u002Fmulti-turn.md) for agentic workflows with tool use.\n\n### Vision-Language Models\n\n```yaml\nactor_rollout_ref:\n  model:\n    path: Qwen\u002FQwen2.5-VL-7B-Instruct\n  rollout:\n    name: vllm\n    enable_vision: true\n```\n\n### LoRA Training\n\n```yaml\nactor_rollout_ref:\n  actor:\n    lora:\n      enabled: true\n      r: 16\n      alpha: 32\n      target_modules: [\"q_proj\", \"v_proj\"]\n```\n\n---\n\n## Resources\n\n- **Documentation**: https:\u002F\u002Fverl.readthedocs.io\u002F\n- **Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2409.19256\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fvolcengine\u002Fverl\n- **Recipes**: https:\u002F\u002Fgithub.com\u002Fverl-project\u002Fverl-recipe (DAPO, GSPO, etc.)\n- **Community**: Slack at verl-project\n\n",{"name":5,"tags":44,"author":47,"license":15,"version":48,"category":49,"description":7,"dependencies":50},[45,35,37,34,36,46],"Reinforcement Learning","Distributed Training","Orchestra Research","1.0.0","ai-research",[51,52,53,54,55],"verl>=0.3.0","torch>=2.0.0","ray>=2.41.0","vllm>=0.8.2","transformers>=4.40.0","main","0da4485785ee599388900be50db1635489d53056",".agents\u002Fskills\u002Forch-verl","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6","https:\u002F\u002Fdisha6.6.thenitishkr.in\u002F","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh","www.thenitishkr","thenitishkr | AI • Cyber-security • Research\r\n","2026-11-08 03:33:25",{"source":8,"skillName":5,"commands":66,"prompt":76,"zipAvailable":77},[67,70,73],{"tool":68,"command":69},"npx","npx skills add Tashima-Tarsh\u002FDisha6.6 --skill verl-rl-training",{"tool":71,"command":72},"bunx","bunx skills add Tashima-Tarsh\u002FDisha6.6 --skill verl-rl-training",{"tool":74,"command":75},"pnpm","pnpm dlx skills add Tashima-Tarsh\u002FDisha6.6 --skill verl-rl-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-verl\u002FSKILL.md",true,[79,85],{"path":80,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Fapi-reference.md","REFERENCE","参考文档",6941,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-verl\u002Freferences\u002Fapi-reference.md",{"path":86,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":87,"sourceHtmlUrl":88},"references\u002Ftroubleshooting.md",6792,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Forch-verl\u002Freferences\u002Ftroubleshooting.md",[90,121,144,171,196,220],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":94,"categories":95,"keywords":101,"publishedAt":39},"01M6QRD07D4TGNDK6VJ4EHF8BV","youtube-full","Use when the user needs YouTube transcripts, video search, channel browsing, playlist extraction, or content monitoring. Trigger phrases: 'get the transcript for', 'search YouTube for', 'what are the latest videos on', 'list this playlist', 'monitor this channel', or any request involving a YouTube URL, video ID, or @handle. Do NOT use for downloading video or audio files, YouTube engagement data (likes, comments), or private\u002Fage-restricted videos.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fyoutube-full\u002FSKILL.md",[96],{"categoryNo":97,"categoryName":98,"taxonomyType":99,"taxonomyTypeDesc":100},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",[102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,24,118,119,120],"youtube full","youtube","full","transcript","search","monitoring","data","Video","Audio","list","files","do","get","age","trigger","URL","browsing","request","youtube transcripts",{"skillNo":122,"name":123,"nameZh":6,"description":124,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":125,"categories":126,"keywords":127,"publishedAt":39},"01M6QRCJ3112VYPA97YG7KGMC2","x-twitter-growth","X\u002FTwitter growth engine for building audience, crafting viral content, and analyzing engagement. Use when the user wants to grow on X\u002FTwitter, write tweets or threads, analyze their X profile, research competitors on X, plan a posting strategy, or optimize engagement. Complements social-content (generic multi-platform) with X-specific depth: algorithm mechanics, thread engineering, reply strategy, profile optimization, and competitive intelligence via web search.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fx-twitter-growth\u002FSKILL.md",[],[128,129,130,131,106,132,133,134,135,136,137,23,138,139,29,140,141,142,143,24],"twitter","x twitter growth","growth","research","content","engineering","plan","web search","competitors","analyze","write","strategy","platform","grow","posting","competitive",{"skillNo":145,"name":146,"nameZh":6,"description":147,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":148,"categories":149,"keywords":153,"publishedAt":39},"01M6QRBN3EHQNJY42RERP7A149","write-a-skill","Create new agent skills with proper structure, progressive disclosure, and bundled resources. Use when user wants to create, write, build, or author a new skill.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrite-a-skill\u002FSKILL.md",[150],{"categoryNo":151,"categoryName":152,"taxonomyType":99,"taxonomyTypeDesc":100},"01JBSEEDCAT000000000000023","Skill与智能体开发",[154,138,155,156,157,158,159,160,29,161,162,56,163,164,165,166,167,168,169,170],"write a skill","github","create","https","productivity","matt pocock","direct","example","build","new skill","structure","new agent","resources","with","tree","progressive disclosure","author",{"skillNo":172,"name":173,"nameZh":6,"description":174,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":175,"categories":176,"keywords":177,"publishedAt":39},"01M6QRB6G8P33PNGAF9Y69EJ9T","wrap-up","Close out a launched Claude Managed Agent — recap every primitive the founder now owns, regenerate the single-file overview page, and suggest the next 1-2 upgrades. Use when the user says \"wrap up\", \"close this out\", \"what do I own now\", \"give me the summary\", \"recap the agent\", or when the orchestrator routes phase=wrap-up. primitives_inventory.py tables everything owned (agent, environment, session, memory, outcome, deployment); overview_page.py regenerates a self-contained .\u002Fmy-agent\u002Fagent-overview.html; upgrade_suggester.py ranks the next moves from recorded deferrals plus standing hardening steps. Companion to run-without-you; the last stop before phase=done.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrap-up\u002FSKILL.md",[],[178,179,180,181,182,183,184,185,186,29,187,113,188,189,190,191,192,193,194,195],"wrap up","up","wrap","gemini","deployment","run","file","hardening","closeout","stop","upgrade","founder","close","companion","you","tables","directions","phase",{"skillNo":197,"name":198,"nameZh":6,"description":199,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":200,"categories":201,"keywords":203,"publishedAt":39},"01M6QRA5AYC1RBFZ39GTWGS3N8","workflow-builder","Design and write deterministic multi-agent workflow scripts (.js files in .claude\u002Fworkflows\u002F) for Claude Code's Workflow tool. Use when a user wants to build, create, author, scaffold, or run a custom Claude Code workflow, orchestrate sub-agents (fan-out, pipeline, loop, judge-panel), or automate a repeatable multi-step task across fresh-context agents.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fworkflow-builder\u002FSKILL.md",[202],{"categoryNo":151,"categoryName":152,"taxonomyType":99,"taxonomyTypeDesc":100},[204,205,206,207,208,156,183,209,210,211,212,213,138,214,112,215,216,217,218,219],"workflow","workflow builder","builder","design","workflows","code","orchestrate","pipeline","loop","multi-agent","context","across","fresh","scripts","judge","claude code workflow",{"skillNo":221,"name":222,"nameZh":6,"description":223,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":224,"categories":225,"keywords":226,"publishedAt":39},"01M6QR7QMCZESKQA0ERW1E657M","weekly-review","Use when someone wants to run a weekly review, close open loops, audit stalled projects and commitments, get their system back to trusted, restart a lapsed review habit, or says \"\u002Fcs:weekly-review\". Walks David Allen's three-phase loop — GET CLEAR, GET CURRENT, GET CREATIVE — with deterministic scripts that inventory open loops, gate the checklist with named gaps, and score commitment health 0-100.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fweekly-review\u002FSKILL.md",[],[227,228,229,230,231,232,233,234,235,212,236,237,238,239,240,241,114,242,243,195],"review","weekly review","weekly","deterministic scripts","reflect","feeds","inventory","open","team","projects","project-management","scanner","conversation","back","project","sprint","cadence"]