[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M72FNJS5DGTD6RS8QNNSE79B":3,"skill-same:01M72FNJS5DGTD6RS8QNNSE79B":85},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":46,"commitSha":47,"skillDirectory":48,"forksCount":14,"repositoryHtmlUrl":49,"repositoryHomepageUrl":6,"ownerHtmlUrl":50,"ownerBlogUrl":6,"ownerDescription":51,"updatedAt":52,"install":53,"resources":66},"01M72FNJS5DGTD6RS8QNNSE79B","openrlhf-training",null,"High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.","cocyuhao\u002Fai-dev-tooling-research-archive","cocyuhao","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F269083740?v=4","01M72F908ET5R76KP9NXS6Y2GJ","Hao Yu","01M72F909376KCNJVKG8MEPY2Q",0,"MIT","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Fpost-training-openrlhf\u002FSKILL.md",[],[],"2026-11-10 18:39:20",false,"---\nname: openrlhf-training\ndescription: High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Post-Training, OpenRLHF, RLHF, PPO, GRPO, RLOO, DPO, Ray, vLLM, Distributed Training, Large Models, ZeRO-3]\ndependencies: [openrlhf, ray, vllm, torch, transformers, deepspeed]\n---\n\n# OpenRLHF - High-Performance RLHF Training\n\n## Quick start\n\nOpenRLHF is a Ray-based RLHF framework optimized for distributed training with vLLM inference acceleration.\n\n**Installation**:\n```bash\n# Launch Docker container\ndocker run --runtime=nvidia -it --rm --shm-size=\"10g\" --cap-add=SYS_ADMIN \\\n  -v $PWD:\u002Fopenrlhf nvcr.io\u002Fnvidia\u002Fpytorch:25.02-py3 bash\n\n# Uninstall conflicts\nsudo pip uninstall xgboost transformer_engine flash_attn pynvml -y\n\n# Install OpenRLHF with vLLM\npip install openrlhf[vllm]\n```\n\n**PPO Training** (Hybrid Engine):\n```bash\nray start --head --node-ip-address 0.0.0.0 --num-gpus 8\n\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  --runtime-env-json='{\"working_dir\": \"\u002Fopenrlhf\"}' \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --vllm_gpu_memory_utilization 0.5 \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain OpenRLHF\u002FLlama-3-8b-rm-700k \\\n  --save_path .\u002Foutput\u002Fllama3-8b-rlhf \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \\\n  --zero_stage 3 --bf16 \\\n  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \\\n  --init_kl_coef 0.01 --normalize_reward \\\n  --gradient_checkpointing --packing_samples \\\n  --vllm_enable_sleep --deepspeed_enable_sleep\n```\n\n**GRPO Training** (Group Normalized Policy Optimization):\n```bash\n# Same command as PPO, but add:\n--advantage_estimator group_norm\n```\n\n## Common workflows\n\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)\n\n**Step 1: Train reward model** (DPO):\n```bash\ndeepspeed --module openrlhf.cli.train_rm \\\n  --save_path .\u002Foutput\u002Fllama3-8b-rm \\\n  --save_steps -1 --logging_steps 1 \\\n  --eval_steps -1 --train_batch_size 256 \\\n  --micro_train_batch_size 1 --pretrain meta-llama\u002FMeta-Llama-3-8B \\\n  --bf16 --max_epochs 1 --max_len 8192 \\\n  --zero_stage 3 --learning_rate 9e-6 \\\n  --dataset OpenRLHF\u002Fpreference_dataset_mixture2_and_safe_pku \\\n  --apply_chat_template --chosen_key chosen \\\n  --rejected_key rejected --flash_attn --gradient_checkpointing\n```\n\n**Step 2: PPO training**:\n```bash\nray start --head --node-ip-address 0.0.0.0 --num-gpus 8\n\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain .\u002Foutput\u002Fllama3-8b-rm \\\n  --save_path .\u002Foutput\u002Fllama3-8b-ppo \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \\\n  --zero_stage 3 --bf16 \\\n  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \\\n  --init_kl_coef 0.01 --normalize_reward \\\n  --vllm_enable_sleep --deepspeed_enable_sleep\n```\n\n### Workflow 2: GRPO training (no critic model needed)\n\nMemory-efficient alternative to PPO:\n\n```bash\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --advantage_estimator group_norm \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain OpenRLHF\u002FLlama-3-8b-rm-700k \\\n  --save_path .\u002Foutput\u002Fllama3-8b-grpo \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --bf16 \\\n  --actor_learning_rate 5e-7 \\\n  --init_kl_coef 0.01 --use_kl_loss --kl_estimator k3 \\\n  --normalize_reward --no_advantage_std_norm\n```\n\n**Key GRPO parameters**:\n- `--advantage_estimator group_norm` - Enables GRPO\n- `--use_kl_loss` - KL loss from GRPO paper\n- `--kl_estimator k3` - Loss function (k2 ≈ k1)\n- `--no_advantage_std_norm` - Disables std normalization\n\n### Workflow 3: DPO training (preference optimization)\n\nSimpler alternative without reward model:\n\n```bash\ndeepspeed --module openrlhf.cli.train_dpo \\\n  --save_path .\u002Foutput\u002Fllama3-8b-dpo \\\n  --save_steps -1 --logging_steps 1 \\\n  --eval_steps -1 --train_batch_size 256 \\\n  --micro_train_batch_size 2 --pretrain meta-llama\u002FMeta-Llama-3-8B \\\n  --bf16 --max_epochs 1 --max_len 8192 \\\n  --zero_stage 3 --learning_rate 5e-7 --beta 0.1 \\\n  --dataset OpenRLHF\u002Fpreference_dataset_mixture2_and_safe_pku \\\n  --apply_chat_template --chosen_key chosen \\\n  --rejected_key rejected --flash_attn --gradient_checkpointing\n```\n\n## When to use vs alternatives\n\n**Use OpenRLHF when**:\n- Training large models (7B-70B+) with RL\n- Need vLLM inference acceleration\n- Want distributed architecture with Ray\n- Have multi-node GPU cluster\n- Need PPO\u002FGRPO\u002FRLOO\u002FDPO in one framework\n\n**Algorithm selection**:\n- **PPO**: Maximum control, best for complex rewards\n- **GRPO**: Memory-efficient, no critic needed\n- **RLOO**: Modified PPO with per-token KL\n- **REINFORCE++**: More stable than GRPO, faster than PPO\n- **DPO**: Simplest, no reward model needed\n\n**Use alternatives instead**:\n- **TRL**: Single-node training, simpler API\n- **veRL**: ByteDance's framework for 671B models\n- **DeepSpeedChat**: Integrated with DeepSpeed ecosystem\n\n## Common issues\n\n**Issue: GPU OOM with large models**\n\nDisable model colocation:\n```bash\n# Remove --colocate_all_models flag\n# Allocate separate GPUs for each model\n--actor_num_gpus_per_node 8 \\\n--critic_num_gpus_per_node 8 \\\n--reward_num_gpus_per_node 8 \\\n--ref_num_gpus_per_node 8\n```\n\n**Issue: DeepSpeed GPU index out of range**\n\nSet environment variable:\n```bash\nexport RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1\n```\n\n**Issue: Training instability**\n\nUse Hybrid Engine instead of async:\n```bash\n--colocate_all_models \\\n--vllm_enable_sleep \\\n--deepspeed_enable_sleep\n```\n\nAdjust KL coefficient:\n```bash\n--init_kl_coef 0.05  # Increase from 0.01\n```\n\n**Issue: Slow generation during PPO**\n\nEnable vLLM acceleration:\n```bash\n--vllm_num_engines 4 \\\n--vllm_tensor_parallel_size 2 \\\n--vllm_gpu_memory_utilization 0.5\n```\n\n## Advanced topics\n\n**Hybrid Engine GPU sharing**: See [references\u002Fhybrid-engine.md](references\u002Fhybrid-engine.md) for vLLM sleep mode, DeepSpeed sleep mode, and optimal node allocation.\n\n**Algorithm comparison**: See [references\u002Falgorithm-comparison.md](references\u002Falgorithm-comparison.md) for PPO vs GRPO vs RLOO vs REINFORCE++ benchmarks and hyperparameters.\n\n**Multi-node setup**: See [references\u002Fmulti-node-training.md](references\u002Fmulti-node-training.md) for Ray cluster configuration and fault tolerance.\n\n**Custom reward functions**: See [references\u002Fcustom-rewards.md](references\u002Fcustom-rewards.md) for reinforced fine-tuning and agent RLHF.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA A100\u002FH100 recommended\n- **VRAM**:\n  - 7B model: 8× A100 40GB (Hybrid Engine)\n  - 70B model: 48× A100 80GB (vLLM:Actor:Critic = 1:1:1)\n- **Multi-node**: Ray cluster with InfiniBand recommended\n- **Docker**: NVIDIA PyTorch container 25.02+\n\n**Performance**:\n- 2× faster than DeepSpeedChat\n- vLLM inference acceleration\n- Hybrid Engine minimizes GPU idle time\n\n## Resources\n\n- Docs: https:\u002F\u002Fgithub.com\u002FOpenRLHF\u002FOpenRLHF\n- Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2405.11143\n- Examples: https:\u002F\u002Fgithub.com\u002FOpenRLHF\u002FOpenRLHF\u002Ftree\u002Fmain\u002Fexamples\n- Discord: Community support\n\n\n\n","# OpenRLHF - High-Performance RLHF Training\n\n## Quick start\n\nOpenRLHF is a Ray-based RLHF framework optimized for distributed training with vLLM inference acceleration.\n\n**Installation**:\n```bash\n# Launch Docker container\ndocker run --runtime=nvidia -it --rm --shm-size=\"10g\" --cap-add=SYS_ADMIN \\\n  -v $PWD:\u002Fopenrlhf nvcr.io\u002Fnvidia\u002Fpytorch:25.02-py3 bash\n\n# Uninstall conflicts\nsudo pip uninstall xgboost transformer_engine flash_attn pynvml -y\n\n# Install OpenRLHF with vLLM\npip install openrlhf[vllm]\n```\n\n**PPO Training** (Hybrid Engine):\n```bash\nray start --head --node-ip-address 0.0.0.0 --num-gpus 8\n\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  --runtime-env-json='{\"working_dir\": \"\u002Fopenrlhf\"}' \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --vllm_gpu_memory_utilization 0.5 \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain OpenRLHF\u002FLlama-3-8b-rm-700k \\\n  --save_path .\u002Foutput\u002Fllama3-8b-rlhf \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \\\n  --zero_stage 3 --bf16 \\\n  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \\\n  --init_kl_coef 0.01 --normalize_reward \\\n  --gradient_checkpointing --packing_samples \\\n  --vllm_enable_sleep --deepspeed_enable_sleep\n```\n\n**GRPO Training** (Group Normalized Policy Optimization):\n```bash\n# Same command as PPO, but add:\n--advantage_estimator group_norm\n```\n\n## Common workflows\n\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)\n\n**Step 1: Train reward model** (DPO):\n```bash\ndeepspeed --module openrlhf.cli.train_rm \\\n  --save_path .\u002Foutput\u002Fllama3-8b-rm \\\n  --save_steps -1 --logging_steps 1 \\\n  --eval_steps -1 --train_batch_size 256 \\\n  --micro_train_batch_size 1 --pretrain meta-llama\u002FMeta-Llama-3-8B \\\n  --bf16 --max_epochs 1 --max_len 8192 \\\n  --zero_stage 3 --learning_rate 9e-6 \\\n  --dataset OpenRLHF\u002Fpreference_dataset_mixture2_and_safe_pku \\\n  --apply_chat_template --chosen_key chosen \\\n  --rejected_key rejected --flash_attn --gradient_checkpointing\n```\n\n**Step 2: PPO training**:\n```bash\nray start --head --node-ip-address 0.0.0.0 --num-gpus 8\n\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --critic_num_nodes 1 --critic_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain .\u002Foutput\u002Fllama3-8b-rm \\\n  --save_path .\u002Foutput\u002Fllama3-8b-ppo \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --prompt_max_len 1024 --generate_max_len 1024 \\\n  --zero_stage 3 --bf16 \\\n  --actor_learning_rate 5e-7 --critic_learning_rate 9e-6 \\\n  --init_kl_coef 0.01 --normalize_reward \\\n  --vllm_enable_sleep --deepspeed_enable_sleep\n```\n\n### Workflow 2: GRPO training (no critic model needed)\n\nMemory-efficient alternative to PPO:\n\n```bash\nray job submit --address=\"http:\u002F\u002F127.0.0.1:8265\" \\\n  -- python3 -m openrlhf.cli.train_ppo_ray \\\n  --advantage_estimator group_norm \\\n  --ref_num_nodes 1 --ref_num_gpus_per_node 8 \\\n  --reward_num_nodes 1 --reward_num_gpus_per_node 8 \\\n  --actor_num_nodes 1 --actor_num_gpus_per_node 8 \\\n  --vllm_num_engines 4 --vllm_tensor_parallel_size 2 \\\n  --colocate_all_models \\\n  --pretrain OpenRLHF\u002FLlama-3-8b-sft-mixture \\\n  --reward_pretrain OpenRLHF\u002FLlama-3-8b-rm-700k \\\n  --save_path .\u002Foutput\u002Fllama3-8b-grpo \\\n  --micro_train_batch_size 8 --train_batch_size 128 \\\n  --micro_rollout_batch_size 16 --rollout_batch_size 1024 \\\n  --max_epochs 1 --bf16 \\\n  --actor_learning_rate 5e-7 \\\n  --init_kl_coef 0.01 --use_kl_loss --kl_estimator k3 \\\n  --normalize_reward --no_advantage_std_norm\n```\n\n**Key GRPO parameters**:\n- `--advantage_estimator group_norm` - Enables GRPO\n- `--use_kl_loss` - KL loss from GRPO paper\n- `--kl_estimator k3` - Loss function (k2 ≈ k1)\n- `--no_advantage_std_norm` - Disables std normalization\n\n### Workflow 3: DPO training (preference optimization)\n\nSimpler alternative without reward model:\n\n```bash\ndeepspeed --module openrlhf.cli.train_dpo \\\n  --save_path .\u002Foutput\u002Fllama3-8b-dpo \\\n  --save_steps -1 --logging_steps 1 \\\n  --eval_steps -1 --train_batch_size 256 \\\n  --micro_train_batch_size 2 --pretrain meta-llama\u002FMeta-Llama-3-8B \\\n  --bf16 --max_epochs 1 --max_len 8192 \\\n  --zero_stage 3 --learning_rate 5e-7 --beta 0.1 \\\n  --dataset OpenRLHF\u002Fpreference_dataset_mixture2_and_safe_pku \\\n  --apply_chat_template --chosen_key chosen \\\n  --rejected_key rejected --flash_attn --gradient_checkpointing\n```\n\n## When to use vs alternatives\n\n**Use OpenRLHF when**:\n- Training large models (7B-70B+) with RL\n- Need vLLM inference acceleration\n- Want distributed architecture with Ray\n- Have multi-node GPU cluster\n- Need PPO\u002FGRPO\u002FRLOO\u002FDPO in one framework\n\n**Algorithm selection**:\n- **PPO**: Maximum control, best for complex rewards\n- **GRPO**: Memory-efficient, no critic needed\n- **RLOO**: Modified PPO with per-token KL\n- **REINFORCE++**: More stable than GRPO, faster than PPO\n- **DPO**: Simplest, no reward model needed\n\n**Use alternatives instead**:\n- **TRL**: Single-node training, simpler API\n- **veRL**: ByteDance's framework for 671B models\n- **DeepSpeedChat**: Integrated with DeepSpeed ecosystem\n\n## Common issues\n\n**Issue: GPU OOM with large models**\n\nDisable model colocation:\n```bash\n# Remove --colocate_all_models flag\n# Allocate separate GPUs for each model\n--actor_num_gpus_per_node 8 \\\n--critic_num_gpus_per_node 8 \\\n--reward_num_gpus_per_node 8 \\\n--ref_num_gpus_per_node 8\n```\n\n**Issue: DeepSpeed GPU index out of range**\n\nSet environment variable:\n```bash\nexport RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=1\n```\n\n**Issue: Training instability**\n\nUse Hybrid Engine instead of async:\n```bash\n--colocate_all_models \\\n--vllm_enable_sleep \\\n--deepspeed_enable_sleep\n```\n\nAdjust KL coefficient:\n```bash\n--init_kl_coef 0.05  # Increase from 0.01\n```\n\n**Issue: Slow generation during PPO**\n\nEnable vLLM acceleration:\n```bash\n--vllm_num_engines 4 \\\n--vllm_tensor_parallel_size 2 \\\n--vllm_gpu_memory_utilization 0.5\n```\n\n## Advanced topics\n\n**Hybrid Engine GPU sharing**: See [references\u002Fhybrid-engine.md](references\u002Fhybrid-engine.md) for vLLM sleep mode, DeepSpeed sleep mode, and optimal node allocation.\n\n**Algorithm comparison**: See [references\u002Falgorithm-comparison.md](references\u002Falgorithm-comparison.md) for PPO vs GRPO vs RLOO vs REINFORCE++ benchmarks and hyperparameters.\n\n**Multi-node setup**: See [references\u002Fmulti-node-training.md](references\u002Fmulti-node-training.md) for Ray cluster configuration and fault tolerance.\n\n**Custom reward functions**: See [references\u002Fcustom-rewards.md](references\u002Fcustom-rewards.md) for reinforced fine-tuning and agent RLHF.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA A100\u002FH100 recommended\n- **VRAM**:\n  - 7B model: 8× A100 40GB (Hybrid Engine)\n  - 70B model: 48× A100 80GB (vLLM:Actor:Critic = 1:1:1)\n- **Multi-node**: Ray cluster with InfiniBand recommended\n- **Docker**: NVIDIA PyTorch container 25.02+\n\n**Performance**:\n- 2× faster than DeepSpeedChat\n- vLLM inference acceleration\n- Hybrid Engine minimizes GPU idle time\n\n## Resources\n\n- Docs: https:\u002F\u002Fgithub.com\u002FOpenRLHF\u002FOpenRLHF\n- Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2405.11143\n- Examples: https:\u002F\u002Fgithub.com\u002FOpenRLHF\u002FOpenRLHF\u002Ftree\u002Fmain\u002Fexamples\n- Discord: Community support\n\n\n\n",{"name":5,"tags":24,"author":37,"license":15,"version":38,"description":7,"dependencies":39},[25,26,27,28,29,30,31,32,33,34,35,36],"Post-Training","OpenRLHF","RLHF","PPO","GRPO","RLOO","DPO","Ray","vLLM","Distributed Training","Large Models","ZeRO-3","Orchestra Research","1.0.0",[40,41,42,43,44,45],"openrlhf","ray","vllm","torch","transformers","deepspeed","main","f4a24bd1ab047411d4ae57ee9a96701a7706e779","claude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Fpost-training-openrlhf","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive","https:\u002F\u002Fgithub.com\u002Fcocyuhao","Beijing Wuzi University","2026-05-12 10:57:02",{"source":8,"skillName":5,"commands":54,"prompt":64,"zipAvailable":65},[55,58,61],{"tool":56,"command":57},"npx","npx skills add cocyuhao\u002Fai-dev-tooling-research-archive --skill openrlhf-training",{"tool":59,"command":60},"bunx","bunx skills add cocyuhao\u002Fai-dev-tooling-research-archive --skill openrlhf-training",{"tool":62,"command":63},"pnpm","pnpm dlx skills add cocyuhao\u002Fai-dev-tooling-research-archive --skill openrlhf-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Fpost-training-openrlhf\u002FSKILL.md",true,[67,73,77,81],{"path":68,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":71,"sourceHtmlUrl":72},"references\u002Falgorithm-comparison.md","REFERENCE","参考文档",9783,"https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Fpost-training-openrlhf\u002Freferences\u002Falgorithm-comparison.md",{"path":74,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":75,"sourceHtmlUrl":76},"references\u002Fcustom-rewards.md",15865,"https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Fpost-training-openrlhf\u002Freferences\u002Fcustom-rewards.md",{"path":78,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":79,"sourceHtmlUrl":80},"references\u002Fhybrid-engine.md",7266,"https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Fpost-training-openrlhf\u002Freferences\u002Fhybrid-engine.md",{"path":82,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Fmulti-node-training.md",11099,"https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Fpost-training-openrlhf\u002Freferences\u002Fmulti-node-training.md",[86,94,102,109,116,123],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":93},"01M72JQ593HE3SRJD3446CCKA8","web-access","所有联网操作必须通过此 skill 处理，包括：搜索、网页抓取、登录后操作、网络交互等。 触发场景：用户要求搜索信息、查看网页内容、访问需要登录的网站、操作网页界面、抓取社交媒体内容（小红书、微博、推特等）、读取动态渲染页面、以及任何需要真实浏览器环境的网络任务。","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fweb-access\u002FSKILL.md",[],[],"2026-11-10 22:39:42",{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":98,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":93},"01M72JKXJYFT2RBJ2HP5ZRA853","last30days","Multi-query social research across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web. Intelligent query planning with Gemini\u002FOpenAI fallback.","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Flast30days-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Flast30days-skill\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":93},"01M72JKMRQSD0ZY66KQ863HVY5","writing-plans","Use when you have a spec or requirements for a multi-step task. Creates comprehensive implementation plans with bite-sized tasks, exact file paths, and complete code examples.","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fsoftware-development\u002Fwriting-plans\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":93},"01M72JKHCV8DT6VWSD4KS5YWT9","test-driven-development","Use when implementing any feature or bugfix, before writing implementation code. Enforces RED-GREEN-REFACTOR cycle with test-first approach.","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fsoftware-development\u002Ftest-driven-development\u002FSKILL.md",[],[],{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":93},"01M72JKD7VX7EQM38R704F3KT1","systematic-debugging","Use when encountering any bug, test failure, or unexpected behavior. 4-phase root cause investigation — NO fixes without understanding the problem first.","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fsoftware-development\u002Fsystematic-debugging\u002FSKILL.md",[],[],{"skillNo":124,"name":125,"nameZh":6,"description":126,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":127,"categories":128,"keywords":129,"publishedAt":93},"01M72JKA9XXB92FS8JHPCYW8CM","subagent-driven-development","Use when executing implementation plans with independent tasks. Dispatches fresh delegate_task per task with two-stage review (spec compliance then code quality).","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fsoftware-development\u002Fsubagent-driven-development\u002FSKILL.md",[],[]]