[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6GBA2ZKTN1F5JDHNTW0Z0BC":3,"skill-same:01M6GBA2ZKTN1F5JDHNTW0Z0BC":84},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":62,"commitSha":63,"skillDirectory":64,"forksCount":65,"repositoryHtmlUrl":66,"repositoryHomepageUrl":6,"ownerHtmlUrl":67,"ownerBlogUrl":6,"ownerDescription":68,"updatedAt":69,"install":70,"resources":83},"01M6GBA2ZKTN1F5JDHNTW0Z0BC","fine-tuning-with-trl",null,"TRL: SFT, DPO, PPO, GRPO, reward modeling for LLM RLHF.","zapabob\u002FSkills","zapabob","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F130830376?v=4","01M6DBMYZAGRTFQDSQYW3W764P","峯岸 亮","01M6DBMZ3GKDSJ01NZ9XTZD7JD",2,"MIT","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Farchive\u002Fskill-md-reviewed\u002Fsha256\u002F97\u002F97554ed569d39c185eb0c3c7033a72975b0902377621b8e0cb8925b535ebd878\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"tuning","with","TRL","fine tuning with trl","fine tuning","DPO","Post-Training","RLHF","PPO","Transformers","macos","learning","Fine-Tuning","alignment","training","modeling","Accelerate","windows","Reward Modeling","linux","2026-11-10 10:12:30",false,"---\r\nname: fine-tuning-with-trl\r\ndescription: \"TRL: SFT, DPO, PPO, GRPO, reward modeling for LLM RLHF.\"\r\nversion: 1.0.0\r\nauthor: Orchestra Research\r\nlicense: MIT\r\ndependencies: [trl, transformers, datasets, peft, accelerate, torch]\r\nplatforms: [linux, macos, windows]\r\nmetadata:\r\n  hermes:\r\n    tags: [Post-Training, TRL, Reinforcement Learning, Fine-Tuning, SFT, DPO, PPO, GRPO, RLHF, Preference Alignment, HuggingFace]\r\n\r\n---\r\n\r\n# TRL - Transformer Reinforcement Learning\r\n\r\n## Quick start\r\n\r\nTRL provides post-training methods for aligning language models with human preferences.\r\n\r\n**Installation**:\r\n```bash\r\npip install trl transformers datasets peft accelerate\r\n```\r\n\r\n**Supervised Fine-Tuning** (instruction tuning):\r\n```python\r\nfrom trl import SFTTrainer\r\n\r\ntrainer = SFTTrainer(\r\n    model=\"Qwen\u002FQwen2.5-0.5B\",\r\n    train_dataset=dataset,  # Prompt-completion pairs\r\n)\r\ntrainer.train()\r\n```\r\n\r\n**DPO** (align with preferences):\r\n```python\r\nfrom trl import DPOTrainer, DPOConfig\r\n\r\nconfig = DPOConfig(output_dir=\"model-dpo\", beta=0.1)\r\ntrainer = DPOTrainer(\r\n    model=model,\r\n    args=config,\r\n    train_dataset=preference_dataset,  # chosen\u002Frejected pairs\r\n    processing_class=tokenizer\r\n)\r\ntrainer.train()\r\n```\r\n\r\n## Common workflows\r\n\r\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)\r\n\r\nComplete pipeline from base model to human-aligned model.\r\n\r\nCopy this checklist:\r\n\r\n```\r\nRLHF Training:\r\n- [ ] Step 1: Supervised fine-tuning (SFT)\r\n- [ ] Step 2: Train reward model\r\n- [ ] Step 3: PPO reinforcement learning\r\n- [ ] Step 4: Evaluate aligned model\r\n```\r\n\r\n**Step 1: Supervised fine-tuning**\r\n\r\nTrain base model on instruction-following data:\r\n\r\n```python\r\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\r\nfrom trl import SFTTrainer, SFTConfig\r\nfrom datasets import load_dataset\r\n\r\n# Load model\r\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\r\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\r\n\r\n# Load instruction dataset\r\ndataset = load_dataset(\"trl-lib\u002FCapybara\", split=\"train\")\r\n\r\n# Configure training\r\ntraining_args = SFTConfig(\r\n    output_dir=\"Qwen2.5-0.5B-SFT\",\r\n    per_device_train_batch_size=4,\r\n    num_train_epochs=1,\r\n    learning_rate=2e-5,\r\n    logging_steps=10,\r\n    save_strategy=\"epoch\"\r\n)\r\n\r\n# Train\r\ntrainer = SFTTrainer(\r\n    model=model,\r\n    args=training_args,\r\n    train_dataset=dataset,\r\n    tokenizer=tokenizer\r\n)\r\ntrainer.train()\r\ntrainer.save_model()\r\n```\r\n\r\n**Step 2: Train reward model**\r\n\r\nTrain model to predict human preferences:\r\n\r\n```python\r\nfrom transformers import AutoModelForSequenceClassification\r\nfrom trl import RewardTrainer, RewardConfig\r\n\r\n# Load SFT model as base\r\nmodel = AutoModelForSequenceClassification.from_pretrained(\r\n    \"Qwen2.5-0.5B-SFT\",\r\n    num_labels=1  # Single reward score\r\n)\r\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen2.5-0.5B-SFT\")\r\n\r\n# Load preference data (chosen\u002Frejected pairs)\r\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\r\n\r\n# Configure training\r\ntraining_args = RewardConfig(\r\n    output_dir=\"Qwen2.5-0.5B-Reward\",\r\n    per_device_train_batch_size=2,\r\n    num_train_epochs=1,\r\n    learning_rate=1e-5\r\n)\r\n\r\n# Train reward model\r\ntrainer = RewardTrainer(\r\n    model=model,\r\n    args=training_args,\r\n    processing_class=tokenizer,\r\n    train_dataset=dataset\r\n)\r\ntrainer.train()\r\ntrainer.save_model()\r\n```\r\n\r\n**Step 3: PPO reinforcement learning**\r\n\r\nOptimize policy using reward model:\r\n\r\n```bash\r\npython -m trl.scripts.ppo \\\r\n    --model_name_or_path Qwen2.5-0.5B-SFT \\\r\n    --reward_model_path Qwen2.5-0.5B-Reward \\\r\n    --dataset_name trl-internal-testing\u002Fdescriptiveness-sentiment-trl-style \\\r\n    --output_dir Qwen2.5-0.5B-PPO \\\r\n    --learning_rate 3e-6 \\\r\n    --per_device_train_batch_size 64 \\\r\n    --total_episodes 10000\r\n```\r\n\r\n**Step 4: Evaluate**\r\n\r\n```python\r\nfrom transformers import pipeline\r\n\r\n# Load aligned model\r\ngenerator = pipeline(\"text-generation\", model=\"Qwen2.5-0.5B-PPO\")\r\n\r\n# Test\r\nprompt = \"Explain quantum computing to a 10-year-old\"\r\noutput = generator(prompt, max_length=200)[0][\"generated_text\"]\r\nprint(output)\r\n```\r\n\r\n### Workflow 2: Simple preference alignment with DPO\r\n\r\nAlign model with preferences without reward model.\r\n\r\nCopy this checklist:\r\n\r\n```\r\nDPO Training:\r\n- [ ] Step 1: Prepare preference dataset\r\n- [ ] Step 2: Configure DPO\r\n- [ ] Step 3: Train with DPOTrainer\r\n- [ ] Step 4: Evaluate alignment\r\n```\r\n\r\n**Step 1: Prepare preference dataset**\r\n\r\nDataset format:\r\n```json\r\n{\r\n  \"prompt\": \"What is the capital of France?\",\r\n  \"chosen\": \"The capital of France is Paris.\",\r\n  \"rejected\": \"I don't know.\"\r\n}\r\n```\r\n\r\nLoad dataset:\r\n```python\r\nfrom datasets import load_dataset\r\n\r\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\r\n# Or load your own\r\n# dataset = load_dataset(\"json\", data_files=\"preferences.json\")\r\n```\r\n\r\n**Step 2: Configure DPO**\r\n\r\n```python\r\nfrom trl import DPOConfig\r\n\r\nconfig = DPOConfig(\r\n    output_dir=\"Qwen2.5-0.5B-DPO\",\r\n    per_device_train_batch_size=4,\r\n    num_train_epochs=1,\r\n    learning_rate=5e-7,\r\n    beta=0.1,  # KL penalty strength\r\n    max_prompt_length=512,\r\n    max_length=1024,\r\n    logging_steps=10\r\n)\r\n```\r\n\r\n**Step 3: Train with DPOTrainer**\r\n\r\n```python\r\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\r\nfrom trl import DPOTrainer\r\n\r\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\r\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\r\n\r\ntrainer = DPOTrainer(\r\n    model=model,\r\n    args=config,\r\n    train_dataset=dataset,\r\n    processing_class=tokenizer\r\n)\r\n\r\ntrainer.train()\r\ntrainer.save_model()\r\n```\r\n\r\n**CLI alternative**:\r\n```bash\r\ntrl dpo \\\r\n    --model_name_or_path Qwen\u002FQwen2.5-0.5B-Instruct \\\r\n    --dataset_name argilla\u002FCapybara-Preferences \\\r\n    --output_dir Qwen2.5-0.5B-DPO \\\r\n    --per_device_train_batch_size 4 \\\r\n    --learning_rate 5e-7 \\\r\n    --beta 0.1\r\n```\r\n\r\n### Workflow 3: Memory-efficient online RL with GRPO\r\n\r\nTrain with reinforcement learning using minimal memory.\r\n\r\nFor in-depth GRPO guidance — reward function design, critical training insights (loss behavior, mode collapse, tuning), and advanced multi-stage patterns — see **[references\u002Fgrpo-training.md](references\u002Fgrpo-training.md)**. A production-ready training script is in **[templates\u002Fbasic_grpo_training.py](templates\u002Fbasic_grpo_training.py)**.\r\n\r\nCopy this checklist:\r\n\r\n```\r\nGRPO Training:\r\n- [ ] Step 1: Define reward function\r\n- [ ] Step 2: Configure GRPO\r\n- [ ] Step 3: Train with GRPOTrainer\r\n```\r\n\r\n**Step 1: Define reward function**\r\n\r\n```python\r\ndef reward_function(completions, **kwargs):\r\n    \"\"\"\r\n    Compute rewards for completions.\r\n\r\n    Args:\r\n        completions: List of generated texts\r\n\r\n    Returns:\r\n        List of reward scores (floats)\r\n    \"\"\"\r\n    rewards = []\r\n    for completion in completions:\r\n        # Example: reward based on length and unique words\r\n        score = len(completion.split())  # Favor longer responses\r\n        score += len(set(completion.lower().split()))  # Reward unique words\r\n        rewards.append(score)\r\n    return rewards\r\n```\r\n\r\nOr use a reward model:\r\n```python\r\nfrom transformers import pipeline\r\n\r\nreward_model = pipeline(\"text-classification\", model=\"reward-model-path\")\r\n\r\ndef reward_from_model(completions, prompts, **kwargs):\r\n    # Combine prompt + completion\r\n    full_texts = [p + c for p, c in zip(prompts, completions)]\r\n    # Get reward scores\r\n    results = reward_model(full_texts)\r\n    return [r[\"score\"] for r in results]\r\n```\r\n\r\n**Step 2: Configure GRPO**\r\n\r\n```python\r\nfrom trl import GRPOConfig\r\n\r\nconfig = GRPOConfig(\r\n    output_dir=\"Qwen2-GRPO\",\r\n    per_device_train_batch_size=4,\r\n    num_train_epochs=1,\r\n    learning_rate=1e-5,\r\n    num_generations=4,  # Generate 4 completions per prompt\r\n    max_new_tokens=128\r\n)\r\n```\r\n\r\n**Step 3: Train with GRPOTrainer**\r\n\r\n```python\r\nfrom datasets import load_dataset\r\nfrom trl import GRPOTrainer\r\n\r\n# Load prompt-only dataset\r\ndataset = load_dataset(\"trl-lib\u002Ftldr\", split=\"train\")\r\n\r\ntrainer = GRPOTrainer(\r\n    model=\"Qwen\u002FQwen2-0.5B-Instruct\",\r\n    reward_funcs=reward_function,  # Your reward function\r\n    args=config,\r\n    train_dataset=dataset\r\n)\r\n\r\ntrainer.train()\r\n```\r\n\r\n**CLI**:\r\n```bash\r\ntrl grpo \\\r\n    --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\r\n    --dataset_name trl-lib\u002Ftldr \\\r\n    --output_dir Qwen2-GRPO \\\r\n    --num_generations 4\r\n```\r\n\r\n## When to use vs alternatives\r\n\r\n**Use TRL when:**\r\n- Need to align model with human preferences\r\n- Have preference data (chosen\u002Frejected pairs)\r\n- Want to use reinforcement learning (PPO, GRPO)\r\n- Need reward model training\r\n- Doing RLHF (full pipeline)\r\n\r\n**Method selection**:\r\n- **SFT**: Have prompt-completion pairs, want basic instruction following\r\n- **DPO**: Have preferences, want simple alignment (no reward model needed)\r\n- **PPO**: Have reward model, need maximum control over RL\r\n- **GRPO**: Memory-constrained, want online RL\r\n- **Reward Model**: Building RLHF pipeline, need to score generations\r\n\r\n**Use alternatives instead:**\r\n- **HuggingFace Trainer**: Basic fine-tuning without RL\r\n- **Axolotl**: YAML-based training configuration\r\n- **LitGPT**: Educational, minimal fine-tuning\r\n- **Unsloth**: Fast LoRA training\r\n\r\n## Common issues\r\n\r\n**Issue: OOM during DPO training**\r\n\r\nReduce batch size and sequence length:\r\n```python\r\nconfig = DPOConfig(\r\n    per_device_train_batch_size=1,  # Reduce from 4\r\n    max_length=512,  # Reduce from 1024\r\n    gradient_accumulation_steps=8  # Maintain effective batch\r\n)\r\n```\r\n\r\nOr use gradient checkpointing:\r\n```python\r\nmodel.gradient_checkpointing_enable()\r\n```\r\n\r\n**Issue: Poor alignment quality**\r\n\r\nTune beta parameter:\r\n```python\r\n# Higher beta = more conservative (stays closer to reference)\r\nconfig = DPOConfig(beta=0.5)  # Default 0.1\r\n\r\n# Lower beta = more aggressive alignment\r\nconfig = DPOConfig(beta=0.01)\r\n```\r\n\r\n**Issue: Reward model not learning**\r\n\r\nCheck loss type and learning rate:\r\n```python\r\nconfig = RewardConfig(\r\n    learning_rate=1e-5,  # Try different LR\r\n    num_train_epochs=3  # Train longer\r\n)\r\n```\r\n\r\nEnsure preference dataset has clear winners:\r\n```python\r\n# Verify dataset\r\nprint(dataset[0])\r\n# Should have clear chosen > rejected\r\n```\r\n\r\n**Issue: PPO training unstable**\r\n\r\nAdjust KL coefficient:\r\n```python\r\nconfig = PPOConfig(\r\n    kl_coef=0.1,  # Increase from 0.05\r\n    cliprange=0.1  # Reduce from 0.2\r\n)\r\n```\r\n\r\n## Advanced topics\r\n\r\n**SFT training guide**: See [references\u002Fsft-training.md](references\u002Fsft-training.md) for dataset formats, chat templates, packing strategies, and multi-GPU training.\r\n\r\n**DPO variants**: See [references\u002Fdpo-variants.md](references\u002Fdpo-variants.md) for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.\r\n\r\n**Reward modeling**: See [references\u002Freward-modeling.md](references\u002Freward-modeling.md) for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.\r\n\r\n**Online RL methods**: See [references\u002Fonline-rl.md](references\u002Fonline-rl.md) for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.\r\n\r\n**GRPO deep dive**: See [references\u002Fgrpo-training.md](references\u002Fgrpo-training.md) for expert-level GRPO patterns — reward function design philosophy, training insights (why loss increases, mode collapse detection), hyperparameter tuning, multi-stage training, and troubleshooting. Production-ready template in [templates\u002Fbasic_grpo_training.py](templates\u002Fbasic_grpo_training.py).\r\n\r\n## Hardware requirements\r\n\r\n- **GPU**: NVIDIA (CUDA required)\r\n- **VRAM**: Depends on model and method\r\n  - SFT 7B: 16GB (with LoRA)\r\n  - DPO 7B: 24GB (stores reference model)\r\n  - PPO 7B: 40GB (policy + reward model)\r\n  - GRPO 7B: 24GB (more memory efficient)\r\n- **Multi-GPU**: Supported via `accelerate`\r\n- **Mixed precision**: BF16 recommended (A100\u002FH100)\r\n\r\n**Memory optimization**:\r\n- Use LoRA\u002FQLoRA for all methods\r\n- Enable gradient checkpointing\r\n- Use smaller batch sizes with gradient accumulation\r\n\r\n## Resources\r\n\r\n- Docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002F\r\n- GitHub: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\r\n- Papers:\r\n  - \"Training language models to follow instructions with human feedback\" (InstructGPT, 2022)\r\n  - \"Direct Preference Optimization: Your Language Model is Secretly a Reward Model\" (DPO, 2023)\r\n  - \"Group Relative Policy Optimization\" (GRPO, 2024)\r\n- Examples: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002Fmain\u002Fexamples\u002Fscripts\r\n","# TRL - Transformer Reinforcement Learning\r\n\r\n## Quick start\r\n\r\nTRL provides post-training methods for aligning language models with human preferences.\r\n\r\n**Installation**:\r\n```bash\r\npip install trl transformers datasets peft accelerate\r\n```\r\n\r\n**Supervised Fine-Tuning** (instruction tuning):\r\n```python\r\nfrom trl import SFTTrainer\r\n\r\ntrainer = SFTTrainer(\r\n    model=\"Qwen\u002FQwen2.5-0.5B\",\r\n    train_dataset=dataset,  # Prompt-completion pairs\r\n)\r\ntrainer.train()\r\n```\r\n\r\n**DPO** (align with preferences):\r\n```python\r\nfrom trl import DPOTrainer, DPOConfig\r\n\r\nconfig = DPOConfig(output_dir=\"model-dpo\", beta=0.1)\r\ntrainer = DPOTrainer(\r\n    model=model,\r\n    args=config,\r\n    train_dataset=preference_dataset,  # chosen\u002Frejected pairs\r\n    processing_class=tokenizer\r\n)\r\ntrainer.train()\r\n```\r\n\r\n## Common workflows\r\n\r\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)\r\n\r\nComplete pipeline from base model to human-aligned model.\r\n\r\nCopy this checklist:\r\n\r\n```\r\nRLHF Training:\r\n- [ ] Step 1: Supervised fine-tuning (SFT)\r\n- [ ] Step 2: Train reward model\r\n- [ ] Step 3: PPO reinforcement learning\r\n- [ ] Step 4: Evaluate aligned model\r\n```\r\n\r\n**Step 1: Supervised fine-tuning**\r\n\r\nTrain base model on instruction-following data:\r\n\r\n```python\r\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\r\nfrom trl import SFTTrainer, SFTConfig\r\nfrom datasets import load_dataset\r\n\r\n# Load model\r\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\r\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\r\n\r\n# Load instruction dataset\r\ndataset = load_dataset(\"trl-lib\u002FCapybara\", split=\"train\")\r\n\r\n# Configure training\r\ntraining_args = SFTConfig(\r\n    output_dir=\"Qwen2.5-0.5B-SFT\",\r\n    per_device_train_batch_size=4,\r\n    num_train_epochs=1,\r\n    learning_rate=2e-5,\r\n    logging_steps=10,\r\n    save_strategy=\"epoch\"\r\n)\r\n\r\n# Train\r\ntrainer = SFTTrainer(\r\n    model=model,\r\n    args=training_args,\r\n    train_dataset=dataset,\r\n    tokenizer=tokenizer\r\n)\r\ntrainer.train()\r\ntrainer.save_model()\r\n```\r\n\r\n**Step 2: Train reward model**\r\n\r\nTrain model to predict human preferences:\r\n\r\n```python\r\nfrom transformers import AutoModelForSequenceClassification\r\nfrom trl import RewardTrainer, RewardConfig\r\n\r\n# Load SFT model as base\r\nmodel = AutoModelForSequenceClassification.from_pretrained(\r\n    \"Qwen2.5-0.5B-SFT\",\r\n    num_labels=1  # Single reward score\r\n)\r\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen2.5-0.5B-SFT\")\r\n\r\n# Load preference data (chosen\u002Frejected pairs)\r\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\r\n\r\n# Configure training\r\ntraining_args = RewardConfig(\r\n    output_dir=\"Qwen2.5-0.5B-Reward\",\r\n    per_device_train_batch_size=2,\r\n    num_train_epochs=1,\r\n    learning_rate=1e-5\r\n)\r\n\r\n# Train reward model\r\ntrainer = RewardTrainer(\r\n    model=model,\r\n    args=training_args,\r\n    processing_class=tokenizer,\r\n    train_dataset=dataset\r\n)\r\ntrainer.train()\r\ntrainer.save_model()\r\n```\r\n\r\n**Step 3: PPO reinforcement learning**\r\n\r\nOptimize policy using reward model:\r\n\r\n```bash\r\npython -m trl.scripts.ppo \\\r\n    --model_name_or_path Qwen2.5-0.5B-SFT \\\r\n    --reward_model_path Qwen2.5-0.5B-Reward \\\r\n    --dataset_name trl-internal-testing\u002Fdescriptiveness-sentiment-trl-style \\\r\n    --output_dir Qwen2.5-0.5B-PPO \\\r\n    --learning_rate 3e-6 \\\r\n    --per_device_train_batch_size 64 \\\r\n    --total_episodes 10000\r\n```\r\n\r\n**Step 4: Evaluate**\r\n\r\n```python\r\nfrom transformers import pipeline\r\n\r\n# Load aligned model\r\ngenerator = pipeline(\"text-generation\", model=\"Qwen2.5-0.5B-PPO\")\r\n\r\n# Test\r\nprompt = \"Explain quantum computing to a 10-year-old\"\r\noutput = generator(prompt, max_length=200)[0][\"generated_text\"]\r\nprint(output)\r\n```\r\n\r\n### Workflow 2: Simple preference alignment with DPO\r\n\r\nAlign model with preferences without reward model.\r\n\r\nCopy this checklist:\r\n\r\n```\r\nDPO Training:\r\n- [ ] Step 1: Prepare preference dataset\r\n- [ ] Step 2: Configure DPO\r\n- [ ] Step 3: Train with DPOTrainer\r\n- [ ] Step 4: Evaluate alignment\r\n```\r\n\r\n**Step 1: Prepare preference dataset**\r\n\r\nDataset format:\r\n```json\r\n{\r\n  \"prompt\": \"What is the capital of France?\",\r\n  \"chosen\": \"The capital of France is Paris.\",\r\n  \"rejected\": \"I don't know.\"\r\n}\r\n```\r\n\r\nLoad dataset:\r\n```python\r\nfrom datasets import load_dataset\r\n\r\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\r\n# Or load your own\r\n# dataset = load_dataset(\"json\", data_files=\"preferences.json\")\r\n```\r\n\r\n**Step 2: Configure DPO**\r\n\r\n```python\r\nfrom trl import DPOConfig\r\n\r\nconfig = DPOConfig(\r\n    output_dir=\"Qwen2.5-0.5B-DPO\",\r\n    per_device_train_batch_size=4,\r\n    num_train_epochs=1,\r\n    learning_rate=5e-7,\r\n    beta=0.1,  # KL penalty strength\r\n    max_prompt_length=512,\r\n    max_length=1024,\r\n    logging_steps=10\r\n)\r\n```\r\n\r\n**Step 3: Train with DPOTrainer**\r\n\r\n```python\r\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\r\nfrom trl import DPOTrainer\r\n\r\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\r\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\r\n\r\ntrainer = DPOTrainer(\r\n    model=model,\r\n    args=config,\r\n    train_dataset=dataset,\r\n    processing_class=tokenizer\r\n)\r\n\r\ntrainer.train()\r\ntrainer.save_model()\r\n```\r\n\r\n**CLI alternative**:\r\n```bash\r\ntrl dpo \\\r\n    --model_name_or_path Qwen\u002FQwen2.5-0.5B-Instruct \\\r\n    --dataset_name argilla\u002FCapybara-Preferences \\\r\n    --output_dir Qwen2.5-0.5B-DPO \\\r\n    --per_device_train_batch_size 4 \\\r\n    --learning_rate 5e-7 \\\r\n    --beta 0.1\r\n```\r\n\r\n### Workflow 3: Memory-efficient online RL with GRPO\r\n\r\nTrain with reinforcement learning using minimal memory.\r\n\r\nFor in-depth GRPO guidance — reward function design, critical training insights (loss behavior, mode collapse, tuning), and advanced multi-stage patterns — see **[references\u002Fgrpo-training.md](references\u002Fgrpo-training.md)**. A production-ready training script is in **[templates\u002Fbasic_grpo_training.py](templates\u002Fbasic_grpo_training.py)**.\r\n\r\nCopy this checklist:\r\n\r\n```\r\nGRPO Training:\r\n- [ ] Step 1: Define reward function\r\n- [ ] Step 2: Configure GRPO\r\n- [ ] Step 3: Train with GRPOTrainer\r\n```\r\n\r\n**Step 1: Define reward function**\r\n\r\n```python\r\ndef reward_function(completions, **kwargs):\r\n    \"\"\"\r\n    Compute rewards for completions.\r\n\r\n    Args:\r\n        completions: List of generated texts\r\n\r\n    Returns:\r\n        List of reward scores (floats)\r\n    \"\"\"\r\n    rewards = []\r\n    for completion in completions:\r\n        # Example: reward based on length and unique words\r\n        score = len(completion.split())  # Favor longer responses\r\n        score += len(set(completion.lower().split()))  # Reward unique words\r\n        rewards.append(score)\r\n    return rewards\r\n```\r\n\r\nOr use a reward model:\r\n```python\r\nfrom transformers import pipeline\r\n\r\nreward_model = pipeline(\"text-classification\", model=\"reward-model-path\")\r\n\r\ndef reward_from_model(completions, prompts, **kwargs):\r\n    # Combine prompt + completion\r\n    full_texts = [p + c for p, c in zip(prompts, completions)]\r\n    # Get reward scores\r\n    results = reward_model(full_texts)\r\n    return [r[\"score\"] for r in results]\r\n```\r\n\r\n**Step 2: Configure GRPO**\r\n\r\n```python\r\nfrom trl import GRPOConfig\r\n\r\nconfig = GRPOConfig(\r\n    output_dir=\"Qwen2-GRPO\",\r\n    per_device_train_batch_size=4,\r\n    num_train_epochs=1,\r\n    learning_rate=1e-5,\r\n    num_generations=4,  # Generate 4 completions per prompt\r\n    max_new_tokens=128\r\n)\r\n```\r\n\r\n**Step 3: Train with GRPOTrainer**\r\n\r\n```python\r\nfrom datasets import load_dataset\r\nfrom trl import GRPOTrainer\r\n\r\n# Load prompt-only dataset\r\ndataset = load_dataset(\"trl-lib\u002Ftldr\", split=\"train\")\r\n\r\ntrainer = GRPOTrainer(\r\n    model=\"Qwen\u002FQwen2-0.5B-Instruct\",\r\n    reward_funcs=reward_function,  # Your reward function\r\n    args=config,\r\n    train_dataset=dataset\r\n)\r\n\r\ntrainer.train()\r\n```\r\n\r\n**CLI**:\r\n```bash\r\ntrl grpo \\\r\n    --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\r\n    --dataset_name trl-lib\u002Ftldr \\\r\n    --output_dir Qwen2-GRPO \\\r\n    --num_generations 4\r\n```\r\n\r\n## When to use vs alternatives\r\n\r\n**Use TRL when:**\r\n- Need to align model with human preferences\r\n- Have preference data (chosen\u002Frejected pairs)\r\n- Want to use reinforcement learning (PPO, GRPO)\r\n- Need reward model training\r\n- Doing RLHF (full pipeline)\r\n\r\n**Method selection**:\r\n- **SFT**: Have prompt-completion pairs, want basic instruction following\r\n- **DPO**: Have preferences, want simple alignment (no reward model needed)\r\n- **PPO**: Have reward model, need maximum control over RL\r\n- **GRPO**: Memory-constrained, want online RL\r\n- **Reward Model**: Building RLHF pipeline, need to score generations\r\n\r\n**Use alternatives instead:**\r\n- **HuggingFace Trainer**: Basic fine-tuning without RL\r\n- **Axolotl**: YAML-based training configuration\r\n- **LitGPT**: Educational, minimal fine-tuning\r\n- **Unsloth**: Fast LoRA training\r\n\r\n## Common issues\r\n\r\n**Issue: OOM during DPO training**\r\n\r\nReduce batch size and sequence length:\r\n```python\r\nconfig = DPOConfig(\r\n    per_device_train_batch_size=1,  # Reduce from 4\r\n    max_length=512,  # Reduce from 1024\r\n    gradient_accumulation_steps=8  # Maintain effective batch\r\n)\r\n```\r\n\r\nOr use gradient checkpointing:\r\n```python\r\nmodel.gradient_checkpointing_enable()\r\n```\r\n\r\n**Issue: Poor alignment quality**\r\n\r\nTune beta parameter:\r\n```python\r\n# Higher beta = more conservative (stays closer to reference)\r\nconfig = DPOConfig(beta=0.5)  # Default 0.1\r\n\r\n# Lower beta = more aggressive alignment\r\nconfig = DPOConfig(beta=0.01)\r\n```\r\n\r\n**Issue: Reward model not learning**\r\n\r\nCheck loss type and learning rate:\r\n```python\r\nconfig = RewardConfig(\r\n    learning_rate=1e-5,  # Try different LR\r\n    num_train_epochs=3  # Train longer\r\n)\r\n```\r\n\r\nEnsure preference dataset has clear winners:\r\n```python\r\n# Verify dataset\r\nprint(dataset[0])\r\n# Should have clear chosen > rejected\r\n```\r\n\r\n**Issue: PPO training unstable**\r\n\r\nAdjust KL coefficient:\r\n```python\r\nconfig = PPOConfig(\r\n    kl_coef=0.1,  # Increase from 0.05\r\n    cliprange=0.1  # Reduce from 0.2\r\n)\r\n```\r\n\r\n## Advanced topics\r\n\r\n**SFT training guide**: See [references\u002Fsft-training.md](references\u002Fsft-training.md) for dataset formats, chat templates, packing strategies, and multi-GPU training.\r\n\r\n**DPO variants**: See [references\u002Fdpo-variants.md](references\u002Fdpo-variants.md) for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.\r\n\r\n**Reward modeling**: See [references\u002Freward-modeling.md](references\u002Freward-modeling.md) for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.\r\n\r\n**Online RL methods**: See [references\u002Fonline-rl.md](references\u002Fonline-rl.md) for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.\r\n\r\n**GRPO deep dive**: See [references\u002Fgrpo-training.md](references\u002Fgrpo-training.md) for expert-level GRPO patterns — reward function design philosophy, training insights (why loss increases, mode collapse detection), hyperparameter tuning, multi-stage training, and troubleshooting. Production-ready template in [templates\u002Fbasic_grpo_training.py](templates\u002Fbasic_grpo_training.py).\r\n\r\n## Hardware requirements\r\n\r\n- **GPU**: NVIDIA (CUDA required)\r\n- **VRAM**: Depends on model and method\r\n  - SFT 7B: 16GB (with LoRA)\r\n  - DPO 7B: 24GB (stores reference model)\r\n  - PPO 7B: 40GB (policy + reward model)\r\n  - GRPO 7B: 24GB (more memory efficient)\r\n- **Multi-GPU**: Supported via `accelerate`\r\n- **Mixed precision**: BF16 recommended (A100\u002FH100)\r\n\r\n**Memory optimization**:\r\n- Use LoRA\u002FQLoRA for all methods\r\n- Enable gradient checkpointing\r\n- Use smaller batch sizes with gradient accumulation\r\n\r\n## Resources\r\n\r\n- Docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002F\r\n- GitHub: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\r\n- Papers:\r\n  - \"Training language models to follow instructions with human feedback\" (InstructGPT, 2022)\r\n  - \"Direct Preference Optimization: Your Language Model is Secretly a Reward Model\" (DPO, 2023)\r\n  - \"Group Relative Policy Optimization\" (GRPO, 2024)\r\n- Examples: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002Fmain\u002Fexamples\u002Fscripts\r\n",{"name":5,"author":44,"license":15,"version":45,"metadata":46,"platforms":54,"description":7,"dependencies":55},"Orchestra Research","1.0.0",{"hermes":47},{"tags":48},[25,21,49,31,50,24,27,51,26,52,53],"Reinforcement Learning","SFT","GRPO","Preference Alignment","HuggingFace",[38,29,36],[56,57,58,59,60,61],"trl","transformers","datasets","peft","accelerate","torch","main","0d1434f89f4bc6b07e9d272accd8c1013768820f","archive\u002Fskill-md-reviewed\u002Fsha256\u002F97\u002F97554ed569d39c185eb0c3c7033a72975b0902377621b8e0cb8925b535ebd878",0,"https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills","https:\u002F\u002Fgithub.com\u002Fzapabob","The Open University of JAPAN　2019-now\r\n\r\nTokyo University of Science　2007-2010","2026-10-26 16:29:01",{"source":8,"skillName":5,"commands":71,"prompt":81,"zipAvailable":82},[72,75,78],{"tool":73,"command":74},"npx","npx skills add zapabob\u002FSkills --skill fine-tuning-with-trl",{"tool":76,"command":77},"bunx","bunx skills add zapabob\u002FSkills --skill fine-tuning-with-trl",{"tool":79,"command":80},"pnpm","pnpm dlx skills add zapabob\u002FSkills --skill fine-tuning-with-trl","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Farchive\u002Fskill-md-reviewed\u002Fsha256\u002F97\u002F97554ed569d39c185eb0c3c7033a72975b0902377621b8e0cb8925b535ebd878\u002FSKILL.md",true,[],[85,120,155,181,207,223],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":104,"publishedAt":39},"01M6GJTE1GESGGF9K028Y7753E","xurl","X\u002FTwitter via xurl CLI: post, search, DM, media, v2 API.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fxurl\u002Fvariants\u002Fhermes-agent\u002FSKILL.md",[91,96,99],{"categoryNo":92,"categoryName":93,"taxonomyType":94,"taxonomyTypeDesc":95},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",{"categoryNo":97,"categoryName":98,"taxonomyType":94,"taxonomyTypeDesc":95},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":100,"categoryName":101,"taxonomyType":102,"taxonomyTypeDesc":103},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[87,105,29,106,107,38,36,108,109,110,111,112,113,114,115,116,62,117,118,119],"openclaw","cli","github","official","search","https","social-media","api","Media","hermes","hermes agent","twitter","social","via","post",{"skillNo":121,"name":122,"nameZh":6,"description":123,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":124,"categories":125,"keywords":137,"publishedAt":39},"01M6GJT8S1PXQ8WPXT1HSNBWH2","writing-plans","Write implementation plans: bite-sized tasks, paths, code.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fwriting-plans\u002Fvariants\u002Fhermes-agent\u002FSKILL.md",[126,129,132,133,134],{"categoryNo":127,"categoryName":128,"taxonomyType":94,"taxonomyTypeDesc":95},"01JBSEEDCAT000000000000015","视觉设计",{"categoryNo":130,"categoryName":131,"taxonomyType":94,"taxonomyTypeDesc":95},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":97,"categoryName":98,"taxonomyType":94,"taxonomyTypeDesc":95},{"categoryNo":100,"categoryName":101,"taxonomyType":102,"taxonomyTypeDesc":103},{"categoryNo":135,"categoryName":136,"taxonomyType":102,"taxonomyTypeDesc":103},"01JBSEEDCAT000000000000036","内容创作者",[138,139,140,141,29,142,38,143,36,144,145,146,147,148,149,150,151,152,153,154],"writing","writing plans","plans","design","review","workflow","documentation","tasks","development","code","planning","code-review","subagent","write","superpowers","paths","test-driven-development",{"skillNo":156,"name":157,"nameZh":6,"description":158,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":159,"categories":160,"keywords":164,"publishedAt":39},"01M6GJST0H5X7WHBJ3BZK0DVKH","whisper","OpenAI's general-purpose speech recognition model. Supports 99 languages, transcription, translation to English, and language identification. Six model sizes from tiny (39M params) to large (1550M params). Use for speech-to-text, podcast transcription, or multilingual audio processing. Best for robust, multilingual ASR.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fwhisper\u002Fvariants\u002Fhermes-agent-optional\u002FSKILL.md",[161],{"categoryNo":162,"categoryName":163,"taxonomyType":94,"taxonomyTypeDesc":95},"01JBSEEDCAT000000000000016","图像与多媒体",[157,29,38,165,166,167,28,168,169,170,171,172,173,174,175,176,177,178,179,180],"openai","speech","multimodal","ASR","Audio","text","model","speech recognition","use","general","multilingual","translation","english","identification","processing","Audio Processing",{"skillNo":182,"name":183,"nameZh":6,"description":184,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":185,"categories":186,"keywords":190,"publishedAt":39},"01M6GJSMRQ8EETZTYMG72P70EA","weights-and-biases","W&B: log ML experiments, sweeps, model registry, dashboards.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fweights-and-biases\u002Fvariants\u002Fhermes-agent\u002FSKILL.md",[187],{"categoryNo":188,"categoryName":189,"taxonomyType":94,"taxonomyTypeDesc":95},"01JBSEEDCAT000000000000013","分析与可视化",[191,29,192,193,36,194,195,196,197,19,198,171,199,200,201,202,203,204,205,206],"weights and biases","tensorflow","experiments","pytorch","visualization","registry","real-time","dashboards","experiment","log","time","Experiment Tracking","Model Registry","Hyperparameter Tuning","WandB","Real-Time Visualization",{"skillNo":208,"name":209,"nameZh":6,"description":210,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":211,"categories":212,"keywords":214,"publishedAt":39},"01M6GJS19KDPRY112JTR326S06","watchers","Poll RSS, JSON APIs, and GitHub with watermark dedup.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fwatchers\u002Fvariants\u002Fhermes-agent-optional\u002FSKILL.md",[213],{"categoryNo":97,"categoryName":98,"taxonomyType":94,"taxonomyTypeDesc":95},[209,29,215,107,38,216,217,218,219,220,221,114,115,222,20],"terminal","automation","http","monitoring","cron","json","rss","devops",{"skillNo":224,"name":225,"nameZh":6,"description":226,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":227,"categories":228,"keywords":231,"publishedAt":39},"01M6GJRHJD2A6Q830SWF9W48BC","serving-llms-vllm","vLLM: high-throughput LLM serving, OpenAI API, quantization.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fvllm\u002Fvariants\u002Fhermes-agent\u002FSKILL.md",[229,230],{"categoryNo":92,"categoryName":93,"taxonomyType":94,"taxonomyTypeDesc":95},{"categoryNo":100,"categoryName":101,"taxonomyType":102,"taxonomyTypeDesc":103},[232,233,234,38,235,165,236,237,238,28,239,112,240,241,242,243,244,245,246,247],"llms","vLLM","serving llms vllm","research","openai api","tensor","Inference Serving","Production","llm serving","High Throughput","PagedAttention","Continuous Batching","Quantization","Tensor Parallelism","high-throughput","throughput"]