[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BG0WD6W3RMBHVBT2M0TMPP":3,"skill-same:01M7BG0WD6W3RMBHVBT2M0TMPP":100},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":51,"commitSha":52,"skillDirectory":53,"forksCount":54,"repositoryHtmlUrl":55,"repositoryHomepageUrl":6,"ownerHtmlUrl":56,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":57,"install":58,"resources":71},"01M7BG0WD6W3RMBHVBT2M0TMPP","trl-fine-tuning",null,"TRL: SFT, DPO, GRPO, RLOO reward modeling for LLM RLHF.","LIJALEN23\u002FBeryl","LIJALEN23","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F141412346?v=4","01M7AQK5G18E0M3NKP1S35F4KE","oneplusnine","01M7AQK5GNQE3MMPFJ9TCCJQCN",1,"MIT","https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Foptional-skills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: trl-fine-tuning\ndescription: \"TRL: SFT, DPO, GRPO, RLOO reward modeling for LLM RLHF.\"\nversion: 1.0.1\nauthor: Orchestra Research\nlicense: MIT\ndependencies: [trl, transformers, datasets, peft, accelerate, torch]\nplatforms: [linux, macos, windows]\nmetadata:\n  hermes:\n    tags: [Post-Training, TRL, Reinforcement Learning, Fine-Tuning, SFT, DPO, GRPO, RLOO, RLHF, Preference Alignment, HuggingFace]\n\n---\n\n# TRL - Transformer Reinforcement Learning\n\n## Quick start\n\nTRL provides post-training methods for aligning language models with human preferences.\n\n**Installation**:\n```bash\npip install trl transformers datasets peft accelerate\n```\n\n**Supervised Fine-Tuning** (instruction tuning):\n```python\nfrom trl import SFTTrainer\n\ntrainer = SFTTrainer(\n    model=\"Qwen\u002FQwen2.5-0.5B\",\n    train_dataset=dataset,  # Prompt-completion pairs\n)\ntrainer.train()\n```\n\n**DPO** (align with preferences):\n```python\nfrom trl import DPOTrainer, DPOConfig\n\nconfig = DPOConfig(output_dir=\"model-dpo\", beta=0.1)\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=preference_dataset,  # chosen\u002Frejected pairs\n    processing_class=tokenizer\n)\ntrainer.train()\n```\n\n## Common workflows\n\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → RLOO)\n\nComplete pipeline from base model to human-aligned model.\n\n> **Note (TRL 1.x):** PPO has been **removed** from TRL — `PPOTrainer`, `PPOConfig`, and\n> `python -m trl.scripts.ppo` no longer exist. Use an online-RL trainer TRL still ships:\n> **RLOO** (`RLOOTrainer` \u002F `trl rloo`) is the closest drop-in for a reward-model-driven\n> RLHF pipeline, and **GRPO** (`GRPOTrainer` \u002F `trl grpo`, see Workflow 3) is the\n> memory-efficient alternative. The step below uses RLOO.\n\nCopy this checklist:\n\n```\nRLHF Training:\n- [ ] Step 1: Supervised fine-tuning (SFT)\n- [ ] Step 2: Train reward model\n- [ ] Step 3: RLOO reinforcement learning\n- [ ] Step 4: Evaluate aligned model\n```\n\n**Step 1: Supervised fine-tuning**\n\nTrain base model on instruction-following data:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import SFTTrainer, SFTConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\n\n# Load instruction dataset\ndataset = load_dataset(\"trl-lib\u002FCapybara\", split=\"train\")\n\n# Configure training\ntraining_args = SFTConfig(\n    output_dir=\"Qwen2.5-0.5B-SFT\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=2e-5,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\n# Train\ntrainer = SFTTrainer(\n    model=model,\n    args=training_args,\n    train_dataset=dataset,\n    processing_class=tokenizer\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 2: Train reward model**\n\nTrain model to predict human preferences:\n\n```python\nfrom transformers import AutoModelForSequenceClassification\nfrom trl import RewardTrainer, RewardConfig\n\n# Load SFT model as base\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    \"Qwen2.5-0.5B-SFT\",\n    num_labels=1  # Single reward score\n)\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen2.5-0.5B-SFT\")\n\n# Load preference data (chosen\u002Frejected pairs)\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n\n# Configure training\ntraining_args = RewardConfig(\n    output_dir=\"Qwen2.5-0.5B-Reward\",\n    per_device_train_batch_size=2,\n    num_train_epochs=1,\n    learning_rate=1e-5\n)\n\n# Train reward model\ntrainer = RewardTrainer(\n    model=model,\n    args=training_args,\n    processing_class=tokenizer,\n    train_dataset=dataset\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 3: RLOO reinforcement learning**\n\nOptimize policy using the reward model. PPO was removed in TRL 1.x; use the RLOO CLI\n(`trl rloo`) with the trained reward model passed via `--reward_model_name_or_path`:\n\n```bash\ntrl rloo \\\n    --model_name_or_path Qwen2.5-0.5B-SFT \\\n    --reward_model_name_or_path Qwen2.5-0.5B-Reward \\\n    --dataset_name trl-internal-testing\u002Fdescriptiveness-sentiment-trl-style \\\n    --output_dir Qwen2.5-0.5B-RLOO \\\n    --learning_rate 3e-6 \\\n    --per_device_train_batch_size 64 \\\n    --num_generations 4\n```\n\nEquivalent Python (`RLOOTrainer` \u002F `RLOOConfig`):\n```python\nfrom trl import RLOOTrainer, RLOOConfig\nfrom transformers import AutoModelForSequenceClassification, AutoTokenizer\n\nreward_model = AutoModelForSequenceClassification.from_pretrained(\n    \"Qwen2.5-0.5B-Reward\", num_labels=1\n)\n\nconfig = RLOOConfig(\n    output_dir=\"Qwen2.5-0.5B-RLOO\",\n    per_device_train_batch_size=64,\n    learning_rate=3e-6,\n    num_generations=4,\n)\n\ntrainer = RLOOTrainer(\n    model=\"Qwen2.5-0.5B-SFT\",\n    reward_funcs=reward_model,   # a reward model (or a callable reward function)\n    args=config,\n    train_dataset=dataset,       # prompt-only dataset\n    processing_class=tokenizer,\n)\ntrainer.train()\n```\n\n**Step 4: Evaluate**\n\n```python\nfrom transformers import pipeline\n\n# Load aligned model\ngenerator = pipeline(\"text-generation\", model=\"Qwen2.5-0.5B-RLOO\")\n\n# Test\nprompt = \"Explain quantum computing to a 10-year-old\"\noutput = generator(prompt, max_length=200)[0][\"generated_text\"]\nprint(output)\n```\n\n### Workflow 2: Simple preference alignment with DPO\n\nAlign model with preferences without reward model.\n\nCopy this checklist:\n\n```\nDPO Training:\n- [ ] Step 1: Prepare preference dataset\n- [ ] Step 2: Configure DPO\n- [ ] Step 3: Train with DPOTrainer\n- [ ] Step 4: Evaluate alignment\n```\n\n**Step 1: Prepare preference dataset**\n\nDataset format:\n```json\n{\n  \"prompt\": \"What is the capital of France?\",\n  \"chosen\": \"The capital of France is Paris.\",\n  \"rejected\": \"I don't know.\"\n}\n```\n\nLoad dataset:\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n# Or load your own\n# dataset = load_dataset(\"json\", data_files=\"preferences.json\")\n```\n\n**Step 2: Configure DPO**\n\n```python\nfrom trl import DPOConfig\n\nconfig = DPOConfig(\n    output_dir=\"Qwen2.5-0.5B-DPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=5e-7,\n    beta=0.1,  # KL penalty strength\n    max_prompt_length=512,\n    max_length=1024,\n    logging_steps=10\n)\n```\n\n**Step 3: Train with DPOTrainer**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import DPOTrainer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\n\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=dataset,\n    processing_class=tokenizer\n)\n\ntrainer.train()\ntrainer.save_model()\n```\n\n**CLI alternative**:\n```bash\ntrl dpo \\\n    --model_name_or_path Qwen\u002FQwen2.5-0.5B-Instruct \\\n    --dataset_name argilla\u002FCapybara-Preferences \\\n    --output_dir Qwen2.5-0.5B-DPO \\\n    --per_device_train_batch_size 4 \\\n    --learning_rate 5e-7 \\\n    --beta 0.1\n```\n\n### Workflow 3: Memory-efficient online RL with GRPO\n\nTrain with reinforcement learning using minimal memory.\n\nFor in-depth GRPO guidance — reward function design, critical training insights (loss behavior, mode collapse, tuning), and advanced multi-stage patterns — see **[references\u002Fgrpo-training.md](references\u002Fgrpo-training.md)**. A production-ready training script is in **[templates\u002Fbasic_grpo_training.py](templates\u002Fbasic_grpo_training.py)**.\n\nCopy this checklist:\n\n```\nGRPO Training:\n- [ ] Step 1: Define reward function\n- [ ] Step 2: Configure GRPO\n- [ ] Step 3: Train with GRPOTrainer\n```\n\n**Step 1: Define reward function**\n\n```python\ndef reward_function(completions, **kwargs):\n    \"\"\"\n    Compute rewards for completions.\n\n    Args:\n        completions: List of generated texts\n\n    Returns:\n        List of reward scores (floats)\n    \"\"\"\n    rewards = []\n    for completion in completions:\n        # Example: reward based on length and unique words\n        score = len(completion.split())  # Favor longer responses\n        score += len(set(completion.lower().split()))  # Reward unique words\n        rewards.append(score)\n    return rewards\n```\n\nOr use a reward model:\n```python\nfrom transformers import pipeline\n\nreward_model = pipeline(\"text-classification\", model=\"reward-model-path\")\n\ndef reward_from_model(completions, prompts, **kwargs):\n    # Combine prompt + completion\n    full_texts = [p + c for p, c in zip(prompts, completions)]\n    # Get reward scores\n    results = reward_model(full_texts)\n    return [r[\"score\"] for r in results]\n```\n\n**Step 2: Configure GRPO**\n\n```python\nfrom trl import GRPOConfig\n\nconfig = GRPOConfig(\n    output_dir=\"Qwen2-GRPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=1e-5,\n    num_generations=4,  # Generate 4 completions per prompt\n    max_new_tokens=128\n)\n```\n\n**Step 3: Train with GRPOTrainer**\n\n```python\nfrom datasets import load_dataset\nfrom trl import GRPOTrainer\n\n# Load prompt-only dataset\ndataset = load_dataset(\"trl-lib\u002Ftldr\", split=\"train\")\n\ntrainer = GRPOTrainer(\n    model=\"Qwen\u002FQwen2-0.5B-Instruct\",\n    reward_funcs=reward_function,  # Your reward function\n    args=config,\n    train_dataset=dataset\n)\n\ntrainer.train()\n```\n\n**CLI**:\n```bash\ntrl grpo \\\n    --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n    --dataset_name trl-lib\u002Ftldr \\\n    --output_dir Qwen2-GRPO \\\n    --num_generations 4\n```\n\n## When to use vs alternatives\n\n**Use TRL when:**\n- Need to align model with human preferences\n- Have preference data (chosen\u002Frejected pairs)\n- Want to use reinforcement learning (RLOO, GRPO)\n- Need reward model training\n- Doing RLHF (full pipeline)\n\n**Method selection**:\n- **SFT**: Have prompt-completion pairs, want basic instruction following\n- **DPO**: Have preferences, want simple alignment (no reward model needed)\n- **RLOO**: Have a reward model, want online RL (the reward-model-driven RLHF path; PPO was removed in TRL 1.x)\n- **GRPO**: Memory-constrained, want online RL with reward functions\n- **Reward Model**: Building RLHF pipeline, need to score generations\n\n**Use alternatives instead:**\n- **HuggingFace Trainer**: Basic fine-tuning without RL\n- **Axolotl**: YAML-based training configuration\n- **LitGPT**: Educational, minimal fine-tuning\n- **Unsloth**: Fast LoRA training\n\n## Common issues\n\n**Issue: OOM during DPO training**\n\nReduce batch size and sequence length:\n```python\nconfig = DPOConfig(\n    per_device_train_batch_size=1,  # Reduce from 4\n    max_length=512,  # Reduce from 1024\n    gradient_accumulation_steps=8  # Maintain effective batch\n)\n```\n\nOr use gradient checkpointing:\n```python\nmodel.gradient_checkpointing_enable()\n```\n\n**Issue: Poor alignment quality**\n\nTune beta parameter:\n```python\n# Higher beta = more conservative (stays closer to reference)\nconfig = DPOConfig(beta=0.5)  # Default 0.1\n\n# Lower beta = more aggressive alignment\nconfig = DPOConfig(beta=0.01)\n```\n\n**Issue: Reward model not learning**\n\nCheck loss type and learning rate:\n```python\nconfig = RewardConfig(\n    learning_rate=1e-5,  # Try different LR\n    num_train_epochs=3  # Train longer\n)\n```\n\nEnsure preference dataset has clear winners:\n```python\n# Verify dataset\nprint(dataset[0])\n# Should have clear chosen > rejected\n```\n\n**Issue: Online RL (RLOO\u002FGRPO) training unstable**\n\nAdjust the KL\u002Fbeta regularization toward the reference policy:\n```python\nfrom trl import RLOOConfig\n\nconfig = RLOOConfig(\n    beta=0.05,          # KL coefficient toward the reference model (increase for stability)\n    num_generations=4,  # more samples per prompt = lower-variance advantage estimates\n)\n```\n\n## Advanced topics\n\n**SFT training guide**: See [references\u002Fsft-training.md](references\u002Fsft-training.md) for dataset formats, chat templates, packing strategies, and multi-GPU training.\n\n**DPO variants**: See [references\u002Fdpo-variants.md](references\u002Fdpo-variants.md) for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.\n\n**Reward modeling**: See [references\u002Freward-modeling.md](references\u002Freward-modeling.md) for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.\n\n**Online RL methods**: See [references\u002Fonline-rl.md](references\u002Fonline-rl.md) for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.\n\n**GRPO deep dive**: See [references\u002Fgrpo-training.md](references\u002Fgrpo-training.md) for expert-level GRPO patterns — reward function design philosophy, training insights (why loss increases, mode collapse detection), hyperparameter tuning, multi-stage training, and troubleshooting. Production-ready template in [templates\u002Fbasic_grpo_training.py](templates\u002Fbasic_grpo_training.py).\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA required)\n- **VRAM**: Depends on model and method\n  - SFT 7B: 16GB (with LoRA)\n  - DPO 7B: 24GB (stores reference model)\n  - RLOO 7B: 40GB (policy + reward model)\n  - GRPO 7B: 24GB (more memory efficient)\n- **Multi-GPU**: Supported via `accelerate`\n- **Mixed precision**: BF16 recommended (A100\u002FH100)\n\n**Memory optimization**:\n- Use LoRA\u002FQLoRA for all methods\n- Enable gradient checkpointing\n- Use smaller batch sizes with gradient accumulation\n\n## Resources\n\n- Docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\n- Papers:\n  - \"Training language models to follow instructions with human feedback\" (InstructGPT, 2022)\n  - \"Direct Preference Optimization: Your Language Model is Secretly a Reward Model\" (DPO, 2023)\n  - \"Group Relative Policy Optimization\" (GRPO, 2024)\n- Examples: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002Fmain\u002Fexamples\u002Fscripts\n\n\n\n","# TRL - Transformer Reinforcement Learning\n\n## Quick start\n\nTRL provides post-training methods for aligning language models with human preferences.\n\n**Installation**:\n```bash\npip install trl transformers datasets peft accelerate\n```\n\n**Supervised Fine-Tuning** (instruction tuning):\n```python\nfrom trl import SFTTrainer\n\ntrainer = SFTTrainer(\n    model=\"Qwen\u002FQwen2.5-0.5B\",\n    train_dataset=dataset,  # Prompt-completion pairs\n)\ntrainer.train()\n```\n\n**DPO** (align with preferences):\n```python\nfrom trl import DPOTrainer, DPOConfig\n\nconfig = DPOConfig(output_dir=\"model-dpo\", beta=0.1)\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=preference_dataset,  # chosen\u002Frejected pairs\n    processing_class=tokenizer\n)\ntrainer.train()\n```\n\n## Common workflows\n\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → RLOO)\n\nComplete pipeline from base model to human-aligned model.\n\n> **Note (TRL 1.x):** PPO has been **removed** from TRL — `PPOTrainer`, `PPOConfig`, and\n> `python -m trl.scripts.ppo` no longer exist. Use an online-RL trainer TRL still ships:\n> **RLOO** (`RLOOTrainer` \u002F `trl rloo`) is the closest drop-in for a reward-model-driven\n> RLHF pipeline, and **GRPO** (`GRPOTrainer` \u002F `trl grpo`, see Workflow 3) is the\n> memory-efficient alternative. The step below uses RLOO.\n\nCopy this checklist:\n\n```\nRLHF Training:\n- [ ] Step 1: Supervised fine-tuning (SFT)\n- [ ] Step 2: Train reward model\n- [ ] Step 3: RLOO reinforcement learning\n- [ ] Step 4: Evaluate aligned model\n```\n\n**Step 1: Supervised fine-tuning**\n\nTrain base model on instruction-following data:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import SFTTrainer, SFTConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\n\n# Load instruction dataset\ndataset = load_dataset(\"trl-lib\u002FCapybara\", split=\"train\")\n\n# Configure training\ntraining_args = SFTConfig(\n    output_dir=\"Qwen2.5-0.5B-SFT\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=2e-5,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\n# Train\ntrainer = SFTTrainer(\n    model=model,\n    args=training_args,\n    train_dataset=dataset,\n    processing_class=tokenizer\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 2: Train reward model**\n\nTrain model to predict human preferences:\n\n```python\nfrom transformers import AutoModelForSequenceClassification\nfrom trl import RewardTrainer, RewardConfig\n\n# Load SFT model as base\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    \"Qwen2.5-0.5B-SFT\",\n    num_labels=1  # Single reward score\n)\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen2.5-0.5B-SFT\")\n\n# Load preference data (chosen\u002Frejected pairs)\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n\n# Configure training\ntraining_args = RewardConfig(\n    output_dir=\"Qwen2.5-0.5B-Reward\",\n    per_device_train_batch_size=2,\n    num_train_epochs=1,\n    learning_rate=1e-5\n)\n\n# Train reward model\ntrainer = RewardTrainer(\n    model=model,\n    args=training_args,\n    processing_class=tokenizer,\n    train_dataset=dataset\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 3: RLOO reinforcement learning**\n\nOptimize policy using the reward model. PPO was removed in TRL 1.x; use the RLOO CLI\n(`trl rloo`) with the trained reward model passed via `--reward_model_name_or_path`:\n\n```bash\ntrl rloo \\\n    --model_name_or_path Qwen2.5-0.5B-SFT \\\n    --reward_model_name_or_path Qwen2.5-0.5B-Reward \\\n    --dataset_name trl-internal-testing\u002Fdescriptiveness-sentiment-trl-style \\\n    --output_dir Qwen2.5-0.5B-RLOO \\\n    --learning_rate 3e-6 \\\n    --per_device_train_batch_size 64 \\\n    --num_generations 4\n```\n\nEquivalent Python (`RLOOTrainer` \u002F `RLOOConfig`):\n```python\nfrom trl import RLOOTrainer, RLOOConfig\nfrom transformers import AutoModelForSequenceClassification, AutoTokenizer\n\nreward_model = AutoModelForSequenceClassification.from_pretrained(\n    \"Qwen2.5-0.5B-Reward\", num_labels=1\n)\n\nconfig = RLOOConfig(\n    output_dir=\"Qwen2.5-0.5B-RLOO\",\n    per_device_train_batch_size=64,\n    learning_rate=3e-6,\n    num_generations=4,\n)\n\ntrainer = RLOOTrainer(\n    model=\"Qwen2.5-0.5B-SFT\",\n    reward_funcs=reward_model,   # a reward model (or a callable reward function)\n    args=config,\n    train_dataset=dataset,       # prompt-only dataset\n    processing_class=tokenizer,\n)\ntrainer.train()\n```\n\n**Step 4: Evaluate**\n\n```python\nfrom transformers import pipeline\n\n# Load aligned model\ngenerator = pipeline(\"text-generation\", model=\"Qwen2.5-0.5B-RLOO\")\n\n# Test\nprompt = \"Explain quantum computing to a 10-year-old\"\noutput = generator(prompt, max_length=200)[0][\"generated_text\"]\nprint(output)\n```\n\n### Workflow 2: Simple preference alignment with DPO\n\nAlign model with preferences without reward model.\n\nCopy this checklist:\n\n```\nDPO Training:\n- [ ] Step 1: Prepare preference dataset\n- [ ] Step 2: Configure DPO\n- [ ] Step 3: Train with DPOTrainer\n- [ ] Step 4: Evaluate alignment\n```\n\n**Step 1: Prepare preference dataset**\n\nDataset format:\n```json\n{\n  \"prompt\": \"What is the capital of France?\",\n  \"chosen\": \"The capital of France is Paris.\",\n  \"rejected\": \"I don't know.\"\n}\n```\n\nLoad dataset:\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n# Or load your own\n# dataset = load_dataset(\"json\", data_files=\"preferences.json\")\n```\n\n**Step 2: Configure DPO**\n\n```python\nfrom trl import DPOConfig\n\nconfig = DPOConfig(\n    output_dir=\"Qwen2.5-0.5B-DPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=5e-7,\n    beta=0.1,  # KL penalty strength\n    max_prompt_length=512,\n    max_length=1024,\n    logging_steps=10\n)\n```\n\n**Step 3: Train with DPOTrainer**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import DPOTrainer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\n\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=dataset,\n    processing_class=tokenizer\n)\n\ntrainer.train()\ntrainer.save_model()\n```\n\n**CLI alternative**:\n```bash\ntrl dpo \\\n    --model_name_or_path Qwen\u002FQwen2.5-0.5B-Instruct \\\n    --dataset_name argilla\u002FCapybara-Preferences \\\n    --output_dir Qwen2.5-0.5B-DPO \\\n    --per_device_train_batch_size 4 \\\n    --learning_rate 5e-7 \\\n    --beta 0.1\n```\n\n### Workflow 3: Memory-efficient online RL with GRPO\n\nTrain with reinforcement learning using minimal memory.\n\nFor in-depth GRPO guidance — reward function design, critical training insights (loss behavior, mode collapse, tuning), and advanced multi-stage patterns — see **[references\u002Fgrpo-training.md](references\u002Fgrpo-training.md)**. A production-ready training script is in **[templates\u002Fbasic_grpo_training.py](templates\u002Fbasic_grpo_training.py)**.\n\nCopy this checklist:\n\n```\nGRPO Training:\n- [ ] Step 1: Define reward function\n- [ ] Step 2: Configure GRPO\n- [ ] Step 3: Train with GRPOTrainer\n```\n\n**Step 1: Define reward function**\n\n```python\ndef reward_function(completions, **kwargs):\n    \"\"\"\n    Compute rewards for completions.\n\n    Args:\n        completions: List of generated texts\n\n    Returns:\n        List of reward scores (floats)\n    \"\"\"\n    rewards = []\n    for completion in completions:\n        # Example: reward based on length and unique words\n        score = len(completion.split())  # Favor longer responses\n        score += len(set(completion.lower().split()))  # Reward unique words\n        rewards.append(score)\n    return rewards\n```\n\nOr use a reward model:\n```python\nfrom transformers import pipeline\n\nreward_model = pipeline(\"text-classification\", model=\"reward-model-path\")\n\ndef reward_from_model(completions, prompts, **kwargs):\n    # Combine prompt + completion\n    full_texts = [p + c for p, c in zip(prompts, completions)]\n    # Get reward scores\n    results = reward_model(full_texts)\n    return [r[\"score\"] for r in results]\n```\n\n**Step 2: Configure GRPO**\n\n```python\nfrom trl import GRPOConfig\n\nconfig = GRPOConfig(\n    output_dir=\"Qwen2-GRPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=1e-5,\n    num_generations=4,  # Generate 4 completions per prompt\n    max_new_tokens=128\n)\n```\n\n**Step 3: Train with GRPOTrainer**\n\n```python\nfrom datasets import load_dataset\nfrom trl import GRPOTrainer\n\n# Load prompt-only dataset\ndataset = load_dataset(\"trl-lib\u002Ftldr\", split=\"train\")\n\ntrainer = GRPOTrainer(\n    model=\"Qwen\u002FQwen2-0.5B-Instruct\",\n    reward_funcs=reward_function,  # Your reward function\n    args=config,\n    train_dataset=dataset\n)\n\ntrainer.train()\n```\n\n**CLI**:\n```bash\ntrl grpo \\\n    --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n    --dataset_name trl-lib\u002Ftldr \\\n    --output_dir Qwen2-GRPO \\\n    --num_generations 4\n```\n\n## When to use vs alternatives\n\n**Use TRL when:**\n- Need to align model with human preferences\n- Have preference data (chosen\u002Frejected pairs)\n- Want to use reinforcement learning (RLOO, GRPO)\n- Need reward model training\n- Doing RLHF (full pipeline)\n\n**Method selection**:\n- **SFT**: Have prompt-completion pairs, want basic instruction following\n- **DPO**: Have preferences, want simple alignment (no reward model needed)\n- **RLOO**: Have a reward model, want online RL (the reward-model-driven RLHF path; PPO was removed in TRL 1.x)\n- **GRPO**: Memory-constrained, want online RL with reward functions\n- **Reward Model**: Building RLHF pipeline, need to score generations\n\n**Use alternatives instead:**\n- **HuggingFace Trainer**: Basic fine-tuning without RL\n- **Axolotl**: YAML-based training configuration\n- **LitGPT**: Educational, minimal fine-tuning\n- **Unsloth**: Fast LoRA training\n\n## Common issues\n\n**Issue: OOM during DPO training**\n\nReduce batch size and sequence length:\n```python\nconfig = DPOConfig(\n    per_device_train_batch_size=1,  # Reduce from 4\n    max_length=512,  # Reduce from 1024\n    gradient_accumulation_steps=8  # Maintain effective batch\n)\n```\n\nOr use gradient checkpointing:\n```python\nmodel.gradient_checkpointing_enable()\n```\n\n**Issue: Poor alignment quality**\n\nTune beta parameter:\n```python\n# Higher beta = more conservative (stays closer to reference)\nconfig = DPOConfig(beta=0.5)  # Default 0.1\n\n# Lower beta = more aggressive alignment\nconfig = DPOConfig(beta=0.01)\n```\n\n**Issue: Reward model not learning**\n\nCheck loss type and learning rate:\n```python\nconfig = RewardConfig(\n    learning_rate=1e-5,  # Try different LR\n    num_train_epochs=3  # Train longer\n)\n```\n\nEnsure preference dataset has clear winners:\n```python\n# Verify dataset\nprint(dataset[0])\n# Should have clear chosen > rejected\n```\n\n**Issue: Online RL (RLOO\u002FGRPO) training unstable**\n\nAdjust the KL\u002Fbeta regularization toward the reference policy:\n```python\nfrom trl import RLOOConfig\n\nconfig = RLOOConfig(\n    beta=0.05,          # KL coefficient toward the reference model (increase for stability)\n    num_generations=4,  # more samples per prompt = lower-variance advantage estimates\n)\n```\n\n## Advanced topics\n\n**SFT training guide**: See [references\u002Fsft-training.md](references\u002Fsft-training.md) for dataset formats, chat templates, packing strategies, and multi-GPU training.\n\n**DPO variants**: See [references\u002Fdpo-variants.md](references\u002Fdpo-variants.md) for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.\n\n**Reward modeling**: See [references\u002Freward-modeling.md](references\u002Freward-modeling.md) for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.\n\n**Online RL methods**: See [references\u002Fonline-rl.md](references\u002Fonline-rl.md) for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.\n\n**GRPO deep dive**: See [references\u002Fgrpo-training.md](references\u002Fgrpo-training.md) for expert-level GRPO patterns — reward function design philosophy, training insights (why loss increases, mode collapse detection), hyperparameter tuning, multi-stage training, and troubleshooting. Production-ready template in [templates\u002Fbasic_grpo_training.py](templates\u002Fbasic_grpo_training.py).\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA required)\n- **VRAM**: Depends on model and method\n  - SFT 7B: 16GB (with LoRA)\n  - DPO 7B: 24GB (stores reference model)\n  - RLOO 7B: 40GB (policy + reward model)\n  - GRPO 7B: 24GB (more memory efficient)\n- **Multi-GPU**: Supported via `accelerate`\n- **Mixed precision**: BF16 recommended (A100\u002FH100)\n\n**Memory optimization**:\n- Use LoRA\u002FQLoRA for all methods\n- Enable gradient checkpointing\n- Use smaller batch sizes with gradient accumulation\n\n## Resources\n\n- Docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\n- Papers:\n  - \"Training language models to follow instructions with human feedback\" (InstructGPT, 2022)\n  - \"Direct Preference Optimization: Your Language Model is Secretly a Reward Model\" (DPO, 2023)\n  - \"Group Relative Policy Optimization\" (GRPO, 2024)\n- Examples: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002Fmain\u002Fexamples\u002Fscripts\n\n\n\n",{"name":5,"author":24,"license":15,"version":25,"metadata":26,"platforms":40,"description":7,"dependencies":44},"Orchestra Research","1.0.1",{"hermes":27},{"tags":28},[29,30,31,32,33,34,35,36,37,38,39],"Post-Training","TRL","Reinforcement Learning","Fine-Tuning","SFT","DPO","GRPO","RLOO","RLHF","Preference Alignment","HuggingFace",[41,42,43],"linux","macos","windows",[45,46,47,48,49,50],"trl","transformers","datasets","peft","accelerate","torch","master","4a5caebee4b5f9068d3a68c9a7ecd8df12bd4962","local-agent\u002Foptional-skills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning",0,"https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl","https:\u002F\u002Fgithub.com\u002FLIJALEN23","2026-09-01 14:02:08",{"source":8,"skillName":5,"commands":59,"prompt":69,"zipAvailable":70},[60,63,66],{"tool":61,"command":62},"npx","npx skills add LIJALEN23\u002FBeryl --skill trl-fine-tuning",{"tool":64,"command":65},"bunx","bunx skills add LIJALEN23\u002FBeryl --skill trl-fine-tuning",{"tool":67,"command":68},"pnpm","pnpm dlx skills add LIJALEN23\u002FBeryl --skill trl-fine-tuning","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Foptional-skills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002FSKILL.md",true,[72,78,82,86,90,94],{"path":73,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":76,"sourceHtmlUrl":77},"references\u002Fdpo-variants.md","REFERENCE","参考文档",4297,"https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Foptional-skills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002Freferences\u002Fdpo-variants.md",{"path":79,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Fgrpo-training.md",15901,"https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Foptional-skills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002Freferences\u002Fgrpo-training.md",{"path":83,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":84,"sourceHtmlUrl":85},"references\u002Fonline-rl.md",1971,"https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Foptional-skills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002Freferences\u002Fonline-rl.md",{"path":87,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Freward-modeling.md",2597,"https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Foptional-skills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002Freferences\u002Freward-modeling.md",{"path":91,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":92,"sourceHtmlUrl":93},"references\u002Fsft-training.md",3237,"https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Foptional-skills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002Freferences\u002Fsft-training.md",{"path":95,"resourceType":96,"resourceTypeDesc":97,"sizeBytes":98,"sourceHtmlUrl":99},"templates\u002Fbasic_grpo_training.py","SCRIPT","脚本",6113,"https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Foptional-skills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002Ftemplates\u002Fbasic_grpo_training.py",[101,108,115,122,129,136],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":19},"01M7BGCRH58RARQTF3R8CZ84DB","test-driven-development","TDD: enforce RED-GREEN-REFACTOR, tests before code.","https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Fskills\u002Fsoftware-development\u002Ftest-driven-development\u002FSKILL.md",[],[],{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":19},"01M7BGCK0GGNV3K6NWMM0MDN3W","systematic-debugging","4-phase root cause debugging: understand bugs before fixing.","https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Fskills\u002Fsoftware-development\u002Fsystematic-debugging\u002FSKILL.md",[],[],{"skillNo":116,"name":117,"nameZh":6,"description":118,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":119,"categories":120,"keywords":121,"publishedAt":19},"01M7BGCF0TAZPKSM4WMYA27XGD","spike","Throwaway experiments to validate an idea before build.","https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Fskills\u002Fsoftware-development\u002Fspike\u002FSKILL.md",[],[],{"skillNo":123,"name":124,"nameZh":6,"description":125,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":126,"categories":127,"keywords":128,"publishedAt":19},"01M7BGCBH01YYYBQWR7ZFRMX27","simplify-code","Parallel 4-agent cleanup of recent code changes.","https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Fskills\u002Fsoftware-development\u002Fsimplify-code\u002FSKILL.md",[],[],{"skillNo":130,"name":131,"nameZh":6,"description":132,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":133,"categories":134,"keywords":135,"publishedAt":19},"01M7BGC8MGHS5SRJ3NZYYNAKJY","requesting-code-review","Pre-commit review: security scan, quality gates, auto-fix.","https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Fskills\u002Fsoftware-development\u002Frequesting-code-review\u002FSKILL.md",[],[],{"skillNo":137,"name":138,"nameZh":6,"description":139,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":140,"categories":141,"keywords":142,"publishedAt":19},"01M7BGC4HB5ZJP5N19MZTZXKZ1","python-debugpy","Debug Python: pdb REPL + debugpy remote (DAP).","https:\u002F\u002Fgithub.com\u002FLIJALEN23\u002FBeryl\u002Fblob\u002Fmaster\u002Flocal-agent\u002Fskills\u002Fsoftware-development\u002Fpython-debugpy\u002FSKILL.md",[],[]]