[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M71TBJCDFVRYAPXV4PW25MBG":3,"skill-same:01M71TBJCDFVRYAPXV4PW25MBG":107},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":38,"linkOnly":39,"markdownContent":40,"markdownBody":41,"frontMatter":42,"refName":59,"commitSha":60,"skillDirectory":61,"forksCount":13,"repositoryHtmlUrl":62,"repositoryHomepageUrl":6,"ownerHtmlUrl":63,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":64,"install":65,"resources":78},"01M71TBJCDFVRYAPXV4PW25MBG","fine-tuning-with-trl",null,"TRL: SFT, DPO, PPO, GRPO, reward modeling for LLM RLHF.","eliottbusiness\u002FDeptFlow-Agent","eliottbusiness","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F263418316?v=4","01M71T5T608RHFA07W6QVC6TYX","01M71T5T6R98RMYZ53PYH851T1",0,"MIT","https:\u002F\u002Fgithub.com\u002Feliottbusiness\u002FDeptFlow-Agent\u002Fblob\u002Fmain\u002Fprofile\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37],"with","TRL","tuning","fine tuning with trl","fine tuning","for","training","orchestra","Reward Modeling","Accelerate","modeling","alignment","learning","Transformers","PPO","RLHF","Post-Training","GRPO","DPO","Fine-Tuning","2026-11-10 13:02:02",false,"---\nname: fine-tuning-with-trl\ndescription: \"TRL: SFT, DPO, PPO, GRPO, reward modeling for LLM RLHF.\"\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ndependencies: [trl, transformers, datasets, peft, accelerate, torch]\nmetadata:\n  hermes:\n    tags: [Post-Training, TRL, Reinforcement Learning, Fine-Tuning, SFT, DPO, PPO, GRPO, RLHF, Preference Alignment, HuggingFace]\n\n---\n\n# TRL - Transformer Reinforcement Learning\n\n## Quick start\n\nTRL provides post-training methods for aligning language models with human preferences.\n\n**Installation**:\n```bash\npip install trl transformers datasets peft accelerate\n```\n\n**Supervised Fine-Tuning** (instruction tuning):\n```python\nfrom trl import SFTTrainer\n\ntrainer = SFTTrainer(\n    model=\"Qwen\u002FQwen2.5-0.5B\",\n    train_dataset=dataset,  # Prompt-completion pairs\n)\ntrainer.train()\n```\n\n**DPO** (align with preferences):\n```python\nfrom trl import DPOTrainer, DPOConfig\n\nconfig = DPOConfig(output_dir=\"model-dpo\", beta=0.1)\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=preference_dataset,  # chosen\u002Frejected pairs\n    processing_class=tokenizer\n)\ntrainer.train()\n```\n\n## Common workflows\n\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)\n\nComplete pipeline from base model to human-aligned model.\n\nCopy this checklist:\n\n```\nRLHF Training:\n- [ ] Step 1: Supervised fine-tuning (SFT)\n- [ ] Step 2: Train reward model\n- [ ] Step 3: PPO reinforcement learning\n- [ ] Step 4: Evaluate aligned model\n```\n\n**Step 1: Supervised fine-tuning**\n\nTrain base model on instruction-following data:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import SFTTrainer, SFTConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\n\n# Load instruction dataset\ndataset = load_dataset(\"trl-lib\u002FCapybara\", split=\"train\")\n\n# Configure training\ntraining_args = SFTConfig(\n    output_dir=\"Qwen2.5-0.5B-SFT\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=2e-5,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\n# Train\ntrainer = SFTTrainer(\n    model=model,\n    args=training_args,\n    train_dataset=dataset,\n    tokenizer=tokenizer\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 2: Train reward model**\n\nTrain model to predict human preferences:\n\n```python\nfrom transformers import AutoModelForSequenceClassification\nfrom trl import RewardTrainer, RewardConfig\n\n# Load SFT model as base\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    \"Qwen2.5-0.5B-SFT\",\n    num_labels=1  # Single reward score\n)\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen2.5-0.5B-SFT\")\n\n# Load preference data (chosen\u002Frejected pairs)\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n\n# Configure training\ntraining_args = RewardConfig(\n    output_dir=\"Qwen2.5-0.5B-Reward\",\n    per_device_train_batch_size=2,\n    num_train_epochs=1,\n    learning_rate=1e-5\n)\n\n# Train reward model\ntrainer = RewardTrainer(\n    model=model,\n    args=training_args,\n    processing_class=tokenizer,\n    train_dataset=dataset\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 3: PPO reinforcement learning**\n\nOptimize policy using reward model:\n\n```bash\npython -m trl.scripts.ppo \\\n    --model_name_or_path Qwen2.5-0.5B-SFT \\\n    --reward_model_path Qwen2.5-0.5B-Reward \\\n    --dataset_name trl-internal-testing\u002Fdescriptiveness-sentiment-trl-style \\\n    --output_dir Qwen2.5-0.5B-PPO \\\n    --learning_rate 3e-6 \\\n    --per_device_train_batch_size 64 \\\n    --total_episodes 10000\n```\n\n**Step 4: Evaluate**\n\n```python\nfrom transformers import pipeline\n\n# Load aligned model\ngenerator = pipeline(\"text-generation\", model=\"Qwen2.5-0.5B-PPO\")\n\n# Test\nprompt = \"Explain quantum computing to a 10-year-old\"\noutput = generator(prompt, max_length=200)[0][\"generated_text\"]\nprint(output)\n```\n\n### Workflow 2: Simple preference alignment with DPO\n\nAlign model with preferences without reward model.\n\nCopy this checklist:\n\n```\nDPO Training:\n- [ ] Step 1: Prepare preference dataset\n- [ ] Step 2: Configure DPO\n- [ ] Step 3: Train with DPOTrainer\n- [ ] Step 4: Evaluate alignment\n```\n\n**Step 1: Prepare preference dataset**\n\nDataset format:\n```json\n{\n  \"prompt\": \"What is the capital of France?\",\n  \"chosen\": \"The capital of France is Paris.\",\n  \"rejected\": \"I don't know.\"\n}\n```\n\nLoad dataset:\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n# Or load your own\n# dataset = load_dataset(\"json\", data_files=\"preferences.json\")\n```\n\n**Step 2: Configure DPO**\n\n```python\nfrom trl import DPOConfig\n\nconfig = DPOConfig(\n    output_dir=\"Qwen2.5-0.5B-DPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=5e-7,\n    beta=0.1,  # KL penalty strength\n    max_prompt_length=512,\n    max_length=1024,\n    logging_steps=10\n)\n```\n\n**Step 3: Train with DPOTrainer**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import DPOTrainer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\n\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=dataset,\n    processing_class=tokenizer\n)\n\ntrainer.train()\ntrainer.save_model()\n```\n\n**CLI alternative**:\n```bash\ntrl dpo \\\n    --model_name_or_path Qwen\u002FQwen2.5-0.5B-Instruct \\\n    --dataset_name argilla\u002FCapybara-Preferences \\\n    --output_dir Qwen2.5-0.5B-DPO \\\n    --per_device_train_batch_size 4 \\\n    --learning_rate 5e-7 \\\n    --beta 0.1\n```\n\n### Workflow 3: Memory-efficient online RL with GRPO\n\nTrain with reinforcement learning using minimal memory.\n\nFor in-depth GRPO guidance — reward function design, critical training insights (loss behavior, mode collapse, tuning), and advanced multi-stage patterns — see **[references\u002Fgrpo-training.md](references\u002Fgrpo-training.md)**. A production-ready training script is in **[templates\u002Fbasic_grpo_training.py](templates\u002Fbasic_grpo_training.py)**.\n\nCopy this checklist:\n\n```\nGRPO Training:\n- [ ] Step 1: Define reward function\n- [ ] Step 2: Configure GRPO\n- [ ] Step 3: Train with GRPOTrainer\n```\n\n**Step 1: Define reward function**\n\n```python\ndef reward_function(completions, **kwargs):\n    \"\"\"\n    Compute rewards for completions.\n\n    Args:\n        completions: List of generated texts\n\n    Returns:\n        List of reward scores (floats)\n    \"\"\"\n    rewards = []\n    for completion in completions:\n        # Example: reward based on length and unique words\n        score = len(completion.split())  # Favor longer responses\n        score += len(set(completion.lower().split()))  # Reward unique words\n        rewards.append(score)\n    return rewards\n```\n\nOr use a reward model:\n```python\nfrom transformers import pipeline\n\nreward_model = pipeline(\"text-classification\", model=\"reward-model-path\")\n\ndef reward_from_model(completions, prompts, **kwargs):\n    # Combine prompt + completion\n    full_texts = [p + c for p, c in zip(prompts, completions)]\n    # Get reward scores\n    results = reward_model(full_texts)\n    return [r[\"score\"] for r in results]\n```\n\n**Step 2: Configure GRPO**\n\n```python\nfrom trl import GRPOConfig\n\nconfig = GRPOConfig(\n    output_dir=\"Qwen2-GRPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=1e-5,\n    num_generations=4,  # Generate 4 completions per prompt\n    max_new_tokens=128\n)\n```\n\n**Step 3: Train with GRPOTrainer**\n\n```python\nfrom datasets import load_dataset\nfrom trl import GRPOTrainer\n\n# Load prompt-only dataset\ndataset = load_dataset(\"trl-lib\u002Ftldr\", split=\"train\")\n\ntrainer = GRPOTrainer(\n    model=\"Qwen\u002FQwen2-0.5B-Instruct\",\n    reward_funcs=reward_function,  # Your reward function\n    args=config,\n    train_dataset=dataset\n)\n\ntrainer.train()\n```\n\n**CLI**:\n```bash\ntrl grpo \\\n    --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n    --dataset_name trl-lib\u002Ftldr \\\n    --output_dir Qwen2-GRPO \\\n    --num_generations 4\n```\n\n## When to use vs alternatives\n\n**Use TRL when:**\n- Need to align model with human preferences\n- Have preference data (chosen\u002Frejected pairs)\n- Want to use reinforcement learning (PPO, GRPO)\n- Need reward model training\n- Doing RLHF (full pipeline)\n\n**Method selection**:\n- **SFT**: Have prompt-completion pairs, want basic instruction following\n- **DPO**: Have preferences, want simple alignment (no reward model needed)\n- **PPO**: Have reward model, need maximum control over RL\n- **GRPO**: Memory-constrained, want online RL\n- **Reward Model**: Building RLHF pipeline, need to score generations\n\n**Use alternatives instead:**\n- **HuggingFace Trainer**: Basic fine-tuning without RL\n- **Axolotl**: YAML-based training configuration\n- **LitGPT**: Educational, minimal fine-tuning\n- **Unsloth**: Fast LoRA training\n\n## Common issues\n\n**Issue: OOM during DPO training**\n\nReduce batch size and sequence length:\n```python\nconfig = DPOConfig(\n    per_device_train_batch_size=1,  # Reduce from 4\n    max_length=512,  # Reduce from 1024\n    gradient_accumulation_steps=8  # Maintain effective batch\n)\n```\n\nOr use gradient checkpointing:\n```python\nmodel.gradient_checkpointing_enable()\n```\n\n**Issue: Poor alignment quality**\n\nTune beta parameter:\n```python\n# Higher beta = more conservative (stays closer to reference)\nconfig = DPOConfig(beta=0.5)  # Default 0.1\n\n# Lower beta = more aggressive alignment\nconfig = DPOConfig(beta=0.01)\n```\n\n**Issue: Reward model not learning**\n\nCheck loss type and learning rate:\n```python\nconfig = RewardConfig(\n    learning_rate=1e-5,  # Try different LR\n    num_train_epochs=3  # Train longer\n)\n```\n\nEnsure preference dataset has clear winners:\n```python\n# Verify dataset\nprint(dataset[0])\n# Should have clear chosen > rejected\n```\n\n**Issue: PPO training unstable**\n\nAdjust KL coefficient:\n```python\nconfig = PPOConfig(\n    kl_coef=0.1,  # Increase from 0.05\n    cliprange=0.1  # Reduce from 0.2\n)\n```\n\n## Advanced topics\n\n**SFT training guide**: See [references\u002Fsft-training.md](references\u002Fsft-training.md) for dataset formats, chat templates, packing strategies, and multi-GPU training.\n\n**DPO variants**: See [references\u002Fdpo-variants.md](references\u002Fdpo-variants.md) for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.\n\n**Reward modeling**: See [references\u002Freward-modeling.md](references\u002Freward-modeling.md) for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.\n\n**Online RL methods**: See [references\u002Fonline-rl.md](references\u002Fonline-rl.md) for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.\n\n**GRPO deep dive**: See [references\u002Fgrpo-training.md](references\u002Fgrpo-training.md) for expert-level GRPO patterns — reward function design philosophy, training insights (why loss increases, mode collapse detection), hyperparameter tuning, multi-stage training, and troubleshooting. Production-ready template in [templates\u002Fbasic_grpo_training.py](templates\u002Fbasic_grpo_training.py).\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA required)\n- **VRAM**: Depends on model and method\n  - SFT 7B: 16GB (with LoRA)\n  - DPO 7B: 24GB (stores reference model)\n  - PPO 7B: 40GB (policy + reward model)\n  - GRPO 7B: 24GB (more memory efficient)\n- **Multi-GPU**: Supported via `accelerate`\n- **Mixed precision**: BF16 recommended (A100\u002FH100)\n\n**Memory optimization**:\n- Use LoRA\u002FQLoRA for all methods\n- Enable gradient checkpointing\n- Use smaller batch sizes with gradient accumulation\n\n## Resources\n\n- Docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\n- Papers:\n  - \"Training language models to follow instructions with human feedback\" (InstructGPT, 2022)\n  - \"Direct Preference Optimization: Your Language Model is Secretly a Reward Model\" (DPO, 2023)\n  - \"Group Relative Policy Optimization\" (GRPO, 2024)\n- Examples: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002Fmain\u002Fexamples\u002Fscripts\n\n\n\n","# TRL - Transformer Reinforcement Learning\n\n## Quick start\n\nTRL provides post-training methods for aligning language models with human preferences.\n\n**Installation**:\n```bash\npip install trl transformers datasets peft accelerate\n```\n\n**Supervised Fine-Tuning** (instruction tuning):\n```python\nfrom trl import SFTTrainer\n\ntrainer = SFTTrainer(\n    model=\"Qwen\u002FQwen2.5-0.5B\",\n    train_dataset=dataset,  # Prompt-completion pairs\n)\ntrainer.train()\n```\n\n**DPO** (align with preferences):\n```python\nfrom trl import DPOTrainer, DPOConfig\n\nconfig = DPOConfig(output_dir=\"model-dpo\", beta=0.1)\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=preference_dataset,  # chosen\u002Frejected pairs\n    processing_class=tokenizer\n)\ntrainer.train()\n```\n\n## Common workflows\n\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)\n\nComplete pipeline from base model to human-aligned model.\n\nCopy this checklist:\n\n```\nRLHF Training:\n- [ ] Step 1: Supervised fine-tuning (SFT)\n- [ ] Step 2: Train reward model\n- [ ] Step 3: PPO reinforcement learning\n- [ ] Step 4: Evaluate aligned model\n```\n\n**Step 1: Supervised fine-tuning**\n\nTrain base model on instruction-following data:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import SFTTrainer, SFTConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\n\n# Load instruction dataset\ndataset = load_dataset(\"trl-lib\u002FCapybara\", split=\"train\")\n\n# Configure training\ntraining_args = SFTConfig(\n    output_dir=\"Qwen2.5-0.5B-SFT\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=2e-5,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\n# Train\ntrainer = SFTTrainer(\n    model=model,\n    args=training_args,\n    train_dataset=dataset,\n    tokenizer=tokenizer\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 2: Train reward model**\n\nTrain model to predict human preferences:\n\n```python\nfrom transformers import AutoModelForSequenceClassification\nfrom trl import RewardTrainer, RewardConfig\n\n# Load SFT model as base\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    \"Qwen2.5-0.5B-SFT\",\n    num_labels=1  # Single reward score\n)\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen2.5-0.5B-SFT\")\n\n# Load preference data (chosen\u002Frejected pairs)\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n\n# Configure training\ntraining_args = RewardConfig(\n    output_dir=\"Qwen2.5-0.5B-Reward\",\n    per_device_train_batch_size=2,\n    num_train_epochs=1,\n    learning_rate=1e-5\n)\n\n# Train reward model\ntrainer = RewardTrainer(\n    model=model,\n    args=training_args,\n    processing_class=tokenizer,\n    train_dataset=dataset\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 3: PPO reinforcement learning**\n\nOptimize policy using reward model:\n\n```bash\npython -m trl.scripts.ppo \\\n    --model_name_or_path Qwen2.5-0.5B-SFT \\\n    --reward_model_path Qwen2.5-0.5B-Reward \\\n    --dataset_name trl-internal-testing\u002Fdescriptiveness-sentiment-trl-style \\\n    --output_dir Qwen2.5-0.5B-PPO \\\n    --learning_rate 3e-6 \\\n    --per_device_train_batch_size 64 \\\n    --total_episodes 10000\n```\n\n**Step 4: Evaluate**\n\n```python\nfrom transformers import pipeline\n\n# Load aligned model\ngenerator = pipeline(\"text-generation\", model=\"Qwen2.5-0.5B-PPO\")\n\n# Test\nprompt = \"Explain quantum computing to a 10-year-old\"\noutput = generator(prompt, max_length=200)[0][\"generated_text\"]\nprint(output)\n```\n\n### Workflow 2: Simple preference alignment with DPO\n\nAlign model with preferences without reward model.\n\nCopy this checklist:\n\n```\nDPO Training:\n- [ ] Step 1: Prepare preference dataset\n- [ ] Step 2: Configure DPO\n- [ ] Step 3: Train with DPOTrainer\n- [ ] Step 4: Evaluate alignment\n```\n\n**Step 1: Prepare preference dataset**\n\nDataset format:\n```json\n{\n  \"prompt\": \"What is the capital of France?\",\n  \"chosen\": \"The capital of France is Paris.\",\n  \"rejected\": \"I don't know.\"\n}\n```\n\nLoad dataset:\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n# Or load your own\n# dataset = load_dataset(\"json\", data_files=\"preferences.json\")\n```\n\n**Step 2: Configure DPO**\n\n```python\nfrom trl import DPOConfig\n\nconfig = DPOConfig(\n    output_dir=\"Qwen2.5-0.5B-DPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=5e-7,\n    beta=0.1,  # KL penalty strength\n    max_prompt_length=512,\n    max_length=1024,\n    logging_steps=10\n)\n```\n\n**Step 3: Train with DPOTrainer**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import DPOTrainer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\n\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=dataset,\n    processing_class=tokenizer\n)\n\ntrainer.train()\ntrainer.save_model()\n```\n\n**CLI alternative**:\n```bash\ntrl dpo \\\n    --model_name_or_path Qwen\u002FQwen2.5-0.5B-Instruct \\\n    --dataset_name argilla\u002FCapybara-Preferences \\\n    --output_dir Qwen2.5-0.5B-DPO \\\n    --per_device_train_batch_size 4 \\\n    --learning_rate 5e-7 \\\n    --beta 0.1\n```\n\n### Workflow 3: Memory-efficient online RL with GRPO\n\nTrain with reinforcement learning using minimal memory.\n\nFor in-depth GRPO guidance — reward function design, critical training insights (loss behavior, mode collapse, tuning), and advanced multi-stage patterns — see **[references\u002Fgrpo-training.md](references\u002Fgrpo-training.md)**. A production-ready training script is in **[templates\u002Fbasic_grpo_training.py](templates\u002Fbasic_grpo_training.py)**.\n\nCopy this checklist:\n\n```\nGRPO Training:\n- [ ] Step 1: Define reward function\n- [ ] Step 2: Configure GRPO\n- [ ] Step 3: Train with GRPOTrainer\n```\n\n**Step 1: Define reward function**\n\n```python\ndef reward_function(completions, **kwargs):\n    \"\"\"\n    Compute rewards for completions.\n\n    Args:\n        completions: List of generated texts\n\n    Returns:\n        List of reward scores (floats)\n    \"\"\"\n    rewards = []\n    for completion in completions:\n        # Example: reward based on length and unique words\n        score = len(completion.split())  # Favor longer responses\n        score += len(set(completion.lower().split()))  # Reward unique words\n        rewards.append(score)\n    return rewards\n```\n\nOr use a reward model:\n```python\nfrom transformers import pipeline\n\nreward_model = pipeline(\"text-classification\", model=\"reward-model-path\")\n\ndef reward_from_model(completions, prompts, **kwargs):\n    # Combine prompt + completion\n    full_texts = [p + c for p, c in zip(prompts, completions)]\n    # Get reward scores\n    results = reward_model(full_texts)\n    return [r[\"score\"] for r in results]\n```\n\n**Step 2: Configure GRPO**\n\n```python\nfrom trl import GRPOConfig\n\nconfig = GRPOConfig(\n    output_dir=\"Qwen2-GRPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=1e-5,\n    num_generations=4,  # Generate 4 completions per prompt\n    max_new_tokens=128\n)\n```\n\n**Step 3: Train with GRPOTrainer**\n\n```python\nfrom datasets import load_dataset\nfrom trl import GRPOTrainer\n\n# Load prompt-only dataset\ndataset = load_dataset(\"trl-lib\u002Ftldr\", split=\"train\")\n\ntrainer = GRPOTrainer(\n    model=\"Qwen\u002FQwen2-0.5B-Instruct\",\n    reward_funcs=reward_function,  # Your reward function\n    args=config,\n    train_dataset=dataset\n)\n\ntrainer.train()\n```\n\n**CLI**:\n```bash\ntrl grpo \\\n    --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n    --dataset_name trl-lib\u002Ftldr \\\n    --output_dir Qwen2-GRPO \\\n    --num_generations 4\n```\n\n## When to use vs alternatives\n\n**Use TRL when:**\n- Need to align model with human preferences\n- Have preference data (chosen\u002Frejected pairs)\n- Want to use reinforcement learning (PPO, GRPO)\n- Need reward model training\n- Doing RLHF (full pipeline)\n\n**Method selection**:\n- **SFT**: Have prompt-completion pairs, want basic instruction following\n- **DPO**: Have preferences, want simple alignment (no reward model needed)\n- **PPO**: Have reward model, need maximum control over RL\n- **GRPO**: Memory-constrained, want online RL\n- **Reward Model**: Building RLHF pipeline, need to score generations\n\n**Use alternatives instead:**\n- **HuggingFace Trainer**: Basic fine-tuning without RL\n- **Axolotl**: YAML-based training configuration\n- **LitGPT**: Educational, minimal fine-tuning\n- **Unsloth**: Fast LoRA training\n\n## Common issues\n\n**Issue: OOM during DPO training**\n\nReduce batch size and sequence length:\n```python\nconfig = DPOConfig(\n    per_device_train_batch_size=1,  # Reduce from 4\n    max_length=512,  # Reduce from 1024\n    gradient_accumulation_steps=8  # Maintain effective batch\n)\n```\n\nOr use gradient checkpointing:\n```python\nmodel.gradient_checkpointing_enable()\n```\n\n**Issue: Poor alignment quality**\n\nTune beta parameter:\n```python\n# Higher beta = more conservative (stays closer to reference)\nconfig = DPOConfig(beta=0.5)  # Default 0.1\n\n# Lower beta = more aggressive alignment\nconfig = DPOConfig(beta=0.01)\n```\n\n**Issue: Reward model not learning**\n\nCheck loss type and learning rate:\n```python\nconfig = RewardConfig(\n    learning_rate=1e-5,  # Try different LR\n    num_train_epochs=3  # Train longer\n)\n```\n\nEnsure preference dataset has clear winners:\n```python\n# Verify dataset\nprint(dataset[0])\n# Should have clear chosen > rejected\n```\n\n**Issue: PPO training unstable**\n\nAdjust KL coefficient:\n```python\nconfig = PPOConfig(\n    kl_coef=0.1,  # Increase from 0.05\n    cliprange=0.1  # Reduce from 0.2\n)\n```\n\n## Advanced topics\n\n**SFT training guide**: See [references\u002Fsft-training.md](references\u002Fsft-training.md) for dataset formats, chat templates, packing strategies, and multi-GPU training.\n\n**DPO variants**: See [references\u002Fdpo-variants.md](references\u002Fdpo-variants.md) for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.\n\n**Reward modeling**: See [references\u002Freward-modeling.md](references\u002Freward-modeling.md) for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.\n\n**Online RL methods**: See [references\u002Fonline-rl.md](references\u002Fonline-rl.md) for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.\n\n**GRPO deep dive**: See [references\u002Fgrpo-training.md](references\u002Fgrpo-training.md) for expert-level GRPO patterns — reward function design philosophy, training insights (why loss increases, mode collapse detection), hyperparameter tuning, multi-stage training, and troubleshooting. Production-ready template in [templates\u002Fbasic_grpo_training.py](templates\u002Fbasic_grpo_training.py).\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA required)\n- **VRAM**: Depends on model and method\n  - SFT 7B: 16GB (with LoRA)\n  - DPO 7B: 24GB (stores reference model)\n  - PPO 7B: 40GB (policy + reward model)\n  - GRPO 7B: 24GB (more memory efficient)\n- **Multi-GPU**: Supported via `accelerate`\n- **Mixed precision**: BF16 recommended (A100\u002FH100)\n\n**Memory optimization**:\n- Use LoRA\u002FQLoRA for all methods\n- Enable gradient checkpointing\n- Use smaller batch sizes with gradient accumulation\n\n## Resources\n\n- Docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\n- Papers:\n  - \"Training language models to follow instructions with human feedback\" (InstructGPT, 2022)\n  - \"Direct Preference Optimization: Your Language Model is Secretly a Reward Model\" (DPO, 2023)\n  - \"Group Relative Policy Optimization\" (GRPO, 2024)\n- Examples: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002Fmain\u002Fexamples\u002Fscripts\n\n\n\n",{"name":5,"author":43,"license":14,"version":44,"metadata":45,"description":7,"dependencies":52},"Orchestra Research","1.0.0",{"hermes":46},{"tags":47},[34,19,48,37,49,36,32,35,33,50,51],"Reinforcement Learning","SFT","Preference Alignment","HuggingFace",[53,54,55,56,57,58],"trl","transformers","datasets","peft","accelerate","torch","main","3ffb682eba417bd920665a7146fecc206b3bccf8","profile\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning","https:\u002F\u002Fgithub.com\u002Feliottbusiness\u002FDeptFlow-Agent","https:\u002F\u002Fgithub.com\u002Feliottbusiness","2026-05-03 19:19:17",{"source":8,"skillName":5,"commands":66,"prompt":76,"zipAvailable":77},[67,70,73],{"tool":68,"command":69},"npx","npx skills add eliottbusiness\u002FDeptFlow-Agent --skill fine-tuning-with-trl",{"tool":71,"command":72},"bunx","bunx skills add eliottbusiness\u002FDeptFlow-Agent --skill fine-tuning-with-trl",{"tool":74,"command":75},"pnpm","pnpm dlx skills add eliottbusiness\u002FDeptFlow-Agent --skill fine-tuning-with-trl","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Feliottbusiness\u002FDeptFlow-Agent\u002Fblob\u002Fmain\u002Fprofile\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002FSKILL.md",true,[79,85,89,93,97,101],{"path":80,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Fdpo-variants.md","REFERENCE","参考文档",4297,"https:\u002F\u002Fgithub.com\u002Feliottbusiness\u002FDeptFlow-Agent\u002Fblob\u002Fmain\u002Fprofile\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002Freferences\u002Fdpo-variants.md",{"path":86,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":87,"sourceHtmlUrl":88},"references\u002Fgrpo-training.md",15901,"https:\u002F\u002Fgithub.com\u002Feliottbusiness\u002FDeptFlow-Agent\u002Fblob\u002Fmain\u002Fprofile\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002Freferences\u002Fgrpo-training.md",{"path":90,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":91,"sourceHtmlUrl":92},"references\u002Fonline-rl.md",1971,"https:\u002F\u002Fgithub.com\u002Feliottbusiness\u002FDeptFlow-Agent\u002Fblob\u002Fmain\u002Fprofile\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002Freferences\u002Fonline-rl.md",{"path":94,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":95,"sourceHtmlUrl":96},"references\u002Freward-modeling.md",2597,"https:\u002F\u002Fgithub.com\u002Feliottbusiness\u002FDeptFlow-Agent\u002Fblob\u002Fmain\u002Fprofile\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002Freferences\u002Freward-modeling.md",{"path":98,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":99,"sourceHtmlUrl":100},"references\u002Fsft-training.md",3237,"https:\u002F\u002Fgithub.com\u002Feliottbusiness\u002FDeptFlow-Agent\u002Fblob\u002Fmain\u002Fprofile\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002Freferences\u002Fsft-training.md",{"path":102,"resourceType":103,"resourceTypeDesc":104,"sizeBytes":105,"sourceHtmlUrl":106},"templates\u002Fbasic_grpo_training.py","SCRIPT","脚本",6113,"https:\u002F\u002Fgithub.com\u002Feliottbusiness\u002FDeptFlow-Agent\u002Fblob\u002Fmain\u002Fprofile\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002Ftemplates\u002Fbasic_grpo_training.py",[108,151,176,193,212,236],{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":112,"categories":113,"keywords":130,"publishedAt":38},"01M71TEDB3XPT9H5R3SH05XWES","writing-plans","Write implementation plans: bite-sized tasks, paths, code.","https:\u002F\u002Fgithub.com\u002Feliottbusiness\u002FDeptFlow-Agent\u002Fblob\u002Fmain\u002Fprofile\u002Fskills\u002Fsoftware-development\u002Fwriting-plans\u002FSKILL.md",[114,119,122,127],{"categoryNo":115,"categoryName":116,"taxonomyType":117,"taxonomyTypeDesc":118},"01JBSEEDCAT000000000000015","视觉设计","SCENARIO","使用场景",{"categoryNo":120,"categoryName":121,"taxonomyType":117,"taxonomyTypeDesc":118},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":123,"categoryName":124,"taxonomyType":125,"taxonomyTypeDesc":126},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",{"categoryNo":128,"categoryName":129,"taxonomyType":125,"taxonomyTypeDesc":126},"01JBSEEDCAT000000000000036","内容创作者",[131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150],"writing","writing plans","plans","test-driven","test-driven-development","superpowers","design","workflow","code","tasks","subagent","code-review","review","documentation","write","paths","planning","development","test","implementation",{"skillNo":152,"name":135,"nameZh":6,"description":153,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":154,"categories":155,"keywords":162,"publishedAt":38},"01M71TEB43BNQKKCRMQZDZ7SAQ","TDD: enforce RED-GREEN-REFACTOR, tests before code.","https:\u002F\u002Fgithub.com\u002Feliottbusiness\u002FDeptFlow-Agent\u002Fblob\u002Fmain\u002Fprofile\u002Fskills\u002Fsoftware-development\u002Ftest-driven-development\u002FSKILL.md",[156,159],{"categoryNo":157,"categoryName":158,"taxonomyType":117,"taxonomyTypeDesc":118},"01JBSEEDCAT000000000000007","代码审查",{"categoryNo":160,"categoryName":161,"taxonomyType":117,"taxonomyTypeDesc":118},"01JBSEEDCAT000000000000008","测试与质量",[163,148,149,164,165,166,167,136,131,168,139,141,169,170,171,172,173,133,174,175],"test driven development","driven","green","red-green-refactor","testing","debugging","systematic","refactor","systematic-debugging","enforce","tests","tdd","quality",{"skillNo":177,"name":171,"nameZh":6,"description":178,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":179,"categories":180,"keywords":182,"publishedAt":38},"01M71TE76KV9R1WY1V5XQXMMTJ","4-phase root cause debugging: understand bugs before fixing.","https:\u002F\u002Fgithub.com\u002Feliottbusiness\u002FDeptFlow-Agent\u002Fblob\u002Fmain\u002Fprofile\u002Fskills\u002Fsoftware-development\u002Fsystematic-debugging\u002FSKILL.md",[181],{"categoryNo":160,"categoryName":161,"taxonomyType":117,"taxonomyTypeDesc":118},[183,168,169,134,184,185,135,136,131,141,186,187,148,188,133,189,149,190,191,192],"systematic debugging","problem","problem-solving","phase","bugs","root cause","root-cause","investigation","understand","hermes",{"skillNo":194,"name":195,"nameZh":6,"description":196,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":197,"categories":198,"keywords":204,"publishedAt":38},"01M71TE2Y2TRFZNT6YRPKQA33R","subagent-driven-development","Execute plans via delegate_task subagents (2-stage review).","https:\u002F\u002Fgithub.com\u002Feliottbusiness\u002FDeptFlow-Agent\u002Fblob\u002Fmain\u002Fprofile\u002Fskills\u002Fsoftware-development\u002Fsubagent-driven-development\u002FSKILL.md",[199,200,203],{"categoryNo":120,"categoryName":121,"taxonomyType":117,"taxonomyTypeDesc":118},{"categoryNo":201,"categoryName":202,"taxonomyType":117,"taxonomyTypeDesc":118},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":128,"categoryName":129,"taxonomyType":125,"taxonomyTypeDesc":126},[205,141,148,164,134,135,206,207,136,131,138,139,142,143,208,209,133,210,211,149],"subagent driven development","delegate","delegation","subagents","parallel","stage","task",{"skillNo":213,"name":214,"nameZh":6,"description":215,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":216,"categories":217,"keywords":221,"publishedAt":38},"01M71TE0E7WRGXY6N0WTMDXY23","spike","Throwaway experiments to validate an idea before build.","https:\u002F\u002Fgithub.com\u002Feliottbusiness\u002FDeptFlow-Agent\u002Fblob\u002Fmain\u002Fprofile\u002Fskills\u002Fsoftware-development\u002Fspike\u002FSKILL.md",[218],{"categoryNo":219,"categoryName":220,"taxonomyType":117,"taxonomyTypeDesc":118},"01JBSEEDCAT000000000000021","资料检索与研究",[214,222,223,224,147,148,133,225,226,131,227,228,141,229,230,231,232,233,234,235],"concept","exploration","get","throwaway","shit","proof-of-concept","done","prototype","plan","research","build","sketch","mvp","validate",{"skillNo":237,"name":238,"nameZh":6,"description":239,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":240,"categories":241,"keywords":244,"publishedAt":38},"01M71TDY3JXZHWSYNF3AZ2XM2D","requesting-code-review","Pre-commit review: security scan, quality gates, auto-fix.","https:\u002F\u002Fgithub.com\u002Feliottbusiness\u002FDeptFlow-Agent\u002Fblob\u002Fmain\u002Fprofile\u002Fskills\u002Fsoftware-development\u002Frequesting-code-review\u002FSKILL.md",[242,243],{"categoryNo":157,"categoryName":158,"taxonomyType":117,"taxonomyTypeDesc":118},{"categoryNo":123,"categoryName":124,"taxonomyType":125,"taxonomyTypeDesc":126},[245,246,139,143,134,135,247,248,136,249,131,250,251,141,142,252,253,148,254,133],"code review","requesting code review","fix","auto-fix","auto","security scan","commit","verification","security","scan"]