[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M71FX3CQZX36DDHC2CPPQMMF":3,"skill-same:01M71FX3CQZX36DDHC2CPPQMMF":102},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":38,"linkOnly":39,"markdownContent":40,"markdownBody":41,"frontMatter":42,"refName":64,"commitSha":65,"skillDirectory":66,"forksCount":13,"repositoryHtmlUrl":67,"repositoryHomepageUrl":6,"ownerHtmlUrl":68,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":69,"install":70,"resources":83},"01M71FX3CQZX36DDHC2CPPQMMF","fine-tuning-with-trl",null,"Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO\u002FGRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.","gjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6","gjbvsjjvgm-sketch","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F278511911?v=4","01M71F3JXKZQ70H2KCHHTXKZDA","01M71F3JY8A79Z3NZBW08TF4JD",0,"MIT","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37],"tuning","TRL","with","fine tuning","fine tuning with trl","Post-Training","learning","Optimization","for","orchestra","feedback","Transformers","human","llms","Accelerate","when","training","align","model","alignment","2026-11-10 10:12:30",false,"---\nname: fine-tuning-with-trl\ndescription: Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO\u002FGRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ndependencies: [trl, transformers, datasets, peft, accelerate, torch]\nmetadata:\n  hermes:\n    tags: [Post-Training, TRL, Reinforcement Learning, Fine-Tuning, SFT, DPO, PPO, GRPO, RLHF, Preference Alignment, HuggingFace]\n\n---\n\n# TRL - Transformer Reinforcement Learning\n\n## Quick start\n\nTRL provides post-training methods for aligning language models with human preferences.\n\n**Installation**:\n```bash\npip install trl transformers datasets peft accelerate\n```\n\n**Supervised Fine-Tuning** (instruction tuning):\n```python\nfrom trl import SFTTrainer\n\ntrainer = SFTTrainer(\n    model=\"Qwen\u002FQwen2.5-0.5B\",\n    train_dataset=dataset,  # Prompt-completion pairs\n)\ntrainer.train()\n```\n\n**DPO** (align with preferences):\n```python\nfrom trl import DPOTrainer, DPOConfig\n\nconfig = DPOConfig(output_dir=\"model-dpo\", beta=0.1)\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=preference_dataset,  # chosen\u002Frejected pairs\n    processing_class=tokenizer\n)\ntrainer.train()\n```\n\n## Common workflows\n\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)\n\nComplete pipeline from base model to human-aligned model.\n\nCopy this checklist:\n\n```\nRLHF Training:\n- [ ] Step 1: Supervised fine-tuning (SFT)\n- [ ] Step 2: Train reward model\n- [ ] Step 3: PPO reinforcement learning\n- [ ] Step 4: Evaluate aligned model\n```\n\n**Step 1: Supervised fine-tuning**\n\nTrain base model on instruction-following data:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import SFTTrainer, SFTConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\n\n# Load instruction dataset\ndataset = load_dataset(\"trl-lib\u002FCapybara\", split=\"train\")\n\n# Configure training\ntraining_args = SFTConfig(\n    output_dir=\"Qwen2.5-0.5B-SFT\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=2e-5,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\n# Train\ntrainer = SFTTrainer(\n    model=model,\n    args=training_args,\n    train_dataset=dataset,\n    tokenizer=tokenizer\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 2: Train reward model**\n\nTrain model to predict human preferences:\n\n```python\nfrom transformers import AutoModelForSequenceClassification\nfrom trl import RewardTrainer, RewardConfig\n\n# Load SFT model as base\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    \"Qwen2.5-0.5B-SFT\",\n    num_labels=1  # Single reward score\n)\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen2.5-0.5B-SFT\")\n\n# Load preference data (chosen\u002Frejected pairs)\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n\n# Configure training\ntraining_args = RewardConfig(\n    output_dir=\"Qwen2.5-0.5B-Reward\",\n    per_device_train_batch_size=2,\n    num_train_epochs=1,\n    learning_rate=1e-5\n)\n\n# Train reward model\ntrainer = RewardTrainer(\n    model=model,\n    args=training_args,\n    processing_class=tokenizer,\n    train_dataset=dataset\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 3: PPO reinforcement learning**\n\nOptimize policy using reward model:\n\n```bash\npython -m trl.scripts.ppo \\\n    --model_name_or_path Qwen2.5-0.5B-SFT \\\n    --reward_model_path Qwen2.5-0.5B-Reward \\\n    --dataset_name trl-internal-testing\u002Fdescriptiveness-sentiment-trl-style \\\n    --output_dir Qwen2.5-0.5B-PPO \\\n    --learning_rate 3e-6 \\\n    --per_device_train_batch_size 64 \\\n    --total_episodes 10000\n```\n\n**Step 4: Evaluate**\n\n```python\nfrom transformers import pipeline\n\n# Load aligned model\ngenerator = pipeline(\"text-generation\", model=\"Qwen2.5-0.5B-PPO\")\n\n# Test\nprompt = \"Explain quantum computing to a 10-year-old\"\noutput = generator(prompt, max_length=200)[0][\"generated_text\"]\nprint(output)\n```\n\n### Workflow 2: Simple preference alignment with DPO\n\nAlign model with preferences without reward model.\n\nCopy this checklist:\n\n```\nDPO Training:\n- [ ] Step 1: Prepare preference dataset\n- [ ] Step 2: Configure DPO\n- [ ] Step 3: Train with DPOTrainer\n- [ ] Step 4: Evaluate alignment\n```\n\n**Step 1: Prepare preference dataset**\n\nDataset format:\n```json\n{\n  \"prompt\": \"What is the capital of France?\",\n  \"chosen\": \"The capital of France is Paris.\",\n  \"rejected\": \"I don't know.\"\n}\n```\n\nLoad dataset:\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n# Or load your own\n# dataset = load_dataset(\"json\", data_files=\"preferences.json\")\n```\n\n**Step 2: Configure DPO**\n\n```python\nfrom trl import DPOConfig\n\nconfig = DPOConfig(\n    output_dir=\"Qwen2.5-0.5B-DPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=5e-7,\n    beta=0.1,  # KL penalty strength\n    max_prompt_length=512,\n    max_length=1024,\n    logging_steps=10\n)\n```\n\n**Step 3: Train with DPOTrainer**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import DPOTrainer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\n\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=dataset,\n    processing_class=tokenizer\n)\n\ntrainer.train()\ntrainer.save_model()\n```\n\n**CLI alternative**:\n```bash\ntrl dpo \\\n    --model_name_or_path Qwen\u002FQwen2.5-0.5B-Instruct \\\n    --dataset_name argilla\u002FCapybara-Preferences \\\n    --output_dir Qwen2.5-0.5B-DPO \\\n    --per_device_train_batch_size 4 \\\n    --learning_rate 5e-7 \\\n    --beta 0.1\n```\n\n### Workflow 3: Memory-efficient online RL with GRPO\n\nTrain with reinforcement learning using minimal memory.\n\nCopy this checklist:\n\n```\nGRPO Training:\n- [ ] Step 1: Define reward function\n- [ ] Step 2: Configure GRPO\n- [ ] Step 3: Train with GRPOTrainer\n```\n\n**Step 1: Define reward function**\n\n```python\ndef reward_function(completions, **kwargs):\n    \"\"\"\n    Compute rewards for completions.\n\n    Args:\n        completions: List of generated texts\n\n    Returns:\n        List of reward scores (floats)\n    \"\"\"\n    rewards = []\n    for completion in completions:\n        # Example: reward based on length and unique words\n        score = len(completion.split())  # Favor longer responses\n        score += len(set(completion.lower().split()))  # Reward unique words\n        rewards.append(score)\n    return rewards\n```\n\nOr use a reward model:\n```python\nfrom transformers import pipeline\n\nreward_model = pipeline(\"text-classification\", model=\"reward-model-path\")\n\ndef reward_from_model(completions, prompts, **kwargs):\n    # Combine prompt + completion\n    full_texts = [p + c for p, c in zip(prompts, completions)]\n    # Get reward scores\n    results = reward_model(full_texts)\n    return [r[\"score\"] for r in results]\n```\n\n**Step 2: Configure GRPO**\n\n```python\nfrom trl import GRPOConfig\n\nconfig = GRPOConfig(\n    output_dir=\"Qwen2-GRPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=1e-5,\n    num_generations=4,  # Generate 4 completions per prompt\n    max_new_tokens=128\n)\n```\n\n**Step 3: Train with GRPOTrainer**\n\n```python\nfrom datasets import load_dataset\nfrom trl import GRPOTrainer\n\n# Load prompt-only dataset\ndataset = load_dataset(\"trl-lib\u002Ftldr\", split=\"train\")\n\ntrainer = GRPOTrainer(\n    model=\"Qwen\u002FQwen2-0.5B-Instruct\",\n    reward_funcs=reward_function,  # Your reward function\n    args=config,\n    train_dataset=dataset\n)\n\ntrainer.train()\n```\n\n**CLI**:\n```bash\ntrl grpo \\\n    --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n    --dataset_name trl-lib\u002Ftldr \\\n    --output_dir Qwen2-GRPO \\\n    --num_generations 4\n```\n\n## When to use vs alternatives\n\n**Use TRL when:**\n- Need to align model with human preferences\n- Have preference data (chosen\u002Frejected pairs)\n- Want to use reinforcement learning (PPO, GRPO)\n- Need reward model training\n- Doing RLHF (full pipeline)\n\n**Method selection**:\n- **SFT**: Have prompt-completion pairs, want basic instruction following\n- **DPO**: Have preferences, want simple alignment (no reward model needed)\n- **PPO**: Have reward model, need maximum control over RL\n- **GRPO**: Memory-constrained, want online RL\n- **Reward Model**: Building RLHF pipeline, need to score generations\n\n**Use alternatives instead:**\n- **HuggingFace Trainer**: Basic fine-tuning without RL\n- **Axolotl**: YAML-based training configuration\n- **LitGPT**: Educational, minimal fine-tuning\n- **Unsloth**: Fast LoRA training\n\n## Common issues\n\n**Issue: OOM during DPO training**\n\nReduce batch size and sequence length:\n```python\nconfig = DPOConfig(\n    per_device_train_batch_size=1,  # Reduce from 4\n    max_length=512,  # Reduce from 1024\n    gradient_accumulation_steps=8  # Maintain effective batch\n)\n```\n\nOr use gradient checkpointing:\n```python\nmodel.gradient_checkpointing_enable()\n```\n\n**Issue: Poor alignment quality**\n\nTune beta parameter:\n```python\n# Higher beta = more conservative (stays closer to reference)\nconfig = DPOConfig(beta=0.5)  # Default 0.1\n\n# Lower beta = more aggressive alignment\nconfig = DPOConfig(beta=0.01)\n```\n\n**Issue: Reward model not learning**\n\nCheck loss type and learning rate:\n```python\nconfig = RewardConfig(\n    learning_rate=1e-5,  # Try different LR\n    num_train_epochs=3  # Train longer\n)\n```\n\nEnsure preference dataset has clear winners:\n```python\n# Verify dataset\nprint(dataset[0])\n# Should have clear chosen > rejected\n```\n\n**Issue: PPO training unstable**\n\nAdjust KL coefficient:\n```python\nconfig = PPOConfig(\n    kl_coef=0.1,  # Increase from 0.05\n    cliprange=0.1  # Reduce from 0.2\n)\n```\n\n## Advanced topics\n\n**SFT training guide**: See [references\u002Fsft-training.md](references\u002Fsft-training.md) for dataset formats, chat templates, packing strategies, and multi-GPU training.\n\n**DPO variants**: See [references\u002Fdpo-variants.md](references\u002Fdpo-variants.md) for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.\n\n**Reward modeling**: See [references\u002Freward-modeling.md](references\u002Freward-modeling.md) for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.\n\n**Online RL methods**: See [references\u002Fonline-rl.md](references\u002Fonline-rl.md) for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA required)\n- **VRAM**: Depends on model and method\n  - SFT 7B: 16GB (with LoRA)\n  - DPO 7B: 24GB (stores reference model)\n  - PPO 7B: 40GB (policy + reward model)\n  - GRPO 7B: 24GB (more memory efficient)\n- **Multi-GPU**: Supported via `accelerate`\n- **Mixed precision**: BF16 recommended (A100\u002FH100)\n\n**Memory optimization**:\n- Use LoRA\u002FQLoRA for all methods\n- Enable gradient checkpointing\n- Use smaller batch sizes with gradient accumulation\n\n## Resources\n\n- Docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\n- Papers:\n  - \"Training language models to follow instructions with human feedback\" (InstructGPT, 2022)\n  - \"Direct Preference Optimization: Your Language Model is Secretly a Reward Model\" (DPO, 2023)\n  - \"Group Relative Policy Optimization\" (GRPO, 2024)\n- Examples: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002Fmain\u002Fexamples\u002Fscripts\n\n\n\n","# TRL - Transformer Reinforcement Learning\n\n## Quick start\n\nTRL provides post-training methods for aligning language models with human preferences.\n\n**Installation**:\n```bash\npip install trl transformers datasets peft accelerate\n```\n\n**Supervised Fine-Tuning** (instruction tuning):\n```python\nfrom trl import SFTTrainer\n\ntrainer = SFTTrainer(\n    model=\"Qwen\u002FQwen2.5-0.5B\",\n    train_dataset=dataset,  # Prompt-completion pairs\n)\ntrainer.train()\n```\n\n**DPO** (align with preferences):\n```python\nfrom trl import DPOTrainer, DPOConfig\n\nconfig = DPOConfig(output_dir=\"model-dpo\", beta=0.1)\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=preference_dataset,  # chosen\u002Frejected pairs\n    processing_class=tokenizer\n)\ntrainer.train()\n```\n\n## Common workflows\n\n### Workflow 1: Full RLHF pipeline (SFT → Reward Model → PPO)\n\nComplete pipeline from base model to human-aligned model.\n\nCopy this checklist:\n\n```\nRLHF Training:\n- [ ] Step 1: Supervised fine-tuning (SFT)\n- [ ] Step 2: Train reward model\n- [ ] Step 3: PPO reinforcement learning\n- [ ] Step 4: Evaluate aligned model\n```\n\n**Step 1: Supervised fine-tuning**\n\nTrain base model on instruction-following data:\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import SFTTrainer, SFTConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B\")\n\n# Load instruction dataset\ndataset = load_dataset(\"trl-lib\u002FCapybara\", split=\"train\")\n\n# Configure training\ntraining_args = SFTConfig(\n    output_dir=\"Qwen2.5-0.5B-SFT\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=2e-5,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\n# Train\ntrainer = SFTTrainer(\n    model=model,\n    args=training_args,\n    train_dataset=dataset,\n    tokenizer=tokenizer\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 2: Train reward model**\n\nTrain model to predict human preferences:\n\n```python\nfrom transformers import AutoModelForSequenceClassification\nfrom trl import RewardTrainer, RewardConfig\n\n# Load SFT model as base\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    \"Qwen2.5-0.5B-SFT\",\n    num_labels=1  # Single reward score\n)\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen2.5-0.5B-SFT\")\n\n# Load preference data (chosen\u002Frejected pairs)\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n\n# Configure training\ntraining_args = RewardConfig(\n    output_dir=\"Qwen2.5-0.5B-Reward\",\n    per_device_train_batch_size=2,\n    num_train_epochs=1,\n    learning_rate=1e-5\n)\n\n# Train reward model\ntrainer = RewardTrainer(\n    model=model,\n    args=training_args,\n    processing_class=tokenizer,\n    train_dataset=dataset\n)\ntrainer.train()\ntrainer.save_model()\n```\n\n**Step 3: PPO reinforcement learning**\n\nOptimize policy using reward model:\n\n```bash\npython -m trl.scripts.ppo \\\n    --model_name_or_path Qwen2.5-0.5B-SFT \\\n    --reward_model_path Qwen2.5-0.5B-Reward \\\n    --dataset_name trl-internal-testing\u002Fdescriptiveness-sentiment-trl-style \\\n    --output_dir Qwen2.5-0.5B-PPO \\\n    --learning_rate 3e-6 \\\n    --per_device_train_batch_size 64 \\\n    --total_episodes 10000\n```\n\n**Step 4: Evaluate**\n\n```python\nfrom transformers import pipeline\n\n# Load aligned model\ngenerator = pipeline(\"text-generation\", model=\"Qwen2.5-0.5B-PPO\")\n\n# Test\nprompt = \"Explain quantum computing to a 10-year-old\"\noutput = generator(prompt, max_length=200)[0][\"generated_text\"]\nprint(output)\n```\n\n### Workflow 2: Simple preference alignment with DPO\n\nAlign model with preferences without reward model.\n\nCopy this checklist:\n\n```\nDPO Training:\n- [ ] Step 1: Prepare preference dataset\n- [ ] Step 2: Configure DPO\n- [ ] Step 3: Train with DPOTrainer\n- [ ] Step 4: Evaluate alignment\n```\n\n**Step 1: Prepare preference dataset**\n\nDataset format:\n```json\n{\n  \"prompt\": \"What is the capital of France?\",\n  \"chosen\": \"The capital of France is Paris.\",\n  \"rejected\": \"I don't know.\"\n}\n```\n\nLoad dataset:\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"trl-lib\u002Fultrafeedback_binarized\", split=\"train\")\n# Or load your own\n# dataset = load_dataset(\"json\", data_files=\"preferences.json\")\n```\n\n**Step 2: Configure DPO**\n\n```python\nfrom trl import DPOConfig\n\nconfig = DPOConfig(\n    output_dir=\"Qwen2.5-0.5B-DPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=5e-7,\n    beta=0.1,  # KL penalty strength\n    max_prompt_length=512,\n    max_length=1024,\n    logging_steps=10\n)\n```\n\n**Step 3: Train with DPOTrainer**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom trl import DPOTrainer\n\nmodel = AutoModelForCausalLM.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"Qwen\u002FQwen2.5-0.5B-Instruct\")\n\ntrainer = DPOTrainer(\n    model=model,\n    args=config,\n    train_dataset=dataset,\n    processing_class=tokenizer\n)\n\ntrainer.train()\ntrainer.save_model()\n```\n\n**CLI alternative**:\n```bash\ntrl dpo \\\n    --model_name_or_path Qwen\u002FQwen2.5-0.5B-Instruct \\\n    --dataset_name argilla\u002FCapybara-Preferences \\\n    --output_dir Qwen2.5-0.5B-DPO \\\n    --per_device_train_batch_size 4 \\\n    --learning_rate 5e-7 \\\n    --beta 0.1\n```\n\n### Workflow 3: Memory-efficient online RL with GRPO\n\nTrain with reinforcement learning using minimal memory.\n\nCopy this checklist:\n\n```\nGRPO Training:\n- [ ] Step 1: Define reward function\n- [ ] Step 2: Configure GRPO\n- [ ] Step 3: Train with GRPOTrainer\n```\n\n**Step 1: Define reward function**\n\n```python\ndef reward_function(completions, **kwargs):\n    \"\"\"\n    Compute rewards for completions.\n\n    Args:\n        completions: List of generated texts\n\n    Returns:\n        List of reward scores (floats)\n    \"\"\"\n    rewards = []\n    for completion in completions:\n        # Example: reward based on length and unique words\n        score = len(completion.split())  # Favor longer responses\n        score += len(set(completion.lower().split()))  # Reward unique words\n        rewards.append(score)\n    return rewards\n```\n\nOr use a reward model:\n```python\nfrom transformers import pipeline\n\nreward_model = pipeline(\"text-classification\", model=\"reward-model-path\")\n\ndef reward_from_model(completions, prompts, **kwargs):\n    # Combine prompt + completion\n    full_texts = [p + c for p, c in zip(prompts, completions)]\n    # Get reward scores\n    results = reward_model(full_texts)\n    return [r[\"score\"] for r in results]\n```\n\n**Step 2: Configure GRPO**\n\n```python\nfrom trl import GRPOConfig\n\nconfig = GRPOConfig(\n    output_dir=\"Qwen2-GRPO\",\n    per_device_train_batch_size=4,\n    num_train_epochs=1,\n    learning_rate=1e-5,\n    num_generations=4,  # Generate 4 completions per prompt\n    max_new_tokens=128\n)\n```\n\n**Step 3: Train with GRPOTrainer**\n\n```python\nfrom datasets import load_dataset\nfrom trl import GRPOTrainer\n\n# Load prompt-only dataset\ndataset = load_dataset(\"trl-lib\u002Ftldr\", split=\"train\")\n\ntrainer = GRPOTrainer(\n    model=\"Qwen\u002FQwen2-0.5B-Instruct\",\n    reward_funcs=reward_function,  # Your reward function\n    args=config,\n    train_dataset=dataset\n)\n\ntrainer.train()\n```\n\n**CLI**:\n```bash\ntrl grpo \\\n    --model_name_or_path Qwen\u002FQwen2-0.5B-Instruct \\\n    --dataset_name trl-lib\u002Ftldr \\\n    --output_dir Qwen2-GRPO \\\n    --num_generations 4\n```\n\n## When to use vs alternatives\n\n**Use TRL when:**\n- Need to align model with human preferences\n- Have preference data (chosen\u002Frejected pairs)\n- Want to use reinforcement learning (PPO, GRPO)\n- Need reward model training\n- Doing RLHF (full pipeline)\n\n**Method selection**:\n- **SFT**: Have prompt-completion pairs, want basic instruction following\n- **DPO**: Have preferences, want simple alignment (no reward model needed)\n- **PPO**: Have reward model, need maximum control over RL\n- **GRPO**: Memory-constrained, want online RL\n- **Reward Model**: Building RLHF pipeline, need to score generations\n\n**Use alternatives instead:**\n- **HuggingFace Trainer**: Basic fine-tuning without RL\n- **Axolotl**: YAML-based training configuration\n- **LitGPT**: Educational, minimal fine-tuning\n- **Unsloth**: Fast LoRA training\n\n## Common issues\n\n**Issue: OOM during DPO training**\n\nReduce batch size and sequence length:\n```python\nconfig = DPOConfig(\n    per_device_train_batch_size=1,  # Reduce from 4\n    max_length=512,  # Reduce from 1024\n    gradient_accumulation_steps=8  # Maintain effective batch\n)\n```\n\nOr use gradient checkpointing:\n```python\nmodel.gradient_checkpointing_enable()\n```\n\n**Issue: Poor alignment quality**\n\nTune beta parameter:\n```python\n# Higher beta = more conservative (stays closer to reference)\nconfig = DPOConfig(beta=0.5)  # Default 0.1\n\n# Lower beta = more aggressive alignment\nconfig = DPOConfig(beta=0.01)\n```\n\n**Issue: Reward model not learning**\n\nCheck loss type and learning rate:\n```python\nconfig = RewardConfig(\n    learning_rate=1e-5,  # Try different LR\n    num_train_epochs=3  # Train longer\n)\n```\n\nEnsure preference dataset has clear winners:\n```python\n# Verify dataset\nprint(dataset[0])\n# Should have clear chosen > rejected\n```\n\n**Issue: PPO training unstable**\n\nAdjust KL coefficient:\n```python\nconfig = PPOConfig(\n    kl_coef=0.1,  # Increase from 0.05\n    cliprange=0.1  # Reduce from 0.2\n)\n```\n\n## Advanced topics\n\n**SFT training guide**: See [references\u002Fsft-training.md](references\u002Fsft-training.md) for dataset formats, chat templates, packing strategies, and multi-GPU training.\n\n**DPO variants**: See [references\u002Fdpo-variants.md](references\u002Fdpo-variants.md) for IPO, cDPO, RPO, and other DPO loss functions with recommended hyperparameters.\n\n**Reward modeling**: See [references\u002Freward-modeling.md](references\u002Freward-modeling.md) for outcome vs process rewards, Bradley-Terry loss, and reward model evaluation.\n\n**Online RL methods**: See [references\u002Fonline-rl.md](references\u002Fonline-rl.md) for PPO, GRPO, RLOO, and OnlineDPO with detailed configurations.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA required)\n- **VRAM**: Depends on model and method\n  - SFT 7B: 16GB (with LoRA)\n  - DPO 7B: 24GB (stores reference model)\n  - PPO 7B: 40GB (policy + reward model)\n  - GRPO 7B: 24GB (more memory efficient)\n- **Multi-GPU**: Supported via `accelerate`\n- **Mixed precision**: BF16 recommended (A100\u002FH100)\n\n**Memory optimization**:\n- Use LoRA\u002FQLoRA for all methods\n- Enable gradient checkpointing\n- Use smaller batch sizes with gradient accumulation\n\n## Resources\n\n- Docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002F\n- GitHub: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\n- Papers:\n  - \"Training language models to follow instructions with human feedback\" (InstructGPT, 2022)\n  - \"Direct Preference Optimization: Your Language Model is Secretly a Reward Model\" (DPO, 2023)\n  - \"Group Relative Policy Optimization\" (GRPO, 2024)\n- Examples: https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Ftree\u002Fmain\u002Fexamples\u002Fscripts\n\n\n\n",{"name":5,"author":43,"license":14,"version":44,"metadata":45,"description":7,"dependencies":57},"Orchestra Research","1.0.0",{"hermes":46},{"tags":47},[23,19,48,49,50,51,52,53,54,55,56],"Reinforcement Learning","Fine-Tuning","SFT","DPO","PPO","GRPO","RLHF","Preference Alignment","HuggingFace",[58,59,60,61,62,63],"trl","transformers","datasets","peft","accelerate","torch","main","93bbc269671fbb340afa15b36a8766f15dc30df2","hermes-agent\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch","2026-04-27 02:51:28",{"source":8,"skillName":5,"commands":71,"prompt":81,"zipAvailable":82},[72,75,78],{"tool":73,"command":74},"npx","npx skills add gjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6 --skill fine-tuning-with-trl",{"tool":76,"command":77},"bunx","bunx skills add gjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6 --skill fine-tuning-with-trl",{"tool":79,"command":80},"pnpm","pnpm dlx skills add gjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6 --skill fine-tuning-with-trl","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002FSKILL.md",true,[84,90,94,98],{"path":85,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Fdpo-variants.md","REFERENCE","参考文档",4297,"https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002Freferences\u002Fdpo-variants.md",{"path":91,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":92,"sourceHtmlUrl":93},"references\u002Fonline-rl.md",1971,"https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002Freferences\u002Fonline-rl.md",{"path":95,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":96,"sourceHtmlUrl":97},"references\u002Freward-modeling.md",2597,"https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002Freferences\u002Freward-modeling.md",{"path":99,"resourceType":86,"resourceTypeDesc":87,"sizeBytes":100,"sourceHtmlUrl":101},"references\u002Fsft-training.md",3237,"https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fmlops\u002Ftraining\u002Ftrl-fine-tuning\u002Freferences\u002Fsft-training.md",[103,144,172,188,207,228],{"skillNo":104,"name":105,"nameZh":6,"description":106,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":107,"categories":108,"keywords":125,"publishedAt":38},"01M71G6TJ2RAQ1J0PV3K6D9GQA","writing-plans","Use when you have a spec or requirements for a multi-step task. Creates comprehensive implementation plans with bite-sized tasks, exact file paths, and complete code examples.","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6\u002Fblob\u002Fmain\u002Fhermes_sovereign_data\u002Fskills\u002Fsoftware-development\u002Fwriting-plans\u002FSKILL.md",[109,114,117,122],{"categoryNo":110,"categoryName":111,"taxonomyType":112,"taxonomyTypeDesc":113},"01JBSEEDCAT000000000000015","视觉设计","SCENARIO","使用场景",{"categoryNo":115,"categoryName":116,"taxonomyType":112,"taxonomyTypeDesc":113},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":118,"categoryName":119,"taxonomyType":120,"taxonomyTypeDesc":121},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",{"categoryNo":123,"categoryName":124,"taxonomyType":120,"taxonomyTypeDesc":121},"01JBSEEDCAT000000000000036","内容创作者",[126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,26,143,33],"writing","writing plans","plans","design","review","workflow","documentation","tasks","development","code","planning","code-review","subagent","superpowers","paths","test-driven-development","you","test-driven",{"skillNo":145,"name":141,"nameZh":6,"description":146,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":147,"categories":148,"keywords":157,"publishedAt":38},"01M71G6R81013R3T7W5VP56GCB","Use when implementing any feature or bugfix, before writing implementation code. Enforces RED-GREEN-REFACTOR cycle with test-first approach.","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6\u002Fblob\u002Fmain\u002Fhermes_sovereign_data\u002Fskills\u002Fsoftware-development\u002Ftest-driven-development\u002FSKILL.md",[149,152,155,156],{"categoryNo":150,"categoryName":151,"taxonomyType":112,"taxonomyTypeDesc":113},"01JBSEEDCAT000000000000007","代码审查",{"categoryNo":153,"categoryName":154,"taxonomyType":112,"taxonomyTypeDesc":113},"01JBSEEDCAT000000000000008","测试与质量",{"categoryNo":115,"categoryName":116,"taxonomyType":112,"taxonomyTypeDesc":113},{"categoryNo":123,"categoryName":124,"taxonomyType":120,"taxonomyTypeDesc":121},[158,159,134,160,161,126,162,135,163,138,139,164,165,166,167,168,169,170,171,33],"test","test driven development","driven","testing","debugging","refactor","red-green-refactor","cycle","green","systematic","bugfix","implementing","systematic-debugging","test-first",{"skillNo":173,"name":170,"nameZh":6,"description":174,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":175,"categories":176,"keywords":178,"publishedAt":38},"01M71G6N0Y546JZEDF25E2PMCE","Use when encountering any bug, test failure, or unexpected behavior. 4-phase root cause investigation — NO fixes without understanding the problem first.","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6\u002Fblob\u002Fmain\u002Fhermes_sovereign_data\u002Fskills\u002Fsoftware-development\u002Fsystematic-debugging\u002FSKILL.md",[177],{"categoryNo":153,"categoryName":154,"taxonomyType":112,"taxonomyTypeDesc":113},[179,162,167,126,158,134,138,180,139,181,182,183,128,184,141,185,143,186,187,33],"systematic debugging","use","bug","root-cause","problem-solving","problem","fixes","phase","root cause",{"skillNo":189,"name":190,"nameZh":6,"description":191,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":192,"categories":193,"keywords":200,"publishedAt":38},"01M71G6J0DXEGY48GH4KVW2SJH","subagent-driven-development","Use when executing implementation plans with independent tasks. Dispatches fresh delegate_task per task with two-stage review (spec compliance then code quality).","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6\u002Fblob\u002Fmain\u002Fhermes_sovereign_data\u002Fskills\u002Fsoftware-development\u002Fsubagent-driven-development\u002FSKILL.md",[194,195,196,197],{"categoryNo":115,"categoryName":116,"taxonomyType":112,"taxonomyTypeDesc":113},{"categoryNo":118,"categoryName":119,"taxonomyType":120,"taxonomyTypeDesc":121},{"categoryNo":123,"categoryName":124,"taxonomyType":120,"taxonomyTypeDesc":121},{"categoryNo":198,"categoryName":199,"taxonomyType":120,"taxonomyTypeDesc":121},"01JBSEEDCAT000000000000037","律师与法务",[134,201,138,160,126,130,131,202,133,135,203,137,204,205,139,128,206,141,143,33],"subagent driven development","code quality","delegation","parallel","delegate","fresh",{"skillNo":208,"name":209,"nameZh":6,"description":210,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":211,"categories":212,"keywords":215,"publishedAt":38},"01M71G6EFFBQX6634E6XHB09Y1","requesting-code-review","Pre-commit verification pipeline — static security scan, baseline-aware quality gates, independent reviewer subagent, and auto-fix loop. Use after code changes and before committing, pushing, or opening a PR.","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6\u002Fblob\u002Fmain\u002Fhermes_sovereign_data\u002Fskills\u002Fsoftware-development\u002Frequesting-code-review\u002FSKILL.md",[213,214],{"categoryNo":150,"categoryName":151,"taxonomyType":112,"taxonomyTypeDesc":113},{"categoryNo":118,"categoryName":119,"taxonomyType":120,"taxonomyTypeDesc":121},[216,130,135,217,126,218,137,219,220,221,222,138,223,224,225,139,141,226,227,143],"code review","requesting code review","commit","verification","security scan","pipeline","loop","reviewer","fix","auto","static","auto-fix",{"skillNo":229,"name":230,"nameZh":6,"description":231,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":232,"categories":233,"keywords":234,"publishedAt":38},"01M71G6CM8ANSWVFXCN9AAD58Y","plan","Plan mode for Hermes — inspect context, write a markdown plan into the active workspace's `.hermes\u002Fplans\u002F` directory, and do not execute the work.","https:\u002F\u002Fgithub.com\u002Fgjbvsjjvgm-sketch\u002Fhermes-sovereign-unified-v13.6\u002Fblob\u002Fmain\u002Fhermes_sovereign_data\u002Fskills\u002Fsoftware-development\u002Fplan\u002FSKILL.md",[],[230,126,235,131,134,136,236,138,237,238,239,240,241,242,128,243,244,245,26,246],"directory","markdown","workspace","work","write","implementation","do","active","plan mode","inspect","mode","plan-mode"]