trl-training
Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning). Supports SFT, DPO, GRPO, KTO, RLOO and Reward Model training via CLI commands.
- License 许可
- Apache-2.0
- 最近更新
- 2026-10-22
- Stars
- ★ 1
- Forks
- 0
01 前言 Front Matter
- name
- trl-training
- risk
- unknown
- source
- https://github.com/huggingface/skills/tree/main/skills/trl-training
- license
- Apache-2.0
- date_added
- 1782864000000
- description
- Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning). Supports SFT, DPO, GRPO, KTO, RLOO and Reward Model training via CLI commands.
- source_repo
- huggingface/skills
- source_type
- official
- license_source
- https://github.com/huggingface/skills/blob/main/LICENSE
03 SKILL.md 正文
TRL Training Skill
When to Use
Use this skill when you need train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning). Supports SFT, DPO, GRPO, KTO, RLOO and Reward Model training via CLI commands.
You are an expert at using the TRL (Transformers Reinforcement Learning) library to train and fine-tune large language models.
Overview
TRL provides CLI commands for post-training foundation models using state-of-the-art techniques:
- SFT (Supervised Fine-Tuning): Fine-tune models on instruction-following or conversational datasets
- DPO (Direct Preference Optimization): Align models using preference data
- GRPO (Group Relative Policy Optimization): Train models by ranking multiple sampled outputs relative to each other and optimizing based on their comparative rewards.
- RLOO (Reinforce Leave One Out): Online RL training with generation-based rewards
- Reward Model Training: Train reward models for RLHF
TRL is built on top of Hugging Face Transformers and Accelerate, providing seamless integration with the Hugging Face ecosystem.
Core Commands
trl sft - Supervised Fine-Tuning
Fine-tune language models on instruction-following or conversational datasets.
Full training:
trl sft \
--model_name_or_path Qwen/Qwen2-0.5B \
--dataset_name trl-lib/Capybara \
--learning_rate 2.0e-5 \
--num_train_epochs 1 \
--packing \
--per_device_train_batch_size 2 \
--gradient_accumulation_steps 8 \
--eos_token '<|im_end|>' \
--eval_strategy steps \
--eval_steps 100 \
--output_dir Qwen2-0.5B-SFT \
--push_to_hub
Train with LoRA adapters:
trl sft \
--model_name_or_path Qwen/Qwen2-0.5B \
--dataset_name trl-lib/Capybara \
--learning_rate 2.0e-4 \
--num_train_epochs 1 \
--packing \
--per_device_train_batch_size 2 \
--gradient_accumulation_steps 8 \
--eos_token '<|im_end|>' \
--eval_strategy steps \
--eval_steps 100 \
--use_peft \
--lora_r 32 \
--lora_alpha 16 \
--output_dir Qwen2-0.5B-SFT \
--push_to_hub
trl dpo - Direct Preference Optimization
Align models using preference data (chosen/rejected pairs).
Full training:
trl dpo \
--dataset_name trl-lib/ultrafeedback_binarized \
--model_name_or_path Qwen/Qwen2-0.5B-Instruct \
--learning_rate 5.0e-7 \
--num_train_epochs 1 \
--per_device_train_batch_size 2 \
--max_steps 1000 \
--gradient_accumulation_steps 8 \
--eval_strategy steps \
--eval_steps 50 \
--output_dir Qwen2-0.5B-DPO \
--no_remove_unused_columns
Train with LoRA adapters:
trl dpo \
--dataset_name trl-lib/ultrafeedback_binarized \
--model_name_or_path Qwen/Qwen2-0.5B-Instruct \
--learning_rate 5.0e-6 \
--num_train_epochs 1 \
--per_device_train_batch_size 2 \
--max_steps 1000 \
--gradient_accumulation_steps 8 \
--eval_strategy steps \
--eval_steps 50 \
--output_dir Qwen2-0.5B-DPO \
--no_remove_unused_columns \
--use_peft \
--lora_r 32 \
--lora_alpha 16
trl grpo - Group Relative Policy Optimization
Train models using reward functions or LLM-as-a-judge for evaluating generations and providing rewards.
Basic usage:
trl grpo \
--model_name_or_path Qwen/Qwen2.5-0.5B \
--dataset_name trl-lib/gsm8k \
--reward_funcs accuracy_reward \
--output_dir Qwen2-0.5B-GRPO \
--push_to_hub
trl rloo - Reinforce Leave One Out
Online RL training where the model generates text and receives rewards based on custom criteria.
Basic usage:
trl rloo \
--model_name_or_path Qwen/Qwen2.5-0.5B \
--dataset_name trl-lib/tldr \
--reward_model_name_or_path sentiment-analysis:nlptown/bert-base-multilingual-uncased-sentiment \
--output_dir Qwen2-0.5B-RLOO \
--push_to_hub
trl reward - Reward Model Training
Train a reward model to score text quality for RLHF.
Full training:
trl reward \
--model_name_or_path Qwen/Qwen2-0.5B-Instruct \
--dataset_name trl-lib/ultrafeedback_binarized \
--output_dir Qwen2-0.5B-Reward \
--per_device_train_batch_size 8 \
--num_train_epochs 1 \
--learning_rate 1.0e-5 \
--eval_strategy steps \
--eval_steps 50 \
--max_length 2048
Train with LoRA adapters:
trl reward \
--model_name_or_path Qwen/Qwen2-0.5B-Instruct \
--dataset_name trl-lib/ultrafeedback_binarized \
--output_dir Qwen2-0.5B-Reward-LoRA \
--per_device_train_batch_size 8 \
--num_train_epochs 1 \
--learning_rate 1.0e-4 \
--eval_strategy steps \
--eval_steps 50 \
--max_length 2048 \
--use_peft \
--lora_task_type SEQ_CLS \
--lora_r 32 \
--lora_alpha 16
Configuration Files
TRL supports YAML configuration files for reproducible training. All CLI arguments can be specified in a config file.
Example config (sft_config.yaml):
model_name_or_path: Qwen/Qwen2.5-0.5B
dataset_name: trl-lib/Capybara
learning_rate: 2.0e-5
num_train_epochs: 1
per_device_train_batch_size: 8
gradient_accumulation_steps: 2
output_dir: ./sft_output
use_peft: true
lora_r: 16
lora_alpha: 16
report_to: trackio
Launch with config:
trl sft --config sft_config.yaml
Override config values:
trl sft --config sft_config.yaml --learning_rate 1.0e-5
Distributed Training
TRL integrates with Accelerate for multi-GPU and multi-node training.
Multi-GPU training:
trl sft \
--config sft_config.yaml \
--num_processes 4
Use predefined Accelerate configs:
TRL provides predefined configs: single_gpu, multi_gpu, fsdp1, fsdp2, zero1, zero2, zero3
trl sft \
--config sft_config.yaml \
--accelerate_config zero2
Custom Accelerate config:
# Generate custom config
accelerate config
# Use custom config
trl sft --config sft_config.yaml --config_file ~/.cache/huggingface/accelerate/default_config.yaml
Fully Sharded Data Parallel (FSDP):
trl sft --config sft_config.yaml --accelerate_config fsdp2
DeepSpeed ZeRO:
trl sft --config sft_config.yaml --accelerate_config zero3
Troubleshooting
CUDA Out of Memory
- Reduce
--per_device_train_batch_sizeand increase--gradient_accumulation_steps - Enable
--use_peftfor LoRA training - Use
--gradient_checkpointingto save memory - Try smaller model or longer sequence truncation
Dataset Loading Issues
- Verify dataset exists: check Hugging Face Hub or local path
- Check dataset format matches expected columns
- Use
--dataset_configfor multi-config datasets - Inspect dataset:
from datasets import load_dataset; ds = load_dataset(name)
Model Loading Issues
- Verify model exists on Hugging Face Hub
- Check if gated model requires authentication:
hf auth login - For local models, provide absolute path
- Ensure sufficient disk space and memory
Slow Training
- Enable dataset
--packingfor short sequences - Use larger
--per_device_train_batch_sizeif memory allows - Enable
--tf32for faster computation on Ampere GPUs - Use
--bf16on supported hardware - Consider multi-GPU training with
--num_processes
Generation Issues (GRPO/RLOO)
- Check prompt format in dataset
- Adjust
--temperatureand--top_pfor generation - Verify the reward function (for GRPO/RLOO)
Additional Resources
- Documentation: https://huggingface.co/docs/trl
- GitHub: https://github.com/huggingface/trl
- Examples: https://github.com/huggingface/trl/tree/main/examples
Best Practices
- Start with SFT: Always fine-tune base models with SFT before preference alignment
- Use LoRA for efficiency: Enable
--use_peftfor faster training and lower memory - Monitor training: Use
--report_to trackio(or--report_to wandbor--report_to tensorboard) for tracking - Save checkpoints: TRL automatically saves checkpoints in
--output_dir - Test on small datasets first: Verify pipeline works before full training
- Use configuration files: Create YAML configs for reproducibility
- Leverage Accelerate: Use multi-GPU training for faster iteration
When helping users with TRL:
- Always check which training method is appropriate for their use case
- Verify dataset format matches the expected schema
- Recommend starting with smaller models for testing
- Suggest LoRA for resource-constrained environments
- Point to specific documentation sections for advanced features
Limitations
- Use this skill only when the task clearly matches its upstream product or API scope.
- Verify commands, API behavior, pricing, quotas, credentials, and deployment effects against current official documentation before making changes.
- Do not treat generated examples as a substitute for environment-specific tests, security review, or user approval for destructive or costly actions.
04 同仓库更多 Skills
testing-best-practices
Laravel test design and review. Use when selecting coverage, naming or structuring tests, choosing assertions or test data, isolating dependencies, testing HTTP or security boundaries, improving suite performance, or reviewing test value. Use framework guidance or search-docs for Pest and PHPUnit syntax.
tailwindcss-development
Always invoke when the user's message includes 'tailwind' in any form. Also invoke for: building responsive grid layouts (multi-column card grids, product grids), flex/grid page structures (dashboards with sidebars, fixed topbars, mobile-toggle navs), styling UI components (cards, tables, navbars, pricing sections, forms, inputs, badges), adding dark mode variants, fixing spacing or typography, and Tailwind v3/v4 work. The core use case: writing or fixing Tailwind utility classes in HTML templates (Blade, JSX, Vue). Skip for backend PHP logic, database queries, API routes, JavaScript with no HTML/CSS component, CSS file audits, build tool configuration, and vanilla CSS.
laravel-best-practices
Apply this skill whenever writing, reviewing, or refactoring Laravel PHP code. This includes creating or modifying controllers, models, migrations, form requests, policies, jobs, scheduled commands, service classes, and Eloquent queries. Triggers for N+1 and query performance issues, caching strategies, authorization and security patterns, validation, error handling, queue and job configuration, route definitions, and architectural decisions. Also use for Laravel code reviews and refactoring existing Laravel code to follow best practices. Covers any task involving Laravel backend PHP code patterns.
infer-conventions
Use this skill to analyze how a Laravel application is actually written and record its conventions as shared rules. Trigger when the user wants to detect, infer, document, or standardize project conventions or coding style, set up or grow `.ai/rules`, resolve mixed or conflicting patterns (e.g. "are we using Form Requests or inline validation?"), or onboard agents and teammates to "how we do things here". Covers: a systematic sweep of ~49 Laravel convention dimensions (validation, models, architecture, testing, frontend, database, console), open-ended house-pattern discovery, conflict reporting, and recording rules scoped to the right paths via the Boost `record-rule` MCP tool. Only run this skill when the user explicitly asks for it; never start a sweep as part of another task. Do not use for one-off code review, enforcing formatting a linter already handles, or editing `.ai/rules` files by hand.
inertia-react-development
Develops Inertia.js v2 React client-side applications. Activates when creating React pages, forms, or navigation; using <Link>, <Form>, useForm, or router; working with deferred props, prefetching, or polling; or when user mentions React with Inertia, React pages, React forms, or React navigation.
deploying-to-cloud
Deploys and manages Laravel applications on Laravel Cloud using the `cloud` CLI. Use when the user wants to deploy an app, ship to Cloud, create or manage applications, environments, databases, caches, object storage, queues, domains, instances, background processes, secrets, compute, scheduled tasks, or Laravel Cloud infrastructure; or check billing, usage, or spend. Triggers on deployment, shipping, Cloud management, environment setup, database provisioning, billing or usage queries, the `cloud` CLI, and Laravel Cloud deployment troubleshooting.