training-and-integrations
Use PEFT with Transformers Trainer, Accelerate, FSDP, DeepSpeed, TRL SFT, Diffusers training examples, memory-efficient training, torch.compile, and backend-specific training caveats.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- training-and-integrations
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Use PEFT with Transformers Trainer, Accelerate, FSDP, DeepSpeed, TRL SFT, Diffusers training examples, memory-efficient training, torch.compile, and backend-specific training caveats.
- disable-model-invocation
- true
02 资源文件 4
- 参考文档references/distributed-and-backends.md6.5 KB
- 参考文档references/training-recipes.md7.0 KB
- 参考文档references/troubleshooting.md6.0 KB
- 脚本scripts/build_training_checklist.py5.4 KB
03 SKILL.md 正文
Training and Integrations
Use this sub-skill when the task is about fitting PEFT into a training workflow rather than choosing adapter method parameters. It covers Trainer, Seq2SeqTrainer, SFTTrainer, raw Accelerate, FSDP, DeepSpeed, Diffusers DreamBooth/ControlNet-style scripts, memory-reduction choices, quantized training backends, and adapting PEFT example workflows safely.
Route Here For
- Converting a single-GPU PEFT training script to
accelerate launch, FSDP, DeepSpeed, or multi-GPU SFT. - Deciding where PEFT wrapping belongs in
Trainer,SFTTrainer, custom training loops, or Diffusers pipelines. - Troubleshooting distributed save behavior, mixed precision gradient errors, gradient checkpointing/offload interactions, optional backend dependencies, or example scripts that drift with upstream APIs.
- Building a training readiness checklist without downloading data/models or running training.
- Deciding whether
torch.compile, quantized loading, CPU/GPU offload, or gradient checkpointing is safe for a PEFT workflow.
Route Elsewhere
- Core
LoraConfig,PeftConfig,get_peft_model, adapter lifecycle, target module discovery, andmodules_to_save: use../adapter-core/SKILL.md. - LoRA variants, QLoRA configuration, LoftQ, QALoRA, DoRA, quantization parameter choices, and trainable tokens: use
../lora-and-quantization/SKILL.md. - Prompt tuning, prefix tuning, p-tuning, and soft prompt method behavior: use
../prompt-and-soft-methods/SKILL.md. - Non-LoRA tuner families such as IA3, BOFT/OFT, LoHa/LoKr, VeRA, HRA, RandLoRA, and similar method-specific choices: use
../specialized-tuners/SKILL.md. - Saving, loading, merging, conversion, deployment packaging, or checkpoint format questions after training: use
../save-load-merge/SKILL.md.
Start Fast
-
Run the bundled no-training checklist builder from this sub-skill directory:
python scripts/build_training_checklist.py --workflow trainer --backend single-gpu python scripts/build_training_checklist.py --workflow trainer --backend fsdp --quantized python scripts/build_training_checklist.py --workflow diffusers --backend deepspeed -
Read
references/training-recipes.mdfor PEFT placement inTrainer,SFTTrainer,Accelerate, and Diffusers-style scripts. -
Read
references/distributed-and-backends.mdbefore changing FSDP, DeepSpeed, quantization, offload, ortorch.compilesettings. -
Read
references/troubleshooting.mdwhen failures mention FP16 gradient unscale, gradient checkpointing, optional packages, FSDP/DeepSpeed save behavior, stale examples, or backend-specific dtype limitations.
Core Workflow Pattern
- Prepare the base model/tokenizer/dataset exactly as the training framework expects, then apply PEFT before the trainer or accelerator prepares the model.
- For Transformers
Trainer, pass aPeftModelasmodel, or use a trainer integration such as TRLSFTTrainer(peft_config=...)when that trainer is responsible for wrapping. - For raw
Accelerate, build the PEFT model beforeaccelerator.prepare(...); after preparation, save adapters with the unwrapped model on the main process. - For FSDP, set PEFT-aware wrapping through
peft.utils.other.fsdp_auto_wrap_policy, keepfsdp_use_orig_params: falsewhen memory savings matter, and switch to a full state dict before final adapter save when using trainer-managed FSDP. - For DeepSpeed ZeRO-3, align
accelerateconfig and training arguments, keep gradient accumulation consistent across both, and use trainer/accelerator save helpers rather than rank-local saves. - For quantized PEFT training, route method/backend parameters to
lora-and-quantization, but keep integration checks here: installed optional backend,prepare_model_for_kbit_training, compatible gradient checkpointing mode, and compatible distributed backend.
Safety Rules For Adapting Examples
- Treat examples as patterns, not stable APIs: re-check current
transformers,accelerate,trl,diffusers,bitsandbytes, andtorchversions when adapting. - Do not run example training as a smoke test unless the user explicitly wants downloads, GPU time, and dataset access; use parser/help checks or dry configuration review first.
- Replace hard-coded model IDs, dataset names, hub push flags, precision flags, and output directories with user-controlled arguments.
- Preserve the final adapter save path and distributed save semantics when porting examples to FSDP or DeepSpeed.
- Keep PEFT adapter configuration decisions separate from launcher/backend decisions so changes are reviewable.
Bundled Materials
references/training-recipes.md: framework-specific recipes forTrainer,Accelerate, TRL SFT, Diffusers training, memory-efficient training, andtorch.compileplacement.references/distributed-and-backends.md: FSDP, DeepSpeed, quantization, offload, precision, and backend compatibility guidance.references/troubleshooting.md: diagnosis paths for stale examples, dtype unscale errors, checkpointing/offload issues, optional packages, distributed saves, and compile limitations.scripts/build_training_checklist.py: parser-only checklist generator fortrainer,accelerate, anddiffusersworkflows acrosssingle-gpu,fsdp,deepspeed, andcpubackends.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).