pytorch-pro
PyTorch guidance — tensors, autograd, training loops, DataLoader, distributed training, and production inference.
- License 许可
- MIT
- 最近更新
- 2026-11-07
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- pytorch-pro
- category
- development
- description
- PyTorch guidance — tensors, autograd, training loops, DataLoader, distributed training, and production inference.
03 SKILL.md 正文
Overview
PyTorch is the dominant framework for deep learning research and increasingly for production: dynamic computation graphs, Pythonic APIs, and an ecosystem (torchvision, torchaudio, Hugging Face, Lightning) that covers most use cases. Its imperative style makes debugging intuitive — you can print tensors mid-forward-pass like any Python code.
This skill covers the PyTorch fundamentals that transfer everywhere: tensors and autograd, correct training loops, efficient data loading, distributed training, and the path to production inference (TorchScript, ONNX, TensorRT, torch.compile).
When to use
- Building or debugging PyTorch models.
- Writing correct training loops (train/eval modes, gradient handling).
- Optimizing data loading (DataLoader bottlenecks).
- Training on multiple GPUs (DDP).
- Exporting models for production inference.
- Choosing between PyTorch, TensorFlow, and JAX.
Core concepts
- Tensors. The core data structure:
torch.tensor, shapes/dtypes/devices. Think in shapes — most bugs are shape mismatches; print shapes liberally while developing.devicediscipline (.to(device)) avoids CPU/GPU mixing errors. - Autograd. Automatic differentiation: operations on
requires_grad=Truetensors build a graph;loss.backward()computes gradients.torch.no_grad()for inference (saves memory, disables graph);detach()to cut graph edges. - nn.Module. The building block:
__init__defines layers,forwarddefines computation.model.train()vsmodel.eval()— dropout/BatchNorm behave differently; forgettingeval()at inference is a classic silent bug. - Training loop anatomy. Zero grads → forward → loss → backward → optimizer step.
optimizer.zero_grad()placement matters (gradients accumulate by default — a feature for accumulation, a bug when forgotten). - DataLoader.
Dataset+DataLoader:num_workersfor parallel loading,pin_memoryfor GPU transfer speed, customcollate_fnfor variable-length data. Data loading is the usual bottleneck — profile it (nvidia-smishowing idle GPU = starved pipeline). - Losses and optimizers. CrossEntropyLoss (classification), MSE/BCE variants; Adam/AdamW defaults, SGD with momentum for some vision tasks; learning-rate schedulers (cosine, OneCycle, ReduceLROnPlateau). AdamW + cosine is the strong default.
- Mixed precision.
torch.cuda.amp(autocast + GradScaler): ~2x speedup, half memory, with loss scaling for stability. Nearly free performance — use it unless you have a reason not to. - Gradient clipping/accumulation. Clip (
clip_grad_norm_) for RNNs/transformers stability; accumulation (loss / k, step every k batches) for effective large batches on limited VRAM. - Checkpointing. Save
model.state_dict()+optimizer.state_dict()+ epoch + scheduler + RNG states — resumable training needs all of these, not just weights. Save best + latest. - Distributed (DDP).
DistributedDataParallel— one process per GPU, gradients averaged.torchrunlauncher;DistributedSamplerfor the DataLoader; sync BatchNorm where needed. DDP (not DataParallel) is the correct multi-GPU approach. - torch.compile. The 2.0+ compiler: often 30%+ speedup with one line (
torch.compile(model)). Graph breaks from data-dependent control flow reduce gains — write compile-friendly code (avoid Python branching on tensor values in hot paths). - Inference optimization.
model.eval()+torch.no_grad()/inference_mode(); export paths: TorchScript (C++ runtime), ONNX (cross-framework), TensorRT (NVIDIA GPUs, max perf), quantization (INT8 for CPU/latency). Choose by deployment target. - Debugging. NaN hunting (check loss per batch,
torch.autograd.set_detect_anomaly(True)for the guilty op), shape printing, overfitting a single batch as a smoke test (if it can't overfit 10 samples, something's broken). - Ecosystem. torchvision/torchaudio/huggingface for pretrained models; PyTorch Lightning for training-loop structure;
torchmetricsfor correct metric computation.
Practical workflow
- Write the correct loop. Train/eval modes, zero grads, no-grad inference — the template:
for epoch in range(epochs): model.train() for x, y in train_loader: x, y = x.to(device), y.to(device) optimizer.zero_grad() with torch.autocast(device_type="cuda", dtype=torch.float16): loss = criterion(model(x), y) scaler.scale(loss).backward() scaler.unscale_(optimizer) torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) scaler.step(optimizer); scaler.update() # validation model.eval() with torch.inference_mode(): val_loss = sum(criterion(model(x.to(device)), y.to(device)) for x, y in val_loader) / len(val_loader) - Smoke-test first. Overfit a single batch — if loss doesn't → ~0, debug before full training (LR, data, labels, architecture).
- Fix data loading.
num_workers=4+,pin_memory=True, profile GPU utilization; move preprocessing (tokenization, augmentation) into the pipeline, not the training step. - Scale with DDP.
torchrun --nproc_per_node=4 train.py; DDP wrap; DistributedSampler; scale LR with batch size (linear scaling rule as starting point). - Checkpoint completely. state_dicts for model+optimizer+scheduler, epoch, RNG states; save best-by-val and latest; test resume once.
- Compile and export.
torch.compilefor training/inference speed; export to ONNX/TensorRT/quantized for deployment targets; benchmark latency/throughput per export. - Monitor training. Loss curves, LR schedule, gradient norms, GPU util/memory — W&B/MLflow from run one. Diverging loss? Check LR, data normalization, and label correctness first.
- Serve efficiently. Batch inference requests,
inference_mode(), right-sized instances; the deployment format (ONNX/TensorRT) chosen by latency requirements.
Common pitfalls
- Forgetting
model.eval()— dropout/BatchNorm active at inference; silent quality drop. - Forgetting
zero_grad()— gradients accumulating unintentionally; loss behaves strangely. - No
torch.no_grad()at inference — graph building wasting memory;inference_mode()is stricter. - DataLoader bottleneck — GPU idle;
num_workers,pin_memory, pipeline preprocessing. - Using DataParallel — deprecated, slower; DDP with
torchrun. - NaNs unchecked — training diverging silently; anomaly detection + per-batch loss logging.
- Wrong device mixing — CPU tensor + CUDA tensor errors; consistent
.to(device). - Saving whole model instead of state_dict — pickle fragility;
state_dict+ architecture code. - Incomplete checkpoints — weights only, can't resume optimizer/scheduler; save everything.
- LR not scaled with batch size — DDP batch growth without LR adjustment; linear scaling starting point.
- Compile-unfriendly code — data-dependent Python branching killing
torch.compilegains; restructure hot paths. - No smoke test — full training on a broken setup; overfit-one-batch first.
- Metric bugs — wrong averaging across batches;
torchmetricsor careful accumulation.
04 同仓库更多 Skills
web-scrape
Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open/fetch fails on JS-heavy pages, or when you need clean markdown/structured extraction from a site. Requires FIRECRAWL_API_KEY.
design-taste-frontend
tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML/CSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, "AI-made", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.
test-driven-development
Use when implementing any feature or bugfix, before writing implementation code
systematic-debugging
Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes
requesting-code-review
Use when completing tasks, implementing major features, or before merging to verify work meets requirements
brainstorming
You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).