[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M73AD4J09M7DD746S0SFE8EP":3,"skill-same:01M73AD4J09M7DD746S0SFE8EP":87},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":39,"commitSha":40,"skillDirectory":41,"forksCount":42,"repositoryHtmlUrl":43,"repositoryHomepageUrl":6,"ownerHtmlUrl":44,"ownerBlogUrl":45,"ownerDescription":6,"updatedAt":46,"install":47,"resources":60},"01M73AD4J09M7DD746S0SFE8EP","ml-training-recipes",null,"Battle-tested PyTorch training recipes for all domains — LLMs, vision, diffusion, medical imaging, protein\u002Fdrug discovery, spatial omics, genomics. Covers training loops, optimizer selection (AdamW, Muon), LR scheduling, mixed precision, debugging, and systematic experimentation. Use when training or fine-tuning neural networks, debugging loss spikes or OOM, choosing architectures, or optimizing GPU throughput.","qhjqhj00\u002Fresearch-skills-pool","qhjqhj00","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F29684092?v=4","01M72AE0ZGXVM5C57N512DGQG8","Tommy Chien","01M72AE107ESJMHT7QC5SXX144",3,"MIT","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--10-optimization__ml-training-recipes\u002FSKILL.md",[],[],"2026-11-11 15:11:14",false,"---\r\nname: ml-training-recipes\r\ndescription: Battle-tested PyTorch training recipes for all domains — LLMs, vision, diffusion, medical imaging, protein\u002Fdrug discovery, spatial omics, genomics. Covers training loops, optimizer selection (AdamW, Muon), LR scheduling, mixed precision, debugging, and systematic experimentation. Use when training or fine-tuning neural networks, debugging loss spikes or OOM, choosing architectures, or optimizing GPU throughput.\r\nversion: 1.0.0\r\nauthor: dailycafi\r\nlicense: MIT\r\ntags: [PyTorch, Training, Optimization, LLM, Vision, Diffusion, Biomedical, Muon, AdamW, Debugging]\r\ndependencies: [torch>=2.0.0]\r\n---\r\n\r\n# ML Training Recipes\r\n\r\nBattle-tested patterns for PyTorch training across domains. Drawn from production codebases\r\n(Karpathy's autoresearch\u002Fnanochat, torchvision, HuggingFace) and modern training practice.\r\n\r\n## Reference files (read when needed)\r\n\r\n- `references\u002Farchitecture.md` — Transformer\u002FLLM architecture code patterns, weight init\r\n- `references\u002Foptimizers.md` — Muon, AdamW hybrid, per-group LR, compiled optimizer steps\r\n- `references\u002Fdomain-specific.md` — Vision, diffusion, contrastive, distributed, checkpointing, data loading\r\n- `references\u002Fscaling-and-selection.md` — Scaling laws, compute budget tables, decision trees, DGX Spark\r\n- `references\u002Fbiomedical.md` — Drug discovery, protein models, medical imaging, genomics, clinical NLP\r\n- `references\u002Fexperiment-loop.md` — Autonomous experiment loop (autoresearch keep\u002Fdiscard\u002Frevert)\r\n\r\n---\r\n\r\n## Architecture Selection\r\n\r\nPick the right model by **data type** and **data scale**:\r\n\r\n| Data Type | \u003C 10K samples | 10K-100K | > 100K |\r\n|-----------|--------------|----------|--------|\r\n| **Images** | Pretrained CNN + fine-tune | Fine-tune ViT or CNN | ViT from scratch |\r\n| **Text (gen)** | Few-shot prompting | Fine-tune GPT\u002FLLaMA (LoRA) | Pretrain from scratch |\r\n| **Tabular** | XGBoost\u002FLightGBM | Still XGBoost | Neural viable |\r\n| **Audio** | Pretrained Whisper | Fine-tune AST | Train from scratch |\r\n| **Molecules** | Pretrained GNN | Fine-tune molecular LM | Train GNN from scratch |\r\n| **Proteins** | ESM-2 embeddings + head | Fine-tune ESM-2 | Train protein LM |\r\n| **Medical img** | Pretrained CNN | nnU-Net (auto-config) | Swin-UNETR \u002F MedSAM |\r\n\r\n**Key principle**: architecture matters less than training recipe at equal compute. A well-tuned\r\nResNet beats a poorly-tuned ViT (ref: \"ResNet Strikes Back\", Wightman 2021).\r\n\r\nFor biomedical domains, see `references\u002Fbiomedical.md`.\r\nFor sequence model selection and compute planning, see `references\u002Fscaling-and-selection.md`.\r\n\r\n---\r\n\r\n## Scaling Laws\r\n\r\n### Chinchilla rule (Hoffmann et al., 2022)\r\n\r\nCompute-optimal training: **~20 tokens per parameter**.\r\n\r\n| Model Size | Compute-Optimal | Inference-Optimal (100×) |\r\n|-----------|----------------|--------------------------|\r\n| 125M | 2.5B tokens | 12.5B tokens |\r\n| 1B | 20B tokens | 100B tokens |\r\n| 7B | 140B tokens | 700B tokens |\r\n\r\n**FLOPs ≈ 6 × N × D** (N=params, D=tokens). Data repetition limit: ~4 epochs before diminishing returns.\r\n\r\n---\r\n\r\n## Training Loop\r\n\r\n```python\r\nimport gc, time, torch\r\n\r\ntorch.manual_seed(42)\r\ntorch.set_float32_matmul_precision(\"high\")  # TF32 on Ampere+\r\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\r\n\r\ngrad_accum_steps = total_batch_size \u002F\u002F (batch_size * seq_len)\r\nstep = 0\r\n\r\nwhile not done:\r\n    t0 = time.time()\r\n    for micro_step in range(grad_accum_steps):\r\n        with autocast_ctx:\r\n            loss = model(x, y)\r\n        (loss \u002F grad_accum_steps).backward()\r\n        x, y = next(train_loader)\r\n\r\n    update_lr(optimizer, progress)\r\n    optimizer.step()\r\n    model.zero_grad(set_to_none=True)  # frees memory vs zeroing\r\n\r\n    if loss.item() > 100:  # fast-fail on divergence\r\n        print(\"FAIL: loss exploded\"); exit(1)\r\n\r\n    torch.cuda.synchronize()\r\n    if step == 0:\r\n        gc.collect(); gc.freeze(); gc.disable()  # avoid ~500ms GC stalls\r\n    step += 1\r\n```\r\n\r\n### Key principles\r\n\r\n- **Gradient clipping**: `clip_grad_norm_(params, 1.0)` — near-universal for Transformers.\r\n  Exception: Muon optimizer normalizes updates via orthogonalization, so clipping is optional.\r\n- **Tensor Core alignment**: batch size, hidden dims should be multiples of 8 (bf16) or 64 (A100).\r\n- **Time-based budgets** make experiments comparable across hardware.\r\n- **`cudnn.benchmark = True`** for fixed-size vision inputs.\r\n\r\n---\r\n\r\n## Optimizer Configuration\r\n\r\nModern LLM training uses different optimizers per parameter group:\r\n\r\n| Parameter Type | Optimizer | LR (base) | Weight Decay |\r\n|---------------|-----------|-----------|--------------|\r\n| 2D weight matrices | Muon | 0.04 | 0.2 |\r\n| Token embeddings | AdamW | 0.6 × scale | 0.0 |\r\n| Unembedding (lm_head) | AdamW | 0.004 × scale | 0.0 |\r\n| Per-layer scalars | AdamW | 0.005 × scale | 0.0 |\r\n\r\n**LR scaling by dimension**: `lr * (d_model \u002F 768)^(-0.5)` — keeps dynamics stable across sizes.\r\n\r\n### Rules of thumb\r\n\r\n- Embeddings need higher LR (sparse updates). Never weight-decay embeddings.\r\n- Weight decay scheduling: linearly decay WD to 0 over training.\r\n- AdamW defaults: β1=0.9, β2=0.95, eps=1e-10 (not default 1e-8 — prevents stale updates in bf16).\r\n\r\nFor Muon details (polar express orthogonalization, NorMuon), see `references\u002Foptimizers.md`.\r\n\r\n---\r\n\r\n## Learning Rate Scheduling\r\n\r\n### Time-based (autoresearch style)\r\n\r\n```python\r\ndef get_lr_multiplier(progress):  # progress = elapsed_time \u002F time_budget\r\n    if progress \u003C warmup_ratio:\r\n        return progress \u002F warmup_ratio\r\n    elif progress \u003C 1.0 - warmdown_ratio:\r\n        return 1.0\r\n    else:\r\n        cooldown = (1.0 - progress) \u002F warmdown_ratio\r\n        return cooldown + (1 - cooldown) * final_lr_frac\r\n```\r\n\r\n### Cosine decay\r\n\r\n```python\r\ndef get_lr(step, total_steps, max_lr, min_lr, warmup_steps):\r\n    if step \u003C warmup_steps:\r\n        return max_lr * step \u002F warmup_steps\r\n    progress = (step - warmup_steps) \u002F (total_steps - warmup_steps)\r\n    return min_lr + 0.5 * (max_lr - min_lr) * (1 + math.cos(math.pi * progress))\r\n```\r\n\r\n**WSD (Warmup-Stable-Decay)**: gaining traction — easier to resume training mid-run.\r\n\r\n### Guidance\r\n\r\n- **Warmup**: 1-5% of training. Zero warmup valid with Muon (autoresearch uses `WARMUP_RATIO=0.0`).\r\n- **Warmdown**: 30-50% of training in LR decay. Matters more than warmup for final quality.\r\n- **Final LR**: 0 or ~10% of peak. Zero is simpler.\r\n\r\n---\r\n\r\n## Mixed Precision & Compilation\r\n\r\n```python\r\nimport os\r\nos.environ[\"PYTORCH_ALLOC_CONF\"] = \"expandable_segments:True\"  # before torch import\r\n\r\nimport torch\r\ntorch.set_float32_matmul_precision(\"high\")\r\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\r\nmodel = torch.compile(model, dynamic=False)\r\n```\r\n\r\n- **bf16** (Ampere+): same exponent as fp32, no loss scaling needed. Preferred over fp16.\r\n- **fp16**: needs GradScaler. Use only on V100 or older.\r\n- `dynamic=False` enables max optimization. Add `fullgraph=True` if no graph breaks.\r\n- First steps are slow (JIT) — exclude from timing.\r\n\r\n---\r\n\r\n## Memory & Performance\r\n\r\n### Meta device init (large models)\r\n\r\n```python\r\nwith torch.device(\"meta\"):\r\n    model = GPT(config)          # zero memory\r\nmodel.to_empty(device=\"cuda\")\r\nmodel.init_weights()\r\n```\r\n\r\n### MFU (Model FLOPs Utilization)\r\n\r\n```python\r\nachieved_flops = model_flops_per_token * batch_tokens \u002F step_time\r\nmfu = achieved_flops \u002F gpu_peak_flops\r\n# H100 SXM: 989.5 TFLOPS | A100: 312 | RTX 4090: 165\r\n```\r\n\r\nGood targets: >30% decent, >40% good, >50% excellent (single-GPU).\r\n\r\n### OOM solutions (in order)\r\n\r\n1. Reduce `DEVICE_BATCH_SIZE`, increase `grad_accum_steps`\r\n2. `PYTORCH_ALLOC_CONF=expandable_segments:True`\r\n3. `model.zero_grad(set_to_none=True)`\r\n4. Meta device init → `to_empty`\r\n5. Activation checkpointing: `torch.utils.checkpoint.checkpoint()`\r\n6. 8-bit optimizer (bitsandbytes): ~30% savings on optimizer states\r\n\r\n---\r\n\r\n## Hyperparameter Search\r\n\r\n### Priority order (tune first → last)\r\n\r\n1. **Learning rate** — most impactful. Always tune first.\r\n2. **Batch size** — largest that fits. Speed knob, not quality knob.\r\n3. **Weight decay** — 0.01-0.1 for AdamW.\r\n4. **Warmup steps** — 1-5% of training.\r\n\r\n### The 2025 default recipe\r\n\r\n| Setting | Value |\r\n|---------|-------|\r\n| Optimizer | AdamW (β1=0.9, β2=0.95, eps=1e-10) |\r\n| Weight decay | 0.1 |\r\n| LR schedule | Cosine decay or WSD |\r\n| Peak LR | 3e-4 (scale down for larger models) |\r\n| Precision | bf16 |\r\n| Grad clipping | max_norm=1.0 |\r\n| Normalization | RMSNorm (pre-norm) |\r\n| Activation | SwiGLU |\r\n| Position encoding | RoPE |\r\n| Attention | Flash Attention, optionally GQA |\r\n\r\n---\r\n\r\n## Debugging Checklist\r\n\r\n### Karpathy's recipe (still canonical)\r\n\r\n1. **Become one with the data** — visualize, check distributions, verify labels\r\n2. **Get end-to-end running first** — verify on a trivial case\r\n3. **Overfit one batch** — if you can't, you have a bug\r\n4. **Then regularize** — add regularization only after overfitting works\r\n5. **Tune hyperparameters** — start with known defaults\r\n\r\n### Loss exploding \u002F NaN\r\n\r\n1. Reduce LR (3-10× smaller)\r\n2. Add gradient clipping: `clip_grad_norm_(params, 1.0)`\r\n3. Check for inf\u002Fnan in inputs\r\n4. Add logit soft capping: `softcap * tanh(logits \u002F softcap)`\r\n5. Add QK-norm in attention\r\n6. Verify weight init (zero-init output projections?)\r\n7. Check loss reduction with gradient accumulation (`loss \u002F grad_accum_steps`)\r\n\r\n### Slow training \u002F Low MFU\r\n\r\n1. Verify `torch.compile` is active\r\n2. Check `torch.set_float32_matmul_precision(\"high\")`\r\n3. Pin memory + non_blocking transfers\r\n4. Profile with `torch.profiler`\r\n5. GC stalls? `gc.freeze(); gc.disable()`\r\n6. Tensor Core alignment: dims multiples of 8\u002F64\r\n\r\n### Loss plateau \u002F Slow convergence\r\n\r\n1. LR too low — try 2-5× larger\r\n2. Warmup too long\r\n3. Weight decay too high\r\n4. Verify LR schedule is actually applied (print each step)\r\n5. Model too small for task\r\n\r\n### Silent failures\r\n\r\n1. **Data leakage** between train\u002Fval\r\n2. **Wrong preprocessing at inference** — augmentation mismatch\r\n3. **Label errors** — use cleanlab to detect\r\n4. **Shuffling bugs** — correlated batches\r\n5. **Tokenizer mismatch** with pretrained model\r\n\r\n### What to monitor\r\n\r\n- **Gradient norms** — spike precedes loss spike\r\n- **Per-layer activation stats** — reveals exploding\u002Fvanishing\r\n- **Dead neurons** — >50% zero ReLU = dying ReLU problem\r\n- **Learning rate** — verify schedule applied (common silent bug)\r\n\r\n---\r\n\r\n## Experiment Management\r\n\r\nTrack experiments in TSV for easy comparison:\r\n\r\n```\r\ncommit  val_bpb  memory_gb  status   description\r\na1b2c3d 0.9979   44.0       keep     baseline\r\nb2c3d4e 0.9932   44.2       keep     increase matrix LR to 0.04\r\nc3d4e5f 1.0050   44.0       discard  switch to GeLU (worse)\r\n```\r\n\r\n**Simplicity criterion**: all else equal, simpler is better. Removing something and getting equal\r\nresults is a great outcome. For systematic agent-driven experimentation, see `references\u002Fexperiment-loop.md`.\r\n\r\n### Evaluation metrics by domain\r\n\r\n| Domain | Primary Metric | Notes |\r\n|--------|---------------|-------|\r\n| LLM | BPB (bits per byte) | Vocab-size-independent |\r\n| Classification | Accuracy \u002F F1 | Macro-F1 for imbalanced |\r\n| Segmentation | mIoU \u002F Dice | Per-class IoU reveals weak spots |\r\n| Generation | FID | Needs >10k samples |\r\n| Regression | RMSE \u002F MAE | Log-transform skewed targets |\r\n","# ML Training Recipes\r\n\r\nBattle-tested patterns for PyTorch training across domains. Drawn from production codebases\r\n(Karpathy's autoresearch\u002Fnanochat, torchvision, HuggingFace) and modern training practice.\r\n\r\n## Reference files (read when needed)\r\n\r\n- `references\u002Farchitecture.md` — Transformer\u002FLLM architecture code patterns, weight init\r\n- `references\u002Foptimizers.md` — Muon, AdamW hybrid, per-group LR, compiled optimizer steps\r\n- `references\u002Fdomain-specific.md` — Vision, diffusion, contrastive, distributed, checkpointing, data loading\r\n- `references\u002Fscaling-and-selection.md` — Scaling laws, compute budget tables, decision trees, DGX Spark\r\n- `references\u002Fbiomedical.md` — Drug discovery, protein models, medical imaging, genomics, clinical NLP\r\n- `references\u002Fexperiment-loop.md` — Autonomous experiment loop (autoresearch keep\u002Fdiscard\u002Frevert)\r\n\r\n---\r\n\r\n## Architecture Selection\r\n\r\nPick the right model by **data type** and **data scale**:\r\n\r\n| Data Type | \u003C 10K samples | 10K-100K | > 100K |\r\n|-----------|--------------|----------|--------|\r\n| **Images** | Pretrained CNN + fine-tune | Fine-tune ViT or CNN | ViT from scratch |\r\n| **Text (gen)** | Few-shot prompting | Fine-tune GPT\u002FLLaMA (LoRA) | Pretrain from scratch |\r\n| **Tabular** | XGBoost\u002FLightGBM | Still XGBoost | Neural viable |\r\n| **Audio** | Pretrained Whisper | Fine-tune AST | Train from scratch |\r\n| **Molecules** | Pretrained GNN | Fine-tune molecular LM | Train GNN from scratch |\r\n| **Proteins** | ESM-2 embeddings + head | Fine-tune ESM-2 | Train protein LM |\r\n| **Medical img** | Pretrained CNN | nnU-Net (auto-config) | Swin-UNETR \u002F MedSAM |\r\n\r\n**Key principle**: architecture matters less than training recipe at equal compute. A well-tuned\r\nResNet beats a poorly-tuned ViT (ref: \"ResNet Strikes Back\", Wightman 2021).\r\n\r\nFor biomedical domains, see `references\u002Fbiomedical.md`.\r\nFor sequence model selection and compute planning, see `references\u002Fscaling-and-selection.md`.\r\n\r\n---\r\n\r\n## Scaling Laws\r\n\r\n### Chinchilla rule (Hoffmann et al., 2022)\r\n\r\nCompute-optimal training: **~20 tokens per parameter**.\r\n\r\n| Model Size | Compute-Optimal | Inference-Optimal (100×) |\r\n|-----------|----------------|--------------------------|\r\n| 125M | 2.5B tokens | 12.5B tokens |\r\n| 1B | 20B tokens | 100B tokens |\r\n| 7B | 140B tokens | 700B tokens |\r\n\r\n**FLOPs ≈ 6 × N × D** (N=params, D=tokens). Data repetition limit: ~4 epochs before diminishing returns.\r\n\r\n---\r\n\r\n## Training Loop\r\n\r\n```python\r\nimport gc, time, torch\r\n\r\ntorch.manual_seed(42)\r\ntorch.set_float32_matmul_precision(\"high\")  # TF32 on Ampere+\r\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\r\n\r\ngrad_accum_steps = total_batch_size \u002F\u002F (batch_size * seq_len)\r\nstep = 0\r\n\r\nwhile not done:\r\n    t0 = time.time()\r\n    for micro_step in range(grad_accum_steps):\r\n        with autocast_ctx:\r\n            loss = model(x, y)\r\n        (loss \u002F grad_accum_steps).backward()\r\n        x, y = next(train_loader)\r\n\r\n    update_lr(optimizer, progress)\r\n    optimizer.step()\r\n    model.zero_grad(set_to_none=True)  # frees memory vs zeroing\r\n\r\n    if loss.item() > 100:  # fast-fail on divergence\r\n        print(\"FAIL: loss exploded\"); exit(1)\r\n\r\n    torch.cuda.synchronize()\r\n    if step == 0:\r\n        gc.collect(); gc.freeze(); gc.disable()  # avoid ~500ms GC stalls\r\n    step += 1\r\n```\r\n\r\n### Key principles\r\n\r\n- **Gradient clipping**: `clip_grad_norm_(params, 1.0)` — near-universal for Transformers.\r\n  Exception: Muon optimizer normalizes updates via orthogonalization, so clipping is optional.\r\n- **Tensor Core alignment**: batch size, hidden dims should be multiples of 8 (bf16) or 64 (A100).\r\n- **Time-based budgets** make experiments comparable across hardware.\r\n- **`cudnn.benchmark = True`** for fixed-size vision inputs.\r\n\r\n---\r\n\r\n## Optimizer Configuration\r\n\r\nModern LLM training uses different optimizers per parameter group:\r\n\r\n| Parameter Type | Optimizer | LR (base) | Weight Decay |\r\n|---------------|-----------|-----------|--------------|\r\n| 2D weight matrices | Muon | 0.04 | 0.2 |\r\n| Token embeddings | AdamW | 0.6 × scale | 0.0 |\r\n| Unembedding (lm_head) | AdamW | 0.004 × scale | 0.0 |\r\n| Per-layer scalars | AdamW | 0.005 × scale | 0.0 |\r\n\r\n**LR scaling by dimension**: `lr * (d_model \u002F 768)^(-0.5)` — keeps dynamics stable across sizes.\r\n\r\n### Rules of thumb\r\n\r\n- Embeddings need higher LR (sparse updates). Never weight-decay embeddings.\r\n- Weight decay scheduling: linearly decay WD to 0 over training.\r\n- AdamW defaults: β1=0.9, β2=0.95, eps=1e-10 (not default 1e-8 — prevents stale updates in bf16).\r\n\r\nFor Muon details (polar express orthogonalization, NorMuon), see `references\u002Foptimizers.md`.\r\n\r\n---\r\n\r\n## Learning Rate Scheduling\r\n\r\n### Time-based (autoresearch style)\r\n\r\n```python\r\ndef get_lr_multiplier(progress):  # progress = elapsed_time \u002F time_budget\r\n    if progress \u003C warmup_ratio:\r\n        return progress \u002F warmup_ratio\r\n    elif progress \u003C 1.0 - warmdown_ratio:\r\n        return 1.0\r\n    else:\r\n        cooldown = (1.0 - progress) \u002F warmdown_ratio\r\n        return cooldown + (1 - cooldown) * final_lr_frac\r\n```\r\n\r\n### Cosine decay\r\n\r\n```python\r\ndef get_lr(step, total_steps, max_lr, min_lr, warmup_steps):\r\n    if step \u003C warmup_steps:\r\n        return max_lr * step \u002F warmup_steps\r\n    progress = (step - warmup_steps) \u002F (total_steps - warmup_steps)\r\n    return min_lr + 0.5 * (max_lr - min_lr) * (1 + math.cos(math.pi * progress))\r\n```\r\n\r\n**WSD (Warmup-Stable-Decay)**: gaining traction — easier to resume training mid-run.\r\n\r\n### Guidance\r\n\r\n- **Warmup**: 1-5% of training. Zero warmup valid with Muon (autoresearch uses `WARMUP_RATIO=0.0`).\r\n- **Warmdown**: 30-50% of training in LR decay. Matters more than warmup for final quality.\r\n- **Final LR**: 0 or ~10% of peak. Zero is simpler.\r\n\r\n---\r\n\r\n## Mixed Precision & Compilation\r\n\r\n```python\r\nimport os\r\nos.environ[\"PYTORCH_ALLOC_CONF\"] = \"expandable_segments:True\"  # before torch import\r\n\r\nimport torch\r\ntorch.set_float32_matmul_precision(\"high\")\r\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\r\nmodel = torch.compile(model, dynamic=False)\r\n```\r\n\r\n- **bf16** (Ampere+): same exponent as fp32, no loss scaling needed. Preferred over fp16.\r\n- **fp16**: needs GradScaler. Use only on V100 or older.\r\n- `dynamic=False` enables max optimization. Add `fullgraph=True` if no graph breaks.\r\n- First steps are slow (JIT) — exclude from timing.\r\n\r\n---\r\n\r\n## Memory & Performance\r\n\r\n### Meta device init (large models)\r\n\r\n```python\r\nwith torch.device(\"meta\"):\r\n    model = GPT(config)          # zero memory\r\nmodel.to_empty(device=\"cuda\")\r\nmodel.init_weights()\r\n```\r\n\r\n### MFU (Model FLOPs Utilization)\r\n\r\n```python\r\nachieved_flops = model_flops_per_token * batch_tokens \u002F step_time\r\nmfu = achieved_flops \u002F gpu_peak_flops\r\n# H100 SXM: 989.5 TFLOPS | A100: 312 | RTX 4090: 165\r\n```\r\n\r\nGood targets: >30% decent, >40% good, >50% excellent (single-GPU).\r\n\r\n### OOM solutions (in order)\r\n\r\n1. Reduce `DEVICE_BATCH_SIZE`, increase `grad_accum_steps`\r\n2. `PYTORCH_ALLOC_CONF=expandable_segments:True`\r\n3. `model.zero_grad(set_to_none=True)`\r\n4. Meta device init → `to_empty`\r\n5. Activation checkpointing: `torch.utils.checkpoint.checkpoint()`\r\n6. 8-bit optimizer (bitsandbytes): ~30% savings on optimizer states\r\n\r\n---\r\n\r\n## Hyperparameter Search\r\n\r\n### Priority order (tune first → last)\r\n\r\n1. **Learning rate** — most impactful. Always tune first.\r\n2. **Batch size** — largest that fits. Speed knob, not quality knob.\r\n3. **Weight decay** — 0.01-0.1 for AdamW.\r\n4. **Warmup steps** — 1-5% of training.\r\n\r\n### The 2025 default recipe\r\n\r\n| Setting | Value |\r\n|---------|-------|\r\n| Optimizer | AdamW (β1=0.9, β2=0.95, eps=1e-10) |\r\n| Weight decay | 0.1 |\r\n| LR schedule | Cosine decay or WSD |\r\n| Peak LR | 3e-4 (scale down for larger models) |\r\n| Precision | bf16 |\r\n| Grad clipping | max_norm=1.0 |\r\n| Normalization | RMSNorm (pre-norm) |\r\n| Activation | SwiGLU |\r\n| Position encoding | RoPE |\r\n| Attention | Flash Attention, optionally GQA |\r\n\r\n---\r\n\r\n## Debugging Checklist\r\n\r\n### Karpathy's recipe (still canonical)\r\n\r\n1. **Become one with the data** — visualize, check distributions, verify labels\r\n2. **Get end-to-end running first** — verify on a trivial case\r\n3. **Overfit one batch** — if you can't, you have a bug\r\n4. **Then regularize** — add regularization only after overfitting works\r\n5. **Tune hyperparameters** — start with known defaults\r\n\r\n### Loss exploding \u002F NaN\r\n\r\n1. Reduce LR (3-10× smaller)\r\n2. Add gradient clipping: `clip_grad_norm_(params, 1.0)`\r\n3. Check for inf\u002Fnan in inputs\r\n4. Add logit soft capping: `softcap * tanh(logits \u002F softcap)`\r\n5. Add QK-norm in attention\r\n6. Verify weight init (zero-init output projections?)\r\n7. Check loss reduction with gradient accumulation (`loss \u002F grad_accum_steps`)\r\n\r\n### Slow training \u002F Low MFU\r\n\r\n1. Verify `torch.compile` is active\r\n2. Check `torch.set_float32_matmul_precision(\"high\")`\r\n3. Pin memory + non_blocking transfers\r\n4. Profile with `torch.profiler`\r\n5. GC stalls? `gc.freeze(); gc.disable()`\r\n6. Tensor Core alignment: dims multiples of 8\u002F64\r\n\r\n### Loss plateau \u002F Slow convergence\r\n\r\n1. LR too low — try 2-5× larger\r\n2. Warmup too long\r\n3. Weight decay too high\r\n4. Verify LR schedule is actually applied (print each step)\r\n5. Model too small for task\r\n\r\n### Silent failures\r\n\r\n1. **Data leakage** between train\u002Fval\r\n2. **Wrong preprocessing at inference** — augmentation mismatch\r\n3. **Label errors** — use cleanlab to detect\r\n4. **Shuffling bugs** — correlated batches\r\n5. **Tokenizer mismatch** with pretrained model\r\n\r\n### What to monitor\r\n\r\n- **Gradient norms** — spike precedes loss spike\r\n- **Per-layer activation stats** — reveals exploding\u002Fvanishing\r\n- **Dead neurons** — >50% zero ReLU = dying ReLU problem\r\n- **Learning rate** — verify schedule applied (common silent bug)\r\n\r\n---\r\n\r\n## Experiment Management\r\n\r\nTrack experiments in TSV for easy comparison:\r\n\r\n```\r\ncommit  val_bpb  memory_gb  status   description\r\na1b2c3d 0.9979   44.0       keep     baseline\r\nb2c3d4e 0.9932   44.2       keep     increase matrix LR to 0.04\r\nc3d4e5f 1.0050   44.0       discard  switch to GeLU (worse)\r\n```\r\n\r\n**Simplicity criterion**: all else equal, simpler is better. Removing something and getting equal\r\nresults is a great outcome. For systematic agent-driven experimentation, see `references\u002Fexperiment-loop.md`.\r\n\r\n### Evaluation metrics by domain\r\n\r\n| Domain | Primary Metric | Notes |\r\n|--------|---------------|-------|\r\n| LLM | BPB (bits per byte) | Vocab-size-independent |\r\n| Classification | Accuracy \u002F F1 | Macro-F1 for imbalanced |\r\n| Segmentation | mIoU \u002F Dice | Per-class IoU reveals weak spots |\r\n| Generation | FID | Needs >10k samples |\r\n| Regression | RMSE \u002F MAE | Log-transform skewed targets |\r\n",{"name":5,"tags":24,"author":35,"license":15,"version":36,"description":7,"dependencies":37},[25,26,27,28,29,30,31,32,33,34],"PyTorch","Training","Optimization","LLM","Vision","Diffusion","Biomedical","Muon","AdamW","Debugging","dailycafi","1.0.0",[38],"torch>=2.0.0","master","c247302c836de6438d746b45670990eb71691166","skills_collected\u002FOrchestra-Research--AI-research-SKILLs--10-optimization__ml-training-recipes",1,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool","https:\u002F\u002Fgithub.com\u002Fqhjqhj00","chien.io","2026-05-11 15:07:59",{"source":8,"skillName":5,"commands":48,"prompt":58,"zipAvailable":59},[49,52,55],{"tool":50,"command":51},"npx","npx skills add qhjqhj00\u002Fresearch-skills-pool --skill ml-training-recipes",{"tool":53,"command":54},"bunx","bunx skills add qhjqhj00\u002Fresearch-skills-pool --skill ml-training-recipes",{"tool":56,"command":57},"pnpm","pnpm dlx skills add qhjqhj00\u002Fresearch-skills-pool --skill ml-training-recipes","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--10-optimization__ml-training-recipes\u002FSKILL.md",true,[61,67,71,75,79,83],{"path":62,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":65,"sourceHtmlUrl":66},"references\u002Farchitecture.md","REFERENCE","参考文档",10523,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--10-optimization__ml-training-recipes\u002Freferences\u002Farchitecture.md",{"path":68,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":69,"sourceHtmlUrl":70},"references\u002Fbiomedical.md",21275,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--10-optimization__ml-training-recipes\u002Freferences\u002Fbiomedical.md",{"path":72,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":73,"sourceHtmlUrl":74},"references\u002Fdomain-specific.md",19643,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--10-optimization__ml-training-recipes\u002Freferences\u002Fdomain-specific.md",{"path":76,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":77,"sourceHtmlUrl":78},"references\u002Fexperiment-loop.md",4574,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--10-optimization__ml-training-recipes\u002Freferences\u002Fexperiment-loop.md",{"path":80,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":81,"sourceHtmlUrl":82},"references\u002Foptimizers.md",10893,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--10-optimization__ml-training-recipes\u002Freferences\u002Foptimizers.md",{"path":84,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":85,"sourceHtmlUrl":86},"references\u002Fscaling-and-selection.md",17259,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--10-optimization__ml-training-recipes\u002Freferences\u002Fscaling-and-selection.md",[88,95,102,109,116,123],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M73JJB6EQPFPPVQK2Y9N67Q0","transformer-lens-interpretability","Provides guidance for mechanistic interpretability research using TransformerLens to inspect and manipulate transformer internals via HookPoints and activation caching. Use when reverse-engineering model algorithms, studying attention patterns, or performing activation patching experiments.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F130-transformer-lens\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":19},"01M73JHTANRRWEE9HP98TPEHAY","wiki-vitepress","Packages generated wiki Markdown into a VitePress static site with dark theme, dark-mode Mermaid diagrams with click-to-zoom, and production build output. Use when the user wants to create a browsable website from generated wiki pages.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F126-wiki-vitepress\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":19},"01M73JHJBMVFW46F1Y8K3SBF43","phoenix-observability","Open-source AI observability platform for LLM tracing, evaluation, and monitoring. Use when debugging LLM applications with detailed traces, running evaluations on datasets, or monitoring production AI systems with real-time insights.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F124-phoenix\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":19},"01M73JHFPA3A2806PMP42ZN86Q","frontend-ui-dark-ts","Build dark-themed React applications using Tailwind CSS with custom theming, glassmorphism effects, and Framer Motion animations. Use when creating dashboards, admin panels, or data-rich interfaces with a refined dark aesthetic.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F123-frontend-ui-dark-ts\u002FSKILL.md",[],[],{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":19},"01M73JFVBVK57NRJS49TT2VDCT","tensorboard","Visualize training metrics, debug models with histograms, compare experiments, visualize model graphs, and profile performance with TensorBoard - Google's ML visualization toolkit","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F113-tensorboard\u002FSKILL.md",[],[],{"skillNo":124,"name":125,"nameZh":6,"description":126,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":127,"licenseFileUrl":6,"sourceHtmlUrl":128,"categories":129,"keywords":130,"publishedAt":19},"01M73JFF6PK6ZCN41C89J7HS2B","markdown-mermaid-writing","Comprehensive markdown and Mermaid diagram writing skill. Use when creating any scientific document, report, analysis, or visualization. Establishes text-based diagrams as the default documentation standard with full style guides (markdown + mermaid), 24 diagram type references, and 9 document templates.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F110-markdown-mermaid-writing\u002FSKILL.md",[],[]]