[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7D2HRF1T67MS0VSYYFP9YN2":3,"skill-same:01M7D2HRF1T67MS0VSYYFP9YN2":84},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":38,"commitSha":39,"skillDirectory":40,"forksCount":13,"repositoryHtmlUrl":41,"repositoryHomepageUrl":6,"ownerHtmlUrl":42,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":43,"install":44,"resources":57},"01M7D2HRF1T67MS0VSYYFP9YN2","ml-training-recipes",null,"Battle-tested PyTorch training recipes for all domains — LLMs, vision, diffusion, medical imaging, protein\u002Fdrug discovery, spatial omics, genomics. Covers training loops, optimizer selection (AdamW, Muon), LR scheduling, mixed precision, debugging, and systematic experimentation. Use when training or fine-tuning neural networks, debugging loss spikes or OOM, choosing architectures, or optimizing GPU throughput.","beyondai\u002Fsandbox","beyondai","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F2845958?v=4","01M7D24SBNM47BJGJ9J2HB13BP","01M7D24SCDC9DYE005XH6R553S",0,"MIT","https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.claude\u002Fskills\u002Fml-training-recipes\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: ml-training-recipes\ndescription: Battle-tested PyTorch training recipes for all domains — LLMs, vision, diffusion, medical imaging, protein\u002Fdrug discovery, spatial omics, genomics. Covers training loops, optimizer selection (AdamW, Muon), LR scheduling, mixed precision, debugging, and systematic experimentation. Use when training or fine-tuning neural networks, debugging loss spikes or OOM, choosing architectures, or optimizing GPU throughput.\nversion: 1.0.0\nauthor: dailycafi\nlicense: MIT\ntags: [PyTorch, Training, Optimization, LLM, Vision, Diffusion, Biomedical, Muon, AdamW, Debugging]\ndependencies: [torch>=2.0.0]\n---\n\n# ML Training Recipes\n\nBattle-tested patterns for PyTorch training across domains. Drawn from production codebases\n(Karpathy's autoresearch\u002Fnanochat, torchvision, HuggingFace) and modern training practice.\n\n## Reference files (read when needed)\n\n- `references\u002Farchitecture.md` — Transformer\u002FLLM architecture code patterns, weight init\n- `references\u002Foptimizers.md` — Muon, AdamW hybrid, per-group LR, compiled optimizer steps\n- `references\u002Fdomain-specific.md` — Vision, diffusion, contrastive, distributed, checkpointing, data loading\n- `references\u002Fscaling-and-selection.md` — Scaling laws, compute budget tables, decision trees, DGX Spark\n- `references\u002Fbiomedical.md` — Drug discovery, protein models, medical imaging, genomics, clinical NLP\n- `references\u002Fexperiment-loop.md` — Autonomous experiment loop (autoresearch keep\u002Fdiscard\u002Frevert)\n\n---\n\n## Architecture Selection\n\nPick the right model by **data type** and **data scale**:\n\n| Data Type | \u003C 10K samples | 10K-100K | > 100K |\n|-----------|--------------|----------|--------|\n| **Images** | Pretrained CNN + fine-tune | Fine-tune ViT or CNN | ViT from scratch |\n| **Text (gen)** | Few-shot prompting | Fine-tune GPT\u002FLLaMA (LoRA) | Pretrain from scratch |\n| **Tabular** | XGBoost\u002FLightGBM | Still XGBoost | Neural viable |\n| **Audio** | Pretrained Whisper | Fine-tune AST | Train from scratch |\n| **Molecules** | Pretrained GNN | Fine-tune molecular LM | Train GNN from scratch |\n| **Proteins** | ESM-2 embeddings + head | Fine-tune ESM-2 | Train protein LM |\n| **Medical img** | Pretrained CNN | nnU-Net (auto-config) | Swin-UNETR \u002F MedSAM |\n\n**Key principle**: architecture matters less than training recipe at equal compute. A well-tuned\nResNet beats a poorly-tuned ViT (ref: \"ResNet Strikes Back\", Wightman 2021).\n\nFor biomedical domains, see `references\u002Fbiomedical.md`.\nFor sequence model selection and compute planning, see `references\u002Fscaling-and-selection.md`.\n\n---\n\n## Scaling Laws\n\n### Chinchilla rule (Hoffmann et al., 2022)\n\nCompute-optimal training: **~20 tokens per parameter**.\n\n| Model Size | Compute-Optimal | Inference-Optimal (100×) |\n|-----------|----------------|--------------------------|\n| 125M | 2.5B tokens | 12.5B tokens |\n| 1B | 20B tokens | 100B tokens |\n| 7B | 140B tokens | 700B tokens |\n\n**FLOPs ≈ 6 × N × D** (N=params, D=tokens). Data repetition limit: ~4 epochs before diminishing returns.\n\n---\n\n## Training Loop\n\n```python\nimport gc, time, torch\n\ntorch.manual_seed(42)\ntorch.set_float32_matmul_precision(\"high\")  # TF32 on Ampere+\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\n\ngrad_accum_steps = total_batch_size \u002F\u002F (batch_size * seq_len)\nstep = 0\n\nwhile not done:\n    t0 = time.time()\n    for micro_step in range(grad_accum_steps):\n        with autocast_ctx:\n            loss = model(x, y)\n        (loss \u002F grad_accum_steps).backward()\n        x, y = next(train_loader)\n\n    update_lr(optimizer, progress)\n    optimizer.step()\n    model.zero_grad(set_to_none=True)  # frees memory vs zeroing\n\n    if loss.item() > 100:  # fast-fail on divergence\n        print(\"FAIL: loss exploded\"); exit(1)\n\n    torch.cuda.synchronize()\n    if step == 0:\n        gc.collect(); gc.freeze(); gc.disable()  # avoid ~500ms GC stalls\n    step += 1\n```\n\n### Key principles\n\n- **Gradient clipping**: `clip_grad_norm_(params, 1.0)` — near-universal for Transformers.\n  Exception: Muon optimizer normalizes updates via orthogonalization, so clipping is optional.\n- **Tensor Core alignment**: batch size, hidden dims should be multiples of 8 (bf16) or 64 (A100).\n- **Time-based budgets** make experiments comparable across hardware.\n- **`cudnn.benchmark = True`** for fixed-size vision inputs.\n\n---\n\n## Optimizer Configuration\n\nModern LLM training uses different optimizers per parameter group:\n\n| Parameter Type | Optimizer | LR (base) | Weight Decay |\n|---------------|-----------|-----------|--------------|\n| 2D weight matrices | Muon | 0.04 | 0.2 |\n| Token embeddings | AdamW | 0.6 × scale | 0.0 |\n| Unembedding (lm_head) | AdamW | 0.004 × scale | 0.0 |\n| Per-layer scalars | AdamW | 0.005 × scale | 0.0 |\n\n**LR scaling by dimension**: `lr * (d_model \u002F 768)^(-0.5)` — keeps dynamics stable across sizes.\n\n### Rules of thumb\n\n- Embeddings need higher LR (sparse updates). Never weight-decay embeddings.\n- Weight decay scheduling: linearly decay WD to 0 over training.\n- AdamW defaults: β1=0.9, β2=0.95, eps=1e-10 (not default 1e-8 — prevents stale updates in bf16).\n\nFor Muon details (polar express orthogonalization, NorMuon), see `references\u002Foptimizers.md`.\n\n---\n\n## Learning Rate Scheduling\n\n### Time-based (autoresearch style)\n\n```python\ndef get_lr_multiplier(progress):  # progress = elapsed_time \u002F time_budget\n    if progress \u003C warmup_ratio:\n        return progress \u002F warmup_ratio\n    elif progress \u003C 1.0 - warmdown_ratio:\n        return 1.0\n    else:\n        cooldown = (1.0 - progress) \u002F warmdown_ratio\n        return cooldown + (1 - cooldown) * final_lr_frac\n```\n\n### Cosine decay\n\n```python\ndef get_lr(step, total_steps, max_lr, min_lr, warmup_steps):\n    if step \u003C warmup_steps:\n        return max_lr * step \u002F warmup_steps\n    progress = (step - warmup_steps) \u002F (total_steps - warmup_steps)\n    return min_lr + 0.5 * (max_lr - min_lr) * (1 + math.cos(math.pi * progress))\n```\n\n**WSD (Warmup-Stable-Decay)**: gaining traction — easier to resume training mid-run.\n\n### Guidance\n\n- **Warmup**: 1-5% of training. Zero warmup valid with Muon (autoresearch uses `WARMUP_RATIO=0.0`).\n- **Warmdown**: 30-50% of training in LR decay. Matters more than warmup for final quality.\n- **Final LR**: 0 or ~10% of peak. Zero is simpler.\n\n---\n\n## Mixed Precision & Compilation\n\n```python\nimport os\nos.environ[\"PYTORCH_ALLOC_CONF\"] = \"expandable_segments:True\"  # before torch import\n\nimport torch\ntorch.set_float32_matmul_precision(\"high\")\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\nmodel = torch.compile(model, dynamic=False)\n```\n\n- **bf16** (Ampere+): same exponent as fp32, no loss scaling needed. Preferred over fp16.\n- **fp16**: needs GradScaler. Use only on V100 or older.\n- `dynamic=False` enables max optimization. Add `fullgraph=True` if no graph breaks.\n- First steps are slow (JIT) — exclude from timing.\n\n---\n\n## Memory & Performance\n\n### Meta device init (large models)\n\n```python\nwith torch.device(\"meta\"):\n    model = GPT(config)          # zero memory\nmodel.to_empty(device=\"cuda\")\nmodel.init_weights()\n```\n\n### MFU (Model FLOPs Utilization)\n\n```python\nachieved_flops = model_flops_per_token * batch_tokens \u002F step_time\nmfu = achieved_flops \u002F gpu_peak_flops\n# H100 SXM: 989.5 TFLOPS | A100: 312 | RTX 4090: 165\n```\n\nGood targets: >30% decent, >40% good, >50% excellent (single-GPU).\n\n### OOM solutions (in order)\n\n1. Reduce `DEVICE_BATCH_SIZE`, increase `grad_accum_steps`\n2. `PYTORCH_ALLOC_CONF=expandable_segments:True`\n3. `model.zero_grad(set_to_none=True)`\n4. Meta device init → `to_empty`\n5. Activation checkpointing: `torch.utils.checkpoint.checkpoint()`\n6. 8-bit optimizer (bitsandbytes): ~30% savings on optimizer states\n\n---\n\n## Hyperparameter Search\n\n### Priority order (tune first → last)\n\n1. **Learning rate** — most impactful. Always tune first.\n2. **Batch size** — largest that fits. Speed knob, not quality knob.\n3. **Weight decay** — 0.01-0.1 for AdamW.\n4. **Warmup steps** — 1-5% of training.\n\n### The 2025 default recipe\n\n| Setting | Value |\n|---------|-------|\n| Optimizer | AdamW (β1=0.9, β2=0.95, eps=1e-10) |\n| Weight decay | 0.1 |\n| LR schedule | Cosine decay or WSD |\n| Peak LR | 3e-4 (scale down for larger models) |\n| Precision | bf16 |\n| Grad clipping | max_norm=1.0 |\n| Normalization | RMSNorm (pre-norm) |\n| Activation | SwiGLU |\n| Position encoding | RoPE |\n| Attention | Flash Attention, optionally GQA |\n\n---\n\n## Debugging Checklist\n\n### Karpathy's recipe (still canonical)\n\n1. **Become one with the data** — visualize, check distributions, verify labels\n2. **Get end-to-end running first** — verify on a trivial case\n3. **Overfit one batch** — if you can't, you have a bug\n4. **Then regularize** — add regularization only after overfitting works\n5. **Tune hyperparameters** — start with known defaults\n\n### Loss exploding \u002F NaN\n\n1. Reduce LR (3-10× smaller)\n2. Add gradient clipping: `clip_grad_norm_(params, 1.0)`\n3. Check for inf\u002Fnan in inputs\n4. Add logit soft capping: `softcap * tanh(logits \u002F softcap)`\n5. Add QK-norm in attention\n6. Verify weight init (zero-init output projections?)\n7. Check loss reduction with gradient accumulation (`loss \u002F grad_accum_steps`)\n\n### Slow training \u002F Low MFU\n\n1. Verify `torch.compile` is active\n2. Check `torch.set_float32_matmul_precision(\"high\")`\n3. Pin memory + non_blocking transfers\n4. Profile with `torch.profiler`\n5. GC stalls? `gc.freeze(); gc.disable()`\n6. Tensor Core alignment: dims multiples of 8\u002F64\n\n### Loss plateau \u002F Slow convergence\n\n1. LR too low — try 2-5× larger\n2. Warmup too long\n3. Weight decay too high\n4. Verify LR schedule is actually applied (print each step)\n5. Model too small for task\n\n### Silent failures\n\n1. **Data leakage** between train\u002Fval\n2. **Wrong preprocessing at inference** — augmentation mismatch\n3. **Label errors** — use cleanlab to detect\n4. **Shuffling bugs** — correlated batches\n5. **Tokenizer mismatch** with pretrained model\n\n### What to monitor\n\n- **Gradient norms** — spike precedes loss spike\n- **Per-layer activation stats** — reveals exploding\u002Fvanishing\n- **Dead neurons** — >50% zero ReLU = dying ReLU problem\n- **Learning rate** — verify schedule applied (common silent bug)\n\n---\n\n## Experiment Management\n\nTrack experiments in TSV for easy comparison:\n\n```\ncommit  val_bpb  memory_gb  status   description\na1b2c3d 0.9979   44.0       keep     baseline\nb2c3d4e 0.9932   44.2       keep     increase matrix LR to 0.04\nc3d4e5f 1.0050   44.0       discard  switch to GeLU (worse)\n```\n\n**Simplicity criterion**: all else equal, simpler is better. Removing something and getting equal\nresults is a great outcome. For systematic agent-driven experimentation, see `references\u002Fexperiment-loop.md`.\n\n### Evaluation metrics by domain\n\n| Domain | Primary Metric | Notes |\n|--------|---------------|-------|\n| LLM | BPB (bits per byte) | Vocab-size-independent |\n| Classification | Accuracy \u002F F1 | Macro-F1 for imbalanced |\n| Segmentation | mIoU \u002F Dice | Per-class IoU reveals weak spots |\n| Generation | FID | Needs >10k samples |\n| Regression | RMSE \u002F MAE | Log-transform skewed targets |\n","# ML Training Recipes\n\nBattle-tested patterns for PyTorch training across domains. Drawn from production codebases\n(Karpathy's autoresearch\u002Fnanochat, torchvision, HuggingFace) and modern training practice.\n\n## Reference files (read when needed)\n\n- `references\u002Farchitecture.md` — Transformer\u002FLLM architecture code patterns, weight init\n- `references\u002Foptimizers.md` — Muon, AdamW hybrid, per-group LR, compiled optimizer steps\n- `references\u002Fdomain-specific.md` — Vision, diffusion, contrastive, distributed, checkpointing, data loading\n- `references\u002Fscaling-and-selection.md` — Scaling laws, compute budget tables, decision trees, DGX Spark\n- `references\u002Fbiomedical.md` — Drug discovery, protein models, medical imaging, genomics, clinical NLP\n- `references\u002Fexperiment-loop.md` — Autonomous experiment loop (autoresearch keep\u002Fdiscard\u002Frevert)\n\n---\n\n## Architecture Selection\n\nPick the right model by **data type** and **data scale**:\n\n| Data Type | \u003C 10K samples | 10K-100K | > 100K |\n|-----------|--------------|----------|--------|\n| **Images** | Pretrained CNN + fine-tune | Fine-tune ViT or CNN | ViT from scratch |\n| **Text (gen)** | Few-shot prompting | Fine-tune GPT\u002FLLaMA (LoRA) | Pretrain from scratch |\n| **Tabular** | XGBoost\u002FLightGBM | Still XGBoost | Neural viable |\n| **Audio** | Pretrained Whisper | Fine-tune AST | Train from scratch |\n| **Molecules** | Pretrained GNN | Fine-tune molecular LM | Train GNN from scratch |\n| **Proteins** | ESM-2 embeddings + head | Fine-tune ESM-2 | Train protein LM |\n| **Medical img** | Pretrained CNN | nnU-Net (auto-config) | Swin-UNETR \u002F MedSAM |\n\n**Key principle**: architecture matters less than training recipe at equal compute. A well-tuned\nResNet beats a poorly-tuned ViT (ref: \"ResNet Strikes Back\", Wightman 2021).\n\nFor biomedical domains, see `references\u002Fbiomedical.md`.\nFor sequence model selection and compute planning, see `references\u002Fscaling-and-selection.md`.\n\n---\n\n## Scaling Laws\n\n### Chinchilla rule (Hoffmann et al., 2022)\n\nCompute-optimal training: **~20 tokens per parameter**.\n\n| Model Size | Compute-Optimal | Inference-Optimal (100×) |\n|-----------|----------------|--------------------------|\n| 125M | 2.5B tokens | 12.5B tokens |\n| 1B | 20B tokens | 100B tokens |\n| 7B | 140B tokens | 700B tokens |\n\n**FLOPs ≈ 6 × N × D** (N=params, D=tokens). Data repetition limit: ~4 epochs before diminishing returns.\n\n---\n\n## Training Loop\n\n```python\nimport gc, time, torch\n\ntorch.manual_seed(42)\ntorch.set_float32_matmul_precision(\"high\")  # TF32 on Ampere+\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\n\ngrad_accum_steps = total_batch_size \u002F\u002F (batch_size * seq_len)\nstep = 0\n\nwhile not done:\n    t0 = time.time()\n    for micro_step in range(grad_accum_steps):\n        with autocast_ctx:\n            loss = model(x, y)\n        (loss \u002F grad_accum_steps).backward()\n        x, y = next(train_loader)\n\n    update_lr(optimizer, progress)\n    optimizer.step()\n    model.zero_grad(set_to_none=True)  # frees memory vs zeroing\n\n    if loss.item() > 100:  # fast-fail on divergence\n        print(\"FAIL: loss exploded\"); exit(1)\n\n    torch.cuda.synchronize()\n    if step == 0:\n        gc.collect(); gc.freeze(); gc.disable()  # avoid ~500ms GC stalls\n    step += 1\n```\n\n### Key principles\n\n- **Gradient clipping**: `clip_grad_norm_(params, 1.0)` — near-universal for Transformers.\n  Exception: Muon optimizer normalizes updates via orthogonalization, so clipping is optional.\n- **Tensor Core alignment**: batch size, hidden dims should be multiples of 8 (bf16) or 64 (A100).\n- **Time-based budgets** make experiments comparable across hardware.\n- **`cudnn.benchmark = True`** for fixed-size vision inputs.\n\n---\n\n## Optimizer Configuration\n\nModern LLM training uses different optimizers per parameter group:\n\n| Parameter Type | Optimizer | LR (base) | Weight Decay |\n|---------------|-----------|-----------|--------------|\n| 2D weight matrices | Muon | 0.04 | 0.2 |\n| Token embeddings | AdamW | 0.6 × scale | 0.0 |\n| Unembedding (lm_head) | AdamW | 0.004 × scale | 0.0 |\n| Per-layer scalars | AdamW | 0.005 × scale | 0.0 |\n\n**LR scaling by dimension**: `lr * (d_model \u002F 768)^(-0.5)` — keeps dynamics stable across sizes.\n\n### Rules of thumb\n\n- Embeddings need higher LR (sparse updates). Never weight-decay embeddings.\n- Weight decay scheduling: linearly decay WD to 0 over training.\n- AdamW defaults: β1=0.9, β2=0.95, eps=1e-10 (not default 1e-8 — prevents stale updates in bf16).\n\nFor Muon details (polar express orthogonalization, NorMuon), see `references\u002Foptimizers.md`.\n\n---\n\n## Learning Rate Scheduling\n\n### Time-based (autoresearch style)\n\n```python\ndef get_lr_multiplier(progress):  # progress = elapsed_time \u002F time_budget\n    if progress \u003C warmup_ratio:\n        return progress \u002F warmup_ratio\n    elif progress \u003C 1.0 - warmdown_ratio:\n        return 1.0\n    else:\n        cooldown = (1.0 - progress) \u002F warmdown_ratio\n        return cooldown + (1 - cooldown) * final_lr_frac\n```\n\n### Cosine decay\n\n```python\ndef get_lr(step, total_steps, max_lr, min_lr, warmup_steps):\n    if step \u003C warmup_steps:\n        return max_lr * step \u002F warmup_steps\n    progress = (step - warmup_steps) \u002F (total_steps - warmup_steps)\n    return min_lr + 0.5 * (max_lr - min_lr) * (1 + math.cos(math.pi * progress))\n```\n\n**WSD (Warmup-Stable-Decay)**: gaining traction — easier to resume training mid-run.\n\n### Guidance\n\n- **Warmup**: 1-5% of training. Zero warmup valid with Muon (autoresearch uses `WARMUP_RATIO=0.0`).\n- **Warmdown**: 30-50% of training in LR decay. Matters more than warmup for final quality.\n- **Final LR**: 0 or ~10% of peak. Zero is simpler.\n\n---\n\n## Mixed Precision & Compilation\n\n```python\nimport os\nos.environ[\"PYTORCH_ALLOC_CONF\"] = \"expandable_segments:True\"  # before torch import\n\nimport torch\ntorch.set_float32_matmul_precision(\"high\")\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\nmodel = torch.compile(model, dynamic=False)\n```\n\n- **bf16** (Ampere+): same exponent as fp32, no loss scaling needed. Preferred over fp16.\n- **fp16**: needs GradScaler. Use only on V100 or older.\n- `dynamic=False` enables max optimization. Add `fullgraph=True` if no graph breaks.\n- First steps are slow (JIT) — exclude from timing.\n\n---\n\n## Memory & Performance\n\n### Meta device init (large models)\n\n```python\nwith torch.device(\"meta\"):\n    model = GPT(config)          # zero memory\nmodel.to_empty(device=\"cuda\")\nmodel.init_weights()\n```\n\n### MFU (Model FLOPs Utilization)\n\n```python\nachieved_flops = model_flops_per_token * batch_tokens \u002F step_time\nmfu = achieved_flops \u002F gpu_peak_flops\n# H100 SXM: 989.5 TFLOPS | A100: 312 | RTX 4090: 165\n```\n\nGood targets: >30% decent, >40% good, >50% excellent (single-GPU).\n\n### OOM solutions (in order)\n\n1. Reduce `DEVICE_BATCH_SIZE`, increase `grad_accum_steps`\n2. `PYTORCH_ALLOC_CONF=expandable_segments:True`\n3. `model.zero_grad(set_to_none=True)`\n4. Meta device init → `to_empty`\n5. Activation checkpointing: `torch.utils.checkpoint.checkpoint()`\n6. 8-bit optimizer (bitsandbytes): ~30% savings on optimizer states\n\n---\n\n## Hyperparameter Search\n\n### Priority order (tune first → last)\n\n1. **Learning rate** — most impactful. Always tune first.\n2. **Batch size** — largest that fits. Speed knob, not quality knob.\n3. **Weight decay** — 0.01-0.1 for AdamW.\n4. **Warmup steps** — 1-5% of training.\n\n### The 2025 default recipe\n\n| Setting | Value |\n|---------|-------|\n| Optimizer | AdamW (β1=0.9, β2=0.95, eps=1e-10) |\n| Weight decay | 0.1 |\n| LR schedule | Cosine decay or WSD |\n| Peak LR | 3e-4 (scale down for larger models) |\n| Precision | bf16 |\n| Grad clipping | max_norm=1.0 |\n| Normalization | RMSNorm (pre-norm) |\n| Activation | SwiGLU |\n| Position encoding | RoPE |\n| Attention | Flash Attention, optionally GQA |\n\n---\n\n## Debugging Checklist\n\n### Karpathy's recipe (still canonical)\n\n1. **Become one with the data** — visualize, check distributions, verify labels\n2. **Get end-to-end running first** — verify on a trivial case\n3. **Overfit one batch** — if you can't, you have a bug\n4. **Then regularize** — add regularization only after overfitting works\n5. **Tune hyperparameters** — start with known defaults\n\n### Loss exploding \u002F NaN\n\n1. Reduce LR (3-10× smaller)\n2. Add gradient clipping: `clip_grad_norm_(params, 1.0)`\n3. Check for inf\u002Fnan in inputs\n4. Add logit soft capping: `softcap * tanh(logits \u002F softcap)`\n5. Add QK-norm in attention\n6. Verify weight init (zero-init output projections?)\n7. Check loss reduction with gradient accumulation (`loss \u002F grad_accum_steps`)\n\n### Slow training \u002F Low MFU\n\n1. Verify `torch.compile` is active\n2. Check `torch.set_float32_matmul_precision(\"high\")`\n3. Pin memory + non_blocking transfers\n4. Profile with `torch.profiler`\n5. GC stalls? `gc.freeze(); gc.disable()`\n6. Tensor Core alignment: dims multiples of 8\u002F64\n\n### Loss plateau \u002F Slow convergence\n\n1. LR too low — try 2-5× larger\n2. Warmup too long\n3. Weight decay too high\n4. Verify LR schedule is actually applied (print each step)\n5. Model too small for task\n\n### Silent failures\n\n1. **Data leakage** between train\u002Fval\n2. **Wrong preprocessing at inference** — augmentation mismatch\n3. **Label errors** — use cleanlab to detect\n4. **Shuffling bugs** — correlated batches\n5. **Tokenizer mismatch** with pretrained model\n\n### What to monitor\n\n- **Gradient norms** — spike precedes loss spike\n- **Per-layer activation stats** — reveals exploding\u002Fvanishing\n- **Dead neurons** — >50% zero ReLU = dying ReLU problem\n- **Learning rate** — verify schedule applied (common silent bug)\n\n---\n\n## Experiment Management\n\nTrack experiments in TSV for easy comparison:\n\n```\ncommit  val_bpb  memory_gb  status   description\na1b2c3d 0.9979   44.0       keep     baseline\nb2c3d4e 0.9932   44.2       keep     increase matrix LR to 0.04\nc3d4e5f 1.0050   44.0       discard  switch to GeLU (worse)\n```\n\n**Simplicity criterion**: all else equal, simpler is better. Removing something and getting equal\nresults is a great outcome. For systematic agent-driven experimentation, see `references\u002Fexperiment-loop.md`.\n\n### Evaluation metrics by domain\n\n| Domain | Primary Metric | Notes |\n|--------|---------------|-------|\n| LLM | BPB (bits per byte) | Vocab-size-independent |\n| Classification | Accuracy \u002F F1 | Macro-F1 for imbalanced |\n| Segmentation | mIoU \u002F Dice | Per-class IoU reveals weak spots |\n| Generation | FID | Needs >10k samples |\n| Regression | RMSE \u002F MAE | Log-transform skewed targets |\n",{"name":5,"tags":23,"author":34,"license":14,"version":35,"description":7,"dependencies":36},[24,25,26,27,28,29,30,31,32,33],"PyTorch","Training","Optimization","LLM","Vision","Diffusion","Biomedical","Muon","AdamW","Debugging","dailycafi","1.0.0",[37],"torch>=2.0.0","main","46e03791269608c5ac886a64c08f5d94147e8786","labs\u002F.claude\u002Fskills\u002Fml-training-recipes","https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox","https:\u002F\u002Fgithub.com\u002Fbeyondai","2026-11-08 04:18:42",{"source":8,"skillName":5,"commands":45,"prompt":55,"zipAvailable":56},[46,49,52],{"tool":47,"command":48},"npx","npx skills add beyondai\u002Fsandbox --skill ml-training-recipes",{"tool":50,"command":51},"bunx","bunx skills add beyondai\u002Fsandbox --skill ml-training-recipes",{"tool":53,"command":54},"pnpm","pnpm dlx skills add beyondai\u002Fsandbox --skill ml-training-recipes","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.claude\u002Fskills\u002Fml-training-recipes\u002FSKILL.md",true,[58,64,68,72,76,80],{"path":59,"resourceType":60,"resourceTypeDesc":61,"sizeBytes":62,"sourceHtmlUrl":63},"references\u002Farchitecture.md","REFERENCE","参考文档",10204,"https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.claude\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Farchitecture.md",{"path":65,"resourceType":60,"resourceTypeDesc":61,"sizeBytes":66,"sourceHtmlUrl":67},"references\u002Fbiomedical.md",20778,"https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.claude\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Fbiomedical.md",{"path":69,"resourceType":60,"resourceTypeDesc":61,"sizeBytes":70,"sourceHtmlUrl":71},"references\u002Fdomain-specific.md",19018,"https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.claude\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Fdomain-specific.md",{"path":73,"resourceType":60,"resourceTypeDesc":61,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Fexperiment-loop.md",4446,"https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.claude\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Fexperiment-loop.md",{"path":77,"resourceType":60,"resourceTypeDesc":61,"sizeBytes":78,"sourceHtmlUrl":79},"references\u002Foptimizers.md",10564,"https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.claude\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Foptimizers.md",{"path":81,"resourceType":60,"resourceTypeDesc":61,"sizeBytes":82,"sourceHtmlUrl":83},"references\u002Fscaling-and-selection.md",16832,"https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.claude\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Fscaling-and-selection.md",[85,92,99,106,113,120],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":18},"01M7D3GMPGJZRG8EPY404YF8TR","weights-and-biases","Track ML experiments with automatic logging, visualize training in real-time, optimize hyperparameters with sweeps, and manage model registry with W&B - collaborative MLOps platform","https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.opencode\u002Fskills\u002Fweights-and-biases\u002FSKILL.md",[],[],{"skillNo":93,"name":94,"nameZh":6,"description":95,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":18},"01M7D3GFZH440V2J532BSZMPVP","serving-llms-vllm","Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency\u002Fthroughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ\u002FAWQ\u002FFP8), and tensor parallelism.","https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.opencode\u002Fskills\u002Fvllm\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":18},"01M7D3GAE8MQCPGF72EP9G9GKY","verl-rl-training","Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.","https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.opencode\u002Fskills\u002Fverl\u002FSKILL.md",[],[],{"skillNo":107,"name":108,"nameZh":6,"description":109,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":18},"01M7D3G7KKNE5RGNYJNXV4F0MM","unsloth","Expert guidance for fast fine-tuning with Unsloth - 2-5x faster training, 50-80% less memory, LoRA\u002FQLoRA optimization","https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.opencode\u002Fskills\u002Funsloth\u002FSKILL.md",[],[],{"skillNo":114,"name":115,"nameZh":6,"description":116,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":117,"categories":118,"keywords":119,"publishedAt":18},"01M7D3G2RCZQ08GS0KAW5CBWSC","fine-tuning-with-trl","Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO\u002FGRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.","https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.opencode\u002Fskills\u002Ftrl-fine-tuning\u002FSKILL.md",[],[],{"skillNo":121,"name":122,"nameZh":6,"description":123,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":124,"categories":125,"keywords":126,"publishedAt":18},"01M7D3FX08BPB172CYGM597NZF","torchforge-rl-training","Provides guidance for PyTorch-native agentic RL using torchforge, Meta's library separating infra from algorithms. Use when you want clean RL abstractions, easy algorithm experimentation, or scalable training with Monarch and TorchTitan.","https:\u002F\u002Fgithub.com\u002Fbeyondai\u002Fsandbox\u002Fblob\u002Fmain\u002Flabs\u002F.opencode\u002Fskills\u002Ftorchforge\u002FSKILL.md",[],[]]