[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MAFT2NFMYJN1GCFSTXKH34":3,"skill-same:01M6MAFT2NFMYJN1GCFSTXKH34":102},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":40,"linkOnly":41,"markdownContent":42,"markdownBody":43,"frontMatter":44,"refName":52,"commitSha":53,"skillDirectory":54,"forksCount":55,"repositoryHtmlUrl":56,"repositoryHomepageUrl":57,"ownerHtmlUrl":58,"ownerBlogUrl":59,"ownerDescription":60,"updatedAt":61,"install":62,"resources":75},"01M6MAFT2NFMYJN1GCFSTXKH34","ml-training-recipes",null,"Battle-tested PyTorch training recipes for all domains — LLMs, vision, diffusion, medical imaging, protein\u002Fdrug discovery, spatial omics, genomics. Covers training loops, optimizer selection (AdamW, Muon), LR scheduling, mixed precision, debugging, and systematic experimentation. Use when training or fine-tuning neural networks, debugging loss spikes or OOM, choosing architectures, or optimizing GPU throughput.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1554,"MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39],"ml","training","ml training recipes","recipes","biomedical","Protein","for","imaging","experimentation","architectures","domains","drug discovery","systematic","neural networks","selection","scheduling","use","discovery","networks","Fine-Tuning","pytorch","2026-11-10 10:12:30",false,"---\nname: ml-training-recipes\ndescription: Battle-tested PyTorch training recipes for all domains — LLMs, vision, diffusion, medical imaging, protein\u002Fdrug discovery, spatial omics, genomics. Covers training loops, optimizer selection (AdamW, Muon), LR scheduling, mixed precision, debugging, and systematic experimentation. Use when training or fine-tuning neural networks, debugging loss spikes or OOM, choosing architectures, or optimizing GPU throughput.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-training\n  maintainer: Kalaris Labs\n  contributor: dailycafi\n  tags: PyTorch, Training, Optimization, LLM, Vision, Diffusion, Biomedical, Muon, AdamW, Debugging\n  dependencies: torch>=2.0.0\n---\n\n# ML Training Recipes\n\nBattle-tested patterns for PyTorch training across domains. Drawn from production codebases\n(Karpathy's autoresearch\u002Fnanochat, torchvision, HuggingFace) and modern training practice.\n\n## Reference files (read when needed)\n\n- `references\u002Farchitecture.md` — Transformer\u002FLLM architecture code patterns, weight init\n- `references\u002Foptimizers.md` — Muon, AdamW hybrid, per-group LR, compiled optimizer steps\n- `references\u002Fdomain-specific.md` — Vision, diffusion, contrastive, distributed, checkpointing, data loading\n- `references\u002Fscaling-and-selection.md` — Scaling laws, compute budget tables, decision trees, DGX Spark\n- `references\u002Fbiomedical.md` — Drug discovery, protein models, medical imaging, genomics, clinical NLP\n- `references\u002Fexperiment-loop.md` — Autonomous experiment loop (autoresearch keep\u002Fdiscard\u002Frevert)\n\n---\n\n## Architecture Selection\n\nPick the right model by **data type** and **data scale**:\n\n| Data Type | \u003C 10K samples | 10K-100K | > 100K |\n|-----------|--------------|----------|--------|\n| **Images** | Pretrained CNN + fine-tune | Fine-tune ViT or CNN | ViT from scratch |\n| **Text (gen)** | Few-shot prompting | Fine-tune GPT\u002FLLaMA (LoRA) | Pretrain from scratch |\n| **Tabular** | XGBoost\u002FLightGBM | Still XGBoost | Neural viable |\n| **Audio** | Pretrained Whisper | Fine-tune AST | Train from scratch |\n| **Molecules** | Pretrained GNN | Fine-tune molecular LM | Train GNN from scratch |\n| **Proteins** | ESM-2 embeddings + head | Fine-tune ESM-2 | Train protein LM |\n| **Medical img** | Pretrained CNN | nnU-Net (auto-config) | Swin-UNETR \u002F MedSAM |\n\n**Key principle**: architecture matters less than training recipe at equal compute. A well-tuned\nResNet beats a poorly-tuned ViT (ref: \"ResNet Strikes Back\", Wightman 2021).\n\nFor biomedical domains, see `references\u002Fbiomedical.md`.\nFor sequence model selection and compute planning, see `references\u002Fscaling-and-selection.md`.\n\n---\n\n## Scaling Laws\n\n### Chinchilla rule (Hoffmann et al., 2022)\n\nCompute-optimal training: **~20 tokens per parameter**.\n\n| Model Size | Compute-Optimal | Inference-Optimal (100×) |\n|-----------|----------------|--------------------------|\n| 125M | 2.5B tokens | 12.5B tokens |\n| 1B | 20B tokens | 100B tokens |\n| 7B | 140B tokens | 700B tokens |\n\n**FLOPs ≈ 6 × N × D** (N=params, D=tokens). Data repetition limit: ~4 epochs before diminishing returns.\n\n---\n\n## Training Loop\n\n```python\nimport gc, time, torch\n\ntorch.manual_seed(42)\ntorch.set_float32_matmul_precision(\"high\")  # TF32 on Ampere+\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\n\ngrad_accum_steps = total_batch_size \u002F\u002F (batch_size * seq_len)\nstep = 0\n\nwhile not done:\n    t0 = time.time()\n    for micro_step in range(grad_accum_steps):\n        with autocast_ctx:\n            loss = model(x, y)\n        (loss \u002F grad_accum_steps).backward()\n        x, y = next(train_loader)\n\n    update_lr(optimizer, progress)\n    optimizer.step()\n    model.zero_grad(set_to_none=True)  # frees memory vs zeroing\n\n    if loss.item() > 100:  # fast-fail on divergence\n        print(\"FAIL: loss exploded\"); exit(1)\n\n    torch.cuda.synchronize()\n    if step == 0:\n        gc.collect(); gc.freeze(); gc.disable()  # avoid ~500ms GC stalls\n    step += 1\n```\n\n### Key principles\n\n- **Gradient clipping**: `clip_grad_norm_(params, 1.0)` — near-universal for Transformers.\n  Exception: Muon optimizer normalizes updates via orthogonalization, so clipping is optional.\n- **Tensor Core alignment**: batch size, hidden dims should be multiples of 8 (bf16) or 64 (A100).\n- **Time-based budgets** make experiments comparable across hardware.\n- **`cudnn.benchmark = True`** for fixed-size vision inputs.\n\n---\n\n## Optimizer Configuration\n\nModern LLM training uses different optimizers per parameter group:\n\n| Parameter Type | Optimizer | LR (base) | Weight Decay |\n|---------------|-----------|-----------|--------------|\n| 2D weight matrices | Muon | 0.04 | 0.2 |\n| Token embeddings | AdamW | 0.6 × scale | 0.0 |\n| Unembedding (lm_head) | AdamW | 0.004 × scale | 0.0 |\n| Per-layer scalars | AdamW | 0.005 × scale | 0.0 |\n\n**LR scaling by dimension**: `lr * (d_model \u002F 768)^(-0.5)` — keeps dynamics stable across sizes.\n\n### Rules of thumb\n\n- Embeddings need higher LR (sparse updates). Never weight-decay embeddings.\n- Weight decay scheduling: linearly decay WD to 0 over training.\n- AdamW defaults: β1=0.9, β2=0.95, eps=1e-10 (not default 1e-8 — prevents stale updates in bf16).\n\nFor Muon details (polar express orthogonalization, NorMuon), see `references\u002Foptimizers.md`.\n\n---\n\n## Learning Rate Scheduling\n\n### Time-based (autoresearch style)\n\n```python\ndef get_lr_multiplier(progress):  # progress = elapsed_time \u002F time_budget\n    if progress \u003C warmup_ratio:\n        return progress \u002F warmup_ratio\n    elif progress \u003C 1.0 - warmdown_ratio:\n        return 1.0\n    else:\n        cooldown = (1.0 - progress) \u002F warmdown_ratio\n        return cooldown + (1 - cooldown) * final_lr_frac\n```\n\n### Cosine decay\n\n```python\ndef get_lr(step, total_steps, max_lr, min_lr, warmup_steps):\n    if step \u003C warmup_steps:\n        return max_lr * step \u002F warmup_steps\n    progress = (step - warmup_steps) \u002F (total_steps - warmup_steps)\n    return min_lr + 0.5 * (max_lr - min_lr) * (1 + math.cos(math.pi * progress))\n```\n\n**WSD (Warmup-Stable-Decay)**: gaining traction — easier to resume training mid-run.\n\n### Guidance\n\n- **Warmup**: 1-5% of training. Zero warmup valid with Muon (autoresearch uses `WARMUP_RATIO=0.0`).\n- **Warmdown**: 30-50% of training in LR decay. Matters more than warmup for final quality.\n- **Final LR**: 0 or ~10% of peak. Zero is simpler.\n\n---\n\n## Mixed Precision & Compilation\n\n```python\nimport os\nos.environ[\"PYTORCH_ALLOC_CONF\"] = \"expandable_segments:True\"  # before torch import\n\nimport torch\ntorch.set_float32_matmul_precision(\"high\")\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\nmodel = torch.compile(model, dynamic=False)\n```\n\n- **bf16** (Ampere+): same exponent as fp32, no loss scaling needed. Preferred over fp16.\n- **fp16**: needs GradScaler. Use only on V100 or older.\n- `dynamic=False` enables max optimization. Add `fullgraph=True` if no graph breaks.\n- First steps are slow (JIT) — exclude from timing.\n\n---\n\n## Memory & Performance\n\n### Meta device init (large models)\n\n```python\nwith torch.device(\"meta\"):\n    model = GPT(config)          # zero memory\nmodel.to_empty(device=\"cuda\")\nmodel.init_weights()\n```\n\n### MFU (Model FLOPs Utilization)\n\n```python\nachieved_flops = model_flops_per_token * batch_tokens \u002F step_time\nmfu = achieved_flops \u002F gpu_peak_flops\n# H100 SXM: 989.5 TFLOPS | A100: 312 | RTX 4090: 165\n```\n\nGood targets: >30% decent, >40% good, >50% excellent (single-GPU).\n\n### OOM solutions (in order)\n\n1. Reduce `DEVICE_BATCH_SIZE`, increase `grad_accum_steps`\n2. `PYTORCH_ALLOC_CONF=expandable_segments:True`\n3. `model.zero_grad(set_to_none=True)`\n4. Meta device init → `to_empty`\n5. Activation checkpointing: `torch.utils.checkpoint.checkpoint()`\n6. 8-bit optimizer (bitsandbytes): ~30% savings on optimizer states\n\n---\n\n## Hyperparameter Search\n\n### Priority order (tune first → last)\n\n1. **Learning rate** — most impactful. Always tune first.\n2. **Batch size** — largest that fits. Speed knob, not quality knob.\n3. **Weight decay** — 0.01-0.1 for AdamW.\n4. **Warmup steps** — 1-5% of training.\n\n### The 2025 default recipe\n\n| Setting | Value |\n|---------|-------|\n| Optimizer | AdamW (β1=0.9, β2=0.95, eps=1e-10) |\n| Weight decay | 0.1 |\n| LR schedule | Cosine decay or WSD |\n| Peak LR | 3e-4 (scale down for larger models) |\n| Precision | bf16 |\n| Grad clipping | max_norm=1.0 |\n| Normalization | RMSNorm (pre-norm) |\n| Activation | SwiGLU |\n| Position encoding | RoPE |\n| Attention | Flash Attention, optionally GQA |\n\n---\n\n## Debugging Checklist\n\n### Karpathy's recipe (still canonical)\n\n1. **Become one with the data** — visualize, check distributions, verify labels\n2. **Get end-to-end running first** — verify on a trivial case\n3. **Overfit one batch** — if you can't, you have a bug\n4. **Then regularize** — add regularization only after overfitting works\n5. **Tune hyperparameters** — start with known defaults\n\n### Loss exploding \u002F NaN\n\n1. Reduce LR (3-10× smaller)\n2. Add gradient clipping: `clip_grad_norm_(params, 1.0)`\n3. Check for inf\u002Fnan in inputs\n4. Add logit soft capping: `softcap * tanh(logits \u002F softcap)`\n5. Add QK-norm in attention\n6. Verify weight init (zero-init output projections?)\n7. Check loss reduction with gradient accumulation (`loss \u002F grad_accum_steps`)\n\n### Slow training \u002F Low MFU\n\n1. Verify `torch.compile` is active\n2. Check `torch.set_float32_matmul_precision(\"high\")`\n3. Pin memory + non_blocking transfers\n4. Profile with `torch.profiler`\n5. GC stalls? `gc.freeze(); gc.disable()`\n6. Tensor Core alignment: dims multiples of 8\u002F64\n\n### Loss plateau \u002F Slow convergence\n\n1. LR too low — try 2-5× larger\n2. Warmup too long\n3. Weight decay too high\n4. Verify LR schedule is actually applied (print each step)\n5. Model too small for task\n\n### Silent failures\n\n1. **Data leakage** between train\u002Fval\n2. **Wrong preprocessing at inference** — augmentation mismatch\n3. **Label errors** — use cleanlab to detect\n4. **Shuffling bugs** — correlated batches\n5. **Tokenizer mismatch** with pretrained model\n\n### What to monitor\n\n- **Gradient norms** — spike precedes loss spike\n- **Per-layer activation stats** — reveals exploding\u002Fvanishing\n- **Dead neurons** — >50% zero ReLU = dying ReLU problem\n- **Learning rate** — verify schedule applied (common silent bug)\n\n---\n\n## Experiment Management\n\nTrack experiments in TSV for easy comparison:\n\n```\ncommit  val_bpb  memory_gb  status   description\na1b2c3d 0.9979   44.0       keep     baseline\nb2c3d4e 0.9932   44.2       keep     increase matrix LR to 0.04\nc3d4e5f 1.0050   44.0       discard  switch to GeLU (worse)\n```\n\n**Simplicity criterion**: all else equal, simpler is better. Removing something and getting equal\nresults is a great outcome. For systematic agent-driven experimentation, see `references\u002Fexperiment-loop.md`.\n\n### Evaluation metrics by domain\n\n| Domain | Primary Metric | Notes |\n|--------|---------------|-------|\n| LLM | BPB (bits per byte) | Vocab-size-independent |\n| Classification | Accuracy \u002F F1 | Macro-F1 for imbalanced |\n| Segmentation | mIoU \u002F Dice | Per-class IoU reveals weak spots |\n| Generation | FID | Needs >10k samples |\n| Regression | RMSE \u002F MAE | Log-transform skewed targets |\n\n## Related skills\n\n- `serving-llms-vllm`: Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching.\n- `llama-factory`: Guides fine-tuning of large language models with LLaMA-Factory, covering the WebUI no-code interface, training across 100+ supported models…\n- `peft-fine-tuning`: Fine-tunes LLMs with Hugging Face PEFT, using LoRA, QLoRA, IA3, AdaLoRA, prefix tuning, and prompt tuning so that under 1% of parameters ar…\n","# ML Training Recipes\n\nBattle-tested patterns for PyTorch training across domains. Drawn from production codebases\n(Karpathy's autoresearch\u002Fnanochat, torchvision, HuggingFace) and modern training practice.\n\n## Reference files (read when needed)\n\n- `references\u002Farchitecture.md` — Transformer\u002FLLM architecture code patterns, weight init\n- `references\u002Foptimizers.md` — Muon, AdamW hybrid, per-group LR, compiled optimizer steps\n- `references\u002Fdomain-specific.md` — Vision, diffusion, contrastive, distributed, checkpointing, data loading\n- `references\u002Fscaling-and-selection.md` — Scaling laws, compute budget tables, decision trees, DGX Spark\n- `references\u002Fbiomedical.md` — Drug discovery, protein models, medical imaging, genomics, clinical NLP\n- `references\u002Fexperiment-loop.md` — Autonomous experiment loop (autoresearch keep\u002Fdiscard\u002Frevert)\n\n---\n\n## Architecture Selection\n\nPick the right model by **data type** and **data scale**:\n\n| Data Type | \u003C 10K samples | 10K-100K | > 100K |\n|-----------|--------------|----------|--------|\n| **Images** | Pretrained CNN + fine-tune | Fine-tune ViT or CNN | ViT from scratch |\n| **Text (gen)** | Few-shot prompting | Fine-tune GPT\u002FLLaMA (LoRA) | Pretrain from scratch |\n| **Tabular** | XGBoost\u002FLightGBM | Still XGBoost | Neural viable |\n| **Audio** | Pretrained Whisper | Fine-tune AST | Train from scratch |\n| **Molecules** | Pretrained GNN | Fine-tune molecular LM | Train GNN from scratch |\n| **Proteins** | ESM-2 embeddings + head | Fine-tune ESM-2 | Train protein LM |\n| **Medical img** | Pretrained CNN | nnU-Net (auto-config) | Swin-UNETR \u002F MedSAM |\n\n**Key principle**: architecture matters less than training recipe at equal compute. A well-tuned\nResNet beats a poorly-tuned ViT (ref: \"ResNet Strikes Back\", Wightman 2021).\n\nFor biomedical domains, see `references\u002Fbiomedical.md`.\nFor sequence model selection and compute planning, see `references\u002Fscaling-and-selection.md`.\n\n---\n\n## Scaling Laws\n\n### Chinchilla rule (Hoffmann et al., 2022)\n\nCompute-optimal training: **~20 tokens per parameter**.\n\n| Model Size | Compute-Optimal | Inference-Optimal (100×) |\n|-----------|----------------|--------------------------|\n| 125M | 2.5B tokens | 12.5B tokens |\n| 1B | 20B tokens | 100B tokens |\n| 7B | 140B tokens | 700B tokens |\n\n**FLOPs ≈ 6 × N × D** (N=params, D=tokens). Data repetition limit: ~4 epochs before diminishing returns.\n\n---\n\n## Training Loop\n\n```python\nimport gc, time, torch\n\ntorch.manual_seed(42)\ntorch.set_float32_matmul_precision(\"high\")  # TF32 on Ampere+\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\n\ngrad_accum_steps = total_batch_size \u002F\u002F (batch_size * seq_len)\nstep = 0\n\nwhile not done:\n    t0 = time.time()\n    for micro_step in range(grad_accum_steps):\n        with autocast_ctx:\n            loss = model(x, y)\n        (loss \u002F grad_accum_steps).backward()\n        x, y = next(train_loader)\n\n    update_lr(optimizer, progress)\n    optimizer.step()\n    model.zero_grad(set_to_none=True)  # frees memory vs zeroing\n\n    if loss.item() > 100:  # fast-fail on divergence\n        print(\"FAIL: loss exploded\"); exit(1)\n\n    torch.cuda.synchronize()\n    if step == 0:\n        gc.collect(); gc.freeze(); gc.disable()  # avoid ~500ms GC stalls\n    step += 1\n```\n\n### Key principles\n\n- **Gradient clipping**: `clip_grad_norm_(params, 1.0)` — near-universal for Transformers.\n  Exception: Muon optimizer normalizes updates via orthogonalization, so clipping is optional.\n- **Tensor Core alignment**: batch size, hidden dims should be multiples of 8 (bf16) or 64 (A100).\n- **Time-based budgets** make experiments comparable across hardware.\n- **`cudnn.benchmark = True`** for fixed-size vision inputs.\n\n---\n\n## Optimizer Configuration\n\nModern LLM training uses different optimizers per parameter group:\n\n| Parameter Type | Optimizer | LR (base) | Weight Decay |\n|---------------|-----------|-----------|--------------|\n| 2D weight matrices | Muon | 0.04 | 0.2 |\n| Token embeddings | AdamW | 0.6 × scale | 0.0 |\n| Unembedding (lm_head) | AdamW | 0.004 × scale | 0.0 |\n| Per-layer scalars | AdamW | 0.005 × scale | 0.0 |\n\n**LR scaling by dimension**: `lr * (d_model \u002F 768)^(-0.5)` — keeps dynamics stable across sizes.\n\n### Rules of thumb\n\n- Embeddings need higher LR (sparse updates). Never weight-decay embeddings.\n- Weight decay scheduling: linearly decay WD to 0 over training.\n- AdamW defaults: β1=0.9, β2=0.95, eps=1e-10 (not default 1e-8 — prevents stale updates in bf16).\n\nFor Muon details (polar express orthogonalization, NorMuon), see `references\u002Foptimizers.md`.\n\n---\n\n## Learning Rate Scheduling\n\n### Time-based (autoresearch style)\n\n```python\ndef get_lr_multiplier(progress):  # progress = elapsed_time \u002F time_budget\n    if progress \u003C warmup_ratio:\n        return progress \u002F warmup_ratio\n    elif progress \u003C 1.0 - warmdown_ratio:\n        return 1.0\n    else:\n        cooldown = (1.0 - progress) \u002F warmdown_ratio\n        return cooldown + (1 - cooldown) * final_lr_frac\n```\n\n### Cosine decay\n\n```python\ndef get_lr(step, total_steps, max_lr, min_lr, warmup_steps):\n    if step \u003C warmup_steps:\n        return max_lr * step \u002F warmup_steps\n    progress = (step - warmup_steps) \u002F (total_steps - warmup_steps)\n    return min_lr + 0.5 * (max_lr - min_lr) * (1 + math.cos(math.pi * progress))\n```\n\n**WSD (Warmup-Stable-Decay)**: gaining traction — easier to resume training mid-run.\n\n### Guidance\n\n- **Warmup**: 1-5% of training. Zero warmup valid with Muon (autoresearch uses `WARMUP_RATIO=0.0`).\n- **Warmdown**: 30-50% of training in LR decay. Matters more than warmup for final quality.\n- **Final LR**: 0 or ~10% of peak. Zero is simpler.\n\n---\n\n## Mixed Precision & Compilation\n\n```python\nimport os\nos.environ[\"PYTORCH_ALLOC_CONF\"] = \"expandable_segments:True\"  # before torch import\n\nimport torch\ntorch.set_float32_matmul_precision(\"high\")\nautocast_ctx = torch.amp.autocast(device_type=\"cuda\", dtype=torch.bfloat16)\nmodel = torch.compile(model, dynamic=False)\n```\n\n- **bf16** (Ampere+): same exponent as fp32, no loss scaling needed. Preferred over fp16.\n- **fp16**: needs GradScaler. Use only on V100 or older.\n- `dynamic=False` enables max optimization. Add `fullgraph=True` if no graph breaks.\n- First steps are slow (JIT) — exclude from timing.\n\n---\n\n## Memory & Performance\n\n### Meta device init (large models)\n\n```python\nwith torch.device(\"meta\"):\n    model = GPT(config)          # zero memory\nmodel.to_empty(device=\"cuda\")\nmodel.init_weights()\n```\n\n### MFU (Model FLOPs Utilization)\n\n```python\nachieved_flops = model_flops_per_token * batch_tokens \u002F step_time\nmfu = achieved_flops \u002F gpu_peak_flops\n# H100 SXM: 989.5 TFLOPS | A100: 312 | RTX 4090: 165\n```\n\nGood targets: >30% decent, >40% good, >50% excellent (single-GPU).\n\n### OOM solutions (in order)\n\n1. Reduce `DEVICE_BATCH_SIZE`, increase `grad_accum_steps`\n2. `PYTORCH_ALLOC_CONF=expandable_segments:True`\n3. `model.zero_grad(set_to_none=True)`\n4. Meta device init → `to_empty`\n5. Activation checkpointing: `torch.utils.checkpoint.checkpoint()`\n6. 8-bit optimizer (bitsandbytes): ~30% savings on optimizer states\n\n---\n\n## Hyperparameter Search\n\n### Priority order (tune first → last)\n\n1. **Learning rate** — most impactful. Always tune first.\n2. **Batch size** — largest that fits. Speed knob, not quality knob.\n3. **Weight decay** — 0.01-0.1 for AdamW.\n4. **Warmup steps** — 1-5% of training.\n\n### The 2025 default recipe\n\n| Setting | Value |\n|---------|-------|\n| Optimizer | AdamW (β1=0.9, β2=0.95, eps=1e-10) |\n| Weight decay | 0.1 |\n| LR schedule | Cosine decay or WSD |\n| Peak LR | 3e-4 (scale down for larger models) |\n| Precision | bf16 |\n| Grad clipping | max_norm=1.0 |\n| Normalization | RMSNorm (pre-norm) |\n| Activation | SwiGLU |\n| Position encoding | RoPE |\n| Attention | Flash Attention, optionally GQA |\n\n---\n\n## Debugging Checklist\n\n### Karpathy's recipe (still canonical)\n\n1. **Become one with the data** — visualize, check distributions, verify labels\n2. **Get end-to-end running first** — verify on a trivial case\n3. **Overfit one batch** — if you can't, you have a bug\n4. **Then regularize** — add regularization only after overfitting works\n5. **Tune hyperparameters** — start with known defaults\n\n### Loss exploding \u002F NaN\n\n1. Reduce LR (3-10× smaller)\n2. Add gradient clipping: `clip_grad_norm_(params, 1.0)`\n3. Check for inf\u002Fnan in inputs\n4. Add logit soft capping: `softcap * tanh(logits \u002F softcap)`\n5. Add QK-norm in attention\n6. Verify weight init (zero-init output projections?)\n7. Check loss reduction with gradient accumulation (`loss \u002F grad_accum_steps`)\n\n### Slow training \u002F Low MFU\n\n1. Verify `torch.compile` is active\n2. Check `torch.set_float32_matmul_precision(\"high\")`\n3. Pin memory + non_blocking transfers\n4. Profile with `torch.profiler`\n5. GC stalls? `gc.freeze(); gc.disable()`\n6. Tensor Core alignment: dims multiples of 8\u002F64\n\n### Loss plateau \u002F Slow convergence\n\n1. LR too low — try 2-5× larger\n2. Warmup too long\n3. Weight decay too high\n4. Verify LR schedule is actually applied (print each step)\n5. Model too small for task\n\n### Silent failures\n\n1. **Data leakage** between train\u002Fval\n2. **Wrong preprocessing at inference** — augmentation mismatch\n3. **Label errors** — use cleanlab to detect\n4. **Shuffling bugs** — correlated batches\n5. **Tokenizer mismatch** with pretrained model\n\n### What to monitor\n\n- **Gradient norms** — spike precedes loss spike\n- **Per-layer activation stats** — reveals exploding\u002Fvanishing\n- **Dead neurons** — >50% zero ReLU = dying ReLU problem\n- **Learning rate** — verify schedule applied (common silent bug)\n\n---\n\n## Experiment Management\n\nTrack experiments in TSV for easy comparison:\n\n```\ncommit  val_bpb  memory_gb  status   description\na1b2c3d 0.9979   44.0       keep     baseline\nb2c3d4e 0.9932   44.2       keep     increase matrix LR to 0.04\nc3d4e5f 1.0050   44.0       discard  switch to GeLU (worse)\n```\n\n**Simplicity criterion**: all else equal, simpler is better. Removing something and getting equal\nresults is a great outcome. For systematic agent-driven experimentation, see `references\u002Fexperiment-loop.md`.\n\n### Evaluation metrics by domain\n\n| Domain | Primary Metric | Notes |\n|--------|---------------|-------|\n| LLM | BPB (bits per byte) | Vocab-size-independent |\n| Classification | Accuracy \u002F F1 | Macro-F1 for imbalanced |\n| Segmentation | mIoU \u002F Dice | Per-class IoU reveals weak spots |\n| Generation | FID | Needs >10k samples |\n| Regression | RMSE \u002F MAE | Log-transform skewed targets |\n\n## Related skills\n\n- `serving-llms-vllm`: Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching.\n- `llama-factory`: Guides fine-tuning of large language models with LLaMA-Factory, covering the WebUI no-code interface, training across 100+ supported models…\n- `peft-fine-tuning`: Fine-tunes LLMs with Hugging Face PEFT, using LoRA, QLoRA, IA3, AdaLoRA, prefix tuning, and prompt tuning so that under 1% of parameters ar…\n",{"name":5,"license":15,"metadata":45,"description":7},{"tags":46,"version":47,"category":48,"maintainer":49,"contributor":50,"dependencies":51},"PyTorch, Training, Optimization, LLM, Vision, Diffusion, Biomedical, Muon, AdamW, Debugging","1.0.0","ml-training","Kalaris Labs","dailycafi","torch>=2.0.0","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes",468,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-11 02:22:54",{"source":8,"skillName":5,"commands":63,"prompt":73,"zipAvailable":74},[64,67,70],{"tool":65,"command":66},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill ml-training-recipes",{"tool":68,"command":69},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill ml-training-recipes",{"tool":71,"command":72},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill ml-training-recipes","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes\u002FSKILL.md",true,[76,82,86,90,94,98],{"path":77,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Farchitecture.md","REFERENCE","参考文档",10204,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Farchitecture.md",{"path":83,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":84,"sourceHtmlUrl":85},"references\u002Fbiomedical.md",20778,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Fbiomedical.md",{"path":87,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Fdomain-specific.md",19018,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Fdomain-specific.md",{"path":91,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":92,"sourceHtmlUrl":93},"references\u002Fexperiment-loop.md",4446,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Fexperiment-loop.md",{"path":95,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":96,"sourceHtmlUrl":97},"references\u002Foptimizers.md",10564,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Foptimizers.md",{"path":99,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":100,"sourceHtmlUrl":101},"references\u002Fscaling-and-selection.md",16832,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fml-training-recipes\u002Freferences\u002Fscaling-and-selection.md",[103,131,156,190,209,223],{"skillNo":104,"name":105,"nameZh":6,"description":106,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":107,"licenseFileUrl":6,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":40},"01M716A6ETNB9B1SQ3GXZC75VR","deja-search","deja-vu memory — search the user's past AI coding sessions with the deja CLI. Holds what they already settled and never wrote into the repo (naming rules, banned or already-tried tools, required checks before merge, preferred values, who owns what). Use before your first edit in a task, before you call a change done or ready to merge, before you recommend a tool, library, design or reviewer, before re-debugging an error that may already be solved, when they say 'didn't we fix this before', 'what did we decide about X' or 'I already have that', and before telling them something on this machine does not exist.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fvshulcz\u002Fdeja-vu\u002Fskills\u002Fdeja-search\u002FSKILL.md",[],[111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130],"search","deja search","error","ready","never","memory","re","decide","fix","design","debugging","when","you","done","reviewer","io","change","cli","merge","past",{"skillNo":132,"name":133,"nameZh":6,"description":134,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":107,"licenseFileUrl":6,"sourceHtmlUrl":135,"categories":136,"keywords":137,"publishedAt":40},"01M716A3J2W5SENSE9VGMXB375","deja-reflect","End-of-session reflection and memory curation with deja-vu. Review what was decided, what commands broke and what repaired them, and record durable project rules or decisions with deja remember so the next session starts with them.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fvshulcz\u002Fdeja-vu\u002Fskills\u002Fdeja-reflect\u002FSKILL.md",[],[138,139,140,141,116,142,143,144,145,146,147,148,149,126,150,151,152,153,154,155],"reflect","deja reflect","brew","remember","project rules","project","session","end","reflection","Rules","decisions","review","with","record","github","next","https","node",{"skillNo":157,"name":158,"nameZh":6,"description":159,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":107,"licenseFileUrl":6,"sourceHtmlUrl":160,"categories":161,"keywords":173,"publishedAt":40},"01M715T3RR4BAYSR57HPPTJ0PF","revid","Make, edit, export and post AI videos with Revid (revid.ai) through the Revid MCP server. Use whenever the user wants a short-form or long-form video made (from an idea, a script, a URL, an article, a PDF, a tweet, a product page, their own clips, or a song), wants to find what is going viral on TikTok, YouTube, Instagram or X before making something, wants to edit or re-export an existing Revid project, generate a standalone AI image or clip, make a series episode, or publish and schedule a finished video to connected social accounts. Covers setup (hosted MCP with OAuth, or an API key), the 36 tools, credits, polling and the rules for publishing. Routes to the more specific revid-* skills when one fits.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftmakersas\u002Frevid-skills\u002Fskills\u002Frevid\u002FSKILL.md",[162,167,170],{"categoryNo":163,"categoryName":164,"taxonomyType":165,"taxonomyTypeDesc":166},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",{"categoryNo":168,"categoryName":169,"taxonomyType":165,"taxonomyTypeDesc":166},"01JBSEEDCAT000000000000016","图像与多媒体",{"categoryNo":171,"categoryName":172,"taxonomyType":165,"taxonomyTypeDesc":166},"01JBSEEDCAT000000000000023","Skill与智能体开发",[158,174,175,176,177,178,143,179,117,180,181,182,183,122,184,185,186,187,188,189],"Video","short-form","standalone","PDF","viral","tiktok","short","social","MCP","song","URL","image","mcp server","schedule","api","publish",{"skillNo":191,"name":192,"nameZh":6,"description":193,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":107,"licenseFileUrl":6,"sourceHtmlUrl":194,"categories":195,"keywords":196,"publishedAt":40},"01M71569ZM2X3H5RBFK8PVKMK2","review-study","Review a QuestionPunk survey before launch for wording, answer choices, branching, and missing settings. Use for check my survey or review this draft before publishing.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fquestionpunk\u002Fcodex-plugin\u002Fskills\u002Freview-study\u002FSKILL.md",[],[197,198,149,25,199,200,201,202,35,203,204,205,206,207,208],"study","review study","answer","missing","survey","publishing","check","draft","settings","launch","my","branching",{"skillNo":210,"name":211,"nameZh":6,"description":212,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":107,"licenseFileUrl":6,"sourceHtmlUrl":213,"categories":214,"keywords":215,"publishedAt":40},"01M71567GV0C35YQBY0DPGR86B","manage-study","Find QuestionPunk studies, check status, publish an explicitly requested draft, or retrieve its participant link. Use for find my survey, publish this study, or how many people responded.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fquestionpunk\u002Fcodex-plugin\u002Fskills\u002Fmanage-study\u002FSKILL.md",[],[216,197,217,25,201,35,218,203,204,189,219,220,221,222,207],"manage study","manage","people","how","find","link","status",{"skillNo":224,"name":225,"nameZh":6,"description":226,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":107,"licenseFileUrl":6,"sourceHtmlUrl":227,"categories":228,"keywords":229,"publishedAt":40},"01M71565A8TCA01AZQZRGRYMFZ","create-study","Create a saved QuestionPunk survey draft from a research brief or edit specified questions. Use for requests such as make a customer feedback survey or shorten my onboarding study.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fquestionpunk\u002Fcodex-plugin\u002Fskills\u002Fcreate-study\u002FSKILL.md",[],[230,197,231,232,233,234,25,235,201,236,35,237,238,204,239,240,241,242,243,207],"create study","create","brief","as","feedback","customer feedback","research brief","shorten","saved","research","requests","customer","questions","edit"]