[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7APJ94JGVH4TCRY6BTKWH9B":3,"skill-same:01M7APJ94JGVH4TCRY6BTKWH9B":48},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":29,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M7APJ94JGVH4TCRY6BTKWH9B","ml-deep-learning-and-training",null,"Use when building or training neural networks. Covers the machinery (autodiff, initialization, normalization, regularization), optimization and schedules, the things that make training work, the updated bias-variance picture, transformers and the rest (CNNs, diffusion, SSMs), PyTorch and torch.compile, JAX and the rest of the stack, training at scale (the parallelism taxonomy — DDP, FSDP, tensor and pipeline parallelism), mixed precision and memory, and fine-tuning and adaptation (full fine-tuning, PEFT\u002FLoRA).","the-vibey-project\u002Fvibey","the-vibey-project","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F329298758?v=4","01M7AJHGZ9722CTJ4JNQZVRH0Q","the vibey project","01M7AJHGZVMG2K7TGXQ6XFHVG3",5,"MIT","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fmachine-learning\u002Fskills\u002Fml-deep-learning-and-training\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: ml-deep-learning-and-training\ndescription: \"Use when building or training neural networks. Covers the machinery (autodiff, initialization, normalization, regularization), optimization and schedules, the things that make training work, the updated bias-variance picture, transformers and the rest (CNNs, diffusion, SSMs), PyTorch and torch.compile, JAX and the rest of the stack, training at scale (the parallelism taxonomy — DDP, FSDP, tensor and pipeline parallelism), mixed precision and memory, and fine-tuning and adaptation (full fine-tuning, PEFT\u002FLoRA).\"\n---\n\n# Machine Learning: Deep Learning, Architectures, the Ecosystem, and Training at Scale\n\n> **Part 2 of 4** of the *Machine Learning* reference (plugin `machine-learning`), covering §4–§8. Sibling skills: `ml-framing-data-and-classical` (§0–§3), `ml-evaluation-serving-mlops-and-safety` (§9–§14), `ml-reference` (§15–§20). Section numbers are shared across the set; a reference written as §N → `skill` points into that sibling skill.\n>\n> **Currency:** Verified August 2026. See §17 → `ml-reference` for the currency snapshot and what goes stale first.\n\n> **How to read this.** Reference, not tutorial. Sections are independent. Three markers:\n> - **[DURABLE]** — statistics, optimization theory, or a lesson the field has relearned\n>   every few years. Does not expire.\n> - **[VERSIONED]** — framework versions, hardware, library APIs. Moving fast; verify.\n> - **[CONTESTED]** — practitioners genuinely disagree.\n>\n> **⚠️ GOTCHA** boxes mark the mistakes that produce a model that looks great offline and\n> fails in production — which is the characteristic failure of this field.\n>\n> **The three framings that organize everything below:**\n> 1. **Your data is the model.** Architecture choices matter far less than data quality,\n>    quantity, and whether your evaluation measures the thing you care about. Most time on\n>    a real project goes to data, and most catastrophic failures originate there (§2 → `ml-framing-data-and-classical`).\n> 2. **The default failure mode is a number that lies.** Leakage, distribution shift, a\n>    metric that doesn't match the objective, a test set you tuned against. **A model that\n>    looks too good almost always is** — and finding out why is the core skill.\n> 3. **Almost everything you'll build should not be a neural network.** For tabular data —\n>    which is most business data — gradient-boosted trees remain the strongest default\n>    (§3.1 → `ml-framing-data-and-classical`). Reach for deep learning when you have perceptual data, sequences, very large\n>    datasets, or a pretrained model to adapt.\n\n---\n\n## §4. Deep Learning Fundamentals\n\n### 4.1 The machinery\n\n```\nforward pass  →  loss  →  BACKPROP (reverse-mode autodiff)  →  optimizer step\n                              ↑\n                    the chain rule, applied over a computation graph\n```\n**[DURABLE] Backpropagation is reverse-mode automatic differentiation**, not a separate\nalgorithm. Understanding it as autodiff over a graph explains why memory scales with\nactivations (you must keep intermediates for the backward pass), why gradient checkpointing\nworks (recompute instead of store), and why `detach()`\u002F`stop_gradient` does what it does.\n\n### 4.2 Optimization\n\n| Optimizer | Notes |\n|---|---|\n| **SGD + momentum** | Still competitive for vision; often generalizes better than adaptive methods |\n| **Adam \u002F AdamW** | **The default.** ⚠️ **Use AdamW, not Adam, whenever you use weight decay** — Adam's L2 penalty is not equivalent to weight decay, and this measurably hurts |\n| **Adafactor \u002F 8-bit Adam** | Memory-efficient for large models — optimizer state is often the largest memory consumer (§7.3) |\n| **Muon, Shampoo, second-order** | Active area; real wins reported at scale, less settled than the marketing |\n\n**Learning rate is the hyperparameter that matters most, by a wide margin.** Practices that\nconsistently pay: **warmup** (especially with transformers and large batches), **cosine or\nlinear decay**, and an **LR-range test** to find the scale. **[DURABLE] If you can tune only\none thing, tune the learning rate.**\n\n**Batch size** interacts with LR (linear scaling rule as a starting heuristic), affects\ngeneralization, and is often set by memory rather than by choice. **Gradient accumulation**\nsimulates a large batch on small hardware.\n\n### 4.3 The things that make training work\n\n- **Initialization** — Xavier\u002FGlorot for tanh-family, **He\u002FKaiming for ReLU-family**.\n  Getting this wrong causes vanishing or exploding activations before you write a single\n  training loop.\n- **Normalization** — **BatchNorm** (⚠️ batch-size dependent, and it behaves differently in\n  train vs. eval — a classic bug source), **LayerNorm** (the transformer standard),\n  **RMSNorm** (cheaper, now widely preferred in LLMs), GroupNorm.\n- **Residual connections** — the innovation that made deep networks trainable at all.\n- **Activations** — ReLU, GELU, **SiLU\u002FSwish**, and **SwiGLU** (the modern LLM FFN default).\n- **Regularization** — weight decay, dropout (⚠️ largely fallen out of favour in large\n  transformers), early stopping, data augmentation (**the most effective regularizer in\n  vision**), label smoothing, and mixup\u002Fcutmix.\n- **Gradient clipping** — by global norm. Cheap insurance against loss spikes.\n\n### 4.4 The bias-variance picture, updated\n\n**[DURABLE, but the classical story is incomplete.]** The textbook U-curve — underfit,\nsweet spot, overfit — is real for classical models. **Double descent** complicates it:\npast the interpolation threshold, test error can *decrease again* as you keep adding\ncapacity, which is part of why enormously overparameterized networks work at all.\n**Practical implication: \"the model is too big, it will overfit\" is not a reliable\nargument for deep networks**, and regularization plus data scale matters more than\nparameter count.\n\n---\n\n## §5. Architectures\n\n### 5.1 Transformers\n\n**[DURABLE] The dominant architecture, and worth understanding mechanically rather than by\nanalogy.**\n```\ntokens → embeddings + POSITIONAL INFORMATION\n  → N × [ self-attention  →  residual+norm  →  FFN  →  residual+norm ]\n    → output head\n\nattention(Q,K,V) = softmax(QKᵀ\u002F√d_k) V\n```\nKey points: **multi-head attention** runs several attention patterns in parallel;\nattention is **O(n²)** in sequence length, which is the central scaling problem; the\n**FFN holds most of the parameters**; and **causal masking** is what makes a decoder\nautoregressive.\n\n**The modern variants you'll meet**: **RoPE** (rotary position embeddings — now standard),\n**GQA \u002F MQA** (grouped\u002Fmulti-query attention — fewer KV heads, **dramatically smaller KV\ncache**, §11.2 → `ml-evaluation-serving-mlops-and-safety`), **FlashAttention** (IO-aware exact attention — not an approximation; it\navoids materializing the n² matrix in HBM), **MoE** (mixture of experts — many parameters,\nfew active per token), **sliding-window** and other sparse attention, and **SSMs\u002FMamba**\nas the main non-attention contender.\n\n### 5.2 The rest\n\n**CNNs** — still the right choice for many vision tasks, especially with limited data and\ncompute; ConvNeXt showed a modernized CNN matches ViTs. Convolution's inductive bias\n(locality, translation equivariance) is a *feature* when data is scarce.\n**Vision Transformers** — win at scale, need more data or heavy augmentation.\n**Diffusion models** — the generative image\u002Fvideo\u002Faudio default; iterative denoising, with\nflow matching as the cleaner modern formulation.\n**GNNs** — for genuinely graph-structured data. ⚠️ Often beaten by feature engineering plus\na GBDT; verify the graph structure actually carries signal.\n**Encoder-decoder vs. decoder-only** — decoder-only won for general LLMs; encoder models\n(BERT-family) remain excellent and much cheaper for classification and retrieval, and are\nbadly underused because attention moved elsewhere.\n**Embedding models** — the workhorse of retrieval, RAG, semantic search, and dedup, and\noften the highest-value-per-FLOP thing you can deploy.\n\n---\n\n## §6. The Ecosystem\n\n### 6.1 PyTorch\n\n**[VERSIONED] PyTorch 2.13 is current (July 2026)**, with a roughly quarterly cadence.\nThe parts that matter:\n\n**`torch.compile`** — the headline feature since 2.0. Traces your model (TorchDynamo),\nlowers it (TorchInductor), and generates fused Triton (or now **CuTeDSL**) kernels.\n**Typically a substantial speedup for free.** ⚠️ **Watch for graph breaks** — Python\nconstructs Dynamo can't trace force it to fall back, silently costing you the speedup;\n`torch._dynamo.error_on_graph_break()` makes that loud rather than silent.\n**Recompilation** on changing shapes is the other tax — mark dynamic dimensions explicitly.\n\n**Other things worth knowing**: `torch.export` is the unified capture path (legacy\n`torch.jit` and old ONNX flows are deprecated or removed); **`weights_only=True` is the\ndefault for `torch.load` since 2.6** — a genuine security improvement, since pickle\ndeserialization is arbitrary code execution; **FlexAttention** for custom attention\npatterns without writing kernels; **AOTInductor** for ahead-of-time compilation.\n\n### 6.2 The rest of the stack\n\n| Layer | Tools |\n|---|---|\n| **Frameworks** | **PyTorch** (dominant in research and increasingly production), **JAX** (functional, `jit`\u002F`grad`\u002F`vmap`\u002F`pmap`, strong on TPU, favored for large-scale research), TensorFlow\u002FKeras (legacy-heavy but alive), **MLX** (Apple Silicon) |\n| **Training loops** | **Lightning**, **HF Accelerate**, **torchtune**, raw loops (fine, and often clearer) |\n| **Models and data** | **Hugging Face** `transformers`, `datasets`, `tokenizers`, `peft`, `trl` — effectively the industry's default distribution channel |\n| **Classical** | scikit-learn, XGBoost, LightGBM, CatBoost, statsmodels |\n| **Numerics** | NumPy, pandas, **Polars** (faster, saner API, increasingly the default for new work), DuckDB, Arrow |\n| **Kernels** | **Triton** (write GPU kernels in Python), CUDA, FlashAttention, **FlashInfer** |\n| **Experiment tracking** | Weights & Biases, MLflow, TensorBoard, Aim |\n| **Serving** | **vLLM**, **SGLang**, TensorRT-LLM, Triton Inference Server, Ray Serve, llama.cpp\u002FOllama (§11 → `ml-evaluation-serving-mlops-and-safety`) |\n| **Orchestration** | Ray, Kubeflow, Airflow, Prefect, SLURM (still the HPC standard) |\n\n**[DURABLE] Pin your versions and record them.** The ML stack breaks compatibility\nconstantly, and \"it worked last month\" is a real and frequent failure.\n\n---\n\n## §7. Training at Scale\n\n### 7.1 The parallelism taxonomy\n\n| Strategy | Splits | Use when |\n|---|---|---|\n| **Data parallel (DDP)** | The batch | **The default.** Model fits on one GPU |\n| **FSDP \u002F ZeRO** | Parameters, gradients, optimizer state | Model doesn't fit. **The mainstream large-model answer** |\n| **Tensor parallel** | Individual matrices, within a layer | Very large layers; needs fast interconnect (NVLink) |\n| **Pipeline parallel** | Layers across devices | Very deep models; ⚠️ introduces bubbles |\n| **Expert parallel** | MoE experts | MoE models |\n| **Context\u002Fsequence parallel** | The sequence dimension | Very long context |\n\n**[DURABLE] Real large-scale training composes several of these** (\"3D parallelism\" and\nbeyond), and the composition is chosen against your interconnect topology, not in the\nabstract. **Communication is usually the bottleneck**, which is why NVLink\u002FInfiniBand\ntopology drives the design.\n\n**[VERSIONED]** PyTorch's **FSDP2** now supports overlapping reduce-scatter and all-gather\nvia separate process groups (opt-in), which increases throughput; and **torchcomms** is a\nnew communications backend for PyTorch Distributed aimed at fault tolerance, scalability,\nand debuggability on large clusters — with plans to make it the default, including\nbreaking changes to how ProcessGroups operate (eager initialization, single backend\ndevice). **If you run large distributed jobs, that migration is on your horizon.**\n\n### 7.2 Mixed precision\n\n**[DURABLE]** Train in low precision, keep a master copy and accumulations in higher\nprecision.\n- **FP16** — needs **loss scaling** (gradients underflow otherwise).\n- **BF16** — same exponent range as FP32, less mantissa. **No loss scaling needed. The\n  default on modern hardware, and the right choice.**\n- **FP8** — real on Hopper and Blackwell; needs careful scaling; increasingly used in\n  production training.\n- **[VERSIONED] NVFP4 \u002F MXFP4** — 4-bit formats with hardware support on Blackwell,\n  now appearing in both training and inference paths.\n\n### 7.3 Memory\n\n**[DURABLE] Know where the memory goes**, because \"CUDA out of memory\" is the most common\nobstacle in practice:\n```\nparameters  +  gradients  +  OPTIMIZER STATE  +  activations  +  fragmentation\n                                    ↑\n              Adam keeps 2 extra copies — often the largest single term\n```\nA rough anchor: **fp32 Adam training costs roughly 16 bytes per parameter** (4 param +\n4 grad + 8 optimizer state), before activations. That's why a 7B model doesn't fit on an\n80 GB card without help.\n\n**The levers**: **gradient checkpointing** (recompute activations — trades ~30% compute for\nlarge memory savings), **gradient accumulation**, **8-bit or fused optimizers**, **FSDP\nsharding**, **`nn.LinearCrossEntropyLoss`** (a 2026 PyTorch addition fusing the final\nprojection and loss — **cuts peak memory by up to 4× for large-vocabulary LM training**,\nwhere the logits tensor is genuinely enormous), and simply **reducing sequence length**.\n\n---\n\n## §8. Fine-Tuning and Adaptation\n\n**[DURABLE] The order to try things, cheapest first:**\n```\n1. Prompting \u002F few-shot                  ← no training. Try this first, seriously\n2. Retrieval (RAG)                       ← when the problem is missing knowledge\n3. PEFT \u002F LoRA                           ← when the problem is behaviour or format\n4. Full fine-tuning                      ← when you have real data and real budget\n5. Continued pretraining                 ← new domain, large corpus\n6. Training from scratch                 ← almost never the right answer\n```\n**⚠️ The most common expensive mistake is fine-tuning to fix a knowledge problem.**\nFine-tuning teaches *behaviour, style, and format* reliably; it teaches *facts* poorly and\nexpensively. **If the model doesn't know something, retrieve it.**\n\n**LoRA** — freeze the base, train low-rank adapters (`W + BA`). Tiny memory footprint,\nswappable adapters, near-full-fine-tuning quality on most tasks. **QLoRA** adds 4-bit\nquantization of the frozen base, putting single-GPU fine-tuning of large models within\nreach. Other PEFT: prefix tuning, prompt tuning, IA³, DoRA.\n\n**Post-training \u002F alignment**: **SFT** (supervised fine-tuning on demonstrations), then\npreference optimization — **RLHF\u002FPPO** (powerful, complex, unstable), **DPO** (much\nsimpler, no reward model, now the common default), and GRPO and relatives for reasoning\nwork. **[DURABLE] Data quality dominates method choice here** — a small, carefully curated\nSFT set routinely beats a large noisy one.\n\n**⚠️ Catastrophic forgetting is real.** Fine-tuning on a narrow task degrades general\ncapability. Mix in general data, use lower learning rates, prefer PEFT, and **evaluate on\ncapabilities you didn't train on** — not just the target task.\n","# Machine Learning: Deep Learning, Architectures, the Ecosystem, and Training at Scale\n\n> **Part 2 of 4** of the *Machine Learning* reference (plugin `machine-learning`), covering §4–§8. Sibling skills: `ml-framing-data-and-classical` (§0–§3), `ml-evaluation-serving-mlops-and-safety` (§9–§14), `ml-reference` (§15–§20). Section numbers are shared across the set; a reference written as §N → `skill` points into that sibling skill.\n>\n> **Currency:** Verified August 2026. See §17 → `ml-reference` for the currency snapshot and what goes stale first.\n\n> **How to read this.** Reference, not tutorial. Sections are independent. Three markers:\n> - **[DURABLE]** — statistics, optimization theory, or a lesson the field has relearned\n>   every few years. Does not expire.\n> - **[VERSIONED]** — framework versions, hardware, library APIs. Moving fast; verify.\n> - **[CONTESTED]** — practitioners genuinely disagree.\n>\n> **⚠️ GOTCHA** boxes mark the mistakes that produce a model that looks great offline and\n> fails in production — which is the characteristic failure of this field.\n>\n> **The three framings that organize everything below:**\n> 1. **Your data is the model.** Architecture choices matter far less than data quality,\n>    quantity, and whether your evaluation measures the thing you care about. Most time on\n>    a real project goes to data, and most catastrophic failures originate there (§2 → `ml-framing-data-and-classical`).\n> 2. **The default failure mode is a number that lies.** Leakage, distribution shift, a\n>    metric that doesn't match the objective, a test set you tuned against. **A model that\n>    looks too good almost always is** — and finding out why is the core skill.\n> 3. **Almost everything you'll build should not be a neural network.** For tabular data —\n>    which is most business data — gradient-boosted trees remain the strongest default\n>    (§3.1 → `ml-framing-data-and-classical`). Reach for deep learning when you have perceptual data, sequences, very large\n>    datasets, or a pretrained model to adapt.\n\n---\n\n## §4. Deep Learning Fundamentals\n\n### 4.1 The machinery\n\n```\nforward pass  →  loss  →  BACKPROP (reverse-mode autodiff)  →  optimizer step\n                              ↑\n                    the chain rule, applied over a computation graph\n```\n**[DURABLE] Backpropagation is reverse-mode automatic differentiation**, not a separate\nalgorithm. Understanding it as autodiff over a graph explains why memory scales with\nactivations (you must keep intermediates for the backward pass), why gradient checkpointing\nworks (recompute instead of store), and why `detach()`\u002F`stop_gradient` does what it does.\n\n### 4.2 Optimization\n\n| Optimizer | Notes |\n|---|---|\n| **SGD + momentum** | Still competitive for vision; often generalizes better than adaptive methods |\n| **Adam \u002F AdamW** | **The default.** ⚠️ **Use AdamW, not Adam, whenever you use weight decay** — Adam's L2 penalty is not equivalent to weight decay, and this measurably hurts |\n| **Adafactor \u002F 8-bit Adam** | Memory-efficient for large models — optimizer state is often the largest memory consumer (§7.3) |\n| **Muon, Shampoo, second-order** | Active area; real wins reported at scale, less settled than the marketing |\n\n**Learning rate is the hyperparameter that matters most, by a wide margin.** Practices that\nconsistently pay: **warmup** (especially with transformers and large batches), **cosine or\nlinear decay**, and an **LR-range test** to find the scale. **[DURABLE] If you can tune only\none thing, tune the learning rate.**\n\n**Batch size** interacts with LR (linear scaling rule as a starting heuristic), affects\ngeneralization, and is often set by memory rather than by choice. **Gradient accumulation**\nsimulates a large batch on small hardware.\n\n### 4.3 The things that make training work\n\n- **Initialization** — Xavier\u002FGlorot for tanh-family, **He\u002FKaiming for ReLU-family**.\n  Getting this wrong causes vanishing or exploding activations before you write a single\n  training loop.\n- **Normalization** — **BatchNorm** (⚠️ batch-size dependent, and it behaves differently in\n  train vs. eval — a classic bug source), **LayerNorm** (the transformer standard),\n  **RMSNorm** (cheaper, now widely preferred in LLMs), GroupNorm.\n- **Residual connections** — the innovation that made deep networks trainable at all.\n- **Activations** — ReLU, GELU, **SiLU\u002FSwish**, and **SwiGLU** (the modern LLM FFN default).\n- **Regularization** — weight decay, dropout (⚠️ largely fallen out of favour in large\n  transformers), early stopping, data augmentation (**the most effective regularizer in\n  vision**), label smoothing, and mixup\u002Fcutmix.\n- **Gradient clipping** — by global norm. Cheap insurance against loss spikes.\n\n### 4.4 The bias-variance picture, updated\n\n**[DURABLE, but the classical story is incomplete.]** The textbook U-curve — underfit,\nsweet spot, overfit — is real for classical models. **Double descent** complicates it:\npast the interpolation threshold, test error can *decrease again* as you keep adding\ncapacity, which is part of why enormously overparameterized networks work at all.\n**Practical implication: \"the model is too big, it will overfit\" is not a reliable\nargument for deep networks**, and regularization plus data scale matters more than\nparameter count.\n\n---\n\n## §5. Architectures\n\n### 5.1 Transformers\n\n**[DURABLE] The dominant architecture, and worth understanding mechanically rather than by\nanalogy.**\n```\ntokens → embeddings + POSITIONAL INFORMATION\n  → N × [ self-attention  →  residual+norm  →  FFN  →  residual+norm ]\n    → output head\n\nattention(Q,K,V) = softmax(QKᵀ\u002F√d_k) V\n```\nKey points: **multi-head attention** runs several attention patterns in parallel;\nattention is **O(n²)** in sequence length, which is the central scaling problem; the\n**FFN holds most of the parameters**; and **causal masking** is what makes a decoder\nautoregressive.\n\n**The modern variants you'll meet**: **RoPE** (rotary position embeddings — now standard),\n**GQA \u002F MQA** (grouped\u002Fmulti-query attention — fewer KV heads, **dramatically smaller KV\ncache**, §11.2 → `ml-evaluation-serving-mlops-and-safety`), **FlashAttention** (IO-aware exact attention — not an approximation; it\navoids materializing the n² matrix in HBM), **MoE** (mixture of experts — many parameters,\nfew active per token), **sliding-window** and other sparse attention, and **SSMs\u002FMamba**\nas the main non-attention contender.\n\n### 5.2 The rest\n\n**CNNs** — still the right choice for many vision tasks, especially with limited data and\ncompute; ConvNeXt showed a modernized CNN matches ViTs. Convolution's inductive bias\n(locality, translation equivariance) is a *feature* when data is scarce.\n**Vision Transformers** — win at scale, need more data or heavy augmentation.\n**Diffusion models** — the generative image\u002Fvideo\u002Faudio default; iterative denoising, with\nflow matching as the cleaner modern formulation.\n**GNNs** — for genuinely graph-structured data. ⚠️ Often beaten by feature engineering plus\na GBDT; verify the graph structure actually carries signal.\n**Encoder-decoder vs. decoder-only** — decoder-only won for general LLMs; encoder models\n(BERT-family) remain excellent and much cheaper for classification and retrieval, and are\nbadly underused because attention moved elsewhere.\n**Embedding models** — the workhorse of retrieval, RAG, semantic search, and dedup, and\noften the highest-value-per-FLOP thing you can deploy.\n\n---\n\n## §6. The Ecosystem\n\n### 6.1 PyTorch\n\n**[VERSIONED] PyTorch 2.13 is current (July 2026)**, with a roughly quarterly cadence.\nThe parts that matter:\n\n**`torch.compile`** — the headline feature since 2.0. Traces your model (TorchDynamo),\nlowers it (TorchInductor), and generates fused Triton (or now **CuTeDSL**) kernels.\n**Typically a substantial speedup for free.** ⚠️ **Watch for graph breaks** — Python\nconstructs Dynamo can't trace force it to fall back, silently costing you the speedup;\n`torch._dynamo.error_on_graph_break()` makes that loud rather than silent.\n**Recompilation** on changing shapes is the other tax — mark dynamic dimensions explicitly.\n\n**Other things worth knowing**: `torch.export` is the unified capture path (legacy\n`torch.jit` and old ONNX flows are deprecated or removed); **`weights_only=True` is the\ndefault for `torch.load` since 2.6** — a genuine security improvement, since pickle\ndeserialization is arbitrary code execution; **FlexAttention** for custom attention\npatterns without writing kernels; **AOTInductor** for ahead-of-time compilation.\n\n### 6.2 The rest of the stack\n\n| Layer | Tools |\n|---|---|\n| **Frameworks** | **PyTorch** (dominant in research and increasingly production), **JAX** (functional, `jit`\u002F`grad`\u002F`vmap`\u002F`pmap`, strong on TPU, favored for large-scale research), TensorFlow\u002FKeras (legacy-heavy but alive), **MLX** (Apple Silicon) |\n| **Training loops** | **Lightning**, **HF Accelerate**, **torchtune**, raw loops (fine, and often clearer) |\n| **Models and data** | **Hugging Face** `transformers`, `datasets`, `tokenizers`, `peft`, `trl` — effectively the industry's default distribution channel |\n| **Classical** | scikit-learn, XGBoost, LightGBM, CatBoost, statsmodels |\n| **Numerics** | NumPy, pandas, **Polars** (faster, saner API, increasingly the default for new work), DuckDB, Arrow |\n| **Kernels** | **Triton** (write GPU kernels in Python), CUDA, FlashAttention, **FlashInfer** |\n| **Experiment tracking** | Weights & Biases, MLflow, TensorBoard, Aim |\n| **Serving** | **vLLM**, **SGLang**, TensorRT-LLM, Triton Inference Server, Ray Serve, llama.cpp\u002FOllama (§11 → `ml-evaluation-serving-mlops-and-safety`) |\n| **Orchestration** | Ray, Kubeflow, Airflow, Prefect, SLURM (still the HPC standard) |\n\n**[DURABLE] Pin your versions and record them.** The ML stack breaks compatibility\nconstantly, and \"it worked last month\" is a real and frequent failure.\n\n---\n\n## §7. Training at Scale\n\n### 7.1 The parallelism taxonomy\n\n| Strategy | Splits | Use when |\n|---|---|---|\n| **Data parallel (DDP)** | The batch | **The default.** Model fits on one GPU |\n| **FSDP \u002F ZeRO** | Parameters, gradients, optimizer state | Model doesn't fit. **The mainstream large-model answer** |\n| **Tensor parallel** | Individual matrices, within a layer | Very large layers; needs fast interconnect (NVLink) |\n| **Pipeline parallel** | Layers across devices | Very deep models; ⚠️ introduces bubbles |\n| **Expert parallel** | MoE experts | MoE models |\n| **Context\u002Fsequence parallel** | The sequence dimension | Very long context |\n\n**[DURABLE] Real large-scale training composes several of these** (\"3D parallelism\" and\nbeyond), and the composition is chosen against your interconnect topology, not in the\nabstract. **Communication is usually the bottleneck**, which is why NVLink\u002FInfiniBand\ntopology drives the design.\n\n**[VERSIONED]** PyTorch's **FSDP2** now supports overlapping reduce-scatter and all-gather\nvia separate process groups (opt-in), which increases throughput; and **torchcomms** is a\nnew communications backend for PyTorch Distributed aimed at fault tolerance, scalability,\nand debuggability on large clusters — with plans to make it the default, including\nbreaking changes to how ProcessGroups operate (eager initialization, single backend\ndevice). **If you run large distributed jobs, that migration is on your horizon.**\n\n### 7.2 Mixed precision\n\n**[DURABLE]** Train in low precision, keep a master copy and accumulations in higher\nprecision.\n- **FP16** — needs **loss scaling** (gradients underflow otherwise).\n- **BF16** — same exponent range as FP32, less mantissa. **No loss scaling needed. The\n  default on modern hardware, and the right choice.**\n- **FP8** — real on Hopper and Blackwell; needs careful scaling; increasingly used in\n  production training.\n- **[VERSIONED] NVFP4 \u002F MXFP4** — 4-bit formats with hardware support on Blackwell,\n  now appearing in both training and inference paths.\n\n### 7.3 Memory\n\n**[DURABLE] Know where the memory goes**, because \"CUDA out of memory\" is the most common\nobstacle in practice:\n```\nparameters  +  gradients  +  OPTIMIZER STATE  +  activations  +  fragmentation\n                                    ↑\n              Adam keeps 2 extra copies — often the largest single term\n```\nA rough anchor: **fp32 Adam training costs roughly 16 bytes per parameter** (4 param +\n4 grad + 8 optimizer state), before activations. That's why a 7B model doesn't fit on an\n80 GB card without help.\n\n**The levers**: **gradient checkpointing** (recompute activations — trades ~30% compute for\nlarge memory savings), **gradient accumulation**, **8-bit or fused optimizers**, **FSDP\nsharding**, **`nn.LinearCrossEntropyLoss`** (a 2026 PyTorch addition fusing the final\nprojection and loss — **cuts peak memory by up to 4× for large-vocabulary LM training**,\nwhere the logits tensor is genuinely enormous), and simply **reducing sequence length**.\n\n---\n\n## §8. Fine-Tuning and Adaptation\n\n**[DURABLE] The order to try things, cheapest first:**\n```\n1. Prompting \u002F few-shot                  ← no training. Try this first, seriously\n2. Retrieval (RAG)                       ← when the problem is missing knowledge\n3. PEFT \u002F LoRA                           ← when the problem is behaviour or format\n4. Full fine-tuning                      ← when you have real data and real budget\n5. Continued pretraining                 ← new domain, large corpus\n6. Training from scratch                 ← almost never the right answer\n```\n**⚠️ The most common expensive mistake is fine-tuning to fix a knowledge problem.**\nFine-tuning teaches *behaviour, style, and format* reliably; it teaches *facts* poorly and\nexpensively. **If the model doesn't know something, retrieve it.**\n\n**LoRA** — freeze the base, train low-rank adapters (`W + BA`). Tiny memory footprint,\nswappable adapters, near-full-fine-tuning quality on most tasks. **QLoRA** adds 4-bit\nquantization of the frozen base, putting single-GPU fine-tuning of large models within\nreach. Other PEFT: prefix tuning, prompt tuning, IA³, DoRA.\n\n**Post-training \u002F alignment**: **SFT** (supervised fine-tuning on demonstrations), then\npreference optimization — **RLHF\u002FPPO** (powerful, complex, unstable), **DPO** (much\nsimpler, no reward model, now the common default), and GRPO and relatives for reasoning\nwork. **[DURABLE] Data quality dominates method choice here** — a small, carefully curated\nSFT set routinely beats a large noisy one.\n\n**⚠️ Catastrophic forgetting is real.** Fine-tuning on a narrow task degrades general\ncapability. Mix in general data, use lower learning rates, prefer PEFT, and **evaluate on\ncapabilities you didn't train on** — not just the target task.\n",{"name":5,"description":7},"develop","ae32736a1aabddabca5d2c96d665aa0ebc7d8700","src\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fmachine-learning\u002Fskills\u002Fml-deep-learning-and-training",3,"https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey","https:\u002F\u002Fthe-vibey-project.github.io\u002Fvibey\u002F","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project","https:\u002F\u002Fvibewithadam.matthewsteinberger.com","freedom for software engineers","2026-11-15 05:47:42",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add the-vibey-project\u002Fvibey --skill ml-deep-learning-and-training",{"tool":40,"command":41},"bunx","bunx skills add the-vibey-project\u002Fvibey --skill ml-deep-learning-and-training",{"tool":43,"command":44},"pnpm","pnpm dlx skills add the-vibey-project\u002Fvibey --skill ml-deep-learning-and-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fmachine-learning\u002Fskills\u002Fml-deep-learning-and-training\u002FSKILL.md",true,[],[49,56,63,70,77,84],{"skillNo":50,"name":51,"nameZh":6,"description":52,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":53,"categories":54,"keywords":55,"publishedAt":19},"01M7AT1FQJAE1FQYAMDTF490Y4","white-paper-writing","Use when writing white papers, technical reports, policy documents, thought leadership pieces, or any long-form authoritative document. Triggers on requests to write, structure, outline, or review white papers, technical reports, position papers, policy briefs, or evidence-driven arguments. Also use for B2B content marketing strategy, technical documentation requiring authority, academic-style reports, or any document that must educate while persuading. Use when someone asks \"how should I structure this\" for a long-form document.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Fwhite-paper-writing\u002FSKILL.md",[],[],{"skillNo":57,"name":58,"nameZh":6,"description":59,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":60,"categories":61,"keywords":62,"publishedAt":19},"01M7AT16BJRJJKK2MASXRZS76G","prose-technical-mechanics","Use when editing or writing prose at a technical\u002Fmechanical level: choosing between Anglo-Saxon and Latinate diction, placing information in topic vs stress positions, writing periodic vs cumulative sentences, handling sentence rhythm, punctuation as logic, transitions, paragraph structure, or phonesthetics. Use when someone wants to understand WHY prose works or doesn't, not just surface-level style advice. Triggers on word choice, sentence structure, emphasis, rhythm, diction, or any deep writing mechanics question.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Fprose-technical-mechanics\u002FSKILL.md",[],[],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":19},"01M7AT0W9HT0ZZ9Y287NW99MR3","prose-mechanics","Use when writing, editing, or analyzing prose style; when trying to avoid generic AI-sounding writing; when applying specific prose styles such as journalistic, academic, narrative, KJV biblical, Hemingway plain, Latinate\u002FJamesian, Orwellian, Augustan, Victorian, Early Modern (Bacon, Browne, Milton), or free indirect discourse; or when coaching someone on writing voice and mechanics. Triggers on any request to improve, analyze, or write in a specific prose style, to fix \"AI-sounding\" writing, to make prose more vivid or concrete, to vary sentence rhythm, or to understand how a particular writer achieves their effects.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Fprose-mechanics\u002FSKILL.md",[],[],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M7AT0KR635JX66X2X6F3E2CP","narrative-structure","Use when structuring stories, pitches, presentations, white papers, or any persuasive narrative; analyzing the hero's journey in fiction or non-fiction; applying StoryBrand for business communication; designing narrative arcs for thought leadership; or understanding Vogler, Snyder, Propp, or Campbell frameworks. Triggers on any narrative structure, storytelling, pitch structure, StoryBrand, hero's journey, three-act structure, Freytag, Save the Cat, narrative arc, or \"how should I structure this story\u002Fpitch\u002Fpresentation\" request.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Fnarrative-structure\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":19},"01M7AT0AKGW09NKKZDYBT8VRDE","legal-brief-writing","Use when writing legal briefs, motions, or any court filing; when analyzing judicial cognition and persuasion; when applying elite brief-writing techniques from Roberts, Clement, Kagan, Garner; or when structuring legal arguments for maximum persuasion. Also use for autistic\u002Fneurodivergent advocates who need to understand how their natural communication style maps to elite legal advocacy. Triggers on any legal writing, brief drafting, motion preparation, or legal argument structuring request.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Flegal-brief-writing\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":19},"01M7AT00VWPS4Q8B5VEMXX9NQT","journal-publishability-criteria","Use whenever a research paper, article or preprint is being written, revised, reviewed or published, and specifically before every publication of docs\u002Fpaper.md. Covers what legitimate journals actually gate on, in order of gating — the editor's desk screen (scope, novelty, package problems), what reviewers judge (methodological soundness, claims the evidence can carry, statistical and qualitative reporting), EQUATOR reporting guidelines, the ICMJE authorship criteria, COPE's view of text recycling, the near-universal generative-AI policy (no AI author, disclosure, author responsibility), TOP 2025 transparency standards, how to verify a journal is legitimate, submitting without an institution, and a pre-submission checklist. Triggers on requests to evaluate, review, prepare or submit a paper, to judge publishability, to write a declarations or data-availability section, to disclose AI use, or to check a journal.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Fjournal-publishability-criteria\u002FSKILL.md",[],[]]