[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7APJFVB5JVARSYMB9NAZFGH":3,"skill-same:01M7APJFVB5JVARSYMB9NAZFGH":48},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":29,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M7APJFVB5JVARSYMB9NAZFGH","ml-evaluation-serving-mlops-and-safety",null,"Use when evaluating, debugging, deploying, or governing models. Covers metrics and doing evaluation honestly, evaluating generative models, debugging training runs (NaNs, divergence, overfitting, data bugs), inference and serving (vLLM, SGLang, batching, the KV cache, quantization, what actually determines inference cost and its economics), MLOps and reproducibility (the pipeline, monitoring, drift), hardware and cost, and interpretability, fairness, security, and safety.","the-vibey-project\u002Fvibey","the-vibey-project","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F329298758?v=4","01M7AJHGZ9722CTJ4JNQZVRH0Q","the vibey project","01M7AJHGZVMG2K7TGXQ6XFHVG3",5,"MIT","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fmachine-learning\u002Fskills\u002Fml-evaluation-serving-mlops-and-safety\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: ml-evaluation-serving-mlops-and-safety\ndescription: \"Use when evaluating, debugging, deploying, or governing models. Covers metrics and doing evaluation honestly, evaluating generative models, debugging training runs (NaNs, divergence, overfitting, data bugs), inference and serving (vLLM, SGLang, batching, the KV cache, quantization, what actually determines inference cost and its economics), MLOps and reproducibility (the pipeline, monitoring, drift), hardware and cost, and interpretability, fairness, security, and safety.\"\n---\n\n# Machine Learning: Evaluation, Debugging, Inference and Serving, MLOps, and Responsible ML\n\n> **Part 3 of 4** of the *Machine Learning* reference (plugin `machine-learning`), covering §9–§14. Sibling skills: `ml-framing-data-and-classical` (§0–§3), `ml-deep-learning-and-training` (§4–§8), `ml-reference` (§15–§20). Section numbers are shared across the set; a reference written as §N → `skill` points into that sibling skill.\n>\n> **Currency:** Verified August 2026. See §17 → `ml-reference` for the currency snapshot and what goes stale first.\n\n> **How to read this.** Reference, not tutorial. Sections are independent. Three markers:\n> - **[DURABLE]** — statistics, optimization theory, or a lesson the field has relearned\n>   every few years. Does not expire.\n> - **[VERSIONED]** — framework versions, hardware, library APIs. Moving fast; verify.\n> - **[CONTESTED]** — practitioners genuinely disagree.\n>\n> **⚠️ GOTCHA** boxes mark the mistakes that produce a model that looks great offline and\n> fails in production — which is the characteristic failure of this field.\n>\n> **The three framings that organize everything below:**\n> 1. **Your data is the model.** Architecture choices matter far less than data quality,\n>    quantity, and whether your evaluation measures the thing you care about. Most time on\n>    a real project goes to data, and most catastrophic failures originate there (§2 → `ml-framing-data-and-classical`).\n> 2. **The default failure mode is a number that lies.** Leakage, distribution shift, a\n>    metric that doesn't match the objective, a test set you tuned against. **A model that\n>    looks too good almost always is** — and finding out why is the core skill.\n> 3. **Almost everything you'll build should not be a neural network.** For tabular data —\n>    which is most business data — gradient-boosted trees remain the strongest default\n>    (§3.1 → `ml-framing-data-and-classical`). Reach for deep learning when you have perceptual data, sequences, very large\n>    datasets, or a pretrained model to adapt.\n\n---\n\n## §9. Evaluation\n\n**[DURABLE] This section is where most projects actually fail, and it fails quietly.**\n\n### 9.1 Metrics\n\n**⚠️ Accuracy is almost always the wrong metric.** On a 99:1 imbalanced problem, predicting\nthe majority class gets 99% accuracy and is useless.\n\n| Task | Metrics |\n|---|---|\n| **Binary classification** | Precision, recall, F1, **PR-AUC** (better than ROC-AUC under heavy imbalance), ROC-AUC, MCC, and the **confusion matrix — always look at it** |\n| **Multi-class** | Macro (treats classes equally) vs. micro (weights by frequency) vs. weighted — **say which you used; they disagree substantially** |\n| **Regression** | MAE (robust), RMSE (penalizes large errors), MAPE (⚠️ breaks near zero, asymmetric), R² |\n| **Ranking \u002F retrieval** | NDCG, MRR, MAP, recall@k, precision@k |\n| **Probabilistic** | **Log loss, Brier score, calibration curves** — use these when the probability matters |\n| **Generative \u002F LLM** | Task-specific benchmarks, human eval, LLM-as-judge (⚠️ §9.3), pairwise preference |\n\n**[DURABLE] Choose the metric from the decision it informs.** If false negatives cost 100×\nfalse positives, optimize for that, and **tune the decision threshold explicitly** — it's a\nfree parameter most people leave at 0.5 for no reason.\n\n### 9.2 Doing it honestly\n\n- **One test-set evaluation.** Every additional look is leakage (§2.1 → `ml-framing-data-and-classical`).\n- **Report variance**, not a point estimate. Multiple seeds, confidence intervals,\n  bootstrapping. **⚠️ A great deal of reported \"improvement\" in ML is within seed noise**,\n  and the field's reproducibility problems are substantially this.\n- **Compare against a strong baseline**, tuned as carefully as your method. An untuned\n  baseline is not a comparison.\n- **Slice your metrics.** Aggregate performance hides systematic failure on subgroups —\n  by user segment, geography, device, class, time period. **This is both an accuracy\n  practice and a fairness practice** (§14.2).\n- **Evaluate on distribution shift** you expect in production: a future time period, a new\n  region, a new source.\n- **Look at the errors.** Sample 50 mistakes and read them. **Consistently the highest\n  information-per-minute activity in applied ML**, and consistently skipped.\n\n### 9.3 Evaluating generative models\n\nHard, and getting harder. **Benchmark contamination** is pervasive — test sets leak into\npretraining corpora, so public benchmark scores are systematically optimistic.\n**LLM-as-judge** is practical and scalable but carries **position bias, verbosity bias,\nand self-preference bias**; mitigate by randomizing order, using multiple judges, and\ncalibrating against human labels on a subset. **Build a private eval set from your actual\ntask**, hold it out, and version it — this beats any public benchmark for deciding whether\nyour system works.\n\n---\n\n## §10. Debugging Training\n\n**[DURABLE] A systematic ladder, in order. Skipping steps is what makes debugging take\nweeks.**\n\n1. **Overfit a single batch.** Train on 2–10 examples until loss ≈ 0. **If you can't, you\n   have a bug, not a learning-rate problem.** This is the single most valuable diagnostic in\n   deep learning and takes two minutes.\n2. **Check shapes and the data.** Print them. Visualize an actual batch after\n   augmentation. **⚠️ Silent broadcasting bugs are endemic** — a `(B,1)` vs. `(B,)` mismatch\n   produces a `(B,B)` tensor and a loss that trains to something meaningless.\n3. **Verify the loss at initialization.** Cross-entropy on k balanced classes should start\n   near `ln(k)`. If it doesn't, your labels, logits, or reduction are wrong.\n4. **Check the label pipeline end-to-end.** Off-by-one, shuffled labels, wrong mapping.\n5. **Watch gradient norms.** Zero → disconnected graph or dead activations. Exploding →\n   clip, lower the LR, check normalization.\n6. **Sweep the learning rate** over orders of magnitude before touching anything else.\n7. **Compare train vs. validation curves**: both high = underfitting (bigger model, train\n   longer, better features); train low + val high = overfitting (more data, more\n   regularization, augmentation); **val better than train** = a bug, or dropout\u002FBN\n   train-mode artifacts.\n8. **Turn off tricks** — augmentation, scheduler, mixed precision, `torch.compile` — and\n   reintroduce them one at a time.\n\n| Symptom | Likely cause |\n|---|---|\n| Loss is NaN | LR too high; log(0)\u002Fdiv-by-zero; fp16 overflow (**use bf16**); bad input data |\n| Loss flat from step 0 | LR too low; frozen params; disconnected graph; `optimizer.zero_grad()` misplaced |\n| Loss decreases then explodes | LR too high; missing gradient clipping; a bad batch |\n| Great train, terrible val | Overfitting, or **leakage in reverse** (val is harder\u002Fdifferent) |\n| Great val, terrible production | **Distribution shift, or leakage** (§2.1 → `ml-framing-data-and-classical`). Check the leakage list first |\n| Works in eager, breaks compiled | Graph breaks, dynamic shapes, or a numerics difference |\n| Non-deterministic across runs | Expected — see §12.1. Report variance |\n| Slower than expected on GPU | Dataloader-bound (check GPU utilization), small batches, unnecessary syncs, `.item()` in the loop |\n\n---\n\n## §11. Inference and Serving\n\n### 11.1 The landscape\n\n**[VERSIONED — this layer moved fast and consolidated in 2025–2026.]**\n\n| Engine | Best for |\n|---|---|\n| **vLLM** | **The practical default.** Broadest hardware support (NVIDIA CUDA, AMD ROCm, Intel XPU, Google TPU, and more), one `pip` install, OpenAI-compatible server immediately |\n| **SGLang** | **Long shared prefixes** — RAG over a fixed corpus, multi-turn agents, structured decoding. **RadixAttention** turns prefix overlap into automatic cache hits. Reported ~29% higher throughput than vLLM on H100 in one comparison, with much larger gains on prefix-heavy workloads |\n| **TensorRT-LLM** | Committed to NVIDIA's newest silicon and want the deepest kernel-level optimization — **paid for in build complexity** |\n| **llama.cpp \u002F Ollama \u002F MLX** | Single-user local inference, CPU and Apple Silicon |\n| **Triton Inference Server \u002F Ray Serve** | General model serving, multi-model, non-LLM |\n\n**⚠️ [VERSIONED] Hugging Face TGI moved to maintenance mode** (announced December 2025,\nrepository archived read-only **21 March 2026**), redirecting users to vLLM, SGLang,\nllama.cpp, and MLX. **If you're on TGI, plan a migration.**\n\n**[DURABLE] The feature set has converged.** Any modern engine gives you **continuous\n(in-flight) batching** — new requests join the running batch the moment a slot frees, which\nis the single biggest throughput unlock over naïve batching — and a **paged KV cache**\n(virtual-memory management for the attention K\u002FV store). Choose on hardware breadth,\nworkload shape, and operational complexity, not on feature checklists.\n\n### 11.2 What actually determines inference cost\n\n**[DURABLE]** LLM inference has two distinct phases with completely different bottlenecks:\n- **Prefill** (processing the prompt) — **compute-bound**, parallel across tokens.\n- **Decode** (generating tokens) — **memory-bandwidth-bound**, one token at a time.\n\n**This asymmetry drives everything**: it's why **disaggregated prefill\u002Fdecode** (routing\neach phase to different hardware) is a major 2026 architecture, why **memory bandwidth\nmatters more than FLOPs** for serving, and why batch size helps decode enormously.\n\n**The KV cache is usually your real memory constraint**, growing linearly with batch size\nand sequence length. **GQA\u002FMQA**, **paged attention**, **prefix caching**, and **KV cache\nquantization** all exist to attack it.\n\n**Other levers**: **quantization** (§11.3), **speculative decoding** (a small draft model\nproposes, the big model verifies — real latency wins), **prefix caching**, and **structured\noutput constraints**.\n\n### 11.3 Quantization\n\n| Format | Use |\n|---|---|\n| **INT8 \u002F W8A8** | Well-established, minimal quality loss |\n| **INT4 \u002F W4A16** (GPTQ, AWQ) | **The common serving choice.** ~4× memory reduction, small quality cost |\n| **FP8** | Native hardware support on Hopper\u002FBlackwell; good quality retention |\n| **NVFP4 \u002F MXFP4** | 4-bit with Blackwell hardware support; increasingly used |\n| **GGUF (k-quants)** | The llama.cpp ecosystem, CPU and Apple Silicon |\n\n**[DURABLE] Quantization-aware training beats post-training quantization on quality and\ncosts more.** For most people PTQ with a good calibration set is sufficient. **⚠️ Always\nevaluate the quantized model on your own task** — published \"negligible degradation\" claims\nare measured on benchmarks that may have nothing to do with your use case, and degradation\nis often concentrated in exactly the hard cases you care about.\n\n### 11.4 The economics\n\n**[VERSIONED, and the trend is the point]**: **inference now accounts for roughly two-thirds\nof all AI compute, up from about one-third in 2023**, and cost-per-token has fallen by\norders of magnitude — one analysis puts GPT-4-equivalent performance at roughly **$0.40 per\nmillion tokens in early 2026 versus ~$20 in late 2022**. Serving-engine improvements are a\nlarge part of it: **GPU utilization went from ~30–40% to ~70–80%** through continuous\nbatching, paged attention, and speculative decoding.\n\n**[DURABLE] The practical consequence: cost-per-token is the KPI**, and a 10× inference cost\nreduction directly enables 10× the users at the same budget. Self-hosting reaches cost\nparity with commercial APIs at moderate volume — one 2026 study found parity within 1–4\nmonths at ~30M tokens\u002Fday — but **only if you actually keep the hardware busy.**\n\n---\n\n## §12. MLOps and Reproducibility\n\n### 12.1 Reproducibility\n\n**[DURABLE] Bit-exact reproducibility on GPU is achievable but costly**, and most teams\nshould aim for *statistical* reproducibility instead: seed everything (Python, NumPy,\nframework, dataloader workers), pin all versions and record them, version data and code\ntogether, log the full config, and **report variance across seeds rather than a single\nrun**. Full determinism requires deterministic algorithms (`torch.use_deterministic_algorithms`),\ndisabling cuDNN benchmarking, and fixed dataloader ordering — and it will slow you down.\n\n**Version the data.** Model artifacts without the exact training data are not reproducible,\nand \"we retrained and got different numbers\" is otherwise unresolvable.\n\n### 12.2 The pipeline\n\n```\ndata ingestion → validation → feature engineering → training → EVALUATION GATE\n  → registry → deployment (shadow → canary → full) → MONITORING → retraining\n```\n**[DURABLE] The evaluation gate and monitoring are the parts people skip**, and they're\nthe parts that prevent silent disasters.\n\n**Feature stores** solve a real problem — **training\u002Fserving skew**, where the features\ncomputed at training time differ subtly from those computed at inference. **⚠️ This is one\nof the most common and hardest-to-find production bugs**, and if you compute features in\ntwo different codepaths you will eventually have it.\n\n### 12.3 Monitoring\n\nMonitor, in roughly this priority order: **prediction distribution** (drifts first and\ncheapest to watch), **input feature distributions** (PSI, KL divergence), **actual\nperformance** where labels eventually arrive, **latency and throughput**, **error rates**,\nand **business metrics** (the only ones that ultimately matter).\n\n**[DURABLE] Models degrade. Plan for retraining from day one** — trigger it on a schedule,\non drift detection, or on performance drop. And **keep the ability to roll back to the\nprevious model instantly**; a bad model deploy is an outage.\n\n---\n\n## §13. Hardware and Cost\n\n**[VERSIONED]**\n\n| Tier | Notes |\n|---|---|\n| **H100 (80 GB)** | ⚠️ **The software ecosystem is most mature here** — vLLM, SGLang, TensorRT-LLM, PyTorch, JAX. **Any optimization technique you read about was probably benchmarked on H100** |\n| **H200 (141 GB)** | Same die as H100 with HBM3e: **141 GB and 4.8 TB\u002Fs (+43% bandwidth)**. Often better per-token cost than 2× H100 for FP16 70B |\n| **Blackwell (B200\u002FGB200\u002FB300)** | **192 GB HBM3e** — a 70B FP16 model with real KV headroom, or 405B FP8 on two chips. NVFP4 support. ~3× faster LLM training than the prior generation by NVIDIA's own MLPerf claims |\n| **Rubin \u002F Vera** | The announced next generation, aimed at agentic training and inference |\n| **RTX PRO 6000 Blackwell (96 GB)** | Workstation-class; 4–8 in a rackmount is a standard 2026 self-hosting configuration |\n| **AMD MI300X \u002F ROCm** | Real and improving; **NVIDIA remains the default for vLLM, Unsloth, and most production workflows** |\n| **TPU v5e\u002Fv6, Trainium\u002FInferentia, Gaudi 3** | Viable, especially via JAX (TPU) or where you have committed cloud spend |\n\n**[DURABLE] For LLM inference, memory capacity and bandwidth dominate the selection\ndecision, not FLOPs** (§11.2). For training, interconnect (NVLink, InfiniBand) often\nmatters more than per-chip performance once you're multi-node.\n\n**Cost discipline**: cloud beats buying unless you keep hardware busy more than roughly\n8 hours\u002Fday; **spot\u002Fpreemptible instances plus checkpointing** are the single biggest\ntraining cost lever; profile before scaling up (most jobs are dataloader-bound or\nbadly-batched, not compute-bound); and **the cheapest optimization is a smaller model that\nmeets the requirement.**\n\n---\n\n## §14. Interpretability, Fairness, Safety\n\n### 14.1 Interpretability\n\n**Intrinsically interpretable models** — linear models, small trees, GAMs — are\nunderrated. **[DURABLE] If interpretability is a hard requirement, use an interpretable\nmodel rather than explaining a black box.**\n\n**Post-hoc methods**: **SHAP** (game-theoretic, the practical standard; ⚠️ correlated\nfeatures make attributions ambiguous), **LIME** (local surrogates; unstable),\n**permutation importance** (⚠️ misleading with correlated features), **partial dependence\nand ICE**, attention maps (⚠️ **attention is not explanation** — a well-known result;\nattention weights don't reliably indicate what drove the output), and **counterfactuals**\n(often the most *useful* form for an affected person).\n\n**⚠️ Feature importance is not causal.** A high-importance feature tells you what the model\nuses, not what drives the outcome. Acting on it as if it were causal is a recurring and\nexpensive error.\n\n### 14.2 Fairness\n\n**[DURABLE] The mathematics constrains you**: demographic parity, equalized odds, and\ncalibration within groups are **provably mutually incompatible** except in degenerate cases.\n**You must choose which fairness definition applies to your context** — there is no\ntechnically neutral option, and pretending otherwise is itself a choice.\n\nPractically: **slice every metric by group** (§9.2 — this is the same practice as good\nevaluation), audit training data for representation and historical bias, remember that\n**removing a protected attribute doesn't remove its influence** (proxies are everywhere),\nand document the model's intended use and limitations (model cards, datasheets).\n\n### 14.3 Security and safety\n\n**Adversarial examples** (small perturbations flip predictions — still largely unsolved),\n**data poisoning**, **model extraction**, **membership inference** and **training data\nextraction** (models memorize; verbatim regurgitation is real), and **prompt injection**\nfor LLM systems (⚠️ **not solved**, and any system giving an LLM tools plus untrusted input\nhas this exposure).\n\n**⚠️ `pickle` deserialization is arbitrary code execution.** Use **safetensors** for model\nweights. PyTorch's `weights_only=True` default since 2.6 addresses this, but third-party\ncheckpoints from unknown sources remain a real supply-chain risk.\n","# Machine Learning: Evaluation, Debugging, Inference and Serving, MLOps, and Responsible ML\n\n> **Part 3 of 4** of the *Machine Learning* reference (plugin `machine-learning`), covering §9–§14. Sibling skills: `ml-framing-data-and-classical` (§0–§3), `ml-deep-learning-and-training` (§4–§8), `ml-reference` (§15–§20). Section numbers are shared across the set; a reference written as §N → `skill` points into that sibling skill.\n>\n> **Currency:** Verified August 2026. See §17 → `ml-reference` for the currency snapshot and what goes stale first.\n\n> **How to read this.** Reference, not tutorial. Sections are independent. Three markers:\n> - **[DURABLE]** — statistics, optimization theory, or a lesson the field has relearned\n>   every few years. Does not expire.\n> - **[VERSIONED]** — framework versions, hardware, library APIs. Moving fast; verify.\n> - **[CONTESTED]** — practitioners genuinely disagree.\n>\n> **⚠️ GOTCHA** boxes mark the mistakes that produce a model that looks great offline and\n> fails in production — which is the characteristic failure of this field.\n>\n> **The three framings that organize everything below:**\n> 1. **Your data is the model.** Architecture choices matter far less than data quality,\n>    quantity, and whether your evaluation measures the thing you care about. Most time on\n>    a real project goes to data, and most catastrophic failures originate there (§2 → `ml-framing-data-and-classical`).\n> 2. **The default failure mode is a number that lies.** Leakage, distribution shift, a\n>    metric that doesn't match the objective, a test set you tuned against. **A model that\n>    looks too good almost always is** — and finding out why is the core skill.\n> 3. **Almost everything you'll build should not be a neural network.** For tabular data —\n>    which is most business data — gradient-boosted trees remain the strongest default\n>    (§3.1 → `ml-framing-data-and-classical`). Reach for deep learning when you have perceptual data, sequences, very large\n>    datasets, or a pretrained model to adapt.\n\n---\n\n## §9. Evaluation\n\n**[DURABLE] This section is where most projects actually fail, and it fails quietly.**\n\n### 9.1 Metrics\n\n**⚠️ Accuracy is almost always the wrong metric.** On a 99:1 imbalanced problem, predicting\nthe majority class gets 99% accuracy and is useless.\n\n| Task | Metrics |\n|---|---|\n| **Binary classification** | Precision, recall, F1, **PR-AUC** (better than ROC-AUC under heavy imbalance), ROC-AUC, MCC, and the **confusion matrix — always look at it** |\n| **Multi-class** | Macro (treats classes equally) vs. micro (weights by frequency) vs. weighted — **say which you used; they disagree substantially** |\n| **Regression** | MAE (robust), RMSE (penalizes large errors), MAPE (⚠️ breaks near zero, asymmetric), R² |\n| **Ranking \u002F retrieval** | NDCG, MRR, MAP, recall@k, precision@k |\n| **Probabilistic** | **Log loss, Brier score, calibration curves** — use these when the probability matters |\n| **Generative \u002F LLM** | Task-specific benchmarks, human eval, LLM-as-judge (⚠️ §9.3), pairwise preference |\n\n**[DURABLE] Choose the metric from the decision it informs.** If false negatives cost 100×\nfalse positives, optimize for that, and **tune the decision threshold explicitly** — it's a\nfree parameter most people leave at 0.5 for no reason.\n\n### 9.2 Doing it honestly\n\n- **One test-set evaluation.** Every additional look is leakage (§2.1 → `ml-framing-data-and-classical`).\n- **Report variance**, not a point estimate. Multiple seeds, confidence intervals,\n  bootstrapping. **⚠️ A great deal of reported \"improvement\" in ML is within seed noise**,\n  and the field's reproducibility problems are substantially this.\n- **Compare against a strong baseline**, tuned as carefully as your method. An untuned\n  baseline is not a comparison.\n- **Slice your metrics.** Aggregate performance hides systematic failure on subgroups —\n  by user segment, geography, device, class, time period. **This is both an accuracy\n  practice and a fairness practice** (§14.2).\n- **Evaluate on distribution shift** you expect in production: a future time period, a new\n  region, a new source.\n- **Look at the errors.** Sample 50 mistakes and read them. **Consistently the highest\n  information-per-minute activity in applied ML**, and consistently skipped.\n\n### 9.3 Evaluating generative models\n\nHard, and getting harder. **Benchmark contamination** is pervasive — test sets leak into\npretraining corpora, so public benchmark scores are systematically optimistic.\n**LLM-as-judge** is practical and scalable but carries **position bias, verbosity bias,\nand self-preference bias**; mitigate by randomizing order, using multiple judges, and\ncalibrating against human labels on a subset. **Build a private eval set from your actual\ntask**, hold it out, and version it — this beats any public benchmark for deciding whether\nyour system works.\n\n---\n\n## §10. Debugging Training\n\n**[DURABLE] A systematic ladder, in order. Skipping steps is what makes debugging take\nweeks.**\n\n1. **Overfit a single batch.** Train on 2–10 examples until loss ≈ 0. **If you can't, you\n   have a bug, not a learning-rate problem.** This is the single most valuable diagnostic in\n   deep learning and takes two minutes.\n2. **Check shapes and the data.** Print them. Visualize an actual batch after\n   augmentation. **⚠️ Silent broadcasting bugs are endemic** — a `(B,1)` vs. `(B,)` mismatch\n   produces a `(B,B)` tensor and a loss that trains to something meaningless.\n3. **Verify the loss at initialization.** Cross-entropy on k balanced classes should start\n   near `ln(k)`. If it doesn't, your labels, logits, or reduction are wrong.\n4. **Check the label pipeline end-to-end.** Off-by-one, shuffled labels, wrong mapping.\n5. **Watch gradient norms.** Zero → disconnected graph or dead activations. Exploding →\n   clip, lower the LR, check normalization.\n6. **Sweep the learning rate** over orders of magnitude before touching anything else.\n7. **Compare train vs. validation curves**: both high = underfitting (bigger model, train\n   longer, better features); train low + val high = overfitting (more data, more\n   regularization, augmentation); **val better than train** = a bug, or dropout\u002FBN\n   train-mode artifacts.\n8. **Turn off tricks** — augmentation, scheduler, mixed precision, `torch.compile` — and\n   reintroduce them one at a time.\n\n| Symptom | Likely cause |\n|---|---|\n| Loss is NaN | LR too high; log(0)\u002Fdiv-by-zero; fp16 overflow (**use bf16**); bad input data |\n| Loss flat from step 0 | LR too low; frozen params; disconnected graph; `optimizer.zero_grad()` misplaced |\n| Loss decreases then explodes | LR too high; missing gradient clipping; a bad batch |\n| Great train, terrible val | Overfitting, or **leakage in reverse** (val is harder\u002Fdifferent) |\n| Great val, terrible production | **Distribution shift, or leakage** (§2.1 → `ml-framing-data-and-classical`). Check the leakage list first |\n| Works in eager, breaks compiled | Graph breaks, dynamic shapes, or a numerics difference |\n| Non-deterministic across runs | Expected — see §12.1. Report variance |\n| Slower than expected on GPU | Dataloader-bound (check GPU utilization), small batches, unnecessary syncs, `.item()` in the loop |\n\n---\n\n## §11. Inference and Serving\n\n### 11.1 The landscape\n\n**[VERSIONED — this layer moved fast and consolidated in 2025–2026.]**\n\n| Engine | Best for |\n|---|---|\n| **vLLM** | **The practical default.** Broadest hardware support (NVIDIA CUDA, AMD ROCm, Intel XPU, Google TPU, and more), one `pip` install, OpenAI-compatible server immediately |\n| **SGLang** | **Long shared prefixes** — RAG over a fixed corpus, multi-turn agents, structured decoding. **RadixAttention** turns prefix overlap into automatic cache hits. Reported ~29% higher throughput than vLLM on H100 in one comparison, with much larger gains on prefix-heavy workloads |\n| **TensorRT-LLM** | Committed to NVIDIA's newest silicon and want the deepest kernel-level optimization — **paid for in build complexity** |\n| **llama.cpp \u002F Ollama \u002F MLX** | Single-user local inference, CPU and Apple Silicon |\n| **Triton Inference Server \u002F Ray Serve** | General model serving, multi-model, non-LLM |\n\n**⚠️ [VERSIONED] Hugging Face TGI moved to maintenance mode** (announced December 2025,\nrepository archived read-only **21 March 2026**), redirecting users to vLLM, SGLang,\nllama.cpp, and MLX. **If you're on TGI, plan a migration.**\n\n**[DURABLE] The feature set has converged.** Any modern engine gives you **continuous\n(in-flight) batching** — new requests join the running batch the moment a slot frees, which\nis the single biggest throughput unlock over naïve batching — and a **paged KV cache**\n(virtual-memory management for the attention K\u002FV store). Choose on hardware breadth,\nworkload shape, and operational complexity, not on feature checklists.\n\n### 11.2 What actually determines inference cost\n\n**[DURABLE]** LLM inference has two distinct phases with completely different bottlenecks:\n- **Prefill** (processing the prompt) — **compute-bound**, parallel across tokens.\n- **Decode** (generating tokens) — **memory-bandwidth-bound**, one token at a time.\n\n**This asymmetry drives everything**: it's why **disaggregated prefill\u002Fdecode** (routing\neach phase to different hardware) is a major 2026 architecture, why **memory bandwidth\nmatters more than FLOPs** for serving, and why batch size helps decode enormously.\n\n**The KV cache is usually your real memory constraint**, growing linearly with batch size\nand sequence length. **GQA\u002FMQA**, **paged attention**, **prefix caching**, and **KV cache\nquantization** all exist to attack it.\n\n**Other levers**: **quantization** (§11.3), **speculative decoding** (a small draft model\nproposes, the big model verifies — real latency wins), **prefix caching**, and **structured\noutput constraints**.\n\n### 11.3 Quantization\n\n| Format | Use |\n|---|---|\n| **INT8 \u002F W8A8** | Well-established, minimal quality loss |\n| **INT4 \u002F W4A16** (GPTQ, AWQ) | **The common serving choice.** ~4× memory reduction, small quality cost |\n| **FP8** | Native hardware support on Hopper\u002FBlackwell; good quality retention |\n| **NVFP4 \u002F MXFP4** | 4-bit with Blackwell hardware support; increasingly used |\n| **GGUF (k-quants)** | The llama.cpp ecosystem, CPU and Apple Silicon |\n\n**[DURABLE] Quantization-aware training beats post-training quantization on quality and\ncosts more.** For most people PTQ with a good calibration set is sufficient. **⚠️ Always\nevaluate the quantized model on your own task** — published \"negligible degradation\" claims\nare measured on benchmarks that may have nothing to do with your use case, and degradation\nis often concentrated in exactly the hard cases you care about.\n\n### 11.4 The economics\n\n**[VERSIONED, and the trend is the point]**: **inference now accounts for roughly two-thirds\nof all AI compute, up from about one-third in 2023**, and cost-per-token has fallen by\norders of magnitude — one analysis puts GPT-4-equivalent performance at roughly **$0.40 per\nmillion tokens in early 2026 versus ~$20 in late 2022**. Serving-engine improvements are a\nlarge part of it: **GPU utilization went from ~30–40% to ~70–80%** through continuous\nbatching, paged attention, and speculative decoding.\n\n**[DURABLE] The practical consequence: cost-per-token is the KPI**, and a 10× inference cost\nreduction directly enables 10× the users at the same budget. Self-hosting reaches cost\nparity with commercial APIs at moderate volume — one 2026 study found parity within 1–4\nmonths at ~30M tokens\u002Fday — but **only if you actually keep the hardware busy.**\n\n---\n\n## §12. MLOps and Reproducibility\n\n### 12.1 Reproducibility\n\n**[DURABLE] Bit-exact reproducibility on GPU is achievable but costly**, and most teams\nshould aim for *statistical* reproducibility instead: seed everything (Python, NumPy,\nframework, dataloader workers), pin all versions and record them, version data and code\ntogether, log the full config, and **report variance across seeds rather than a single\nrun**. Full determinism requires deterministic algorithms (`torch.use_deterministic_algorithms`),\ndisabling cuDNN benchmarking, and fixed dataloader ordering — and it will slow you down.\n\n**Version the data.** Model artifacts without the exact training data are not reproducible,\nand \"we retrained and got different numbers\" is otherwise unresolvable.\n\n### 12.2 The pipeline\n\n```\ndata ingestion → validation → feature engineering → training → EVALUATION GATE\n  → registry → deployment (shadow → canary → full) → MONITORING → retraining\n```\n**[DURABLE] The evaluation gate and monitoring are the parts people skip**, and they're\nthe parts that prevent silent disasters.\n\n**Feature stores** solve a real problem — **training\u002Fserving skew**, where the features\ncomputed at training time differ subtly from those computed at inference. **⚠️ This is one\nof the most common and hardest-to-find production bugs**, and if you compute features in\ntwo different codepaths you will eventually have it.\n\n### 12.3 Monitoring\n\nMonitor, in roughly this priority order: **prediction distribution** (drifts first and\ncheapest to watch), **input feature distributions** (PSI, KL divergence), **actual\nperformance** where labels eventually arrive, **latency and throughput**, **error rates**,\nand **business metrics** (the only ones that ultimately matter).\n\n**[DURABLE] Models degrade. Plan for retraining from day one** — trigger it on a schedule,\non drift detection, or on performance drop. And **keep the ability to roll back to the\nprevious model instantly**; a bad model deploy is an outage.\n\n---\n\n## §13. Hardware and Cost\n\n**[VERSIONED]**\n\n| Tier | Notes |\n|---|---|\n| **H100 (80 GB)** | ⚠️ **The software ecosystem is most mature here** — vLLM, SGLang, TensorRT-LLM, PyTorch, JAX. **Any optimization technique you read about was probably benchmarked on H100** |\n| **H200 (141 GB)** | Same die as H100 with HBM3e: **141 GB and 4.8 TB\u002Fs (+43% bandwidth)**. Often better per-token cost than 2× H100 for FP16 70B |\n| **Blackwell (B200\u002FGB200\u002FB300)** | **192 GB HBM3e** — a 70B FP16 model with real KV headroom, or 405B FP8 on two chips. NVFP4 support. ~3× faster LLM training than the prior generation by NVIDIA's own MLPerf claims |\n| **Rubin \u002F Vera** | The announced next generation, aimed at agentic training and inference |\n| **RTX PRO 6000 Blackwell (96 GB)** | Workstation-class; 4–8 in a rackmount is a standard 2026 self-hosting configuration |\n| **AMD MI300X \u002F ROCm** | Real and improving; **NVIDIA remains the default for vLLM, Unsloth, and most production workflows** |\n| **TPU v5e\u002Fv6, Trainium\u002FInferentia, Gaudi 3** | Viable, especially via JAX (TPU) or where you have committed cloud spend |\n\n**[DURABLE] For LLM inference, memory capacity and bandwidth dominate the selection\ndecision, not FLOPs** (§11.2). For training, interconnect (NVLink, InfiniBand) often\nmatters more than per-chip performance once you're multi-node.\n\n**Cost discipline**: cloud beats buying unless you keep hardware busy more than roughly\n8 hours\u002Fday; **spot\u002Fpreemptible instances plus checkpointing** are the single biggest\ntraining cost lever; profile before scaling up (most jobs are dataloader-bound or\nbadly-batched, not compute-bound); and **the cheapest optimization is a smaller model that\nmeets the requirement.**\n\n---\n\n## §14. Interpretability, Fairness, Safety\n\n### 14.1 Interpretability\n\n**Intrinsically interpretable models** — linear models, small trees, GAMs — are\nunderrated. **[DURABLE] If interpretability is a hard requirement, use an interpretable\nmodel rather than explaining a black box.**\n\n**Post-hoc methods**: **SHAP** (game-theoretic, the practical standard; ⚠️ correlated\nfeatures make attributions ambiguous), **LIME** (local surrogates; unstable),\n**permutation importance** (⚠️ misleading with correlated features), **partial dependence\nand ICE**, attention maps (⚠️ **attention is not explanation** — a well-known result;\nattention weights don't reliably indicate what drove the output), and **counterfactuals**\n(often the most *useful* form for an affected person).\n\n**⚠️ Feature importance is not causal.** A high-importance feature tells you what the model\nuses, not what drives the outcome. Acting on it as if it were causal is a recurring and\nexpensive error.\n\n### 14.2 Fairness\n\n**[DURABLE] The mathematics constrains you**: demographic parity, equalized odds, and\ncalibration within groups are **provably mutually incompatible** except in degenerate cases.\n**You must choose which fairness definition applies to your context** — there is no\ntechnically neutral option, and pretending otherwise is itself a choice.\n\nPractically: **slice every metric by group** (§9.2 — this is the same practice as good\nevaluation), audit training data for representation and historical bias, remember that\n**removing a protected attribute doesn't remove its influence** (proxies are everywhere),\nand document the model's intended use and limitations (model cards, datasheets).\n\n### 14.3 Security and safety\n\n**Adversarial examples** (small perturbations flip predictions — still largely unsolved),\n**data poisoning**, **model extraction**, **membership inference** and **training data\nextraction** (models memorize; verbatim regurgitation is real), and **prompt injection**\nfor LLM systems (⚠️ **not solved**, and any system giving an LLM tools plus untrusted input\nhas this exposure).\n\n**⚠️ `pickle` deserialization is arbitrary code execution.** Use **safetensors** for model\nweights. PyTorch's `weights_only=True` default since 2.6 addresses this, but third-party\ncheckpoints from unknown sources remain a real supply-chain risk.\n",{"name":5,"description":7},"develop","ae32736a1aabddabca5d2c96d665aa0ebc7d8700","src\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fmachine-learning\u002Fskills\u002Fml-evaluation-serving-mlops-and-safety",3,"https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey","https:\u002F\u002Fthe-vibey-project.github.io\u002Fvibey\u002F","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project","https:\u002F\u002Fvibewithadam.matthewsteinberger.com","freedom for software engineers","2026-11-15 05:47:42",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add the-vibey-project\u002Fvibey --skill ml-evaluation-serving-mlops-and-safety",{"tool":40,"command":41},"bunx","bunx skills add the-vibey-project\u002Fvibey --skill ml-evaluation-serving-mlops-and-safety",{"tool":43,"command":44},"pnpm","pnpm dlx skills add the-vibey-project\u002Fvibey --skill ml-evaluation-serving-mlops-and-safety","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fmachine-learning\u002Fskills\u002Fml-evaluation-serving-mlops-and-safety\u002FSKILL.md",true,[],[49,56,63,70,77,84],{"skillNo":50,"name":51,"nameZh":6,"description":52,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":53,"categories":54,"keywords":55,"publishedAt":19},"01M7AT1FQJAE1FQYAMDTF490Y4","white-paper-writing","Use when writing white papers, technical reports, policy documents, thought leadership pieces, or any long-form authoritative document. Triggers on requests to write, structure, outline, or review white papers, technical reports, position papers, policy briefs, or evidence-driven arguments. Also use for B2B content marketing strategy, technical documentation requiring authority, academic-style reports, or any document that must educate while persuading. Use when someone asks \"how should I structure this\" for a long-form document.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Fwhite-paper-writing\u002FSKILL.md",[],[],{"skillNo":57,"name":58,"nameZh":6,"description":59,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":60,"categories":61,"keywords":62,"publishedAt":19},"01M7AT16BJRJJKK2MASXRZS76G","prose-technical-mechanics","Use when editing or writing prose at a technical\u002Fmechanical level: choosing between Anglo-Saxon and Latinate diction, placing information in topic vs stress positions, writing periodic vs cumulative sentences, handling sentence rhythm, punctuation as logic, transitions, paragraph structure, or phonesthetics. Use when someone wants to understand WHY prose works or doesn't, not just surface-level style advice. Triggers on word choice, sentence structure, emphasis, rhythm, diction, or any deep writing mechanics question.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Fprose-technical-mechanics\u002FSKILL.md",[],[],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":19},"01M7AT0W9HT0ZZ9Y287NW99MR3","prose-mechanics","Use when writing, editing, or analyzing prose style; when trying to avoid generic AI-sounding writing; when applying specific prose styles such as journalistic, academic, narrative, KJV biblical, Hemingway plain, Latinate\u002FJamesian, Orwellian, Augustan, Victorian, Early Modern (Bacon, Browne, Milton), or free indirect discourse; or when coaching someone on writing voice and mechanics. Triggers on any request to improve, analyze, or write in a specific prose style, to fix \"AI-sounding\" writing, to make prose more vivid or concrete, to vary sentence rhythm, or to understand how a particular writer achieves their effects.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Fprose-mechanics\u002FSKILL.md",[],[],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M7AT0KR635JX66X2X6F3E2CP","narrative-structure","Use when structuring stories, pitches, presentations, white papers, or any persuasive narrative; analyzing the hero's journey in fiction or non-fiction; applying StoryBrand for business communication; designing narrative arcs for thought leadership; or understanding Vogler, Snyder, Propp, or Campbell frameworks. Triggers on any narrative structure, storytelling, pitch structure, StoryBrand, hero's journey, three-act structure, Freytag, Save the Cat, narrative arc, or \"how should I structure this story\u002Fpitch\u002Fpresentation\" request.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Fnarrative-structure\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":19},"01M7AT0AKGW09NKKZDYBT8VRDE","legal-brief-writing","Use when writing legal briefs, motions, or any court filing; when analyzing judicial cognition and persuasion; when applying elite brief-writing techniques from Roberts, Clement, Kagan, Garner; or when structuring legal arguments for maximum persuasion. Also use for autistic\u002Fneurodivergent advocates who need to understand how their natural communication style maps to elite legal advocacy. Triggers on any legal writing, brief drafting, motion preparation, or legal argument structuring request.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Flegal-brief-writing\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":19},"01M7AT00VWPS4Q8B5VEMXX9NQT","journal-publishability-criteria","Use whenever a research paper, article or preprint is being written, revised, reviewed or published, and specifically before every publication of docs\u002Fpaper.md. Covers what legitimate journals actually gate on, in order of gating — the editor's desk screen (scope, novelty, package problems), what reviewers judge (methodological soundness, claims the evidence can carry, statistical and qualitative reporting), EQUATOR reporting guidelines, the ICMJE authorship criteria, COPE's view of text recycling, the near-universal generative-AI policy (no AI author, disclosure, author responsibility), TOP 2025 transparency standards, how to verify a journal is legitimate, submitting without an institution, and a pre-submission checklist. Triggers on requests to evaluate, review, prepare or submit a paper, to judge publishability, to write a declarations or data-availability section, to disclose AI use, or to check a journal.","https:\u002F\u002Fgithub.com\u002Fthe-vibey-project\u002Fvibey\u002Fblob\u002Fdevelop\u002Fsrc\u002Fvibey_tools\u002Fskills\u002Fplugins\u002Fwriting-craft\u002Fskills\u002Fjournal-publishability-criteria\u002FSKILL.md",[],[]]