[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7EMW8WGD71CKBQW7WXCDRQJ":3,"skill-same:01M7EMW8WGD71CKBQW7WXCDRQJ":47},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":28,"repositoryHtmlUrl":29,"repositoryHomepageUrl":6,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":6,"updatedAt":32,"install":33,"resources":46},"01M7EMW8WGD71CKBQW7WXCDRQJ","dpo-guide",null,"Align language models with human preferences using Direct Preference Optimization — no reward model, no RL loop.","TanNhatCMS\u002Fjarvis","TanNhatCMS","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F113796420?v=4","01M753MV17KG5CD3152G3WQEQY","KietNT","01M7EKEDYVYQ6PWVJJMZ911VMX",0,"MIT","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fams-dpo-guide\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: dpo-guide\ndescription: Align language models with human preferences using Direct Preference Optimization — no reward model, no RL loop.\ncategory: ai-research\n---\n\n## Overview\n\nDirect Preference Optimization (DPO) trains a language model directly on preference\npairs — (prompt, chosen, rejected) — without training a separate reward model or\nrunning reinforcement learning. The key mathematical insight is that the optimal\npolicy under a KL-constrained reward objective has a closed form, which lets you\nreframe \"maximize this implicit reward\" as a simple classification loss on the\npolicy itself: increase the likelihood of the chosen response relative to the\nrejected one, scaled by how far the policy has moved from a reference model.\n\nIn practice, DPO is dramatically simpler to run than PPO-based RLHF: it's a single\nsupervised-style training loop with standard hyperparameters, stable gradients,\nand no value network, advantage estimation, or KL scheduling. It has become the\ndefault preference-tuning method for most practitioners. Its limits: it's an\noffline method (it can't explore beyond the training pairs), and like all\npreference methods it's only as good as the preference data.\n\nThink of DPO as the 80\u002F20 of preference tuning: most of RLHF's practical benefit\nat a fraction of the complexity. Reach for PPO when you need online exploration;\nreach for DPO for everything else.\n\n## When to use\n\n- Preference-tuning a model when you lack the infrastructure or appetite for a full\n  PPO pipeline.\n- Fine-tuning open models on human or AI-generated preference pairs for style,\n  helpfulness, or safety.\n- Rapid iteration on alignment behavior: DPO trains in hours, letting you test\n  data and hyperparameter changes quickly.\n- Distilling preferences from a stronger model (generate pairs, have the strong\n  model judge, DPO-tune the smaller one).\n- As the final stage after SFT in a standard post-training recipe.\n- Small teams and researchers who need preference tuning without RL expertise.\n\n## Core concepts\n\n- **The DPO loss**: for each triple, the loss pushes up log π(chosen)\u002Fπ_ref(chosen)\n  − log π(rejected)\u002Fπ_ref(rejected), passed through a sigmoid scaled by β.\n  Intuitively: make the good answer relatively more likely than the bad one,\n  without drifting far from the reference model.\n- **β (beta)**: the temperature controlling how strongly preferences reshape the\n  policy. Low β → aggressive updates, risk of overfitting to pairs. High β →\n  conservative, stays near the reference. Typical starting range: 0.1–0.5.\n- **Reference model**: usually the SFT checkpoint you started from, kept frozen.\n  The KL-to-reference term is what prevents collapse — don't skip it by using a\n  mismatched reference.\n- **Implicit reward**: DPO never materializes a reward function, but you can read\n  one off: r(x,y) = β·log(π(y|x)\u002Fπ_ref(y|x)). Useful for debugging which pairs the\n  model \"believes.\"\n- **On-policy vs. off-policy**: DPO is off-policy — it learns from fixed pairs. If\n  the policy drifts far from the data distribution, the implicit reward\n  extrapolates badly. Iterative DPO (re-collect pairs from the current policy)\n  fixes this at the cost of complexity.\n- **Variants**: IPO (adds regularization against overfitting to deterministic\n  preferences), KTO (works with binary good\u002Fbad labels instead of pairs), ORPO\n  (merges SFT and preference objectives, no reference model needed), SimPO\n  (reference-free with length normalization).\n- **Pair quality**: the entire method hinges on what distinguishes chosen from\n  rejected. Informative pairs differ on the target behavior; noisy pairs teach\n  noise.\n- **Length dynamics**: like all preference methods, DPO can learn \"longer is\n  better\" if chosen responses are systematically longer. Watch output length as a\n  training metric.\n\n## Practical workflow\n\n1. **Prepare preference triples.** (prompt, chosen, rejected). Chosen and rejected\n   should be genuinely comparable — pairs where one is obviously garbage teach the\n   model nothing about the decision boundary you care about.\n2. **Audit pair quality.** Sample 100+ pairs manually. Check: does the chosen\n   response actually demonstrate the desired behavior? Is the rejected one wrong in\n   the relevant way, not just worse overall?\n3. **Start from a good SFT model.** DPO refines; it doesn't create capabilities.\n   The reference model is this SFT checkpoint.\n4. **Set hyperparameters.** Learning rate ~5e-7 to 1e-6 (lower than SFT),\n   β ≈ 0.1–0.5, 1–3 epochs. Watch the implicit reward margin: it should grow\n   steadily, not explode.\n5. **Train and monitor.** Track: loss, reward margin (chosen − rejected implicit\n   reward), output length, and — critically — win-rate on a held-out eval judged\n   by humans or a strong judge model. Also track KL from reference; a collapsing\n   KL with rising margin means overfitting.\n6. **Evaluate on behavior, not loss.** Run your safety and quality evals. Check\n   for the classic DPO pathologies: verbosity increase, sycophancy, and degraded\n   performance on tasks absent from the preference data.\n7. **Iterate on data.** If a behavior isn't changing, the pairs probably don't\n   isolate it. Add targeted pairs for that specific behavior rather than more\n   generic data.\n\nChecklist for a DPO run:\n- SFT baseline evaluated before starting, so you can measure the delta.\n- Pair quality audited manually on a sample.\n- Held-out preference accuracy improves without KL collapse.\n- Human or strong-judge win-rate confirms the improvement is real.\n- Capability benchmarks (reasoning, knowledge) haven't regressed.\n- Output length monitored for verbosity drift.\n\n## Common pitfalls\n\n- **Garbage pairs.** DPO amplifies whatever distinguishes chosen from rejected. If\n  chosen responses are just longer, you train a verbosity maximizer. Audit pair\n  quality first.\n- **β too low.** The policy overfits to the pairs, producing extreme probability\n  shifts and degenerate outputs on anything slightly off-distribution.\n- **Too many epochs.** DPO overfits fast — often 1 epoch is enough. More epochs\n  rarely help and frequently hurt.\n- **Mismatched reference.** Using a different checkpoint as reference than the one\n  training started from breaks the KL anchoring the theory relies on.\n- **Expecting exploration.** DPO can't discover behaviors absent from the data. If\n  no pair demonstrates the desired behavior, DPO won't invent it — that's a data\n  problem, not a hyperparameter problem.\n- **Judging by loss alone.** Training loss decreasing while human-judged quality\n  flatlines is common. Always keep an independent behavioral eval.\n- **Ignoring length.** Not tracking output length during training, then wondering\n  why the model rambles. Length is a first-class metric here.\n- **One big undifferentiated dataset.** Mixing safety pairs, style pairs, and\n  helpfulness pairs without tracking which behavior each subset drives. When\n  something goes wrong, you can't attribute it.\n","## Overview\n\nDirect Preference Optimization (DPO) trains a language model directly on preference\npairs — (prompt, chosen, rejected) — without training a separate reward model or\nrunning reinforcement learning. The key mathematical insight is that the optimal\npolicy under a KL-constrained reward objective has a closed form, which lets you\nreframe \"maximize this implicit reward\" as a simple classification loss on the\npolicy itself: increase the likelihood of the chosen response relative to the\nrejected one, scaled by how far the policy has moved from a reference model.\n\nIn practice, DPO is dramatically simpler to run than PPO-based RLHF: it's a single\nsupervised-style training loop with standard hyperparameters, stable gradients,\nand no value network, advantage estimation, or KL scheduling. It has become the\ndefault preference-tuning method for most practitioners. Its limits: it's an\noffline method (it can't explore beyond the training pairs), and like all\npreference methods it's only as good as the preference data.\n\nThink of DPO as the 80\u002F20 of preference tuning: most of RLHF's practical benefit\nat a fraction of the complexity. Reach for PPO when you need online exploration;\nreach for DPO for everything else.\n\n## When to use\n\n- Preference-tuning a model when you lack the infrastructure or appetite for a full\n  PPO pipeline.\n- Fine-tuning open models on human or AI-generated preference pairs for style,\n  helpfulness, or safety.\n- Rapid iteration on alignment behavior: DPO trains in hours, letting you test\n  data and hyperparameter changes quickly.\n- Distilling preferences from a stronger model (generate pairs, have the strong\n  model judge, DPO-tune the smaller one).\n- As the final stage after SFT in a standard post-training recipe.\n- Small teams and researchers who need preference tuning without RL expertise.\n\n## Core concepts\n\n- **The DPO loss**: for each triple, the loss pushes up log π(chosen)\u002Fπ_ref(chosen)\n  − log π(rejected)\u002Fπ_ref(rejected), passed through a sigmoid scaled by β.\n  Intuitively: make the good answer relatively more likely than the bad one,\n  without drifting far from the reference model.\n- **β (beta)**: the temperature controlling how strongly preferences reshape the\n  policy. Low β → aggressive updates, risk of overfitting to pairs. High β →\n  conservative, stays near the reference. Typical starting range: 0.1–0.5.\n- **Reference model**: usually the SFT checkpoint you started from, kept frozen.\n  The KL-to-reference term is what prevents collapse — don't skip it by using a\n  mismatched reference.\n- **Implicit reward**: DPO never materializes a reward function, but you can read\n  one off: r(x,y) = β·log(π(y|x)\u002Fπ_ref(y|x)). Useful for debugging which pairs the\n  model \"believes.\"\n- **On-policy vs. off-policy**: DPO is off-policy — it learns from fixed pairs. If\n  the policy drifts far from the data distribution, the implicit reward\n  extrapolates badly. Iterative DPO (re-collect pairs from the current policy)\n  fixes this at the cost of complexity.\n- **Variants**: IPO (adds regularization against overfitting to deterministic\n  preferences), KTO (works with binary good\u002Fbad labels instead of pairs), ORPO\n  (merges SFT and preference objectives, no reference model needed), SimPO\n  (reference-free with length normalization).\n- **Pair quality**: the entire method hinges on what distinguishes chosen from\n  rejected. Informative pairs differ on the target behavior; noisy pairs teach\n  noise.\n- **Length dynamics**: like all preference methods, DPO can learn \"longer is\n  better\" if chosen responses are systematically longer. Watch output length as a\n  training metric.\n\n## Practical workflow\n\n1. **Prepare preference triples.** (prompt, chosen, rejected). Chosen and rejected\n   should be genuinely comparable — pairs where one is obviously garbage teach the\n   model nothing about the decision boundary you care about.\n2. **Audit pair quality.** Sample 100+ pairs manually. Check: does the chosen\n   response actually demonstrate the desired behavior? Is the rejected one wrong in\n   the relevant way, not just worse overall?\n3. **Start from a good SFT model.** DPO refines; it doesn't create capabilities.\n   The reference model is this SFT checkpoint.\n4. **Set hyperparameters.** Learning rate ~5e-7 to 1e-6 (lower than SFT),\n   β ≈ 0.1–0.5, 1–3 epochs. Watch the implicit reward margin: it should grow\n   steadily, not explode.\n5. **Train and monitor.** Track: loss, reward margin (chosen − rejected implicit\n   reward), output length, and — critically — win-rate on a held-out eval judged\n   by humans or a strong judge model. Also track KL from reference; a collapsing\n   KL with rising margin means overfitting.\n6. **Evaluate on behavior, not loss.** Run your safety and quality evals. Check\n   for the classic DPO pathologies: verbosity increase, sycophancy, and degraded\n   performance on tasks absent from the preference data.\n7. **Iterate on data.** If a behavior isn't changing, the pairs probably don't\n   isolate it. Add targeted pairs for that specific behavior rather than more\n   generic data.\n\nChecklist for a DPO run:\n- SFT baseline evaluated before starting, so you can measure the delta.\n- Pair quality audited manually on a sample.\n- Held-out preference accuracy improves without KL collapse.\n- Human or strong-judge win-rate confirms the improvement is real.\n- Capability benchmarks (reasoning, knowledge) haven't regressed.\n- Output length monitored for verbosity drift.\n\n## Common pitfalls\n\n- **Garbage pairs.** DPO amplifies whatever distinguishes chosen from rejected. If\n  chosen responses are just longer, you train a verbosity maximizer. Audit pair\n  quality first.\n- **β too low.** The policy overfits to the pairs, producing extreme probability\n  shifts and degenerate outputs on anything slightly off-distribution.\n- **Too many epochs.** DPO overfits fast — often 1 epoch is enough. More epochs\n  rarely help and frequently hurt.\n- **Mismatched reference.** Using a different checkpoint as reference than the one\n  training started from breaks the KL anchoring the theory relies on.\n- **Expecting exploration.** DPO can't discover behaviors absent from the data. If\n  no pair demonstrates the desired behavior, DPO won't invent it — that's a data\n  problem, not a hyperparameter problem.\n- **Judging by loss alone.** Training loss decreasing while human-judged quality\n  flatlines is common. Always keep an independent behavioral eval.\n- **Ignoring length.** Not tracking output length during training, then wondering\n  why the model rambles. Length is a first-class metric here.\n- **One big undifferentiated dataset.** Mixing safety pairs, style pairs, and\n  helpfulness pairs without tracking which behavior each subset drives. When\n  something goes wrong, you can't attribute it.\n",{"name":5,"category":24,"description":7},"ai-research","main","af388a77c2fe5e0b4fd879cca4300a0ffa9a6579","skills\u002Fams-dpo-guide",1,"https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis","https:\u002F\u002Fgithub.com\u002FTanNhatCMS","https:\u002F\u002Fmrkiet.name.vn","2026-11-07 14:23:08",{"source":8,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add TanNhatCMS\u002Fjarvis --skill dpo-guide",{"tool":39,"command":40},"bunx","bunx skills add TanNhatCMS\u002Fjarvis --skill dpo-guide",{"tool":42,"command":43},"pnpm","pnpm dlx skills add TanNhatCMS\u002Fjarvis --skill dpo-guide","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fams-dpo-guide\u002FSKILL.md",true,[],[48,55,63,70,77,84],{"skillNo":49,"name":50,"nameZh":6,"description":51,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":52,"categories":53,"keywords":54,"publishedAt":19},"01M7EWYJR0J8DARY7E59Z6Y9MV","web-scrape","Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open\u002Ffetch fails on JS-heavy pages, or when you need clean markdown\u002Fstructured extraction from a site. Requires FIRECRAWL_API_KEY.","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fweb-scrape\u002FSKILL.md",[],[],{"skillNo":56,"name":57,"nameZh":6,"description":58,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":59,"sourceHtmlUrl":60,"categories":61,"keywords":62,"publishedAt":19},"01M7EWYBN5P6F2AJG3FR24T9BP","design-taste-frontend","tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML\u002FCSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, \"AI-made\", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Ftaste-design\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Ftaste-design\u002FSKILL.md",[],[],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":19},"01M7EWY8FQFNQ2GB7NZZWGSSDX","test-driven-development","Use when implementing any feature or bugfix, before writing implementation code","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-tdd\u002FSKILL.md",[],[],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M7EWY5SHBFRH1HDHVWM9K7TV","systematic-debugging","Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-debugging\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":19},"01M7EWY3NM0Z0JAKS7HDZ24K66","requesting-code-review","Use when completing tasks, implementing major features, or before merging to verify work meets requirements","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-code-review\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":19},"01M7EWY0WRMWX4M9ZFY0WKC593","brainstorming","You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-brainstorming\u002FSKILL.md",[],[]]