[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BRM3Q7Q0VC24FFJM0D85F8":3,"skill-same:01M7BRM3Q7Q0VC24FFJM0D85F8":61},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":6,"ownerHtmlUrl":29,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":30,"install":31,"resources":44},"01M7BRM3Q7Q0VC24FFJM0D85F8","rl-post-training",null,"Diagnostic guide for RL-based post-training of language models (GRPO, PPO, REINFORCE, DPO). Use proactively when debugging a training pipeline that shows no improvement, loss anomalies, reward stagnation, NaN gradients, or other unexpected behavior during reinforcement learning fine-tuning. Work through all pipeline stages — reward, advantages, log-probs, loss, generation\u002Fdecoding — before concluding the diagnosis is complete; stopping after one or two fixes is a common failure mode. Covers log-probability math, advantage estimation, numerical stability, reward processing, and generation\u002Fdecoding pipeline issues.","linyouxue\u002Fskill-repair-benchmark","linyouxue","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F216151464?v=4","01M7BJEVCXKD74KH6FG9CBVSQB","林宥学","01M7BJEVE02NP217V93BKZFC0J",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fevaluation\u002Fresults\u002FHumanAnnotated-Claude-Gold28-20261007\u002Ftasks\u002Fdebug-trl-grpo\u002Frepaired_skill\u002Frl-post-training\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: rl-post-training\ndescription: Diagnostic guide for RL-based post-training of language models (GRPO, PPO, REINFORCE, DPO). Use proactively when debugging a training pipeline that shows no improvement, loss anomalies, reward stagnation, NaN gradients, or other unexpected behavior during reinforcement learning fine-tuning. Work through all pipeline stages — reward, advantages, log-probs, loss, generation\u002Fdecoding — before concluding the diagnosis is complete; stopping after one or two fixes is a common failure mode. Covers log-probability math, advantage estimation, numerical stability, reward processing, and generation\u002Fdecoding pipeline issues.\n---\n\n# RL Post-Training — Concepts & Diagnostic Guide\n\n## Core Concepts\n\nRL post-training optimizes a language model's policy using reward signals. The standard pipeline:\n\n```\nprompt → generate completions → score with reward → compute advantages → policy gradient update\n```\n\nEach stage has distinct failure modes. When a model \"shows no improvement,\" the bug could be anywhere in this pipeline.\n\n## Diagnostic Methodology\n\nWhen RL training produces no improvement, work through these stages in order. Each stage depends on the previous one being correct.\n\n### Stage 1: Verify Reward Signal\n- Are rewards non-constant? If all rewards are identical, there is no learning signal.\n- Do rewards correlate with completion quality? Spot-check decoded completions against their scores.\n- Is the reward function being called on the correct text? Check that decoding\u002Fstripping preserves the content the reward function needs to evaluate.\n\n### Stage 2: Verify Advantage Computation\n- Are advantages non-zero when rewards vary? If they collapse to ~0, the policy gradient vanishes.\n- Check the magnitude and dtype of every numerical-stability constant in the advantage path (additive epsilons, clipping bounds). Compare each to what the math requires.\n- Check the group size `G`. `G ≤ 2` makes `std` either undefined or extremely noisy.\n\n### Stage 3: Verify Log-Probability Computation\n- Verify bounds: log-probs of valid tokens must be non-positive.\n- Compare your implementation against `F.log_softmax` on a small deterministic input — a numerical match rules out sign errors, wrong gathering axis, and off-by-one subtraction.\n- On a near-one-hot input, confirm the dominant token's log-prob is close to 0 (not close to the min).\n\n### Stage 4: Verify Loss Computation\n- Is the loss changing across steps? Flat loss suggests zero gradients upstream.\n- Log the KL term and the policy-gradient term separately. Either dominating the other is diagnostic.\n- Check the fraction of clipped samples. Near-100% clipping means the clip range is starving the signal.\n\n### Stage 5: Verify Generation and Decoding\n- Are padding and decoder artefacts stripped from the text the reward function sees?\n- Does every completion shape your model can emit survive the decoding path with non-empty output where a human would expect non-empty output? Include degenerate cases (no formatting markers, only a prefix, only a suffix) in the round-trip test.\n- Print or log a sample of the actual strings handed to the reward function — mismatches between \"what the model generated\" and \"what the reward saw\" are often visible at a glance.\n\n## Fixing, Not Rewriting\n\nA bug in a branched function is a bug in one branch, not a verdict on the whole function. Before editing:\n\n- Enumerate the input shapes the function handles today and the output each branch produces. The branches exist because callers rely on them.\n- Identify which input-output pairs violate the intended contract. Those are the only branches you need to change.\n- If your diff collapses a multi-branch function to a one-liner, you have almost certainly broken a contract a different caller depends on. Re-read the call sites before committing.\n\nThe same principle applies to epsilon values, sign conventions, and clipping bounds — if a constant looks wrong, replace it with a correct constant; don't remove the surrounding numerical-stability logic.\n\n## Key Mathematical Invariants\n\nThese invariants should always hold. If any is violated, there is a bug.\n\n| Invariant | What it means | How to check |\n|-----------|--------------|-------------|\n| `log_prob \u003C= 0` | Log of a probability is non-positive | `assert (log_probs \u003C= 1e-6).all()` |\n| log-probs match `F.log_softmax` | Manual implementation equals the reference | `torch.allclose(manual, F.log_softmax(...).gather(...))` |\n| `sum(softmax(logits)) == 1` | Probabilities sum to 1 | `assert torch.allclose(softmax.sum(-1), ones)` |\n| `0 \u003C epsilon \u003C\u003C 1` | Additive epsilons exist for numerical stability only | `assert 0 \u003C epsilon \u003C 1` |\n| `advantages != 0` when rewards vary | Non-constant rewards yield non-zero advantages | `assert advantages.abs().max() > 0.1` |\n| Decoding preserves content | Every generation shape the model emits survives the decode path with the content a human reader would expect | Round-trip representative samples; confirm non-empty where non-empty is expected |\n\n## Common Pitfall Categories\n\nDetailed pitfall catalog with examples is in `references\u002Fcommon-pitfalls.md`. Summary:\n\n1. **Sign errors in log-space** — log-softmax, KL divergence, DPO log-ratio\n2. **Numerical stability constants out of range** — additive epsilons, clip bounds, temperature\n3. **String processing in decoding** — cleanup that blanks shapes the rules didn't anticipate\n4. **Reward \u002F decoding format mismatch** — reward function sees text with the shape it needs removed\n5. **Reference model drift** — reference model not frozen, or gradients flowing through it\n6. **Gradient flow breakage** — detached tensors, in-place ops, `.item()`\u002Fnumpy round-trips in the loss path\n7. **Configuration misuse** — SFT-scale LR, KL coefficient dominating, clip range too tight, `G` too small\n\n## Available References\n\n| File | Contents | When to load |\n|------|----------|-------------|\n| `references\u002Fcommon-pitfalls.md` | Catalog of pitfall categories with symptoms and detection strategies | When you have a suspicious area and want to match it against known pattern shapes |\n| `references\u002Fdiagnostic-workflow.md` | Stage-by-stage diagnostic procedures with verification snippets | When you need guidance on which invariant to check in which order |\n| `scripts\u002Fverify_pipeline.py` | Runnable diagnostic that prints log-prob \u002F advantage \u002F decoding values on small fixed inputs so you can compare them against the invariants above | Before declaring a fix done: run this and inspect the output, paying attention to lines marked `?` |\n","# RL Post-Training — Concepts & Diagnostic Guide\n\n## Core Concepts\n\nRL post-training optimizes a language model's policy using reward signals. The standard pipeline:\n\n```\nprompt → generate completions → score with reward → compute advantages → policy gradient update\n```\n\nEach stage has distinct failure modes. When a model \"shows no improvement,\" the bug could be anywhere in this pipeline.\n\n## Diagnostic Methodology\n\nWhen RL training produces no improvement, work through these stages in order. Each stage depends on the previous one being correct.\n\n### Stage 1: Verify Reward Signal\n- Are rewards non-constant? If all rewards are identical, there is no learning signal.\n- Do rewards correlate with completion quality? Spot-check decoded completions against their scores.\n- Is the reward function being called on the correct text? Check that decoding\u002Fstripping preserves the content the reward function needs to evaluate.\n\n### Stage 2: Verify Advantage Computation\n- Are advantages non-zero when rewards vary? If they collapse to ~0, the policy gradient vanishes.\n- Check the magnitude and dtype of every numerical-stability constant in the advantage path (additive epsilons, clipping bounds). Compare each to what the math requires.\n- Check the group size `G`. `G ≤ 2` makes `std` either undefined or extremely noisy.\n\n### Stage 3: Verify Log-Probability Computation\n- Verify bounds: log-probs of valid tokens must be non-positive.\n- Compare your implementation against `F.log_softmax` on a small deterministic input — a numerical match rules out sign errors, wrong gathering axis, and off-by-one subtraction.\n- On a near-one-hot input, confirm the dominant token's log-prob is close to 0 (not close to the min).\n\n### Stage 4: Verify Loss Computation\n- Is the loss changing across steps? Flat loss suggests zero gradients upstream.\n- Log the KL term and the policy-gradient term separately. Either dominating the other is diagnostic.\n- Check the fraction of clipped samples. Near-100% clipping means the clip range is starving the signal.\n\n### Stage 5: Verify Generation and Decoding\n- Are padding and decoder artefacts stripped from the text the reward function sees?\n- Does every completion shape your model can emit survive the decoding path with non-empty output where a human would expect non-empty output? Include degenerate cases (no formatting markers, only a prefix, only a suffix) in the round-trip test.\n- Print or log a sample of the actual strings handed to the reward function — mismatches between \"what the model generated\" and \"what the reward saw\" are often visible at a glance.\n\n## Fixing, Not Rewriting\n\nA bug in a branched function is a bug in one branch, not a verdict on the whole function. Before editing:\n\n- Enumerate the input shapes the function handles today and the output each branch produces. The branches exist because callers rely on them.\n- Identify which input-output pairs violate the intended contract. Those are the only branches you need to change.\n- If your diff collapses a multi-branch function to a one-liner, you have almost certainly broken a contract a different caller depends on. Re-read the call sites before committing.\n\nThe same principle applies to epsilon values, sign conventions, and clipping bounds — if a constant looks wrong, replace it with a correct constant; don't remove the surrounding numerical-stability logic.\n\n## Key Mathematical Invariants\n\nThese invariants should always hold. If any is violated, there is a bug.\n\n| Invariant | What it means | How to check |\n|-----------|--------------|-------------|\n| `log_prob \u003C= 0` | Log of a probability is non-positive | `assert (log_probs \u003C= 1e-6).all()` |\n| log-probs match `F.log_softmax` | Manual implementation equals the reference | `torch.allclose(manual, F.log_softmax(...).gather(...))` |\n| `sum(softmax(logits)) == 1` | Probabilities sum to 1 | `assert torch.allclose(softmax.sum(-1), ones)` |\n| `0 \u003C epsilon \u003C\u003C 1` | Additive epsilons exist for numerical stability only | `assert 0 \u003C epsilon \u003C 1` |\n| `advantages != 0` when rewards vary | Non-constant rewards yield non-zero advantages | `assert advantages.abs().max() > 0.1` |\n| Decoding preserves content | Every generation shape the model emits survives the decode path with the content a human reader would expect | Round-trip representative samples; confirm non-empty where non-empty is expected |\n\n## Common Pitfall Categories\n\nDetailed pitfall catalog with examples is in `references\u002Fcommon-pitfalls.md`. Summary:\n\n1. **Sign errors in log-space** — log-softmax, KL divergence, DPO log-ratio\n2. **Numerical stability constants out of range** — additive epsilons, clip bounds, temperature\n3. **String processing in decoding** — cleanup that blanks shapes the rules didn't anticipate\n4. **Reward \u002F decoding format mismatch** — reward function sees text with the shape it needs removed\n5. **Reference model drift** — reference model not frozen, or gradients flowing through it\n6. **Gradient flow breakage** — detached tensors, in-place ops, `.item()`\u002Fnumpy round-trips in the loss path\n7. **Configuration misuse** — SFT-scale LR, KL coefficient dominating, clip range too tight, `G` too small\n\n## Available References\n\n| File | Contents | When to load |\n|------|----------|-------------|\n| `references\u002Fcommon-pitfalls.md` | Catalog of pitfall categories with symptoms and detection strategies | When you have a suspicious area and want to match it against known pattern shapes |\n| `references\u002Fdiagnostic-workflow.md` | Stage-by-stage diagnostic procedures with verification snippets | When you need guidance on which invariant to check in which order |\n| `scripts\u002Fverify_pipeline.py` | Runnable diagnostic that prints log-prob \u002F advantage \u002F decoding values on small fixed inputs so you can compare them against the invariants above | Before declaring a fix done: run this and inspect the output, paying attention to lines marked `?` |\n",{"name":5,"description":7},"main","cdcb6d230f3b15c3c33f8a2bdd16439f43602cbe","evaluation\u002Fresults\u002FHumanAnnotated-Claude-Gold28-20261007\u002Ftasks\u002Fdebug-trl-grpo\u002Frepaired_skill\u002Frl-post-training",1,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark","https:\u002F\u002Fgithub.com\u002Flinyouxue","2026-11-14 20:28:35",{"source":8,"skillName":5,"commands":32,"prompt":42,"zipAvailable":43},[33,36,39],{"tool":34,"command":35},"npx","npx skills add linyouxue\u002Fskill-repair-benchmark --skill rl-post-training",{"tool":37,"command":38},"bunx","bunx skills add linyouxue\u002Fskill-repair-benchmark --skill rl-post-training",{"tool":40,"command":41},"pnpm","pnpm dlx skills add linyouxue\u002Fskill-repair-benchmark --skill rl-post-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fevaluation\u002Fresults\u002FHumanAnnotated-Claude-Gold28-20261007\u002Ftasks\u002Fdebug-trl-grpo\u002Frepaired_skill\u002Frl-post-training\u002FSKILL.md",true,[45,51,55],{"path":46,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":49,"sourceHtmlUrl":50},"references\u002Fcommon-pitfalls.md","REFERENCE","参考文档",11361,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fevaluation\u002Fresults\u002FHumanAnnotated-Claude-Gold28-20261007\u002Ftasks\u002Fdebug-trl-grpo\u002Frepaired_skill\u002Frl-post-training\u002Freferences\u002Fcommon-pitfalls.md",{"path":52,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":53,"sourceHtmlUrl":54},"references\u002Fdiagnostic-workflow.md",7007,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fevaluation\u002Fresults\u002FHumanAnnotated-Claude-Gold28-20261007\u002Ftasks\u002Fdebug-trl-grpo\u002Frepaired_skill\u002Frl-post-training\u002Freferences\u002Fdiagnostic-workflow.md",{"path":56,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":59,"sourceHtmlUrl":60},"scripts\u002Fverify_pipeline.py","SCRIPT","脚本",8294,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fevaluation\u002Fresults\u002FHumanAnnotated-Claude-Gold28-20261007\u002Ftasks\u002Fdebug-trl-grpo\u002Frepaired_skill\u002Frl-post-training\u002Fscripts\u002Fverify_pipeline.py",[62,69,76,83,88,93],{"skillNo":63,"name":64,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":68,"publishedAt":19},"01M7E1HGVRY73NBJ022CJG13X6","video-frame-extraction","Extract frames from video files and save them as images using OpenCV","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_03\u002Finputs\u002Fskills\u002Fvideo-frame-extraction\u002FSKILL.md",[],[],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":19},"01M7E1HAG5R7CJ9C8NYJS771MB","openai-vision","Analyze images and multi-frame sequences using OpenAI GPT vision models","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_03\u002Finputs\u002Fskills\u002Fopenai-vision\u002FSKILL.md",[],[],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":19},"01M7E1H7DADKB6QBWKC2KQ63SZ","image-ocr","Extract text content from images using Tesseract OCR via Python, including structured field extraction from receipts and forms","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_03\u002Finputs\u002Fskills\u002Fimage-ocr\u002FSKILL.md",[],[],{"skillNo":84,"name":64,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M7E1H0WBNT43TENGQY7AVZHD","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_02\u002Finputs\u002Fskills\u002Fvideo-frame-extraction\u002FSKILL.md",[],[],{"skillNo":89,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":19},"01M7E1GTC7D4N4Q2EF03875J9E","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_02\u002Finputs\u002Fskills\u002Fopenai-vision\u002FSKILL.md",[],[],{"skillNo":94,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":19},"01M7E1GQ92CWFMGH53KSSE4WER","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_02\u002Finputs\u002Fskills\u002Fimage-ocr\u002FSKILL.md",[],[]]