[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BWHTDT1BASJY1W7436MF2T":3,"skill-same:01M7BWHTDT1BASJY1W7436MF2T":55},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":6,"ownerHtmlUrl":29,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":30,"install":31,"resources":44},"01M7BWHTDT1BASJY1W7436MF2T","grpo",null,"Reference for the GRPO (Group Relative Policy Optimization) algorithm. Use when implementing, debugging, or verifying a GRPO training pipeline — covers the mathematical formulation (group-relative advantages, clipped surrogate loss, KL penalty), the training loop (generate → score → advantage → loss), log-probability computation, advantage estimation, and relationship to PPO\u002FREINFORCE.","linyouxue\u002Fskill-repair-benchmark","linyouxue","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F216151464?v=4","01M7BJEVCXKD74KH6FG9CBVSQB","林宥学","01M7BJEVE02NP217V93BKZFC0J",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fevaluation\u002Fresults\u002FOpus47-OriginalSkill-79Tasks-20260920\u002Fruns\u002Fopus47-original-skill\u002Fdebug-trl-grpo-opus47-original-skill-r002\u002Finputs\u002Fskills\u002Fgrpo\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: grpo\ndescription: Reference for the GRPO (Group Relative Policy Optimization) algorithm. Use when implementing, debugging, or verifying a GRPO training pipeline — covers the mathematical formulation (group-relative advantages, clipped surrogate loss, KL penalty), the training loop (generate → score → advantage → loss), log-probability computation, advantage estimation, and relationship to PPO\u002FREINFORCE.\n---\n\n# GRPO Algorithm Reference\n\n## Overview\n\nGroup Relative Policy Optimization (GRPO) is a policy gradient method that eliminates the need for a learned critic by computing advantages from group statistics. For each prompt, multiple completions are sampled and their rewards are normalized within the group to produce advantages.\n\n**Key insight:** Instead of training a value function to estimate `V(s)`, GRPO uses the mean reward of the group as the baseline. This removes the critic entirely, reducing memory and avoiding value function approximation errors.\n\n## Training Loop\n\nEach step follows this pipeline:\n\n```\n1. Sample G completions per prompt from current policy\n2. Decode completions to text\n3. Score completions with reward function\n4. Compute group-relative advantages\n5. Compute per-token log-probs (current policy + reference policy)\n6. Compute clipped surrogate loss with KL penalty\n7. Backpropagate and update\n```\n\n## Advantage Estimation\n\nRewards are normalized within each group of `G` completions for the same prompt:\n\n```\nmu   = mean(r_1, ..., r_G)\nsigma = std(r_1, ..., r_G)\nA_i  = (r_i - mu) \u002F (sigma + epsilon)\n```\n\n**`epsilon`** is a small constant (typically 1e-4 to 1e-8) for numerical stability only. It prevents division by zero when all rewards in a group are identical.\n\n**Properties:**\n- Advantages within each group sum to approximately zero;\n- High-reward completions get positive advantages, low-reward get negative;\n- The learning signal vanishes if epsilon is too large (dominates denominator) or if rewards are constant. For example, if all rewards in a group are identical (or nearly identical), then causing all advantages to collapse to 0;\n\n## Log-Probability Computation\n\nPer-token log probabilities via log-softmax:\n```\nlog_prob(token_i) = logit(token_i) - logsumexp(logits)\n```\n\n**Critical invariant:** `log_prob \u003C= 0` always, since it's the log of a probability in (0, 1].\n\nSequence-level log probability: `log_pi(y|x) = sum_t log_pi(t_j | x, t_\u003Cj)`\n\n## Loss Function\n\nThe GRPO loss combines a clipped surrogate objective with a KL divergence penalty:\n\n```\nratio = exp(log_pi_theta(y|x) - log_pi_old(y|x))\nL_clip = min(ratio * A, clip(ratio, 1-eps, 1+eps) * A)\nL_kl = beta * KL(pi_theta || pi_ref)\nloss = -E[L_clip] + L_kl\n```\n\n| Component | Purpose |\n|-----------|---------|\n| `ratio` | How much the policy has changed from the generation policy |\n| Clipping | Prevents destructively large policy updates |\n| `beta * KL` | Keeps the policy close to the reference (prevents degeneration) |\n\n## Key Hyperparameters\n\n| Parameter | Typical Range | Effect |\n|-----------|--------------|--------|\n| `num_generations` (G) | 4–16 | More = lower variance advantages, higher compute cost |\n| `beta` | 0.01–0.1 | Higher = more conservative updates (closer to reference) |\n| `epsilon` (clip) | 0.1–0.2 | Narrower = more conservative updates |\n| `epsilon` (advantage) | 1e-8–1e-4 | Must be small; only for numerical stability |\n| `learning_rate` | 1e-7–5e-6 | Much lower than SFT; RL is sensitive to LR |\n\n## Available References\n\n| File | Contents                                                                                                                                          | When to load |\n|------|---------------------------------------------------------------------------------------------------------------------------------------------------|-------------|\n| `references\u002Fgrpo-algorithm.md` | Full mathematical formulation with notation table, step-by-step derivations, comparison to PPO and REINFORCE                                      | When you need to verify whether a specific implementation detail matches the algorithm specification |\n| `references\u002Fgrpo-trainer-internals.md` | GRPOTrainer implementation from popular frameworks (TRL): method-by-method breakdown, data flow diagram, and how each algorithm step maps to code | When tracing bugs through a GRPOTrainer implementation or understanding how the algorithm maps to specific code paths |\n","# GRPO Algorithm Reference\n\n## Overview\n\nGroup Relative Policy Optimization (GRPO) is a policy gradient method that eliminates the need for a learned critic by computing advantages from group statistics. For each prompt, multiple completions are sampled and their rewards are normalized within the group to produce advantages.\n\n**Key insight:** Instead of training a value function to estimate `V(s)`, GRPO uses the mean reward of the group as the baseline. This removes the critic entirely, reducing memory and avoiding value function approximation errors.\n\n## Training Loop\n\nEach step follows this pipeline:\n\n```\n1. Sample G completions per prompt from current policy\n2. Decode completions to text\n3. Score completions with reward function\n4. Compute group-relative advantages\n5. Compute per-token log-probs (current policy + reference policy)\n6. Compute clipped surrogate loss with KL penalty\n7. Backpropagate and update\n```\n\n## Advantage Estimation\n\nRewards are normalized within each group of `G` completions for the same prompt:\n\n```\nmu   = mean(r_1, ..., r_G)\nsigma = std(r_1, ..., r_G)\nA_i  = (r_i - mu) \u002F (sigma + epsilon)\n```\n\n**`epsilon`** is a small constant (typically 1e-4 to 1e-8) for numerical stability only. It prevents division by zero when all rewards in a group are identical.\n\n**Properties:**\n- Advantages within each group sum to approximately zero;\n- High-reward completions get positive advantages, low-reward get negative;\n- The learning signal vanishes if epsilon is too large (dominates denominator) or if rewards are constant. For example, if all rewards in a group are identical (or nearly identical), then causing all advantages to collapse to 0;\n\n## Log-Probability Computation\n\nPer-token log probabilities via log-softmax:\n```\nlog_prob(token_i) = logit(token_i) - logsumexp(logits)\n```\n\n**Critical invariant:** `log_prob \u003C= 0` always, since it's the log of a probability in (0, 1].\n\nSequence-level log probability: `log_pi(y|x) = sum_t log_pi(t_j | x, t_\u003Cj)`\n\n## Loss Function\n\nThe GRPO loss combines a clipped surrogate objective with a KL divergence penalty:\n\n```\nratio = exp(log_pi_theta(y|x) - log_pi_old(y|x))\nL_clip = min(ratio * A, clip(ratio, 1-eps, 1+eps) * A)\nL_kl = beta * KL(pi_theta || pi_ref)\nloss = -E[L_clip] + L_kl\n```\n\n| Component | Purpose |\n|-----------|---------|\n| `ratio` | How much the policy has changed from the generation policy |\n| Clipping | Prevents destructively large policy updates |\n| `beta * KL` | Keeps the policy close to the reference (prevents degeneration) |\n\n## Key Hyperparameters\n\n| Parameter | Typical Range | Effect |\n|-----------|--------------|--------|\n| `num_generations` (G) | 4–16 | More = lower variance advantages, higher compute cost |\n| `beta` | 0.01–0.1 | Higher = more conservative updates (closer to reference) |\n| `epsilon` (clip) | 0.1–0.2 | Narrower = more conservative updates |\n| `epsilon` (advantage) | 1e-8–1e-4 | Must be small; only for numerical stability |\n| `learning_rate` | 1e-7–5e-6 | Much lower than SFT; RL is sensitive to LR |\n\n## Available References\n\n| File | Contents                                                                                                                                          | When to load |\n|------|---------------------------------------------------------------------------------------------------------------------------------------------------|-------------|\n| `references\u002Fgrpo-algorithm.md` | Full mathematical formulation with notation table, step-by-step derivations, comparison to PPO and REINFORCE                                      | When you need to verify whether a specific implementation detail matches the algorithm specification |\n| `references\u002Fgrpo-trainer-internals.md` | GRPOTrainer implementation from popular frameworks (TRL): method-by-method breakdown, data flow diagram, and how each algorithm step maps to code | When tracing bugs through a GRPOTrainer implementation or understanding how the algorithm maps to specific code paths |\n",{"name":5,"description":7},"main","cdcb6d230f3b15c3c33f8a2bdd16439f43602cbe","evaluation\u002Fresults\u002FOpus47-OriginalSkill-79Tasks-20260920\u002Fruns\u002Fopus47-original-skill\u002Fdebug-trl-grpo-opus47-original-skill-r002\u002Finputs\u002Fskills\u002Fgrpo",1,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark","https:\u002F\u002Fgithub.com\u002Flinyouxue","2026-11-14 20:28:35",{"source":8,"skillName":5,"commands":32,"prompt":42,"zipAvailable":43},[33,36,39],{"tool":34,"command":35},"npx","npx skills add linyouxue\u002Fskill-repair-benchmark --skill grpo",{"tool":37,"command":38},"bunx","bunx skills add linyouxue\u002Fskill-repair-benchmark --skill grpo",{"tool":40,"command":41},"pnpm","pnpm dlx skills add linyouxue\u002Fskill-repair-benchmark --skill grpo","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fevaluation\u002Fresults\u002FOpus47-OriginalSkill-79Tasks-20260920\u002Fruns\u002Fopus47-original-skill\u002Fdebug-trl-grpo-opus47-original-skill-r002\u002Finputs\u002Fskills\u002Fgrpo\u002FSKILL.md",true,[45,51],{"path":46,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":49,"sourceHtmlUrl":50},"references\u002Fgrpo-algorithm.md","REFERENCE","参考文档",4980,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fevaluation\u002Fresults\u002FOpus47-OriginalSkill-79Tasks-20260920\u002Fruns\u002Fopus47-original-skill\u002Fdebug-trl-grpo-opus47-original-skill-r002\u002Finputs\u002Fskills\u002Fgrpo\u002Freferences\u002Fgrpo-algorithm.md",{"path":52,"resourceType":47,"resourceTypeDesc":48,"sizeBytes":53,"sourceHtmlUrl":54},"references\u002Fgrpo-trainer-internals.md",3954,"https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fevaluation\u002Fresults\u002FOpus47-OriginalSkill-79Tasks-20260920\u002Fruns\u002Fopus47-original-skill\u002Fdebug-trl-grpo-opus47-original-skill-r002\u002Finputs\u002Fskills\u002Fgrpo\u002Freferences\u002Fgrpo-trainer-internals.md",[56,63,70,77,82,87],{"skillNo":57,"name":58,"nameZh":6,"description":59,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":60,"categories":61,"keywords":62,"publishedAt":19},"01M7E1HGVRY73NBJ022CJG13X6","video-frame-extraction","Extract frames from video files and save them as images using OpenCV","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_03\u002Finputs\u002Fskills\u002Fvideo-frame-extraction\u002FSKILL.md",[],[],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":19},"01M7E1HAG5R7CJ9C8NYJS771MB","openai-vision","Analyze images and multi-frame sequences using OpenAI GPT vision models","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_03\u002Finputs\u002Fskills\u002Fopenai-vision\u002FSKILL.md",[],[],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M7E1H7DADKB6QBWKC2KQ63SZ","image-ocr","Extract text content from images using Tesseract OCR via Python, including structured field extraction from receipts and forms","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_03\u002Finputs\u002Fskills\u002Fimage-ocr\u002FSKILL.md",[],[],{"skillNo":78,"name":58,"nameZh":6,"description":59,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":81,"publishedAt":19},"01M7E1H0WBNT43TENGQY7AVZHD","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_02\u002Finputs\u002Fskills\u002Fvideo-frame-extraction\u002FSKILL.md",[],[],{"skillNo":83,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":19},"01M7E1GTC7D4N4Q2EF03875J9E","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_02\u002Finputs\u002Fskills\u002Fopenai-vision\u002FSKILL.md",[],[],{"skillNo":88,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":19},"01M7E1GQ92CWFMGH53KSSE4WER","https:\u002F\u002Fgithub.com\u002Flinyouxue\u002Fskill-repair-benchmark\u002Fblob\u002Fmain\u002Fjpg-ocr-stat\u002Fresult\u002Fskilltta-claude4.7\u002Fgroup_03\u002Frun_02\u002Finputs\u002Fskills\u002Fimage-ocr\u002FSKILL.md",[],[]]