[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6KJDSG9F9X3TP47KZ3ABPPE":3,"skill-same:01M6KJDSG9F9X3TP47KZ3ABPPE":65},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":38,"linkOnly":39,"markdownContent":40,"markdownBody":41,"frontMatter":42,"refName":43,"commitSha":44,"skillDirectory":45,"forksCount":46,"repositoryHtmlUrl":47,"repositoryHomepageUrl":48,"ownerHtmlUrl":49,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":50,"install":51,"resources":64},"01M6KJDSG9F9X3TP47KZ3ABPPE","simpo-loss-implementation",null,"Implement the SimPO loss function for preference-aligned language models. Use this skill when implementing SimPO training objectives, computing length-normalized rewards, or translating the SimPO paper's mathematical formulas into PyTorch code. Essential for LLM alignment tasks that require preference-based learning with reference-free rewards.","cxcscmu\u002FSkillLearnBench","cxcscmu","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F142838272?v=4","01M6KCQDFQWPDSN0MKZY96MSQP","01M6KCQDM7G70TKJ28NTSQC0RZ",93,"MIT","https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench\u002Fblob\u002Fmain\u002Fskills\u002Fb4-skill-creator-claude-haiku-4-5\u002Fnlp-paper-reproduction\u002Fsimpo-loss-implementation\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37],"simpo loss implementation","simpo loss","implementation","simpo","essential","for","implementing","formulas","LLM Alignment","Reference-Free","free","implement","simpo training","alignment","use","learning","code","tasks","training","pytorch","2026-11-10 10:12:30",false,"---\nname: simpo-loss-implementation\ndescription: Implement the SimPO loss function for preference-aligned language models. Use this skill when implementing SimPO training objectives, computing length-normalized rewards, or translating the SimPO paper's mathematical formulas into PyTorch code. Essential for LLM alignment tasks that require preference-based learning with reference-free rewards.\n---\n\n# SimPO Loss Implementation\n\n## Overview\n\nSimPO (Simple Preference Optimization) is a reference-free preference optimization method that trains language models using pairwise preference data. The loss function combines length-normalized rewards with a Bradley-Terry preference model.\n\n## Core Mathematical Formulas\n\n### 1. Length-Normalized Reward\n```\nr_SimPO(x, y) = (β\u002F|y|) * log π_θ(y|x)\n```\n\nWhere:\n- **β**: Scaling constant (typically 2.0)\n- **|y|**: Response length (number of tokens)\n- **log π_θ(y|x)**: Average log probability per token (not summed)\n\n### 2. Bradley-Terry Preference with Margin\n```\np(y_w ≻ y_l | x) = σ(r(x, y_w) - r(x, y_l) - γ)\n```\n\nWhere:\n- **γ**: Target reward margin (calculated as γ = β * gamma_beta_ratio)\n- **σ(·)**: Sigmoid function\n- **y_w**: Chosen\u002Fwinning response\n- **y_l**: Rejected\u002Flosing response\n\n### 3. SimPO Loss Objective\n```\nL_SimPO = -log(σ(reward_chosen - reward_rejected - γ))\n```\n\nThis is equivalent to binary cross-entropy loss where the target is always 1.\n\n## Implementation Pattern\n\n### Input Parameters\n```python\ndef simpo_loss(\n    self,\n    policy_chosen_logps: torch.FloatTensor,  # Shape: (batch_size,)\n    policy_rejected_logps: torch.FloatTensor  # Shape: (batch_size,)\n) -> Tuple[torch.FloatTensor, torch.FloatTensor, torch.FloatTensor]:\n```\n\n### Output\nReturns tuple of:\n- `losses`: Individual losses for each pair (batch_size,)\n- `chosen_rewards`: Rewards for chosen responses (batch_size,)\n- `rejected_rewards`: Rewards for rejected responses (batch_size,)\n\n## Implementation Steps\n\n### Step 1: Extract Configuration Values\nAccess `self.beta` and `gamma_beta_ratio` from the trainer's config to compute:\n- `gamma = self.beta * self.config.gamma_beta_ratio`\n\n### Step 2: Calculate Length-Normalized Rewards\nFor chosen responses:\n```python\nchosen_rewards = (self.beta \u002F chosen_length) * policy_chosen_logps\n```\n\nFor rejected responses:\n```python\nrejected_rewards = (self.beta \u002F rejected_length) * policy_rejected_logps\n```\n\n**Important**: The log probabilities passed are already **averaged per token** (not summed), so division by length has already been applied. Therefore, you multiply by `beta` only:\n```python\nchosen_rewards = self.beta * policy_chosen_logps\nrejected_rewards = self.beta * policy_rejected_logps\n```\n\n### Step 3: Compute Log Differences with Margin\n```python\nlog_odds = chosen_rewards - rejected_rewards - gamma\n```\n\n### Step 4: Apply Sigmoid and Compute Loss\nUsing binary cross-entropy with logits:\n```python\nlosses = -torch.nn.functional.logsigmoid(log_odds)\n```\n\nThis computes: `-log(σ(x))` which is the negative log-likelihood that the sigmoid evaluates to 1.\n\n## Configuration Defaults\n\nFrom `SimPOConfig`:\n- `beta`: 2.0 (reward scaling factor)\n- `gamma_beta_ratio`: 0.25 (ratio between margin and beta)\n- Default gamma: 0.5\n\n## Key Implementation Details\n\n1. **Log Probabilities Are Pre-Normalized**: The input `policy_chosen_logps` and `policy_rejected_logps` are already averaged log probabilities per token. No additional length normalization is needed in the loss function.\n\n2. **No Reference Model**: Unlike DPO, SimPO uses absolute log-likelihoods, not log-ratios. This eliminates the need for a reference model.\n\n3. **Vectorized Operations**: All operations should be vectorized across the batch dimension for efficiency.\n\n4. **Device Consistency**: Ensure all tensors remain on the same device (CPU\u002FGPU).\n\n## Common Pitfalls to Avoid\n\n1. **Double Length Normalization**: Don't divide by length again if log probs are already averaged\n2. **Dimension Mismatches**: Ensure rewards and log_odds maintain shape (batch_size,)\n3. **Missing Margin**: The `-gamma` term is critical; without it, the model ignores preference margins\n4. **Wrong Loss Function**: Use `-logsigmoid()` not `log_sigmoid()` for negative log-likelihood\n\n## Testing Pattern\n\nVerify implementation by:\n1. Checking output shapes match input batch sizes\n2. Confirming losses are positive values\n3. Validating that worse log-probability pairs produce higher losses\n4. Testing with unit tests using fixed input tensors\n","# SimPO Loss Implementation\n\n## Overview\n\nSimPO (Simple Preference Optimization) is a reference-free preference optimization method that trains language models using pairwise preference data. The loss function combines length-normalized rewards with a Bradley-Terry preference model.\n\n## Core Mathematical Formulas\n\n### 1. Length-Normalized Reward\n```\nr_SimPO(x, y) = (β\u002F|y|) * log π_θ(y|x)\n```\n\nWhere:\n- **β**: Scaling constant (typically 2.0)\n- **|y|**: Response length (number of tokens)\n- **log π_θ(y|x)**: Average log probability per token (not summed)\n\n### 2. Bradley-Terry Preference with Margin\n```\np(y_w ≻ y_l | x) = σ(r(x, y_w) - r(x, y_l) - γ)\n```\n\nWhere:\n- **γ**: Target reward margin (calculated as γ = β * gamma_beta_ratio)\n- **σ(·)**: Sigmoid function\n- **y_w**: Chosen\u002Fwinning response\n- **y_l**: Rejected\u002Flosing response\n\n### 3. SimPO Loss Objective\n```\nL_SimPO = -log(σ(reward_chosen - reward_rejected - γ))\n```\n\nThis is equivalent to binary cross-entropy loss where the target is always 1.\n\n## Implementation Pattern\n\n### Input Parameters\n```python\ndef simpo_loss(\n    self,\n    policy_chosen_logps: torch.FloatTensor,  # Shape: (batch_size,)\n    policy_rejected_logps: torch.FloatTensor  # Shape: (batch_size,)\n) -> Tuple[torch.FloatTensor, torch.FloatTensor, torch.FloatTensor]:\n```\n\n### Output\nReturns tuple of:\n- `losses`: Individual losses for each pair (batch_size,)\n- `chosen_rewards`: Rewards for chosen responses (batch_size,)\n- `rejected_rewards`: Rewards for rejected responses (batch_size,)\n\n## Implementation Steps\n\n### Step 1: Extract Configuration Values\nAccess `self.beta` and `gamma_beta_ratio` from the trainer's config to compute:\n- `gamma = self.beta * self.config.gamma_beta_ratio`\n\n### Step 2: Calculate Length-Normalized Rewards\nFor chosen responses:\n```python\nchosen_rewards = (self.beta \u002F chosen_length) * policy_chosen_logps\n```\n\nFor rejected responses:\n```python\nrejected_rewards = (self.beta \u002F rejected_length) * policy_rejected_logps\n```\n\n**Important**: The log probabilities passed are already **averaged per token** (not summed), so division by length has already been applied. Therefore, you multiply by `beta` only:\n```python\nchosen_rewards = self.beta * policy_chosen_logps\nrejected_rewards = self.beta * policy_rejected_logps\n```\n\n### Step 3: Compute Log Differences with Margin\n```python\nlog_odds = chosen_rewards - rejected_rewards - gamma\n```\n\n### Step 4: Apply Sigmoid and Compute Loss\nUsing binary cross-entropy with logits:\n```python\nlosses = -torch.nn.functional.logsigmoid(log_odds)\n```\n\nThis computes: `-log(σ(x))` which is the negative log-likelihood that the sigmoid evaluates to 1.\n\n## Configuration Defaults\n\nFrom `SimPOConfig`:\n- `beta`: 2.0 (reward scaling factor)\n- `gamma_beta_ratio`: 0.25 (ratio between margin and beta)\n- Default gamma: 0.5\n\n## Key Implementation Details\n\n1. **Log Probabilities Are Pre-Normalized**: The input `policy_chosen_logps` and `policy_rejected_logps` are already averaged log probabilities per token. No additional length normalization is needed in the loss function.\n\n2. **No Reference Model**: Unlike DPO, SimPO uses absolute log-likelihoods, not log-ratios. This eliminates the need for a reference model.\n\n3. **Vectorized Operations**: All operations should be vectorized across the batch dimension for efficiency.\n\n4. **Device Consistency**: Ensure all tensors remain on the same device (CPU\u002FGPU).\n\n## Common Pitfalls to Avoid\n\n1. **Double Length Normalization**: Don't divide by length again if log probs are already averaged\n2. **Dimension Mismatches**: Ensure rewards and log_odds maintain shape (batch_size,)\n3. **Missing Margin**: The `-gamma` term is critical; without it, the model ignores preference margins\n4. **Wrong Loss Function**: Use `-logsigmoid()` not `log_sigmoid()` for negative log-likelihood\n\n## Testing Pattern\n\nVerify implementation by:\n1. Checking output shapes match input batch sizes\n2. Confirming losses are positive values\n3. Validating that worse log-probability pairs produce higher losses\n4. Testing with unit tests using fixed input tensors\n",{"name":5,"description":7},"main","726e1f79b96b0cb3d729d0e55a1c847e0b5353ae","skills\u002Fb4-skill-creator-claude-haiku-4-5\u002Fnlp-paper-reproduction\u002Fsimpo-loss-implementation",8,"https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench","https:\u002F\u002Fcxcscmu.github.io\u002FSkillLearnBench","https:\u002F\u002Fgithub.com\u002Fcxcscmu","2026-10-02 22:55:36",{"source":8,"skillName":5,"commands":52,"prompt":62,"zipAvailable":63},[53,56,59],{"tool":54,"command":55},"npx","npx skills add cxcscmu\u002FSkillLearnBench --skill simpo-loss-implementation",{"tool":57,"command":58},"bunx","bunx skills add cxcscmu\u002FSkillLearnBench --skill simpo-loss-implementation",{"tool":60,"command":61},"pnpm","pnpm dlx skills add cxcscmu\u002FSkillLearnBench --skill simpo-loss-implementation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench\u002Fblob\u002Fmain\u002Fskills\u002Fb4-skill-creator-claude-haiku-4-5\u002Fnlp-paper-reproduction\u002Fsimpo-loss-implementation\u002FSKILL.md",true,[],[66,84,107,125,144,156],{"skillNo":67,"name":68,"nameZh":6,"description":69,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":70,"categories":71,"keywords":77,"publishedAt":38},"01M6KM5KH175P5ABP5JMX2SBDQ","object_counter","Count occurrences of an object in the image using computer vision algorithm.","https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench\u002Fblob\u002Fmain\u002Fskills\u002Fhuman_authored\u002Fvideo-object-counting\u002Fobject_counter\u002FSKILL.md",[72],{"categoryNo":73,"categoryName":74,"taxonomyType":75,"taxonomyTypeDesc":76},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",[78,79,80,81,82,83],"object counter","object","vision","computer","Computer Vision","image",{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":93,"publishedAt":38},"01M6KM51TMVXG79MMPCS6Y8G86","travel-planning-workflow","Invoke this skill when performing travel planning. It defines a structured workflow for generating itineraries. Following this workflow is essential to ensure the final plan satisfies user requirements.","https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench\u002Fblob\u002Fmain\u002Fskills\u002Fhuman_authored\u002Ftravel-planning\u002Ftravel-planning-workflow\u002FSKILL.md",[90],{"categoryNo":91,"categoryName":92,"taxonomyType":75,"taxonomyTypeDesc":76},"01JBSEEDCAT000000000000023","Skill与智能体开发",[94,95,96,97,98,99,100,101,102,23,103,104,105,22,106],"workflow","planning","planning workflow","travel","travel planning","travel planning workflow","structured","performing","plan","user","it","requirements","itineraries",{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":111,"categories":112,"keywords":113,"publishedAt":38},"01M6KM4DY820XHS9YTSK5QP5HD","bundled-dataset-search-tool","invoke this skill when you need to perform database search for travel planning. This skill provides some useful pre-packaged tools to look up accommodations, attractions, cities, driving distance, flights, and restaurants from the bundled dataset.","https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench\u002Fblob\u002Fmain\u002Fskills\u002Fhuman_authored\u002Ftravel-planning\u002Fbundled-dataset-search-tool\u002FSKILL.md",[],[114,115,116,117,118,119,120,97,121,122,98,123,23,124,95],"search","tool","dataset","bundled dataset search tool","restaurants","Tools","up","look","flights","you","database",{"skillNo":126,"name":127,"nameZh":6,"description":128,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":129,"categories":130,"keywords":131,"publishedAt":38},"01M6KM465BB9PQ0B548HQSVAWZ","glm-output","Output-processing guidance for GLM tasks. Especially useful after glm-basics and glm-calibration when you need verifier-matching metrics from output.nc and a final \u002Froot\u002Fmetrics.json.","https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench\u002Fblob\u002Fmain\u002Fskills\u002Fhuman_authored\u002Ftemperature-simulation\u002Fglm-output\u002FSKILL.md",[],[132,133,134,23,35,135,136,137,138,139,140,141,123,142,143],"glm output","output","glm","metrics","json","basics","guidance","matching","verifier","processing","root","calibration",{"skillNo":145,"name":146,"nameZh":6,"description":147,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":148,"categories":149,"keywords":151,"publishedAt":38},"01M6KM40KY3QWDW5W97S39R2BC","glm-calibration","Calibration guidance for GLM tasks. Often most effective after glm-basics has clarified the setup; glm-output is the companion skill for exact final metric computation.","https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench\u002Fblob\u002Fmain\u002Fskills\u002Fhuman_authored\u002Ftemperature-simulation\u002Fglm-calibration\u002FSKILL.md",[150],{"categoryNo":91,"categoryName":92,"taxonomyType":75,"taxonomyTypeDesc":76},[152,143,134,133,138,137,35,153,154,23,155],"glm calibration","setup","companion","metric",{"skillNo":157,"name":158,"nameZh":6,"description":159,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":160,"categories":161,"keywords":162,"publishedAt":38},"01M6KM3VHHVSGMGHJQB3PPT6YG","glm-basics","A good starting point for GLM calibration tasks. Use to inspect glm3.nml, confirm how GLM runs, and identify the relevant files before moving on to calibration and output evaluation.","https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench\u002Fblob\u002Fmain\u002Fskills\u002Fhuman_authored\u002Ftemperature-simulation\u002Fglm-basics\u002FSKILL.md",[],[163,134,137,164,152,143,23,133,165,166,167,32,168,169,35],"glm basics","inspect","relevant","files","good","how","Evaluation"]