[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6KDWGQANKAS1Q48PBX1BYVM":3,"skill-same:01M6KDWGQANKAS1Q48PBX1BYVM":61},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":34,"linkOnly":35,"markdownContent":36,"markdownBody":37,"frontMatter":38,"refName":39,"commitSha":40,"skillDirectory":41,"forksCount":42,"repositoryHtmlUrl":43,"repositoryHomepageUrl":44,"ownerHtmlUrl":45,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":46,"install":47,"resources":60},"01M6KDWGQANKAS1Q48PBX1BYVM","pytorch-preference-optimization",null,"Guide for implementing preference optimization methods (DPO, SimPO, IPO) in PyTorch. Use when implementing loss functions for RLHF-style training.","cxcscmu\u002FSkillLearnBench","cxcscmu","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F142838272?v=4","01M6KCQDFQWPDSN0MKZY96MSQP","01M6KCQDM7G70TKJ28NTSQC0RZ",93,"MIT","https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench\u002Fblob\u002Fmain\u002Fskills\u002Fb1-one-shot-claude-sonnet-4-6\u002Fnlp-paper-reproduction\u002Fpytorch-preference-optimization\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33],"pytorch","Optimization","Preference Optimization","pytorch preference optimization","training","DPO","RLHF","guide","use","functions","methods","style","IPO","simpo","implementing","for","2026-11-10 10:12:30",false,"---\nname: pytorch-preference-optimization\ndescription: Guide for implementing preference optimization methods (DPO, SimPO, IPO) in PyTorch. Use when implementing loss functions for RLHF-style training.\n---\n\n# PyTorch Preference Optimization\n\n## Common Pattern\nAll preference optimization methods take:\n- `policy_chosen_logps`: log probs for preferred responses\n- `policy_rejected_logps`: log probs for rejected responses\n\n## DPO Loss\n```python\n# Requires reference model log probs\nlogits = (policy_chosen_logps - ref_chosen_logps) - (policy_rejected_logps - ref_rejected_logps)\nlosses = -F.logsigmoid(beta * logits)\n```\n\n## SimPO Loss (Reference-Free)\n```python\n# No reference model needed - uses length-normalized log probs\npi_logratios = policy_chosen_logps - policy_rejected_logps  # already avg'd per token\nlogits = pi_logratios - gamma_beta_ratio  # gamma_beta_ratio = gamma\u002Fbeta\nlosses = -F.logsigmoid(beta * logits)\n```\n\n## Tips\n- `F.logsigmoid(x)` is numerically more stable than `torch.log(torch.sigmoid(x))`\n- For label smoothing: `loss = -logsigmoid(logits)*(1-ls) - logsigmoid(-logits)*ls`\n- Hinge loss: `torch.relu(1 - beta * logits)`\n- Always `.detach()` reward tensors to prevent gradient flow through them\n","# PyTorch Preference Optimization\n\n## Common Pattern\nAll preference optimization methods take:\n- `policy_chosen_logps`: log probs for preferred responses\n- `policy_rejected_logps`: log probs for rejected responses\n\n## DPO Loss\n```python\n# Requires reference model log probs\nlogits = (policy_chosen_logps - ref_chosen_logps) - (policy_rejected_logps - ref_rejected_logps)\nlosses = -F.logsigmoid(beta * logits)\n```\n\n## SimPO Loss (Reference-Free)\n```python\n# No reference model needed - uses length-normalized log probs\npi_logratios = policy_chosen_logps - policy_rejected_logps  # already avg'd per token\nlogits = pi_logratios - gamma_beta_ratio  # gamma_beta_ratio = gamma\u002Fbeta\nlosses = -F.logsigmoid(beta * logits)\n```\n\n## Tips\n- `F.logsigmoid(x)` is numerically more stable than `torch.log(torch.sigmoid(x))`\n- For label smoothing: `loss = -logsigmoid(logits)*(1-ls) - logsigmoid(-logits)*ls`\n- Hinge loss: `torch.relu(1 - beta * logits)`\n- Always `.detach()` reward tensors to prevent gradient flow through them\n",{"name":5,"description":7},"main","726e1f79b96b0cb3d729d0e55a1c847e0b5353ae","skills\u002Fb1-one-shot-claude-sonnet-4-6\u002Fnlp-paper-reproduction\u002Fpytorch-preference-optimization",8,"https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench","https:\u002F\u002Fcxcscmu.github.io\u002FSkillLearnBench","https:\u002F\u002Fgithub.com\u002Fcxcscmu","2026-10-02 22:55:36",{"source":8,"skillName":5,"commands":48,"prompt":58,"zipAvailable":59},[49,52,55],{"tool":50,"command":51},"npx","npx skills add cxcscmu\u002FSkillLearnBench --skill pytorch-preference-optimization",{"tool":53,"command":54},"bunx","bunx skills add cxcscmu\u002FSkillLearnBench --skill pytorch-preference-optimization",{"tool":56,"command":57},"pnpm","pnpm dlx skills add cxcscmu\u002FSkillLearnBench --skill pytorch-preference-optimization","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench\u002Fblob\u002Fmain\u002Fskills\u002Fb1-one-shot-claude-sonnet-4-6\u002Fnlp-paper-reproduction\u002Fpytorch-preference-optimization\u002FSKILL.md",true,[],[62,80,104,122,142,154],{"skillNo":63,"name":64,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":73,"publishedAt":34},"01M6KM5KH175P5ABP5JMX2SBDQ","object_counter","Count occurrences of an object in the image using computer vision algorithm.","https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench\u002Fblob\u002Fmain\u002Fskills\u002Fhuman_authored\u002Fvideo-object-counting\u002Fobject_counter\u002FSKILL.md",[68],{"categoryNo":69,"categoryName":70,"taxonomyType":71,"taxonomyTypeDesc":72},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",[74,75,76,77,78,79],"object counter","object","vision","computer","Computer Vision","image",{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":89,"publishedAt":34},"01M6KM51TMVXG79MMPCS6Y8G86","travel-planning-workflow","Invoke this skill when performing travel planning. It defines a structured workflow for generating itineraries. Following this workflow is essential to ensure the final plan satisfies user requirements.","https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench\u002Fblob\u002Fmain\u002Fskills\u002Fhuman_authored\u002Ftravel-planning\u002Ftravel-planning-workflow\u002FSKILL.md",[86],{"categoryNo":87,"categoryName":88,"taxonomyType":71,"taxonomyTypeDesc":72},"01JBSEEDCAT000000000000023","Skill与智能体开发",[90,91,92,93,94,95,96,97,98,33,99,100,101,102,103],"workflow","planning","planning workflow","travel","travel planning","travel planning workflow","structured","performing","plan","user","it","requirements","essential","itineraries",{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":34},"01M6KM4DY820XHS9YTSK5QP5HD","bundled-dataset-search-tool","invoke this skill when you need to perform database search for travel planning. This skill provides some useful pre-packaged tools to look up accommodations, attractions, cities, driving distance, flights, and restaurants from the bundled dataset.","https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench\u002Fblob\u002Fmain\u002Fskills\u002Fhuman_authored\u002Ftravel-planning\u002Fbundled-dataset-search-tool\u002FSKILL.md",[],[111,112,113,114,115,116,117,93,118,119,94,120,33,121,91],"search","tool","dataset","bundled dataset search tool","restaurants","Tools","up","look","flights","you","database",{"skillNo":123,"name":124,"nameZh":6,"description":125,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":126,"categories":127,"keywords":128,"publishedAt":34},"01M6KM465BB9PQ0B548HQSVAWZ","glm-output","Output-processing guidance for GLM tasks. Especially useful after glm-basics and glm-calibration when you need verifier-matching metrics from output.nc and a final \u002Froot\u002Fmetrics.json.","https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench\u002Fblob\u002Fmain\u002Fskills\u002Fhuman_authored\u002Ftemperature-simulation\u002Fglm-output\u002FSKILL.md",[],[129,130,131,33,132,133,134,135,136,137,138,139,120,140,141],"glm output","output","glm","tasks","metrics","json","basics","guidance","matching","verifier","processing","root","calibration",{"skillNo":143,"name":144,"nameZh":6,"description":145,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":146,"categories":147,"keywords":149,"publishedAt":34},"01M6KM40KY3QWDW5W97S39R2BC","glm-calibration","Calibration guidance for GLM tasks. Often most effective after glm-basics has clarified the setup; glm-output is the companion skill for exact final metric computation.","https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench\u002Fblob\u002Fmain\u002Fskills\u002Fhuman_authored\u002Ftemperature-simulation\u002Fglm-calibration\u002FSKILL.md",[148],{"categoryNo":87,"categoryName":88,"taxonomyType":71,"taxonomyTypeDesc":72},[150,141,131,130,136,135,132,151,152,33,153],"glm calibration","setup","companion","metric",{"skillNo":155,"name":156,"nameZh":6,"description":157,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":158,"categories":159,"keywords":160,"publishedAt":34},"01M6KM3VHHVSGMGHJQB3PPT6YG","glm-basics","A good starting point for GLM calibration tasks. Use to inspect glm3.nml, confirm how GLM runs, and identify the relevant files before moving on to calibration and output evaluation.","https:\u002F\u002Fgithub.com\u002Fcxcscmu\u002FSkillLearnBench\u002Fblob\u002Fmain\u002Fskills\u002Fhuman_authored\u002Ftemperature-simulation\u002Fglm-basics\u002FSKILL.md",[],[161,131,135,162,150,141,33,130,163,164,165,26,166,167,132],"glm basics","inspect","relevant","files","good","how","Evaluation"]