[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BZT6N2N509YFF2F71PECYX":3,"skill-same:01M7BZT6N2N509YFF2F71PECYX":48},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":29,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M7BZT6N2N509YFF2F71PECYX","shift-evals",null,"Run and read Shift's evaluations — the SWE-bench slice, dogfood tickets, judge agreement, compaction scoring, and Terminal-Bench or DeepSWE through Harbor. Use when asked to measure a harness change, investigate a miss, or interpret an eval result.","thrashr888\u002Fshift","thrashr888","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F10213?v=4","01M7BZSZGHT71RQ58BNZX66ACN","Paul Thrasher","01M7BZSZH5MARAKQHD9SR9VXVJ",1,"MIT","https:\u002F\u002Fgithub.com\u002Fthrashr888\u002Fshift\u002Fblob\u002Fmain\u002F.shift-plugin\u002Fskills\u002Fshift-evals\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: shift-evals\ndescription: Run and read Shift's evaluations — the SWE-bench slice, dogfood tickets, judge agreement, compaction scoring, and Terminal-Bench or DeepSWE through Harbor. Use when asked to measure a harness change, investigate a miss, or interpret an eval result.\n---\n\n# Evaluations\n\n`docs\u002Fevals-rfc.md` is the source of truth: the slices, every recorded run and\nwhat each one showed. This skill is the operating instructions and the traps\nthat cost the most time; read the RFC for results and history rather than\ntrusting a summary here.\n\n## The driver\n\n```sh\npython3 scripts\u002Fevals.py --help\n```\n\n```sh\npython3 scripts\u002Fevals.py fetch                       # cache SWE-bench Verified\npython3 scripts\u002Fevals.py slice                       # the seeded 25-instance slice\npython3 scripts\u002Fevals.py run [--instances A,B]       # print-mode run per instance\npython3 scripts\u002Fevals.py grade RUN_ID                # official harness\npython3 scripts\u002Fevals.py dogfood [--tasks A,B]       # current-tree tickets, hidden local tests\npython3 scripts\u002Fevals.py session [NAME|--all]        # per-turn review\npython3 scripts\u002Fevals.py judge [--cases|--session N] # judge agreement\npython3 scripts\u002Fevals.py compaction [--replay]       # summary fact coverage\npython3 scripts\u002Fevals.py live-repair [--tasks A,B]   # live_eval vs edit-plus-reload\npython3 scripts\u002Fevals.py terminal-bench [--tasks ..] # through Harbor\npython3 scripts\u002Fevals.py deepswe [--tasks A,B]       # through Harbor\n```\n\nResults land in `evals\u002Fresults\u002FRUN_ID\u002F`: `predictions.jsonl` is what the\nharness grades, `results.jsonl` is one metrics record per instance. Both come\nfrom the receipt Shift writes with `--receipt`, never from the transcript.\n\n## Validate a slice before believing a miss\n\n```sh\npython3 scripts\u002Fevals.py terminal-bench --tasks NAME --oracle\n```\n\n`--oracle` applies the task's reference solution through Harbor. If the oracle\nrun does not pass, the task is broken in this environment and a Shift failure\non it says nothing about Shift. Do this before investigating any new miss.\n\n## Harbor gotchas\n\nThese are environmental and will waste a run each if forgotten. The RFC's\nTerminal-Bench and DeepSWE section has the full reasoning.\n\n- **Guile version.** Task images ship Debian 12's Guile 3.0.8 as often as not,\n  and Shift's `run` tool needs the `spawn` that 3.0.9 added. The adapter\n  installs a pinned conda-forge Guile through micromamba for this reason; if a\n  container's toolchain looks like it should work and does not, check which\n  Guile is actually on `PATH` there.\n- **The gost 15-second timeout.** DeepSWE's agent phase runs with no network,\n  and reaching a local Ollama goes through a gost sidecar that drops any\n  connection whose response headers take over 15 seconds — which a local\n  model's prefill routinely does. An extra compose file mounts\n  `evals\u002Fharbor\u002Fgost.yaml` over the sidecar's own config to fix it. Every model\n  call in the first DeepSWE run died at exactly 15.0 seconds.\n- **Uncommitted work scores zero.** DeepSWE grades `git diff BASE HEAD`, so the\n  adapter commits whatever changed after the turn. A change left in the working\n  tree is invisible to the verifier.\n- **Non-Python DeepSWE images** have shipped zero-byte toolchains in this\n  environment. Confirm with `--oracle` before reading a failure as Shift's.\n\n## Reading a result\n\nA benchmark turn leaves a session folder next to its receipt, so\n`scripts\u002Fevals.py session` reviews it like any other session. Prefer the\nreceipt's own numbers — rounds, tokens, cost, status — over anything narrated\nin the transcript; the receipt is projected from the ledger and the trace,\nand a model's account of what it did is not evidence.\n","# Evaluations\n\n`docs\u002Fevals-rfc.md` is the source of truth: the slices, every recorded run and\nwhat each one showed. This skill is the operating instructions and the traps\nthat cost the most time; read the RFC for results and history rather than\ntrusting a summary here.\n\n## The driver\n\n```sh\npython3 scripts\u002Fevals.py --help\n```\n\n```sh\npython3 scripts\u002Fevals.py fetch                       # cache SWE-bench Verified\npython3 scripts\u002Fevals.py slice                       # the seeded 25-instance slice\npython3 scripts\u002Fevals.py run [--instances A,B]       # print-mode run per instance\npython3 scripts\u002Fevals.py grade RUN_ID                # official harness\npython3 scripts\u002Fevals.py dogfood [--tasks A,B]       # current-tree tickets, hidden local tests\npython3 scripts\u002Fevals.py session [NAME|--all]        # per-turn review\npython3 scripts\u002Fevals.py judge [--cases|--session N] # judge agreement\npython3 scripts\u002Fevals.py compaction [--replay]       # summary fact coverage\npython3 scripts\u002Fevals.py live-repair [--tasks A,B]   # live_eval vs edit-plus-reload\npython3 scripts\u002Fevals.py terminal-bench [--tasks ..] # through Harbor\npython3 scripts\u002Fevals.py deepswe [--tasks A,B]       # through Harbor\n```\n\nResults land in `evals\u002Fresults\u002FRUN_ID\u002F`: `predictions.jsonl` is what the\nharness grades, `results.jsonl` is one metrics record per instance. Both come\nfrom the receipt Shift writes with `--receipt`, never from the transcript.\n\n## Validate a slice before believing a miss\n\n```sh\npython3 scripts\u002Fevals.py terminal-bench --tasks NAME --oracle\n```\n\n`--oracle` applies the task's reference solution through Harbor. If the oracle\nrun does not pass, the task is broken in this environment and a Shift failure\non it says nothing about Shift. Do this before investigating any new miss.\n\n## Harbor gotchas\n\nThese are environmental and will waste a run each if forgotten. The RFC's\nTerminal-Bench and DeepSWE section has the full reasoning.\n\n- **Guile version.** Task images ship Debian 12's Guile 3.0.8 as often as not,\n  and Shift's `run` tool needs the `spawn` that 3.0.9 added. The adapter\n  installs a pinned conda-forge Guile through micromamba for this reason; if a\n  container's toolchain looks like it should work and does not, check which\n  Guile is actually on `PATH` there.\n- **The gost 15-second timeout.** DeepSWE's agent phase runs with no network,\n  and reaching a local Ollama goes through a gost sidecar that drops any\n  connection whose response headers take over 15 seconds — which a local\n  model's prefill routinely does. An extra compose file mounts\n  `evals\u002Fharbor\u002Fgost.yaml` over the sidecar's own config to fix it. Every model\n  call in the first DeepSWE run died at exactly 15.0 seconds.\n- **Uncommitted work scores zero.** DeepSWE grades `git diff BASE HEAD`, so the\n  adapter commits whatever changed after the turn. A change left in the working\n  tree is invisible to the verifier.\n- **Non-Python DeepSWE images** have shipped zero-byte toolchains in this\n  environment. Confirm with `--oracle` before reading a failure as Shift's.\n\n## Reading a result\n\nA benchmark turn leaves a session folder next to its receipt, so\n`scripts\u002Fevals.py session` reviews it like any other session. Prefer the\nreceipt's own numbers — rounds, tokens, cost, status — over anything narrated\nin the transcript; the receipt is projected from the ledger and the trace,\nand a model's account of what it did is not evidence.\n",{"name":5,"description":7},"main","9c095495f5562627526cdf6f0673419a760c1d3e",".shift-plugin\u002Fskills\u002Fshift-evals",0,"https:\u002F\u002Fgithub.com\u002Fthrashr888\u002Fshift","https:\u002F\u002Fthrashr888.github.io\u002Fshift\u002F","https:\u002F\u002Fgithub.com\u002Fthrashr888","https:\u002F\u002Fpaul.thrasher.dev\u002F","Building many websites.","2026-09-25 11:00:23",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add thrashr888\u002Fshift --skill shift-evals",{"tool":40,"command":41},"bunx","bunx skills add thrashr888\u002Fshift --skill shift-evals",{"tool":43,"command":44},"pnpm","pnpm dlx skills add thrashr888\u002Fshift --skill shift-evals","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fthrashr888\u002Fshift\u002Fblob\u002Fmain\u002F.shift-plugin\u002Fskills\u002Fshift-evals\u002FSKILL.md",true,[],[49,55,62,69,76,82],{"skillNo":50,"name":51,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":52,"categories":53,"keywords":54,"publishedAt":19},"01M7BZV5FJS0T12SJB2X3V8K2B","tauri-browser","https:\u002F\u002Fgithub.com\u002Fthrashr888\u002Fshift\u002Fblob\u002Fmain\u002Fplugins\u002Ftauri-browser\u002Fskills\u002Ftauri-browser\u002FSKILL.md",[],[],{"skillNo":56,"name":57,"nameZh":6,"description":58,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":59,"categories":60,"keywords":61,"publishedAt":19},"01M7BZV2R1DK48G0BVA9HPZFDT","ripwire","Answer \"who calls this\", \"is it safe to change\", \"which tests cover it\" and \"where do I start in this repo\" from a call graph instead of grepping and reading files. Use before editing a symbol, when a test fails with a trace, and when landing in an unfamiliar codebase; not for Scheme, which it cannot parse.","https:\u002F\u002Fgithub.com\u002Fthrashr888\u002Fshift\u002Fblob\u002Fmain\u002Fplugins\u002Fripwire\u002Fskills\u002Fripwire\u002FSKILL.md",[],[],{"skillNo":63,"name":64,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":68,"publishedAt":19},"01M7BZTZ8JTQBWYR1Z9XMPWWAJ","onepassword","Work with the user's 1Password through the op CLI for non-secret operations (which vaults and items exist, whether a session is signed in) and know how Shift gets secrets from it. Use when a server or command needs a credential, or the user asks about their vaults.","https:\u002F\u002Fgithub.com\u002Fthrashr888\u002Fshift\u002Fblob\u002Fmain\u002Fplugins\u002Fonepassword\u002Fskills\u002Fonepassword\u002FSKILL.md",[],[],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":19},"01M7BZTWX2AS9CHHR300PNQ389","kanban","Keep and change this project's board — read the columns, move a card, add or retire one. Use when asked what is in progress, what to pick up next, or to move a card between columns.","https:\u002F\u002Fgithub.com\u002Fthrashr888\u002Fshift\u002Fblob\u002Fmain\u002Fplugins\u002Fkanban\u002Fskills\u002Fkanban\u002FSKILL.md",[],[],{"skillNo":77,"name":78,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":81,"publishedAt":19},"01M7BZTW46MQVNWJKNN4QAZ99C","cortex","https:\u002F\u002Fgithub.com\u002Fthrashr888\u002Fshift\u002Fblob\u002Fmain\u002Fplugins\u002Fcortex\u002Fskills\u002Fcortex\u002FSKILL.md",[],[],{"skillNo":83,"name":84,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":19},"01M7BZTRZ7Y48S3SYY5PDNBFZ6","cider","Read and act on macOS Apple apps (Calendar, Reminders, Contacts, Notes, Mail, Music, Safari, Home and 30 more) with the cider CLI, which prints JSON. Use when the user asks about their calendar, reminders, contacts, notes, mail or Mac state, or wants one of those changed.","https:\u002F\u002Fgithub.com\u002Fthrashr888\u002Fshift\u002Fblob\u002Fmain\u002Fplugins\u002Fcider\u002Fskills\u002Fcider\u002FSKILL.md",[],[]]