[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6C38F02AX26DZHMC39W0SQK":3,"skill-same:01M6C38F02AX26DZHMC39W0SQK":103},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":22,"publishedAt":53,"linkOnly":54,"markdownContent":55,"markdownBody":56,"frontMatter":57,"refName":76,"commitSha":77,"skillDirectory":78,"forksCount":79,"repositoryHtmlUrl":80,"repositoryHomepageUrl":6,"ownerHtmlUrl":81,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":82,"install":83,"resources":96},"01M6C38F02AX26DZHMC39W0SQK","eval-vendor-tooling",null,"Evaluate a third-party AI\u002FML repo, vendor claim, or article headline — clone, run inference, contract-verify the README vs source vs shipped checkpoint, and publish a self-contained reproducible eval to jleechanorg\u002Fevals. Trigger when a user message pairs a URL\u002Frepo reference with an action verb like 'evaluate', 'replay', 'does X actually work', or 'publish findings to evals'.","jleechanorg\u002Fjleechan-skills","jleechanorg","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F221404966?v=4","01M6C1VJ99XG950X1DWPEBCBGX","01M6C1VJDK5YVDNT3362RT5WSB",6,"MIT","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002Fhermes\u002Fskills\u002Fdevops\u002Feval-vendor-tooling\u002FSKILL.md",[17],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000037","律师与法务","PROFESSION","适用职业",[23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52],"tooling","eval vendor tooling","vendor","eval","repo","when","headless","URL","article","self","message","claim","attach","default","evidence","README","github","work","model","checkpoint","verify","Reproducible","publish","publish findings","verification","browser","pr","ml","workflow","linux","2026-11-10 10:12:30",false,"---\nname: eval-vendor-tooling\ndescription: \"Evaluate a third-party AI\u002FML repo, vendor claim, or article headline — clone, run inference, contract-verify the README vs source vs shipped checkpoint, and publish a self-contained reproducible eval to jleechanorg\u002Fevals. Trigger when a user message pairs a URL\u002Frepo reference with an action verb like 'evaluate', 'replay', 'does X actually work', or 'publish findings to evals'.\"\nversion: 1.0.0\nauthor: Hermes Agent\nlicense: MIT\nplatforms: [linux, macos]\nmetadata:\n  hermes:\n    tags: [eval, vendor, third-party, reproducibility, claim-verification, ml-model, public-publish]\n    related_skills: [browser-headless-default, github-repo-management, github-pr-workflow, evidence-attach-not-path-cite, outbout-secret-publication-gate]\n---\n\n# Evaluate third-party vendor tooling (class-level)\n\nWhen the user gives you a URL, repo link, or vendor claim (\"X cuts cost by 60%\") and asks you to evaluate, this is the workflow. The session-proven lesson: **just do the work** — clone, fetch, infer, capture raw outputs. Don't ask permission menus before the first tool call. After a tool call fails, don't fall back to asking the user — pivot to the next reasonable tool.\n\n## Phase 0 — Pick the right first tool. Don't ask.\n\nWhen the user's first message contains a URL or repo reference AND an action verb (\"clone\", \"evaluate\", \"replay\", \"test\"), your first turn must contain a real tool call. Pick the cheapest correct tool:\n\n| Source type | First tool |\n|---|---|\n| GitHub repo (`github.com\u002F\u003Cowner>\u002F\u003Crepo>`) | `terminal: git clone \u003Curl>` (into a clean workspace like `~\u002Fprojects\u002F\u003Cslug>\u002F`) |\n| Plain-text endpoint (`.md`, `.yaml`, `raw.githubusercontent.com`, docs JSON) | `web_extract` or `terminal: curl` |\n| Paywalled \u002F JS-rendered (Medium, vendor blog, dynamic docs) | `browser_navigate` (headless) |\n| Any other URL | `browser_navigate` headless (per `browser-headless-default`) |\n\nDo NOT post a clarifying menu first. Do NOT ask \"paste vs I try harder\". Just pick and go.\n\nAfter a tool fails:\n- `web_extract` returns \"ddgs is search-only\" → pivot to `browser_navigate` headless\n- `curl` to a known bot-wall (Medium, Cloudflare-protected) returns 5-10 KB of HTML → pivot to `browser_navigate`\n- `browser_navigate` gets the \"Just a moment…\" Cloudflare interstitial → say so, don't fake content from there\n- `gh repo view` returns 404 → `gh repo create ...` if user wants to publish; otherwise ask once\n\nOnly if the **pivot path itself fails** should you surface a blocker to the user.\n\n## Phase 1 — Capture raw inputs as you go\n\nDon't re-fetch the article, README, or paper more than once. Save everything that you'll need later:\n\n- The HTML\u002FMD response → `\u002Ftmp\u002F\u003Cslug>.html` for later grep\n- Downloaded source files (`*.py`, `*.yaml`, `*.json`) → keep them in their cloned locations\n- Mine `session_search` for related past work in the user's projects if relevant — the user's real prompts are usually the best eval inputs\n\n## Phase 2 — Read source, not just README\n\nREADME is a marketing claim. Source is the contract. Always read:\n\n- The model definition file (architecture, head shape, layer config) — for ML repos this is the inference entrypoint\n- The download-model.sh or equivalent — confirms what was actually shipped\n- The model-loading call — `strict=True` (will raise on mismatch) vs `strict=False` (will silently ignore)\n- The version-pinning files (`requirements.txt`, `pyproject.toml`, `Dockerfile`)\n\nFor ML repos specifically, run a contract verification step before any inference:\n\n```python\nimport torch\nsd = torch.load(MODEL_PATH, map_location=\"cpu\", weights_only=False)\nfor k, v in sd.items():\n    if \"classifier\" in k or \"head\" in k or \"output\" in k:\n        print(f\"{k}: {tuple(v.shape)}\")\n# If README says Linear(768-4) but shape is [6, 768], the repo lies.\n```\n\nAlso check class counts: `NUM_LABELS` constants in code, label dictionaries, class head width in `state_dict`, sample labels in tests.\n\n## Phase 3 — Run inference in-process, not via HTTP\n\nIf the eval involves running a model, **don't spin up the vendor's full FastAPI \u002F agent \u002F router stack first**. Transport is not the question. Model behavior is.\n\nLoad the model in-process via `torch.load` + a re-implemented architecture (or a faithful copy of the vendor's `nn.Module` classes), and run inference directly. Capture per-input:\n\n- The prompt as sent\n- Tokens used (and whether it was truncated)\n- Per-class probabilities (not just argmax — the distribution tells you what the model is actually uncertain about)\n- Latency per inference\n\nFor latency: GPU latencies vendor cites are usually `g4dn.xlarge` at batch=1; CPU is ~10× slower. Don't trust vendor latency claims without reading the test setup.\n\n## Phase 4 — Catch the contract mismatch (the most common failure)\n\nThe single most common failure pattern in third-party ML repos:\n\n> README claims N classes. Code says N classes. **Checkpoint has M ≠ N classes.**\n\nIf `model.load_state_dict(sd)` raises `size mismatch … copying a param with shape torch.Size([M, …]) from checkpoint, the shape in current model is torch.Size([N, …])`:\n\n- The repo does not run end-to-end out of the box. Document this as a **Failure** in the report.\n- Don't try to fix it silently. Either:\n  1. Patch the code to `NUM_LABELS=M` (and document), or\n  2. Run with `load_state_dict(strict=False)` and a matching head shape, but flag the divergence in the report.\n- Either way, the **published interpretation of the model's output is unverified** when there's a class-count mismatch. Index 0..M-1 from the checkpoint is the only ground truth.\n\nOther common contract violations to look for:\n- Head shape ↔ labels dict mismatch\n- Tokenizer vocabulary size ↔ embedding matrix rows mismatch\n- Max-seq-len in code ↔ positional embeddings\n- \"`strict=True`\" load that will raise on the user's machine\n- Pre-claimed accuracy metrics without a test set in the repo\n\n## Phase 5 — Capture outputs in a real shell, not a sandbox\n\nThe agent's `execute_code` runs in a sandboxed Python process that **does not persist files to `\u002Ftmp` across calls reliably**. To keep raw outputs:\n\n```bash\n# Write the inference script to a real .sh file\nwrite_file(path=\"\u002Ftmp\u002Frun_\u003Cslug>.sh\", content=\"\u003C\u003CSCRIPT>>...\u003C\u003CPYEOF>>...PYEOF\")\n\n# Run it via real bash so \u002Ftmp output survives\nbash \u002Ftmp\u002Frun_\u003Cslug>.sh\n```\n\nWhy: `execute_code` outputs vanish after the call returns. Anything you want to ship in the eval (raw logits, per-prompt timings, prompts themselves) must be written via a real shell script and survive into the next tool call.\n\n## Phase 6 — Publish to `jleechanorg\u002Fevals`\n\nSelf-contained eval repo structure (matches the canonical layout landed in this repo's own session):\n\n```\njleechanorg\u002Fevals\u002F\n├── README.md                      # top-level index + conventions\n├── LICENSE                        # MIT, with note that vendored code keeps its own license\n├── .gitignore                     # never commit model weights or upstream clones\n└── docs\u002F\u003Ceval-slug>\u002F\n    ├── README.md                  # folder-level entry, TL;DR, link map\n    ├── report.md                  # full narrative (what we tested, what we found, verdict)\n    ├── repro.sh                   # one-command reproduction (one real shell script)\n    └── raw\u002F\n        ├── prompts.json\n        └── results.json\n```\n\nPer-eval folder naming: `docs\u002F\u003Ceval-type>-\u003Csubject>\u002F` where `\u003Ceval-type>` is one of:\n- `cli-eval-\u003Ctool-name>` for CLI tools\n- `router-eval-\u003Cvendor-repo>` for LLM routers\n- `model-eval-\u003Cmodel-name>` for raw model evaluations\n- `tool-eval-\u003Ctool-name>` for SaaS \u002F hosted tools\n\nCommit message convention: `[agento] feat: \u003Ceval-name> — \u003Cone-line summary>`. The `[agento]` prefix matches the convention from worldarchitect\u002Fagent-orchestrator work.\n\n**Before `git push`**, per `pr-ci-fix-autopush` and `push-pr-donot-stop-halfway`:\n- `git rev-parse origin\u002Fmain` must equal `HEAD` after push\n- `gh repo view jleechanorg\u002Fevals` shows the new description and default branch\n- The README renders correctly on github.com (visit it once via `web_extract` against the raw URL if you want to verify)\n- No model weights, venvs, or `upstream-clone\u002F` directories in the diff (`.gitignore` MUST cover them)\n\n## Pitfalls\n\n### P1. Don't ask permission menus when the user gave a directive\n\nBad first turn after curl fails:\n> \"Do you want me to (A) try harder on the article, (B) wait for you to paste, or (C) try a different approach?\"\n\nGood first turn:\n> `browser_navigate(url)` (the call itself, with a 1-line summary)\n\nThe `task-ack-and-execute` SOUL.md COMMIT already requires the first turn to contain a tool call. This pitfall reinforces: **after a tool fails, don't pivot to asking — pivot to the next tool.**\n\n### P2. README ≠ contract\n\nAlways run a contract verification step (Phase 2) before trusting any headline numbers. Common violations: claimed parameter count is loose, claimed accuracy has no test set, claimed latency is GPU-only at batch=1.\n\n### P3. Sandbox outputs don't survive\n\n`execute_code` outputs vanish after the call. Use a real `bash` subprocess via `write_file` → `terminal bash` if you need raw outputs to persist.\n\n### P4. Class-count mismatches are the #1 contract violation in ML repos\n\nAlways print the checkpoint head shape BEFORE running inference. Save it to the eval's `raw\u002Fresults.json` so future readers can verify your setup.\n\n### P5. Latency claims are scope-dependent\n\n\"10ms router\" might be GPU-only, batch=1, on a specific instance type. Don't trust without reading the test setup. Always re-measure on the eval's own machine.\n\n### P6. Paywalled content is a real obstacle\n\nMedium and similar sites have aggressive bot-walls. If `browser_navigate` headless + `BROWSERBASE_ADVANCED_STEALTH` can't get past it, the article itself is unreadable. **Say so** in the report rather than fabricating a summary.\n\n### P7. Don't leak credentials in eval artifacts\n\nPer the `outbound-secret-publication-gate` SOUL.md COMMIT, before pushing anything to `jleechanorg\u002Fevals`, scan all `.md` and `.sh` files for `ghp_`, `xoxp-`, `sk-`, `AKIA` patterns. Never commit tokens, no matter how \"obvious\" they seem in a private context.\n\n## Verification\n\nAfter publishing:\n\n1. `git -C ~\u002Fprojects\u002Fevals log --oneline -1` — commit exists\n2. `git -C ~\u002Fprojects\u002Fevals rev-parse origin\u002Fmain` equals `HEAD` (no local commits behind)\n3. `gh repo view jleechanorg\u002Fevals --json description,defaultBranchRef` — description + main branch set\n4. Web-fetch the raw README URL on github.com to verify markdown renders\n5. `bash docs\u002F\u003Cslug>\u002Frepro.sh` from a clean checkout — runs end-to-end and reproduces `raw\u002Fresults.json`\n\n## Related references\n\n- `references\u002Frouter-eval-deycoding-2026-07-22.md` — first worked example, in this repo's own session: a ~140M encoder BFSI router that doesn't load out of the box, can't do whole-convo classification, and has a contract mismatch.\n\n## Related skills\n\n- `browser-headless-default` — for the headless browser pivot path\n- `github-repo-management` — for `gh repo create`, `git push`, etc.\n- `evidence-attach-not-path-cite` — for embedding screenshots \u002F evidence in reports\n- `outbound-secret-publication-gate` (SOUL.md COMMIT) — for the pre-publish credential scan\n- `pr-clean-branch-from-main-no-history-bloat` (SOUL.md COMMIT) — if the eval repo itself needs branch discipline","# Evaluate third-party vendor tooling (class-level)\n\nWhen the user gives you a URL, repo link, or vendor claim (\"X cuts cost by 60%\") and asks you to evaluate, this is the workflow. The session-proven lesson: **just do the work** — clone, fetch, infer, capture raw outputs. Don't ask permission menus before the first tool call. After a tool call fails, don't fall back to asking the user — pivot to the next reasonable tool.\n\n## Phase 0 — Pick the right first tool. Don't ask.\n\nWhen the user's first message contains a URL or repo reference AND an action verb (\"clone\", \"evaluate\", \"replay\", \"test\"), your first turn must contain a real tool call. Pick the cheapest correct tool:\n\n| Source type | First tool |\n|---|---|\n| GitHub repo (`github.com\u002F\u003Cowner>\u002F\u003Crepo>`) | `terminal: git clone \u003Curl>` (into a clean workspace like `~\u002Fprojects\u002F\u003Cslug>\u002F`) |\n| Plain-text endpoint (`.md`, `.yaml`, `raw.githubusercontent.com`, docs JSON) | `web_extract` or `terminal: curl` |\n| Paywalled \u002F JS-rendered (Medium, vendor blog, dynamic docs) | `browser_navigate` (headless) |\n| Any other URL | `browser_navigate` headless (per `browser-headless-default`) |\n\nDo NOT post a clarifying menu first. Do NOT ask \"paste vs I try harder\". Just pick and go.\n\nAfter a tool fails:\n- `web_extract` returns \"ddgs is search-only\" → pivot to `browser_navigate` headless\n- `curl` to a known bot-wall (Medium, Cloudflare-protected) returns 5-10 KB of HTML → pivot to `browser_navigate`\n- `browser_navigate` gets the \"Just a moment…\" Cloudflare interstitial → say so, don't fake content from there\n- `gh repo view` returns 404 → `gh repo create ...` if user wants to publish; otherwise ask once\n\nOnly if the **pivot path itself fails** should you surface a blocker to the user.\n\n## Phase 1 — Capture raw inputs as you go\n\nDon't re-fetch the article, README, or paper more than once. Save everything that you'll need later:\n\n- The HTML\u002FMD response → `\u002Ftmp\u002F\u003Cslug>.html` for later grep\n- Downloaded source files (`*.py`, `*.yaml`, `*.json`) → keep them in their cloned locations\n- Mine `session_search` for related past work in the user's projects if relevant — the user's real prompts are usually the best eval inputs\n\n## Phase 2 — Read source, not just README\n\nREADME is a marketing claim. Source is the contract. Always read:\n\n- The model definition file (architecture, head shape, layer config) — for ML repos this is the inference entrypoint\n- The download-model.sh or equivalent — confirms what was actually shipped\n- The model-loading call — `strict=True` (will raise on mismatch) vs `strict=False` (will silently ignore)\n- The version-pinning files (`requirements.txt`, `pyproject.toml`, `Dockerfile`)\n\nFor ML repos specifically, run a contract verification step before any inference:\n\n```python\nimport torch\nsd = torch.load(MODEL_PATH, map_location=\"cpu\", weights_only=False)\nfor k, v in sd.items():\n    if \"classifier\" in k or \"head\" in k or \"output\" in k:\n        print(f\"{k}: {tuple(v.shape)}\")\n# If README says Linear(768-4) but shape is [6, 768], the repo lies.\n```\n\nAlso check class counts: `NUM_LABELS` constants in code, label dictionaries, class head width in `state_dict`, sample labels in tests.\n\n## Phase 3 — Run inference in-process, not via HTTP\n\nIf the eval involves running a model, **don't spin up the vendor's full FastAPI \u002F agent \u002F router stack first**. Transport is not the question. Model behavior is.\n\nLoad the model in-process via `torch.load` + a re-implemented architecture (or a faithful copy of the vendor's `nn.Module` classes), and run inference directly. Capture per-input:\n\n- The prompt as sent\n- Tokens used (and whether it was truncated)\n- Per-class probabilities (not just argmax — the distribution tells you what the model is actually uncertain about)\n- Latency per inference\n\nFor latency: GPU latencies vendor cites are usually `g4dn.xlarge` at batch=1; CPU is ~10× slower. Don't trust vendor latency claims without reading the test setup.\n\n## Phase 4 — Catch the contract mismatch (the most common failure)\n\nThe single most common failure pattern in third-party ML repos:\n\n> README claims N classes. Code says N classes. **Checkpoint has M ≠ N classes.**\n\nIf `model.load_state_dict(sd)` raises `size mismatch … copying a param with shape torch.Size([M, …]) from checkpoint, the shape in current model is torch.Size([N, …])`:\n\n- The repo does not run end-to-end out of the box. Document this as a **Failure** in the report.\n- Don't try to fix it silently. Either:\n  1. Patch the code to `NUM_LABELS=M` (and document), or\n  2. Run with `load_state_dict(strict=False)` and a matching head shape, but flag the divergence in the report.\n- Either way, the **published interpretation of the model's output is unverified** when there's a class-count mismatch. Index 0..M-1 from the checkpoint is the only ground truth.\n\nOther common contract violations to look for:\n- Head shape ↔ labels dict mismatch\n- Tokenizer vocabulary size ↔ embedding matrix rows mismatch\n- Max-seq-len in code ↔ positional embeddings\n- \"`strict=True`\" load that will raise on the user's machine\n- Pre-claimed accuracy metrics without a test set in the repo\n\n## Phase 5 — Capture outputs in a real shell, not a sandbox\n\nThe agent's `execute_code` runs in a sandboxed Python process that **does not persist files to `\u002Ftmp` across calls reliably**. To keep raw outputs:\n\n```bash\n# Write the inference script to a real .sh file\nwrite_file(path=\"\u002Ftmp\u002Frun_\u003Cslug>.sh\", content=\"\u003C\u003CSCRIPT>>...\u003C\u003CPYEOF>>...PYEOF\")\n\n# Run it via real bash so \u002Ftmp output survives\nbash \u002Ftmp\u002Frun_\u003Cslug>.sh\n```\n\nWhy: `execute_code` outputs vanish after the call returns. Anything you want to ship in the eval (raw logits, per-prompt timings, prompts themselves) must be written via a real shell script and survive into the next tool call.\n\n## Phase 6 — Publish to `jleechanorg\u002Fevals`\n\nSelf-contained eval repo structure (matches the canonical layout landed in this repo's own session):\n\n```\njleechanorg\u002Fevals\u002F\n├── README.md                      # top-level index + conventions\n├── LICENSE                        # MIT, with note that vendored code keeps its own license\n├── .gitignore                     # never commit model weights or upstream clones\n└── docs\u002F\u003Ceval-slug>\u002F\n    ├── README.md                  # folder-level entry, TL;DR, link map\n    ├── report.md                  # full narrative (what we tested, what we found, verdict)\n    ├── repro.sh                   # one-command reproduction (one real shell script)\n    └── raw\u002F\n        ├── prompts.json\n        └── results.json\n```\n\nPer-eval folder naming: `docs\u002F\u003Ceval-type>-\u003Csubject>\u002F` where `\u003Ceval-type>` is one of:\n- `cli-eval-\u003Ctool-name>` for CLI tools\n- `router-eval-\u003Cvendor-repo>` for LLM routers\n- `model-eval-\u003Cmodel-name>` for raw model evaluations\n- `tool-eval-\u003Ctool-name>` for SaaS \u002F hosted tools\n\nCommit message convention: `[agento] feat: \u003Ceval-name> — \u003Cone-line summary>`. The `[agento]` prefix matches the convention from worldarchitect\u002Fagent-orchestrator work.\n\n**Before `git push`**, per `pr-ci-fix-autopush` and `push-pr-donot-stop-halfway`:\n- `git rev-parse origin\u002Fmain` must equal `HEAD` after push\n- `gh repo view jleechanorg\u002Fevals` shows the new description and default branch\n- The README renders correctly on github.com (visit it once via `web_extract` against the raw URL if you want to verify)\n- No model weights, venvs, or `upstream-clone\u002F` directories in the diff (`.gitignore` MUST cover them)\n\n## Pitfalls\n\n### P1. Don't ask permission menus when the user gave a directive\n\nBad first turn after curl fails:\n> \"Do you want me to (A) try harder on the article, (B) wait for you to paste, or (C) try a different approach?\"\n\nGood first turn:\n> `browser_navigate(url)` (the call itself, with a 1-line summary)\n\nThe `task-ack-and-execute` SOUL.md COMMIT already requires the first turn to contain a tool call. This pitfall reinforces: **after a tool fails, don't pivot to asking — pivot to the next tool.**\n\n### P2. README ≠ contract\n\nAlways run a contract verification step (Phase 2) before trusting any headline numbers. Common violations: claimed parameter count is loose, claimed accuracy has no test set, claimed latency is GPU-only at batch=1.\n\n### P3. Sandbox outputs don't survive\n\n`execute_code` outputs vanish after the call. Use a real `bash` subprocess via `write_file` → `terminal bash` if you need raw outputs to persist.\n\n### P4. Class-count mismatches are the #1 contract violation in ML repos\n\nAlways print the checkpoint head shape BEFORE running inference. Save it to the eval's `raw\u002Fresults.json` so future readers can verify your setup.\n\n### P5. Latency claims are scope-dependent\n\n\"10ms router\" might be GPU-only, batch=1, on a specific instance type. Don't trust without reading the test setup. Always re-measure on the eval's own machine.\n\n### P6. Paywalled content is a real obstacle\n\nMedium and similar sites have aggressive bot-walls. If `browser_navigate` headless + `BROWSERBASE_ADVANCED_STEALTH` can't get past it, the article itself is unreadable. **Say so** in the report rather than fabricating a summary.\n\n### P7. Don't leak credentials in eval artifacts\n\nPer the `outbound-secret-publication-gate` SOUL.md COMMIT, before pushing anything to `jleechanorg\u002Fevals`, scan all `.md` and `.sh` files for `ghp_`, `xoxp-`, `sk-`, `AKIA` patterns. Never commit tokens, no matter how \"obvious\" they seem in a private context.\n\n## Verification\n\nAfter publishing:\n\n1. `git -C ~\u002Fprojects\u002Fevals log --oneline -1` — commit exists\n2. `git -C ~\u002Fprojects\u002Fevals rev-parse origin\u002Fmain` equals `HEAD` (no local commits behind)\n3. `gh repo view jleechanorg\u002Fevals --json description,defaultBranchRef` — description + main branch set\n4. Web-fetch the raw README URL on github.com to verify markdown renders\n5. `bash docs\u002F\u003Cslug>\u002Frepro.sh` from a clean checkout — runs end-to-end and reproduces `raw\u002Fresults.json`\n\n## Related references\n\n- `references\u002Frouter-eval-deycoding-2026-07-22.md` — first worked example, in this repo's own session: a ~140M encoder BFSI router that doesn't load out of the box, can't do whole-convo classification, and has a contract mismatch.\n\n## Related skills\n\n- `browser-headless-default` — for the headless browser pivot path\n- `github-repo-management` — for `gh repo create`, `git push`, etc.\n- `evidence-attach-not-path-cite` — for embedding screenshots \u002F evidence in reports\n- `outbound-secret-publication-gate` (SOUL.md COMMIT) — for the pre-publish credential scan\n- `pr-clean-branch-from-main-no-history-bloat` (SOUL.md COMMIT) — if the eval repo itself needs branch discipline",{"name":5,"author":58,"license":14,"version":59,"metadata":60,"platforms":74,"description":7},"Hermes Agent","1.0.0",{"hermes":61},{"tags":62,"related_skills":68},[26,25,63,64,65,66,67],"third-party","reproducibility","claim-verification","ml-model","public-publish",[69,70,71,72,73],"browser-headless-default","github-repo-management","github-pr-workflow","evidence-attach-not-path-cite","outbout-secret-publication-gate",[52,75],"macos","main","42ccd02cbf5e8e47839dc3eab6fc754dc6b339cf","hermes\u002Fskills\u002Fdevops\u002Feval-vendor-tooling",0,"https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills","https:\u002F\u002Fgithub.com\u002Fjleechanorg","2026-11-11 22:19:36",{"source":8,"skillName":5,"commands":84,"prompt":94,"zipAvailable":95},[85,88,91],{"tool":86,"command":87},"npx","npx skills add jleechanorg\u002Fjleechan-skills --skill eval-vendor-tooling",{"tool":89,"command":90},"bunx","bunx skills add jleechanorg\u002Fjleechan-skills --skill eval-vendor-tooling",{"tool":92,"command":93},"pnpm","pnpm dlx skills add jleechanorg\u002Fjleechan-skills --skill eval-vendor-tooling","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002Fhermes\u002Fskills\u002Fdevops\u002Feval-vendor-tooling\u002FSKILL.md",true,[97],{"path":98,"resourceType":99,"resourceTypeDesc":100,"sizeBytes":101,"sourceHtmlUrl":102},"references\u002Frouter-eval-deycoding-2026-07-22.md","REFERENCE","参考文档",6295,"https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002Fhermes\u002Fskills\u002Fdevops\u002Feval-vendor-tooling\u002Freferences\u002Frouter-eval-deycoding-2026-07-22.md",[104,120,140,154,166,180],{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":53},"01M716Y5MTJ6GYJZ2ATXCJK8G4","instinct","Use when the user invokes \u002Finstinct or \u002Fin, or asks to \"message Instinct\", \"ask Instinct\", \"tell Instinct\", \"check Instinct\", or read replies from the configured peer assistant. iMessage first, WhatsApp Web as fallback.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Finstinct\u002FSKILL.md",[],[106,111,28,112,113,114,115,116,117,118,33,119],"user","replies","check","use","identity","whatsapp","gmail","read","ask",{"skillNo":121,"name":122,"nameZh":6,"description":123,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":124,"categories":125,"keywords":129,"publishedAt":53},"01M716Y4HF7FVVMJSA2CCVN5VF","hr","Compatibility alias for history-resume; produce the same fresh-state, provenance-aware resume briefing.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fhr\u002FSKILL.md",[126],{"categoryNo":127,"categoryName":128,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000038","人力资源",[122,130,131,132,133,134,135,136,137,138,139],"resume","compatibility","provenance","for","history","produce","alias","briefing","state","fresh",{"skillNo":141,"name":142,"nameZh":6,"description":143,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":144,"categories":145,"keywords":147,"publishedAt":53},"01M716Y1FFNFKFJYYKF8HQA6T8","history-resume","Resume briefing for the current repository branch or worktree using fresh state and admitted history sources.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fhistory-resume\u002FSKILL.md",[146],{"categoryNo":127,"categoryName":128,"taxonomyType":20,"taxonomyTypeDesc":21},[134,130,148,149,111,150,137,151,133,152,153,138,139],"history resume","branch","analysis","repository","worktree","sources",{"skillNo":155,"name":156,"nameZh":6,"description":157,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":158,"categories":159,"keywords":160,"publishedAt":53},"01M716XWXJ8EPHK4RH9T14TR85","grok","Alias for the canonical grokbot browser conversation workflow; \u002Fgrok talks to the Grok bot the same way \u002Fdot talks to Dot.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fgrok\u002FSKILL.md",[],[156,161,162,51,133,163,136,164,48,165],"dot","bot","canonical","conversation","grokbot",{"skillNo":167,"name":168,"nameZh":6,"description":169,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":170,"categories":171,"keywords":172,"publishedAt":53},"01M70N9Q07DTE0EW5KFXMMMSJ0","codexa-bench","Compare `codexa` and normal `codex` on a fair split of comparable work by default; use `--strict` for duplicate runs of the exact same task.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fcodexa-bench\u002FSKILL.md",[],[173,40,174,114,36,133,175,176,177,178,179],"codexa bench","split","duplicate","task","compare","normal","codex",{"skillNo":181,"name":182,"nameZh":6,"description":183,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":184,"licenseFileUrl":185,"sourceHtmlUrl":186,"categories":187,"keywords":199,"publishedAt":53},"01M6C24S2X60SP8BBFGAEKPEGW","frontend-design","Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, artifacts, posters, or applications (examples include websites, landing pages, dashboards, React components, HTML\u002FCSS layouts, or when styling\u002Fbeautifying any web UI). Generates creative, polished code and UI design that avoids generic AI aesthetics.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ffrontend-design\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ffrontend-design\u002FSKILL.md",[188,193,196],{"categoryNo":189,"categoryName":190,"taxonomyType":191,"taxonomyTypeDesc":192},"01JBSEEDCAT000000000000009","前端开发","SCENARIO","使用场景",{"categoryNo":194,"categoryName":195,"taxonomyType":191,"taxonomyTypeDesc":192},"01JBSEEDCAT000000000000015","视觉设计",{"categoryNo":197,"categoryName":198,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000032","设计师",[200,201,202,114,203,204,205,111,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,28,224],"frontend","design","frontend design","generic","ui design","react","html","components","code","artifacts","ui","aesthetics","Production","web components","landing","applications","quality","web","css","create","examples","with","react components","web ui","pages"]