[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6CATFE8AYDXA5P1AH2212FR":3,"skill-same:01M6CATFE8AYDXA5P1AH2212FR":123},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":22,"publishedAt":53,"linkOnly":54,"markdownContent":55,"markdownBody":56,"frontMatter":57,"refName":90,"commitSha":91,"skillDirectory":92,"forksCount":93,"repositoryHtmlUrl":94,"repositoryHomepageUrl":6,"ownerHtmlUrl":95,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":96,"install":97,"resources":110},"01M6CATFE8AYDXA5P1AH2212FR","vendor-ml-artifact-eval",null,"Pre-adoption evaluation recipe for any third-party ML artifact (router, classifier, embedding model, RAG pipeline). Before recommending adoption — or even running it on real production traffic — load this skill and run the 6-gate checklist: (1) README-vs-claim triangulation, (2) code-level load-path inspection, (3) shipped-checkpoint-vs-claimed-class-count gate (the silent killer — 2026-07-22 incident: NUM_LABELS=4 in code, NUM_LABELS=6 in shipped weights), (4) per-turn architecture boundary check (truncation, fallback policy, confidence threshold), (5) on-distribution label-coverage audit against YOUR real workload, (6) realistic cost-cut estimate for YOUR workload (not the vendor's headline figure).","jleechanorg\u002Fjleechan-skills","jleechanorg","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F221404966?v=4","01M6C1VJ99XG950X1DWPEBCBGX","01M6C1VJDK5YVDNT3362RT5WSB",6,"MIT","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002Fhermes\u002Fskills\u002Fvendor-ml-artifact-eval\u002FSKILL.md",[17],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000009","前端开发","SCENARIO","使用场景",[23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52],"artifact","vendor","eval","vendor ml artifact eval","ml","traffic","port","cut","adopt","fix","pre-adoption","classifier","patch","labels","good","router","inspection","frontend","coverage","choose","model","projects","pipeline","audit","verification","code","run","rag","architecture","cli","2026-11-10 10:12:30",false,"---\nname: vendor-ml-artifact-eval\nversion: 0.1.0\ndescription: \"Pre-adoption evaluation recipe for any third-party ML artifact (router, classifier, embedding model, RAG pipeline). Before recommending adoption — or even running it on real production traffic — load this skill and run the 6-gate checklist: (1) README-vs-claim triangulation, (2) code-level load-path inspection, (3) shipped-checkpoint-vs-claimed-class-count gate (the silent killer — 2026-07-22 incident: NUM_LABELS=4 in code, NUM_LABELS=6 in shipped weights), (4) per-turn architecture boundary check (truncation, fallback policy, confidence threshold), (5) on-distribution label-coverage audit against YOUR real workload, (6) realistic cost-cut estimate for YOUR workload (not the vendor's headline figure).\"\ntags: [\"eval\", \"vendor\", \"router\", \"classifier\", \"model\", \"adoption\", \"pre-adoption\", \"ml-artifact\"]\ncategory: software-development\ntriggers:\n  - evaluate this model\n  - evaluate this router\n  - evaluate this classifier\n  - is this good for our use case\n  - should we adopt this\n  - replay our convos through this\n  - does it pass for classification\n  - check if the claims hold\n  - vendor model eval\n  - third-party router eval\n  - run it on my past convos\n  - what would it choose on my data\n  - are the choices right\n  - eval before adoption\n  - pre-adoption check\nchangelog:\n  - \"0.1.0 (2026-07-22): Initial version. Recipe distilled from `deycoding\u002Fdeycoding-compliance-classifier-router` evaluation — the first end-to-end run of this checklist. Six gates + the headline finding pattern (NUM_LABELS=4 vs shipped-checkpoint head=6) that the eval surfaced. Companion to `finish-the-job` (the eval-stall anti-pattern was patched there). Verified end-state: 11 prompts run against the actual model, 441MB weights loaded from HuggingFace, eval report at `~\u002Fprojects\u002Frouter-eval\u002FEVAL_REPORT.md`.\"\nrelated_skills:\n  - finish-the-job\n  - verify-deployed-frontend-fix\n  - cli-env-var-verification\n  - convergent-bug-triage\n  - pr-description-validator-gate6b\n  - patch-port-protocol\n  - skillify\n  - harness-engineering\n---\n\n# vendor-ml-artifact-eval\n\n**Pre-adoption evaluation recipe for third-party ML artifacts.** Run this skill when a user asks to evaluate \u002F adopt \u002F replay-against-real-data \u002F check-claims-hold on a third-party model, router, classifier, embedding, or pipeline. The class-of-failure mode this skill catches: **vendor claims that pass the headline test but break under code-level inspection.**\n\n## Why this skill exists\n\nHeadline claims (\"99.2% accuracy\", \"65-73% cost savings\", \"\u003C10ms latency\", \"supports whole-convo classification\") are the **first** layer of evaluation, not the last. The 2026-07-22 incident (this skill's originating event) had every headline claim — and the shipped repo **does not run end-to-end as-documented**. Specifically:\n\n- `server\u002Fserver.py` sets `NUM_LABELS = 4` and a 4-entry `LABELS` dict (simple_no_pii, simple_pii, complex_no_pii, complex_pii).\n- The shipped `state_dict` from HuggingFace has a **6-class** classification head (`classifier.3.weight: torch.Size([6, 768])`).\n- The README documents the architecture as `Linear(768-768) - Tanh - Dropout - Linear(768-4)`.\n- **Three sources of truth disagree.** The FastAPI server cannot start; `load_state_dict` raises on shape mismatch with the default `strict=True`.\n\nThis skill catches that class of failure systematically, **before** recommending adoption. The same checklist catches: short-context truncation that makes \"whole-convo\" impossible, fallback policies that mask real ambiguity, label sets that don't cover the user's actual distribution, and headline cost-savings numbers that only hold for the vendor's narrow workload.\n\n## When to load\n\nLoad this skill when **any** of these signals fire:\n\n| Signal | Example |\n|---|---|\n| User shares a third-party ML artifact and asks \"is this good\" | \"evaluate this: https:\u002F\u002Fgithub.com\u002F\u003Cvendor>\u002F\u003Crouter>\" |\n| User asks to replay their data through a third-party classifier | \"can we replay it against some of my older convos\" |\n| User asks for adoption guidance on a vendor router | \"should we adopt this for our cost-cut\" |\n| User asks to verify vendor claims | \"do the claims hold \u002F does it actually do what they say\" |\n| User shares a Medium \u002F blog post about a vendor's ML pipeline | \"look at this article and tell me if it would work for us\" |\n\nDo **not** load this skill for: evaluating your own model (use `verify-deployed-frontend-fix` or the harness-engineering skill), CLI tool env var claims (use `cli-env-var-verification`), sibling-bug triage on your own campaign (use `convergent-bug-triage`).\n\n## The 6-gate checklist\n\nRun these in order. **Each gate is independently load-bearing** — a fail at gate N invalidates downstream gate results.\n\n### Gate 1 — README-vs-claim triangulation\n\n**Goal:** Verify the artifact's docs match its claims. Even before reading code, mine the README (or article, or paper) for:\n\n- Architecture numbers (parameters, dimensions, layers, max sequence length, vocabulary size)\n- Latency claims (GPU, CPU, throughput)\n- Accuracy claims (overall accuracy, per-class precision\u002Frecall, PII recall)\n- Cost-savings claims (typically stated as a percentage on the vendor's workload)\n- Label set \u002F class taxonomy (what the model is supposed to output)\n- Deployment story (one command? CloudFormation? Docker? local-only?)\n\nWrite these into a 1-row table at the top of the eval report. You'll cross-reference each claim against the code at Gate 3.\n\n**Pitfall:** Medium \u002F blog articles often lose nuance that the README preserves. If the article is bot-walled or paywalled, the **README IS the article** — read it as the canonical source. Do not ask the user to paste the article.\n\n### Gate 2 — Code-level load-path inspection\n\nRead the actual file(s) that load and use the model. For PyTorch: `model = SomeClass(...)` + `model.load_state_dict(torch.load(MODEL_PATH, ...))` + `model.eval()`. For HuggingFace transformers: `AutoModel.from_pretrained(...)` + `AutoTokenizer.from_pretrained(...)`. For ONNX: `ort.InferenceSession(...)`. For TFLite: `tf.lite.Interpreter(...)`.\n\nExtract and tabulate:\n\n| Field | Value | Source |\n|---|---|---|\n| `NUM_LABELS` \u002F `num_classes` | integer | constants or config |\n| `MAX_SEQ_LEN` \u002F `max_position_embeddings` | integer | config dataclass or argparse |\n| `LABELS` dict \u002F label mapping | name → index | constants or yaml |\n| `MODEL_PATH` env var default | string path | env var block |\n| `TOKENIZER_PATH` env var default | string path | env var block |\n| `CONFIDENCE_THRESHOLD` default | float | agent config |\n| `FALLBACK_LABEL` default | string | agent config |\n| `device` resolution | cuda \u002F cpu \u002F mps | runtime check |\n\n### Gate 3 — Shipped-checkpoint-vs-claimed-class-count gate (the silent killer)\n\n**This gate is the load-bearing one.** It catches the `NUM_LABELS=4 in code, NUM_LABELS=6 in weights` class of bug. The recipe:\n\n```python\nimport torch\nsd = torch.load('\u003Cmodel_path>', map_location='cpu', weights_only=False)\n# Find the classifier head's weight tensor (last linear layer's W)\n# Typical names: 'classifier.3.weight', 'classifier.classifier.weight',\n# 'head.weight', 'output.weight', 'pooler.dense.weight'\nfor k, v in sd.items():\n    if 'classifier' in k.lower() or 'head' in k.lower() or 'output' in k.lower():\n        if v.ndim == 2:\n            print(k, v.shape)  # should be (NUM_CLASSES, HIDDEN_DIM)\n            break\n```\n\nIf the head shape does NOT match the claimed `NUM_LABELS`, **the repo is broken end-to-end**. The FastAPI server will raise on load. **Surface this in the verdict as a ? red finding**, not as a paper-over. Two possible resolutions:\n\n1. **Update `NUM_LABELS` in code** to match the checkpoint. This unblocks inference but breaks the documented label mapping — you don't know what indices 4, 5, etc. mean unless the vendor published a separate label dictionary.\n2. **Re-train or re-export** the checkpoint to match the documented label set. This unblocks the documented label mapping but requires the vendor's training code.\n\nEither way, do not adopt the repo as-is. Surface this to the user before any deployment.\n\n### Gate 4 — Per-turn architecture boundary check\n\nMost vendor routers are **per-turn**, not per-conversation. Verify by checking:\n\n- `MAX_SEQ_LEN` \u002F `max_position_embeddings` \u002F `truncation` defaults — if it's small (e.g. 128, 512), the model silently truncates anything longer.\n- The API contract (`POST \u002Fclassify` or `POST \u002Fchat` or `POST \u002Froute`) — does it accept a single query, a list of queries, or a multi-message conversation?\n- The model architecture — bidirectional encoders (BERT, RoBERTa, MiniMax-M3-embed) are per-segment. Causal decoders (GPT, LLaMA) are autoregressive. Long-context variants (Longformer, BigBird, RoPE-extended) are the only ones that can do whole-convo.\n\nIf the user asked \"does it pass in the whole convo\" and the architecture is per-turn, **the answer is no, by design**. That's not a bug; that's the wrong product. Don't recommend a per-turn router for a whole-convo problem; recommend a different product.\n\n### Gate 5 — On-distribution label-coverage audit\n\nThe vendor's published label set may not match the user's actual task distribution. Mine the user's past session_search for ~30-100 recent real prompts and bucket them by task type. Compare that distribution against the vendor's class taxonomy.\n\nExample from the 2026-07-22 incident: the vendor's 4-class taxonomy was `{simple_no_pii, simple_pii, complex_no_pii, complex_pii}`. The user's real distribution was roughly `{git\u002FCLI ops, Slack digests, AO orchestration, CSS\u002FUI, PR review, beads, memory\u002Fskill management}`. **Zero overlap.** Even if the model classifies perfectly, the routing decisions don't help — a `gh pr view` query routed to \"small model\" is wrong because the user already prefers the long-context model for that query type.\n\nRecipe:\n\n1. Mine `session_search` for the user's top 5-10 active projects over the last 30 days.\n2. Extract the first user prompt from each session.\n3. Bucket them by hand-labeled task type (you can use a quick LLM call to label them in batches, or eyeball it for 30-50 prompts).\n4. Compare the bucket distribution to the vendor's label taxonomy.\n5. Flag as ? if \u003C30% of user tasks have a natural mapping to a vendor class; flag as ? if \u003C10%.\n\n### Gate 6 — Realistic cost-cut estimate for YOUR workload\n\nVendor headline numbers are measured on **the vendor's workload**, not yours. The 60% cost cut on `deycoding-compliance-classifier-router` is a BFSI banking-chat figure — your dev-ops \u002F Slack-digest \u002F browser-automation distribution won't see the same savings.\n\nRecipe for a realistic estimate:\n\n1. Look at the vendor's cost-cut derivation: it's typically `(small-model-cost * %-small-traffic) + (large-model-cost * %-large-traffic) - routing-overhead`.\n2. Estimate your %-small-traffic by applying Gate 5's bucket distribution to the vendor's small\u002Flarge routing rules.\n3. Estimate routing overhead (typically 5-15ms per request).\n4. Compare against your **current** cost distribution (large model for everything → 100% large cost).\n5. Realistic savings on a non-matching workload: **5-20%**, not the vendor's headline.\n\nBe honest with the user about this. A 60% saving on the vendor's workload is a marketing number; a 10-20% saving on yours is the engineering reality. Do not adopt based on the vendor's figure.\n\n## Verdict shape (mandatory)\n\nEvery eval run ends with a verdict in this shape:\n\n```\n? \u002F ? \u002F ?  verdict on whether to adopt\n\n- Headline finding: \u003Cthe single most damning or interesting thing the eval turned up>\n- Gate-by-gate: 1-6 with pass\u002Ffail and one-line evidence\n- Realistic cost-cut for your workload: \u003Cnumber>%\n- Recommendation: adopt \u002F fork-and-patch \u002F do-not-adopt \u002F build-our-own\n- Proof: \u003Cpath to eval report, raw logits, log dump>\n```\n\nIf the verdict is \"do not adopt\" or \"fork-and-patch\", also include a one-line \"what would change my mind\" or \"what fork-and-patch requires\". The user wants to know the minimum delta that would make adoption viable.\n\n## Anti-patterns (do not do)\n\n- ❌ **\"Trust the README.\"** The README is the first source of truth, not the only one. Always triangulate with code + weights + behavior.\n- ❌ **\"Trust the headline accuracy.\"** Headline accuracy is measured on the vendor's test set. Your distribution is different. Gate 5 catches this.\n- ❌ **\"Skip Gate 3 because the README says it works.\"** Gate 3 is the silent killer. The 2026-07-22 incident proves a vendor can ship a README that documents the wrong number of classes while the shipped weights have the right (different) number.\n- ❌ **\"Classify the user's data and report accuracy.\"** Without Gate 5's label-coverage audit, the accuracy number is meaningless — you're measuring how well the model classifies into a taxonomy the user doesn't actually use.\n- ❌ **\"Recommend adoption if the architecture sounds right.\"** Architecture rightness ≠ workload rightness. The same router can be production-grade for BFSI and a bad fit for engineering dev work.\n- ❌ **\"Stall on a pre-execution menu.\"** When the user says \"evaluate this, replay my convos, are the choices right\", execute the verbs in sequence. See `finish-the-job` `references\u002Feval-stall-on-preflight-menu-2026-07-22.md`.\n\n## Support files\n\n- `scripts\u002Fcheck_checkpoint_class_count.py` — Gate 3 reproducer. Detects PyTorch \u002F HuggingFace classifier heads, walks the state_dict or config, and reports (or asserts, with `--expected N`) the class count. Catches the 2026-07-22 incident's class of bug in one command: `python3 scripts\u002Fcheck_checkpoint_class_count.py \u003Cmodel_path> --expected 4`. Exit 0 = pass, exit 1 = mismatch.\n\n## Cross-references\n\n- `finish-the-job` — the eval-stall anti-pattern (pre-execution menu) was patched there with a companion reference.\n- `verify-deployed-frontend-fix` — three-layer proof pattern (merged code + bundle hash + runtime computed style) is analogous: Gate 1 (README) + Gate 2 (code) + Gate 3 (weights) + Gates 4-5 (runtime behavior + coverage).\n- `cli-env-var-verification` — binary-string grep on compiled executable + official vendor docs + live process test + config-file presence. Same triangulation discipline, applied to env vars instead of model artifacts.\n- `convergent-bug-triage` — for when the eval surfaces multiple sibling issues (e.g. three separate failures all traceable to the same root cause like the class-count mismatch).\n- `pr-description-validator-gate6b` — same gate discipline (read-the-validator-locally, don't trust the headline PASS\u002FFAIL summary), applied to PR description validators instead of model artifacts.\n\n## Worked example — the 2026-07-22 incident\n\nUser asked: *\"clone this locally and evaluate it. Does it pass in the whole convo for classification? Can we replay it against some of my older convos\u002Fprojects to see what it would choose and if the choices right?\"* with a Medium link.\n\nPhase 0: Classified as vendor-ml-artifact-eval (no existing skill covered this class). Loaded this skill.\n\nGate 1 (README-vs-claim): Cloned `deycoding\u002Fdeycoding-compliance-classifier-router`. README documents `NUM_LABELS=4` (simple_no_pii \u002F simple_pii \u002F complex_no_pii \u002F complex_pii), `MAX_SEQ_LEN=128`, ~140M params, ~7ms GPU latency, 99.2% accuracy.\n\nGate 2 (code-level load path): `server\u002Fserver.py:18-19` shows `NUM_LABELS = 4; LABELS = {0: \"simple_no_pii\", 1: \"simple_pii\", 2: \"complex_no_pii\", 3: \"complex_pii\"}`. `MAX_SEQ_LEN = 128` at line 17. `model.load_state_dict(torch.load(MODEL_PATH, ...), weights_only=False)` at line 118 — note `weights_only=False` (security smell, but not the issue here).\n\nGate 3 (checkpoint-vs-claimed-class-count): Loaded the model weights from HuggingFace. The final classification layer `classifier.3.weight` has shape `torch.Size([6, 768])`. **MISMATCH.** The shipped code wants `NUM_LABELS=4` but the shipped weights have a 6-class head. `load_state_dict` will raise. **? Block.**\n\nGate 4 (per-turn architecture boundary): `MAX_SEQ_LEN=128` tokens. README example for `Design fraud detection system for UPI transactions` (8 tokens) fits; `Read my convo history to see what wasn't optimal...` (65 tokens) fits but is heavily truncated; a full conversation thread (2000-4000 words) does NOT fit. **The model cannot do whole-convo classification. Different product.**\n\nGate 5 (on-distribution label coverage): Mined 8 real past Jeffrey prompts via `session_search`. Buckets: `git\u002FCLI ops` (3), `AO orchestration` (2), `Slack digests` (1), `PR work` (1), `other` (1). Vendor's 4-class taxonomy (complexity × PII) has **zero direct overlap** with this distribution. Routing a `git push origin \u003Cbranch>` to \"small model\" because it has no PII is wrong.\n\nGate 6 (realistic cost-cut estimate): Vendor's 60% figure is BFSI-derived. Engineering dev workload (long-context code review, file diff analysis, multi-file refactors) overwhelmingly wants the long-context model — the small-model tier wouldn't even fit most past prompts in its 128-token context window. Realistic estimate for the user's actual distribution: **10-20%**, not 60%.\n\n**Verdict:** ? *workable compliance-routing primitive; **does not, and cannot, do whole-conversation classification** as the user asked; shipped code and shipped weights are out of sync; not ready for drop-in. Recommend fork + re-train on a task-type taxonomy matching the user's session distribution (~2-week AO dispatch).*\n\n**Proof:** Eval report at `~\u002Fprojects\u002Frouter-eval\u002FEVAL_REPORT.md` (16 KB). Raw logits at `\u002Ftmp\u002Frouter_eval_results.json`. Model weights + tokenizer at `~\u002Fprojects\u002Frouter-eval\u002Fdeycoding-compliance-classifier-router\u002Fserver\u002Fdeycoding.compliance-classifier-in-1-0.{pt,json}`. Reproducer recipe documented in the report.","# vendor-ml-artifact-eval\n\n**Pre-adoption evaluation recipe for third-party ML artifacts.** Run this skill when a user asks to evaluate \u002F adopt \u002F replay-against-real-data \u002F check-claims-hold on a third-party model, router, classifier, embedding, or pipeline. The class-of-failure mode this skill catches: **vendor claims that pass the headline test but break under code-level inspection.**\n\n## Why this skill exists\n\nHeadline claims (\"99.2% accuracy\", \"65-73% cost savings\", \"\u003C10ms latency\", \"supports whole-convo classification\") are the **first** layer of evaluation, not the last. The 2026-07-22 incident (this skill's originating event) had every headline claim — and the shipped repo **does not run end-to-end as-documented**. Specifically:\n\n- `server\u002Fserver.py` sets `NUM_LABELS = 4` and a 4-entry `LABELS` dict (simple_no_pii, simple_pii, complex_no_pii, complex_pii).\n- The shipped `state_dict` from HuggingFace has a **6-class** classification head (`classifier.3.weight: torch.Size([6, 768])`).\n- The README documents the architecture as `Linear(768-768) - Tanh - Dropout - Linear(768-4)`.\n- **Three sources of truth disagree.** The FastAPI server cannot start; `load_state_dict` raises on shape mismatch with the default `strict=True`.\n\nThis skill catches that class of failure systematically, **before** recommending adoption. The same checklist catches: short-context truncation that makes \"whole-convo\" impossible, fallback policies that mask real ambiguity, label sets that don't cover the user's actual distribution, and headline cost-savings numbers that only hold for the vendor's narrow workload.\n\n## When to load\n\nLoad this skill when **any** of these signals fire:\n\n| Signal | Example |\n|---|---|\n| User shares a third-party ML artifact and asks \"is this good\" | \"evaluate this: https:\u002F\u002Fgithub.com\u002F\u003Cvendor>\u002F\u003Crouter>\" |\n| User asks to replay their data through a third-party classifier | \"can we replay it against some of my older convos\" |\n| User asks for adoption guidance on a vendor router | \"should we adopt this for our cost-cut\" |\n| User asks to verify vendor claims | \"do the claims hold \u002F does it actually do what they say\" |\n| User shares a Medium \u002F blog post about a vendor's ML pipeline | \"look at this article and tell me if it would work for us\" |\n\nDo **not** load this skill for: evaluating your own model (use `verify-deployed-frontend-fix` or the harness-engineering skill), CLI tool env var claims (use `cli-env-var-verification`), sibling-bug triage on your own campaign (use `convergent-bug-triage`).\n\n## The 6-gate checklist\n\nRun these in order. **Each gate is independently load-bearing** — a fail at gate N invalidates downstream gate results.\n\n### Gate 1 — README-vs-claim triangulation\n\n**Goal:** Verify the artifact's docs match its claims. Even before reading code, mine the README (or article, or paper) for:\n\n- Architecture numbers (parameters, dimensions, layers, max sequence length, vocabulary size)\n- Latency claims (GPU, CPU, throughput)\n- Accuracy claims (overall accuracy, per-class precision\u002Frecall, PII recall)\n- Cost-savings claims (typically stated as a percentage on the vendor's workload)\n- Label set \u002F class taxonomy (what the model is supposed to output)\n- Deployment story (one command? CloudFormation? Docker? local-only?)\n\nWrite these into a 1-row table at the top of the eval report. You'll cross-reference each claim against the code at Gate 3.\n\n**Pitfall:** Medium \u002F blog articles often lose nuance that the README preserves. If the article is bot-walled or paywalled, the **README IS the article** — read it as the canonical source. Do not ask the user to paste the article.\n\n### Gate 2 — Code-level load-path inspection\n\nRead the actual file(s) that load and use the model. For PyTorch: `model = SomeClass(...)` + `model.load_state_dict(torch.load(MODEL_PATH, ...))` + `model.eval()`. For HuggingFace transformers: `AutoModel.from_pretrained(...)` + `AutoTokenizer.from_pretrained(...)`. For ONNX: `ort.InferenceSession(...)`. For TFLite: `tf.lite.Interpreter(...)`.\n\nExtract and tabulate:\n\n| Field | Value | Source |\n|---|---|---|\n| `NUM_LABELS` \u002F `num_classes` | integer | constants or config |\n| `MAX_SEQ_LEN` \u002F `max_position_embeddings` | integer | config dataclass or argparse |\n| `LABELS` dict \u002F label mapping | name → index | constants or yaml |\n| `MODEL_PATH` env var default | string path | env var block |\n| `TOKENIZER_PATH` env var default | string path | env var block |\n| `CONFIDENCE_THRESHOLD` default | float | agent config |\n| `FALLBACK_LABEL` default | string | agent config |\n| `device` resolution | cuda \u002F cpu \u002F mps | runtime check |\n\n### Gate 3 — Shipped-checkpoint-vs-claimed-class-count gate (the silent killer)\n\n**This gate is the load-bearing one.** It catches the `NUM_LABELS=4 in code, NUM_LABELS=6 in weights` class of bug. The recipe:\n\n```python\nimport torch\nsd = torch.load('\u003Cmodel_path>', map_location='cpu', weights_only=False)\n# Find the classifier head's weight tensor (last linear layer's W)\n# Typical names: 'classifier.3.weight', 'classifier.classifier.weight',\n# 'head.weight', 'output.weight', 'pooler.dense.weight'\nfor k, v in sd.items():\n    if 'classifier' in k.lower() or 'head' in k.lower() or 'output' in k.lower():\n        if v.ndim == 2:\n            print(k, v.shape)  # should be (NUM_CLASSES, HIDDEN_DIM)\n            break\n```\n\nIf the head shape does NOT match the claimed `NUM_LABELS`, **the repo is broken end-to-end**. The FastAPI server will raise on load. **Surface this in the verdict as a ? red finding**, not as a paper-over. Two possible resolutions:\n\n1. **Update `NUM_LABELS` in code** to match the checkpoint. This unblocks inference but breaks the documented label mapping — you don't know what indices 4, 5, etc. mean unless the vendor published a separate label dictionary.\n2. **Re-train or re-export** the checkpoint to match the documented label set. This unblocks the documented label mapping but requires the vendor's training code.\n\nEither way, do not adopt the repo as-is. Surface this to the user before any deployment.\n\n### Gate 4 — Per-turn architecture boundary check\n\nMost vendor routers are **per-turn**, not per-conversation. Verify by checking:\n\n- `MAX_SEQ_LEN` \u002F `max_position_embeddings` \u002F `truncation` defaults — if it's small (e.g. 128, 512), the model silently truncates anything longer.\n- The API contract (`POST \u002Fclassify` or `POST \u002Fchat` or `POST \u002Froute`) — does it accept a single query, a list of queries, or a multi-message conversation?\n- The model architecture — bidirectional encoders (BERT, RoBERTa, MiniMax-M3-embed) are per-segment. Causal decoders (GPT, LLaMA) are autoregressive. Long-context variants (Longformer, BigBird, RoPE-extended) are the only ones that can do whole-convo.\n\nIf the user asked \"does it pass in the whole convo\" and the architecture is per-turn, **the answer is no, by design**. That's not a bug; that's the wrong product. Don't recommend a per-turn router for a whole-convo problem; recommend a different product.\n\n### Gate 5 — On-distribution label-coverage audit\n\nThe vendor's published label set may not match the user's actual task distribution. Mine the user's past session_search for ~30-100 recent real prompts and bucket them by task type. Compare that distribution against the vendor's class taxonomy.\n\nExample from the 2026-07-22 incident: the vendor's 4-class taxonomy was `{simple_no_pii, simple_pii, complex_no_pii, complex_pii}`. The user's real distribution was roughly `{git\u002FCLI ops, Slack digests, AO orchestration, CSS\u002FUI, PR review, beads, memory\u002Fskill management}`. **Zero overlap.** Even if the model classifies perfectly, the routing decisions don't help — a `gh pr view` query routed to \"small model\" is wrong because the user already prefers the long-context model for that query type.\n\nRecipe:\n\n1. Mine `session_search` for the user's top 5-10 active projects over the last 30 days.\n2. Extract the first user prompt from each session.\n3. Bucket them by hand-labeled task type (you can use a quick LLM call to label them in batches, or eyeball it for 30-50 prompts).\n4. Compare the bucket distribution to the vendor's label taxonomy.\n5. Flag as ? if \u003C30% of user tasks have a natural mapping to a vendor class; flag as ? if \u003C10%.\n\n### Gate 6 — Realistic cost-cut estimate for YOUR workload\n\nVendor headline numbers are measured on **the vendor's workload**, not yours. The 60% cost cut on `deycoding-compliance-classifier-router` is a BFSI banking-chat figure — your dev-ops \u002F Slack-digest \u002F browser-automation distribution won't see the same savings.\n\nRecipe for a realistic estimate:\n\n1. Look at the vendor's cost-cut derivation: it's typically `(small-model-cost * %-small-traffic) + (large-model-cost * %-large-traffic) - routing-overhead`.\n2. Estimate your %-small-traffic by applying Gate 5's bucket distribution to the vendor's small\u002Flarge routing rules.\n3. Estimate routing overhead (typically 5-15ms per request).\n4. Compare against your **current** cost distribution (large model for everything → 100% large cost).\n5. Realistic savings on a non-matching workload: **5-20%**, not the vendor's headline.\n\nBe honest with the user about this. A 60% saving on the vendor's workload is a marketing number; a 10-20% saving on yours is the engineering reality. Do not adopt based on the vendor's figure.\n\n## Verdict shape (mandatory)\n\nEvery eval run ends with a verdict in this shape:\n\n```\n? \u002F ? \u002F ?  verdict on whether to adopt\n\n- Headline finding: \u003Cthe single most damning or interesting thing the eval turned up>\n- Gate-by-gate: 1-6 with pass\u002Ffail and one-line evidence\n- Realistic cost-cut for your workload: \u003Cnumber>%\n- Recommendation: adopt \u002F fork-and-patch \u002F do-not-adopt \u002F build-our-own\n- Proof: \u003Cpath to eval report, raw logits, log dump>\n```\n\nIf the verdict is \"do not adopt\" or \"fork-and-patch\", also include a one-line \"what would change my mind\" or \"what fork-and-patch requires\". The user wants to know the minimum delta that would make adoption viable.\n\n## Anti-patterns (do not do)\n\n- ❌ **\"Trust the README.\"** The README is the first source of truth, not the only one. Always triangulate with code + weights + behavior.\n- ❌ **\"Trust the headline accuracy.\"** Headline accuracy is measured on the vendor's test set. Your distribution is different. Gate 5 catches this.\n- ❌ **\"Skip Gate 3 because the README says it works.\"** Gate 3 is the silent killer. The 2026-07-22 incident proves a vendor can ship a README that documents the wrong number of classes while the shipped weights have the right (different) number.\n- ❌ **\"Classify the user's data and report accuracy.\"** Without Gate 5's label-coverage audit, the accuracy number is meaningless — you're measuring how well the model classifies into a taxonomy the user doesn't actually use.\n- ❌ **\"Recommend adoption if the architecture sounds right.\"** Architecture rightness ≠ workload rightness. The same router can be production-grade for BFSI and a bad fit for engineering dev work.\n- ❌ **\"Stall on a pre-execution menu.\"** When the user says \"evaluate this, replay my convos, are the choices right\", execute the verbs in sequence. See `finish-the-job` `references\u002Feval-stall-on-preflight-menu-2026-07-22.md`.\n\n## Support files\n\n- `scripts\u002Fcheck_checkpoint_class_count.py` — Gate 3 reproducer. Detects PyTorch \u002F HuggingFace classifier heads, walks the state_dict or config, and reports (or asserts, with `--expected N`) the class count. Catches the 2026-07-22 incident's class of bug in one command: `python3 scripts\u002Fcheck_checkpoint_class_count.py \u003Cmodel_path> --expected 4`. Exit 0 = pass, exit 1 = mismatch.\n\n## Cross-references\n\n- `finish-the-job` — the eval-stall anti-pattern (pre-execution menu) was patched there with a companion reference.\n- `verify-deployed-frontend-fix` — three-layer proof pattern (merged code + bundle hash + runtime computed style) is analogous: Gate 1 (README) + Gate 2 (code) + Gate 3 (weights) + Gates 4-5 (runtime behavior + coverage).\n- `cli-env-var-verification` — binary-string grep on compiled executable + official vendor docs + live process test + config-file presence. Same triangulation discipline, applied to env vars instead of model artifacts.\n- `convergent-bug-triage` — for when the eval surfaces multiple sibling issues (e.g. three separate failures all traceable to the same root cause like the class-count mismatch).\n- `pr-description-validator-gate6b` — same gate discipline (read-the-validator-locally, don't trust the headline PASS\u002FFAIL summary), applied to PR description validators instead of model artifacts.\n\n## Worked example — the 2026-07-22 incident\n\nUser asked: *\"clone this locally and evaluate it. Does it pass in the whole convo for classification? Can we replay it against some of my older convos\u002Fprojects to see what it would choose and if the choices right?\"* with a Medium link.\n\nPhase 0: Classified as vendor-ml-artifact-eval (no existing skill covered this class). Loaded this skill.\n\nGate 1 (README-vs-claim): Cloned `deycoding\u002Fdeycoding-compliance-classifier-router`. README documents `NUM_LABELS=4` (simple_no_pii \u002F simple_pii \u002F complex_no_pii \u002F complex_pii), `MAX_SEQ_LEN=128`, ~140M params, ~7ms GPU latency, 99.2% accuracy.\n\nGate 2 (code-level load path): `server\u002Fserver.py:18-19` shows `NUM_LABELS = 4; LABELS = {0: \"simple_no_pii\", 1: \"simple_pii\", 2: \"complex_no_pii\", 3: \"complex_pii\"}`. `MAX_SEQ_LEN = 128` at line 17. `model.load_state_dict(torch.load(MODEL_PATH, ...), weights_only=False)` at line 118 — note `weights_only=False` (security smell, but not the issue here).\n\nGate 3 (checkpoint-vs-claimed-class-count): Loaded the model weights from HuggingFace. The final classification layer `classifier.3.weight` has shape `torch.Size([6, 768])`. **MISMATCH.** The shipped code wants `NUM_LABELS=4` but the shipped weights have a 6-class head. `load_state_dict` will raise. **? Block.**\n\nGate 4 (per-turn architecture boundary): `MAX_SEQ_LEN=128` tokens. README example for `Design fraud detection system for UPI transactions` (8 tokens) fits; `Read my convo history to see what wasn't optimal...` (65 tokens) fits but is heavily truncated; a full conversation thread (2000-4000 words) does NOT fit. **The model cannot do whole-convo classification. Different product.**\n\nGate 5 (on-distribution label coverage): Mined 8 real past Jeffrey prompts via `session_search`. Buckets: `git\u002FCLI ops` (3), `AO orchestration` (2), `Slack digests` (1), `PR work` (1), `other` (1). Vendor's 4-class taxonomy (complexity × PII) has **zero direct overlap** with this distribution. Routing a `git push origin \u003Cbranch>` to \"small model\" because it has no PII is wrong.\n\nGate 6 (realistic cost-cut estimate): Vendor's 60% figure is BFSI-derived. Engineering dev workload (long-context code review, file diff analysis, multi-file refactors) overwhelmingly wants the long-context model — the small-model tier wouldn't even fit most past prompts in its 128-token context window. Realistic estimate for the user's actual distribution: **10-20%**, not 60%.\n\n**Verdict:** ? *workable compliance-routing primitive; **does not, and cannot, do whole-conversation classification** as the user asked; shipped code and shipped weights are out of sync; not ready for drop-in. Recommend fork + re-train on a task-type taxonomy matching the user's session distribution (~2-week AO dispatch).*\n\n**Proof:** Eval report at `~\u002Fprojects\u002Frouter-eval\u002FEVAL_REPORT.md` (16 KB). Raw logits at `\u002Ftmp\u002Frouter_eval_results.json`. Model weights + tokenizer at `~\u002Fprojects\u002Frouter-eval\u002Fdeycoding-compliance-classifier-router\u002Fserver\u002Fdeycoding.compliance-classifier-in-1-0.{pt,json}`. Reproducer recipe documented in the report.",{"name":5,"tags":58,"version":61,"category":62,"triggers":63,"changelog":79,"description":7,"related_skills":81},[25,24,38,34,43,59,33,60],"adoption","ml-artifact","0.1.0","software-development",[64,65,66,67,68,69,70,71,72,73,74,75,76,77,78],"evaluate this model","evaluate this router","evaluate this classifier","is this good for our use case","should we adopt this","replay our convos through this","does it pass for classification","check if the claims hold","vendor model eval","third-party router eval","run it on my past convos","what would it choose on my data","are the choices right","eval before adoption","pre-adoption check",[80],"0.1.0 (2026-07-22): Initial version. Recipe distilled from `deycoding\u002Fdeycoding-compliance-classifier-router` evaluation — the first end-to-end run of this checklist. Six gates + the headline finding pattern (NUM_LABELS=4 vs shipped-checkpoint head=6) that the eval surfaced. Companion to `finish-the-job` (the eval-stall anti-pattern was patched there). Verified end-state: 11 prompts run against the actual model, 441MB weights loaded from HuggingFace, eval report at `~\u002Fprojects\u002Frouter-eval\u002FEVAL_REPORT.md`.",[82,83,84,85,86,87,88,89],"finish-the-job","verify-deployed-frontend-fix","cli-env-var-verification","convergent-bug-triage","pr-description-validator-gate6b","patch-port-protocol","skillify","harness-engineering","main","05df7380981a2c4af868ce0fa57962ae0eea62a9","hermes\u002Fskills\u002Fvendor-ml-artifact-eval",0,"https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills","https:\u002F\u002Fgithub.com\u002Fjleechanorg","2026-11-14 03:18:13",{"source":8,"skillName":5,"commands":98,"prompt":108,"zipAvailable":109},[99,102,105],{"tool":100,"command":101},"npx","npx skills add jleechanorg\u002Fjleechan-skills --skill vendor-ml-artifact-eval",{"tool":103,"command":104},"bunx","bunx skills add jleechanorg\u002Fjleechan-skills --skill vendor-ml-artifact-eval",{"tool":106,"command":107},"pnpm","pnpm dlx skills add jleechanorg\u002Fjleechan-skills --skill vendor-ml-artifact-eval","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002Fhermes\u002Fskills\u002Fvendor-ml-artifact-eval\u002FSKILL.md",true,[111,117],{"path":112,"resourceType":113,"resourceTypeDesc":114,"sizeBytes":115,"sourceHtmlUrl":116},"agents\u002Fopenai.yaml","REFERENCE","参考文档",272,"https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fconfig_packs\u002Fcontent-creation\u002Fshared\u002Fskills\u002Fstrategic-compact\u002Fagents\u002Fopenai.yaml",{"path":118,"resourceType":119,"resourceTypeDesc":120,"sizeBytes":121,"sourceHtmlUrl":122},"scripts\u002Fcheck_checkpoint_class_count.py","SCRIPT","脚本",5678,"https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002Fhermes\u002Fskills\u002Fvendor-ml-artifact-eval\u002Fscripts\u002Fcheck_checkpoint_class_count.py",[124,142,164,178,192,208],{"skillNo":125,"name":126,"nameZh":6,"description":127,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":128,"categories":129,"keywords":130,"publishedAt":53},"01M716Y5MTJ6GYJZ2ATXCJK8G4","instinct","Use when the user invokes \u002Finstinct or \u002Fin, or asks to \"message Instinct\", \"ask Instinct\", \"tell Instinct\", \"check Instinct\", or read replies from the configured peer assistant. iMessage first, WhatsApp Web as fallback.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Finstinct\u002FSKILL.md",[],[126,131,132,133,134,135,136,137,138,139,140,141],"user","when","replies","check","use","identity","whatsapp","gmail","read","message","ask",{"skillNo":143,"name":144,"nameZh":6,"description":145,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":146,"categories":147,"keywords":153,"publishedAt":53},"01M716Y4HF7FVVMJSA2CCVN5VF","hr","Compatibility alias for history-resume; produce the same fresh-state, provenance-aware resume briefing.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fhr\u002FSKILL.md",[148],{"categoryNo":149,"categoryName":150,"taxonomyType":151,"taxonomyTypeDesc":152},"01JBSEEDCAT000000000000038","人力资源","PROFESSION","适用职业",[144,154,155,156,157,158,159,160,161,162,163],"resume","compatibility","provenance","for","history","produce","alias","briefing","state","fresh",{"skillNo":165,"name":166,"nameZh":6,"description":167,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":168,"categories":169,"keywords":171,"publishedAt":53},"01M716Y1FFNFKFJYYKF8HQA6T8","history-resume","Resume briefing for the current repository branch or worktree using fresh state and admitted history sources.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fhistory-resume\u002FSKILL.md",[170],{"categoryNo":149,"categoryName":150,"taxonomyType":151,"taxonomyTypeDesc":152},[158,154,172,173,131,174,161,175,157,176,177,162,163],"history resume","branch","analysis","repository","worktree","sources",{"skillNo":179,"name":180,"nameZh":6,"description":181,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":182,"categories":183,"keywords":184,"publishedAt":53},"01M716XWXJ8EPHK4RH9T14TR85","grok","Alias for the canonical grokbot browser conversation workflow; \u002Fgrok talks to the Grok bot the same way \u002Fdot talks to Dot.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fgrok\u002FSKILL.md",[],[180,185,186,187,157,188,160,189,190,191],"dot","bot","workflow","canonical","conversation","browser","grokbot",{"skillNo":193,"name":194,"nameZh":6,"description":195,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":196,"categories":197,"keywords":198,"publishedAt":53},"01M70N9Q07DTE0EW5KFXMMMSJ0","codexa-bench","Compare `codexa` and normal `codex` on a fair split of comparable work by default; use `--strict` for duplicate runs of the exact same task.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fcodexa-bench\u002FSKILL.md",[],[199,200,201,135,202,157,203,204,205,206,207],"codexa bench","work","split","default","duplicate","task","compare","normal","codex",{"skillNo":209,"name":210,"nameZh":6,"description":211,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":212,"licenseFileUrl":213,"sourceHtmlUrl":214,"categories":215,"keywords":223,"publishedAt":53},"01M6C24S2X60SP8BBFGAEKPEGW","frontend-design","Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, artifacts, posters, or applications (examples include websites, landing pages, dashboards, React components, HTML\u002FCSS layouts, or when styling\u002Fbeautifying any web UI). Generates creative, polished code and UI design that avoids generic AI aesthetics.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ffrontend-design\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ffrontend-design\u002FSKILL.md",[216,217,220],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},{"categoryNo":218,"categoryName":219,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000015","视觉设计",{"categoryNo":221,"categoryName":222,"taxonomyType":151,"taxonomyTypeDesc":152},"01JBSEEDCAT000000000000032","设计师",[40,224,225,135,226,227,228,131,229,230,48,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,132,246],"design","frontend design","generic","ui design","react","html","components","artifacts","ui","aesthetics","Production","web components","landing","applications","quality","web","css","create","examples","with","react components","web ui","pages"]