[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7CG2E43WNJJ9B2Z82F1Y0RH":3,"skill-same:01M7CG2E43WNJJ9B2Z82F1Y0RH":47},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":6,"ownerHtmlUrl":29,"ownerBlogUrl":30,"ownerDescription":31,"updatedAt":32,"install":33,"resources":46},"01M7CG2E43WNJJ9B2Z82F1Y0RH","benchmarking-methodology",null,"Design and run a reproducible benchmark comparing systems, models, or configurations. Use this Skill whenever the user asks for a benchmark, an eval suite, a performance comparison, an A\u002FB of two models or prompts, or a scored task run — it defines the pre-registration step, the run matrix, workflow-driven execution, scoring, and honest statistical reporting.","ibrahimsaleem\u002Fmetaharnessfactory","ibrahimsaleem","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F27896839?v=4","01M7AABSBMTHXGZH3XCN8AZNKS","Mohammad Ibrahim Saleem","01M7CG0H9FAQKW0KK2V8ERBXJ4",1,"MIT","https:\u002F\u002Fgithub.com\u002Fibrahimsaleem\u002Fmetaharnessfactory\u002Fblob\u002Fmain\u002Fapps\u002Fcli\u002Fconfig\u002Fagent-presets\u002Fharness-bench\u002Fskills\u002Fbenchmarking-methodology\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: benchmarking-methodology\ndescription: Design and run a reproducible benchmark comparing systems, models, or configurations. Use this Skill whenever the user asks for a benchmark, an eval suite, a performance comparison, an A\u002FB of two models or prompts, or a scored task run — it defines the pre-registration step, the run matrix, workflow-driven execution, scoring, and honest statistical reporting.\n---\n\n# Benchmarking Methodology\n\nA benchmark is a claim about the world backed by artifacts. If a number in the report cannot be\ntraced to a run on disk, it does not belong in the report.\n\nFollow the phases in order. Phase 1 happens **before** any run executes — that ordering is the\nwhole point.\n\n## Phase 0 — Question\n\nPin down what is actually being asked:\n\n- **What is under test?** Models, prompts, configurations, code versions, or whole systems.\n- **What is the comparison?** Against each other, against a baseline, or against a fixed threshold.\n- **What decision does this inform?** A benchmark built for \"which is faster\" looks different from\n  one built for \"is this good enough to ship\".\n- **What does \"better\" mean here?** Accuracy, latency, cost, token efficiency, pass rate, or a\n  weighted combination the user must choose — not one you invent.\n\nOpen a goal (`goal` tool) and record the question. Use `ask_user_question` for the metric weighting\nif the user has not specified it; that is a genuine user-owned decision, not something to infer.\n\n## Phase 1 — Pre-registration (do this before running anything)\n\nWrite a file that fixes, in advance:\n\n- **The task suite**: every task, with its input and its success criterion. A task whose criterion\n  is decided after seeing the output is not a task, it is a rationalization.\n- **The systems under test**, with exact versions and full configuration.\n- **Controlled conditions**: identical prompts, identical context, identical limits (timeout, max\n  tokens, retries) for every system. Any intentional difference is a named variable.\n- **Sample size**: runs per task per system. Stochastic systems need repetition — 1 run is an\n  anecdote. Justify the number.\n- **The metrics and the scoring rubric**, including how partial credit works.\n- **What counts as a failure**: timeout, error, malformed output, refusal. Each gets a category.\n\nThis file is the contract. If you change it mid-benchmark, the change and its reason go in the\nreport — you do not silently re-register.\n\n## Phase 2 — Harness construction\n\nBuild the execution scaffold before the real run:\n\n1. Write the runner so a single `(system, task, run-index)` triple executes end to end and writes\n   one artifact: input sent, raw output, latency, token counts, cost, exit status.\n2. **Smoke-run the matrix at N=1** across every system and a couple of tasks. Confirm artifacts\n   land, scoring parses them, and no system is misconfigured. A benchmark that ran for an hour on a\n   broken adapter has produced nothing.\n3. Fix the artifact schema now. Post-hoc schema changes invalidate earlier runs.\n\n## Phase 3 — Execution (workflow-driven)\n\nUse `workflow` as the primary driver: a benchmark matrix is naturally one deterministic script that\nfans runs across systems and tasks, which is exactly what workflows are for. Within it, each\nindividual run can be a `subagent` when the task itself requires agentic work.\n\nRules during execution:\n\n- Long batches run as background jobs (`run_in_background`) so the suite keeps moving.\n- **Never tune a system mid-run.** If you discover a misconfiguration, stop, fix it, and re-run that\n  system's full matrix. Reporting a mixed set of conditions as one result is the most common way a\n  benchmark becomes false.\n- Record every run, including failures, timeouts, and errors. Dropping bad runs inflates every\n  score that survives.\n- Checkpoint progress to disk with `todo_write` and artifact files; a long matrix outlives your\n  context window.\n\n## Phase 4 — Scoring\n\nScore from artifacts, never from memory of what you saw.\n\n- **Deterministic criteria** (exact match, test passes, schema valid): score programmatically.\n- **Judgement criteria**: dispatch `subagent_fork` graders that inherit the rubric. Each grade\n  carries `task-id`, `system`, `run`, `score`, and the reasoning.\n- **Measure grader reliability**: re-score a random sample with a second grader and report the\n  agreement rate. Low agreement means the rubric is weak — report that rather than hiding it.\n- Blind the grader to which system produced an output wherever it is practical to do so.\n\n## Phase 5 — Analysis\n\n- Report **mean and spread**, not just the mean. A system that is better on average and wildly\n  inconsistent is a different product from one that is steadily slightly worse.\n- State sample size beside every number.\n- **Do not rank differences inside the noise.** If the gap is smaller than the run-to-run variance,\n  the honest finding is \"no measurable difference at this sample size\".\n- Break results down by task category — an aggregate score usually hides the interesting structure.\n- Report failure modes separately: timeouts and errors are results, and a system that is fastest\n  when it works and fails a fifth of the time should not look fastest.\n\n## Phase 6 — Report\n\nWrite to a file in the workspace:\n\n1. **Question** — what was asked, what decision it informs.\n2. **Pre-registration** — the suite, conditions, sample size and metrics as fixed in Phase 1, plus\n   any mid-run change and why.\n3. **Results** — tables with sample sizes and spread. Per-category breakdown.\n4. **Failures** — counts by category, per system.\n5. **Findings** — what the data supports, stated no more strongly than the data allows.\n6. **Limitations** — sample size, conditions not controlled, tasks not covered, version drift.\n7. **Reproduction** — the exact commands and artifact locations to re-run and re-score.\n\nIf the result is inconclusive, the report says it is inconclusive. That is a successful benchmark.\n\n## Working rules\n\n- `workflow` first for the matrix; `subagent` for an individual agentic run; `subagent_fork` for\n  rubric graders; `ralph` for bounded repeat-until-stable sampling.\n- Every number traces to an artifact path. Include the path in the results table.\n- Web search is available for published benchmark definitions and baseline numbers; fetch is off.\n","# Benchmarking Methodology\n\nA benchmark is a claim about the world backed by artifacts. If a number in the report cannot be\ntraced to a run on disk, it does not belong in the report.\n\nFollow the phases in order. Phase 1 happens **before** any run executes — that ordering is the\nwhole point.\n\n## Phase 0 — Question\n\nPin down what is actually being asked:\n\n- **What is under test?** Models, prompts, configurations, code versions, or whole systems.\n- **What is the comparison?** Against each other, against a baseline, or against a fixed threshold.\n- **What decision does this inform?** A benchmark built for \"which is faster\" looks different from\n  one built for \"is this good enough to ship\".\n- **What does \"better\" mean here?** Accuracy, latency, cost, token efficiency, pass rate, or a\n  weighted combination the user must choose — not one you invent.\n\nOpen a goal (`goal` tool) and record the question. Use `ask_user_question` for the metric weighting\nif the user has not specified it; that is a genuine user-owned decision, not something to infer.\n\n## Phase 1 — Pre-registration (do this before running anything)\n\nWrite a file that fixes, in advance:\n\n- **The task suite**: every task, with its input and its success criterion. A task whose criterion\n  is decided after seeing the output is not a task, it is a rationalization.\n- **The systems under test**, with exact versions and full configuration.\n- **Controlled conditions**: identical prompts, identical context, identical limits (timeout, max\n  tokens, retries) for every system. Any intentional difference is a named variable.\n- **Sample size**: runs per task per system. Stochastic systems need repetition — 1 run is an\n  anecdote. Justify the number.\n- **The metrics and the scoring rubric**, including how partial credit works.\n- **What counts as a failure**: timeout, error, malformed output, refusal. Each gets a category.\n\nThis file is the contract. If you change it mid-benchmark, the change and its reason go in the\nreport — you do not silently re-register.\n\n## Phase 2 — Harness construction\n\nBuild the execution scaffold before the real run:\n\n1. Write the runner so a single `(system, task, run-index)` triple executes end to end and writes\n   one artifact: input sent, raw output, latency, token counts, cost, exit status.\n2. **Smoke-run the matrix at N=1** across every system and a couple of tasks. Confirm artifacts\n   land, scoring parses them, and no system is misconfigured. A benchmark that ran for an hour on a\n   broken adapter has produced nothing.\n3. Fix the artifact schema now. Post-hoc schema changes invalidate earlier runs.\n\n## Phase 3 — Execution (workflow-driven)\n\nUse `workflow` as the primary driver: a benchmark matrix is naturally one deterministic script that\nfans runs across systems and tasks, which is exactly what workflows are for. Within it, each\nindividual run can be a `subagent` when the task itself requires agentic work.\n\nRules during execution:\n\n- Long batches run as background jobs (`run_in_background`) so the suite keeps moving.\n- **Never tune a system mid-run.** If you discover a misconfiguration, stop, fix it, and re-run that\n  system's full matrix. Reporting a mixed set of conditions as one result is the most common way a\n  benchmark becomes false.\n- Record every run, including failures, timeouts, and errors. Dropping bad runs inflates every\n  score that survives.\n- Checkpoint progress to disk with `todo_write` and artifact files; a long matrix outlives your\n  context window.\n\n## Phase 4 — Scoring\n\nScore from artifacts, never from memory of what you saw.\n\n- **Deterministic criteria** (exact match, test passes, schema valid): score programmatically.\n- **Judgement criteria**: dispatch `subagent_fork` graders that inherit the rubric. Each grade\n  carries `task-id`, `system`, `run`, `score`, and the reasoning.\n- **Measure grader reliability**: re-score a random sample with a second grader and report the\n  agreement rate. Low agreement means the rubric is weak — report that rather than hiding it.\n- Blind the grader to which system produced an output wherever it is practical to do so.\n\n## Phase 5 — Analysis\n\n- Report **mean and spread**, not just the mean. A system that is better on average and wildly\n  inconsistent is a different product from one that is steadily slightly worse.\n- State sample size beside every number.\n- **Do not rank differences inside the noise.** If the gap is smaller than the run-to-run variance,\n  the honest finding is \"no measurable difference at this sample size\".\n- Break results down by task category — an aggregate score usually hides the interesting structure.\n- Report failure modes separately: timeouts and errors are results, and a system that is fastest\n  when it works and fails a fifth of the time should not look fastest.\n\n## Phase 6 — Report\n\nWrite to a file in the workspace:\n\n1. **Question** — what was asked, what decision it informs.\n2. **Pre-registration** — the suite, conditions, sample size and metrics as fixed in Phase 1, plus\n   any mid-run change and why.\n3. **Results** — tables with sample sizes and spread. Per-category breakdown.\n4. **Failures** — counts by category, per system.\n5. **Findings** — what the data supports, stated no more strongly than the data allows.\n6. **Limitations** — sample size, conditions not controlled, tasks not covered, version drift.\n7. **Reproduction** — the exact commands and artifact locations to re-run and re-score.\n\nIf the result is inconclusive, the report says it is inconclusive. That is a successful benchmark.\n\n## Working rules\n\n- `workflow` first for the matrix; `subagent` for an individual agentic run; `subagent_fork` for\n  rubric graders; `ralph` for bounded repeat-until-stable sampling.\n- Every number traces to an artifact path. Include the path in the results table.\n- Web search is available for published benchmark definitions and baseline numbers; fetch is off.\n",{"name":5,"description":7},"main","a933d10855d84d7df34f5fcdf6bb5c032328f027","apps\u002Fcli\u002Fconfig\u002Fagent-presets\u002Fharness-bench\u002Fskills\u002Fbenchmarking-methodology",0,"https:\u002F\u002Fgithub.com\u002Fibrahimsaleem\u002Fmetaharnessfactory","https:\u002F\u002Fgithub.com\u002Fibrahimsaleem","https:\u002F\u002Fibrahimsaleem-portfolio.web.app\u002F","Hi Everyone, I am Mohammad Ibrahim Saleem from Houston, USA.\r\nAI Security Engineer @AT&T and Master's in Cybersecurity from University of Houston.","2026-09-12 14:43:38",{"source":8,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add ibrahimsaleem\u002Fmetaharnessfactory --skill benchmarking-methodology",{"tool":39,"command":40},"bunx","bunx skills add ibrahimsaleem\u002Fmetaharnessfactory --skill benchmarking-methodology",{"tool":42,"command":43},"pnpm","pnpm dlx skills add ibrahimsaleem\u002Fmetaharnessfactory --skill benchmarking-methodology","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fibrahimsaleem\u002Fmetaharnessfactory\u002Fblob\u002Fmain\u002Fapps\u002Fcli\u002Fconfig\u002Fagent-presets\u002Fharness-bench\u002Fskills\u002Fbenchmarking-methodology\u002FSKILL.md",true,[],[48,55,62,69,76,83],{"skillNo":49,"name":50,"nameZh":6,"description":51,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":52,"categories":53,"keywords":54,"publishedAt":19},"01M7CG35SWBJEDBREJQJJ5VTH6","user-only-skill","Prove model-disabled skills stay outside the model catalog.","https:\u002F\u002Fgithub.com\u002Fibrahimsaleem\u002Fmetaharnessfactory\u002Fblob\u002Fmain\u002Fexamples\u002Facp-agent\u002Ftests\u002Fsnapshots\u002Fskill-load\u002Fworkspace\u002F.dsh\u002Fskills\u002Fuser-only-skill\u002FSKILL.md",[],[],{"skillNo":56,"name":57,"nameZh":6,"description":58,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":59,"categories":60,"keywords":61,"publishedAt":19},"01M7CG33W5820XY0VWDA6YV7YT","snapshot-skill","Exercise project skill discovery and loading in snapshot tests.","https:\u002F\u002Fgithub.com\u002Fibrahimsaleem\u002Fmetaharnessfactory\u002Fblob\u002Fmain\u002Fexamples\u002Facp-agent\u002Ftests\u002Fsnapshots\u002Fskill-load\u002Fworkspace\u002F.dsh\u002Fskills\u002Fsnapshot-skill\u002FSKILL.md",[],[],{"skillNo":63,"name":64,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":68,"publishedAt":19},"01M7CG32C1XKY66SWA5A27N7Y7","model-only-skill","Prove user-disabled skills remain available to the model.","https:\u002F\u002Fgithub.com\u002Fibrahimsaleem\u002Fmetaharnessfactory\u002Fblob\u002Fmain\u002Fexamples\u002Facp-agent\u002Ftests\u002Fsnapshots\u002Fskill-load\u002Fworkspace\u002F.dsh\u002Fskills\u002Fmodel-only-skill\u002FSKILL.md",[],[],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":19},"01M7CG2X5YR4YH40PT029S6706","security-review-methodology","Run a complete, evidence-driven security review of a code repository. Use this Skill whenever the user asks for a security review, vulnerability scan, threat assessment, or penetration-test-style analysis of a codebase — it defines the phase order, the subagent fan-out, the judge triage that removes false positives, and the report format.","https:\u002F\u002Fgithub.com\u002Fibrahimsaleem\u002Fmetaharnessfactory\u002Fblob\u002Fmain\u002Fapps\u002Fcli\u002Fconfig\u002Fagent-presets\u002Fharness-security\u002Fskills\u002Fsecurity-review-methodology\u002FSKILL.md",[],[],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":19},"01M7CG2RCVC0HSM17Z2FN03ASM","code-review-methodology","Run a thorough, evidence-driven code review that finds real bugs. Use this Skill whenever the user asks for a code review, a bug hunt, a regression check, a PR review, or an assessment of test coverage — it defines the phase order, the subagent fan-out, the adversarial verification that removes false positives, and the report format.","https:\u002F\u002Fgithub.com\u002Fibrahimsaleem\u002Fmetaharnessfactory\u002Fblob\u002Fmain\u002Fapps\u002Fcli\u002Fconfig\u002Fagent-presets\u002Fharness-review\u002Fskills\u002Fcode-review-methodology\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":19},"01M7CG2K5SH7TGFFS3G7N972H8","redteam-methodology","Run a structured, contained adversarial evaluation of a target model or AI system. Use this Skill whenever the user asks for red-teaming, safety testing, jailbreak probing, refusal-consistency measurement, or model-behaviour evaluation — it defines the scope gate, probe-suite design, the subagent fan-out, grading, and the report format.","https:\u002F\u002Fgithub.com\u002Fibrahimsaleem\u002Fmetaharnessfactory\u002Fblob\u002Fmain\u002Fapps\u002Fcli\u002Fconfig\u002Fagent-presets\u002Fharness-redteam\u002Fskills\u002Fredteam-methodology\u002FSKILL.md",[],[]]