AGENT SKILL

ac-model-evaluation

Building evaluation suites that decide between models for a task: what the eval measures before any prompt is written, constructing cases from real failure modes and coding them from task-native artifacts, contamination and why held-out data no longer guarantees freshness, automatic scoring with exact-match and judge limits, pass-at-k and aggregate metrics read honestly, human comparison and preference data as a separate instrument, and the written evaluation record. It designs the measurement; it never certifies a model as good and never recommends a vendor. Use when asking "model evaluation", "eval suite design", "build a benchmark", "contamination", "held-out set", "LLM as judge", "pass at k", "model comparison", "regression suite for model upgrades", "golden set", "cost of evaluation". Not for improving a prompt against fixed cases (ac-prompt-engineer), not for running the harness code itself (ac-llm-api-integration), not for evaluating datasets as research evidence (ac-alternative-data).

License 许可
MIT
最近更新
2026-11-15
Stars
★ 81
Forks
17

01 前言 Front Matter

name
ac-model-evaluation
status
seedling
license
MIT
metadata
{ "domain": "prompt-engineering" }
description
Building evaluation suites that decide between models for a task: what the eval measures before any prompt is written, constructing cases from real failure modes and coding them from task-native artifacts, contamination and why held-out data no longer guarantees freshness, automatic scoring with exact-match and judge limits, pass-at-k and aggregate metrics read honestly, human comparison and preference data as a separate instrument, and the written evaluation record. It designs the measurement; it never certifies a model as good and never recommends a vendor. Use when asking "model evaluation", "eval suite design", "build a benchmark", "contamination", "held-out set", "LLM as judge", "pass at k", "model comparison", "regression suite for model upgrades", "golden set", "cost of evaluation". Not for improving a prompt against fixed cases (ac-prompt-engineer), not for running the harness code itself (ac-llm-api-integration), not for evaluating datasets as research evidence (ac-alternative-data).

02 资源文件 11

03 SKILL.md 正文

Model evaluation

Overview

Choosing a model for a task is a measurement problem, and the measurement decides more than the choice of models does: an evaluation built from convenient examples measures convenience, one built from real failure modes measures the task, and the two rankings they produce are rarely the same. An eval suite is a small, task-shaped body of evidence — cases with known answers or graded criteria — that turns "model A feels better" into a number, and the number is only as good as its case design.

This skill covers building that instrument: deciding what the eval must measure, constructing cases from the task's real failure surface rather than from invented ease, the contamination problem that quietly erodes benchmarks built from public data, scoring tiers from deterministic checks through model judges to humans, reading aggregate metrics like pass-at-k without fooling yourself, and recording the whole thing so a later model choice can be re-decided on new evidence. Improving a prompt against a fixed set is a different skill; this one is about the set.

Primary sources: https://docs.anthropic.com/en/docs/about-claude/evaluate and https://github.com/EleutherAI/lm-evaluation-harness.

When to use

  • When choosing between models or model versions for a specific task.
  • When building a regression suite to guard model upgrades.
  • When a benchmark's public score is being mapped to your task.
  • When deciding how much evaluation a decision deserves.

When NOT to use

  • For improving a prompt against fixed cases (use ac-prompt-engineer).
  • For the harness and API mechanics of running evals (use ac-llm-api-integration).
  • For evaluating datasets as research evidence (use ac-alternative-data).

Subskill Index

Leaf Domain segment Scope (one line)
subskills/what-the-eval-measures.md prompt-engineering/what-the-eval-measures Deciding the question and decision the eval serves.
subskills/case-construction.md prompt-engineering/case-construction Building cases from real failure modes, not ease.
subskills/contamination.md prompt-engineering/contamination Why held-out data no longer guarantees a fresh test.
subskills/scoring-and-judges.md prompt-engineering/scoring-and-judges Deterministic checks first, judges second, humans for the residue.
subskills/metrics-and-reading.md prompt-engineering/eval-metrics-and-reading Pass-at-k, aggregate scores, and honest reading.
subskills/human-comparison.md prompt-engineering/human-comparison Preference data and side-by-side as a separate instrument.
subskills/eval-economics.md prompt-engineering/eval-economics Cost, cadence, and how much eval a decision needs.
subskills/evaluation-record.md prompt-engineering/evaluation-record The written record: cases, scoring, results, and provenance.

Status and honest limits

An eval measures the cases it contains, and nothing else: a model can be excellent on your suite and poor on the population of requests you actually receive, if the suite is not drawn from that population. Public benchmarks add a second limit — the score on a contaminated benchmark measures recall of training data as much as capability. Treat every score as evidence about the cases measured, and re-decide when the task shifts. Nothing here certifies a model; it builds the instrument a decider can trust or rightly distrust.

04 同仓库更多 Skills

jiayaoqijia
jiayaoqijia/cryptoskill

troves

Deposit and withdraw from Troves DeFi yield vault strategies on Starknet. View available strategies, check positions, and manage vault deposits. Use this skill when the user mentions Troves, vault, yield vault, DeFi vault, strategy, yield farming, vault deposit, vault withdraw, earning yield through vaults, passive yield strategies, or wants to deposit into or withdraw from a yield strategy — even if they don't say "Troves" explicitly.

★ 81
jiayaoqijia
jiayaoqijia/cryptoskill

trade

Swap tokens on Starknet via Fibrous DEX aggregator. Supports per-provider selection (avnu, ekubo, or auto to race all). Use this skill when the user wants to swap, exchange, trade, convert, buy, or sell one token for another on Starknet. Also use when the user asks about token prices, exchange rates, DEX routing, best swap route, or wants to check how much they would receive for a given amount — even if they don't explicitly say "swap".

★ 81
jiayaoqijia
jiayaoqijia/cryptoskill

staking

Stake, unstake, claim, and compound rewards for STRK, WBTC, tBTC, SolvBTC, and LBTC on Starknet. View validators, pools, and staking positions. Use this skill when the user mentions staking, delegating, validators, staking pools, rewards, compounding, unstaking, earning yield, passive income on crypto, or wants to earn on their STRK or BTC tokens — even if they don't say "stake" explicitly.

★ 81
jiayaoqijia
jiayaoqijia/cryptoskill

send

Send STRK, ETH, or any ERC-20 token to a Starknet address. Supports simulation (dry-run) before execution. Use this skill when the user wants to send, transfer, pay, tip, or move tokens to another wallet address on Starknet. Also trigger when the user mentions sending funds to someone, making a payment, transferring assets, or moving crypto to a specific address — even if they just say "pay 10 STRK to 0x...".

★ 81
jiayaoqijia
jiayaoqijia/cryptoskill

portfolio

View a comprehensive DeFi portfolio dashboard and optimize portfolio allocation via automated rebalancing. Use this skill when the user wants an overview, summary, dashboard, total value, net worth, portfolio rebalancing, target allocation, or wants to see all their DeFi positions across Starknet at once. Also trigger when the user asks "what do I have", "show me everything", "how much am I worth", "my positions", "my investments", "rebalance my portfolio", "I want 50% ETH", "optimize my holdings", or any request for a holistic view of their Starknet holdings — even if they don't explicitly say "portfolio".

★ 81
jiayaoqijia
jiayaoqijia/cryptoskill

multi-swap

Execute multiple token swaps in a single Starknet transaction via Fibrous (default). Supports per-provider selection (avnu, ekubo, or auto to race all). Supports 2-3 swap pairs bundled into one multicall. Use this skill when the user wants to swap multiple token pairs at once, do batch swaps, execute several trades simultaneously, perform parallel swaps, or do bulk trading in one transaction. Also trigger when the user mentions "two swaps", "three swaps", "swap X and Y at the same time", or wants to convert multiple tokens in a single call.

★ 81