eval-harness-current

Objective agent evaluation

Define the decision the experiment should support: a skill's benefit over the same agent without it, a regression between revisions, or a model comparison. Changing the model and skill simultaneously cannot isolate either effect.

Define the experiment

• Choose tasks from real pinned issues, workflows, or independently authored user requirements. Record source provenance, environment, and task coverage.
• Define graders before reading the exact candidate instructions. Score actual behavior with executable acceptance checks, including relevant regressions. A function name, code pattern, plausible explanation, or agent self-report is not proof of correctness. Use blinded review only for criteria that need it.
• Keep development cases separate from final holdouts. Any case used to tune instructions becomes a development case, even if originally called a holdout.
• Run the original input through the grader to prove the task's failing behavior; check a known valid outcome and plausible wrong outcomes to test the grader. Keep oracle artifacts outside the agent's accessible task environment.

Execute matched trials

Use fresh workspaces and sessions with the same task input, tool permissions, model version, reasoning effort, limits, and dependency versions. Declare which instructions each condition loads; record effective configuration and instruction hashes. Development history, memory, prior patches, and unrelated skills can contaminate a baseline. If isolation cannot be verified, label that limitation.

Rotate condition order across repeated tasks. Preserve each attempt's transcript, patch, grader output, exit status, actual usage, and elapsed time. Record failures, timeouts, environment errors, and unavailable measurements rather than deleting them or substituting zero. Validate inputs and grader files against their frozen hashes after execution, and detect changes outside the permitted patch scope.

Analyze and iterate

Report the number of planned, attempted, validly graded, successful, and failed trials for each condition. Distinguish task failure from harness failure. Show paired task results and uncertainty before claiming superiority; disclose category coverage, exclusions, repeat count, and any contaminated tasks.

For each task, let n be the number of recorded attempts and c the number that passed a complete grader. With no unscored attempts and 1 <= k <= n, the fraction of uniformly chosen k-subsets with any success is 1 - C(n-c, k) / C(n, k); the fraction where all succeed is C(c, k) / C(n, k), taking combinations with fewer than k items as zero. Here n counts agent attempts, not individual tests within one attempt. Compute per task first, then average tasks equally unless different task weights were declared in advance. Do not pool attempts across tasks to calculate these scores.

For example, if one task has two passes in two attempts and another has no passes in eight, equal-task pass@1 is (1 + 0) / 2 = 1/2, not the pooled attempt rate 2/10. When n < k, the task's k-result is unavailable. Keep unscored attempts visible; report incomplete outcomes or explicit best/worst assignment bounds, not an unconditional pass rate based only on scored runs.

The subset fractions describe the recorded attempts. Interpreting them as future-run probabilities requires suitable independent, identically distributed sampling and stable conditions; adaptive retries need their own analysis. An any-of-k score credits a task if a passing output exists; using that success requires a feasible selection method. Account for the cost of k attempts and selection separately. The score is not an all-runs reliability measure. All-of-k describes consistency under the stated setup, not a worst-case guarantee. Do not relabel retry success or a selected best attempt as pass@1. See the original pass@k estimator and the agent evaluation discussion.

Compare quality first, then tokens, latency, tool use, and cost for comparable outcomes. Do not infer speed gains from host-load noise or efficiency from shorter answers alone. Keep package readiness and harness smoke checks separate from agent task performance.

Revise against demonstrated development failures, rerun affected matched cases and regressions, then evaluate untouched holdouts. Preserve negative results and version each iteration. If measured gains are absent, report that outcome.