plugin-evaluation
Use when designing systematic evaluation methodology for Codex agents and commands — constructing weighted rubrics, choosing between direct scoring and pairwise comparison, building LLM-as-judge pipelines, or mitigating position, verbosity, and self-enhancement bias. NOT for empirical benchmarks with timing/token metrics (use skill-tester) or for iterating on skill content during development (use skill-development).
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-15
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- plugin-evaluation
- description
- Use when designing systematic evaluation methodology for Codex agents and commands — constructing weighted rubrics, choosing between direct scoring and pairwise comparison, building LLM-as-judge pipelines, or mitigating position, verbosity, and self-enhancement bias. NOT for empirical benchmarks with timing/token metrics (use skill-tester) or for iterating on skill content during development (use skill-development).
- allowed-tools
- Read Glob Grep
02 资源文件 3
- 参考文档references/bias-mitigation.md8.6 KB
- 参考文档references/llm-judge-patterns.md24.2 KB
- 参考文档references/metric-selection.md6.0 KB
03 SKILL.md 正文
Evaluation Methods for Codex Agents
Evaluation of agent systems requires outcome-focused approaches that account for non-determinism and multiple valid paths. Agents may reach goals through different routes — evaluation should judge whether they achieve right outcomes through reasonable processes, not match exact execution steps.
Quick Start
- Identify what to evaluate — a command's outputs, an agent's behavior, or a prompt variant
- Design test cases — cover at least two complexity levels: simple (single tool call) and complex (multi-step reasoning)
- Choose method — LLM-as-judge (scale + speed), human evaluation (catches edge cases), or end-state (artifact correctness)
- Apply rubric — instruction following (0.30), completeness (0.25), tool efficiency (0.20), reasoning (0.15), coherence (0.10)
- Iterate — compare before/after scores on the same test set; check no dimension regressed
When to Use
- Testing whether a command or skill produces consistently good outputs
- Validating that context engineering changes improve (not regress) quality
- Scoring agent outputs systematically with LLM-as-judge
- Comparing prompt variants objectively before choosing one
- Detecting or mitigating position bias, verbosity bias, or self-enhancement bias
When NOT to Use
- Debugging why a single command invocation failed — use
Codex --debugand troubleshooting instead - For one-off gut-check tests — evaluation is for systematic, repeatable quality measurement
- For evaluating plugin structure or manifest correctness — use
plugin-developmentorplugin-validatorinstead
Core Concepts
Agent evaluation captures multiple quality dimensions: factual accuracy, completeness, citation accuracy, source quality, and tool efficiency. LLM-as-judge provides scalable evaluation while human evaluation catches edge cases.
Performance Drivers: The 95% Finding
Research on the BrowseComp evaluation found three factors explain 95% of performance variance:
| Factor | Variance Explained | Implication |
|---|---|---|
| Token usage | 80% | More tokens = better performance |
| Number of tool calls | ~10% | More exploration helps |
| Model choice | ~5% | Better models multiply efficiency |
Evaluate with realistic token constraints. Model upgrades beat token increases. Multi-agent architectures distribute work across separate context windows.
Evaluation Challenges
Non-Determinism and Multiple Valid Paths
Agents may take completely different valid paths to the same goal. Traditional step-by-step evaluation fails.
Solution: Evaluate outcomes, not execution paths. Judge whether the agent achieves the right result through a reasonable process.
Context-Dependent Failures
Failures often emerge after extended interaction or at specific complexity levels.
Solution: Cover a range of complexity levels. Test extended interactions, not just isolated queries.
Composite Quality Dimensions
An agent might score high on accuracy but low in efficiency, or vice versa.
Solution: Multi-dimensional rubrics with appropriate weighting per use case.
Evaluation Rubric Design
Default weights: instruction following (0.30), completeness (0.25), tool efficiency (0.20), reasoning (0.15), coherence (0.10). Convert assessments to 0.0–1.0 with weighting. Typical passing thresholds: 0.7 for general use, 0.85 for critical operations.
For per-criterion score bands and rubric generation patterns, see references/llm-judge-patterns.md (Pattern 1, Step 1).
Evaluation Methodologies
LLM-as-Judge
Scales to large test sets and provides consistent judgments. Design evaluation prompts that capture the dimensions of interest.
For the full evaluation prompt template, see references/llm-judge-patterns.md (Pattern 1, Step 3).
Chain-of-Thought Requirement: Always require justification before the score — improves reliability by 15-25%.
Human Evaluation
Human evaluation catches what automation misses: hallucinated answers on unusual queries, subtle context misunderstandings, edge cases, qualitative issues with tone or approach. Sample systematically across complexity levels.
End-State Evaluation
For commands that produce artifacts, evaluate the final output rather than the process: does the code work? Is the configuration valid? Does the output meet requirements?
Test Set Design
Sample Selection: Start small during development — early changes have large impact. Sample from real usage patterns. Add known edge cases. Ensure coverage across complexity levels.
Complexity Stratification: simple (single tool call), medium (multiple tool calls), complex (many tool calls, ambiguity), very complex (extended interaction, deep reasoning).
Context Engineering Evaluation
Testing Prompt Variations: Run current prompt as baseline, then modified prompt on same cases. Measure quality scores, token usage, efficiency. Identify which changes improved which dimensions.
Degradation Testing: Run agents at different context sizes. Identify performance cliffs. Establish safe operating limits.
Evaluation Pitfalls
Scoring without justification → Require evidence-based justification before every score.
Single-pass pairwise comparison → Position bias corrupts results. Always swap positions and check consistency.
Overloaded criteria → One criterion = one measurable aspect.
Missing edge case guidance → Include edge cases in rubrics explicitly.
Ignoring confidence calibration → Calibrate confidence to position consistency and evidence strength.
Decision Framework: Direct vs. Pairwise
Is there an objective ground truth?
├── Yes → Direct Scoring
│ └── Examples: factual accuracy, instruction following, format compliance
└── No → Is it a preference or quality judgment?
├── Yes → Pairwise Comparison
│ └── Examples: tone, style, persuasiveness, creativity
└── No → Reference-based evaluation
└── Examples: summarization, translation
For pairwise: always run two passes with swapped positions. If passes disagree → TIE (position bias detected).
Guidelines
- Always require justification before scores — chain-of-thought improves reliability by 15-25%
- Always swap positions in pairwise comparison — single-pass is corrupted by position bias
- Match scale granularity to rubric specificity — don't use 1-10 without detailed levels
- Separate objective and subjective criteria — direct scoring for objective, pairwise for subjective
- Include confidence scores — calibrate to position consistency and evidence strength
- Define edge cases explicitly — ambiguous situations cause the most evaluation variance
- Use domain-specific rubrics — generic rubrics produce generic (less useful) evaluations
- Validate against human judgments — automated evaluation is only valuable if it correlates
- Monitor for systematic bias — track disagreement patterns by criterion and response type
- Design for iteration — evaluation systems improve with feedback loops
Example: Evaluating a Codex Command
Test Cases for a /refactor command:
- Simple: Rename a variable across a single file
- Medium: Extract a function from existing code
- Complex: Refactor a class to use a new design pattern
- Very Complex: Restructure module dependencies
Evaluation Rubric: Correctness (does it work?), Completeness (all instances updated?), Style (follows conventions?), Efficiency (unnecessary changes avoided?).
Iteration: If evaluation reveals the command often misses instances, add the instruction: "Search the entire codebase for all occurrences." Re-evaluate on same test cases. Compare completeness scores. Check correctness didn't regress.
Iterative Improvement Workflow
- Identify weakness: Use evaluation to find where agent struggles
- Hypothesize cause: Is it the prompt? The context? The examples?
- Modify prompt: Make targeted changes based on hypothesis
- Re-evaluate: Run same test cases with modified prompt
- Compare: Did the change improve the target dimension?
- Check regression: Did other dimensions suffer?
- Iterate: Repeat until quality meets threshold
Testing & Validation
After designing or running an evaluation:
- Justification present — every score has evidence-based justification written before it (not after)
- Pairwise consistency — if using pairwise: ran both position orders and checked for agreement
- Rubric coverage — all 5 criteria rated on every test case
- Complexity coverage — test cases span at least simple and complex levels
- Baseline comparison — before/after improvement uses the same test set (not a new one)
Quality gates:
- [ ] Chain-of-thought justification required before every score
- [ ] Pairwise comparisons run with swapped positions; disagreement → TIE
- [ ] Rubric weights sum to 1.0 (0.30 + 0.25 + 0.20 + 0.15 + 0.10)
- [ ] Test cases cover at least two complexity levels
- [ ] Improvement measurement compares against the same baseline test set
- [ ] At least one dimension validated against human judgment
Reference Guide
| Resource | Purpose |
|---|---|
references/bias-mitigation.md |
Position bias, length bias, self-enhancement, verbosity, authority bias — detection and mitigation |
references/llm-judge-patterns.md |
Structured workflow, hierarchical, PoLL, confidence calibration, detailed prompt templates, meta-evaluation |
references/metric-selection.md |
Classification metrics (precision/recall/F1), agreement (Cohen's κ), correlation (Spearman's ρ), decision tree |
04 同仓库更多 Skills
demo
Fixture skill used by scripts.marketplace_ci sync tests.
export-demo
Fixture skill used by scripts.marketplace_ci export tests.
work-to-development
Validate an accepted Linear Issue's readiness for implementation (criteria, dependencies, duplicate/existing artifacts) and request governed branch/worktree creation through git-kit:starting-work — recording work-started Git/GitHub evidence and a deliberate Linear Started transition only after Git identity is read back and verified. Use when asked to start work on a Linear issue, begin development for an accepted issue, or prepare an accepted issue for implementation. Never creates a branch directly — always delegates to git-kit.
work-linking
Create and maintain stable cross-system links and authority labels between Notion and Linear records, and detect/repair drift between them. Use when asked to link a Notion record to a Linear record, check for drift between Notion and Linear, or repair a broken/stale link. Repairs only the bounded non-authoritative copy or link — never chooses by newest timestamp, and never creates a reverse-write loop against either system's own authority.
status-and-learning
Produce a dated, explicitly non-live Linear-to-Notion progress summary, or capture an outcome/learning after work completes. Use when asked to summarize progress to Notion, report portfolio status, or capture what was learned from a completed piece of work — not to disposition open follow-up items (see open-item-management for that). Never runs automatically or continuously — every summary is a deliberate, approved snapshot, not a live sync.
repository-gates
Discover, delegate, record, and invalidate this repository's actual Git/GitHub lifecycle gates (pre-commit, pre-push, PR required checks, Codex delta review, review requirements, merge rights) and the repository-policy provider profile that maps governed operations to git-kit. Use when asked what gates apply to this repository, to discover repository policy before starting Wave 2 work, or to check whether prior gate evidence is still valid after new commits. Discovery and delegation only — never executes a gate's own internal logic, and never invents a gate the repository doesn't actually define (in particular, never assumes a "Review Changes" gate exists).