benchmark-evaluation
Validate DeepResearch benchmark rollout files and choose official evaluation routes before spending judge API credits.
- License 许可
- Apache-2.0
- 最近更新
- 2026-10-08
- Stars
- ★ 11
- Forks
- 3
01 前言 Front Matter
- name
- benchmark-evaluation
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Validate DeepResearch benchmark rollout files and choose official evaluation routes before spending judge API credits.
- disable-model-invocation
- true
02 资源文件 3
- 参考文档references/evaluation-workflows.md7.4 KB
- 参考文档references/prediction-format.md4.5 KB
- 参考文档references/troubleshooting.md4.8 KB
03 SKILL.md 正文
Benchmark Evaluation
Use this sub-skill when a future agent already has DeepResearch prediction rollout files and needs to decide whether they are ready for official LLM-as-judge evaluation. It is a guidance and preflight skill: it does not generate predictions and it does not call external judge APIs.
Route to this sub-skill when
- The task mentions
evaluate_deepsearch_official.py,evaluate_hle_official.py, pass@k, HLE, GAIA, BrowseComp, WebWalker, XBench DeepSearch, scored JSONL, or judge-model costs. - A rollout folder contains
iter1.jsonl,iter2.jsonl,iter3.jsonl, or split-suffixed files such asiter2_split3of8.jsonl. - The user wants to inspect malformed JSONL, missing predictions/messages,
invalid
<answer>...</answer>tags, missing round files, or termination frequencies before running official judging.
Do not use this sub-skill for
- Creating DeepResearch predictions or running the ReAct agent; route to the
sibling
react-inferencesub-skill. - Choosing among WebAgent family projects or understanding their project-specific
evaluators; route to
webagent-family, then return here for common rollout and LLM-judge concepts. - Replacing the official judge scripts. The bundled validator is a safe schema preflight only, not an implementation of official benchmark metrics.
Fast safe preflight
From this sub-skill directory, validate an unsplit DeepResearch rollout folder:
python scripts/validate_prediction_rollouts.py <rollout-folder> --dataset gaia
Validate distributed rollouts written by run_multi_react.py --total_splits N:
python scripts/validate_prediction_rollouts.py <rollout-folder> --dataset webwalker --allow-splits
Validate an HLE single JSONL file before the HLE judge route:
python scripts/validate_prediction_rollouts.py <hle-predictions.jsonl> --dataset hle
A passing validator result means the files satisfy local shape checks. It does not mean the official LLM judge has run or that pass@k metrics are available.
Read next
references/prediction-format.mdfor rollout file names, fields, split behavior, and validator interpretation.references/evaluation-workflows.mdfor official DeepSearch and HLE evaluator routes, judge-model selection, metrics, and output files.references/troubleshooting.mdfor missing rounds, malformed JSON, absent fields, invalid answer tags, unsupported datasets, missing API variables, and tokenizer caveats.
04 同仓库更多 Skills
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
python-api
Programmatic Object-Detection-Metrics API guidance for BoundingBox, BoundingBoxes, Evaluator, enums, coordinate conversion, AP/mAP semantics, and source-style import diagnostics.
file-evaluation
Folder-based PASCAL VOC AP/mAP evaluation from ground-truth and detection text files.
object-detection-metrics
Guides legacy Object-Detection-Metrics PASCAL VOC AP/mAP evaluation from detection text folders and direct Python API classes.
nx-workspace-patterns
Configure and optimize Nx monorepo workspaces. Use when setting up Nx, configuring project boundaries, optimizing build caching, or implementing affected commands.