ageval-config-package
Author ageval datasets (ageval.yaml + tasks/*/task.yaml, run.py/evaluator.py, profiles.yaml, environment kinds local|docker|e2b|ssh|daytona, limits, gold isolation, script vs LLM-as-judge scoring). Dataset README Run commands start with `ageval` (CLI already installed). Never `uv run ageval` or `.venv/bin/ageval`. Triggers: ageval.yaml, task.yaml, profiles.yaml, dataset, environment kind, evaluator.py. Never secrets in yaml. Never provider.kind.
- License 许可
- MIT
- 所属仓库
- ZJU-REAL/ageval
- 最近更新
- 2026-11-14
- Stars
- ★ 81
- Forks
- 3
01 前言 Front Matter
- name
- ageval-config-package
- description
- Author ageval datasets (ageval.yaml + tasks/*/task.yaml, run.py/evaluator.py, profiles.yaml, environment kinds local|docker|e2b|ssh|daytona, limits, gold isolation, script vs LLM-as-judge scoring). Dataset README Run commands start with `ageval` (CLI already installed). Never `uv run ageval` or `.venv/bin/ageval`. Triggers: ageval.yaml, task.yaml, profiles.yaml, dataset, environment kind, evaluator.py. Never secrets in yaml. Never provider.kind.
02 资源文件 4
- 参考文档agents/openai.yaml246 B
- 参考文档references/ageval-yaml.md1.7 KB
- 参考文档references/conversion.md3.6 KB
- 参考文档references/isolation.md3.6 KB
03 SKILL.md 正文
Config / dataset + task
Config Core is the only normative reader. Delivery unit is a dataset root.
my-dataset/ # CLI path (ageval.dataset/1)
├── ageval.yaml
├── profiles.yaml # THE job (lock/run default). ageval publish also packs every root profiles*.yaml
├── env.example
├── shared/lib/ # optional; import shared.lib.*
└── tasks/<task_id>/
├── task.yaml # ageval.task/1 — roles + intent
├── run.py # async def run(ctx)
├── evaluator.py
├── environment/ # Dockerfile, setup.sh, compose
├── evaluation/ # gold — not for Agent
└── data/ # seed
CLI: ageval lock|run <dataset-root> --task <id>.
Reader-facing README / docs: commands start with ageval. The reader already has the CLI. Do not write uv run ageval, uv run ageval lock, .venv/bin/ageval, or AGEVAL=/path/to/... wrappers. Agent-internal notes (this skill, AGENTS.md) may name a venv binary; dataset README may not.
Dataset README introduces this dataset (identity, profiles.yaml, layout, ageval lock|run, conversion, known gaps). It does not document local .ageval/suite-runs ids, P/F/E counts, or leaderboard rows. Those are upload-suite later, not package copy.
Files present are recognized. Do not re-declare run.py entrypoints in yaml. Do not put executor / api_key on the member task. Do not write provider.kind or assurance.
# ageval.yaml
format: ageval.dataset/1
dataset_id: org/my-suite
version: "0.1.0"
tasks:
root: tasks
# profiles.yaml
format: ageval.profiles/1
environment: docker
agent_profiles:
solver:
executor: acp
api_key: ${ZHIPU_API_KEY}
options: { entry: pi }
extensions:
- plugin: acp
- plugin: docker
Gold lives in evaluation/ and is uploaded at evaluate, not before. setup.sh is the last environment slot.
Two scoring styles in evaluator.py (same barrier; PASS still the return value):
| Style | When | Evidence |
|---|---|---|
| Deterministic script | Default. Compare artifacts to gold. No Agent.session. |
result.json. No evaluation/observation.jsonl. Optional checks via evaluation_check(...) → evaluation/checks.json. |
| LLM-as-judge | Extra role on the same profiles.yaml (e.g. judge). After gold lands, Agent.session("judge").invoke. |
evaluation/observation.jsonl (no user rows). Not merged into Agent trajectory.jsonl. May also return checks. |
Judge binding stays in profiles.yaml (not on the member task). limits.agent_invocations counts run-phase invokes only; judge invokes are bounded by limits.evaluate_seconds. limits.agent_concurrency caps simultaneous run-phase executor invokes (default 1). Judge admission stays serial. SDK session into the evaluator process is projected on environment: local. No Agent.session = script path.
--profiles swaps the job for this lock/run. --agent is mutually exclusive with --profiles. --model requires --agent.
profiles.yaml vs extra profiles*.yaml
Default job (ageval run <root>) |
ageval publish / packageDigest |
|
|---|---|---|
profiles.yaml |
this file only | included |
sibling profiles*.yaml at dataset root |
ignored unless --profiles |
included (member_paths_for_digest / tarball). Overlays those docs declare also enter. |
Hub consumers who ageval run official/…@ver without --profiles get profiles.yaml. Extra profiles.acp.*.yaml at the root still ship in the package and change digest. Experiment jobs that must not be on Hub do not live as profiles*.yaml under the dataset root.
Hub comparability still uses suite fingerprint / upload-suite. Do not add n_attempts to yaml.
Field catalog: references/ageval-yaml.md. Isolation: references/isolation.md. Conversion: references/conversion.md.
04 同仓库更多 Skills
echo
暂无简介
verify-ageval
Verify ageval by driving the public CLI: doctor, a lock that rejects a bad profile, and one minimal-demo task on local plus openai-http against the in-repo loopback gateway. Use when checking lock/run, seal invariants, or whether a credential skip was reported honestly. Triggers: verify ageval, loopback gateway, AGEVAL_SKIP_REAL_ACP, not run.
eval-ops-pitfalls
Pre-run environment checks and suite troubleshooting: dead proxies, invocation timeouts, package mirrors, model routing, environment_image_unresolved, widespread failures, evaluator dependencies, permissions, and CLI/schema mismatches. Use when preparing an evaluation run or diagnosing acp_timeout, HTTP_PROXY, AGEVAL_AGENT_INVOKE_TIMEOUT, AGEVAL_APT_MIRROR, PEP668, or test collection failures.
ageval-sdk-harness
Write dataset run.py with ageval_sdk (RunContext, Agent/AgentSession, ToolSet, RunTerminal, publish). Use for task run.py, sessions, tool guards, optional evaluator.py LLM-as-judge. Triggers: AgentSession, RunTerminal, ScriptError, ctx.publish_json, write run.py, evaluator.py. SDK never decides PASS or holds host credentials.
ageval-plugin
Author ageval.plugin/1: exclusive and chain slots, registry resolve, environment or executor implementations, image_layers bake. Use for plugin.yaml, attach_stdio hosts, trajectory_collect, README capability and parameter tables. Not an app store. Never invent slots.
ageval-platform
ageval platform map: authority (docs/design vs ARCHITECTURE vs Issues), Core vs dataset ownership, red lines (no secrets, mechanism-named adapters), evidence grades, which sibling skill to load. Use when entering the repo, asking "what is ageval", "who owns PASS", "read order", or "evidence grade". Not a substitute for docs/design.