ageval-sdk-harness
Write dataset run.py with ageval_sdk (RunContext, Agent/AgentSession, ToolSet, RunTerminal, publish). Use for task run.py, sessions, tool guards, optional evaluator.py LLM-as-judge. Triggers: AgentSession, RunTerminal, ScriptError, ctx.publish_json, write run.py, evaluator.py. SDK never decides PASS or holds host credentials.
- License 许可
- MIT
- 所属仓库
- ZJU-REAL/ageval
- 最近更新
- 2026-11-14
- Stars
- ★ 81
- Forks
- 3
01 前言 Front Matter
- name
- ageval-sdk-harness
- description
- Write dataset run.py with ageval_sdk (RunContext, Agent/AgentSession, ToolSet, RunTerminal, publish). Use for task run.py, sessions, tool guards, optional evaluator.py LLM-as-judge. Triggers: AgentSession, RunTerminal, ScriptError, ctx.publish_json, write run.py, evaluator.py. SDK never decides PASS or holds host credentials.
02 资源文件 3
- 参考文档agents/openai.yaml243 B
- 参考文档references/antipatterns.md937 B
- 参考文档references/api.md3.1 KB
03 SKILL.md 正文
SDK / run.py
from ageval_sdk import RunContext, RunTerminal
async def run(ctx: RunContext) -> RunTerminal:
async with ctx.agent.session("solver", max_turns=1) as session:
reply = await session.invoke(ctx.params["instruction"])
ctx.publish_json("reply", {"ok": bool(reply.get("ok")), "text": reply.get("text") or ""})
if not reply.get("ok"):
return RunTerminal.failed(str(reply.get("error") or "invoke_failed"))
return RunTerminal.completed("ok")
Default evaluator.py is a script: read artifacts + gold, return {status, score, metrics}. Optional extra key checks (use evaluation_check(...)) — parent writes evaluation/checks.json; bind ignores it. Optional LLM-as-judge: after gold is uploaded, Agent.session(<role>).invoke on a role declared in the same profiles.yaml. Parent injects inputs["agent"] when the socket is projected (environment: local). Still return {status, score, metrics}. invoke kwargs must not override profile_id / executor.
| May | Must not |
|---|---|
| Open sessions, tools, publish | Decide PASS |
Use ctx.params |
Re-read lock or secrets |
| Return completed/failed | Override Core limits |
Raise ScriptError when the script cannot continue |
Use ScriptError for an Agent that fell short |
The Agent fell short: return RunTerminal.failed from run.py, or {"status": "FAIL", ...} from evaluator.py. The script, the dataset, or scoring itself cannot continue:
from ageval_sdk import ScriptError
raise ScriptError(title="Judge response empty", message="judge returned no text")
title is the same label on every occurrence. Variable detail goes in message. The same class is raised from either script. The phase is whichever worker caught it.
| Evaluator Agent.session after gold | Bind PASS from judge prose / observation.jsonl |
API: references/api.md. Antipatterns: references/antipatterns.md.
04 同仓库更多 Skills
echo
暂无简介
verify-ageval
Verify ageval by driving the public CLI: doctor, a lock that rejects a bad profile, and one minimal-demo task on local plus openai-http against the in-repo loopback gateway. Use when checking lock/run, seal invariants, or whether a credential skip was reported honestly. Triggers: verify ageval, loopback gateway, AGEVAL_SKIP_REAL_ACP, not run.
eval-ops-pitfalls
Pre-run environment checks and suite troubleshooting: dead proxies, invocation timeouts, package mirrors, model routing, environment_image_unresolved, widespread failures, evaluator dependencies, permissions, and CLI/schema mismatches. Use when preparing an evaluation run or diagnosing acp_timeout, HTTP_PROXY, AGEVAL_AGENT_INVOKE_TIMEOUT, AGEVAL_APT_MIRROR, PEP668, or test collection failures.
ageval-plugin
Author ageval.plugin/1: exclusive and chain slots, registry resolve, environment or executor implementations, image_layers bake. Use for plugin.yaml, attach_stdio hosts, trajectory_collect, README capability and parameter tables. Not an app store. Never invent slots.
ageval-platform
ageval platform map: authority (docs/design vs ARCHITECTURE vs Issues), Core vs dataset ownership, red lines (no secrets, mechanism-named adapters), evidence grades, which sibling skill to load. Use when entering the repo, asking "what is ageval", "who owns PASS", "read order", or "evidence grade". Not a substitute for docs/design.
ageval-design-system
ageval web UI style rules for coding agents editing website (landing + docs), apps/hub, or apps/viewer: canonical color tokens (cool paper/ink + IKB #1B54E8/#5B7BFF), font stacks (Geist/Anton wordmark-only), radius, role-based focus (scan chrome keeps hairline; edit fields use IKB border), reuse of shipped components, the ten invariants, and the machine check scripts/check_design_tokens.py. Use when adding or restyling UI in any of the three web surfaces, picking colors or fonts, adding icons, or when a change touches raw hex, focus rings, buttons, search fields, or the owl brand assets. Authority is docs/design/09-web-ui-design-system.md; apps/shared/DESIGN.md YAML is the SPA token listing (Hub/Viewer inherit it); SPA DESIGN.md Taste is anti-slop (product chrome, not landing) plus the reuse map. This skill only routes and summarizes — it is not a page inventory.