AGENT SKILL

ageval-sdk-harness

Write dataset run.py with ageval_sdk (RunContext, Agent/AgentSession, ToolSet, RunTerminal, publish). Use for task run.py, sessions, tool guards, optional evaluator.py LLM-as-judge. Triggers: AgentSession, RunTerminal, ScriptError, ctx.publish_json, write run.py, evaluator.py. SDK never decides PASS or holds host credentials.

License 许可
MIT
所属仓库
ZJU-REAL/ageval
最近更新
2026-11-14
Stars
★ 81
Forks
3

01 前言 Front Matter

name
ageval-sdk-harness
description
Write dataset run.py with ageval_sdk (RunContext, Agent/AgentSession, ToolSet, RunTerminal, publish). Use for task run.py, sessions, tool guards, optional evaluator.py LLM-as-judge. Triggers: AgentSession, RunTerminal, ScriptError, ctx.publish_json, write run.py, evaluator.py. SDK never decides PASS or holds host credentials.

02 资源文件 3

03 SKILL.md 正文

SDK / run.py

from ageval_sdk import RunContext, RunTerminal


async def run(ctx: RunContext) -> RunTerminal:
    async with ctx.agent.session("solver", max_turns=1) as session:
        reply = await session.invoke(ctx.params["instruction"])
    ctx.publish_json("reply", {"ok": bool(reply.get("ok")), "text": reply.get("text") or ""})
    if not reply.get("ok"):
        return RunTerminal.failed(str(reply.get("error") or "invoke_failed"))
    return RunTerminal.completed("ok")

Default evaluator.py is a script: read artifacts + gold, return {status, score, metrics}. Optional extra key checks (use evaluation_check(...)) — parent writes evaluation/checks.json; bind ignores it. Optional LLM-as-judge: after gold is uploaded, Agent.session(<role>).invoke on a role declared in the same profiles.yaml. Parent injects inputs["agent"] when the socket is projected (environment: local). Still return {status, score, metrics}. invoke kwargs must not override profile_id / executor.

May Must not
Open sessions, tools, publish Decide PASS
Use ctx.params Re-read lock or secrets
Return completed/failed Override Core limits
Raise ScriptError when the script cannot continue Use ScriptError for an Agent that fell short

The Agent fell short: return RunTerminal.failed from run.py, or {"status": "FAIL", ...} from evaluator.py. The script, the dataset, or scoring itself cannot continue:

from ageval_sdk import ScriptError

raise ScriptError(title="Judge response empty", message="judge returned no text")

title is the same label on every occurrence. Variable detail goes in message. The same class is raised from either script. The phase is whichever worker caught it. | Evaluator Agent.session after gold | Bind PASS from judge prose / observation.jsonl |

API: references/api.md. Antipatterns: references/antipatterns.md.

04 同仓库更多 Skills

ZJU-REAL
ZJU-REAL/ageval

echo

暂无简介

★ 81
ZJU-REAL
ZJU-REAL/ageval

verify-ageval

Verify ageval by driving the public CLI: doctor, a lock that rejects a bad profile, and one minimal-demo task on local plus openai-http against the in-repo loopback gateway. Use when checking lock/run, seal invariants, or whether a credential skip was reported honestly. Triggers: verify ageval, loopback gateway, AGEVAL_SKIP_REAL_ACP, not run.

★ 81
ZJU-REAL
ZJU-REAL/ageval

eval-ops-pitfalls

Pre-run environment checks and suite troubleshooting: dead proxies, invocation timeouts, package mirrors, model routing, environment_image_unresolved, widespread failures, evaluator dependencies, permissions, and CLI/schema mismatches. Use when preparing an evaluation run or diagnosing acp_timeout, HTTP_PROXY, AGEVAL_AGENT_INVOKE_TIMEOUT, AGEVAL_APT_MIRROR, PEP668, or test collection failures.

★ 81
ZJU-REAL
ZJU-REAL/ageval

ageval-plugin

Author ageval.plugin/1: exclusive and chain slots, registry resolve, environment or executor implementations, image_layers bake. Use for plugin.yaml, attach_stdio hosts, trajectory_collect, README capability and parameter tables. Not an app store. Never invent slots.

★ 81
ZJU-REAL
ZJU-REAL/ageval

ageval-platform

ageval platform map: authority (docs/design vs ARCHITECTURE vs Issues), Core vs dataset ownership, red lines (no secrets, mechanism-named adapters), evidence grades, which sibling skill to load. Use when entering the repo, asking "what is ageval", "who owns PASS", "read order", or "evidence grade". Not a substitute for docs/design.

★ 81
ZJU-REAL
ZJU-REAL/ageval

ageval-design-system

ageval web UI style rules for coding agents editing website (landing + docs), apps/hub, or apps/viewer: canonical color tokens (cool paper/ink + IKB #1B54E8/#5B7BFF), font stacks (Geist/Anton wordmark-only), radius, role-based focus (scan chrome keeps hairline; edit fields use IKB border), reuse of shipped components, the ten invariants, and the machine check scripts/check_design_tokens.py. Use when adding or restyling UI in any of the three web surfaces, picking colors or fonts, adding icons, or when a change touches raw hex, focus rings, buttons, search fields, or the owl brand assets. Authority is docs/design/09-web-ui-design-system.md; apps/shared/DESIGN.md YAML is the SPA token listing (Hub/Viewer inherit it); SPA DESIGN.md Taste is anti-slop (product chrome, not landing) plus the reuse map. This skill only routes and summarizes — it is not a page inventory.

★ 81