clawpathy-autoresearch
Eval-driven skill tuning. Given a task and an LLM-judge rubric, iteratively rewrites a SKILL.md until a downstream executor agent performs well against the judge. Low-code: all evaluation is LLM-as-judge, not deterministic Python.
- License 许可
- MIT
- 最近更新
- 2026-11-14
- Stars
- ★ 12
- Forks
- 5
01 前言 Front Matter
- name
- clawpathy-autoresearch
- tags
- [ "meta", "autoresearch", "skill-tuning", "llm-judge", "eval-driven" ]
- author
- Jay Moore
- inputs
- [ { "name": "paper_query_or_task", "type": "string", "required": true, "description": "Paper title/URL/PMID/DOI, or a freeform task description" } ]
- license
- MIT
- outputs
- [ { "name": "workspace/", "type": "directory", "description": "Tuned skill/SKILL.md plus history.jsonl, snapshots, executor_runs" } ]
- version
- 1.0.0
- metadata
- { "openclaw": { "os": [ "darwin", "linux" ], "emoji": "🔁", "always": false, "homepage": "https://github.com/ClawBio/ClawBio", "requires": { "bins": [ "python3", "claude" ] }, "trigger_keywords": [ "auto research", "autoresearch", "tune a skill", "skill tuning", "improve a skill", "eval-driven", "clawpathy", "replicate paper", "reproduce paper" ] } }
- description
- Eval-driven skill tuning. Given a task and an LLM-judge rubric, iteratively rewrites a SKILL.md until a downstream executor agent performs well against the judge. Low-code: all evaluation is LLM-as-judge, not deterministic Python.
02 资源文件 24
- 脚本__init__.py383 B
- 脚本__main__.py1.8 KB
- 其他.gitignore957 B
- 脚本dispatcher.py2.7 KB
- 参考文档examples/champions/README.md1.4 KB
- 参考文档examples/champions/trubetskoy_scz_finemap_0.235.md49.8 KB
- 参考文档examples/champions/yengo_height_ldsc_h2.md25.7 KB
- 参考文档examples/demo_task/README.md690 B
- 参考文档examples/demo_task/rubric.md1.7 KB
- 参考文档examples/demo_task/task.json463 B
- 参考文档examples/presentation_intro.md3.6 KB
- 资产文件examples/skill_transfer_summary.png117.6 KB
- 脚本executor.py1.1 KB
- 脚本judge.py2.6 KB
- 脚本loop_parallel.py9.2 KB
- 脚本loop.py5.1 KB
- 脚本multitask_loop.py15.1 KB
- 参考文档prompts/builder.md2.0 KB
- 参考文档prompts/executor.md927 B
- 参考文档prompts/judge.md3.6 KB
- 参考文档prompts/proposer.md1.2 KB
- 参考文档prompts/scout.md797 B
- 脚本proposer.py1.5 KB
- 脚本workspace.py3.5 KB
03 SKILL.md 正文
clawpathy-autoresearch
[!note] Vault audit 2026-07-24 — USE-10 Use this for eval-driven tuning that iteratively rewrites an existing SKILL.md against an LLM-judge rubric; to scaffold a new skill from scratch use
skill-builder, to auto-draft from observed workflows useautoskill, to package a plugin bundle useplugin-creator. Distinguishing axis: authoring mode (eval-tuning vs manual scaffold vs observation vs plugin packaging).
Eval-driven skill development. The system iteratively rewrites a SKILL.md
so a downstream executor agent performs better at a task class, as judged
by an LLM against a paper/task-specific rubric.
Core idea
propose (sonnet) → execute (sonnet, shell) → judge (opus, rubric)
↑ │
└──────── feedback: verdict + recommended edits ────────┘
- Proposer rewrites SKILL.md based on the last judge verdict.
- Executor runs the new SKILL.md end-to-end inside a workspace.
- Judge scores methodology (primary) and outputs (secondary) against a per-task rubric. Lower is better; 0 = perfect.
- Keep the new SKILL.md only if it strictly beats the best score; else
revert. Stop on target_score or on
early_stop_nconsecutive regressions.
You are the orchestrator
You (the agent reading this) don't run the loop yourself. You dispatch subagents to build the workspace, then hand off to the Python loop.
Phase 1 — Scout
Dispatch a subagent with prompts/scout.md to research the paper/task.
Report key findings to the user in a few lines.
Phase 2 — Scope (you + user)
Have a conversation. Ask ONE question at a time, multiple-choice where helpful. Agree on:
- what to reproduce / what success looks like
- which data sources are in-bounds
- what methodology expectations belong in the rubric
- iteration budget and target_score (if any)
Present a summary and get approval.
Phase 3 — Build
Dispatch a builder subagent with prompts/builder.md and the agreed
scope. It writes:
task.jsonrubric.md— the authoritative scoring rubric for the LLM judgereference/(optional; judge-only)skill/SKILL.md— seed
Validate:
import sys
from importlib import import_module
from pathlib import Path
# `skills` resolves only from the vault root; the hyphen blocks a plain import.
sys.path.insert(0, str(Path.home() / ".agents"))
validate_workspace = import_module("skills.clawpathy-autoresearch").validate_workspace
print(validate_workspace(Path("WORKSPACE"))) # [] means valid
Phase 4 — Loop
cd ~/.agents # run from the vault root, else `No module named 'skills'`
python -m skills.clawpathy-autoresearch WORKSPACE_DIR
# or with custom models:
python -m skills.clawpathy-autoresearch WORKSPACE_DIR \
--proposer-model sonnet --executor-model sonnet --judge-model opus
The loop streams progress to WORKSPACE/history.jsonl, snapshots every
iteration's skill to WORKSPACE/snapshots/iter-NNN.md, and writes the
executor's full transcript to WORKSPACE/executor_runs/iter-NNN.log.
Workspace layout
workspace/
task.json # task metadata + loop knobs
rubric.md # LLM-judge rubric (the heart of the system)
reference/ # optional ground truth, judge-only
skill/SKILL.md # iterated by the loop
output/ # executor outputs (cleared each iter)
executor_runs/iter-NNN.log # transcripts (judge reads these)
snapshots/iter-NNN.md # per-iter SKILL.md snapshots
history.jsonl # one row per iter: score, kept, verdict
Key principles
- LLM judge only. No deterministic Python scorers. All evaluation goes
through
judge.md+ opus. This keeps the system low-code and lets the rubric carry paper-specific nuance without adding code. - Methodology is primary. The rubric weights "did the agent use sound methods?" above "did the numbers match?". Ground-truth match is a signal, not the objective — the goal is better SKILL.md files.
- Never leak ground truth.
reference/is judge-only. The executor prompt says not to read it, and the judge penalises leakage. - No hardcoded answers in SKILL.md. The proposer prompt and the judge both enforce this. The executor must derive results by running methods.
- Snapshots + strict-better revert. Score on the first iter becomes the floor. Later iters that tie or regress revert to the best.
Safety
- All processing is local except scout web fetches for public resources.
- ClawBio disclaimer: research/education tool, not a medical device.
- The executor subagent holds
Bash,Write, andEdit, and the instructions it follows are the proposer's output, not human-reviewed text. It runs once per iteration, up tomax_iterations(default 30), and--parallel Kruns K at once. - Subagents run under
--permission-mode acceptEditsby default.--yoloswitches them tobypassPermissions, removing every approval prompt including for shell commands. Get the user's explicit consent before passing it, and only for genuinely unattended runs — the Phase 2 approval covers research scope, not this.
Gotchas
- Do not skip scoping. The rubric is paper-specific; a generic rubric tunes nothing. Get the user to agree on methodology expectations.
- Do not write a Python scorer. Earlier versions of this project did. They rewarded API-fetching, not methodology. The judge is the scorer.
- Do not hand-pick the "best" snapshot yourself. Trust the loop. If the judge is calibrated wrong, fix the rubric, not the history.
04 同仓库更多 Skills
zarr-python
Stores and queries chunked N-D scientific arrays with Zarr-Python 3, including codecs, sharding, S3/GCS storage, and NumPy/Dask/Xarray integration. Use for array layout, bounded I/O, format migration, or scientific metadata preservation.
yeet
Publish local changes to GitHub by confirming scope, committing intentionally, pushing the branch, and opening a draft PR through the GitHub app from this plugin, with `gh` used only as a fallback where connector coverage is insufficient.
xarray
N-dimensional labeled arrays and datasets in Python. Built on top of NumPy and Dask. It introduces labels in the form of dimensions, coordinates, and attributes on top of raw NumPy-like arrays, making data analysis in physical sciences more intuitive and less error-prone. Use for working with multi-dimensional scientific data, NetCDF/GRIB/Zarr files, climate/weather/oceanographic datasets, remote sensing, geospatial imaging, large out-of-memory datasets with Dask, and labeled array operations.
writing-mstest-tests
Use when asked to fix, rewrite, update, improve, modernize, show corrected code for, or explain existing MSTest tests or MSTest-specific configuration. Use for "review" when corrected code or edits are wanted, even for one pasted assertion or passing tests with bad failure output. Covers expected/actual labels; generic Boolean, collection, string, numeric, null, identity, exception, hard-cast, and object[] checks; TestContext/lifecycle; timeout/cancellation; OS/CI conditions, retry, cleanup, parallelization, MSTest.Sdk project setup, and MSTESTxxxx. Honor the installed MSTest version. DO NOT USE to design new test cases (code-testing), perform report-only audits, create project files rather than explain MSTest setup, run tests, migrate frameworks, or handle non-MSTest/non-.NET code.
wgs-prs
End-to-end WGS to polygenic risk score pipeline. Takes paired-end FASTQ files (or a pre-existing VCF) through nf-core/sarek for variant calling, applies VCF QC (normalisation, hard filtering, Ti/Tv and Het/Hom checks), then computes polygenic risk scores via the PGS Catalog. Fills the FASTQ to VCF gap upstream of the gwas-prs skill.
webapp-testing
Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.