bench-skills
Benchmark stellar-build skills to measure quality and the lift from optimization. Use when the user says "bench my skills", "bench-skills", "benchmark skills", "did optimization help", "measure my skills", "stellar-loop bench", or wants to quantify whether /optimize-skills actually improved things. Held-out eval prompts scored by an LLM judge, reproducible with fixed seeds. Fully local.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-15
- Stars
- ★ 4
- Forks
- 2
01 前言 Front Matter
- name
- bench-skills
- description
- Benchmark stellar-build skills to measure quality and the lift from optimization. Use when the user says "bench my skills", "bench-skills", "benchmark skills", "did optimization help", "measure my skills", "stellar-loop bench", or wants to quantify whether /optimize-skills actually improved things. Held-out eval prompts scored by an LLM judge, reproducible with fixed seeds. Fully local.
03 SKILL.md 正文
What this skill does
This is the measurement half of the stellar-build learning loop - the analog of
jarvis bench skills --max-samples 5 --seeds 42. It scores a skill's quality on a
held-out set of eval prompts so you can tell whether /optimize-skills actually
helped, instead of guessing.
It is read-only: it never edits skills. It only produces scores and a report.
Arguments (optional, natural language is fine)
- a skill name to bench just one skill (default: all skills that have an eval set)
--max-samples N- cap eval prompts per skill (default 5)--seeds a,b,...- fixed seeds for reproducible sampling/judging (default 42)
How to run it
1. Assemble the eval set per skill
For each target skill, build a small held-out set of representative prompts:
- If
~/.stellar/bench/<skill>.jsonlexists, use it. Each line:{ "prompt": "...", "rubric": "what a great response must do" }. - Otherwise, mine candidates from
~/.stellar/traces/*.jsonl. Records are grouped into spans (see optimize-skills for the schema): take the real openingpromptof each span the user triggered this skill with, hold them out (do not also use them as optimization demos), and pair each with a short rubric you infer from the skill's stated purpose. Use one prompt per span, not per turn, and skip legacy / coverage-only records (nokind, mount-stubresult_preview) - they carry no usable trigger. Then persist them to~/.stellar/bench/<skill>.jsonlso future runs are stable. - If there is neither a saved set nor enough traces, synthesize 3 plausible prompts
from the skill's
descriptionand note that the set is synthetic.
Sample up to --max-samples prompts using the given --seeds deterministically
(e.g. stable sort by a hash of seed + prompt, take the first N). Reuse the same
seed across before/after runs so comparisons are apples-to-apples.
2. Score the skill
For each eval prompt, read the skill's current ~/.claude/skills/<skill>/SKILL.md
and reason about how that skill, as written, would handle the prompt. Then act as a
strict judge and score the expected response quality on a 1–5 scale against the
rubric:
- 5 - nails the rubric, no gaps
- 4 - strong, minor gaps
- 3 - acceptable, real gaps
- 2 - weak / partially off-target
- 1 - wrong or unusable
Judge the skill as instructions - clarity, correct triggering, whether its steps
and any LEARNED block would produce a rubric-satisfying result. Be consistent
across runs; the same skill text and seed should yield the same score.
3. Before/after mode
If the user wants the lift from a recent optimization (or just asks "did it help"):
- Score the current installed skill (the "after").
- Score the most recent backup at
~/.stellar/backups/<skill>/SKILL.md.<latest>as the "before", using the same prompts and seeds. - Report both and the delta.
4. Report
Per skill: mean score, per-prompt scores, and - in before/after mode - the lift
(after − before) with a one-line interpretation. Roll up an overall mean across
skills. Call out any skill that regressed (after < before) and suggest
stellar-loop restore <skill>.
Optionally append a run record to ~/.stellar/bench/results.jsonl:
{"ts":"<UTC>","skill":"<name>","mean":4.2,"n":5,"seeds":[42],"mode":"after"}
Guardrails
- Read-only. Never modify a skill from this flow.
- Reproducible. Same skill text + same seed ⇒ same score. Hold seeds fixed across before/after.
- Honest. A synthetic or tiny eval set is a weak signal - say so in the report.
- Local only. Everything reads from
~/.stellarand the installed skills.
04 同仓库更多 Skills
zk-proofs
Zero-knowledge proofs and privacy patterns on Stellar. Covers Groth16 verification in smart contracts via BLS12-381 host functions (CAP-0059, available), the BN254 + Poseidon proposals (CAP-0074/0075, status-sensitive), and concrete toolchain walkthroughs for Circom (on-chain verifiable today), Noir, and RISC Zero (attestation pattern until BN254 lands). Use when building privacy-preserving applications, ZK-verifier contracts, or wiring a proving toolchain to Stellar.
tyler-architect
System architect and technical design leader. Use when the user asks to talk to Tyler or requests the architect.
stellar-help
Analyzes current state and user query to answer BMad questions or recommend the next skill(s) to use. Use when user asks for help, bmad help, what to do next, or what to start with in BMad.
stellar-competitive-landscape
Map the competitive landscape for a Stellar project idea. Use when a user says "who are my competitors on Stellar", "competitive analysis Stellar", "what already exists in this space on Stellar", "show me similar Stellar projects", "is anyone else building this on Stellar", or "map the Stellar landscape for X". Queries the 728-project LumenLoop ecosystem database and Electric Capital developer activity to rank competitors by SCF funding history and repo activity.
standards
Stellar standards, ecosystem, and reference. Covers SEPs (Stellar Ecosystem Proposals), CAPs (Core Advancement Proposals), and a quick map for picking the right standard for wallets, anchors, payments, deposits/withdrawals, federation, deep links, and KYC. Also bundles ecosystem references (DeFi protocols, dev tools, wallets, infra, community projects), curated documentation links, and MCP servers (live tools such as Raven). Use when you need to know which SEP applies, or want a starting point for ecosystem integrations, official docs, or live MCP tooling.
smart-contracts
Stellar smart contract development (Rust, soroban-sdk). Entry point with project setup, contract anatomy, and build/deploy workflow, routing to three companion files in this directory - development.md (storage/TTL, authorization, cross-contract calls, tokens, events, errors, upgrades, fees, troubleshooting), testing.md (unit, fuzz, property, fork, mutation, integration), and security.md (vulnerability classes, checklists, tooling, audits). Use when writing, testing, reviewing, securing, debugging, or shipping Stellar smart contracts, including anything the user calls "Soroban" - Soroban contracts, soroban-sdk, Soroban auth/storage/TTL errors, SEP-41 tokens, or SAC integration from contract code.