AGENT SKILL

bench-skills

Benchmark stellar-build skills to measure quality and the lift from optimization. Use when the user says "bench my skills", "bench-skills", "benchmark skills", "did optimization help", "measure my skills", "stellar-loop bench", or wants to quantify whether /optimize-skills actually improved things. Held-out eval prompts scored by an LLM judge, reproducible with fixed seeds. Fully local.

License 许可
Apache-2.0
最近更新
2026-11-15
Stars
★ 4
Forks
2

01 前言 Front Matter

name
bench-skills
description
Benchmark stellar-build skills to measure quality and the lift from optimization. Use when the user says "bench my skills", "bench-skills", "benchmark skills", "did optimization help", "measure my skills", "stellar-loop bench", or wants to quantify whether /optimize-skills actually improved things. Held-out eval prompts scored by an LLM judge, reproducible with fixed seeds. Fully local.

03 SKILL.md 正文

What this skill does

This is the measurement half of the stellar-build learning loop - the analog of jarvis bench skills --max-samples 5 --seeds 42. It scores a skill's quality on a held-out set of eval prompts so you can tell whether /optimize-skills actually helped, instead of guessing.

It is read-only: it never edits skills. It only produces scores and a report.

Arguments (optional, natural language is fine)

  • a skill name to bench just one skill (default: all skills that have an eval set)
  • --max-samples N - cap eval prompts per skill (default 5)
  • --seeds a,b,... - fixed seeds for reproducible sampling/judging (default 42)

How to run it

1. Assemble the eval set per skill

For each target skill, build a small held-out set of representative prompts:

  1. If ~/.stellar/bench/<skill>.jsonl exists, use it. Each line: { "prompt": "...", "rubric": "what a great response must do" }.
  2. Otherwise, mine candidates from ~/.stellar/traces/*.jsonl. Records are grouped into spans (see optimize-skills for the schema): take the real opening prompt of each span the user triggered this skill with, hold them out (do not also use them as optimization demos), and pair each with a short rubric you infer from the skill's stated purpose. Use one prompt per span, not per turn, and skip legacy / coverage-only records (no kind, mount-stub result_preview) - they carry no usable trigger. Then persist them to ~/.stellar/bench/<skill>.jsonl so future runs are stable.
  3. If there is neither a saved set nor enough traces, synthesize 3 plausible prompts from the skill's description and note that the set is synthetic.

Sample up to --max-samples prompts using the given --seeds deterministically (e.g. stable sort by a hash of seed + prompt, take the first N). Reuse the same seed across before/after runs so comparisons are apples-to-apples.

2. Score the skill

For each eval prompt, read the skill's current ~/.claude/skills/<skill>/SKILL.md and reason about how that skill, as written, would handle the prompt. Then act as a strict judge and score the expected response quality on a 1–5 scale against the rubric:

  • 5 - nails the rubric, no gaps
  • 4 - strong, minor gaps
  • 3 - acceptable, real gaps
  • 2 - weak / partially off-target
  • 1 - wrong or unusable

Judge the skill as instructions - clarity, correct triggering, whether its steps and any LEARNED block would produce a rubric-satisfying result. Be consistent across runs; the same skill text and seed should yield the same score.

3. Before/after mode

If the user wants the lift from a recent optimization (or just asks "did it help"):

  1. Score the current installed skill (the "after").
  2. Score the most recent backup at ~/.stellar/backups/<skill>/SKILL.md.<latest> as the "before", using the same prompts and seeds.
  3. Report both and the delta.

4. Report

Per skill: mean score, per-prompt scores, and - in before/after mode - the lift (after − before) with a one-line interpretation. Roll up an overall mean across skills. Call out any skill that regressed (after < before) and suggest stellar-loop restore <skill>.

Optionally append a run record to ~/.stellar/bench/results.jsonl:

{"ts":"<UTC>","skill":"<name>","mean":4.2,"n":5,"seeds":[42],"mode":"after"}

Guardrails

  • Read-only. Never modify a skill from this flow.
  • Reproducible. Same skill text + same seed ⇒ same score. Hold seeds fixed across before/after.
  • Honest. A synthetic or tiny eval set is a weak signal - say so in the report.
  • Local only. Everything reads from ~/.stellar and the installed skills.

04 同仓库更多 Skills

LumenWipe
LumenWipe/lumenwipe

zk-proofs

Zero-knowledge proofs and privacy patterns on Stellar. Covers Groth16 verification in smart contracts via BLS12-381 host functions (CAP-0059, available), the BN254 + Poseidon proposals (CAP-0074/0075, status-sensitive), and concrete toolchain walkthroughs for Circom (on-chain verifiable today), Noir, and RISC Zero (attestation pattern until BN254 lands). Use when building privacy-preserving applications, ZK-verifier contracts, or wiring a proving toolchain to Stellar.

★ 4
LumenWipe
LumenWipe/lumenwipe

tyler-architect

System architect and technical design leader. Use when the user asks to talk to Tyler or requests the architect.

★ 4
LumenWipe
LumenWipe/lumenwipe

stellar-help

Analyzes current state and user query to answer BMad questions or recommend the next skill(s) to use. Use when user asks for help, bmad help, what to do next, or what to start with in BMad.

★ 4
LumenWipe
LumenWipe/lumenwipe

stellar-competitive-landscape

Map the competitive landscape for a Stellar project idea. Use when a user says "who are my competitors on Stellar", "competitive analysis Stellar", "what already exists in this space on Stellar", "show me similar Stellar projects", "is anyone else building this on Stellar", or "map the Stellar landscape for X". Queries the 728-project LumenLoop ecosystem database and Electric Capital developer activity to rank competitors by SCF funding history and repo activity.

★ 4
LumenWipe
LumenWipe/lumenwipe

standards

Stellar standards, ecosystem, and reference. Covers SEPs (Stellar Ecosystem Proposals), CAPs (Core Advancement Proposals), and a quick map for picking the right standard for wallets, anchors, payments, deposits/withdrawals, federation, deep links, and KYC. Also bundles ecosystem references (DeFi protocols, dev tools, wallets, infra, community projects), curated documentation links, and MCP servers (live tools such as Raven). Use when you need to know which SEP applies, or want a starting point for ecosystem integrations, official docs, or live MCP tooling.

★ 4
LumenWipe
LumenWipe/lumenwipe

smart-contracts

Stellar smart contract development (Rust, soroban-sdk). Entry point with project setup, contract anatomy, and build/deploy workflow, routing to three companion files in this directory - development.md (storage/TTL, authorization, cross-contract calls, tokens, events, errors, upgrades, fees, troubleshooting), testing.md (unit, fuzz, property, fork, mutation, integration), and security.md (vulnerability classes, checklists, tooling, audits). Use when writing, testing, reviewing, securing, debugging, or shipping Stellar smart contracts, including anything the user calls "Soroban" - Soroban contracts, soroban-sdk, Soroban auth/storage/TTL errors, SEP-41 tokens, or SAC integration from contract code.

★ 4