rubric-evaluator
Evaluate a skill folder that contains SKILL.md with a 6-section rubric, deterministic rule checks, semantic model checks, S/A/B/C/F grades, and fixable reports. Use when asked to grade, audit, review, dogfood, or improve a skill, skill directory, SKILL.md, rubric score, trigger quality, resource structure, or safety gate.
- License 许可
- MIT
- 最近更新
- 2026-09-21
- Stars
- ★ 8
- Forks
- 2
01 前言 Front Matter
- name
- rubric-evaluator
- description
- Evaluate a skill folder that contains SKILL.md with a 6-section rubric, deterministic rule checks, semantic model checks, S/A/B/C/F grades, and fixable reports. Use when asked to grade, audit, review, dogfood, or improve a skill, skill directory, SKILL.md, rubric score, trigger quality, resource structure, or safety gate.
02 资源文件 9
- 参考文档agents/openai.yaml224 B
- 参考文档references/automation.md3.8 KB
- 参考文档references/fallbacks.md2.6 KB
- 参考文档references/model-rubric.md13.0 KB
- 脚本scripts/check_rules.py24.2 KB
- 脚本scripts/compare_results.py2.8 KB
- 脚本scripts/render_report.py5.5 KB
- 脚本scripts/results.py4.4 KB
- 脚本scripts/run_checks.sh598 B
03 SKILL.md 正文
Rubric Evaluator
Evaluate a skill directory with a 6-section, 31-item rubric. Use deterministic scripts for structural and safety checks, then apply semantic model checks from the bundled rubric reference. If the runtime or installation path is incomplete, use the fallback playbook and label the result as provisional.
Script paths
The bundled scripts live in this skill's own scripts/ directory. On Claude Code, reference them through the plugin root so they resolve after installation: ${CLAUDE_PLUGIN_ROOT}/skills/rubric-evaluator/scripts/<name>. If ${CLAUDE_PLUGIN_ROOT} is unset — a plain checkout or the Codex host — drop the prefix and run scripts/<name> from this skill's directory. The commands below use the ${CLAUDE_PLUGIN_ROOT} form.
Workflow
-
Confirm the target is a directory containing
SKILL.md. If the path is missing or ambiguous, ask for the exact skill directory. -
Run the deterministic checks with the portable wrapper:
sh "${CLAUDE_PLUGIN_ROOT}/skills/rubric-evaluator/scripts/run_checks.sh" <target-skill-dir>The wrapper tries
python3, thenpython, then the Windowspy -3launcher. If the wrapper cannot run, readreferences/fallbacks.mdand continue in the documented degraded mode. -
If the wrapper is unavailable but a Python interpreter is known, run the script directly:
python3 "${CLAUDE_PLUGIN_ROOT}/skills/rubric-evaluator/scripts/check_rules.py" <target-skill-dir>The script returns JSON with
findings, a provisional rule-onlygrade,coverage, and provenance (rubric version/hash and target hash). Rule-only output is always partial: 14 semantic checks remain. -
Read
references/model-rubric.mdafter the rule output is available. Apply all 14 model checks and produce findings with the same schema as the rule findings. -
Combine rule and model findings into the rule result's
findingsarray, preserving its provenance fields. Keep every finding, includingpassandna, so the final grade is auditable. If rule finding 6.1 isna, it lists suspected credential lines that need review: read them, set 6.1 tofailfor a real secret orpassfor a documented example value, and change itscheckertomodel. Replace 6.1 rather than appending a duplicate. Evaluate the same target snapshot throughout; if files change, rerun the evaluation. -
Validate and refresh the combined result with
scripts/check_rules.py <combined-findings.json> --finalize, saving stdout to a different file. Add--require-completewhen all 31 checks must be judged. A result is complete only when all IDs are present and none isna; otherwise label the grade provisional. The script computes the grade from failed findings:- Any failed
BLOCKERmeansF. - No failed
BLOCKERand no failedMAJORmeansS. - No failed
BLOCKERand 1-2 failedMAJORmeansA. - No failed
BLOCKERand 3-4 failedMAJORmeansB. - No failed
BLOCKERand 5 or more failedMAJORmeansC. - Failed
MINORfindings do not affect the grade.
- Any failed
-
Render the report. If you have written combined findings to a JSON file, run:
python3 "${CLAUDE_PLUGIN_ROOT}/skills/rubric-evaluator/scripts/render_report.py" <combined-findings.json> --skill-name <skill-name>If no file is written, mirror the same report structure manually.
Fallbacks
Read references/fallbacks.md when scripts/run_checks.sh fails, Python is unavailable, the skill is being used from an unpacked repo instead of an installed skill, the target path is outside the current workspace, or the user needs a clear next-hardening plan.
When fallback mode is used, state it in the TL;DR. Do not pretend a manual scan is equivalent to the deterministic rule script. If deterministic rule checks cannot run, mark the report as provisional, keep any uncertain rule findings as na, and list the missing runtime or installation condition as residual risk.
Finding Schema
Use this schema for every rule and model finding:
{
"id": "3.4",
"section": "trigger",
"item": "body-only trigger anti-pattern absent",
"severity": "BLOCKER",
"status": "fail",
"checker": "model",
"why": "The description does not include the invocation condition.",
"how_to_fix": "Move both the skill capability and trigger condition into the description."
}
Allowed values:
severity:BLOCKER,MAJOR,MINORstatus:pass,fail,nachecker:rule,model
Report Rules
Lead with the grade and failed finding counts. Put failed BLOCKER findings first, failed MAJOR findings second, and failed MINOR findings last. For each failed finding, include both:
- why the issue matters
- how to fix it
Keep passed findings collapsed into section summaries unless the user asks for the full matrix.
State evaluation coverage beside the grade. Never present missing or na checks as PASS. Empty or invalid findings are input errors, not an S grade.
Resource Use
- Use
scripts/check_rules.pyfor deterministic checks, rule-only grading, and JSON output. - Use
scripts/run_checks.shas the first-choice launcher for deterministic checks across common Python command names. - Use
scripts/render_report.pywhen combined findings are available as JSON and a markdown report is useful. - Use
scripts/compare_results.pyto compare saved before/after results; only a failed check that now passes is resolved. scripts/results.pyprovides shared schema validation, coverage, provenance, and quality gates for these commands; it is imported, not run directly.- Read
references/automation.mdwhen comparing evaluations, evaluating multiple skills, using CI gates, or interpreting provenance. - Read
references/model-rubric.mdonly after running deterministic checks or when authoring model findings; it contains the 14 semantic checks, pass/fail criteria, examples, and wording guidance. - Read
references/fallbacks.mdonly when runtime, installation, path, or platform constraints prevent the normal workflow, or when the user asks for follow-up hardening work.
04 同仓库更多 Skills
unmentioned-script
Evaluate reusable sample skills when checking a complete rubric workflow.
todo-leftover
Evaluate reusable sample skills when checking a complete rubric workflow.
secret-leak
Evaluate reusable sample skills when checking a complete rubric workflow.
reserved-word
Evaluate Claude-style sample skills when checking a complete rubric workflow.
overbroad-trigger
Improve quality for any review, audit, documentation, or coding task. Use whenever anything needs checking or improvement.
operational-precondition
Reconcile CSV invoice exports against CSV bank transactions. Use when asked to match invoice payments or identify unpaid invoices from CSV exports.