skill-comply
Visualize whether skills, rules, and agent definitions are actually followed — auto-generates scenarios at 3 prompt strictness levels, runs agents, classifies behavioral sequences, and reports compliance rates with full tool call timelines
- License 许可
- MIT
- 最近更新
- 2026-11-15
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- skill-comply
- tools
- Read, Bash
- metadata
- { "origin": "ECC" }
- description
- Visualize whether skills, rules, and agent definitions are actually followed — auto-generates scenarios at 3 prompt strictness levels, runs agents, classifies behavioral sequences, and reports compliance rates with full tool call timelines
02 资源文件 21
- 其他.gitignore76 B
- 其他fixtures/compliant_trace.jsonl1.0 KB
- 其他fixtures/noncompliant_trace.jsonl629 B
- 参考文档fixtures/tdd_spec.yaml1.2 KB
- 参考文档prompts/classifier.md1.0 KB
- 参考文档prompts/scenario_generator.md1.9 KB
- 参考文档prompts/spec_generator.md1.6 KB
- 其他pyproject.toml299 B
- 脚本scripts/__init__.py—
- 脚本scripts/classifier.py2.4 KB
- 脚本scripts/grader.py4.3 KB
- 脚本scripts/parser.py2.7 KB
- 脚本scripts/report.py6.3 KB
- 脚本scripts/run.py4.2 KB
- 脚本scripts/runner.py6.9 KB
- 脚本scripts/scenario_generator.py1.8 KB
- 脚本scripts/spec_generator.py2.2 KB
- 脚本scripts/utils.py411 B
- 脚本tests/test_grader.py8.0 KB
- 脚本tests/test_parser.py3.1 KB
- 脚本tests/test_runner.py6.5 KB
03 SKILL.md 正文
skill-comply: Automated Compliance Measurement
Measures whether coding agents actually follow skills, rules, or agent definitions by:
- Auto-generating expected behavioral sequences (specs) from any .md file
- Auto-generating scenarios with decreasing prompt strictness (supportive → neutral → competing)
- Running
claude -pand capturing tool call traces via stream-json - Classifying tool calls against spec steps using LLM (not regex)
- Checking temporal ordering deterministically
- Generating self-contained reports with spec, prompts, and timelines
Supported Targets
- Skills (
skills/*/SKILL.md): Workflow skills like search-first, TDD guides - Rules (
rules/common/*.md): Mandatory rules like testing.md, security.md, git-workflow.md - Agent definitions (
agents/*.md): Whether an agent gets invoked when expected (internal workflow verification not yet supported)
When to Activate
- User runs
/skill-comply <path> - User asks "is this rule actually being followed?"
- After adding new rules/skills, to verify agent compliance
- Periodically as part of quality maintenance
Usage
# Full run
uv run python -m scripts.run ~/.claude/rules/common/testing.md
# Dry run (no cost, spec + scenarios only)
uv run python -m scripts.run --dry-run ~/.claude/skills/search-first/SKILL.md
# Custom models
uv run python -m scripts.run --gen-model haiku --model sonnet <path>
Key Concept: Prompt Independence
Measures whether a skill/rule is followed even when the prompt doesn't explicitly support it.
Report Contents
Reports are self-contained and include:
- Expected behavioral sequence (auto-generated spec)
- Scenario prompts (what was asked at each strictness level)
- Compliance scores per scenario
- Tool call timelines with LLM classification labels
Advanced (optional)
For users familiar with hooks, reports also include hook promotion recommendations for steps with low compliance. This is informational — the main value is the compliance visibility itself.
04 同仓库更多 Skills
rag-architect-agent
Agente Arquiteto de RAG, Embeddings & Busca Vetorial. Especialista em design e implementação de pipelines de Retrieval-Augmented Generation de alta fidelidade: chunking semântico, modelagem pgvector no PostgreSQL (Neon/Supabase), busca híbrida (cosseno + BM25), reranking, validação via TypeSafe AI e filosofia Ponytail (anti-overengineering).
rag-architecture-patterns
Padrões arquiteturais e implementação de pipelines RAG (Retrieval-Augmented Generation) de alta fidelidade para qualquer projeto ou sistema. Cobre chunking semântico, embeddings, busca híbrida (pgvector + BM25), reranking, validação via TypeSafe AI e filosofia Ponytail (anti-overengineering).
skill-repair
Use this to fix and re-install agent skills that have failed installation. This skill provides the necessary context and permissions to surgically update the `manifest.json` after a fix has been applied.
security-scan
Scan your Claude Code configuration (.claude/ directory) for security vulnerabilities, misconfigurations, and injection risks using AgentShield. Checks CLAUDE.md, settings.json, MCP servers, hooks, and agent definitions.
security-review
Use this skill when adding authentication, handling user input, working with secrets, creating API endpoints, or implementing payment/sensitive features. Provides comprehensive security checklist and patterns.
security-bounty-hunter
Hunt for exploitable, bounty-worthy security issues in repositories. Focuses on remotely reachable vulnerabilities that qualify for real reports instead of noisy local-only findings.