skill-tester
Validate, test, and score the quality of skills within the claude-skills ecosystem. Comprehensive meta-skill: structure validation, Python script testing (syntax + imports + runtime + output format), multi-dimensional quality scoring with letter grades and tier classification (BASIC/STANDARD/POWERFUL). Use when authoring a new skill, auditing existing skills for tier promotion, setting up pre-commit hooks for skill quality, or integrating skill QA into CI.
- License 许可
- MIT
- 最近更新
- 2026-09-22
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- skill-tester
- description
- Validate, test, and score the quality of skills within the claude-skills ecosystem. Comprehensive meta-skill: structure validation, Python script testing (syntax + imports + runtime + output format), multi-dimensional quality scoring with letter grades and tier classification (BASIC/STANDARD/POWERFUL). Use when authoring a new skill, auditing existing skills for tier promotion, setting up pre-commit hooks for skill quality, or integrating skill QA into CI.
02 资源文件 10
- 参考文档expected_outputs/sample_validation_report.json1.7 KB
- 参考文档README.md6.7 KB
- 参考文档references/quality-scoring-rubric.md20.2 KB
- 参考文档references/skill-structure-specification.md11.7 KB
- 参考文档references/tier-requirements-matrix.md17.6 KB
- 脚本scripts/quality_scorer.py49.2 KB
- 脚本scripts/script_tester.py30.4 KB
- 脚本scripts/security_scorer.py22.9 KB
- 脚本scripts/skill_validator.py28.4 KB
- 脚本tests/test_security_scorer.py27.7 KB
03 SKILL.md 正文
Skill Tester
Name: skill-tester Tier: POWERFUL Category: Engineering Quality Assurance Dependencies: None (Python Standard Library Only) Author: Claude Skills Engineering Team Version: 1.0.0 Last Updated: 2026-02-16
Description
The Skill Tester is a comprehensive meta-skill designed to validate, test, and score the quality of skills within the claude-skills ecosystem. This powerful quality assurance tool ensures that all skills meet the rigorous standards required for BASIC, STANDARD, and POWERFUL tier classifications through automated validation, testing, and scoring mechanisms.
As the gatekeeping system for skill quality, this meta-skill provides three core capabilities:
- Structure Validation - Ensures skills conform to required directory structures, file formats, and documentation standards
- Script Testing - Validates Python scripts for syntax, imports, functionality, and output format compliance
- Quality Scoring - Provides comprehensive quality assessment across multiple dimensions with letter grades and improvement recommendations
This skill is essential for maintaining ecosystem consistency, enabling automated CI/CD integration, and supporting both manual and automated quality assurance workflows. It serves as the foundation for pre-commit hooks, pull request validation, and continuous integration processes that maintain the high-quality standards of the claude-skills repository.
Core Features
Comprehensive Skill Validation
- Structure Compliance: Validates directory structure, required files (SKILL.md, README.md, scripts/, references/, assets/, expected_outputs/)
- Documentation Standards: Checks SKILL.md frontmatter, section completeness, minimum line counts per tier
- File Format Validation: Ensures proper Markdown formatting, YAML frontmatter syntax, and file naming conventions
Advanced Script Testing
- Syntax Validation: Compiles Python scripts to detect syntax errors before execution
- Import Analysis: Enforces standard library only policy, identifies external dependencies
- Runtime Testing: Executes scripts with sample data, validates argparse implementation, tests --help functionality
- Output Format Compliance: Verifies dual output support (JSON + human-readable), proper error handling
Multi-Dimensional Quality Scoring
- Documentation Quality (25%): SKILL.md depth and completeness, README clarity, reference documentation quality
- Code Quality (25%): Script complexity, error handling robustness, output format consistency, maintainability
- Completeness (25%): Required directory presence, sample data adequacy, expected output verification
- Usability (25%): Example clarity, argparse help text quality, installation simplicity, user experience
Tier Classification System
Automatically classifies skills based on complexity and functionality:
BASIC Tier Requirements
- Minimum 100 lines in SKILL.md
- At least 1 Python script (100-300 LOC)
- Basic argparse implementation
- Simple input/output handling
- Essential documentation coverage
STANDARD Tier Requirements
- Minimum 200 lines in SKILL.md
- 1-2 Python scripts (300-500 LOC each)
- Advanced argparse with subcommands
- JSON + text output formats
- Comprehensive examples and references
- Error handling and edge case management
POWERFUL Tier Requirements
- Minimum 300 lines in SKILL.md
- 2-3 Python scripts (500-800 LOC each)
- Complex argparse with multiple modes
- Sophisticated output formatting and validation
- Extensive documentation and reference materials
- Advanced error handling and recovery mechanisms
- CI/CD integration capabilities
Architecture & Design
Modular Design Philosophy
The skill-tester follows a modular architecture where each component serves a specific validation purpose:
- skill_validator.py: Core structural and documentation validation engine
- script_tester.py: Runtime testing and execution validation framework
- quality_scorer.py: Multi-dimensional quality assessment and scoring system
Standards Enforcement
All validation is performed against well-defined standards documented in the references/ directory:
- Skill Structure Specification: Defines mandatory and optional components
- Tier Requirements Matrix: Detailed requirements for each skill tier
- Quality Scoring Rubric: Comprehensive scoring methodology and weightings
Integration Capabilities
Designed for seamless integration into existing development workflows:
- Pre-commit Hooks: Prevents substandard skills from being committed
- CI/CD Pipelines: Automated quality gates in pull request workflows
- Manual Validation: Interactive command-line tools for development-time validation
- Batch Processing: Bulk validation and scoring of existing skill repositories
Implementation Details
skill_validator.py Core Functions
# Primary validation workflow
validate_skill_structure() -> ValidationReport
check_skill_md_compliance() -> DocumentationReport
validate_python_scripts() -> ScriptReport
generate_compliance_score() -> float
Key validation checks include:
- SKILL.md frontmatter parsing and validation
- Required section presence (Description, Features, Usage, etc.)
- Minimum line count enforcement per tier
- Python script argparse implementation verification
- Standard library import enforcement
- Directory structure compliance
- README.md quality assessment
script_tester.py Testing Framework
# Core testing functions
syntax_validation() -> SyntaxReport
import_validation() -> ImportReport
runtime_testing() -> RuntimeReport
output_format_validation() -> OutputReport
Testing capabilities encompass:
- Python AST-based syntax validation
- Import statement analysis and external dependency detection
- Controlled script execution with timeout protection
- Argparse --help functionality verification
- Sample data processing and output validation
- Expected output comparison and difference reporting
quality_scorer.py Scoring System
# Multi-dimensional scoring
score_documentation() -> float # 25% weight
score_code_quality() -> float # 25% weight
score_completeness() -> float # 25% weight
score_usability() -> float # 25% weight
calculate_overall_grade() -> str # A-F grade
Scoring dimensions include:
- Documentation: Completeness, clarity, examples, reference quality
- Code Quality: Complexity, maintainability, error handling, output consistency
- Completeness: Required files, sample data, expected outputs, test coverage
- Usability: Help text quality, example clarity, installation simplicity
Usage Scenarios
Development Workflow Integration
# Pre-commit hook validation
skill_validator.py path/to/skill --tier POWERFUL --json
# Comprehensive skill testing
script_tester.py path/to/skill --timeout 30 --sample-data
# Quality assessment and scoring
quality_scorer.py path/to/skill --detailed --recommendations
CI/CD Pipeline Integration
# GitHub Actions workflow example
- name: "validate-skill-quality"
run: |
python skill_validator.py engineering/${{ matrix.skill }} --json | tee validation.json
python script_tester.py engineering/${{ matrix.skill }} | tee testing.json
python quality_scorer.py engineering/${{ matrix.skill }} --json | tee scoring.json
Batch Repository Analysis
# Validate all skills in repository
find engineering/ -type d -maxdepth 1 | xargs -I {} skill_validator.py {}
# Generate repository quality report
quality_scorer.py engineering/ --batch --output-format json > repo_quality.json
Output Formats & Reporting
Dual Output Support
All tools provide both human-readable and machine-parseable output:
Human-Readable Format
=== SKILL VALIDATION REPORT ===
Skill: engineering/example-skill
Tier: STANDARD
Overall Score: 85/100 (B)
Structure Validation: ✓ PASS
├─ SKILL.md: ✓ EXISTS (247 lines)
├─ README.md: ✓ EXISTS
├─ scripts/: ✓ EXISTS (2 files)
└─ references/: ⚠ MISSING (recommended)
Documentation Quality: 22/25 (88%)
Code Quality: 20/25 (80%)
Completeness: 18/25 (72%)
Usability: 21/25 (84%)
Recommendations:
• Add references/ directory with documentation
• Improve error handling in main.py
• Include more comprehensive examples
JSON Format
{
"skill_path": "engineering/example-skill",
"timestamp": "2026-02-16T16:41:00Z",
"validation_results": {
"structure_compliance": {
"score": 0.95,
"checks": {
"skill_md_exists": true,
"readme_exists": true,
"scripts_directory": true,
"references_directory": false
}
},
"overall_score": 85,
"letter_grade": "B",
"tier_recommendation": "STANDARD",
"improvement_suggestions": [
"Add references/ directory",
"Improve error handling",
"Include comprehensive examples"
]
}
}
Quality Assurance Standards
Code Quality Requirements
- Standard Library Only: No external dependencies (pip packages)
- Error Handling: Comprehensive exception handling with meaningful error messages
- Output Consistency: Standardized JSON schema and human-readable formatting
- Performance: Efficient validation algorithms with reasonable execution time
- Maintainability: Clear code structure, comprehensive docstrings, type hints where appropriate
Testing Standards
- Self-Testing: The skill-tester validates itself (meta-validation)
- Sample Data Coverage: Comprehensive test cases covering edge cases and error conditions
- Expected Output Verification: All sample runs produce verifiable, reproducible outputs
- Timeout Protection: Safe execution of potentially problematic scripts with timeout limits
Documentation Standards
- Comprehensive Coverage: All functions, classes, and modules documented
- Usage Examples: Clear, practical examples for all use cases
- Integration Guides: Step-by-step CI/CD and workflow integration instructions
- Reference Materials: Complete specification documents for standards and requirements
Integration Examples
Pre-Commit Hook Setup
#!/bin/bash
# .git/hooks/pre-commit
echo "Running skill validation..."
python engineering/skill-tester/scripts/skill_validator.py engineering/new-skill --tier STANDARD
if [ $? -ne 0 ]; then
echo "Skill validation failed. Commit blocked."
exit 1
fi
echo "Validation passed. Proceeding with commit."
GitHub Actions Workflow
name: "skill-quality-gate"
on:
pull_request:
paths: ['engineering/**']
jobs:
validate-skills:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: "setup-python"
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: "validate-changed-skills"
run: |
changed_skills=$(git diff --name-only ${{ github.event.before }} | grep -E '^engineering/[^/]+/' | cut -d'/' -f1-2 | sort -u)
for skill in $changed_skills; do
echo "Validating $skill..."
python engineering/skill-tester/scripts/skill_validator.py $skill --json
python engineering/skill-tester/scripts/script_tester.py $skill
python engineering/skill-tester/scripts/quality_scorer.py $skill --minimum-score 75
done
Continuous Quality Monitoring
#!/bin/bash
# Daily quality report generation
echo "Generating daily skill quality report..."
timestamp=$(date +"%Y-%m-%d")
python engineering/skill-tester/scripts/quality_scorer.py engineering/ \
--batch --json > "reports/quality_report_${timestamp}.json"
echo "Quality trends analysis..."
python engineering/skill-tester/scripts/trend_analyzer.py reports/ \
--days 30 > "reports/quality_trends_${timestamp}.md"
Performance & Scalability
Execution Performance
- Fast Validation: Structure validation completes in <1 second per skill
- Efficient Testing: Script testing with timeout protection (configurable, default 30s)
- Batch Processing: Optimized for repository-wide analysis with parallel processing support
- Memory Efficiency: Minimal memory footprint for large-scale repository analysis
Scalability Considerations
- Repository Size: Designed to handle repositories with 100+ skills
- Concurrent Execution: Thread-safe implementation supports parallel validation
- Resource Management: Automatic cleanup of temporary files and subprocess resources
- Configuration Flexibility: Configurable timeouts, memory limits, and validation strictness
Security & Safety
Safe Execution Environment
- Sandboxed Testing: Scripts execute in controlled environment with timeout protection
- Resource Limits: Memory and CPU usage monitoring to prevent resource exhaustion
- Input Validation: All inputs sanitized and validated before processing
- No Network Access: Offline operation ensures no external dependencies or network calls
Security Best Practices
- No Code Injection: Static analysis only, no dynamic code generation
- Path Traversal Protection: Secure file system access with path validation
- Minimal Privileges: Operates with minimal required file system permissions
- Audit Logging: Comprehensive logging for security monitoring and troubleshooting
Troubleshooting & Support
Common Issues & Solutions
Validation Failures
- Missing Files: Check directory structure against tier requirements
- Import Errors: Ensure only standard library imports are used
- Documentation Issues: Verify SKILL.md frontmatter and section completeness
Script Testing Problems
- Timeout Errors: Increase timeout limit or optimize script performance
- Execution Failures: Check script syntax and import statement validity
- Output Format Issues: Ensure proper JSON formatting and dual output support
Quality Scoring Discrepancies
- Low Scores: Review scoring rubric and improvement recommendations
- Tier Misclassification: Verify skill complexity against tier requirements
- Inconsistent Results: Check for recent changes in quality standards or scoring weights
Debugging Support
- Verbose Mode: Detailed logging and execution tracing available
- Dry Run Mode: Validation without execution for debugging purposes
- Debug Output: Comprehensive error reporting with file locations and suggestions
Future Enhancements
Planned Features
- Machine Learning Quality Prediction: AI-powered quality assessment using historical data
- Performance Benchmarking: Execution time and resource usage tracking across skills
- Dependency Analysis: Automated detection and validation of skill interdependencies
- Quality Trend Analysis: Historical quality tracking and regression detection
Integration Roadmap
- IDE Plugins: Real-time validation in popular development environments
- Web Dashboard: Centralized quality monitoring and reporting interface
- API Endpoints: RESTful API for external integration and automation
- Notification Systems: Automated alerts for quality degradation or validation failures
Conclusion
The Skill Tester represents a critical infrastructure component for maintaining the high-quality standards of the claude-skills ecosystem. By providing comprehensive validation, testing, and scoring capabilities, it ensures that all skills meet or exceed the rigorous requirements for their respective tiers.
This meta-skill not only serves as a quality gate but also as a development tool that guides skill authors toward best practices and helps maintain consistency across the entire repository. Through its integration capabilities and comprehensive reporting, it enables both manual and automated quality assurance workflows that scale with the growing claude-skills ecosystem.
The combination of structural validation, runtime testing, and multi-dimensional quality scoring provides unparalleled visibility into skill quality while maintaining the flexibility needed for diverse skill types and complexity levels. As the claude-skills repository continues to grow, the Skill Tester will remain the cornerstone of quality assurance and ecosystem integrity.
04 同仓库更多 Skills
syllabus
Generates a curated supplementary reading list from any course syllabus using Consensus academic search. Grill-me intake (syllabus input format + course audience + year range) plus a grouping forcing-options checkpoint before any search runs — so the reading list matches the course's level and recency need. Parses the syllabus to extract topics and learning outcomes, searches Consensus for recent peer-reviewed papers per topic, and produces a professionally formatted .docx with clickable Consensus links, plain-language summaries calibrated to audience level, and Bloom-higher-order discussion questions tied to course learning goals. Triggers whenever a user uploads a syllabus, course outline, or curriculum document and wants supplementary readings. Also triggers on: 'syllabus reading list', 'find papers for my course', 'create a reading list from this syllabus', 'recent research for my class', 'supplementary readings', 'find journal articles for these topics', 'what recent papers cover this material', 'any new research on these course topics', 'update my syllabus with recent papers'. Even casual mentions when a syllabus is attached should trigger this skill.
research
Default entry point for any research request — a hybrid router that classifies the question deterministically and either delegates to a specialist research skill (pulse for trends/sentiment, grants for NIH funding, litreview for academic literature, syllabus for course reading, patent for prior-art + IP landscape, dossier for entity research) or runs its own plan-decompose-multi-source-search-synthesize-cite fallback workflow when no specialist matches. Always surfaces the routing decision so users can override. Triggers — "research [topic]", "look into [topic]", "what do we know about [topic]", "investigate [topic]", "find me information on [topic]", "do some research on [topic]", "I need to understand [topic]", or any research request that doesn't obviously match a more-specific specialist skill. Output is a markdown briefing (default) or .docx document (on request) with full citations and an audit log.
pulse
Multi-source recency research skill that takes the pulse of any topic across Reddit, Hacker News, the open web, and optionally X/Twitter within a configurable recent window (default 30 days). Forcing intake clarifies topic specificity, angle (trend/sentiment/problems/opportunities/comparison), time window, and platform scope before searching. Returns a synthesized briefing with citations, engagement metrics, and cross-platform pattern analysis. Triggers: 'pulse on [topic]', 'what's happening with [topic]', 'what are people saying about [topic]', 'current conversation about [topic]', 'take the pulse of [topic]', 'trending: [topic]', 'find me info on [topic]', or any variation requesting multi-source recency intelligence on a topic. Also use for competitor research, trend discovery, tool comparisons, and audience sentiment analysis.
patent
Patent prior-art and landscape intelligence skill — not generic patent help. Commits to one of five sub-use-cases via forcing intake (novelty search / freedom-to-operate / competitive landscape / acquisition diligence / litigation prior-art) before any search runs. Searches Google Patents, Espacenet, USPTO, and optionally Lens.org for citation-graph signals. Output is an editable Word document (.docx) with verdict, ranked closest art (claim-text extracted), CPC-class-aware landscape, family-resolved hits, geographic coverage, FTO flags where applicable, strategy recommendations, and full audit log. Triggers: 'prior art search for [invention]', 'patent search on [topic]', 'freedom to operate analysis', 'FTO for [product]', 'patent landscape for [field]', 'is [invention] novel', 'patents on [topic]', 'competitive patent analysis', 'prior art for litigation', 'patent diligence on [company]'. Produces search signal, not legal advice — always recommends consulting a patent attorney before filing or licensing decisions. Trademark, copyright, and trade-secret questions are out of scope.
notebooklm
Browser automation skill for controlling Google's NotebookLM. Handles reading and querying notebooks, adding sources (URLs, text, files, YouTube links, synthesized content), generating Studio outputs (Audio Overview, infographics, slide decks, study guides, briefing docs, mind maps, timelines, FAQs), and creating new notebooks. Triggers on any phrase involving NotebookLM — 'open NotebookLM', 'check my [name] notebook', 'pull info from NotebookLM', 'ask my notebook about X', 'add [source] to NotebookLM', 'create an infographic in NotebookLM', 'use NotebookLM Studio', 'generate a slide deck from my notebook', or any variation where the goal involves NotebookLM. Requires browser automation environment — fails gracefully when unavailable.
litreview
Academic literature orientation skill that searches papers via Consensus, builds a strategic search plan using PICO (default) or SPIDER / Decomposition / hybrid as fallbacks, and synthesizes findings into a professionally formatted Word document (.docx) research guide. Grill-me intake (research question specificity + framework hint + tentative depth) before the recon search; a second forcing checkpoint after Phase 2 confirms framework + sub-areas + depth before searches consume budget. Configurable depth (5/10/20 queries) controls coverage vs. speed. Output is a 'launching pad' — not a finished review, but an orientation guide that lets a researcher dive in confidently. Triggers: 'litreview on [topic]', 'literature review on [topic]', 'I'm starting a literature review on X', 'I'm writing a paper on X', 'help me research X', 'I'm doing research on X', 'can you help me research X'. Do NOT trigger for single one-off paper searches where the user just wants a quick list — that's a plain Consensus search.