api-benchmark
Benchmark token generation speed across multiple LLM API providers. Measures TTFT (Time To First Token), tokens-per-second throughput, and total generation time. Use when comparing performance of different API providers, models, or testing API connectivity. Requires OpenCLAW config with provider definitions.
- License 许可
- MIT
- 最近更新
- 2026-06-07
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- api-benchmark
- metadata
- { "author": "Polar", "version": "1.0.2", "requires": { "env": [ "OPENCLAW_CONFIG" ], "config": [ "~/.openclaw/openclaw.json" ] } }
- description
- Benchmark token generation speed across multiple LLM API providers. Measures TTFT (Time To First Token), tokens-per-second throughput, and total generation time. Use when comparing performance of different API providers, models, or testing API connectivity. Requires OpenCLAW config with provider definitions.
- compatibility
- Requires Python 3 with requests library. Reads target configuration from ~/.openclaw/openclaw.json. Supports anthropic-messages, openai-completions, and openai-responses API formats.
03 SKILL.md 正文
API Token Speed Benchmark
This skill benchmarks token generation speed across multiple LLM API providers.
When to use this skill
Use this skill when you need to:
- Compare token generation speed across different API providers
- Measure latency and throughput of LLM models
- Verify API connectivity and authentication
- Test new API endpoints or models
How to run benchmarks
List available targets
python3 main.py --targets
Run benchmark on a specific target
python3 main.py run --label <target-label>
Run benchmark on all targets
python3 main.py run --all
Run preflight check (verify API connectivity)
python3 main.py check --label <target-label>
python3 main.py check --all
Options
-l, --label: Specific target label to benchmark-a, --all: Run on all available targets-r, --repeat: Number of runs per prompt level (default: 1)-c, --category: Run specific prompt category (can repeat: -c short -c medium). Options: short, medium, long-q, --quiet: Quiet mode - suppress progress output--timeout N: Request timeout in seconds (default: 120)--table: Output as formatted table (default: JSON)
Configuration
The tool reads configuration from ~/.openclaw/openclaw.json. Targets are defined in the models.providers section with:
baseUrl: API base URLapiKey: Authentication key (or${ENV_VAR}to read from environment variable)api: API format (anthropic-messages, openai-completions, openai-responses)models: List of model configurations
Security Note: Instead of hardcoding API keys in the config file, use environment variable placeholders:
"apiKey": "${ANTHROPIC_API_KEY}"will read from theANTHROPIC_API_KEYenvironment variable
Example provider config:
{
"models": {
"providers": {
"my-provider": {
"baseUrl": "https://api.example.com",
"apiKey": "sk-xxx",
"api": "openai-completions",
"models": [
{ "id": "model-name", "api": "openai-completions" }
]
}
}
}
}
Output Metrics
- TTFT (Time To First Token): Latency before first token arrives (seconds)
- TPS (Tokens Per Second): Generation throughput
- Total Time: Full generation duration (seconds)
- Input/Output Tokens: Token counts from API usage data (or estimated at 4 chars/token if not provided by API)
Note: Token counts are reported by the API when available. If the API doesn't return token counts, they are estimated at 4 characters per token.
Example Usage
# Check if a specific target is reachable
python3 main.py check --label my-provider
# Benchmark a single target
python3 main.py run --label my-provider --repeat 3
# Compare all targets
python3 main.py run --all --table
04 同仓库更多 Skills
crowd-prompting
A marketplace where AI agents improve prompts, system instructions, tool descriptions, and other text-based content with domain expertise from real-world operations — and earn tokens for valuable contributions.
ceaser
Interact with the Ceaser privacy protocol on Base L2 using the ceaser-mcp MCP tools. This skill uses the ceaser-mcp npm package for ALL operations -- shield, unshield, note management, and protocol queries. All ceaser tool calls use CLI subcommands (npx -y ceaser-mcp <subcommand>). Ceaser is a privacy-preserving ETH wrapper using Noir/UltraHonk zero-knowledge proofs on Base L2 (chain ID 8453).
ceaser-send
Fully automated private ETH transfer via Ceaser Protocol on Base L2 using the ceaser-mcp MCP tools. This skill uses the ceaser-mcp npm package for ALL operations -- all ceaser tool calls use CLI subcommands (npx -y ceaser-mcp <subcommand>). WARNING -- creates observable on-chain links between user wallet and hot wallet (see Privacy Warning). Generates an ephemeral hot wallet with BIP-39 mnemonic recovery, user funds it, agent signs and broadcasts the Shield TX automatically. Exactly one manual step required (funding the hot wallet with ETH). Uses Noir/UltraHonk zero-knowledge proofs.
dilemm-ai
Compete in DilemmAI, the prisoner's dilemma AI arena at dilemm.ai. Use when an OpenClaw agent wants to create an account, design and submit strategy prompts for their bot, enter matchmaking, analyze historical matches and leaderboards, and iterate on bot strategy. Requires headless browser automation (no API available yet).
moldium
Post and manage content on the Moldium blog platform. Triggered by "post to Moldium", "write a blog post", "publish an article", etc.
arxiv-paper-reviews
Interact with arXiv Crawler API to fetch papers, read reviews, submit comments, search papers, and import papers. Use when working with arXiv papers, fetching paper lists by date/category/interest, viewing paper details with comments, submitting paper reviews, searching papers by title, or importing papers from arXiv URLs via API at http://weakaccept.top:8000/.