[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6F3PHE82GASFRF531ZG2T70":3,"skill-same:01M6F3PHE82GASFRF531ZG2T70":75},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":26,"publishedAt":47,"linkOnly":48,"markdownContent":49,"markdownBody":50,"frontMatter":51,"refName":52,"commitSha":53,"skillDirectory":54,"forksCount":55,"repositoryHtmlUrl":56,"repositoryHomepageUrl":57,"ownerHtmlUrl":58,"ownerBlogUrl":6,"ownerDescription":59,"updatedAt":60,"install":61,"resources":74},"01M6F3PHE82GASFRF531ZG2T70","promptfoo-evaluation",null,"Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom assertions, implementing llm-rubric for LLM-as-judge, or managing few-shot examples in prompts. Triggers on keywords like \"promptfoo\", \"eval\", \"LLM evaluation\", \"prompt testing\", or \"model comparison\".","ComeOnOliver\u002Fskillshub","ComeOnOliver","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F43215937?v=4","01M6E5JNFECJAAVFE0W7SMWWD2","Haoran Shu","01M6E5JNNDQS7AJZDYFQ8TPZCK",67,"MIT","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fdaymade\u002Fclaude-code-skills\u002Fpromptfoo-evaluation\u002FSKILL.md",[18,23],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000008","测试与质量","SCENARIO","使用场景",{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发",[27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46],"promptfoo evaluation","Evaluation","promptfoo","rubric","comparison","judge","few-shot","as","triggers","writing python","up","use","testing","prompt","eval","model","examples","framework","python","writing","2026-11-10 10:12:30",false,"---\nname: promptfoo-evaluation\ndescription: Configures and runs LLM evaluation using Promptfoo framework. Use when setting up prompt testing, creating evaluation configs (promptfooconfig.yaml), writing Python custom assertions, implementing llm-rubric for LLM-as-judge, or managing few-shot examples in prompts. Triggers on keywords like \"promptfoo\", \"eval\", \"LLM evaluation\", \"prompt testing\", or \"model comparison\".\n---\n\n# Promptfoo Evaluation\n\n## Overview\n\nThis skill provides guidance for configuring and running LLM evaluations using [Promptfoo](https:\u002F\u002Fwww.promptfoo.dev\u002F), an open-source CLI tool for testing and comparing LLM outputs.\n\n## Quick Start\n\n```bash\n# Initialize a new evaluation project\nnpx promptfoo@latest init\n\n# Run evaluation\nnpx promptfoo@latest eval\n\n# View results in browser\nnpx promptfoo@latest view\n```\n\n## Configuration Structure\n\nA typical Promptfoo project structure:\n\n```\nproject\u002F\n├── promptfooconfig.yaml    # Main configuration\n├── prompts\u002F\n│   ├── system.md           # System prompt\n│   └── chat.json           # Chat format prompt\n├── tests\u002F\n│   └── cases.yaml          # Test cases\n└── scripts\u002F\n    └── metrics.py          # Custom Python assertions\n```\n\n## Core Configuration (promptfooconfig.yaml)\n\n```yaml\n# yaml-language-server: $schema=https:\u002F\u002Fpromptfoo.dev\u002Fconfig-schema.json\ndescription: \"My LLM Evaluation\"\n\n# Prompts to test\nprompts:\n  - file:\u002F\u002Fprompts\u002Fsystem.md\n  - file:\u002F\u002Fprompts\u002Fchat.json\n\n# Models to compare\nproviders:\n  - id: anthropic:messages:claude-sonnet-4-6\n    label: Claude-Sonnet-4.6\n  - id: openai:gpt-4.1\n    label: GPT-4.1\n\n# Test cases\ntests: file:\u002F\u002Ftests\u002Fcases.yaml\n\n# Concurrency control (MUST be under commandLineOptions, NOT top-level)\ncommandLineOptions:\n  maxConcurrency: 2\n\n# Default assertions for all tests\ndefaultTest:\n  assert:\n    - type: python\n      value: file:\u002F\u002Fscripts\u002Fmetrics.py:custom_assert\n    - type: llm-rubric\n      value: |\n        Evaluate the response quality on a 0-1 scale.\n      threshold: 0.7\n\n# Output path\noutputPath: results\u002Feval-results.json\n```\n\n## Prompt Formats\n\n### Text Prompt (system.md)\n\n```markdown\nYou are a helpful assistant.\n\nTask: {{task}}\nContext: {{context}}\n```\n\n### Chat Format (chat.json)\n\n```json\n[\n  {\"role\": \"system\", \"content\": \"{{system_prompt}}\"},\n  {\"role\": \"user\", \"content\": \"{{user_input}}\"}\n]\n```\n\n### Few-Shot Pattern\n\nEmbed examples directly in prompt or use chat format with assistant messages:\n\n```json\n[\n  {\"role\": \"system\", \"content\": \"{{system_prompt}}\"},\n  {\"role\": \"user\", \"content\": \"Example input: {{example_input}}\"},\n  {\"role\": \"assistant\", \"content\": \"{{example_output}}\"},\n  {\"role\": \"user\", \"content\": \"Now process: {{actual_input}}\"}\n]\n```\n\n## Test Cases (tests\u002Fcases.yaml)\n\n```yaml\n- description: \"Test case 1\"\n  vars:\n    system_prompt: file:\u002F\u002Fprompts\u002Fsystem.md\n    user_input: \"Hello world\"\n    # Load content from files\n    context: file:\u002F\u002Fdata\u002Fcontext.txt\n  assert:\n    - type: contains\n      value: \"expected text\"\n    - type: python\n      value: file:\u002F\u002Fscripts\u002Fmetrics.py:custom_check\n      threshold: 0.8\n```\n\n## Python Custom Assertions\n\nCreate a Python file for custom assertions (e.g., `scripts\u002Fmetrics.py`):\n\n```python\ndef get_assert(output: str, context: dict) -> dict:\n    \"\"\"Default assertion function.\"\"\"\n    vars_dict = context.get('vars', {})\n\n    # Access test variables\n    expected = vars_dict.get('expected', '')\n\n    # Return result\n    return {\n        \"pass\": expected in output,\n        \"score\": 0.8,\n        \"reason\": \"Contains expected content\",\n        \"named_scores\": {\"relevance\": 0.9}\n    }\n\ndef custom_check(output: str, context: dict) -> dict:\n    \"\"\"Custom named assertion.\"\"\"\n    word_count = len(output.split())\n    passed = 100 \u003C= word_count \u003C= 500\n\n    return {\n        \"pass\": passed,\n        \"score\": min(1.0, word_count \u002F 300),\n        \"reason\": f\"Word count: {word_count}\"\n    }\n```\n\n**Key points:**\n- Default function name is `get_assert`\n- Specify function with `file:\u002F\u002Fpath.py:function_name`\n- Return `bool`, `float` (score), or `dict` with pass\u002Fscore\u002Freason\n- Access variables via `context['vars']`\n\n## LLM-as-Judge (llm-rubric)\n\n```yaml\nassert:\n  - type: llm-rubric\n    value: |\n      Evaluate the response based on:\n      1. Accuracy of information\n      2. Clarity of explanation\n      3. Completeness\n\n      Score 0.0-1.0 where 0.7+ is passing.\n    threshold: 0.7\n    provider: openai:gpt-4.1  # Optional: override grader model\n```\n\n**When using a relay\u002Fproxy API**, each `llm-rubric` assertion needs its own `provider` config with `apiBaseUrl`. Otherwise the grader falls back to the default Anthropic\u002FOpenAI endpoint and gets 401 errors:\n\n```yaml\nassert:\n  - type: llm-rubric\n    value: |\n      Evaluate quality on a 0-1 scale.\n    threshold: 0.7\n    provider:\n      id: anthropic:messages:claude-sonnet-4-6\n      config:\n        apiBaseUrl: https:\u002F\u002Fyour-relay.example.com\u002Fapi\n```\n\n**Best practices:**\n- Provide clear scoring criteria\n- Use `threshold` to set minimum passing score\n- Default grader uses available API keys (OpenAI → Anthropic → Google)\n- **When using relay\u002Fproxy**: every `llm-rubric` must have its own `provider` with `apiBaseUrl` — the main provider's `apiBaseUrl` is NOT inherited\n\n## Common Assertion Types\n\n| Type | Usage | Example |\n|------|-------|---------|\n| `contains` | Check substring | `value: \"hello\"` |\n| `icontains` | Case-insensitive | `value: \"HELLO\"` |\n| `equals` | Exact match | `value: \"42\"` |\n| `regex` | Pattern match | `value: \"\\\\d{4}\"` |\n| `python` | Custom logic | `value: file:\u002F\u002Fscript.py` |\n| `llm-rubric` | LLM grading | `value: \"Is professional\"` |\n| `latency` | Response time | `threshold: 1000` |\n\n## File References\n\nAll `file:\u002F\u002F` paths are resolved relative to `promptfooconfig.yaml` location (NOT the YAML file containing the reference). This is a common gotcha when `tests:` references a separate YAML file — the `file:\u002F\u002F` paths inside that test file still resolve from the config root.\n\n```yaml\n# Load file content as variable\nvars:\n  content: file:\u002F\u002Fdata\u002Finput.txt\n\n# Load prompt from file\nprompts:\n  - file:\u002F\u002Fprompts\u002Fmain.md\n\n# Load test cases from file\ntests: file:\u002F\u002Ftests\u002Fcases.yaml\n\n# Load Python assertion\nassert:\n  - type: python\n    value: file:\u002F\u002Fscripts\u002Fcheck.py:validate\n```\n\n## Running Evaluations\n\n```bash\n# Basic run\nnpx promptfoo@latest eval\n\n# With specific config\nnpx promptfoo@latest eval --config path\u002Fto\u002Fconfig.yaml\n\n# Output to file\nnpx promptfoo@latest eval --output results.json\n\n# Filter tests\nnpx promptfoo@latest eval --filter-metadata category=math\n\n# View results\nnpx promptfoo@latest view\n```\n\n## Relay \u002F Proxy API Configuration\n\nWhen using an API relay or proxy instead of direct Anthropic\u002FOpenAI endpoints:\n\n```yaml\nproviders:\n  - id: anthropic:messages:claude-sonnet-4-6\n    label: Claude-Sonnet-4.6\n    config:\n      max_tokens: 4096\n      apiBaseUrl: https:\u002F\u002Fyour-relay.example.com\u002Fapi  # Promptfoo appends \u002Fv1\u002Fmessages\n\n# CRITICAL: maxConcurrency MUST be under commandLineOptions (NOT top-level)\ncommandLineOptions:\n  maxConcurrency: 1  # Respect relay rate limits\n```\n\n**Key rules:**\n- `apiBaseUrl` goes in `providers[].config` — Promptfoo appends `\u002Fv1\u002Fmessages` automatically\n- `maxConcurrency` must be under `commandLineOptions:` — placing it at top level is silently ignored\n- When using relay with LLM-as-judge, set `maxConcurrency: 1` to avoid concurrent request limits (generation + grading share the same pool)\n- Pass relay token as `ANTHROPIC_API_KEY` env var\n\n## Troubleshooting\n\n**Python not found:**\n```bash\nexport PROMPTFOO_PYTHON=python3\n```\n\n**Large outputs truncated:**\nOutputs over 30000 characters are truncated. Use `head_limit` in assertions.\n\n**File not found errors:**\nAll `file:\u002F\u002F` paths resolve relative to `promptfooconfig.yaml` location.\n\n**maxConcurrency ignored (shows \"up to N at a time\"):**\n`maxConcurrency` must be under `commandLineOptions:`, not at the YAML top level. This is a common mistake.\n\n**LLM-as-judge returns 401 with relay API:**\nEach `llm-rubric` assertion must have its own `provider` with `apiBaseUrl`. The main provider config is not inherited by grader assertions.\n\n**HTML tags in model output inflating metrics:**\nModels may output `\u003Cbr>`, `\u003Cb>`, etc. in structured content. Strip HTML in Python assertions before measuring:\n```python\nimport re\nclean_text = re.sub(r'\u003C[^>]+>', '', raw_text)\n```\n\n## Echo Provider (Preview Mode)\n\nUse the **echo provider** to preview rendered prompts without making API calls:\n\n```yaml\n# promptfooconfig-preview.yaml\nproviders:\n  - echo  # Returns prompt as output, no API calls\n\ntests:\n  - vars:\n      input: \"test content\"\n```\n\n**Use cases:**\n- Preview prompt rendering before expensive API calls\n- Verify Few-shot examples are loaded correctly\n- Debug variable substitution issues\n- Validate prompt structure\n\n```bash\n# Run preview mode\nnpx promptfoo@latest eval --config promptfooconfig-preview.yaml\n```\n\n**Cost:** Free - no API tokens consumed.\n\n## Advanced Few-Shot Implementation\n\n### Multi-turn Conversation Pattern\n\nFor complex few-shot learning with full examples:\n\n```json\n[\n  {\"role\": \"system\", \"content\": \"{{system_prompt}}\"},\n\n  \u002F\u002F Few-shot Example 1\n  {\"role\": \"user\", \"content\": \"Task: {{example_input_1}}\"},\n  {\"role\": \"assistant\", \"content\": \"{{example_output_1}}\"},\n\n  \u002F\u002F Few-shot Example 2 (optional)\n  {\"role\": \"user\", \"content\": \"Task: {{example_input_2}}\"},\n  {\"role\": \"assistant\", \"content\": \"{{example_output_2}}\"},\n\n  \u002F\u002F Actual test\n  {\"role\": \"user\", \"content\": \"Task: {{actual_input}}\"}\n]\n```\n\n**Test case configuration:**\n\n```yaml\ntests:\n  - vars:\n      system_prompt: file:\u002F\u002Fprompts\u002Fsystem.md\n      # Few-shot examples\n      example_input_1: file:\u002F\u002Fdata\u002Fexamples\u002Finput1.txt\n      example_output_1: file:\u002F\u002Fdata\u002Fexamples\u002Foutput1.txt\n      example_input_2: file:\u002F\u002Fdata\u002Fexamples\u002Finput2.txt\n      example_output_2: file:\u002F\u002Fdata\u002Fexamples\u002Foutput2.txt\n      # Actual test\n      actual_input: file:\u002F\u002Fdata\u002Ftest1.txt\n```\n\n**Best practices:**\n- Use 1-3 few-shot examples (more may dilute effectiveness)\n- Ensure examples match the task format exactly\n- Load examples from files for better maintainability\n- Use echo provider first to verify structure\n\n## Long Text Handling\n\nFor Chinese\u002Flong-form content evaluations (10k+ characters):\n\n**Configuration:**\n\n```yaml\nproviders:\n  - id: anthropic:messages:claude-sonnet-4-6\n    config:\n      max_tokens: 8192  # Increase for long outputs\n\ndefaultTest:\n  assert:\n    - type: python\n      value: file:\u002F\u002Fscripts\u002Fmetrics.py:check_length\n```\n\n**Python assertion for text metrics:**\n\n```python\nimport re\n\ndef strip_tags(text: str) -> str:\n    \"\"\"Remove HTML tags for pure text.\"\"\"\n    return re.sub(r'\u003C[^>]+>', '', text)\n\ndef check_length(output: str, context: dict) -> dict:\n    \"\"\"Check output length constraints.\"\"\"\n    raw_input = context['vars'].get('raw_input', '')\n\n    input_len = len(strip_tags(raw_input))\n    output_len = len(strip_tags(output))\n\n    reduction_ratio = 1 - (output_len \u002F input_len) if input_len > 0 else 0\n\n    return {\n        \"pass\": 0.7 \u003C= reduction_ratio \u003C= 0.9,\n        \"score\": reduction_ratio,\n        \"reason\": f\"Reduction: {reduction_ratio:.1%} (target: 70-90%)\",\n        \"named_scores\": {\n            \"input_length\": input_len,\n            \"output_length\": output_len,\n            \"reduction_ratio\": reduction_ratio\n        }\n    }\n```\n\n## Real-World Example\n\n**Project:** Chinese short-video content curation from long transcripts\n\n**Structure:**\n```\ntiaogaoren\u002F\n├── promptfooconfig.yaml          # Production config\n├── promptfooconfig-preview.yaml  # Preview config (echo provider)\n├── prompts\u002F\n│   ├── tiaogaoren-prompt.json   # Chat format with few-shot\n│   └── v4\u002Fsystem-v4.md          # System prompt\n├── tests\u002Fcases.yaml              # 3 test samples\n├── scripts\u002Fmetrics.py            # Custom metrics (reduction ratio, etc.)\n├── data\u002F                         # 5 samples (2 few-shot, 3 eval)\n└── results\u002F\n```\n\n**See:** `.\u002Ftiaogaoren\u002F` (example project root) for full implementation.\n\n## Resources\n\nFor detailed API reference and advanced patterns, see [references\u002Fpromptfoo_api.md](references\u002Fpromptfoo_api.md).\n\n","# Promptfoo Evaluation\n\n## Overview\n\nThis skill provides guidance for configuring and running LLM evaluations using [Promptfoo](https:\u002F\u002Fwww.promptfoo.dev\u002F), an open-source CLI tool for testing and comparing LLM outputs.\n\n## Quick Start\n\n```bash\n# Initialize a new evaluation project\nnpx promptfoo@latest init\n\n# Run evaluation\nnpx promptfoo@latest eval\n\n# View results in browser\nnpx promptfoo@latest view\n```\n\n## Configuration Structure\n\nA typical Promptfoo project structure:\n\n```\nproject\u002F\n├── promptfooconfig.yaml    # Main configuration\n├── prompts\u002F\n│   ├── system.md           # System prompt\n│   └── chat.json           # Chat format prompt\n├── tests\u002F\n│   └── cases.yaml          # Test cases\n└── scripts\u002F\n    └── metrics.py          # Custom Python assertions\n```\n\n## Core Configuration (promptfooconfig.yaml)\n\n```yaml\n# yaml-language-server: $schema=https:\u002F\u002Fpromptfoo.dev\u002Fconfig-schema.json\ndescription: \"My LLM Evaluation\"\n\n# Prompts to test\nprompts:\n  - file:\u002F\u002Fprompts\u002Fsystem.md\n  - file:\u002F\u002Fprompts\u002Fchat.json\n\n# Models to compare\nproviders:\n  - id: anthropic:messages:claude-sonnet-4-6\n    label: Claude-Sonnet-4.6\n  - id: openai:gpt-4.1\n    label: GPT-4.1\n\n# Test cases\ntests: file:\u002F\u002Ftests\u002Fcases.yaml\n\n# Concurrency control (MUST be under commandLineOptions, NOT top-level)\ncommandLineOptions:\n  maxConcurrency: 2\n\n# Default assertions for all tests\ndefaultTest:\n  assert:\n    - type: python\n      value: file:\u002F\u002Fscripts\u002Fmetrics.py:custom_assert\n    - type: llm-rubric\n      value: |\n        Evaluate the response quality on a 0-1 scale.\n      threshold: 0.7\n\n# Output path\noutputPath: results\u002Feval-results.json\n```\n\n## Prompt Formats\n\n### Text Prompt (system.md)\n\n```markdown\nYou are a helpful assistant.\n\nTask: {{task}}\nContext: {{context}}\n```\n\n### Chat Format (chat.json)\n\n```json\n[\n  {\"role\": \"system\", \"content\": \"{{system_prompt}}\"},\n  {\"role\": \"user\", \"content\": \"{{user_input}}\"}\n]\n```\n\n### Few-Shot Pattern\n\nEmbed examples directly in prompt or use chat format with assistant messages:\n\n```json\n[\n  {\"role\": \"system\", \"content\": \"{{system_prompt}}\"},\n  {\"role\": \"user\", \"content\": \"Example input: {{example_input}}\"},\n  {\"role\": \"assistant\", \"content\": \"{{example_output}}\"},\n  {\"role\": \"user\", \"content\": \"Now process: {{actual_input}}\"}\n]\n```\n\n## Test Cases (tests\u002Fcases.yaml)\n\n```yaml\n- description: \"Test case 1\"\n  vars:\n    system_prompt: file:\u002F\u002Fprompts\u002Fsystem.md\n    user_input: \"Hello world\"\n    # Load content from files\n    context: file:\u002F\u002Fdata\u002Fcontext.txt\n  assert:\n    - type: contains\n      value: \"expected text\"\n    - type: python\n      value: file:\u002F\u002Fscripts\u002Fmetrics.py:custom_check\n      threshold: 0.8\n```\n\n## Python Custom Assertions\n\nCreate a Python file for custom assertions (e.g., `scripts\u002Fmetrics.py`):\n\n```python\ndef get_assert(output: str, context: dict) -> dict:\n    \"\"\"Default assertion function.\"\"\"\n    vars_dict = context.get('vars', {})\n\n    # Access test variables\n    expected = vars_dict.get('expected', '')\n\n    # Return result\n    return {\n        \"pass\": expected in output,\n        \"score\": 0.8,\n        \"reason\": \"Contains expected content\",\n        \"named_scores\": {\"relevance\": 0.9}\n    }\n\ndef custom_check(output: str, context: dict) -> dict:\n    \"\"\"Custom named assertion.\"\"\"\n    word_count = len(output.split())\n    passed = 100 \u003C= word_count \u003C= 500\n\n    return {\n        \"pass\": passed,\n        \"score\": min(1.0, word_count \u002F 300),\n        \"reason\": f\"Word count: {word_count}\"\n    }\n```\n\n**Key points:**\n- Default function name is `get_assert`\n- Specify function with `file:\u002F\u002Fpath.py:function_name`\n- Return `bool`, `float` (score), or `dict` with pass\u002Fscore\u002Freason\n- Access variables via `context['vars']`\n\n## LLM-as-Judge (llm-rubric)\n\n```yaml\nassert:\n  - type: llm-rubric\n    value: |\n      Evaluate the response based on:\n      1. Accuracy of information\n      2. Clarity of explanation\n      3. Completeness\n\n      Score 0.0-1.0 where 0.7+ is passing.\n    threshold: 0.7\n    provider: openai:gpt-4.1  # Optional: override grader model\n```\n\n**When using a relay\u002Fproxy API**, each `llm-rubric` assertion needs its own `provider` config with `apiBaseUrl`. Otherwise the grader falls back to the default Anthropic\u002FOpenAI endpoint and gets 401 errors:\n\n```yaml\nassert:\n  - type: llm-rubric\n    value: |\n      Evaluate quality on a 0-1 scale.\n    threshold: 0.7\n    provider:\n      id: anthropic:messages:claude-sonnet-4-6\n      config:\n        apiBaseUrl: https:\u002F\u002Fyour-relay.example.com\u002Fapi\n```\n\n**Best practices:**\n- Provide clear scoring criteria\n- Use `threshold` to set minimum passing score\n- Default grader uses available API keys (OpenAI → Anthropic → Google)\n- **When using relay\u002Fproxy**: every `llm-rubric` must have its own `provider` with `apiBaseUrl` — the main provider's `apiBaseUrl` is NOT inherited\n\n## Common Assertion Types\n\n| Type | Usage | Example |\n|------|-------|---------|\n| `contains` | Check substring | `value: \"hello\"` |\n| `icontains` | Case-insensitive | `value: \"HELLO\"` |\n| `equals` | Exact match | `value: \"42\"` |\n| `regex` | Pattern match | `value: \"\\\\d{4}\"` |\n| `python` | Custom logic | `value: file:\u002F\u002Fscript.py` |\n| `llm-rubric` | LLM grading | `value: \"Is professional\"` |\n| `latency` | Response time | `threshold: 1000` |\n\n## File References\n\nAll `file:\u002F\u002F` paths are resolved relative to `promptfooconfig.yaml` location (NOT the YAML file containing the reference). This is a common gotcha when `tests:` references a separate YAML file — the `file:\u002F\u002F` paths inside that test file still resolve from the config root.\n\n```yaml\n# Load file content as variable\nvars:\n  content: file:\u002F\u002Fdata\u002Finput.txt\n\n# Load prompt from file\nprompts:\n  - file:\u002F\u002Fprompts\u002Fmain.md\n\n# Load test cases from file\ntests: file:\u002F\u002Ftests\u002Fcases.yaml\n\n# Load Python assertion\nassert:\n  - type: python\n    value: file:\u002F\u002Fscripts\u002Fcheck.py:validate\n```\n\n## Running Evaluations\n\n```bash\n# Basic run\nnpx promptfoo@latest eval\n\n# With specific config\nnpx promptfoo@latest eval --config path\u002Fto\u002Fconfig.yaml\n\n# Output to file\nnpx promptfoo@latest eval --output results.json\n\n# Filter tests\nnpx promptfoo@latest eval --filter-metadata category=math\n\n# View results\nnpx promptfoo@latest view\n```\n\n## Relay \u002F Proxy API Configuration\n\nWhen using an API relay or proxy instead of direct Anthropic\u002FOpenAI endpoints:\n\n```yaml\nproviders:\n  - id: anthropic:messages:claude-sonnet-4-6\n    label: Claude-Sonnet-4.6\n    config:\n      max_tokens: 4096\n      apiBaseUrl: https:\u002F\u002Fyour-relay.example.com\u002Fapi  # Promptfoo appends \u002Fv1\u002Fmessages\n\n# CRITICAL: maxConcurrency MUST be under commandLineOptions (NOT top-level)\ncommandLineOptions:\n  maxConcurrency: 1  # Respect relay rate limits\n```\n\n**Key rules:**\n- `apiBaseUrl` goes in `providers[].config` — Promptfoo appends `\u002Fv1\u002Fmessages` automatically\n- `maxConcurrency` must be under `commandLineOptions:` — placing it at top level is silently ignored\n- When using relay with LLM-as-judge, set `maxConcurrency: 1` to avoid concurrent request limits (generation + grading share the same pool)\n- Pass relay token as `ANTHROPIC_API_KEY` env var\n\n## Troubleshooting\n\n**Python not found:**\n```bash\nexport PROMPTFOO_PYTHON=python3\n```\n\n**Large outputs truncated:**\nOutputs over 30000 characters are truncated. Use `head_limit` in assertions.\n\n**File not found errors:**\nAll `file:\u002F\u002F` paths resolve relative to `promptfooconfig.yaml` location.\n\n**maxConcurrency ignored (shows \"up to N at a time\"):**\n`maxConcurrency` must be under `commandLineOptions:`, not at the YAML top level. This is a common mistake.\n\n**LLM-as-judge returns 401 with relay API:**\nEach `llm-rubric` assertion must have its own `provider` with `apiBaseUrl`. The main provider config is not inherited by grader assertions.\n\n**HTML tags in model output inflating metrics:**\nModels may output `\u003Cbr>`, `\u003Cb>`, etc. in structured content. Strip HTML in Python assertions before measuring:\n```python\nimport re\nclean_text = re.sub(r'\u003C[^>]+>', '', raw_text)\n```\n\n## Echo Provider (Preview Mode)\n\nUse the **echo provider** to preview rendered prompts without making API calls:\n\n```yaml\n# promptfooconfig-preview.yaml\nproviders:\n  - echo  # Returns prompt as output, no API calls\n\ntests:\n  - vars:\n      input: \"test content\"\n```\n\n**Use cases:**\n- Preview prompt rendering before expensive API calls\n- Verify Few-shot examples are loaded correctly\n- Debug variable substitution issues\n- Validate prompt structure\n\n```bash\n# Run preview mode\nnpx promptfoo@latest eval --config promptfooconfig-preview.yaml\n```\n\n**Cost:** Free - no API tokens consumed.\n\n## Advanced Few-Shot Implementation\n\n### Multi-turn Conversation Pattern\n\nFor complex few-shot learning with full examples:\n\n```json\n[\n  {\"role\": \"system\", \"content\": \"{{system_prompt}}\"},\n\n  \u002F\u002F Few-shot Example 1\n  {\"role\": \"user\", \"content\": \"Task: {{example_input_1}}\"},\n  {\"role\": \"assistant\", \"content\": \"{{example_output_1}}\"},\n\n  \u002F\u002F Few-shot Example 2 (optional)\n  {\"role\": \"user\", \"content\": \"Task: {{example_input_2}}\"},\n  {\"role\": \"assistant\", \"content\": \"{{example_output_2}}\"},\n\n  \u002F\u002F Actual test\n  {\"role\": \"user\", \"content\": \"Task: {{actual_input}}\"}\n]\n```\n\n**Test case configuration:**\n\n```yaml\ntests:\n  - vars:\n      system_prompt: file:\u002F\u002Fprompts\u002Fsystem.md\n      # Few-shot examples\n      example_input_1: file:\u002F\u002Fdata\u002Fexamples\u002Finput1.txt\n      example_output_1: file:\u002F\u002Fdata\u002Fexamples\u002Foutput1.txt\n      example_input_2: file:\u002F\u002Fdata\u002Fexamples\u002Finput2.txt\n      example_output_2: file:\u002F\u002Fdata\u002Fexamples\u002Foutput2.txt\n      # Actual test\n      actual_input: file:\u002F\u002Fdata\u002Ftest1.txt\n```\n\n**Best practices:**\n- Use 1-3 few-shot examples (more may dilute effectiveness)\n- Ensure examples match the task format exactly\n- Load examples from files for better maintainability\n- Use echo provider first to verify structure\n\n## Long Text Handling\n\nFor Chinese\u002Flong-form content evaluations (10k+ characters):\n\n**Configuration:**\n\n```yaml\nproviders:\n  - id: anthropic:messages:claude-sonnet-4-6\n    config:\n      max_tokens: 8192  # Increase for long outputs\n\ndefaultTest:\n  assert:\n    - type: python\n      value: file:\u002F\u002Fscripts\u002Fmetrics.py:check_length\n```\n\n**Python assertion for text metrics:**\n\n```python\nimport re\n\ndef strip_tags(text: str) -> str:\n    \"\"\"Remove HTML tags for pure text.\"\"\"\n    return re.sub(r'\u003C[^>]+>', '', text)\n\ndef check_length(output: str, context: dict) -> dict:\n    \"\"\"Check output length constraints.\"\"\"\n    raw_input = context['vars'].get('raw_input', '')\n\n    input_len = len(strip_tags(raw_input))\n    output_len = len(strip_tags(output))\n\n    reduction_ratio = 1 - (output_len \u002F input_len) if input_len > 0 else 0\n\n    return {\n        \"pass\": 0.7 \u003C= reduction_ratio \u003C= 0.9,\n        \"score\": reduction_ratio,\n        \"reason\": f\"Reduction: {reduction_ratio:.1%} (target: 70-90%)\",\n        \"named_scores\": {\n            \"input_length\": input_len,\n            \"output_length\": output_len,\n            \"reduction_ratio\": reduction_ratio\n        }\n    }\n```\n\n## Real-World Example\n\n**Project:** Chinese short-video content curation from long transcripts\n\n**Structure:**\n```\ntiaogaoren\u002F\n├── promptfooconfig.yaml          # Production config\n├── promptfooconfig-preview.yaml  # Preview config (echo provider)\n├── prompts\u002F\n│   ├── tiaogaoren-prompt.json   # Chat format with few-shot\n│   └── v4\u002Fsystem-v4.md          # System prompt\n├── tests\u002Fcases.yaml              # 3 test samples\n├── scripts\u002Fmetrics.py            # Custom metrics (reduction ratio, etc.)\n├── data\u002F                         # 5 samples (2 few-shot, 3 eval)\n└── results\u002F\n```\n\n**See:** `.\u002Ftiaogaoren\u002F` (example project root) for full implementation.\n\n## Resources\n\nFor detailed API reference and advanced patterns, see [references\u002Fpromptfoo_api.md](references\u002Fpromptfoo_api.md).\n\n",{"name":5,"description":7},"main","def8531e65114c0fca8fb8551c1871ee0eed705c","skills\u002Fdaymade\u002Fclaude-code-skills\u002Fpromptfoo-evaluation",24,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub","https:\u002F\u002Fskillshub.wtf","https:\u002F\u002Fgithub.com\u002FComeOnOliver","AI & Blockchain Engineer\r\n","2026-06-24 22:07:19",{"source":8,"skillName":5,"commands":62,"prompt":72,"zipAvailable":73},[63,66,69],{"tool":64,"command":65},"npx","npx skills add ComeOnOliver\u002Fskillshub --skill promptfoo-evaluation",{"tool":67,"command":68},"bunx","bunx skills add ComeOnOliver\u002Fskillshub --skill promptfoo-evaluation",{"tool":70,"command":71},"pnpm","pnpm dlx skills add ComeOnOliver\u002Fskillshub --skill promptfoo-evaluation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fdaymade\u002Fclaude-code-skills\u002Fpromptfoo-evaluation\u002FSKILL.md",true,[],[76,85,95,104,110,117],{"skillNo":77,"name":78,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":82,"publishedAt":47},"01M6FKJN13GMG4KY57J3QD8GY5","model-hierarchy-skill","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzscole\u002Fmodel-hierarchy-skill\u002Fmodel-hierarchy-skill\u002FSKILL.md",[81],{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},[42,83,84],"model hierarchy","model hierarchy skill",{"skillNo":86,"name":87,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":91,"publishedAt":47},"01M6FKJJJ6GCAMR4PPPRD12M5X","zoom-mcp","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-mcp\u002FSKILL.md",[90],{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},[92,93,94],"MCP","zoom","zoom mcp",{"skillNo":96,"name":97,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":47},"01M6FKJE3KYT3KPEZMEEHYKA6K","zoom-apps-sdk","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-apps-sdk\u002FSKILL.md",[],[101,102,103,93],"sdk","zoom apps sdk","apps",{"skillNo":105,"name":106,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":107,"categories":108,"keywords":109,"publishedAt":47},"01M6FKJ9EDH9Q5VWDF8HAJRSP0","windows","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwindows\u002FSKILL.md",[],[106],{"skillNo":111,"name":112,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":116,"publishedAt":47},"01M6FKJ666BEEFC3AE9GJ72Q93","whiteboard","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwhiteboard\u002FSKILL.md",[115],{"categoryNo":24,"categoryName":25,"taxonomyType":21,"taxonomyTypeDesc":22},[112],{"skillNo":118,"name":119,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":47},"01M6FKJ21WFCDGTJQ2Q91T9DVW","websockets","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwebsockets\u002FSKILL.md",[],[119]]