[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EE9TD8SCDV82AEPTAXSS76":3,"skill-same:01M6EE9TD8SCDV82AEPTAXSS76":48},{"skillNo":4,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":21,"linkOnly":22,"markdownContent":23,"markdownBody":23,"frontMatter":24,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":28,"repositoryHtmlUrl":29,"repositoryHomepageUrl":30,"ownerHtmlUrl":31,"ownerBlogUrl":6,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M6EE9TD8SCDV82AEPTAXSS76","ai-eval-ci",null,"ComeOnOliver\u002Fskillshub","ComeOnOliver","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F43215937?v=4","01M6E5JNFECJAAVFE0W7SMWWD2","Haoran Shu","01M6E5JNNDQS7AJZDYFQ8TPZCK",67,"MIT","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002FTerminalSkills\u002Fskills\u002Fai-eval-ci\u002FSKILL.md",[],[18,19,20],"ci","eval","ai eval ci","2026-11-10 10:12:30",false,"# AI Eval in CI\n\n## Overview\n\nTest AI agents and LLM outputs the same way you test code — automated evaluations that run in CI, compare against baselines, and fail the build when quality drops. No dashboards to check manually. Just `npx eval run --ci` and a red or green build.\n\n## When to Use\n\n- Adding quality gates before deploying AI features to production\n- Catching prompt regressions when system prompts or models change\n- Comparing model performance (GPT-4o vs Claude Sonnet vs local Llama)\n- Validating RAG pipeline accuracy against a test dataset\n- Benchmarking agent tool-calling accuracy and latency\n\n## Instructions\n\n### Strategy 1: Promptfoo (Config-Driven Evals)\n\nPromptfoo is the most popular open-source eval framework. Define test cases in YAML, run against multiple providers, get a comparison matrix.\n\n```yaml\n# promptfooconfig.yaml — Eval configuration\n# Tests a customer support agent across 3 models with quality assertions\ndescription: \"Customer support agent eval\"\n\nproviders:\n  - id: openai:gpt-4o\n  - id: anthropic:messages:claude-sonnet-4-20250514\n  - id: ollama:llama3.1:8b\n\nprompts:\n  - |\n    You are a customer support agent for a SaaS product.\n    Respond helpfully and accurately. If you don't know, say so.\n    \n    Customer message: {{message}}\n\ntests:\n  - vars:\n      message: \"How do I reset my password?\"\n    assert:\n      - type: llm-rubric\n        value: \"Response explains the password reset process clearly\"\n      - type: not-contains\n        value: \"I don't know\"\n      - type: latency\n        threshold: 3000  # Must respond within 3 seconds\n\n  - vars:\n      message: \"Can I get a refund for my annual plan?\"\n    assert:\n      - type: llm-rubric\n        value: \"Response acknowledges the refund request and explains the policy\"\n      - type: not-contains\n        value: \"I'm an AI\"  # Don't break character\n\n  - vars:\n      message: \"Your product deleted all my data!\"\n    assert:\n      - type: llm-rubric\n        value: \"Response shows empathy, takes the issue seriously, and offers next steps\"\n      - type: sentiment\n        threshold: 0.3  # Must not be dismissive\n\n  - vars:\n      message: \"What's the weather in Tokyo?\"\n    assert:\n      - type: llm-rubric\n        value: \"Response politely redirects to product-related topics\"\n      - type: not-contains\n        value: \"Tokyo\"  # Should not answer off-topic questions\n```\n\n```bash\n# Run evals locally\nnpx promptfoo@latest eval\n\n# Run in CI with threshold — exits non-zero if any test fails\nnpx promptfoo@latest eval --ci --output results.json\n\n# Compare two prompt versions\nnpx promptfoo@latest eval --prompts prompt-v1.txt prompt-v2.txt --share\n```\n\n### Strategy 2: Custom Eval Framework (TypeScript)\n\nWhen you need full control — custom scoring logic, database-backed test sets, domain-specific metrics.\n\n```typescript\n\u002F\u002F eval.ts — Custom AI eval framework with CI integration\n\u002F**\n * Runs evaluation suites against AI agents\u002FLLMs.\n * Each eval defines inputs, expected behavior, and scoring criteria.\n * Exits with code 1 if any score drops below threshold.\n *\u002F\nimport OpenAI from \"openai\";\n\ninterface EvalCase {\n  name: string;\n  input: string;\n  rubric: string;          \u002F\u002F What \"good\" looks like\n  threshold: number;       \u002F\u002F Minimum score 0-1\n  metadata?: Record\u003Cstring, unknown>;\n}\n\ninterface EvalResult {\n  name: string;\n  score: number;\n  pass: boolean;\n  output: string;\n  reasoning: string;\n  latencyMs: number;\n}\n\nconst openai = new OpenAI();\n\n\u002F**\n * Score an AI output using LLM-as-judge.\n * Returns a score 0-1 with reasoning.\n *\u002F\nasync function judge(output: string, rubric: string): Promise\u003C{ score: number; reasoning: string }> {\n  const response = await openai.chat.completions.create({\n    model: \"gpt-4o-mini\",  \u002F\u002F Cheap model for judging\n    messages: [\n      {\n        role: \"system\",\n        content: `You are an eval judge. Score the AI output against the rubric.\nReturn JSON: {\"score\": 0.0-1.0, \"reasoning\": \"brief explanation\"}\nScore 1.0 = perfect match. Score 0.0 = complete failure.`,\n      },\n      {\n        role: \"user\",\n        content: `Rubric: ${rubric}\\n\\nAI Output:\\n${output}`,\n      },\n    ],\n    response_format: { type: \"json_object\" },\n    temperature: 0,  \u002F\u002F Deterministic judging\n  });\n\n  return JSON.parse(response.choices[0].message.content!);\n}\n\n\u002F**\n * Run a single eval case against your AI agent.\n *\u002F\nasync function runEval(\n  agentFn: (input: string) => Promise\u003Cstring>,\n  evalCase: EvalCase\n): Promise\u003CEvalResult> {\n  const start = Date.now();\n  const output = await agentFn(evalCase.input);\n  const latencyMs = Date.now() - start;\n\n  const { score, reasoning } = await judge(output, evalCase.rubric);\n\n  return {\n    name: evalCase.name,\n    score,\n    pass: score >= evalCase.threshold,\n    output: output.slice(0, 200),\n    reasoning,\n    latencyMs,\n  };\n}\n\n\u002F**\n * Run all evals and exit with appropriate code for CI.\n *\u002F\nasync function runSuite(\n  agentFn: (input: string) => Promise\u003Cstring>,\n  cases: EvalCase[]\n): Promise\u003Cvoid> {\n  console.log(`Running ${cases.length} evals...\\n`);\n\n  const results: EvalResult[] = [];\n  for (const evalCase of cases) {\n    const result = await runEval(agentFn, evalCase);\n    results.push(result);\n    const icon = result.pass ? \"✅\" : \"❌\";\n    console.log(`${icon} ${result.name}: ${result.score.toFixed(2)} (threshold: ${evalCase.threshold}) [${result.latencyMs}ms]`);\n    if (!result.pass) {\n      console.log(`   Reasoning: ${result.reasoning}`);\n    }\n  }\n\n  \u002F\u002F Summary\n  const passed = results.filter((r) => r.pass).length;\n  const failed = results.filter((r) => !r.pass).length;\n  const avgScore = results.reduce((s, r) => s + r.score, 0) \u002F results.length;\n\n  console.log(`\\n? Results: ${passed} passed, ${failed} failed (avg score: ${avgScore.toFixed(2)})`);\n\n  \u002F\u002F CI exit code\n  if (failed > 0) {\n    console.log(\"\\n❌ Eval suite FAILED — quality below threshold\");\n    process.exit(1);\n  } else {\n    console.log(\"\\n✅ Eval suite PASSED\");\n  }\n}\n\nexport { runSuite, EvalCase };\n```\n\n### Strategy 3: GitHub Actions Integration\n\n```yaml\n# .github\u002Fworkflows\u002Fai-eval.yml\nname: AI Eval\non:\n  pull_request:\n    paths:\n      - \"prompts\u002F**\"\n      - \"src\u002Fagents\u002F**\"\n      - \"eval\u002F**\"\n\njobs:\n  eval:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions\u002Fcheckout@v4\n      - uses: actions\u002Fsetup-node@v4\n        with: { node-version: 20 }\n      - run: npm ci\n\n      - name: Run AI evals\n        run: npx tsx eval\u002Frun.ts --ci\n        env:\n          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\n          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}\n\n      - name: Post results to PR\n        if: always()\n        uses: actions\u002Fgithub-script@v7\n        with:\n          script: |\n            const fs = require('fs');\n            const results = JSON.parse(fs.readFileSync('eval\u002Fresults.json'));\n            const body = results.map(r => \n              `${r.pass ? '✅' : '❌'} **${r.name}**: ${r.score.toFixed(2)} (${r.latencyMs}ms)`\n            ).join('\\n');\n            github.rest.issues.createComment({\n              ...context.repo,\n              issue_number: context.issue.number,\n              body: `## AI Eval Results\\n\\n${body}`\n            });\n```\n\n## Examples\n\n### Example 1: Add quality gates to a RAG chatbot\n\n**User prompt:** \"Set up automated evals for our RAG customer support bot. It should test accuracy on 50 known Q&A pairs and fail the deploy if accuracy drops below 85%.\"\n\nThe agent will:\n- Create a test dataset from the 50 known Q&A pairs\n- Write promptfoo config with llm-rubric assertions for each\n- Set pass threshold at 0.85\n- Add GitHub Actions workflow that runs on PR to `prompts\u002F` or `src\u002Fagents\u002F`\n- Post eval results as PR comment\n\n### Example 2: Compare models before switching\n\n**User prompt:** \"We're considering switching from GPT-4o to Claude Sonnet. Run our eval suite against both and show me which performs better.\"\n\nThe agent will:\n- Configure promptfoo with both providers\n- Run the existing eval suite against both models\n- Generate comparison table with per-test scores, latency, and cost\n- Recommend based on score-to-cost ratio\n\n## Guidelines\n\n- **Eval every prompt change** — treat prompts like code; test before deploying\n- **LLM-as-judge is good enough** — GPT-4o-mini costs pennies and correlates well with human judgment\n- **Use temperature 0 for judges** — deterministic scoring reduces noise\n- **Keep test sets diverse** — happy path, edge cases, adversarial inputs, off-topic\n- **Set realistic thresholds** — start at 0.7, tighten as the agent improves\n- **Track scores over time** — log results to detect gradual quality drift\n- **Separate eval cost from production cost** — eval uses cheap judge models, production uses the best\n- **Cache eval results** — don't re-run unchanged tests; hash input+prompt for cache keys\n- **Run evals on PRs, not just main** — catch regressions before merge\n",{},"main","def8531e65114c0fca8fb8551c1871ee0eed705c","skills\u002FTerminalSkills\u002Fskills\u002Fai-eval-ci",24,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub","https:\u002F\u002Fskillshub.wtf","https:\u002F\u002Fgithub.com\u002FComeOnOliver","AI & Blockchain Engineer\r\n","2026-06-24 22:07:19",{"source":7,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add ComeOnOliver\u002Fskillshub --skill ai-eval-ci",{"tool":40,"command":41},"bunx","bunx skills add ComeOnOliver\u002Fskillshub --skill ai-eval-ci",{"tool":43,"command":44},"pnpm","pnpm dlx skills add ComeOnOliver\u002Fskillshub --skill ai-eval-ci","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002FTerminalSkills\u002Fskills\u002Fai-eval-ci\u002FSKILL.md",true,[],[49,63,73,82,88,95],{"skillNo":50,"name":51,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":52,"categories":53,"keywords":59,"publishedAt":21},"01M6FKJN13GMG4KY57J3QD8GY5","model-hierarchy-skill","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzscole\u002Fmodel-hierarchy-skill\u002Fmodel-hierarchy-skill\u002FSKILL.md",[54],{"categoryNo":55,"categoryName":56,"taxonomyType":57,"taxonomyTypeDesc":58},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[60,61,62],"model","model hierarchy","model hierarchy skill",{"skillNo":64,"name":65,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":69,"publishedAt":21},"01M6FKJJJ6GCAMR4PPPRD12M5X","zoom-mcp","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-mcp\u002FSKILL.md",[68],{"categoryNo":55,"categoryName":56,"taxonomyType":57,"taxonomyTypeDesc":58},[70,71,72],"MCP","zoom","zoom mcp",{"skillNo":74,"name":75,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":76,"categories":77,"keywords":78,"publishedAt":21},"01M6FKJE3KYT3KPEZMEEHYKA6K","zoom-apps-sdk","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-apps-sdk\u002FSKILL.md",[],[79,80,81,71],"sdk","zoom apps sdk","apps",{"skillNo":83,"name":84,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":21},"01M6FKJ9EDH9Q5VWDF8HAJRSP0","windows","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwindows\u002FSKILL.md",[],[84],{"skillNo":89,"name":90,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":94,"publishedAt":21},"01M6FKJ666BEEFC3AE9GJ72Q93","whiteboard","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwhiteboard\u002FSKILL.md",[93],{"categoryNo":55,"categoryName":56,"taxonomyType":57,"taxonomyTypeDesc":58},[90],{"skillNo":96,"name":97,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":21},"01M6FKJ21WFCDGTJQ2Q91T9DVW","websockets","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwebsockets\u002FSKILL.md",[],[97]]