behavioral-evals
Guidance for creating, running, fixing, and promoting behavioral evaluations. Use when verifying agent decision logic, debugging failures, debugging prompt steering, or adding workspace regression tests.
- License 许可
- MIT
- 最近更新
- 2026-11-12
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- behavioral-evals
- description
- Guidance for creating, running, fixing, and promoting behavioral evaluations. Use when verifying agent decision logic, debugging failures, debugging prompt steering, or adding workspace regression tests.
02 资源文件 6
- 资产文件assets/interactive_eval.ts.txt882 B
- 资产文件assets/standard_eval.ts.txt996 B
- 参考文档references/creating.md4.5 KB
- 参考文档references/fixing.md4.6 KB
- 参考文档references/promoting.md1.9 KB
- 参考文档references/running.md2.5 KB
03 SKILL.md 正文
Behavioral Evals
Overview
Behavioral evaluations (evals) are tests that validate the agent's decision-making (e.g., tool choice) rather than pure functionality. They are critical for verifying prompt changes, debugging steerability, and preventing regressions.
[!NOTE] Single Source of Truth: For core concepts, policies, running tests, and general best practices, always refer to [evals/README.md](file:///Users/abhipatel/code/gemini-cli/docs/evals/README.md).
🔄 Workflow Decision Tree
- Does a prompt/tool change need validation?
- No -> Normal integration tests.
- Yes -> Continue below.
- Is it UI/Interaction heavy?
- Yes -> Use
appEvalTest(AppRig). See creating.md. - No -> Use
evalTest(TestRig). See creating.md.
- Yes -> Use
- Is it a new test?
- Yes -> Set policy to
USUALLY_PASSES. - No ->
ALWAYS_PASSES(locks in regression).
- Yes -> Set policy to
- Are you fixing a failure or promoting a test?
- Fixing -> See fixing.md.
- Promoting -> See promoting.md.
📋 Quick Checklist
1. Setup Workspace
Seed the workspace with necessary files using the files object to simulate a realistic scenario (e.g., NodeJS project with package.json).
- Details in creating.md
2. Write Assertions
Audit agent decisions using rig.setBreakpoint() (AppRig only) or index verification on rig.readToolLogs().
- Details in creating.md
3. Verify
Run single tests locally with Vitest. Confirm stability locally before relying on CI workflows.
- See [evals/README.md](file:///Users/abhipatel/code/gemini-cli/docs/evals/README.md) for running commands.
📦 Bundled Resources
Detailed procedural guides:
- creating.md: Assertion strategies, Rig selection, Mock MCPs.
- fixing.md: Step-by-step automated investigation, architecture diagnosis guidelines.
- promoting.md: Candidate identification criteria and threshold guidelines.
04 同仓库更多 Skills
winui-app
Bootstrap, develop, and design modern WinUI 3 desktop applications with C# and the Windows App SDK using official Microsoft guidance, WinUI Gallery patterns, Windows App SDK samples, and CommunityToolkit components. Use when creating a brand new app, preparing a machine for WinUI, reviewing, refactoring, planning, troubleshooting, environment-checking, or setting up WinUI 3 XAML, controls, navigation, windowing, theming, accessibility, responsiveness, performance, deployment, or related Windows app design and development work.
vercel-deploy
Deploy applications and websites to Vercel. Use when the user requests deployment actions like "deploy my app", "deploy and give me the link", "push this live", or "create a preview deployment".
security-best-practices
Perform language and framework specific security best-practice reviews and suggest improvements. Trigger only when the user explicitly requests security best practices guidance, a security review/report, or secure-by-default coding help. Trigger only for supported languages (python, javascript/typescript, go). Do not trigger for general code review, debugging, or non-security tasks.
playwright
Use when the task requires automating a real browser from the terminal (navigation, form filling, snapshots, screenshots, data extraction, UI-flow debugging) via `playwright-cli` or the bundled wrapper script.
openai-docs
Use when the user asks how to build with OpenAI products or APIs and needs up-to-date official documentation with citations, help choosing the latest model for a use case, or model upgrade and prompt-upgrade guidance; prioritize OpenAI docs MCP tools, use bundled references only as helper context, and restrict any fallback browsing to official OpenAI domains.
netlify-deploy
Deploy web projects to Netlify using the Netlify CLI (`npx netlify`). Use when the user asks to deploy, host, publish, or link a site/repo on Netlify, including preview and production deploys.