[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M77SEWHP54GYPXY3NCMSYBPJ":3,"skill-same:01M77SEWHP54GYPXY3NCMSYBPJ":73},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":29,"commitSha":30,"skillDirectory":31,"forksCount":32,"repositoryHtmlUrl":33,"repositoryHomepageUrl":6,"ownerHtmlUrl":34,"ownerBlogUrl":35,"ownerDescription":6,"updatedAt":36,"install":37,"resources":50},"01M77SEWHP54GYPXY3NCMSYBPJ","google-agents-cli-eval",null,"This skill should be used when the user wants to \"run an evaluation\", \"evaluate my ADK agent\", \"write an eval dataset\", \"analyze eval failures\", \"compare eval results\", \"optimize agent\", or needs guidance on the Agent Platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Do NOT use for API code patterns (use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).","Reihantt6\u002Fsns-myagent","Reihantt6","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F219822095?v=4","01M77KK94PV8B7C2D9XWXNS8DJ","Mohamad Reihan tegar prasetya","01M77KK958QFMTHYXGXFV7KZ79",2,"MIT","https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fgoogle-agents-cli-eval\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\ndescription: \"This skill should be used when the user wants to \\\"run an evaluation\\\", \\\"evaluate my ADK agent\\\", \\\"write an eval dataset\\\", \\\"analyze eval failures\\\", \\\"compare eval results\\\", \\\"optimize agent\\\", or needs guidance on the Agent Platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Do NOT use for API code patterns (use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).\\n\"\nmetadata: {\"author\":\"Google\",\"license\":\"Apache-2.0\",\"version\":\"1.3.1\"}\n---\n# Agent Evaluation Guide\n\n> **Requires:** `agents-cli` (`uv tool install google-agents-cli`) — [install uv](https:\u002F\u002Fdocs.astral.sh\u002Fuv\u002Fgetting-started\u002Finstallation\u002Findex.md) first if needed.\n\n> **Scaffolded project?** If you used `\u002Fgoogle-agents-cli-scaffold`, you already have `agents-cli eval run` (chains `generate` + `grade`), `tests\u002Feval\u002Fdatasets\u002F`, and `tests\u002Feval\u002Feval_config.yaml`. Start with executing `eval run` and iterate from there.\n\n## Reference Files\n\n| File | Contents |\n|------|----------|\n| `references\u002Fdataset_schema.md` | Canonical EvaluationDataset schema — all field types, JSON examples for single-turn \u002F multi-turn \u002F multi-agent, common mistakes |\n| `references\u002Fmetrics-guide.md` | Complete metrics reference — all built-in metrics, match types, custom metrics, judge model config |\n| `references\u002Fuser-simulation.md` | Dynamic conversation testing — `eval dataset synthesize` flags, what scenarios are, compatible metrics |\n| `references\u002Fbuiltin-tools-eval.md` | google_search and model-internal tools — trajectory behavior, metric compatibility |\n| `references\u002Fmultimodal-eval.md` | Multimodal inputs — eval dataset schema, built-in metric limitations, custom evaluator pattern |\n\n---\n\n## The Quality Flywheel\n\nImproving agent quality is iterative. The 5 stages below describe the loop. Each stage has a Default path (you, the coding agent, do the work directly) and an Opt-in CLI command that delegates to the Agent Platform Eval Service for better quality and scale.\n\n### 1. Prepare Data\n\n**Default:** Use or edit the scaffolded `tests\u002Feval\u002Fdatasets\u002Fbasic-dataset.json` to define single-turn eval inputs. Start with 1–2 cases.\n\n**Opt-in:** `agents-cli eval dataset synthesize` — user-simulate multi-turn datasets when you lack data; its output includes traces, so skip Stage 2 and grade directly. See *Eval Commands* and `references\u002Fuser-simulation.md`.\n\n### 2. Run Inference\n\n`agents-cli eval generate` — executes the agent over the dataset and writes traces to `artifacts\u002Ftraces\u002F`. Run this when you wrote the dataset by hand in Stage 1 (default path). **Skip this stage if you used `eval dataset synthesize`** — that command already produced traces.\n\n### 3. Grade Traces (always run)\n\n`agents-cli eval grade` — scores the traces and writes `results_\u003Cts>.{json,html}` to `artifacts\u002Fgrade_results\u002F`. No opt-in alternative; this is the core. Always run, regardless of how Stages 1 and 2 produced the traces.\n\n> **Shortcut:** `agents-cli eval run` chains Stages 2 + 3 in one command using the default `artifacts\u002Ftraces\u002F` directory between them. Use it for the common path; drop back to the two-step form when you need a custom traces location or want to grade an existing traces file.\n\n### 4. Analyze Failures\n\n**Default:** Open the latest `artifacts\u002Fgrade_results\u002Fresults_\u003Cts>.html` (or `.json`) and identify failed metrics — see *What to fix when scores fail* below for the fix table.\n\n**Opt-in:** `agents-cli eval analyze` — LLM-based failure clustering; prefer when you have 10+ failing cases and want categorized failure modes. See *Eval Commands*.\n\n### 5. Optimize & Code Fix\n\n**Default:** Edit the agent — adjust prompts, tool descriptions, instructions, or eval dataset based on the failure analysis. See *What to fix when scores fail* below for the failure → fix mapping.\n\n**Opt-in:** `agents-cli eval optimize` — runs ADK GEPA prompt optimization against a target metric. Suitable for prompt-only failures. The optimized prompt appears in the command output; capture it and apply it to the agent. For the full per-iteration trace, set `print_detailed_results: true` in your optimization config file.\n\n> **Long-running and expensive.** GEPA optimization makes many LLM calls and can take a long time. Do not run it unless the user explicitly asks for prompt optimization. When you do run it, iterate as far as possible with manual fixes first, then run a **single** final `eval optimize` — never loop on this command.\n\n### Running the loop\n\nIterate stages 2 → 3 → 4 → 5 → 2 (or 1 → 3 → 4 → 5 → 1 if using `synthesize`). After each fix, run `agents-cli eval compare \u003Cprev_results>.json \u003Cnew_results>.json` to confirm the target metric improved without regressing others. Expect 5–10+ iterations per case before it passes — this is normal. Only after a case passes should you expand coverage with more eval cases.\n\nWhen doing 5+ iterations, maintain a task list of which cases are fixed, which are still failing, and what fixes you've tried. Prevents re-attempting the same fix.\n\n### Shortcuts That Waste Time\n\nRecognize these rationalizations and push back — they always cost more time than they save:\n\n| Shortcut | Why it fails |\n|----------|-------------|\n| \"I'll tune the eval thresholds down to make it pass\" | Lowering thresholds hides real failures. If the agent can't meet the bar, fix the agent — don't move the bar. |\n| \"This eval case is flaky, I'll skip it\" | Flaky evals reveal non-determinism in your agent. Fix with `temperature=0`, rubric-based metrics, or more specific instructions — don't delete the signal. |\n| \"I just need to fix the eval dataset, not the agent\" | If you're always adjusting expected outputs, your agent has a behavior problem. Fix the instructions or tool logic first. |\n\n## Choosing the Right Metrics\n\nPick built-in metrics by what you want to measure. Multi-turn metrics evaluate the full conversation; single-turn metrics evaluate one prompt-response pair (with intermediate tool calls). When no built-in fits, write a custom metric (see *Evaluation Configuration Schema* below).\n\n| Goal | Recommended built-in metrics |\n|------|------------------------------|\n| **Did the agent achieve the user's goal?** (catch-all for multi-turn agents) | `multi_turn_task_success` |\n| **Was the agent's reasoning path logical and efficient?** | `multi_turn_trajectory_quality` |\n| **Quality of tool \u002F function calling across turns** | `multi_turn_tool_use_quality` |\n| **Final response quality** (no ground-truth reference needed) | `final_response_quality` |\n| **Factual grounding** (catch hallucinated claims, e.g., RAG agents) | `hallucination` |\n| **Safety policy compliance** | `safety` |\n| **Domain-specific check no built-in covers** | Write a custom `LLMMetric` (LLM-judge) or `CodeExecutionMetric` (deterministic Python). See *Evaluation Configuration Schema* below. |\n\nRun `agents-cli eval metric list` to see all available built-ins. For full metric definitions and rubric details, see the [Agent Platform metric docs](https:\u002F\u002Fcloud.google.com\u002Fgemini-enterprise-agent-platform\u002Foptimize\u002Fevaluation\u002Fmanage-metrics) and `references\u002Fmetrics-guide.md`.\n\n---\n\n## What to fix when scores fail\n\nAfter `agents-cli eval grade` completes, inspect the latest `artifacts\u002Fgrade_results\u002Fresults_\u003Ctimestamp>.json` (or open the `.html` file) for per-case scores and judge rationales — that's the input to every fix decision below.\n\n| Failure | What to change |\n|---------|---------------|\n| `multi_turn_task_success` low | The agent isn't completing the user's goal — fix orchestration, missing tool calls, premature termination, or wrong tool selection |\n| `multi_turn_trajectory_quality` low | The agent reaches the goal inefficiently or takes wrong steps — refine planning prompts, tighten instruction order, or remove redundant tool calls |\n| `multi_turn_tool_use_quality` low | Fix tool descriptions, parameter docstrings, or agent instructions for tool selection |\n| `final_response_quality` low | Read the auto-generated rubric verdicts; refine agent instructions to address the worst-scoring criterion (often clarity, completeness, or instruction-following) |\n| `hallucination` low | Tighten agent instructions to stay grounded in tool output; verify the tool actually returned the data the agent claimed |\n| `safety` low | Add safety guardrails to instructions; review the violating content category in the rubric verdict |\n| Agent calls wrong tools | Fix tool descriptions, agent instructions, or `tool_config` |\n| Agent calls extra tools | Add strict stop instructions, or switch to `multi_turn_tool_use_quality` |\n\nAfter applying a fix, rerun `agents-cli eval generate && agents-cli eval grade` and use `agents-cli eval compare \u003Cprev_results>.json \u003Cnew_results>.json` to confirm the fix improved the target metric without regressing others.\n\n---\n\n## Eval Commands\n\nAll `agents-cli eval` subcommands support `--help` for the authoritative flag list and defaults — run `agents-cli eval \u003Csubcommand> --help` (or `agents-cli eval dataset \u003Csubcommand> --help`) when in doubt. The examples below show the most common invocations; flags can change between releases.\n\n### `eval generate`\n\nRuns an agent over an evaluation dataset and writes traces to disk.\n\nBy default, runs the agent in a local HTTP server (launches the project's `fast_api_app.py` if it exists, or falls back to `adk api_server`) and sends each evaluation case over HTTP. You can generate traces from an already-running agent by passing its HTTP endpoint and app name to `--url` and `--app-name`.\n\n```bash\n# Basic — uses tests\u002Feval\u002Fdatasets\u002F, writes to artifacts\u002Ftraces\u002F\nagents-cli eval generate\n\n# Advanced — custom dataset and output dir\nagents-cli eval generate --dataset tests\u002Feval\u002Fdatasets\u002Fcustom.json -o .\u002Fcustom_traces\u002F\n\n# Against a deployed agent (or one you started manually)\nagents-cli eval generate --url https:\u002F\u002Fmy-agent.run.app --app-name app\n```\n\n### `eval grade`\n\nScores generated traces against built-in or custom metrics. Writes timestamped `results_\u003CYYYYMMDD_HHMMSS>.json` (consumed by `eval compare`) and `.html` (open in a browser) into the output dir, and prints a summary table to the console.\n\n```bash\n# Basic — defaults: traces from artifacts\u002Ftraces\u002F, results to artifacts\u002Fgrade_results\u002F,\n# metrics from tests\u002Feval\u002Feval_config.yaml's metrics_to_run\nagents-cli eval grade\n\n# Advanced 1 — grade traces from a non-default location (the canonical\n# pairing for `eval generate --output custom_traces\u002F`)\nagents-cli eval grade --traces custom_traces\u002F\n\n# Advanced 2 — pick built-in metrics, custom output dir\nagents-cli eval grade --metrics tool_use_quality,safety --output .\u002Fout\u002F\n\n# Advanced 3 — load metrics to run from a config file (YAML or JSON) on a specified trace file.\nagents-cli eval grade --traces .\u002Fartifacts\u002Ftraces\u002Ftrace_1.json --config tests\u002Feval\u002Feval_config.yaml\n```\n\nSee *Evaluation Configuration Schema* below for the config file format.\n\n### `eval compare`\n\nDiffs two `results_*.json` files produced by `eval grade`. Run it after a fix to confirm the target metric improved without regressing others.\n\n```bash\nagents-cli eval compare baseline.json candidate.json\n```\n\n### `eval metric list`\n\nLists the built-in metric names usable with `eval grade --metrics`.\n\n```bash\nagents-cli eval metric list\n```\n\n### `eval analyze`\n\nRuns LLM-based failure clustering and root-cause analysis over a `results_*.json` produced by `eval grade`. Use when you have 10+ failing cases and want categorized failure modes instead of reading the HTML case-by-case. Supported `--metric` values: `multi_turn_task_success`, `multi_turn_tool_use_quality`.\n\n```bash\n# Basic — analyze a results file with default settings\nagents-cli eval analyze --eval-result artifacts\u002Fgrade_results\u002Fresults_\u003Cts>.json\n\n# Advanced — restrict to a specific metric and cap loss clusters\nagents-cli eval analyze \\\n  --eval-result artifacts\u002Fgrade_results\u002Fresults_\u003Cts>.json \\\n  --metric multi_turn_tool_use_quality \\\n  --top-k 5 \\\n  --output artifacts\u002Fanalysis_\u003Cts>.json\n```\n\n### `eval dataset synthesize`\n\nGenerates user scenarios server-side from your agent's tools and instructions, then plays each scenario against an LLM-backed user simulator. The output is a graded-ready trace file with full `agent_data.turns` populated — feed it directly to `eval grade` (skip `eval generate`).\n\n```bash\n# Basic — generate 3 default scenarios (up to 5 turns each) into artifacts\u002Ftraces\u002F\n# (where eval grade reads from by default, so synthesize → grade works without flags)\nagents-cli eval dataset synthesize\n\n# Advanced — guide scenario generation with optional instruction and environment context\nagents-cli eval dataset synthesize \\\n  -n 5 \\\n  --instruction \"Customer asking about refunds\" \\\n  --environment-context \"E-commerce support\" \\\n  --max-turns 8 \\\n  -o tests\u002Feval\u002Fdatasets\u002Frefund_scenarios.json\n```\n\nFor scenario semantics, the full `eval dataset synthesize` flag table, and which simulator internals are not user-configurable, see `references\u002Fuser-simulation.md`.\n\n### `eval optimize`\n\nRuns ADK GEPA prompt optimization against a target metric. Suitable after `eval grade` identifies prompt-only failures (wording, not tool\u002Forchestration logic). `--dataset` and `--target-metric` override values in `--config` when both are passed. **Long-running and expensive — see Stage 5 of the Quality Flywheel for usage guidance.**\n\n```bash\n# Basic — optimize against a single metric on a dataset\nagents-cli eval optimize --dataset tests\u002Feval\u002Fdatasets\u002Fbasic-dataset.json --target-metric final_response_quality\n\n# Advanced — drive multi-metric \u002F multi-dataset optimization from a config file\nagents-cli eval optimize --config tests\u002Feval\u002Foptimization_config.json\n```\n\n### `eval submit` \u002F `eval results` (cloud-side)\n\nThe managed, asynchronous counterpart to the local path, for large or CI-driven runs: `eval submit` hands the dataset and metrics to the Agent Platform Eval Service, and `eval results` polls and downloads the scores. Pass `--resource-name \u003Cagent>` to also run inference server-side (managed `generate` + `grade`); omit it to grade an existing trace (managed `grade`).\n\n```bash\n# Grade an existing trace server-side; returns a run resource name to poll\nagents-cli eval submit --dataset tests\u002Feval\u002Fdatasets\u002Fbasic-dataset.json --dest gs:\u002F\u002Fmy-bucket\n# Add --resource-name projects\u002F\u003Cp>\u002Flocations\u002F\u003Cl>\u002FreasoningEngines\u002F\u003Cid> to run inference too\n\nagents-cli eval results --run-id \u003Crun-resource-name>\n```\n\n---\n\n## Evaluation Dataset Format\n\nAn `EvaluationDataset` is a JSON file with an `eval_cases` array. Cases come in two shapes depending on how they're used:\n\n- **Inference input** (what you give to `eval generate`) — a user prompt or a partial conversation ending in a user prompt. The agent runs and produces traces.\n- **Grading input** (what you give to `eval grade`) — a complete trace including the agent's responses and tool calls. Normally produced by `eval generate` or `eval dataset synthesize`; you don't write these by hand.\n\nSee `references\u002Fdataset_schema.md` for the full canonical schema, all field types, and common mistakes.\n\n### Inference input format\n\nTwo shapes are supported.\n\n**(a) Simple single-turn prompt** — what the scaffolded `tests\u002Feval\u002Fdatasets\u002Fbasic-dataset.json` uses. The agent runs from scratch.\n\n```json\n{\n  \"eval_cases\": [\n    {\n      \"eval_case_id\": \"greeting\",\n      \"prompt\": {\n        \"role\": \"user\",\n        \"parts\": [{\"text\": \"Hello, what can you help me with?\"}]\n      }\n    },\n    {\n      \"eval_case_id\": \"weather_query\",\n      \"prompt\": {\n        \"role\": \"user\",\n        \"parts\": [{\"text\": \"What's the weather like in San Francisco?\"}]\n      }\n    }\n  ]\n}\n```\n\n**(b) Multi-turn continuation via `agent_data`** — a partial conversation whose last turn ends with a user message; the agent's next response is evaluated. See `references\u002Fdataset_schema.md` (*Multi-Turn \u002F Multi-Agent Dataset*) for the JSON shape.\n\n### Grading input format (traces)\n\nA complete trace — agent responses plus `function_call` \u002F `function_response` parts — normally produced by `eval generate` \u002F `eval dataset synthesize` (you don't write these by hand). Authors are `\"user\"`, an agent ID from the `agents` map, or `\"tool\"`. See `references\u002Fdataset_schema.md` for the trace shape, multi-agent examples, and the full type reference.\n\n---\n\n## Evaluation Configuration Schema\n\n`agents-cli eval grade --config \u003Cpath>` accepts a single configuration file in either **YAML** (`.yaml` \u002F `.yml`) or **JSON** (`.json`). The file declares two parts:\n\n- `metrics_to_run` — the **selection list** of metric names to execute on this run. Names resolve to built-in metrics first, then to entries in `custom_metrics`.\n- `custom_metrics` — a **definition pool** of custom metrics available to this project. Defining a metric here does **not** run it; it must also appear in `metrics_to_run` (or be passed via `--metrics name1,name2` on the CLI, which is equivalent to overriding `metrics_to_run` for that invocation).\n\n**Minimal example (YAML preferred — human-readable, no JSON escaping for prompts and Python):**\n\n```yaml\nmetrics_to_run:\n  - multi_turn_task_success     # built-in\n  - example_llm_metric          # selected from custom_metrics pool below\n  - agent_turn_count            # selected from custom_metrics pool below\n\ncustom_metrics:\n  - name: example_llm_metric\n    prompt_template: |\n      Rate the agent's response 1-5 for helpfulness and accuracy.\n      Prompt: {prompt}\n      Final response: {response}\n      Full trace (for tool-call and reasoning context): {agent_data}\n      Return JSON: {\"score\": \u003C1|2|3|4|5>, \"explanation\": \"\u003Creason>\"}\n\n  - name: agent_turn_count\n    custom_function: |\n      def evaluate(instance):\n          turns = (instance.get(\"agent_data\") or {}).get(\"turns\", [])\n          return {'score': len(turns)}\n```\n\nJSON is also accepted (same field names, with `prompt_template` and `custom_function` as escaped strings) — but **always prefer YAML** for human-readable configs.\n\nEach entry in `custom_metrics` is dispatched by field: presence of `custom_function` makes it a `CodeExecutionMetric` (deterministic Python); otherwise it's an `LLMMetric` (LLM-as-judge with `prompt_template`). Run `agents-cli eval metric list` to see available built-ins. For full custom-metric field reference (judge model options, sampling counts), see `references\u002Fmetrics-guide.md`.\n\n**Agent trace field model.** For datasets produced by `agents-cli eval generate` (or `eval dataset synthesize`), each eval case exposes three standard fields to a metric:\n\n- `{prompt}` — the user message (or first user turn).\n- `{response}` — the agent's final text response, extracted from the last text-bearing event. In `custom_function` callbacks this is `instance['response']` with shape `{\"role\": \"model\", \"parts\": [{\"text\": \"...\"}]}`.\n- `{agent_data}` — the full structured `turns`\u002F`events` trace, useful when the judge needs to reason about tool calls or intermediate reasoning.\n\n`{reference}` and `{context}` resolve only when the eval case has `reference` \u002F `context` fields populated (e.g., golden-answer datasets); they are not populated by `eval generate` \u002F `eval dataset synthesize`.\n\nCode-based metrics default to **local in-process execution** (no GCP project or region required, but the `evaluate(instance)` function runs with the CLI's privileges). Set `execution: \"remote\"` on the metric to run it server-side in Vertex AI's `CodeExecutionMetric` sandbox instead — that path requires a configured GCP project + region.\n\n---\n\n\n## Common Gotchas\n\n### Use Rubric-Based Tool Evaluation instead of Hardcoded Sequences\n\nEvaluating agent tool usage using strict sequence matching is fragile because agents may call helper tools (like searches or geocoding) in different orders or perform extra proactive steps.\n\nInstead, use **`multi_turn_tool_use_quality`** \u002F **`multi_turn_trajectory_quality`**. These metrics automatically generate content-based and intent-based adaptive rubrics, assessing technical correctness and technical sequence logic semantically using an LLM judge rather than forcing a rigid match.\n\n### App name must match directory name\n\nThe `App` object's `name` parameter MUST match the directory containing your agent:\n\n```python\n# CORRECT - matches the \"app\" directory\napp = App(root_agent=root_agent, name=\"app\")\n\n# WRONG - causes \"Session not found\" errors\napp = App(root_agent=root_agent, name=\"flight_booking_assistant\")\n```\n\n### Vertex eval region\n\n`eval grade` and `eval submit` **default to the `global` endpoint** — they don't inherit the manifest `region` (the eval services support only a subset of regions), and `eval analyze` is `global`-only. Override these per run with `--region \u003CREGION>` (e.g. data residency); the service rejects an unsupported one:\n\n```\n400 FAILED_PRECONDITION: Unsupported region for Vertex Evaluation Service: \u003Cregion>\n```\n\n`eval generate` (without the `--url` flag) and `eval dataset synthesize` run your agent locally, so they honor the agent's own `.env` — notably `GOOGLE_CLOUD_LOCATION`, which selects the model endpoint **when the agent uses Vertex AI** (`GOOGLE_GENAI_USE_VERTEXAI=true`); it's unused with a `GEMINI_API_KEY` (AI Studio). They take **no** `--region` and never override your `.env` with the manifest `region`; change the model region by editing `.env`. One caveat for `synthesize`: its scenario-generation step is a **server-side** eval call at `GOOGLE_CLOUD_LOCATION`, so keep that an eval-supported region (`global` by default) even though the agent itself could run elsewhere.\n\n**No eval region fits your data-residency rules?** Fall back to **local custom metrics** — a `custom_metrics` entry with a `custom_function` (`execution: local`, the default) grades in-process with no GCP region required. You lose the managed built-in metrics, but your `custom_function` can still call an LLM judge in a compliant region itself — so LLM-as-judge grading stays available anywhere.\n\n### The `before_agent_callback` Pattern (State Initialization)\n\nAlways use a callback to initialize session state variables used in your instruction template. This prevents `KeyError` crashes on the first turn:\n\n```python\nasync def initialize_state(callback_context: CallbackContext) -> None:\n    state = callback_context.state\n    if \"user_preferences\" not in state:\n        state[\"user_preferences\"] = {}\n\nroot_agent = Agent(\n    name=\"my_agent\",\n    before_agent_callback=initialize_state,\n    instruction=\"Based on preferences: {user_preferences}...\",\n)\n```\n\n### Model thinking mode may bypass tools\n\nModels with \"thinking\" enabled may skip tool calls. Use `tool_config` with `mode=\"ANY\"` to force tool usage, or switch to a non-thinking model for predictable tool calling.\n\n---\n\n## Common Eval Failure Causes\n\n| Symptom | Cause | Fix |\n|---------|-------|-----|\n| Agent mentions data not in tool output | Hallucination | Tighten agent instructions; add `hallucination` metric |\n| \"Session not found\" error | App name mismatch | Ensure App `name` matches directory name |\n| Score fluctuates between runs | Non-deterministic model | Set `temperature=0` or use rubric-based eval with multiple samples |\n| `tool_use_quality` score low | Wrong tool selected or invalid arguments passed | Refine tool descriptions, instructions, or parameter documentation |\n| LLM judge ignores image\u002Faudio in eval | `get_text_from_content()` skips non-text parts | Use custom metric with vision-capable judge (see `references\u002Fmultimodal-eval.md`) |\n\n---\n\n## Debugging Example\n\nUser says: \"tool_use_quality is low, what's wrong?\"\n\n1. Open the latest `artifacts\u002Fgrade_results\u002Fresults_\u003Ctimestamp>.html` (or read the `.json`) and find the rubric verdicts the adaptive metric generated for the failing case.\n2. Verify whether the agent selected the wrong tool, or called it with wrong arguments — the trace lives in `artifacts\u002Ftraces\u002F`.\n3. Refine the tool's parameters, Python docstring description, or the agent's tool selection instructions to guide the model better.\n4. Rerun `agents-cli eval generate && agents-cli eval grade`.\n5. `agents-cli eval compare \u003Cprev>.json \u003Cnew>.json` to confirm the score improved.\n\n---\n\n## Proving Your Work\n\nDon't assert that eval passes — show the evidence. Concrete output prevents false confidence and catches issues early.\n\n- **After running eval:** Paste the scores table output so the user can see exactly what passed and failed.\n- **After fixing a failure:** Show before\u002Fafter scores for the specific case you fixed, and confirm no other cases regressed.\n- **Before declaring \"eval passes\":** Confirm ALL cases pass, not just the one you were working on. Run `agents-cli eval generate` and `agents-cli eval grade` one final time.\n- **Before moving to deploy:** Show the final `agents-cli eval grade` output with all cases above threshold. This is the gate — no exceptions.\n\n---\n\n## Related Skills\n\n- `\u002Fgoogle-agents-cli-workflow` — Development workflow and the spec-driven build-evaluate-deploy lifecycle\n- `\u002Fgoogle-agents-cli-adk-code` — ADK Python API quick reference for writing agent code\n- `\u002Fgoogle-agents-cli-scaffold` — Project creation and enhancement with `agents-cli scaffold create` \u002F `scaffold enhance`\n- `\u002Fgoogle-agents-cli-deploy` — Deployment targets, CI\u002FCD pipelines, and production workflows\n- `\u002Fgoogle-agents-cli-observability` — Cloud Trace, logging, and monitoring for debugging agent behavior\n","# Agent Evaluation Guide\n\n> **Requires:** `agents-cli` (`uv tool install google-agents-cli`) — [install uv](https:\u002F\u002Fdocs.astral.sh\u002Fuv\u002Fgetting-started\u002Finstallation\u002Findex.md) first if needed.\n\n> **Scaffolded project?** If you used `\u002Fgoogle-agents-cli-scaffold`, you already have `agents-cli eval run` (chains `generate` + `grade`), `tests\u002Feval\u002Fdatasets\u002F`, and `tests\u002Feval\u002Feval_config.yaml`. Start with executing `eval run` and iterate from there.\n\n## Reference Files\n\n| File | Contents |\n|------|----------|\n| `references\u002Fdataset_schema.md` | Canonical EvaluationDataset schema — all field types, JSON examples for single-turn \u002F multi-turn \u002F multi-agent, common mistakes |\n| `references\u002Fmetrics-guide.md` | Complete metrics reference — all built-in metrics, match types, custom metrics, judge model config |\n| `references\u002Fuser-simulation.md` | Dynamic conversation testing — `eval dataset synthesize` flags, what scenarios are, compatible metrics |\n| `references\u002Fbuiltin-tools-eval.md` | google_search and model-internal tools — trajectory behavior, metric compatibility |\n| `references\u002Fmultimodal-eval.md` | Multimodal inputs — eval dataset schema, built-in metric limitations, custom evaluator pattern |\n\n---\n\n## The Quality Flywheel\n\nImproving agent quality is iterative. The 5 stages below describe the loop. Each stage has a Default path (you, the coding agent, do the work directly) and an Opt-in CLI command that delegates to the Agent Platform Eval Service for better quality and scale.\n\n### 1. Prepare Data\n\n**Default:** Use or edit the scaffolded `tests\u002Feval\u002Fdatasets\u002Fbasic-dataset.json` to define single-turn eval inputs. Start with 1–2 cases.\n\n**Opt-in:** `agents-cli eval dataset synthesize` — user-simulate multi-turn datasets when you lack data; its output includes traces, so skip Stage 2 and grade directly. See *Eval Commands* and `references\u002Fuser-simulation.md`.\n\n### 2. Run Inference\n\n`agents-cli eval generate` — executes the agent over the dataset and writes traces to `artifacts\u002Ftraces\u002F`. Run this when you wrote the dataset by hand in Stage 1 (default path). **Skip this stage if you used `eval dataset synthesize`** — that command already produced traces.\n\n### 3. Grade Traces (always run)\n\n`agents-cli eval grade` — scores the traces and writes `results_\u003Cts>.{json,html}` to `artifacts\u002Fgrade_results\u002F`. No opt-in alternative; this is the core. Always run, regardless of how Stages 1 and 2 produced the traces.\n\n> **Shortcut:** `agents-cli eval run` chains Stages 2 + 3 in one command using the default `artifacts\u002Ftraces\u002F` directory between them. Use it for the common path; drop back to the two-step form when you need a custom traces location or want to grade an existing traces file.\n\n### 4. Analyze Failures\n\n**Default:** Open the latest `artifacts\u002Fgrade_results\u002Fresults_\u003Cts>.html` (or `.json`) and identify failed metrics — see *What to fix when scores fail* below for the fix table.\n\n**Opt-in:** `agents-cli eval analyze` — LLM-based failure clustering; prefer when you have 10+ failing cases and want categorized failure modes. See *Eval Commands*.\n\n### 5. Optimize & Code Fix\n\n**Default:** Edit the agent — adjust prompts, tool descriptions, instructions, or eval dataset based on the failure analysis. See *What to fix when scores fail* below for the failure → fix mapping.\n\n**Opt-in:** `agents-cli eval optimize` — runs ADK GEPA prompt optimization against a target metric. Suitable for prompt-only failures. The optimized prompt appears in the command output; capture it and apply it to the agent. For the full per-iteration trace, set `print_detailed_results: true` in your optimization config file.\n\n> **Long-running and expensive.** GEPA optimization makes many LLM calls and can take a long time. Do not run it unless the user explicitly asks for prompt optimization. When you do run it, iterate as far as possible with manual fixes first, then run a **single** final `eval optimize` — never loop on this command.\n\n### Running the loop\n\nIterate stages 2 → 3 → 4 → 5 → 2 (or 1 → 3 → 4 → 5 → 1 if using `synthesize`). After each fix, run `agents-cli eval compare \u003Cprev_results>.json \u003Cnew_results>.json` to confirm the target metric improved without regressing others. Expect 5–10+ iterations per case before it passes — this is normal. Only after a case passes should you expand coverage with more eval cases.\n\nWhen doing 5+ iterations, maintain a task list of which cases are fixed, which are still failing, and what fixes you've tried. Prevents re-attempting the same fix.\n\n### Shortcuts That Waste Time\n\nRecognize these rationalizations and push back — they always cost more time than they save:\n\n| Shortcut | Why it fails |\n|----------|-------------|\n| \"I'll tune the eval thresholds down to make it pass\" | Lowering thresholds hides real failures. If the agent can't meet the bar, fix the agent — don't move the bar. |\n| \"This eval case is flaky, I'll skip it\" | Flaky evals reveal non-determinism in your agent. Fix with `temperature=0`, rubric-based metrics, or more specific instructions — don't delete the signal. |\n| \"I just need to fix the eval dataset, not the agent\" | If you're always adjusting expected outputs, your agent has a behavior problem. Fix the instructions or tool logic first. |\n\n## Choosing the Right Metrics\n\nPick built-in metrics by what you want to measure. Multi-turn metrics evaluate the full conversation; single-turn metrics evaluate one prompt-response pair (with intermediate tool calls). When no built-in fits, write a custom metric (see *Evaluation Configuration Schema* below).\n\n| Goal | Recommended built-in metrics |\n|------|------------------------------|\n| **Did the agent achieve the user's goal?** (catch-all for multi-turn agents) | `multi_turn_task_success` |\n| **Was the agent's reasoning path logical and efficient?** | `multi_turn_trajectory_quality` |\n| **Quality of tool \u002F function calling across turns** | `multi_turn_tool_use_quality` |\n| **Final response quality** (no ground-truth reference needed) | `final_response_quality` |\n| **Factual grounding** (catch hallucinated claims, e.g., RAG agents) | `hallucination` |\n| **Safety policy compliance** | `safety` |\n| **Domain-specific check no built-in covers** | Write a custom `LLMMetric` (LLM-judge) or `CodeExecutionMetric` (deterministic Python). See *Evaluation Configuration Schema* below. |\n\nRun `agents-cli eval metric list` to see all available built-ins. For full metric definitions and rubric details, see the [Agent Platform metric docs](https:\u002F\u002Fcloud.google.com\u002Fgemini-enterprise-agent-platform\u002Foptimize\u002Fevaluation\u002Fmanage-metrics) and `references\u002Fmetrics-guide.md`.\n\n---\n\n## What to fix when scores fail\n\nAfter `agents-cli eval grade` completes, inspect the latest `artifacts\u002Fgrade_results\u002Fresults_\u003Ctimestamp>.json` (or open the `.html` file) for per-case scores and judge rationales — that's the input to every fix decision below.\n\n| Failure | What to change |\n|---------|---------------|\n| `multi_turn_task_success` low | The agent isn't completing the user's goal — fix orchestration, missing tool calls, premature termination, or wrong tool selection |\n| `multi_turn_trajectory_quality` low | The agent reaches the goal inefficiently or takes wrong steps — refine planning prompts, tighten instruction order, or remove redundant tool calls |\n| `multi_turn_tool_use_quality` low | Fix tool descriptions, parameter docstrings, or agent instructions for tool selection |\n| `final_response_quality` low | Read the auto-generated rubric verdicts; refine agent instructions to address the worst-scoring criterion (often clarity, completeness, or instruction-following) |\n| `hallucination` low | Tighten agent instructions to stay grounded in tool output; verify the tool actually returned the data the agent claimed |\n| `safety` low | Add safety guardrails to instructions; review the violating content category in the rubric verdict |\n| Agent calls wrong tools | Fix tool descriptions, agent instructions, or `tool_config` |\n| Agent calls extra tools | Add strict stop instructions, or switch to `multi_turn_tool_use_quality` |\n\nAfter applying a fix, rerun `agents-cli eval generate && agents-cli eval grade` and use `agents-cli eval compare \u003Cprev_results>.json \u003Cnew_results>.json` to confirm the fix improved the target metric without regressing others.\n\n---\n\n## Eval Commands\n\nAll `agents-cli eval` subcommands support `--help` for the authoritative flag list and defaults — run `agents-cli eval \u003Csubcommand> --help` (or `agents-cli eval dataset \u003Csubcommand> --help`) when in doubt. The examples below show the most common invocations; flags can change between releases.\n\n### `eval generate`\n\nRuns an agent over an evaluation dataset and writes traces to disk.\n\nBy default, runs the agent in a local HTTP server (launches the project's `fast_api_app.py` if it exists, or falls back to `adk api_server`) and sends each evaluation case over HTTP. You can generate traces from an already-running agent by passing its HTTP endpoint and app name to `--url` and `--app-name`.\n\n```bash\n# Basic — uses tests\u002Feval\u002Fdatasets\u002F, writes to artifacts\u002Ftraces\u002F\nagents-cli eval generate\n\n# Advanced — custom dataset and output dir\nagents-cli eval generate --dataset tests\u002Feval\u002Fdatasets\u002Fcustom.json -o .\u002Fcustom_traces\u002F\n\n# Against a deployed agent (or one you started manually)\nagents-cli eval generate --url https:\u002F\u002Fmy-agent.run.app --app-name app\n```\n\n### `eval grade`\n\nScores generated traces against built-in or custom metrics. Writes timestamped `results_\u003CYYYYMMDD_HHMMSS>.json` (consumed by `eval compare`) and `.html` (open in a browser) into the output dir, and prints a summary table to the console.\n\n```bash\n# Basic — defaults: traces from artifacts\u002Ftraces\u002F, results to artifacts\u002Fgrade_results\u002F,\n# metrics from tests\u002Feval\u002Feval_config.yaml's metrics_to_run\nagents-cli eval grade\n\n# Advanced 1 — grade traces from a non-default location (the canonical\n# pairing for `eval generate --output custom_traces\u002F`)\nagents-cli eval grade --traces custom_traces\u002F\n\n# Advanced 2 — pick built-in metrics, custom output dir\nagents-cli eval grade --metrics tool_use_quality,safety --output .\u002Fout\u002F\n\n# Advanced 3 — load metrics to run from a config file (YAML or JSON) on a specified trace file.\nagents-cli eval grade --traces .\u002Fartifacts\u002Ftraces\u002Ftrace_1.json --config tests\u002Feval\u002Feval_config.yaml\n```\n\nSee *Evaluation Configuration Schema* below for the config file format.\n\n### `eval compare`\n\nDiffs two `results_*.json` files produced by `eval grade`. Run it after a fix to confirm the target metric improved without regressing others.\n\n```bash\nagents-cli eval compare baseline.json candidate.json\n```\n\n### `eval metric list`\n\nLists the built-in metric names usable with `eval grade --metrics`.\n\n```bash\nagents-cli eval metric list\n```\n\n### `eval analyze`\n\nRuns LLM-based failure clustering and root-cause analysis over a `results_*.json` produced by `eval grade`. Use when you have 10+ failing cases and want categorized failure modes instead of reading the HTML case-by-case. Supported `--metric` values: `multi_turn_task_success`, `multi_turn_tool_use_quality`.\n\n```bash\n# Basic — analyze a results file with default settings\nagents-cli eval analyze --eval-result artifacts\u002Fgrade_results\u002Fresults_\u003Cts>.json\n\n# Advanced — restrict to a specific metric and cap loss clusters\nagents-cli eval analyze \\\n  --eval-result artifacts\u002Fgrade_results\u002Fresults_\u003Cts>.json \\\n  --metric multi_turn_tool_use_quality \\\n  --top-k 5 \\\n  --output artifacts\u002Fanalysis_\u003Cts>.json\n```\n\n### `eval dataset synthesize`\n\nGenerates user scenarios server-side from your agent's tools and instructions, then plays each scenario against an LLM-backed user simulator. The output is a graded-ready trace file with full `agent_data.turns` populated — feed it directly to `eval grade` (skip `eval generate`).\n\n```bash\n# Basic — generate 3 default scenarios (up to 5 turns each) into artifacts\u002Ftraces\u002F\n# (where eval grade reads from by default, so synthesize → grade works without flags)\nagents-cli eval dataset synthesize\n\n# Advanced — guide scenario generation with optional instruction and environment context\nagents-cli eval dataset synthesize \\\n  -n 5 \\\n  --instruction \"Customer asking about refunds\" \\\n  --environment-context \"E-commerce support\" \\\n  --max-turns 8 \\\n  -o tests\u002Feval\u002Fdatasets\u002Frefund_scenarios.json\n```\n\nFor scenario semantics, the full `eval dataset synthesize` flag table, and which simulator internals are not user-configurable, see `references\u002Fuser-simulation.md`.\n\n### `eval optimize`\n\nRuns ADK GEPA prompt optimization against a target metric. Suitable after `eval grade` identifies prompt-only failures (wording, not tool\u002Forchestration logic). `--dataset` and `--target-metric` override values in `--config` when both are passed. **Long-running and expensive — see Stage 5 of the Quality Flywheel for usage guidance.**\n\n```bash\n# Basic — optimize against a single metric on a dataset\nagents-cli eval optimize --dataset tests\u002Feval\u002Fdatasets\u002Fbasic-dataset.json --target-metric final_response_quality\n\n# Advanced — drive multi-metric \u002F multi-dataset optimization from a config file\nagents-cli eval optimize --config tests\u002Feval\u002Foptimization_config.json\n```\n\n### `eval submit` \u002F `eval results` (cloud-side)\n\nThe managed, asynchronous counterpart to the local path, for large or CI-driven runs: `eval submit` hands the dataset and metrics to the Agent Platform Eval Service, and `eval results` polls and downloads the scores. Pass `--resource-name \u003Cagent>` to also run inference server-side (managed `generate` + `grade`); omit it to grade an existing trace (managed `grade`).\n\n```bash\n# Grade an existing trace server-side; returns a run resource name to poll\nagents-cli eval submit --dataset tests\u002Feval\u002Fdatasets\u002Fbasic-dataset.json --dest gs:\u002F\u002Fmy-bucket\n# Add --resource-name projects\u002F\u003Cp>\u002Flocations\u002F\u003Cl>\u002FreasoningEngines\u002F\u003Cid> to run inference too\n\nagents-cli eval results --run-id \u003Crun-resource-name>\n```\n\n---\n\n## Evaluation Dataset Format\n\nAn `EvaluationDataset` is a JSON file with an `eval_cases` array. Cases come in two shapes depending on how they're used:\n\n- **Inference input** (what you give to `eval generate`) — a user prompt or a partial conversation ending in a user prompt. The agent runs and produces traces.\n- **Grading input** (what you give to `eval grade`) — a complete trace including the agent's responses and tool calls. Normally produced by `eval generate` or `eval dataset synthesize`; you don't write these by hand.\n\nSee `references\u002Fdataset_schema.md` for the full canonical schema, all field types, and common mistakes.\n\n### Inference input format\n\nTwo shapes are supported.\n\n**(a) Simple single-turn prompt** — what the scaffolded `tests\u002Feval\u002Fdatasets\u002Fbasic-dataset.json` uses. The agent runs from scratch.\n\n```json\n{\n  \"eval_cases\": [\n    {\n      \"eval_case_id\": \"greeting\",\n      \"prompt\": {\n        \"role\": \"user\",\n        \"parts\": [{\"text\": \"Hello, what can you help me with?\"}]\n      }\n    },\n    {\n      \"eval_case_id\": \"weather_query\",\n      \"prompt\": {\n        \"role\": \"user\",\n        \"parts\": [{\"text\": \"What's the weather like in San Francisco?\"}]\n      }\n    }\n  ]\n}\n```\n\n**(b) Multi-turn continuation via `agent_data`** — a partial conversation whose last turn ends with a user message; the agent's next response is evaluated. See `references\u002Fdataset_schema.md` (*Multi-Turn \u002F Multi-Agent Dataset*) for the JSON shape.\n\n### Grading input format (traces)\n\nA complete trace — agent responses plus `function_call` \u002F `function_response` parts — normally produced by `eval generate` \u002F `eval dataset synthesize` (you don't write these by hand). Authors are `\"user\"`, an agent ID from the `agents` map, or `\"tool\"`. See `references\u002Fdataset_schema.md` for the trace shape, multi-agent examples, and the full type reference.\n\n---\n\n## Evaluation Configuration Schema\n\n`agents-cli eval grade --config \u003Cpath>` accepts a single configuration file in either **YAML** (`.yaml` \u002F `.yml`) or **JSON** (`.json`). The file declares two parts:\n\n- `metrics_to_run` — the **selection list** of metric names to execute on this run. Names resolve to built-in metrics first, then to entries in `custom_metrics`.\n- `custom_metrics` — a **definition pool** of custom metrics available to this project. Defining a metric here does **not** run it; it must also appear in `metrics_to_run` (or be passed via `--metrics name1,name2` on the CLI, which is equivalent to overriding `metrics_to_run` for that invocation).\n\n**Minimal example (YAML preferred — human-readable, no JSON escaping for prompts and Python):**\n\n```yaml\nmetrics_to_run:\n  - multi_turn_task_success     # built-in\n  - example_llm_metric          # selected from custom_metrics pool below\n  - agent_turn_count            # selected from custom_metrics pool below\n\ncustom_metrics:\n  - name: example_llm_metric\n    prompt_template: |\n      Rate the agent's response 1-5 for helpfulness and accuracy.\n      Prompt: {prompt}\n      Final response: {response}\n      Full trace (for tool-call and reasoning context): {agent_data}\n      Return JSON: {\"score\": \u003C1|2|3|4|5>, \"explanation\": \"\u003Creason>\"}\n\n  - name: agent_turn_count\n    custom_function: |\n      def evaluate(instance):\n          turns = (instance.get(\"agent_data\") or {}).get(\"turns\", [])\n          return {'score': len(turns)}\n```\n\nJSON is also accepted (same field names, with `prompt_template` and `custom_function` as escaped strings) — but **always prefer YAML** for human-readable configs.\n\nEach entry in `custom_metrics` is dispatched by field: presence of `custom_function` makes it a `CodeExecutionMetric` (deterministic Python); otherwise it's an `LLMMetric` (LLM-as-judge with `prompt_template`). Run `agents-cli eval metric list` to see available built-ins. For full custom-metric field reference (judge model options, sampling counts), see `references\u002Fmetrics-guide.md`.\n\n**Agent trace field model.** For datasets produced by `agents-cli eval generate` (or `eval dataset synthesize`), each eval case exposes three standard fields to a metric:\n\n- `{prompt}` — the user message (or first user turn).\n- `{response}` — the agent's final text response, extracted from the last text-bearing event. In `custom_function` callbacks this is `instance['response']` with shape `{\"role\": \"model\", \"parts\": [{\"text\": \"...\"}]}`.\n- `{agent_data}` — the full structured `turns`\u002F`events` trace, useful when the judge needs to reason about tool calls or intermediate reasoning.\n\n`{reference}` and `{context}` resolve only when the eval case has `reference` \u002F `context` fields populated (e.g., golden-answer datasets); they are not populated by `eval generate` \u002F `eval dataset synthesize`.\n\nCode-based metrics default to **local in-process execution** (no GCP project or region required, but the `evaluate(instance)` function runs with the CLI's privileges). Set `execution: \"remote\"` on the metric to run it server-side in Vertex AI's `CodeExecutionMetric` sandbox instead — that path requires a configured GCP project + region.\n\n---\n\n\n## Common Gotchas\n\n### Use Rubric-Based Tool Evaluation instead of Hardcoded Sequences\n\nEvaluating agent tool usage using strict sequence matching is fragile because agents may call helper tools (like searches or geocoding) in different orders or perform extra proactive steps.\n\nInstead, use **`multi_turn_tool_use_quality`** \u002F **`multi_turn_trajectory_quality`**. These metrics automatically generate content-based and intent-based adaptive rubrics, assessing technical correctness and technical sequence logic semantically using an LLM judge rather than forcing a rigid match.\n\n### App name must match directory name\n\nThe `App` object's `name` parameter MUST match the directory containing your agent:\n\n```python\n# CORRECT - matches the \"app\" directory\napp = App(root_agent=root_agent, name=\"app\")\n\n# WRONG - causes \"Session not found\" errors\napp = App(root_agent=root_agent, name=\"flight_booking_assistant\")\n```\n\n### Vertex eval region\n\n`eval grade` and `eval submit` **default to the `global` endpoint** — they don't inherit the manifest `region` (the eval services support only a subset of regions), and `eval analyze` is `global`-only. Override these per run with `--region \u003CREGION>` (e.g. data residency); the service rejects an unsupported one:\n\n```\n400 FAILED_PRECONDITION: Unsupported region for Vertex Evaluation Service: \u003Cregion>\n```\n\n`eval generate` (without the `--url` flag) and `eval dataset synthesize` run your agent locally, so they honor the agent's own `.env` — notably `GOOGLE_CLOUD_LOCATION`, which selects the model endpoint **when the agent uses Vertex AI** (`GOOGLE_GENAI_USE_VERTEXAI=true`); it's unused with a `GEMINI_API_KEY` (AI Studio). They take **no** `--region` and never override your `.env` with the manifest `region`; change the model region by editing `.env`. One caveat for `synthesize`: its scenario-generation step is a **server-side** eval call at `GOOGLE_CLOUD_LOCATION`, so keep that an eval-supported region (`global` by default) even though the agent itself could run elsewhere.\n\n**No eval region fits your data-residency rules?** Fall back to **local custom metrics** — a `custom_metrics` entry with a `custom_function` (`execution: local`, the default) grades in-process with no GCP region required. You lose the managed built-in metrics, but your `custom_function` can still call an LLM judge in a compliant region itself — so LLM-as-judge grading stays available anywhere.\n\n### The `before_agent_callback` Pattern (State Initialization)\n\nAlways use a callback to initialize session state variables used in your instruction template. This prevents `KeyError` crashes on the first turn:\n\n```python\nasync def initialize_state(callback_context: CallbackContext) -> None:\n    state = callback_context.state\n    if \"user_preferences\" not in state:\n        state[\"user_preferences\"] = {}\n\nroot_agent = Agent(\n    name=\"my_agent\",\n    before_agent_callback=initialize_state,\n    instruction=\"Based on preferences: {user_preferences}...\",\n)\n```\n\n### Model thinking mode may bypass tools\n\nModels with \"thinking\" enabled may skip tool calls. Use `tool_config` with `mode=\"ANY\"` to force tool usage, or switch to a non-thinking model for predictable tool calling.\n\n---\n\n## Common Eval Failure Causes\n\n| Symptom | Cause | Fix |\n|---------|-------|-----|\n| Agent mentions data not in tool output | Hallucination | Tighten agent instructions; add `hallucination` metric |\n| \"Session not found\" error | App name mismatch | Ensure App `name` matches directory name |\n| Score fluctuates between runs | Non-deterministic model | Set `temperature=0` or use rubric-based eval with multiple samples |\n| `tool_use_quality` score low | Wrong tool selected or invalid arguments passed | Refine tool descriptions, instructions, or parameter documentation |\n| LLM judge ignores image\u002Faudio in eval | `get_text_from_content()` skips non-text parts | Use custom metric with vision-capable judge (see `references\u002Fmultimodal-eval.md`) |\n\n---\n\n## Debugging Example\n\nUser says: \"tool_use_quality is low, what's wrong?\"\n\n1. Open the latest `artifacts\u002Fgrade_results\u002Fresults_\u003Ctimestamp>.html` (or read the `.json`) and find the rubric verdicts the adaptive metric generated for the failing case.\n2. Verify whether the agent selected the wrong tool, or called it with wrong arguments — the trace lives in `artifacts\u002Ftraces\u002F`.\n3. Refine the tool's parameters, Python docstring description, or the agent's tool selection instructions to guide the model better.\n4. Rerun `agents-cli eval generate && agents-cli eval grade`.\n5. `agents-cli eval compare \u003Cprev>.json \u003Cnew>.json` to confirm the score improved.\n\n---\n\n## Proving Your Work\n\nDon't assert that eval passes — show the evidence. Concrete output prevents false confidence and catches issues early.\n\n- **After running eval:** Paste the scores table output so the user can see exactly what passed and failed.\n- **After fixing a failure:** Show before\u002Fafter scores for the specific case you fixed, and confirm no other cases regressed.\n- **Before declaring \"eval passes\":** Confirm ALL cases pass, not just the one you were working on. Run `agents-cli eval generate` and `agents-cli eval grade` one final time.\n- **Before moving to deploy:** Show the final `agents-cli eval grade` output with all cases above threshold. This is the gate — no exceptions.\n\n---\n\n## Related Skills\n\n- `\u002Fgoogle-agents-cli-workflow` — Development workflow and the spec-driven build-evaluate-deploy lifecycle\n- `\u002Fgoogle-agents-cli-adk-code` — ADK Python API quick reference for writing agent code\n- `\u002Fgoogle-agents-cli-scaffold` — Project creation and enhancement with `agents-cli scaffold create` \u002F `scaffold enhance`\n- `\u002Fgoogle-agents-cli-deploy` — Deployment targets, CI\u002FCD pipelines, and production workflows\n- `\u002Fgoogle-agents-cli-observability` — Cloud Trace, logging, and monitoring for debugging agent behavior\n",{"metadata":24,"description":28},{"author":25,"license":26,"version":27},"Google","Apache-2.0","1.3.1","This skill should be used when the user wants to \"run an evaluation\", \"evaluate my ADK agent\", \"write an eval dataset\", \"analyze eval failures\", \"compare eval results\", \"optimize agent\", or needs guidance on the Agent Platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Do NOT use for API code patterns (use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).\n","main","58843b159d7a4050d6b785a98b908cce505c0b92","agent\u002Fskills\u002Fgoogle-agents-cli-eval",0,"https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent","https:\u002F\u002Fgithub.com\u002FReihantt6","reihsns.web.id","2026-08-19 20:08:18",{"source":8,"skillName":5,"commands":38,"prompt":48,"zipAvailable":49},[39,42,45],{"tool":40,"command":41},"npx","npx skills add Reihantt6\u002Fsns-myagent --skill google-agents-cli-eval",{"tool":43,"command":44},"bunx","bunx skills add Reihantt6\u002Fsns-myagent --skill google-agents-cli-eval",{"tool":46,"command":47},"pnpm","pnpm dlx skills add Reihantt6\u002Fsns-myagent --skill google-agents-cli-eval","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fgoogle-agents-cli-eval\u002FSKILL.md",true,[51,57,61,65,69],{"path":52,"resourceType":53,"resourceTypeDesc":54,"sizeBytes":55,"sourceHtmlUrl":56},"references\u002Fbuiltin-tools-eval.md","REFERENCE","参考文档",5906,"https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fgoogle-agents-cli-eval\u002Freferences\u002Fbuiltin-tools-eval.md",{"path":58,"resourceType":53,"resourceTypeDesc":54,"sizeBytes":59,"sourceHtmlUrl":60},"references\u002Fdataset_schema.md",8168,"https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fgoogle-agents-cli-eval\u002Freferences\u002Fdataset_schema.md",{"path":62,"resourceType":53,"resourceTypeDesc":54,"sizeBytes":63,"sourceHtmlUrl":64},"references\u002Fmetrics-guide.md",8299,"https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fgoogle-agents-cli-eval\u002Freferences\u002Fmetrics-guide.md",{"path":66,"resourceType":53,"resourceTypeDesc":54,"sizeBytes":67,"sourceHtmlUrl":68},"references\u002Fmultimodal-eval.md",5382,"https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fgoogle-agents-cli-eval\u002Freferences\u002Fmultimodal-eval.md",{"path":70,"resourceType":53,"resourceTypeDesc":54,"sizeBytes":71,"sourceHtmlUrl":72},"references\u002Fuser-simulation.md",6345,"https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fgoogle-agents-cli-eval\u002Freferences\u002Fuser-simulation.md",[74,81,88,95,102,109],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":19},"01M77XDCYZ407XYMF2T4P6F7VT","youtube-transcript","Extract transcripts from YouTube videos. Use when the user asks for a transcript, subtitles, or captions of a YouTube video and provides a YouTube URL (youtube.com\u002Fwatch?v=, youtu.be\u002F, or similar). Supports output with or without timestamps.","https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fyoutube-transcript\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M77XD7PBC98J6MXGJJ2TJ8DX","youtube-thumbnail-design","YouTube thumbnail design with specific dimensions, contrast rules, and mobile preview optimization. Covers safe zones, text placement, face expression psychology, and A\u002FB testing. Use for: YouTube thumbnails, video cover images, click-through optimization. Triggers: youtube thumbnail, thumbnail design, video thumbnail, click through rate, ctr optimization, youtube cover, video cover image, thumbnail maker, thumbnail tips, youtube design, video preview image","https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fyoutube-thumbnail-design\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M77XD2541C8HXJ6RHYZP4NGJ","youtube-short-maker","Make YouTube Shorts that hold attention with Pexo — vertical 9:16, fast-paced, hook in the first 3 seconds. Describe your topic and Pexo writes the script, generates the shots, picks the models, and assembles a finished Short with music and captions. Use for YouTube Shorts: \"youtube short\", \"make a youtube short\", \"shorts video\", \"short-form video\". NOT for long-form or landscape video.","https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fyoutube-short-maker\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":19},"01M77XCXKVSYK7KTZ3PQBVVGFD","xcode-project-setup","Safely modifies Xcode projects (.pbxproj) to add Swift Packages and link files. Use this skill whenever an iOS project needs dependencies installed (e.g. Firebase, Alamofire).","https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fxcode-project-setup\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":19},"01M77XCR1DV8ZQK2NHWKVY4EWX","x402","Set up Browser Use Cloud payments with x402 — pay per request from a crypto wallet (USDC on Base mainnet), no signup or API key. Two setups it works out up front — \"just use it\" (set up a wallet so you or Claude Code can run cloud browser tasks paid from the wallet — Claude writes and runs throwaway scripts, nothing touches your codebase) or \"build it in\" (install the SDK and write the key + code into your project). Walks through wallet setup, funding, .env, and a ~$1 test run. Use when the user asks about x402, pay-per-use, USDC payments, or wants Browser Use Cloud without an API key. For the free-tier signup (reverse-CAPTCHA → API key), use `browser-use cloud signup` or the `cloud` skill instead.","https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fx402\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":19},"01M77XCMD3R3ZBZQQSW6E0793B","x-twitter-scraper","Build GitHub Copilot workflows with Xquik X API SDKs, REST endpoints, hosted Apify Actor runs, MCP tools, TweetClaw OpenClaw plugin installs, signed webhooks, tweet search, user lookup, follower exports, media actions, and agent automation.","https:\u002F\u002Fgithub.com\u002FReihantt6\u002Fsns-myagent\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fx-twitter-scraper\u002FSKILL.md",[],[]]