[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M69AVMVG4RZ9VYBT2GKWT0M8":3,"skill-same:01M69AVMVG4RZ9VYBT2GKWT0M8":54},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":26,"linkOnly":27,"markdownContent":28,"markdownBody":29,"frontMatter":30,"refName":31,"commitSha":32,"skillDirectory":33,"forksCount":34,"repositoryHtmlUrl":35,"repositoryHomepageUrl":6,"ownerHtmlUrl":36,"ownerBlogUrl":37,"ownerDescription":38,"updatedAt":39,"install":40,"resources":53},"01M69AVMVG4RZ9VYBT2GKWT0M8","eval-analysis",null,"Comprehensive guide for analyzing SABER evaluation results — model comparison, agent architecture comparison, domain-specific analysis, and cross-domain aggregate analysis. Use this when asked to analyze eval results, compare models, generate visualizations, interpret scores, investigate cost-efficiency trade-offs, or extend analysis to new domains.","microsoft\u002FACESEvals","microsoft","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F6154722?v=4","01M698VYDXZ6JW5WH3CRV650R8","Microsoft","01M69AVF6DCZVH3H6ZKYRQD2VV",22,"MIT","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FACESEvals\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Feval-analysis\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[24,25],"eval analysis","architecture","2026-11-10 10:12:30",false,"---\nname: eval-analysis\ndescription: >\n  Comprehensive guide for analyzing SABER evaluation results — model comparison, agent architecture\n  comparison, domain-specific analysis, and cross-domain aggregate analysis. Use this when asked to\n  analyze eval results, compare models, generate visualizations, interpret scores, investigate\n  cost-efficiency trade-offs, or extend analysis to new domains.\n---\n\n# Eval Analysis Skill\n\n## Overview\n\nSABER evaluation analysis is a structured data science framework for comparing AI agent performance across cybersecurity benchmark domains. The analysis pipeline transforms `.eval` log files into actionable insights through 12+ standardized experiments plus domain-specific analyses.\n\n## Getting Started: Sample Eval Data\n\n**No evals to analyze yet?** The repo ships with pre-run sample `.eval` files and can auto-download more from HuggingFace.\n\n### Option 1: Use bundled sample evals (fastest)\n\nThe `eval_samples\u002F` directory contains pre-run evaluation results for all 3 domains across 5 models and 3 agent architectures:\n\n```\neval_samples\u002F\n├── excytin\u002F          # 20 eval files (5 models × default + reasoning baselines + 3 agents)\n├── cybench\u002F          # 17 eval files\n└── cti_realm\u002F        # 11 eval files\n```\n\n**Models included**: Claude Haiku 4.5, Claude Sonnet 4.6, Claude Opus 4.6, GPT-5.4, GPT-5.4-mini  \n**Agent architectures included**: React, GH Copilot, Claude Code (for Sonnet 4.6)  \n**Baselines**: No-reasoning\u002Fno-thinking variants for extended thinking comparison\n\nThe notebooks are pre-configured to load from `eval_samples\u002F`. Just run any notebook — no configuration needed.\n\n### Option 2: Auto-download from HuggingFace\n\nIf `.eval` files are missing locally, `ensure_eval_files()` automatically downloads them from the **[AcesEvals HuggingFace dataset](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fanandmudgerikar\u002FAcesEvals)**:\n\n```python\nfrom saber_analysis import ensure_eval_files\n\nensure_eval_files(\n    eval_logs=EVAL_LOGS,       # {display_name: filename.eval}\n    log_dir=LOG_DIR,           # local directory\n    domain='excytin',          # HuggingFace subfolder\n    fallback_dirs=[            # check these before downloading\n        str(REPO_ROOT \u002F 'eval_samples' \u002F 'excytin'),\n        str(REPO_ROOT \u002F 'latest_experiments' \u002F 'excytin'),\n    ],\n)\n```\n\n**Resolution order**: local `log_dir` → `fallback_dirs` → HuggingFace download  \n**HuggingFace repo**: `anandmudgerikar\u002FAcesEvals` (dataset type)  \n**HuggingFace path**: `latest_experiment_samples\u002F{domain}\u002F{filename}.eval`\n\n### Option 3: Run your own evaluations\n\nIf you need results for a model or agent architecture not in the sample evals:\n\n```bash\n# Run for a new model\nuv run inspect eval domains\u002F\u003Cdomain> --model \u003Capi\u002Fmodel-id> --display plain\n\n# Run for a specific agent architecture\nuv run inspect eval domains\u002F\u003Cdomain> --model \u003Capi\u002Fmodel-id> -T agent=react --display plain\nuv run inspect eval domains\u002F\u003Cdomain> --model \u003Capi\u002Fmodel-id> -T agent=copilot --display plain\n\n# Run with extended reasoning disabled (for baseline comparison)\n# Set INSPECT_EVAL_MODEL_ARGS=reasoning_effort=none or configure in .env\n```\n\nThen copy the `.eval` file to `eval_samples\u002F\u003Cdomain>\u002F` or `latest_experiments\u002F\u003Cdomain>\u002F` and add it to the notebook's `EVAL_LOGS` configuration.\n\n### What's available vs. what you need to run\n\n| Domain | Models (sample evals) | Agent Architectures (sample evals) | What to run yourself |\n|--------|-----------------------|-------------------------------------|----------------------|\n| Excytin | Haiku 4.5, Sonnet 4.6, Opus 4.6, GPT-5.4, GPT-5.4-mini | React, Copilot, Claude Code (Sonnet) | New models, new agents on non-Sonnet models |\n| CyBench | Haiku 4.5, Sonnet 4.6, Opus 4.6, GPT-5.4, GPT-5.4-mini | React, Copilot, Claude Code (Sonnet) | New models, new agents, more challenges |\n| CTI Realm | Haiku 4.5, Sonnet 4.6, Opus 4.6, GPT-5.4, GPT-5.4-mini | Copilot, Claude Code (Sonnet) | React for non-Sonnet models, new models |\n\n### Notebook Inventory\n\n| Notebook | Scope | When to use |\n|----------|-------|-------------|\n| `notebooks\u002Feval_analysis.ipynb` | Domain-agnostic template | Analyzing any single domain's model results; starting point for new domains |\n| `notebooks\u002Fexcytin_analysis.ipynb` | Excytin-specific (self-contained) | Full analysis of Excytin incident response domain |\n| `notebooks\u002Fcybench_analysis.ipynb` | CyBench-specific (self-contained) | Full analysis of CyBench CTF challenges |\n| `notebooks\u002Fcti_realm_analysis.ipynb` | CTI Realm-specific (self-contained) | Full analysis of CTI Realm threat intel domain |\n| `notebooks\u002Fagent_architecture_analysis.ipynb` | Agent comparison template | Comparing agent architectures (React, Copilot, Claude Code) on any domain |\n| `notebooks\u002Fexcytin_agent_architecture_analysis.ipynb` | Excytin agent comparison | Agent architecture analysis specific to Excytin |\n| `notebooks\u002Fcybench_agent_architecture_analysis.ipynb` | CyBench agent comparison | Agent architecture analysis specific to CyBench |\n| `notebooks\u002Fcti_realm_agent_architecture_analysis.ipynb` | CTI Realm agent comparison | Agent architecture analysis specific to CTI Realm |\n| `notebooks\u002Faggregate_model_analysis.ipynb` | Cross-domain model comparison | Aggregate model analysis across all domains (normalized) |\n| `notebooks\u002Faggregate_agent_architecture_analysis.ipynb` | Cross-domain agent comparison | Aggregate agent architecture analysis across all domains |\n| `notebooks\u002Fmodel_safety_filters_analysis.ipynb` | Safety filter investigation | Analyzing model safety refusals and guardrail triggers |\n| `notebooks\u002Fexcytin_basic_analysis.ipynb` | Quick Excytin analysis | Lightweight Excytin analysis for rapid iteration |\n\n### Shared Analysis Library\n\nThe `notebooks\u002Fsaber_analysis\u002F` package provides reusable utilities:\n\n| Module | Functions | Purpose |\n|--------|-----------|---------|\n| `data_loader.py` | `ensure_eval_files()`, `load_eval_logs()`, `load_baseline_logs()`, `load_trajectory_data()` | Standardized .eval parsing; HuggingFace fallback downloads; typed inspect_ai log API |\n| `cost.py` | `calc_cost()`, `extract_cost_rows()` | Token→dollar conversion using per-model pricing |\n| `plots.py` | `setup_plotting()`, `make_legend_patches()`, `classify_score_type()` | Consistent matplotlib styling across all notebooks |\n\n---\n\n## Data Pipeline\n\n```\n.eval ZIP archives\n    │\n    ├── header.json          → run metadata, overall saber_overall scores\n    ├── samples\u002F*.json       → per-sample messages, scores, timing, tool calls\n    └── summaries.json       → aggregated score summaries\n    │\n    ▼\ninspect_ai.log API\n    ├── read_eval_log(path, header_only=True)       → EvalLog with results.scores\n    ├── read_eval_log_sample_summaries(path)         → per-sample scores & timing\n    └── read_eval_log_sample(path, id=...)           → full messages & tool calls\n    │\n    ▼\nsaber_analysis library\n    ├── ensure_eval_files()  → locates locally or downloads from HuggingFace\n    ├── load_eval_logs()     → parses into DataFrames (overall_df, samples_df, subtasks_df)\n    ├── load_trajectory_data() → extracts tool calls, steps, timing per sample\n    └── extract_cost_rows()  → builds cost DataFrame from token usage + pricing\n    │\n    ▼\nNotebook Configuration\n    ├── EVAL_LOGS: dict[str, str]      → {display_name: filename.eval}\n    ├── COLORS: dict[str, str]         → {display_name: hex_color}\n    ├── PRICING: dict[str, dict]       → {api_model_id: {input, output, cache_read, cache_write}}\n    ├── GROUP_FN: Callable             → domain-specific sample ID → group label\n    └── NO_THINKING_LOGS (optional)    → baseline without extended reasoning\n    │\n    ▼\nAnalysis Cells (Experiments 1–14+)\n    │\n    ▼\nArtifacts saved to notebooks\u002Fartifacts\u002F{domain}\u002F*.png\n```\n\n### Key DataFrames\n\n| DataFrame | Columns | Source |\n|-----------|---------|--------|\n| `overall_df` | model, mean, stderr | `load_eval_logs()` — header-level saber_overall |\n| `samples_df` | model, sample_id, group, score | `load_eval_logs()` — per-sample saber_overall |\n| `subtasks_df` | model, sample_id, group, score_type, score | `load_eval_logs()` — submission\u002Fcheckpoint\u002Faggregate breakdown |\n| `traj_df` | model, sample_id, n_tool_calls, n_steps, total_time, tool_counts | `load_trajectory_data()` — agent behavior data |\n| `cost_df` | model, score, input_tokens, output_tokens, cache_read, cache_write, reasoning_tokens, total_cost, cost_per_sample | `extract_cost_rows()` |\n\n---\n\n## Domain-Agnostic Analysis Types (Experiments 1–14)\n\nThese 14 core analyses apply to **any** SABER domain. They appear in `eval_analysis.ipynb` and every domain-specific notebook.\n\n### Experiment 1: Overall Reward Distribution\n\n- **Metric**: Mean `saber_overall` score per model\n- **Visualization**: Bar chart with error bars (stderr) + optional hatched overlay for reasoning delta\n- **Interpretation**:\n  - Higher bars = better performance\n  - Overlapping error bars = statistically insignificant differences\n  - Reasoning delta (hatched) shows impact of extended thinking (positive = reasoning helps)\n- **Code pattern**:\n  ```python\n  overall_df.plot.bar(x='model', y='mean', yerr='stderr')\n  ```\n\n### Experiment 2: Per-Group Breakdown (Domain-Specific Grouping)\n\n- **Metric**: Scores segmented by `GROUP_FN` dimension\n- **Visualization**: Clustered bar chart (models × groups)\n- **Group functions by domain**:\n  - **Excytin**: `'_'.join(id.split('_')[:2])` → 8 security incidents\n  - **CyBench**: `'_'.join(id.split('_')[:-2])` → challenge types\n  - **CTI Realm**: `id.split('_')[0]` → 3 platforms (Linux, AKS, Cloud)\n- **Interpretation**:\n  - Large within-group variance = model-dependent difficulty\n  - Shifting model rankings across groups = no single dominant model\n  - Uniform clusters = inherently hard\u002Feasy groups\n- **Requires**: Setting `GROUP_FN` in configuration; set `GROUP_FN = None` to skip\n\n### Experiment 3: Cost Analysis\n\n- **Metrics**: Total cost ($), cost per sample, token breakdown (input\u002Foutput\u002Fcache\u002Freasoning)\n- **Visualizations**:\n  1. Total cost bar chart\n  2. Cost per sample bar chart\n  3. Pareto frontier (Score vs. Cost — upper-left corner = best value)\n- **Interpretation**:\n  - Pareto-dominant models: no other model is both cheaper AND better\n  - Arrows on Pareto chart show reasoning cost delta\n  - Diminishing returns visible from frontier plateau\n- **Code pattern**:\n  ```python\n  cost_df = pd.DataFrame(extract_cost_rows(EVAL_LOGS, LOG_DIR, PRICING, N_SAMPLES))\n  ```\n\n### Experiment 4: Token Usage Breakdown\n\n- **Metrics**: Input, output, cache read\u002Fwrite, reasoning tokens per model\n- **Visualization**: Horizontal stacked bar chart\n- **Interpretation**:\n  - **Input tokens**: Prompts, tool results, conversation history\n  - **Output tokens**: Generated text, tool calls, answers\n  - **Cache read\u002Fwrite**: Prompt caching (Claude > GPT generally)\n  - **Reasoning tokens**: Extended thinking overhead; highly variable by model family\n  - Large hatched sections = reasoning adds substantial overhead\n\n### Experiment 5: Cost Efficiency (Reward per Dollar)\n\n- **Formula**: `Cost Efficiency = Mean SABER Score \u002F Total Cost ($)`\n- **Visualization**: Bar chart (taller = more efficient)\n- **Interpretation**:\n  - Highest-cost model is not always least efficient\n  - Budget-constrained deployments should prioritize high-efficiency models\n  - Reasoning may increase OR decrease efficiency (depends on score gain vs. cost increase)\n\n### Experiment 6: Sub-Task Score Breakdown\n\n- **Metrics**: Submission score, checkpoint scores, aggregate score per model\n- **Visualization**: Grouped bars (submission\u002Fcheckpoints\u002Faggregate) with reasoning delta overlay\n- **Interpretation**:\n  - **Submission >> Checkpoints**: Model reaches correct conclusion without thorough investigation (may be guessing; risky in production)\n  - **Checkpoints >> Submission**: Model investigates well but fails at synthesis\n  - **Aligned**: Investigation matches answer quality (ideal)\n\n### Experiment 7: Checkpoint Score Distributions\n\n- **Visualization**: Violin plots per model\n- **Interpretation**:\n  - **Bimodal** (peaks at 0 and 1): Binary pass\u002Ffail checkpoint; few partial successes\n  - **Continuous**: Fine-grained scoring; various intermediate scores\n  - **Wide violins**: Inconsistent across samples; **narrow**: consistent\n  - **Long upper tails**: Some samples max out the checkpoint score\n\n### Experiment 8: Domain-Specific Gap Analysis\n\n- **Purpose**: Submission vs. checkpoint gap per domain grouping dimension\n- **Visualization**: Heatmap (models × groups) showing gap = submission - checkpoint\n- **Interpretation**:\n  - Red (positive gap): Model guesses correctly without thorough investigation\n  - Green (negative gap): Model investigates well but fails at synthesis\n  - Yellow (aligned): Investigation matches answer quality\n\n### Experiment 9: Agent Trajectory Analysis\n\nData loading step that extracts per-sample behavioral data:\n- `n_tool_calls`: Actual tool invocations per sample\n- `n_steps`: Assistant message turns per sample\n- `tool_counts`: Per-tool breakdown (bash, python, domain tools)\n- `total_time`: Wall-clock seconds per sample\n\n### Experiment 9a: Score vs. Tool-Call Limit (Effort Budget)\n\n- **Calculation**: For each hypothetical limit N, mean score where samples needing >N calls score 0\n- **Visualization**: Curves per model converging to actual mean at real tool-call cap\n- **Interpretation**:\n  - **Steep early rise**: Efficient model (solves many tasks in first 5–15 calls)\n  - **Plateau**: Budget increase yields diminishing returns\n  - **Key finding**: All models typically reach 95% of final score by ~20–25 calls (Excytin) or ~33–35 (CTI Realm)\n\n### Experiment 9b: Effort Distribution per Model\n\n- **Visualization**: Violin plots of tool calls and steps per sample per model\n- **Interpretation**:\n  - **Narrow, low**: Efficient (quick solutions)\n  - **Wide, high**: Variable effort or struggling\n  - **Long upper tail**: Some samples hit tool-call limit (binding constraint)\n\n### Experiment 10: Tool Usage Analysis\n\n- **Visualization**: Dual panel — left: mean tool calls per tool (absolute); right: tool call ratio (fraction)\n- **Interpretation**:\n  - Domain determines tool preferences (Excytin → 95%+ bash; CTI Realm → 48% KQL tools)\n  - Cross-model consistency = task design funnels agents toward certain tools\n  - Model-specific tool preferences reveal different investigation strategies\n\n### Experiment 11: Time Efficiency & Reward per Tool Call\n\n- **Formulas**:\n  - `Reward\u002FCall = Mean SABER Score \u002F Mean Tool Calls`\n  - `Reward\u002FMinute = Mean Score \u002F Wall-Clock Time (min)`\n- **Visualizations**: Bar charts, scatter plots (effort vs. score with trend line), time distribution violin plots\n- **Interpretation**:\n  - High reward\u002Fcall = doesn't waste invocations\n  - High reward\u002Fminute = fast AND accurate\n  - Time outliers reveal latency issues (e.g., GPT 10+ min vs. Claude 2–3 min)\n\n### Experiment 12: Score Outcome Segmentation by Effort\n\n- **Methodology**: Segment by tool-call quartile (Q1–Q4); show perfect\u002Fpartial\u002Fzero score rates\n- **Visualization**: Stacked bar chart per model per quartile\n- **Interpretation**:\n  - Perfect-score rate drops Q1→Q4: High effort = harder task (effort doesn't help)\n  - Zero-score rate doubles Q1→Q4: Tool-call limit is binding constraint on hard tasks\n  - Partial scores rise in Q4: Upper-effort tasks succeed only partially within budget\n\n### Experiment 13: Cross-Model Difficulty Agreement\n\n- **Calculation**: Spearman rank correlation of per-sample scores for each model pair\n- **Visualization**: Heatmap with ρ values and significance markers\n- **Interpretation**:\n  - **High ρ (>0.7)**: Models agree — difficulty is intrinsic to task\n  - **Low ρ (\u003C0.3)**: Models struggle differently — complementary strengths, good for ensembles\n  - **Within-family > cross-family**: Same-vendor models agree more than different vendors\n\n---\n\n## Domain-Specific Analyses\n\n### Excytin (Incident Response Forensics)\n\n**Domain context**: 8 security incidents, 599 tasks, agents investigate via SQL queries through bash tool.\n\n| Analysis | What it reveals |\n|----------|-----------------|\n| **Per-Incident Breakdown** (Exp 2) | Score variation across 8 incident types |\n| **Submission vs. Checkpoint Gap Heatmap** (Exp 8) | Per-incident × per-model gap showing investigation thoroughness |\n| **SQL Query Analysis** (Exp 14) | Classifies all SQL queries into success\u002Flarge-result\u002Fempty\u002Ferror; reveals query quality differences across models |\n\n**SQL Query Classification**:\n- **Success (≤100 rows)**: Manageable, agent can parse\n- **Large result (>100 rows)**: Unrefined query; should add LIMIT\u002FWHERE\n- **Empty result**: Valid syntax but no matching data (dominant failure mode)\n- **Error**: Syntax error, missing table, permission denied\n\n**Key Excytin insights**:\n- All models show large positive submission-checkpoint gaps (0.459–0.573) — bypassing investigation\n- Bash tool dominates (95–100% of tool calls); Haiku uniquely uses python (~5%)\n- Query quality separates models: Opus 79% success rate vs. GPT-5.4-mini 61%\n- Empty results are more problematic than syntax errors (agents waste effort on unproductive queries)\n\n### CyBench (CTF Challenges)\n\n**Domain context**: CTF exploitation challenges with multi-checkpoint progressive investigation tracking.\n\n| Analysis | What it reveals |\n|----------|-----------------|\n| **Per-Challenge Breakdown** (Exp 2) | Score per challenge type |\n| **Submission vs. Checkpoint Gap** (Exp 8) | Per-challenge gap showing investigation process |\n\n**CyBench checkpoint progression** (e.g., Labyrinth Linguist):\n1. Contextualizing — understanding challenge environment\n2. Understanding templating — recognizing SSTI\u002Flanguage context\n3. Identifying attack vectors — finding injection points\n4. Exploring templates — testing template engines\n5. Locating the flag — finding flag file on disk\n6. Crafting exploit — building final payload\n\n**Key CyBench insights**:\n- All models achieve 1.0 on submission but 0.167–0.500 on checkpoints — solving without demonstrating full process\n- Currently 1 sample per model; distribution analyses produce summary tables instead of charts\n\n### CTI Realm (Threat Intelligence & Detection)\n\n**Domain context**: 25 tasks across 3 platforms (Linux, AKS, Cloud), 5 scored checkpoints totaling 10.0 points.\n\n| Analysis | What it reveals |\n|----------|-----------------|\n| **Per-Platform Breakdown** (Exp 2) | Score variation across Linux\u002FAKS\u002FCloud |\n| **Per-Checkpoint Score Heatmap** (Exp 8) | C0–C4 × Model heatmap showing checkpoint-level performance |\n| **CTI Tool Usage & Report Analysis** (Exp 14) | Tool call patterns across 4 categories (KQL\u002FData, CTI\u002FMITRE, Sigma, General) |\n| **KQL Query Quality Analysis** (Exp 15) | Query success\u002Ferror rates, correlation with F1 scores |\n| **Checkpoint Correlation Analysis** (Exp 16) | Independence of C0–C4 checkpoints (validates scoring design) |\n\n**CTI Realm 5-checkpoint scoring**:\n- **C0 CTI Alignment** (max 1.25): Did agent use CTI reports? Relevance to objective?\n- **C1 MITRE Techniques** (max 0.75): Jaccard similarity of found vs. expected ATT&CK techniques\n- **C2 Data Exploration** (max 1.0): Coverage of expected data sources queried\n- **C3 Query Iteration** (max 0.5): Binary — did agent run ≥2 unique successful KQL queries?\n- **C4 Detection Quality** (max 6.5): KQL F1 (weight 5.0) + Sigma rule quality (weight 1.5)\n\n**CTI Realm tool categories**:\n- **KQL\u002FData** (48%): execute_kql_query, list_kusto_tables, get_table_schema, sample_table_data\n- **CTI\u002FMITRE** (29%): list_cti_report_tags, get_cti_reports_by_tag, search_mitre_techniques\n- **Sigma\u002FDetection** (13%): search_sigma_rules, validate_output_json\n- **General** (10%): bash, python\n\n**Key CTI Realm insights**:\n- C4 (Detection Quality) dominates total score (65% of max) but max achieved is ~59%\n- Checkpoints are remarkably independent (all pairwise |ρ| \u003C 0.4) — validates scoring design\n- C0 (CTI reading) does NOT predict C4 (detection quality) — distinct skills\n- More KQL queries don't linearly improve detection: Opus achieves 58% C4 with only 4.6 queries\u002Fsample vs. GPT-5.4's 9.7\n\n---\n\n## Agent Architecture Analysis\n\n### Comparison Framework\n\nAgent architecture analysis uses the **same 12 experiments** as model comparison but slices by agent architecture instead of model identity. Primary comparison axis: **React vs. GH Copilot vs. Claude Code** for a fixed model (currently Claude Sonnet 4.6).\n\n### Agent Architecture Findings (Aggregate)\n\n| Metric | React | GH Copilot | Claude Code |\n|--------|-------|------------|-------------|\n| Aggregate Score | 0.801 ± 0.134 | 0.648 | 0.441 |\n| Cost\u002FSample | $0.251 (cheapest) | $0.350 | $0.420 |\n| Reward\u002F$ | 3.19 (highest) | 1.85 | 1.05 |\n| Tool Calls\u002FSample | ~18.8 | ~18.7 | ~18.9 |\n| Time\u002FSample | 3.3 min (fastest) | 5.0 min | 4.2 min |\n| Ranking Consistency | 1.0 ± 0.0 (perfect) | 2.33 ± 1.15 (volatile) | 2.67 ± 0.58 |\n\n**Key agent architecture insights**:\n- React dominates on score, cost, and consistency across all domains\n- All architectures use nearly identical tool-call budgets (~18.8 calls); speed difference is thinking\u002Fplanning overhead\n- Copilot matches React on submission scores (0.846 vs 0.852) but lags on checkpoints (0.380 vs 0.548)\n- Claude Code's primary failure mode is invalid final answers (lowest submission score: 0.348)\n\n---\n\n## Cross-Domain Aggregate Analysis\n\n### Normalization Method\n\nDomains have vastly different sample counts (Excytin: 599, CTI Realm: 25, CyBench: 1). To prevent sample-rich domains from dominating:\n\n1. Compute **per-domain mean** for each model\u002Fagent on each metric\n2. Average per-domain means with **equal weight per domain**\n\n$$\\text{Aggregate Score}(m) = \\frac{1}{|\\mathcal{D}|} \\sum_{d \\in \\mathcal{D}} \\overline{\\text{score}}(m, d)$$\n\n### Aggregate Experiment Types\n\n| # | Analysis | What it shows |\n|---|----------|---------------|\n| 1 | Normalized Overall Reward | Domain-weighted mean saber_overall per model |\n| 2 | Per-Domain Score Breakdown | Heatmap + grouped bar showing model × domain scores |\n| 3 | Normalized Cost Analysis | Per-sample cost averaged across domains; Pareto chart |\n| 4 | Normalized Cost Efficiency | Reward per dollar, domain-weighted |\n| 5 | Sub-Task Breakdown | Submission\u002FCheckpoints\u002FAggregate, domain-averaged |\n| 6 | Agent Trajectory Efficiency | Tool calls, steps, time (domain-averaged) |\n| 7 | Cross-Domain Ranking Consistency | Bump chart: do models rank the same across domains? |\n| 8 | Cross-Domain Radar Chart | Spider chart showing per-domain strength per model |\n| 9 | Reasoning Impact (models only) | Per-domain reasoning delta aggregated with equal weight |\n\n**Partial coverage**: If a model doesn't appear in all domains, its aggregate is computed over only contributing domains (annotated in charts).\n\n---\n\n## Key Insights Patterns\n\nThese are recurring patterns to look for and communicate when interpreting results:\n\n### 1. Cost-Efficiency Trade-offs\n- Higher-cost models often provide diminishing returns\n- Typical pattern: Haiku (cheapest) → Sonnet (mid) → Opus (high) → GPT (highest)\n- Haiku gets 12–18× more reward per dollar than GPT-5.4\n- Marginal score gains diminish as cost increases\n\n### 2. Effort vs. Outcome (Non-linear)\n- More tool calls do NOT guarantee better scores\n- High-effort (Q4) tasks have 15–20% more zero scores than low-effort (Q1) tasks\n- Tool-call budget is a binding constraint on hard tasks, not a surplus resource\n\n### 3. Investigation vs. Final Answer Gap\n- Submission scores >> Checkpoint scores across ALL domains\n- Typical gaps: Excytin 0.459–0.573, CyBench 0.500–0.833\n- Models reach correct conclusions without thorough investigation — risky in production\n- Models may be using prior knowledge or pattern-matching instead of genuine analysis\n\n### 4. Speed vs. Capability (Model-Family Patterns)\n- Claude models: 1.4–2.7 min\u002Fsample, lower token budgets\n- GPT models: 10+ min\u002Fsample (4–6× slower), higher token budgets\n- GPT's extended reasoning overhead adds latency without proportional score gains in some domains\n\n### 5. Task Difficulty Varies by Domain Structure\n- Easy domains (all models score high): Excytin (0.77–0.88)\n- Hard domains (massive variance): CTI Realm (0.26–0.63)\n- Domain complexity (tool count, checkpoint interdependencies) drives difficulty ceiling\n\n### 6. Model Agreement on Difficulty\n- Spearman correlations moderate (0.4–0.8) — difficulty is partly intrinsic, partly model-dependent\n- Same-family models agree more than cross-family (Sonnet↔Opus: ρ=0.64 vs. Claude↔GPT: ρ=0.47–0.53)\n- Implication: Ensemble strategies should pair different model families\n\n### 7. Reasoning Impact is Model & Domain-Dependent\n- Excytin: Minimal reasoning boost for most models\n- CTI Realm: Dramatic reasoning boost for GPT (+0.16–0.18), none for Claude\n- Multi-step collaborative workflows benefit more from explicit chain-of-thought\n\n### 8. Architecture Consistency\n- React agent: Rank 1 across all domains (perfect consistency)\n- GH Copilot: Rank 1–3 depending on domain (volatile)\n- Agent architecture choice has domain-dependent trade-offs\n\n---\n\n## Configuration Guide\n\n### Adding a New Model to Analysis\n\n1. **Check if sample evals already exist** — look in `eval_samples\u002F\u003Cdomain>\u002F` or try running the notebook (it auto-downloads from HuggingFace)\n2. **If not available**, run the evaluation:\n   ```bash\n   uv run inspect eval domains\u002F\u003Cdomain> --model \u003Capi\u002Fmodel-id> --display plain\n   ```\n3. Copy the `.eval` file to `eval_samples\u002F\u003Cdomain>\u002F` or `latest_experiments\u002F\u003Cdomain>\u002F` with a descriptive name\n4. Edit the notebook's Configuration cell:\n   ```python\n   EVAL_LOGS['Display Name'] = 'filename.eval'\n   COLORS['Display Name'] = '#HexColor'\n   PRICING['api\u002Fmodel-id'] = {'input': X, 'output': Y, 'cache_read': Z, 'cache_write': W}\n   ```\n5. (Optional) Add a no-reasoning baseline to `NO_THINKING_LOGS`\n\n### Adding a New Domain\n\n1. Create a domain-specific notebook by copying `eval_analysis.ipynb`\n2. Define `GROUP_FN` for your domain's sample ID grouping:\n   ```python\n   # Inspect sample IDs from your .eval files\n   # Write a function: sample_id → group_label\n   GROUP_FN = lambda id: id.split('_')[0]  # Example\n   ```\n3. Add domain-specific experiments after the generic 14\n4. Register the notebook in the inventory table above\n\n### Adding a New Agent Architecture\n\n1. **Check if sample evals already exist** — look in `eval_samples\u002F\u003Cdomain>\u002F` for `*_react.eval`, `*_copilot.eval`, `*_claude_code.eval`\n2. **If not available**, run evals for each agent:\n   ```bash\n   uv run inspect eval domains\u002F\u003Cdomain> --model \u003Cmodel> -T agent=react --display plain\n   uv run inspect eval domains\u002F\u003Cdomain> --model \u003Cmodel> -T agent=copilot --display plain\n   ```\n3. Add entries to the agent architecture notebook's `AGENT_EVAL_LOGS`\n4. Add a color in `AGENT_COLORS`\n\n---\n\n## Artifact Output Structure\n\nAnalysis notebooks save visualizations to `notebooks\u002Fartifacts\u002F`:\n\n```\nnotebooks\u002Fartifacts\u002F\n├── {domain}\u002F                     # Per-domain model comparison\n│   ├── overall_reward.png\n│   ├── per_{group}_scores.png\n│   ├── cost_analysis.png\n│   ├── token_usage_breakdown.png\n│   ├── cost_efficiency.png\n│   ├── subtask_breakdown.png\n│   ├── checkpoint_distributions.png\n│   ├── submission_checkpoint_gap.png\n│   ├── score_vs_tool_call_limit.png\n│   ├── effort_distributions.png\n│   ├── tool_usage.png\n│   ├── steps_vs_reward.png\n│   ├── effort_outcome_segmentation.png\n│   └── cross_model_correlation.png\n│\n├── {domain}_agent\u002F               # Per-domain agent architecture comparison\n│   ├── overall_reward_by_agent.png\n│   ├── cost_analysis_by_agent.png\n│   └── ... (same experiment set, sliced by agent)\n│\n├── aggregate_model\u002F              # Cross-domain model comparison (normalized)\n│   ├── aggregate_overall_reward.png\n│   ├── per_domain_breakdown.png\n│   ├── aggregate_cost_analysis.png\n│   ├── aggregate_cost_efficiency.png\n│   ├── cross_domain_radar.png\n│   ├── ranking_consistency.png\n│   └── reasoning_impact.png\n│\n└── aggregate_agent\u002F              # Cross-domain agent comparison (normalized)\n    ├── aggregate_overall_reward.png\n    ├── per_domain_breakdown.png\n    ├── cross_domain_radar.png\n    └── ranking_consistency.png\n```\n\n---\n\n## Quick Analysis Recipes\n\n### \"Which model should I use for domain X?\"\n\n1. Open `notebooks\u002F{domain}_analysis.ipynb`\n2. Look at Experiment 1 (overall reward) for raw scores\n3. Look at Experiment 5 (cost efficiency) for value\n4. Check Experiment 3 (Pareto frontier) for dominated vs. non-dominated models\n5. Consider Experiment 11 (time efficiency) if latency matters\n\n### \"How do agent architectures compare?\"\n\n1. Open `notebooks\u002Faggregate_agent_architecture_analysis.ipynb`\n2. Look at overall reward per agent (Exp 1) — React typically dominates\n3. Check per-domain breakdown (Exp 2) — some agents may excel in specific domains\n4. Review cost analysis (Exp 3) — architecture affects cost\n5. Check trajectory efficiency (Exp 6) — same tool calls, different speed\n\n### \"Is extended reasoning worth it?\"\n\n1. Open the domain-specific notebook with `NO_THINKING_LOGS` configured\n2. Look at reasoning delta (hatched bars) in Exp 1\n3. Check cost-efficiency impact in Exp 5\n4. CTI Realm benefits most from reasoning; Excytin shows minimal impact\n\n### \"Why is my model scoring low?\"\n\n1. Check Experiment 6 (sub-task breakdown): Submission vs. checkpoint gap reveals whether the model investigates or guesses\n2. Check domain-specific analysis: SQL errors (Excytin), KQL quality (CTI Realm), checkpoint progression (CyBench)\n3. Check Experiment 9a (effort budget): Is the model running out of tool calls?\n4. Check Experiment 12 (effort segmentation): Does the model fail on high-effort tasks?\n\n### \"Programmatic analysis without notebooks\"\n\nUse the `saber_analysis` library directly:\n\n```python\nimport sys\nsys.path.insert(0, \"notebooks\")\nfrom saber_analysis import load_eval_logs, extract_cost_rows, setup_plotting\n\n# Load data\noverall_df, samples_df, subtasks_df = load_eval_logs(\n    eval_logs={'Model A': 'model_a.eval'},\n    log_dir='latest_experiments\u002Fexcytin',\n    group_fn=lambda id: '_'.join(id.split('_')[:2]),\n)\n\n# Cost analysis\ncost_rows = extract_cost_rows(\n    log_dict={'Model A': 'model_a.eval'},\n    log_dir='latest_experiments\u002Fexcytin',\n    pricing={'anthropic\u002Fclaude-sonnet-4-6': {'input': 3, 'output': 15, 'cache_read': 0.3, 'cache_write': 3.75}},\n    n_samples=599,\n)\n```\n","# Eval Analysis Skill\n\n## Overview\n\nSABER evaluation analysis is a structured data science framework for comparing AI agent performance across cybersecurity benchmark domains. The analysis pipeline transforms `.eval` log files into actionable insights through 12+ standardized experiments plus domain-specific analyses.\n\n## Getting Started: Sample Eval Data\n\n**No evals to analyze yet?** The repo ships with pre-run sample `.eval` files and can auto-download more from HuggingFace.\n\n### Option 1: Use bundled sample evals (fastest)\n\nThe `eval_samples\u002F` directory contains pre-run evaluation results for all 3 domains across 5 models and 3 agent architectures:\n\n```\neval_samples\u002F\n├── excytin\u002F          # 20 eval files (5 models × default + reasoning baselines + 3 agents)\n├── cybench\u002F          # 17 eval files\n└── cti_realm\u002F        # 11 eval files\n```\n\n**Models included**: Claude Haiku 4.5, Claude Sonnet 4.6, Claude Opus 4.6, GPT-5.4, GPT-5.4-mini  \n**Agent architectures included**: React, GH Copilot, Claude Code (for Sonnet 4.6)  \n**Baselines**: No-reasoning\u002Fno-thinking variants for extended thinking comparison\n\nThe notebooks are pre-configured to load from `eval_samples\u002F`. Just run any notebook — no configuration needed.\n\n### Option 2: Auto-download from HuggingFace\n\nIf `.eval` files are missing locally, `ensure_eval_files()` automatically downloads them from the **[AcesEvals HuggingFace dataset](https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Fanandmudgerikar\u002FAcesEvals)**:\n\n```python\nfrom saber_analysis import ensure_eval_files\n\nensure_eval_files(\n    eval_logs=EVAL_LOGS,       # {display_name: filename.eval}\n    log_dir=LOG_DIR,           # local directory\n    domain='excytin',          # HuggingFace subfolder\n    fallback_dirs=[            # check these before downloading\n        str(REPO_ROOT \u002F 'eval_samples' \u002F 'excytin'),\n        str(REPO_ROOT \u002F 'latest_experiments' \u002F 'excytin'),\n    ],\n)\n```\n\n**Resolution order**: local `log_dir` → `fallback_dirs` → HuggingFace download  \n**HuggingFace repo**: `anandmudgerikar\u002FAcesEvals` (dataset type)  \n**HuggingFace path**: `latest_experiment_samples\u002F{domain}\u002F{filename}.eval`\n\n### Option 3: Run your own evaluations\n\nIf you need results for a model or agent architecture not in the sample evals:\n\n```bash\n# Run for a new model\nuv run inspect eval domains\u002F\u003Cdomain> --model \u003Capi\u002Fmodel-id> --display plain\n\n# Run for a specific agent architecture\nuv run inspect eval domains\u002F\u003Cdomain> --model \u003Capi\u002Fmodel-id> -T agent=react --display plain\nuv run inspect eval domains\u002F\u003Cdomain> --model \u003Capi\u002Fmodel-id> -T agent=copilot --display plain\n\n# Run with extended reasoning disabled (for baseline comparison)\n# Set INSPECT_EVAL_MODEL_ARGS=reasoning_effort=none or configure in .env\n```\n\nThen copy the `.eval` file to `eval_samples\u002F\u003Cdomain>\u002F` or `latest_experiments\u002F\u003Cdomain>\u002F` and add it to the notebook's `EVAL_LOGS` configuration.\n\n### What's available vs. what you need to run\n\n| Domain | Models (sample evals) | Agent Architectures (sample evals) | What to run yourself |\n|--------|-----------------------|-------------------------------------|----------------------|\n| Excytin | Haiku 4.5, Sonnet 4.6, Opus 4.6, GPT-5.4, GPT-5.4-mini | React, Copilot, Claude Code (Sonnet) | New models, new agents on non-Sonnet models |\n| CyBench | Haiku 4.5, Sonnet 4.6, Opus 4.6, GPT-5.4, GPT-5.4-mini | React, Copilot, Claude Code (Sonnet) | New models, new agents, more challenges |\n| CTI Realm | Haiku 4.5, Sonnet 4.6, Opus 4.6, GPT-5.4, GPT-5.4-mini | Copilot, Claude Code (Sonnet) | React for non-Sonnet models, new models |\n\n### Notebook Inventory\n\n| Notebook | Scope | When to use |\n|----------|-------|-------------|\n| `notebooks\u002Feval_analysis.ipynb` | Domain-agnostic template | Analyzing any single domain's model results; starting point for new domains |\n| `notebooks\u002Fexcytin_analysis.ipynb` | Excytin-specific (self-contained) | Full analysis of Excytin incident response domain |\n| `notebooks\u002Fcybench_analysis.ipynb` | CyBench-specific (self-contained) | Full analysis of CyBench CTF challenges |\n| `notebooks\u002Fcti_realm_analysis.ipynb` | CTI Realm-specific (self-contained) | Full analysis of CTI Realm threat intel domain |\n| `notebooks\u002Fagent_architecture_analysis.ipynb` | Agent comparison template | Comparing agent architectures (React, Copilot, Claude Code) on any domain |\n| `notebooks\u002Fexcytin_agent_architecture_analysis.ipynb` | Excytin agent comparison | Agent architecture analysis specific to Excytin |\n| `notebooks\u002Fcybench_agent_architecture_analysis.ipynb` | CyBench agent comparison | Agent architecture analysis specific to CyBench |\n| `notebooks\u002Fcti_realm_agent_architecture_analysis.ipynb` | CTI Realm agent comparison | Agent architecture analysis specific to CTI Realm |\n| `notebooks\u002Faggregate_model_analysis.ipynb` | Cross-domain model comparison | Aggregate model analysis across all domains (normalized) |\n| `notebooks\u002Faggregate_agent_architecture_analysis.ipynb` | Cross-domain agent comparison | Aggregate agent architecture analysis across all domains |\n| `notebooks\u002Fmodel_safety_filters_analysis.ipynb` | Safety filter investigation | Analyzing model safety refusals and guardrail triggers |\n| `notebooks\u002Fexcytin_basic_analysis.ipynb` | Quick Excytin analysis | Lightweight Excytin analysis for rapid iteration |\n\n### Shared Analysis Library\n\nThe `notebooks\u002Fsaber_analysis\u002F` package provides reusable utilities:\n\n| Module | Functions | Purpose |\n|--------|-----------|---------|\n| `data_loader.py` | `ensure_eval_files()`, `load_eval_logs()`, `load_baseline_logs()`, `load_trajectory_data()` | Standardized .eval parsing; HuggingFace fallback downloads; typed inspect_ai log API |\n| `cost.py` | `calc_cost()`, `extract_cost_rows()` | Token→dollar conversion using per-model pricing |\n| `plots.py` | `setup_plotting()`, `make_legend_patches()`, `classify_score_type()` | Consistent matplotlib styling across all notebooks |\n\n---\n\n## Data Pipeline\n\n```\n.eval ZIP archives\n    │\n    ├── header.json          → run metadata, overall saber_overall scores\n    ├── samples\u002F*.json       → per-sample messages, scores, timing, tool calls\n    └── summaries.json       → aggregated score summaries\n    │\n    ▼\ninspect_ai.log API\n    ├── read_eval_log(path, header_only=True)       → EvalLog with results.scores\n    ├── read_eval_log_sample_summaries(path)         → per-sample scores & timing\n    └── read_eval_log_sample(path, id=...)           → full messages & tool calls\n    │\n    ▼\nsaber_analysis library\n    ├── ensure_eval_files()  → locates locally or downloads from HuggingFace\n    ├── load_eval_logs()     → parses into DataFrames (overall_df, samples_df, subtasks_df)\n    ├── load_trajectory_data() → extracts tool calls, steps, timing per sample\n    └── extract_cost_rows()  → builds cost DataFrame from token usage + pricing\n    │\n    ▼\nNotebook Configuration\n    ├── EVAL_LOGS: dict[str, str]      → {display_name: filename.eval}\n    ├── COLORS: dict[str, str]         → {display_name: hex_color}\n    ├── PRICING: dict[str, dict]       → {api_model_id: {input, output, cache_read, cache_write}}\n    ├── GROUP_FN: Callable             → domain-specific sample ID → group label\n    └── NO_THINKING_LOGS (optional)    → baseline without extended reasoning\n    │\n    ▼\nAnalysis Cells (Experiments 1–14+)\n    │\n    ▼\nArtifacts saved to notebooks\u002Fartifacts\u002F{domain}\u002F*.png\n```\n\n### Key DataFrames\n\n| DataFrame | Columns | Source |\n|-----------|---------|--------|\n| `overall_df` | model, mean, stderr | `load_eval_logs()` — header-level saber_overall |\n| `samples_df` | model, sample_id, group, score | `load_eval_logs()` — per-sample saber_overall |\n| `subtasks_df` | model, sample_id, group, score_type, score | `load_eval_logs()` — submission\u002Fcheckpoint\u002Faggregate breakdown |\n| `traj_df` | model, sample_id, n_tool_calls, n_steps, total_time, tool_counts | `load_trajectory_data()` — agent behavior data |\n| `cost_df` | model, score, input_tokens, output_tokens, cache_read, cache_write, reasoning_tokens, total_cost, cost_per_sample | `extract_cost_rows()` |\n\n---\n\n## Domain-Agnostic Analysis Types (Experiments 1–14)\n\nThese 14 core analyses apply to **any** SABER domain. They appear in `eval_analysis.ipynb` and every domain-specific notebook.\n\n### Experiment 1: Overall Reward Distribution\n\n- **Metric**: Mean `saber_overall` score per model\n- **Visualization**: Bar chart with error bars (stderr) + optional hatched overlay for reasoning delta\n- **Interpretation**:\n  - Higher bars = better performance\n  - Overlapping error bars = statistically insignificant differences\n  - Reasoning delta (hatched) shows impact of extended thinking (positive = reasoning helps)\n- **Code pattern**:\n  ```python\n  overall_df.plot.bar(x='model', y='mean', yerr='stderr')\n  ```\n\n### Experiment 2: Per-Group Breakdown (Domain-Specific Grouping)\n\n- **Metric**: Scores segmented by `GROUP_FN` dimension\n- **Visualization**: Clustered bar chart (models × groups)\n- **Group functions by domain**:\n  - **Excytin**: `'_'.join(id.split('_')[:2])` → 8 security incidents\n  - **CyBench**: `'_'.join(id.split('_')[:-2])` → challenge types\n  - **CTI Realm**: `id.split('_')[0]` → 3 platforms (Linux, AKS, Cloud)\n- **Interpretation**:\n  - Large within-group variance = model-dependent difficulty\n  - Shifting model rankings across groups = no single dominant model\n  - Uniform clusters = inherently hard\u002Feasy groups\n- **Requires**: Setting `GROUP_FN` in configuration; set `GROUP_FN = None` to skip\n\n### Experiment 3: Cost Analysis\n\n- **Metrics**: Total cost ($), cost per sample, token breakdown (input\u002Foutput\u002Fcache\u002Freasoning)\n- **Visualizations**:\n  1. Total cost bar chart\n  2. Cost per sample bar chart\n  3. Pareto frontier (Score vs. Cost — upper-left corner = best value)\n- **Interpretation**:\n  - Pareto-dominant models: no other model is both cheaper AND better\n  - Arrows on Pareto chart show reasoning cost delta\n  - Diminishing returns visible from frontier plateau\n- **Code pattern**:\n  ```python\n  cost_df = pd.DataFrame(extract_cost_rows(EVAL_LOGS, LOG_DIR, PRICING, N_SAMPLES))\n  ```\n\n### Experiment 4: Token Usage Breakdown\n\n- **Metrics**: Input, output, cache read\u002Fwrite, reasoning tokens per model\n- **Visualization**: Horizontal stacked bar chart\n- **Interpretation**:\n  - **Input tokens**: Prompts, tool results, conversation history\n  - **Output tokens**: Generated text, tool calls, answers\n  - **Cache read\u002Fwrite**: Prompt caching (Claude > GPT generally)\n  - **Reasoning tokens**: Extended thinking overhead; highly variable by model family\n  - Large hatched sections = reasoning adds substantial overhead\n\n### Experiment 5: Cost Efficiency (Reward per Dollar)\n\n- **Formula**: `Cost Efficiency = Mean SABER Score \u002F Total Cost ($)`\n- **Visualization**: Bar chart (taller = more efficient)\n- **Interpretation**:\n  - Highest-cost model is not always least efficient\n  - Budget-constrained deployments should prioritize high-efficiency models\n  - Reasoning may increase OR decrease efficiency (depends on score gain vs. cost increase)\n\n### Experiment 6: Sub-Task Score Breakdown\n\n- **Metrics**: Submission score, checkpoint scores, aggregate score per model\n- **Visualization**: Grouped bars (submission\u002Fcheckpoints\u002Faggregate) with reasoning delta overlay\n- **Interpretation**:\n  - **Submission >> Checkpoints**: Model reaches correct conclusion without thorough investigation (may be guessing; risky in production)\n  - **Checkpoints >> Submission**: Model investigates well but fails at synthesis\n  - **Aligned**: Investigation matches answer quality (ideal)\n\n### Experiment 7: Checkpoint Score Distributions\n\n- **Visualization**: Violin plots per model\n- **Interpretation**:\n  - **Bimodal** (peaks at 0 and 1): Binary pass\u002Ffail checkpoint; few partial successes\n  - **Continuous**: Fine-grained scoring; various intermediate scores\n  - **Wide violins**: Inconsistent across samples; **narrow**: consistent\n  - **Long upper tails**: Some samples max out the checkpoint score\n\n### Experiment 8: Domain-Specific Gap Analysis\n\n- **Purpose**: Submission vs. checkpoint gap per domain grouping dimension\n- **Visualization**: Heatmap (models × groups) showing gap = submission - checkpoint\n- **Interpretation**:\n  - Red (positive gap): Model guesses correctly without thorough investigation\n  - Green (negative gap): Model investigates well but fails at synthesis\n  - Yellow (aligned): Investigation matches answer quality\n\n### Experiment 9: Agent Trajectory Analysis\n\nData loading step that extracts per-sample behavioral data:\n- `n_tool_calls`: Actual tool invocations per sample\n- `n_steps`: Assistant message turns per sample\n- `tool_counts`: Per-tool breakdown (bash, python, domain tools)\n- `total_time`: Wall-clock seconds per sample\n\n### Experiment 9a: Score vs. Tool-Call Limit (Effort Budget)\n\n- **Calculation**: For each hypothetical limit N, mean score where samples needing >N calls score 0\n- **Visualization**: Curves per model converging to actual mean at real tool-call cap\n- **Interpretation**:\n  - **Steep early rise**: Efficient model (solves many tasks in first 5–15 calls)\n  - **Plateau**: Budget increase yields diminishing returns\n  - **Key finding**: All models typically reach 95% of final score by ~20–25 calls (Excytin) or ~33–35 (CTI Realm)\n\n### Experiment 9b: Effort Distribution per Model\n\n- **Visualization**: Violin plots of tool calls and steps per sample per model\n- **Interpretation**:\n  - **Narrow, low**: Efficient (quick solutions)\n  - **Wide, high**: Variable effort or struggling\n  - **Long upper tail**: Some samples hit tool-call limit (binding constraint)\n\n### Experiment 10: Tool Usage Analysis\n\n- **Visualization**: Dual panel — left: mean tool calls per tool (absolute); right: tool call ratio (fraction)\n- **Interpretation**:\n  - Domain determines tool preferences (Excytin → 95%+ bash; CTI Realm → 48% KQL tools)\n  - Cross-model consistency = task design funnels agents toward certain tools\n  - Model-specific tool preferences reveal different investigation strategies\n\n### Experiment 11: Time Efficiency & Reward per Tool Call\n\n- **Formulas**:\n  - `Reward\u002FCall = Mean SABER Score \u002F Mean Tool Calls`\n  - `Reward\u002FMinute = Mean Score \u002F Wall-Clock Time (min)`\n- **Visualizations**: Bar charts, scatter plots (effort vs. score with trend line), time distribution violin plots\n- **Interpretation**:\n  - High reward\u002Fcall = doesn't waste invocations\n  - High reward\u002Fminute = fast AND accurate\n  - Time outliers reveal latency issues (e.g., GPT 10+ min vs. Claude 2–3 min)\n\n### Experiment 12: Score Outcome Segmentation by Effort\n\n- **Methodology**: Segment by tool-call quartile (Q1–Q4); show perfect\u002Fpartial\u002Fzero score rates\n- **Visualization**: Stacked bar chart per model per quartile\n- **Interpretation**:\n  - Perfect-score rate drops Q1→Q4: High effort = harder task (effort doesn't help)\n  - Zero-score rate doubles Q1→Q4: Tool-call limit is binding constraint on hard tasks\n  - Partial scores rise in Q4: Upper-effort tasks succeed only partially within budget\n\n### Experiment 13: Cross-Model Difficulty Agreement\n\n- **Calculation**: Spearman rank correlation of per-sample scores for each model pair\n- **Visualization**: Heatmap with ρ values and significance markers\n- **Interpretation**:\n  - **High ρ (>0.7)**: Models agree — difficulty is intrinsic to task\n  - **Low ρ (\u003C0.3)**: Models struggle differently — complementary strengths, good for ensembles\n  - **Within-family > cross-family**: Same-vendor models agree more than different vendors\n\n---\n\n## Domain-Specific Analyses\n\n### Excytin (Incident Response Forensics)\n\n**Domain context**: 8 security incidents, 599 tasks, agents investigate via SQL queries through bash tool.\n\n| Analysis | What it reveals |\n|----------|-----------------|\n| **Per-Incident Breakdown** (Exp 2) | Score variation across 8 incident types |\n| **Submission vs. Checkpoint Gap Heatmap** (Exp 8) | Per-incident × per-model gap showing investigation thoroughness |\n| **SQL Query Analysis** (Exp 14) | Classifies all SQL queries into success\u002Flarge-result\u002Fempty\u002Ferror; reveals query quality differences across models |\n\n**SQL Query Classification**:\n- **Success (≤100 rows)**: Manageable, agent can parse\n- **Large result (>100 rows)**: Unrefined query; should add LIMIT\u002FWHERE\n- **Empty result**: Valid syntax but no matching data (dominant failure mode)\n- **Error**: Syntax error, missing table, permission denied\n\n**Key Excytin insights**:\n- All models show large positive submission-checkpoint gaps (0.459–0.573) — bypassing investigation\n- Bash tool dominates (95–100% of tool calls); Haiku uniquely uses python (~5%)\n- Query quality separates models: Opus 79% success rate vs. GPT-5.4-mini 61%\n- Empty results are more problematic than syntax errors (agents waste effort on unproductive queries)\n\n### CyBench (CTF Challenges)\n\n**Domain context**: CTF exploitation challenges with multi-checkpoint progressive investigation tracking.\n\n| Analysis | What it reveals |\n|----------|-----------------|\n| **Per-Challenge Breakdown** (Exp 2) | Score per challenge type |\n| **Submission vs. Checkpoint Gap** (Exp 8) | Per-challenge gap showing investigation process |\n\n**CyBench checkpoint progression** (e.g., Labyrinth Linguist):\n1. Contextualizing — understanding challenge environment\n2. Understanding templating — recognizing SSTI\u002Flanguage context\n3. Identifying attack vectors — finding injection points\n4. Exploring templates — testing template engines\n5. Locating the flag — finding flag file on disk\n6. Crafting exploit — building final payload\n\n**Key CyBench insights**:\n- All models achieve 1.0 on submission but 0.167–0.500 on checkpoints — solving without demonstrating full process\n- Currently 1 sample per model; distribution analyses produce summary tables instead of charts\n\n### CTI Realm (Threat Intelligence & Detection)\n\n**Domain context**: 25 tasks across 3 platforms (Linux, AKS, Cloud), 5 scored checkpoints totaling 10.0 points.\n\n| Analysis | What it reveals |\n|----------|-----------------|\n| **Per-Platform Breakdown** (Exp 2) | Score variation across Linux\u002FAKS\u002FCloud |\n| **Per-Checkpoint Score Heatmap** (Exp 8) | C0–C4 × Model heatmap showing checkpoint-level performance |\n| **CTI Tool Usage & Report Analysis** (Exp 14) | Tool call patterns across 4 categories (KQL\u002FData, CTI\u002FMITRE, Sigma, General) |\n| **KQL Query Quality Analysis** (Exp 15) | Query success\u002Ferror rates, correlation with F1 scores |\n| **Checkpoint Correlation Analysis** (Exp 16) | Independence of C0–C4 checkpoints (validates scoring design) |\n\n**CTI Realm 5-checkpoint scoring**:\n- **C0 CTI Alignment** (max 1.25): Did agent use CTI reports? Relevance to objective?\n- **C1 MITRE Techniques** (max 0.75): Jaccard similarity of found vs. expected ATT&CK techniques\n- **C2 Data Exploration** (max 1.0): Coverage of expected data sources queried\n- **C3 Query Iteration** (max 0.5): Binary — did agent run ≥2 unique successful KQL queries?\n- **C4 Detection Quality** (max 6.5): KQL F1 (weight 5.0) + Sigma rule quality (weight 1.5)\n\n**CTI Realm tool categories**:\n- **KQL\u002FData** (48%): execute_kql_query, list_kusto_tables, get_table_schema, sample_table_data\n- **CTI\u002FMITRE** (29%): list_cti_report_tags, get_cti_reports_by_tag, search_mitre_techniques\n- **Sigma\u002FDetection** (13%): search_sigma_rules, validate_output_json\n- **General** (10%): bash, python\n\n**Key CTI Realm insights**:\n- C4 (Detection Quality) dominates total score (65% of max) but max achieved is ~59%\n- Checkpoints are remarkably independent (all pairwise |ρ| \u003C 0.4) — validates scoring design\n- C0 (CTI reading) does NOT predict C4 (detection quality) — distinct skills\n- More KQL queries don't linearly improve detection: Opus achieves 58% C4 with only 4.6 queries\u002Fsample vs. GPT-5.4's 9.7\n\n---\n\n## Agent Architecture Analysis\n\n### Comparison Framework\n\nAgent architecture analysis uses the **same 12 experiments** as model comparison but slices by agent architecture instead of model identity. Primary comparison axis: **React vs. GH Copilot vs. Claude Code** for a fixed model (currently Claude Sonnet 4.6).\n\n### Agent Architecture Findings (Aggregate)\n\n| Metric | React | GH Copilot | Claude Code |\n|--------|-------|------------|-------------|\n| Aggregate Score | 0.801 ± 0.134 | 0.648 | 0.441 |\n| Cost\u002FSample | $0.251 (cheapest) | $0.350 | $0.420 |\n| Reward\u002F$ | 3.19 (highest) | 1.85 | 1.05 |\n| Tool Calls\u002FSample | ~18.8 | ~18.7 | ~18.9 |\n| Time\u002FSample | 3.3 min (fastest) | 5.0 min | 4.2 min |\n| Ranking Consistency | 1.0 ± 0.0 (perfect) | 2.33 ± 1.15 (volatile) | 2.67 ± 0.58 |\n\n**Key agent architecture insights**:\n- React dominates on score, cost, and consistency across all domains\n- All architectures use nearly identical tool-call budgets (~18.8 calls); speed difference is thinking\u002Fplanning overhead\n- Copilot matches React on submission scores (0.846 vs 0.852) but lags on checkpoints (0.380 vs 0.548)\n- Claude Code's primary failure mode is invalid final answers (lowest submission score: 0.348)\n\n---\n\n## Cross-Domain Aggregate Analysis\n\n### Normalization Method\n\nDomains have vastly different sample counts (Excytin: 599, CTI Realm: 25, CyBench: 1). To prevent sample-rich domains from dominating:\n\n1. Compute **per-domain mean** for each model\u002Fagent on each metric\n2. Average per-domain means with **equal weight per domain**\n\n$$\\text{Aggregate Score}(m) = \\frac{1}{|\\mathcal{D}|} \\sum_{d \\in \\mathcal{D}} \\overline{\\text{score}}(m, d)$$\n\n### Aggregate Experiment Types\n\n| # | Analysis | What it shows |\n|---|----------|---------------|\n| 1 | Normalized Overall Reward | Domain-weighted mean saber_overall per model |\n| 2 | Per-Domain Score Breakdown | Heatmap + grouped bar showing model × domain scores |\n| 3 | Normalized Cost Analysis | Per-sample cost averaged across domains; Pareto chart |\n| 4 | Normalized Cost Efficiency | Reward per dollar, domain-weighted |\n| 5 | Sub-Task Breakdown | Submission\u002FCheckpoints\u002FAggregate, domain-averaged |\n| 6 | Agent Trajectory Efficiency | Tool calls, steps, time (domain-averaged) |\n| 7 | Cross-Domain Ranking Consistency | Bump chart: do models rank the same across domains? |\n| 8 | Cross-Domain Radar Chart | Spider chart showing per-domain strength per model |\n| 9 | Reasoning Impact (models only) | Per-domain reasoning delta aggregated with equal weight |\n\n**Partial coverage**: If a model doesn't appear in all domains, its aggregate is computed over only contributing domains (annotated in charts).\n\n---\n\n## Key Insights Patterns\n\nThese are recurring patterns to look for and communicate when interpreting results:\n\n### 1. Cost-Efficiency Trade-offs\n- Higher-cost models often provide diminishing returns\n- Typical pattern: Haiku (cheapest) → Sonnet (mid) → Opus (high) → GPT (highest)\n- Haiku gets 12–18× more reward per dollar than GPT-5.4\n- Marginal score gains diminish as cost increases\n\n### 2. Effort vs. Outcome (Non-linear)\n- More tool calls do NOT guarantee better scores\n- High-effort (Q4) tasks have 15–20% more zero scores than low-effort (Q1) tasks\n- Tool-call budget is a binding constraint on hard tasks, not a surplus resource\n\n### 3. Investigation vs. Final Answer Gap\n- Submission scores >> Checkpoint scores across ALL domains\n- Typical gaps: Excytin 0.459–0.573, CyBench 0.500–0.833\n- Models reach correct conclusions without thorough investigation — risky in production\n- Models may be using prior knowledge or pattern-matching instead of genuine analysis\n\n### 4. Speed vs. Capability (Model-Family Patterns)\n- Claude models: 1.4–2.7 min\u002Fsample, lower token budgets\n- GPT models: 10+ min\u002Fsample (4–6× slower), higher token budgets\n- GPT's extended reasoning overhead adds latency without proportional score gains in some domains\n\n### 5. Task Difficulty Varies by Domain Structure\n- Easy domains (all models score high): Excytin (0.77–0.88)\n- Hard domains (massive variance): CTI Realm (0.26–0.63)\n- Domain complexity (tool count, checkpoint interdependencies) drives difficulty ceiling\n\n### 6. Model Agreement on Difficulty\n- Spearman correlations moderate (0.4–0.8) — difficulty is partly intrinsic, partly model-dependent\n- Same-family models agree more than cross-family (Sonnet↔Opus: ρ=0.64 vs. Claude↔GPT: ρ=0.47–0.53)\n- Implication: Ensemble strategies should pair different model families\n\n### 7. Reasoning Impact is Model & Domain-Dependent\n- Excytin: Minimal reasoning boost for most models\n- CTI Realm: Dramatic reasoning boost for GPT (+0.16–0.18), none for Claude\n- Multi-step collaborative workflows benefit more from explicit chain-of-thought\n\n### 8. Architecture Consistency\n- React agent: Rank 1 across all domains (perfect consistency)\n- GH Copilot: Rank 1–3 depending on domain (volatile)\n- Agent architecture choice has domain-dependent trade-offs\n\n---\n\n## Configuration Guide\n\n### Adding a New Model to Analysis\n\n1. **Check if sample evals already exist** — look in `eval_samples\u002F\u003Cdomain>\u002F` or try running the notebook (it auto-downloads from HuggingFace)\n2. **If not available**, run the evaluation:\n   ```bash\n   uv run inspect eval domains\u002F\u003Cdomain> --model \u003Capi\u002Fmodel-id> --display plain\n   ```\n3. Copy the `.eval` file to `eval_samples\u002F\u003Cdomain>\u002F` or `latest_experiments\u002F\u003Cdomain>\u002F` with a descriptive name\n4. Edit the notebook's Configuration cell:\n   ```python\n   EVAL_LOGS['Display Name'] = 'filename.eval'\n   COLORS['Display Name'] = '#HexColor'\n   PRICING['api\u002Fmodel-id'] = {'input': X, 'output': Y, 'cache_read': Z, 'cache_write': W}\n   ```\n5. (Optional) Add a no-reasoning baseline to `NO_THINKING_LOGS`\n\n### Adding a New Domain\n\n1. Create a domain-specific notebook by copying `eval_analysis.ipynb`\n2. Define `GROUP_FN` for your domain's sample ID grouping:\n   ```python\n   # Inspect sample IDs from your .eval files\n   # Write a function: sample_id → group_label\n   GROUP_FN = lambda id: id.split('_')[0]  # Example\n   ```\n3. Add domain-specific experiments after the generic 14\n4. Register the notebook in the inventory table above\n\n### Adding a New Agent Architecture\n\n1. **Check if sample evals already exist** — look in `eval_samples\u002F\u003Cdomain>\u002F` for `*_react.eval`, `*_copilot.eval`, `*_claude_code.eval`\n2. **If not available**, run evals for each agent:\n   ```bash\n   uv run inspect eval domains\u002F\u003Cdomain> --model \u003Cmodel> -T agent=react --display plain\n   uv run inspect eval domains\u002F\u003Cdomain> --model \u003Cmodel> -T agent=copilot --display plain\n   ```\n3. Add entries to the agent architecture notebook's `AGENT_EVAL_LOGS`\n4. Add a color in `AGENT_COLORS`\n\n---\n\n## Artifact Output Structure\n\nAnalysis notebooks save visualizations to `notebooks\u002Fartifacts\u002F`:\n\n```\nnotebooks\u002Fartifacts\u002F\n├── {domain}\u002F                     # Per-domain model comparison\n│   ├── overall_reward.png\n│   ├── per_{group}_scores.png\n│   ├── cost_analysis.png\n│   ├── token_usage_breakdown.png\n│   ├── cost_efficiency.png\n│   ├── subtask_breakdown.png\n│   ├── checkpoint_distributions.png\n│   ├── submission_checkpoint_gap.png\n│   ├── score_vs_tool_call_limit.png\n│   ├── effort_distributions.png\n│   ├── tool_usage.png\n│   ├── steps_vs_reward.png\n│   ├── effort_outcome_segmentation.png\n│   └── cross_model_correlation.png\n│\n├── {domain}_agent\u002F               # Per-domain agent architecture comparison\n│   ├── overall_reward_by_agent.png\n│   ├── cost_analysis_by_agent.png\n│   └── ... (same experiment set, sliced by agent)\n│\n├── aggregate_model\u002F              # Cross-domain model comparison (normalized)\n│   ├── aggregate_overall_reward.png\n│   ├── per_domain_breakdown.png\n│   ├── aggregate_cost_analysis.png\n│   ├── aggregate_cost_efficiency.png\n│   ├── cross_domain_radar.png\n│   ├── ranking_consistency.png\n│   └── reasoning_impact.png\n│\n└── aggregate_agent\u002F              # Cross-domain agent comparison (normalized)\n    ├── aggregate_overall_reward.png\n    ├── per_domain_breakdown.png\n    ├── cross_domain_radar.png\n    └── ranking_consistency.png\n```\n\n---\n\n## Quick Analysis Recipes\n\n### \"Which model should I use for domain X?\"\n\n1. Open `notebooks\u002F{domain}_analysis.ipynb`\n2. Look at Experiment 1 (overall reward) for raw scores\n3. Look at Experiment 5 (cost efficiency) for value\n4. Check Experiment 3 (Pareto frontier) for dominated vs. non-dominated models\n5. Consider Experiment 11 (time efficiency) if latency matters\n\n### \"How do agent architectures compare?\"\n\n1. Open `notebooks\u002Faggregate_agent_architecture_analysis.ipynb`\n2. Look at overall reward per agent (Exp 1) — React typically dominates\n3. Check per-domain breakdown (Exp 2) — some agents may excel in specific domains\n4. Review cost analysis (Exp 3) — architecture affects cost\n5. Check trajectory efficiency (Exp 6) — same tool calls, different speed\n\n### \"Is extended reasoning worth it?\"\n\n1. Open the domain-specific notebook with `NO_THINKING_LOGS` configured\n2. Look at reasoning delta (hatched bars) in Exp 1\n3. Check cost-efficiency impact in Exp 5\n4. CTI Realm benefits most from reasoning; Excytin shows minimal impact\n\n### \"Why is my model scoring low?\"\n\n1. Check Experiment 6 (sub-task breakdown): Submission vs. checkpoint gap reveals whether the model investigates or guesses\n2. Check domain-specific analysis: SQL errors (Excytin), KQL quality (CTI Realm), checkpoint progression (CyBench)\n3. Check Experiment 9a (effort budget): Is the model running out of tool calls?\n4. Check Experiment 12 (effort segmentation): Does the model fail on high-effort tasks?\n\n### \"Programmatic analysis without notebooks\"\n\nUse the `saber_analysis` library directly:\n\n```python\nimport sys\nsys.path.insert(0, \"notebooks\")\nfrom saber_analysis import load_eval_logs, extract_cost_rows, setup_plotting\n\n# Load data\noverall_df, samples_df, subtasks_df = load_eval_logs(\n    eval_logs={'Model A': 'model_a.eval'},\n    log_dir='latest_experiments\u002Fexcytin',\n    group_fn=lambda id: '_'.join(id.split('_')[:2]),\n)\n\n# Cost analysis\ncost_rows = extract_cost_rows(\n    log_dict={'Model A': 'model_a.eval'},\n    log_dir='latest_experiments\u002Fexcytin',\n    pricing={'anthropic\u002Fclaude-sonnet-4-6': {'input': 3, 'output': 15, 'cache_read': 0.3, 'cache_write': 3.75}},\n    n_samples=599,\n)\n```\n",{"name":5,"description":7},"main","2c76fbdea415d6fa6a87bbad846686fefd6ee9bb",".github\u002Fskills\u002Feval-analysis",3,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FACESEvals","https:\u002F\u002Fgithub.com\u002Fmicrosoft","https:\u002F\u002Fopensource.microsoft.com","Open source projects and samples from Microsoft","2026-11-06 05:01:42",{"source":8,"skillName":5,"commands":41,"prompt":51,"zipAvailable":52},[42,45,48],{"tool":43,"command":44},"npx","npx skills add microsoft\u002FACESEvals --skill eval-analysis",{"tool":46,"command":47},"bunx","bunx skills add microsoft\u002FACESEvals --skill eval-analysis",{"tool":49,"command":50},"pnpm","pnpm dlx skills add microsoft\u002FACESEvals --skill eval-analysis","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FACESEvals\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Feval-analysis\u002FSKILL.md",true,[],[55,63,71,85],{"skillNo":56,"name":57,"nameZh":6,"description":58,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":59,"categories":60,"keywords":61,"publishedAt":26},"01M69AVY1HBGF9RSCDNM7DH2XK","inspect-eval-log-analysis","Guide for parsing and analyzing inspect_ai .eval log files. Use this when asked to interpret eval results, extract tool calls, find scores, or investigate agent behavior from .eval logs.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FACESEvals\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Finspect-eval-log-analysis\u002FSKILL.md",[],[62],"inspect eval log analysis",{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":26},"01M69AVW490PYQ0055P22Z9DYQ","inspect-eval-execution","Guide for running SABER inspect_ai evaluations locally. Use this when asked to run, re-run, or configure an inspect eval for any SABER domain.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FACESEvals\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Finspect-eval-execution\u002FSKILL.md",[],[70],"inspect eval execution",{"skillNo":72,"name":73,"nameZh":6,"description":74,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":82,"publishedAt":26},"01M69AVSJA51SMERVDQAYWYF4W","inspect-eval-debugging","Guide for debugging inspect_ai evaluation failures, score issues, and model behavior. Use this when eval results are unexpected, scores are wrong, scoring fails, or model output appears corrupted.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FACESEvals\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Finspect-eval-debugging\u002FSKILL.md",[77],{"categoryNo":78,"categoryName":79,"taxonomyType":80,"taxonomyTypeDesc":81},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[83,84],"inspect eval debugging","ai evaluation",{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":94,"publishedAt":26},"01M69AVQEWW91QZX96MWCZTA5S","eval-monitoring","Guide for monitoring running SABER evaluations, checking progress, and managing eval batches. Use this when asked to monitor evals, check progress, produce a status report, or manage concurrent eval runs. Also covers Docker health and resource management.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FACESEvals\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Feval-monitoring\u002FSKILL.md",[91],{"categoryNo":92,"categoryName":93,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000011","运维与部署",[95,96,97],"eval monitoring","docker","status"]