[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M69AVQEWW91QZX96MWCZTA5S":3,"skill-same:01M69AVQEWW91QZX96MWCZTA5S":69},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":27,"linkOnly":28,"markdownContent":29,"markdownBody":30,"frontMatter":31,"refName":32,"commitSha":33,"skillDirectory":34,"forksCount":35,"repositoryHtmlUrl":36,"repositoryHomepageUrl":6,"ownerHtmlUrl":37,"ownerBlogUrl":38,"ownerDescription":39,"updatedAt":40,"install":41,"resources":54},"01M69AVQEWW91QZX96MWCZTA5S","eval-monitoring",null,"Guide for monitoring running SABER evaluations, checking progress, and managing eval batches. Use this when asked to monitor evals, check progress, produce a status report, or manage concurrent eval runs. Also covers Docker health and resource management.","microsoft\u002FACESEvals","microsoft","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F6154722?v=4","01M698VYDXZ6JW5WH3CRV650R8","Microsoft","01M69AVF6DCZVH3H6ZKYRQD2VV",22,"MIT","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FACESEvals\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Feval-monitoring\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000011","运维与部署","SCENARIO","使用场景",[24,25,26],"eval monitoring","docker","status","2026-11-10 10:12:30",false,"---\nname: eval-monitoring\ndescription: >\n  Guide for monitoring running SABER evaluations, checking progress, and managing eval batches.\n  Use this when asked to monitor evals, check progress, produce a status report, or manage\n  concurrent eval runs. Also covers Docker health and resource management.\n---\n\n# Eval Monitoring Skill\n\n## Quick Status Check\n\nRun the monitoring script for an instant report:\n\n```bash\n.\u002Fscripts\u002Fmonitor_evals.sh\n```\n\nFor continuous monitoring (every 15 minutes):\n```bash\n.\u002Fscripts\u002Fmonitor_evals.sh --watch 15\n```\n\n## Manual Monitoring Commands\n\n### 1. Check Running Processes\n\n```bash\n# Count running evals\nps aux | grep \"inspect eval\" | grep -v grep | wc -l\n\n# See model and agent for each\nps aux | grep \"inspect eval\" | grep -v grep | \\\n  grep -oP '(?\u003C=--model )\"?[^\\s\"]+' | tr -d '\"'\n```\n\n### 2. Check Eval File Progress\n\n> **Staleness note:** `.eval` ZIP files are flushed in batches by inspect_ai, so sample\n> counts read from them lag **2–3 minutes** behind the live results visible in the\n> `inspect view` UI. This is expected — treat `.eval` reads as \"near-real-time\" rather\n> than exact.\n\n```python\n# Quick sample count for an in-progress eval\npython3 -c \"\nimport zipfile, json\nwith zipfile.ZipFile('path\u002Fto\u002Ffile.eval') as z:\n    samples = [n for n in z.namelist() if n.startswith('samples\u002F') and n.endswith('.json')]\n    header = json.loads(z.read('header.json'))\n    print(f'Status: {header.get(\\\"status\\\")}, Samples: {len(samples)}')\n\"\n```\n\n### 3. Check Docker Health\n\n```bash\n# Container count (>120 = resource pressure)\ndocker ps -q | wc -l\n\n# Check for stuck containers\ndocker ps --format '{{.Names}} {{.Status}}' | grep -i \"unhealthy\\|restarting\"\n```\n\n### 4. Check Log Files for Errors\n\n```bash\n# Find recently active logs\nfind latest_experiments\u002F -name \"*_log.txt\" -mmin -15 -exec tail -3 {} +\n\n# Check a specific eval log\ntail -20 latest_experiments\u002Fexcytin\u002Fsonnet_4.6_claude_code_log.txt\n```\n\n## Batch Running\n\n### Launch All Missing Evals (3 concurrent)\n\n```bash\n.\u002Fscripts\u002Frun_eval_batch.sh --domain excytin --max-concurrent 3\n```\n\n### Dry Run (see what would be launched)\n\n```bash\n.\u002Fscripts\u002Frun_eval_batch.sh --domain excytin --dry-run\n```\n\n### Override Concurrency\n\n```bash\n# Conservative: 2 at a time\n.\u002Fscripts\u002Frun_eval_batch.sh --max-concurrent 2\n\n# Only if Docker resources allow: 4 at a time\n.\u002Fscripts\u002Frun_eval_batch.sh --max-concurrent 4\n```\n\n## Concurrency Guidelines\n\n| Concurrent Evals | Docker Resources | When to Use |\n|-------------------|------------------|-------------|\n| 1 | Minimal (~20 containers) | Debugging, resource-constrained |\n| 2-3 | Moderate (~60-100) | **Recommended default** |\n| 4-5 | Heavy (~100-160) | Only with fast models + good hardware |\n| 6+ | Excessive (160+) | **AVOID** — Docker compose timeouts |\n\n**Key rule: Never run more than 3 concurrent evals with `--max-samples 64`.** Each eval\ncan spawn up to 16 Docker sandbox containers simultaneously, plus the permanent services.\n\n## Eval File Naming Convention\n\nEval files follow this pattern: `{model_slug}_{agent}.eval`\n\n| Model Slug | Model ID |\n|------------|----------|\n| `sonnet_4.6` | `anthropic\u002Fclaude-sonnet-4-6` |\n| `opus_4.6` | `anthropic\u002Fclaude-opus-4-6` |\n| `haiku_4.5` | `anthropic\u002Fclaude-haiku-4-5` |\n| `gpt_5.4` | `openai\u002Fazure\u002Fgpt-5.4` |\n| `gpt_5.4_mini` | `openai\u002Fazure\u002Fgpt-5.4-mini` |\n\n| Agent | Description |\n|-------|-------------|\n| `react` | Default React agent (step-by-step reasoning) |\n| `copilot` | GH Copilot agent |\n| `claude_code` | Claude Code agent |\n\n## Interpreting Eval File Status\n\n| File Size | Status | Action |\n|-----------|--------|--------|\n| \u003C50KB | Error\u002Fempty | Delete and retry |\n| 50KB-1MB | Partial (few samples) | Delete and retry |\n| >1MB | Likely has data | Check `header.json` status field |\n| Status: `success` | Complete | Skip |\n| Status: `error` | Failed mid-run | Check error, may need retry |\n| Status: `started` | In progress or abandoned | Check if process alive |\n\n## Common Failure Modes\n\n### Docker Compose Timeout\n**Symptom:** `TimeoutError: Docker compose command ... timed out after 45 seconds`\n**Cause:** Too many concurrent evals or Docker resource exhaustion\n**Fix:** Kill all evals, `docker container prune -f`, relaunch with lower concurrency\n\n### DeploymentNotFound (Azure)\n**Symptom:** `DeploymentNotFound` in pre-flight check\n**Cause:** Wrong deployment name or API version\n**Fix:** Check deployment name matches Azure portal. Use `AZUREAI_OPENAI_API_VERSION` override.\n\n### Safety Refusals (0% scores)\n**Symptom:** All scores 0.0, answers contain \"I cannot\", \"social engineering\"\n**Cause:** Model safety guardrails triggered by benchmark prompts\n**Fix:** Document in `model_safety_filters_analysis.ipynb`, not a retry-able error\n\n## Docker Reset Procedure\n\nWhen Docker is in a bad state:\n\n```bash\n# 1. Gracefully stop all eval processes (allow cleanup of .eval files and Docker resources)\npkill -f \"inspect eval\"\nsleep 10  # Give processes time to flush and clean up\n\n# 2. Force-kill only if any are still running\npkill -9 -f \"inspect eval\" 2>\u002Fdev\u002Fnull\n\n# 3. Stop all containers (xargs -r skips if no containers are running)\ndocker ps -q | xargs -r docker stop\n\n# 4. Remove stopped containers\ndocker container prune -f\n\n# 5. Verify clean state\ndocker ps -q | wc -l  # Should be 0\n\n# 6. Relaunch evals\n.\u002Fscripts\u002Frun_eval_batch.sh --domain excytin --max-concurrent 3\n```\n","# Eval Monitoring Skill\n\n## Quick Status Check\n\nRun the monitoring script for an instant report:\n\n```bash\n.\u002Fscripts\u002Fmonitor_evals.sh\n```\n\nFor continuous monitoring (every 15 minutes):\n```bash\n.\u002Fscripts\u002Fmonitor_evals.sh --watch 15\n```\n\n## Manual Monitoring Commands\n\n### 1. Check Running Processes\n\n```bash\n# Count running evals\nps aux | grep \"inspect eval\" | grep -v grep | wc -l\n\n# See model and agent for each\nps aux | grep \"inspect eval\" | grep -v grep | \\\n  grep -oP '(?\u003C=--model )\"?[^\\s\"]+' | tr -d '\"'\n```\n\n### 2. Check Eval File Progress\n\n> **Staleness note:** `.eval` ZIP files are flushed in batches by inspect_ai, so sample\n> counts read from them lag **2–3 minutes** behind the live results visible in the\n> `inspect view` UI. This is expected — treat `.eval` reads as \"near-real-time\" rather\n> than exact.\n\n```python\n# Quick sample count for an in-progress eval\npython3 -c \"\nimport zipfile, json\nwith zipfile.ZipFile('path\u002Fto\u002Ffile.eval') as z:\n    samples = [n for n in z.namelist() if n.startswith('samples\u002F') and n.endswith('.json')]\n    header = json.loads(z.read('header.json'))\n    print(f'Status: {header.get(\\\"status\\\")}, Samples: {len(samples)}')\n\"\n```\n\n### 3. Check Docker Health\n\n```bash\n# Container count (>120 = resource pressure)\ndocker ps -q | wc -l\n\n# Check for stuck containers\ndocker ps --format '{{.Names}} {{.Status}}' | grep -i \"unhealthy\\|restarting\"\n```\n\n### 4. Check Log Files for Errors\n\n```bash\n# Find recently active logs\nfind latest_experiments\u002F -name \"*_log.txt\" -mmin -15 -exec tail -3 {} +\n\n# Check a specific eval log\ntail -20 latest_experiments\u002Fexcytin\u002Fsonnet_4.6_claude_code_log.txt\n```\n\n## Batch Running\n\n### Launch All Missing Evals (3 concurrent)\n\n```bash\n.\u002Fscripts\u002Frun_eval_batch.sh --domain excytin --max-concurrent 3\n```\n\n### Dry Run (see what would be launched)\n\n```bash\n.\u002Fscripts\u002Frun_eval_batch.sh --domain excytin --dry-run\n```\n\n### Override Concurrency\n\n```bash\n# Conservative: 2 at a time\n.\u002Fscripts\u002Frun_eval_batch.sh --max-concurrent 2\n\n# Only if Docker resources allow: 4 at a time\n.\u002Fscripts\u002Frun_eval_batch.sh --max-concurrent 4\n```\n\n## Concurrency Guidelines\n\n| Concurrent Evals | Docker Resources | When to Use |\n|-------------------|------------------|-------------|\n| 1 | Minimal (~20 containers) | Debugging, resource-constrained |\n| 2-3 | Moderate (~60-100) | **Recommended default** |\n| 4-5 | Heavy (~100-160) | Only with fast models + good hardware |\n| 6+ | Excessive (160+) | **AVOID** — Docker compose timeouts |\n\n**Key rule: Never run more than 3 concurrent evals with `--max-samples 64`.** Each eval\ncan spawn up to 16 Docker sandbox containers simultaneously, plus the permanent services.\n\n## Eval File Naming Convention\n\nEval files follow this pattern: `{model_slug}_{agent}.eval`\n\n| Model Slug | Model ID |\n|------------|----------|\n| `sonnet_4.6` | `anthropic\u002Fclaude-sonnet-4-6` |\n| `opus_4.6` | `anthropic\u002Fclaude-opus-4-6` |\n| `haiku_4.5` | `anthropic\u002Fclaude-haiku-4-5` |\n| `gpt_5.4` | `openai\u002Fazure\u002Fgpt-5.4` |\n| `gpt_5.4_mini` | `openai\u002Fazure\u002Fgpt-5.4-mini` |\n\n| Agent | Description |\n|-------|-------------|\n| `react` | Default React agent (step-by-step reasoning) |\n| `copilot` | GH Copilot agent |\n| `claude_code` | Claude Code agent |\n\n## Interpreting Eval File Status\n\n| File Size | Status | Action |\n|-----------|--------|--------|\n| \u003C50KB | Error\u002Fempty | Delete and retry |\n| 50KB-1MB | Partial (few samples) | Delete and retry |\n| >1MB | Likely has data | Check `header.json` status field |\n| Status: `success` | Complete | Skip |\n| Status: `error` | Failed mid-run | Check error, may need retry |\n| Status: `started` | In progress or abandoned | Check if process alive |\n\n## Common Failure Modes\n\n### Docker Compose Timeout\n**Symptom:** `TimeoutError: Docker compose command ... timed out after 45 seconds`\n**Cause:** Too many concurrent evals or Docker resource exhaustion\n**Fix:** Kill all evals, `docker container prune -f`, relaunch with lower concurrency\n\n### DeploymentNotFound (Azure)\n**Symptom:** `DeploymentNotFound` in pre-flight check\n**Cause:** Wrong deployment name or API version\n**Fix:** Check deployment name matches Azure portal. Use `AZUREAI_OPENAI_API_VERSION` override.\n\n### Safety Refusals (0% scores)\n**Symptom:** All scores 0.0, answers contain \"I cannot\", \"social engineering\"\n**Cause:** Model safety guardrails triggered by benchmark prompts\n**Fix:** Document in `model_safety_filters_analysis.ipynb`, not a retry-able error\n\n## Docker Reset Procedure\n\nWhen Docker is in a bad state:\n\n```bash\n# 1. Gracefully stop all eval processes (allow cleanup of .eval files and Docker resources)\npkill -f \"inspect eval\"\nsleep 10  # Give processes time to flush and clean up\n\n# 2. Force-kill only if any are still running\npkill -9 -f \"inspect eval\" 2>\u002Fdev\u002Fnull\n\n# 3. Stop all containers (xargs -r skips if no containers are running)\ndocker ps -q | xargs -r docker stop\n\n# 4. Remove stopped containers\ndocker container prune -f\n\n# 5. Verify clean state\ndocker ps -q | wc -l  # Should be 0\n\n# 6. Relaunch evals\n.\u002Fscripts\u002Frun_eval_batch.sh --domain excytin --max-concurrent 3\n```\n",{"name":5,"description":7},"main","2c76fbdea415d6fa6a87bbad846686fefd6ee9bb",".github\u002Fskills\u002Feval-monitoring",3,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FACESEvals","https:\u002F\u002Fgithub.com\u002Fmicrosoft","https:\u002F\u002Fopensource.microsoft.com","Open source projects and samples from Microsoft","2026-11-06 05:01:42",{"source":8,"skillName":5,"commands":42,"prompt":52,"zipAvailable":53},[43,46,49],{"tool":44,"command":45},"npx","npx skills add microsoft\u002FACESEvals --skill eval-monitoring",{"tool":47,"command":48},"bunx","bunx skills add microsoft\u002FACESEvals --skill eval-monitoring",{"tool":50,"command":51},"pnpm","pnpm dlx skills add microsoft\u002FACESEvals --skill eval-monitoring","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FACESEvals\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Feval-monitoring\u002FSKILL.md",true,[55,61,65],{"path":56,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":59,"sourceHtmlUrl":60},"references\u002Fdashboard-archetypes.md","REFERENCE","参考文档",6210,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fawesome-rayfin\u002Fblob\u002Fmain\u002Ftemplates\u002Funiversal-app\u002F.agents\u002Fskills\u002Fapp-design\u002Freferences\u002Fdashboard-archetypes.md",{"path":62,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":63,"sourceHtmlUrl":64},"references\u002Fui-style-recipes.md",12792,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fawesome-rayfin\u002Fblob\u002Fmain\u002Ftemplates\u002Funiversal-app\u002F.agents\u002Fskills\u002Fapp-design\u002Freferences\u002Fui-style-recipes.md",{"path":66,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":67,"sourceHtmlUrl":68},"references\u002Fvisual-style-recipes.md",12751,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fawesome-rayfin\u002Fblob\u002Fmain\u002Ftemplates\u002Funiversal-app\u002F.agents\u002Fskills\u002Fapp-design\u002Freferences\u002Fvisual-style-recipes.md",[70,78,86,100],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":27},"01M69AVY1HBGF9RSCDNM7DH2XK","inspect-eval-log-analysis","Guide for parsing and analyzing inspect_ai .eval log files. Use this when asked to interpret eval results, extract tool calls, find scores, or investigate agent behavior from .eval logs.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FACESEvals\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Finspect-eval-log-analysis\u002FSKILL.md",[],[77],"inspect eval log analysis",{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":27},"01M69AVW490PYQ0055P22Z9DYQ","inspect-eval-execution","Guide for running SABER inspect_ai evaluations locally. Use this when asked to run, re-run, or configure an inspect eval for any SABER domain.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FACESEvals\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Finspect-eval-execution\u002FSKILL.md",[],[85],"inspect eval execution",{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":97,"publishedAt":27},"01M69AVSJA51SMERVDQAYWYF4W","inspect-eval-debugging","Guide for debugging inspect_ai evaluation failures, score issues, and model behavior. Use this when eval results are unexpected, scores are wrong, scoring fails, or model output appears corrupted.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FACESEvals\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Finspect-eval-debugging\u002FSKILL.md",[92],{"categoryNo":93,"categoryName":94,"taxonomyType":95,"taxonomyTypeDesc":96},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[98,99],"inspect eval debugging","ai evaluation",{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":109,"publishedAt":27},"01M69AVMVG4RZ9VYBT2GKWT0M8","eval-analysis","Comprehensive guide for analyzing SABER evaluation results — model comparison, agent architecture comparison, domain-specific analysis, and cross-domain aggregate analysis. Use this when asked to analyze eval results, compare models, generate visualizations, interpret scores, investigate cost-efficiency trade-offs, or extend analysis to new domains.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FACESEvals\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Feval-analysis\u002FSKILL.md",[106],{"categoryNo":107,"categoryName":108,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发",[110,111],"eval analysis","architecture"]