[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M69BKT1MSCAN07KS096R4DG9":3,"skill-same:01M69BKT1MSCAN07KS096R4DG9":52},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":23,"linkOnly":24,"markdownContent":25,"markdownBody":26,"frontMatter":27,"refName":28,"commitSha":29,"skillDirectory":30,"forksCount":31,"repositoryHtmlUrl":32,"repositoryHomepageUrl":33,"ownerHtmlUrl":34,"ownerBlogUrl":35,"ownerDescription":36,"updatedAt":37,"install":38,"resources":51},"01M69BKT1MSCAN07KS096R4DG9","benchmark-qed-autoe",null,"Evaluate RAG system outputs using benchmark-qed scoring methods. Use when: running pairwise comparisons, reference-based scoring, assertion-based evaluation (flat or hierarchical), retrieval metrics, or statistical significance tests on RAG outputs. Also use when the user wants to score, compare, or evaluate RAG methods, measure retrieval quality, or run significance tests on benchmark results — even if they don't say \"autoe\" explicitly.","microsoft\u002Fbenchmark-qed","microsoft","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F6154722?v=4","01M698VYDXZ6JW5WH3CRV650R8","Microsoft","01M69BKPH852RZE1NZFW5DFVSA",104,"MIT","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\u002Fblob\u002Fmain\u002F.apm\u002Fskills\u002Fbenchmark-qed-autoe\u002FSKILL.md",[],[19,20,21,22],"benchmark","benchmark qed autoe","rag","run","2026-11-10 10:12:30",false,"---\nname: benchmark-qed-autoe\ndescription: >\n  Evaluate RAG system outputs using benchmark-qed scoring methods. Use when:\n  running pairwise comparisons, reference-based scoring, assertion-based\n  evaluation (flat or hierarchical), retrieval metrics, or statistical\n  significance tests on RAG outputs. Also use when the user wants to score,\n  compare, or evaluate RAG methods, measure retrieval quality, or run\n  significance tests on benchmark results — even if they don't say \"autoe\"\n  explicitly.\n---\n\n# Benchmark-QED Evaluation (autoe)\n\nEvaluate and compare RAG system outputs using LLM-judged scoring, assertion-based evaluation, and retrieval metrics — all with built-in statistical significance testing.\n\n## Prerequisites\n\n- Generated questions\u002Fassertions from the autoq pipeline (or your own)\n- RAG method answer files (JSON, one per method per question set)\n- A valid `settings.yaml` for the evaluation type\n- A configured workspace with valid `settings.yaml` (use the `benchmark-qed-setup` skill to initialize and configure)\n- LLM API key configured\n\nRun all commands with:\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed \u003Ccommand>\n```\n\n## Evaluation Methods Overview\n\n| Method | Command | Best for |\n|--------|---------|----------|\n| Pairwise comparison | `autoe pairwise-scores` | Comparing two RAG methods head-to-head |\n| Differential pairwise | `autoe differential-pairwise-scores` | Reducing length\u002Fformat bias with extract-and-judge |\n| Reference scoring | `autoe reference-scores` | Scoring against gold-standard answers |\n| Assertion scoring | `autoe assertion-scores` | Evaluating with ground-truth assertions (single or multi-RAG) |\n| Hierarchical assertions | `autoe hierarchical-assertion-scores` | Global + local assertion hierarchies |\n| Retrieval metrics | `autoe retrieval-scores` | Precision, recall, fidelity of retrieval |\n| Significance tests | `autoe assertion-significance` | Post-hoc significance on existing scores |\n\n## Commands\n\n### 1. Pairwise Scores\n\nCompare RAG methods using LLM-judged pairwise comparisons.\n\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe pairwise-scores \u003Cconfig.yaml> \u003Coutput_dir> [OPTIONS]\n```\n\n**Options:**\n| Option | Default | Description |\n|--------|---------|-------------|\n| `--alpha` | `0.05` | P-value threshold for significance |\n| `--exclude-criteria` | `[]` | Criteria to exclude (repeatable) |\n| `--print-model-usage` | `false` | Print LLM token usage |\n| `--include-score-id-in-prompt` | `true` | Add a unique ID to provider requests; does not bypass `cache_config` |\n| `--account-url` | `null` | Azure Blob Storage account URL (managed-identity auth). Use when the config path is a `blob:\u002F\u002F` URI. |\n| `--connection-string` | `null` | Azure Blob Storage connection string. Use when the config path is a `blob:\u002F\u002F` URI. |\n\n**Config requires**: `base` (reference method), `others` (methods to compare), `question_sets`, `criteria`, `trials` (must be even), `llm_config`, `prompt_config`. Generated configurations also include `cache_config`, with persistent SQLite enabled by default.\n\nDefault criteria: `comprehensiveness`, `diversity`, `empowerment`, `relevance`\n\n**Output**: `{question_set}_{base}--{other}.csv`, `win_rates.csv`, `winrates_sig_tests.csv`\n\n**Pairwise caching**:\n\n- Keep the generated SQLite `cache_config` to resume interrupted runs and reuse\n  identical question\u002Fcriterion\u002Ftrial judgments.\n- Use `cache_config: {type: none, storage: null}` to request fresh judgments.\n- Each trial is cached separately; caching does not collapse repeated trials or\n  counterbalanced answer ordering.\n- Existing comparison CSVs are checked before the judgment cache. For a fully\n  fresh comparison, remove\u002Fuse a new output CSV and disable or relocate the\n  cache.\n- Inspect the default cache with\n  `benchmark-qed cache inspect .benchmark_qed_cache\u002Fpairwise\u002Fpairwise.sqlite3`.\n- `--include-score-id-in-prompt` may avoid provider-side prompt caching but does\n  not bypass BenchmarkQED's configured pairwise cache.\n\n### 1a. Differential Pairwise Scores\n\nUse `autoe differential-pairwise-scores` for the two-stage extract-and-judge\nmethod. Generated configs use a separate SQLite cache at\n`.benchmark_qed_cache\u002Fdifferential_pairwise\u002Fdifferential_pairwise.sqlite3`.\n\n- Extraction and verdict stages are cached independently.\n- Judge failures resume from completed extraction.\n- Criteria or judge-prompt changes reuse extraction and recompute verdicts.\n- Trials remain separate and counterbalanced.\n- Use `cache_config: {type: none, storage: null}` for fully fresh calls, and\n  also remove\u002Fuse a new output CSV.\n\n### 2. Reference Scores\n\nScore generated answers against reference (gold-standard) answers.\n\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe reference-scores \u003Cconfig.yaml> \u003Coutput_dir> [OPTIONS]\n```\n\n**Config requires**: `reference`, `generated` (list), `criteria`, `score_min`\u002F`score_max`, `trials`, `llm_config`. Generated configs include a persistent SQLite `cache_config`.\n\nDefault criteria: `correctness`, `completeness`. Default score range: 1–10.\n\n**Output**: `reference_scores-{name}.csv`, `model_usage.json`\n\n**Reference caching**:\n\n- Each question\u002Fcriterion\u002Ftrial is cached separately under the `reference`\n  namespace.\n- Cache identity includes both answers, score range, prompts, model\u002Fprovider,\n  call arguments, and score-ID mode.\n- Rerun after a failure to reuse completed judgments.\n- Use `cache_config: {type: none, storage: null}` for fresh judgments.\n- Inspect the default cache with\n  `benchmark-qed cache inspect .benchmark_qed_cache\u002Freference\u002Freference.sqlite3`.\n\n### 3. Assertion Scores\n\nEvaluate RAG methods using assertion-based scoring. Auto-detects single-RAG vs multi-RAG config.\n\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe assertion-scores \u003Cconfig.yaml> \u003Coutput_dir> [OPTIONS]\n```\n\n**Options:**\n| Option | Default | Description |\n|--------|---------|-------------|\n| `--alpha` | `0.05` | Significance threshold (multi-RAG) |\n| `--print-model-usage` | `false` | Print LLM token usage |\n| `--account-url` | `null` | Azure Blob Storage account URL (managed-identity auth). Use when the config path is a `blob:\u002F\u002F` URI. |\n| `--connection-string` | `null` | Azure Blob Storage connection string. Use when the config path is a `blob:\u002F\u002F` URI. |\n\n**Auto-detection**: If the YAML contains a `rag_methods` key, it runs in multi-RAG mode with automated significance testing. Otherwise, single-RAG mode.\n\n**Single-RAG output**: `assertion_scores.csv`, `assertion_summary_by_question.csv`, `eval_summary.json`\n\n**Multi-RAG output**: Per-method scores + significance tests in structured `output_dir\u002F`\n\n**Single-RAG assertion caching**:\n\n- Generated configs use SQLite at\n  `.benchmark_qed_cache\u002Fassertion\u002Fassertion.sqlite3`.\n- Each question\u002Fassertion\u002Ftrial is cached independently; the identity also\n  includes the answer, prompts, model\u002Fprovider, call arguments, and score-ID\n  mode.\n- Rerun after a failure to reuse completed judgments.\n- Inspect with `benchmark-qed cache inspect\n  .benchmark_qed_cache\u002Fassertion\u002Fassertion.sqlite3`.\n- Use `cache_config: {type: none, storage: null}` for fresh judgments.\n- This cache is not used by multi-RAG or hierarchical assertion scoring.\n  Chunk-assertion scoring has a separate assertion\u002Fchunk cache.\n\n### 4. Hierarchical Assertion Scores\n\nScore hierarchical assertions (global assertions with supporting local assertions).\n\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe hierarchical-assertion-scores \u003Cconfig.yaml> \u003Coutput_dir> [OPTIONS]\n```\n\n**Modes**: `staged` (default — evaluate local first, then global) or `joint` (evaluate together)\n\n**Extra field**: `detect_discovery: true` enables detection of novel findings not covered by assertions.\n\nAlso auto-detects single vs multi-RAG config (same as assertion-scores).\n\n### 5. Assertion Significance\n\nRun statistical significance tests on existing assertion scores (no LLM calls).\n\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe assertion-significance \u003Cconfig.yaml>\n```\n\n**Config requires**: `output_dir`, `rag_methods`, `question_sets`, `alpha`, `correction_method`\n\n**Correction methods**: `holm` (default, recommended), `bonferroni`, `fdr_bh`\n\n### 6. Hierarchical Assertion Significance\n\nSignificance tests on hierarchical assertion scores.\n\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe hierarchical-assertion-significance \u003Cconfig.yaml>\n```\n\n**Config requires**: `scores_dir`, `rag_methods`, `scores_filename_template`, `alpha`, `correction_method`, `output_dir`\n\n### 7. Generate Retrieval Reference\n\nGenerate cluster relevance reference data for retrieval evaluation (one-off prep step).\n\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe generate-retrieval-reference \u003Cconfig.yaml>\n```\n\n**Config requires**: `llm_config`, `embedding_config`, question source (`questions_path` or `question_sets`), `text_units_path`\n\n**Key settings**: `num_clusters`, `assessor_type` (`rationale` or `bing`), `semantic_neighbors`, `centroid_neighbors`\n\n### 8. Retrieval Scores\n\nEvaluate retrieval precision, recall, and fidelity for RAG methods.\n\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe retrieval-scores \u003Cconfig.yaml>\n```\n\n**Config requires**: `rag_methods`, `question_sets`, `reference_dir`, `text_units_path`, `output_dir`\n\n**Fidelity metrics**: `js` (Jensen-Shannon divergence) or `tvd` (total variation distance)\n\n## Workflow\n\n### Quick Evaluation (Assertion-Based)\n\n- [ ] Step 1: Verify questions and answers exist — list the workspace and confirm a `settings.yaml` (or `config.yaml`), question JSON files (typically under `output\u002F`), and your RAG method answer JSONs are present.\n- [ ] Step 2: Initialize eval config — use the `benchmark-qed-setup` skill to create and configure an assertion evaluation workspace.\n- [ ] Step 3: Configure settings.yaml with answer paths and assertion paths\n- [ ] Step 4: Run evaluation — `uvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe assertion-scores .\u002Feval_workspace\u002Fsettings.yaml .\u002Feval_output`\n- [ ] Step 5: Summarize results — read the CSVs in `\u003Coutput_dir>` (e.g. `assertion_scores.csv`, `assertion_summary_by_question.csv`) and `eval_summary.json` directly.\n\n### Multi-RAG Comparison\n\nFor comparing multiple RAG methods, use multi-RAG config format (include `rag_methods` key in YAML). This gives you automated pairwise significance testing.\n\n## Gotchas\n\n- **Config auto-detection**: `assertion-scores` and `hierarchical-assertion-scores` detect single vs multi-RAG based on the `rag_methods` key in YAML. Ensure your config matches your intent.\n- **Trials must be even**: For pairwise scores, `trials` must be even (for counterbalancing). Use 4 as default.\n- **Pairwise cache identity**: Questions, answers and labels, criterion, trial,\n  prompts, model\u002Fprovider, call arguments, and score-ID mode determine reuse.\n  Changing any of them creates a cache miss.\n- **Blob storage does not relocate SQLite caches**: Pairwise\n  `cache_config.storage.base_dir` remains a local path even when input\u002Foutput\n  storage uses Azure Blob Storage.\n- **Cache storage is independent**: JSON caches use the storage configured\n  under `cache_config.storage`; they do not inherit input\u002Foutput blob settings.\n  `type: file` resolves a relative `base_dir` from the command's working\n  directory. Use `cache_config.storage.type: blob` with an explicit container\n  and credentials to store JSON entries in Azure Blob Storage.\n- **Stale outputs**: Several commands skip existing output files. Use a fresh output directory or delete specific files to force re-evaluation.\n- **Output is in files**: All scores are written to CSV\u002FJSON files. Parse output files, not CLI stdout.\n- **Long-running**: Evaluation with many questions and trials can take hours. Use background execution.\n- **No `config init` for hierarchical\u002Fretrieval**: The `benchmark-qed-setup` skill only supports `autoe_assertion`, `autoe_pairwise`, and `autoe_reference`. For hierarchical, multi-RAG, and retrieval configs, create YAML manually.\n- **Advanced config types**: Use the `benchmark-qed-setup` skill for configuration guidance on advanced config types.\n\n## Azure Blob Storage\n\nAll `autoe` commands support reading config files from Azure Blob Storage using `blob:\u002F\u002F` URIs:\n\n```bash\n# Config file in blob storage\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe assertion-scores \\\n  blob:\u002F\u002Fmy-container\u002Feval\u002Fsettings.yaml .\u002Feval_output \\\n  --account-url https:\u002F\u002Fmyaccount.blob.core.windows.net\n```\n\nIn addition, `settings.yaml` supports `input_storage` and `output_storage` blocks so the evaluation pipeline can read answers\u002Fassertions from and write results to Azure Blob Storage:\n\n```yaml\n# Read answers and assertions from blob storage\ninput_storage:\n  type: blob\n  container_name: my-datasets\n  connection_string: ${AZURE_STORAGE_CONNECTION_STRING}\n  # Or use managed identity:\n  # account_url: https:\u002F\u002Fmyaccount.blob.core.windows.net\n\n# Write evaluation output to blob storage\noutput_storage:\n  type: blob\n  container_name: my-output\n  connection_string: ${AZURE_STORAGE_CONNECTION_STRING}\n```\n\nWhen using storage blocks, `answer_base_path` and `assertions_path` in the config are resolved relative to the storage container (not the local filesystem).\n\nSee [references\u002Fconfig-reference.md](..\u002Fbenchmark-qed-setup\u002Freferences\u002Fconfig-reference.md) for full `StorageConfig` fields.\n","# Benchmark-QED Evaluation (autoe)\n\nEvaluate and compare RAG system outputs using LLM-judged scoring, assertion-based evaluation, and retrieval metrics — all with built-in statistical significance testing.\n\n## Prerequisites\n\n- Generated questions\u002Fassertions from the autoq pipeline (or your own)\n- RAG method answer files (JSON, one per method per question set)\n- A valid `settings.yaml` for the evaluation type\n- A configured workspace with valid `settings.yaml` (use the `benchmark-qed-setup` skill to initialize and configure)\n- LLM API key configured\n\nRun all commands with:\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed \u003Ccommand>\n```\n\n## Evaluation Methods Overview\n\n| Method | Command | Best for |\n|--------|---------|----------|\n| Pairwise comparison | `autoe pairwise-scores` | Comparing two RAG methods head-to-head |\n| Differential pairwise | `autoe differential-pairwise-scores` | Reducing length\u002Fformat bias with extract-and-judge |\n| Reference scoring | `autoe reference-scores` | Scoring against gold-standard answers |\n| Assertion scoring | `autoe assertion-scores` | Evaluating with ground-truth assertions (single or multi-RAG) |\n| Hierarchical assertions | `autoe hierarchical-assertion-scores` | Global + local assertion hierarchies |\n| Retrieval metrics | `autoe retrieval-scores` | Precision, recall, fidelity of retrieval |\n| Significance tests | `autoe assertion-significance` | Post-hoc significance on existing scores |\n\n## Commands\n\n### 1. Pairwise Scores\n\nCompare RAG methods using LLM-judged pairwise comparisons.\n\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe pairwise-scores \u003Cconfig.yaml> \u003Coutput_dir> [OPTIONS]\n```\n\n**Options:**\n| Option | Default | Description |\n|--------|---------|-------------|\n| `--alpha` | `0.05` | P-value threshold for significance |\n| `--exclude-criteria` | `[]` | Criteria to exclude (repeatable) |\n| `--print-model-usage` | `false` | Print LLM token usage |\n| `--include-score-id-in-prompt` | `true` | Add a unique ID to provider requests; does not bypass `cache_config` |\n| `--account-url` | `null` | Azure Blob Storage account URL (managed-identity auth). Use when the config path is a `blob:\u002F\u002F` URI. |\n| `--connection-string` | `null` | Azure Blob Storage connection string. Use when the config path is a `blob:\u002F\u002F` URI. |\n\n**Config requires**: `base` (reference method), `others` (methods to compare), `question_sets`, `criteria`, `trials` (must be even), `llm_config`, `prompt_config`. Generated configurations also include `cache_config`, with persistent SQLite enabled by default.\n\nDefault criteria: `comprehensiveness`, `diversity`, `empowerment`, `relevance`\n\n**Output**: `{question_set}_{base}--{other}.csv`, `win_rates.csv`, `winrates_sig_tests.csv`\n\n**Pairwise caching**:\n\n- Keep the generated SQLite `cache_config` to resume interrupted runs and reuse\n  identical question\u002Fcriterion\u002Ftrial judgments.\n- Use `cache_config: {type: none, storage: null}` to request fresh judgments.\n- Each trial is cached separately; caching does not collapse repeated trials or\n  counterbalanced answer ordering.\n- Existing comparison CSVs are checked before the judgment cache. For a fully\n  fresh comparison, remove\u002Fuse a new output CSV and disable or relocate the\n  cache.\n- Inspect the default cache with\n  `benchmark-qed cache inspect .benchmark_qed_cache\u002Fpairwise\u002Fpairwise.sqlite3`.\n- `--include-score-id-in-prompt` may avoid provider-side prompt caching but does\n  not bypass BenchmarkQED's configured pairwise cache.\n\n### 1a. Differential Pairwise Scores\n\nUse `autoe differential-pairwise-scores` for the two-stage extract-and-judge\nmethod. Generated configs use a separate SQLite cache at\n`.benchmark_qed_cache\u002Fdifferential_pairwise\u002Fdifferential_pairwise.sqlite3`.\n\n- Extraction and verdict stages are cached independently.\n- Judge failures resume from completed extraction.\n- Criteria or judge-prompt changes reuse extraction and recompute verdicts.\n- Trials remain separate and counterbalanced.\n- Use `cache_config: {type: none, storage: null}` for fully fresh calls, and\n  also remove\u002Fuse a new output CSV.\n\n### 2. Reference Scores\n\nScore generated answers against reference (gold-standard) answers.\n\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe reference-scores \u003Cconfig.yaml> \u003Coutput_dir> [OPTIONS]\n```\n\n**Config requires**: `reference`, `generated` (list), `criteria`, `score_min`\u002F`score_max`, `trials`, `llm_config`. Generated configs include a persistent SQLite `cache_config`.\n\nDefault criteria: `correctness`, `completeness`. Default score range: 1–10.\n\n**Output**: `reference_scores-{name}.csv`, `model_usage.json`\n\n**Reference caching**:\n\n- Each question\u002Fcriterion\u002Ftrial is cached separately under the `reference`\n  namespace.\n- Cache identity includes both answers, score range, prompts, model\u002Fprovider,\n  call arguments, and score-ID mode.\n- Rerun after a failure to reuse completed judgments.\n- Use `cache_config: {type: none, storage: null}` for fresh judgments.\n- Inspect the default cache with\n  `benchmark-qed cache inspect .benchmark_qed_cache\u002Freference\u002Freference.sqlite3`.\n\n### 3. Assertion Scores\n\nEvaluate RAG methods using assertion-based scoring. Auto-detects single-RAG vs multi-RAG config.\n\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe assertion-scores \u003Cconfig.yaml> \u003Coutput_dir> [OPTIONS]\n```\n\n**Options:**\n| Option | Default | Description |\n|--------|---------|-------------|\n| `--alpha` | `0.05` | Significance threshold (multi-RAG) |\n| `--print-model-usage` | `false` | Print LLM token usage |\n| `--account-url` | `null` | Azure Blob Storage account URL (managed-identity auth). Use when the config path is a `blob:\u002F\u002F` URI. |\n| `--connection-string` | `null` | Azure Blob Storage connection string. Use when the config path is a `blob:\u002F\u002F` URI. |\n\n**Auto-detection**: If the YAML contains a `rag_methods` key, it runs in multi-RAG mode with automated significance testing. Otherwise, single-RAG mode.\n\n**Single-RAG output**: `assertion_scores.csv`, `assertion_summary_by_question.csv`, `eval_summary.json`\n\n**Multi-RAG output**: Per-method scores + significance tests in structured `output_dir\u002F`\n\n**Single-RAG assertion caching**:\n\n- Generated configs use SQLite at\n  `.benchmark_qed_cache\u002Fassertion\u002Fassertion.sqlite3`.\n- Each question\u002Fassertion\u002Ftrial is cached independently; the identity also\n  includes the answer, prompts, model\u002Fprovider, call arguments, and score-ID\n  mode.\n- Rerun after a failure to reuse completed judgments.\n- Inspect with `benchmark-qed cache inspect\n  .benchmark_qed_cache\u002Fassertion\u002Fassertion.sqlite3`.\n- Use `cache_config: {type: none, storage: null}` for fresh judgments.\n- This cache is not used by multi-RAG or hierarchical assertion scoring.\n  Chunk-assertion scoring has a separate assertion\u002Fchunk cache.\n\n### 4. Hierarchical Assertion Scores\n\nScore hierarchical assertions (global assertions with supporting local assertions).\n\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe hierarchical-assertion-scores \u003Cconfig.yaml> \u003Coutput_dir> [OPTIONS]\n```\n\n**Modes**: `staged` (default — evaluate local first, then global) or `joint` (evaluate together)\n\n**Extra field**: `detect_discovery: true` enables detection of novel findings not covered by assertions.\n\nAlso auto-detects single vs multi-RAG config (same as assertion-scores).\n\n### 5. Assertion Significance\n\nRun statistical significance tests on existing assertion scores (no LLM calls).\n\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe assertion-significance \u003Cconfig.yaml>\n```\n\n**Config requires**: `output_dir`, `rag_methods`, `question_sets`, `alpha`, `correction_method`\n\n**Correction methods**: `holm` (default, recommended), `bonferroni`, `fdr_bh`\n\n### 6. Hierarchical Assertion Significance\n\nSignificance tests on hierarchical assertion scores.\n\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe hierarchical-assertion-significance \u003Cconfig.yaml>\n```\n\n**Config requires**: `scores_dir`, `rag_methods`, `scores_filename_template`, `alpha`, `correction_method`, `output_dir`\n\n### 7. Generate Retrieval Reference\n\nGenerate cluster relevance reference data for retrieval evaluation (one-off prep step).\n\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe generate-retrieval-reference \u003Cconfig.yaml>\n```\n\n**Config requires**: `llm_config`, `embedding_config`, question source (`questions_path` or `question_sets`), `text_units_path`\n\n**Key settings**: `num_clusters`, `assessor_type` (`rationale` or `bing`), `semantic_neighbors`, `centroid_neighbors`\n\n### 8. Retrieval Scores\n\nEvaluate retrieval precision, recall, and fidelity for RAG methods.\n\n```bash\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe retrieval-scores \u003Cconfig.yaml>\n```\n\n**Config requires**: `rag_methods`, `question_sets`, `reference_dir`, `text_units_path`, `output_dir`\n\n**Fidelity metrics**: `js` (Jensen-Shannon divergence) or `tvd` (total variation distance)\n\n## Workflow\n\n### Quick Evaluation (Assertion-Based)\n\n- [ ] Step 1: Verify questions and answers exist — list the workspace and confirm a `settings.yaml` (or `config.yaml`), question JSON files (typically under `output\u002F`), and your RAG method answer JSONs are present.\n- [ ] Step 2: Initialize eval config — use the `benchmark-qed-setup` skill to create and configure an assertion evaluation workspace.\n- [ ] Step 3: Configure settings.yaml with answer paths and assertion paths\n- [ ] Step 4: Run evaluation — `uvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe assertion-scores .\u002Feval_workspace\u002Fsettings.yaml .\u002Feval_output`\n- [ ] Step 5: Summarize results — read the CSVs in `\u003Coutput_dir>` (e.g. `assertion_scores.csv`, `assertion_summary_by_question.csv`) and `eval_summary.json` directly.\n\n### Multi-RAG Comparison\n\nFor comparing multiple RAG methods, use multi-RAG config format (include `rag_methods` key in YAML). This gives you automated pairwise significance testing.\n\n## Gotchas\n\n- **Config auto-detection**: `assertion-scores` and `hierarchical-assertion-scores` detect single vs multi-RAG based on the `rag_methods` key in YAML. Ensure your config matches your intent.\n- **Trials must be even**: For pairwise scores, `trials` must be even (for counterbalancing). Use 4 as default.\n- **Pairwise cache identity**: Questions, answers and labels, criterion, trial,\n  prompts, model\u002Fprovider, call arguments, and score-ID mode determine reuse.\n  Changing any of them creates a cache miss.\n- **Blob storage does not relocate SQLite caches**: Pairwise\n  `cache_config.storage.base_dir` remains a local path even when input\u002Foutput\n  storage uses Azure Blob Storage.\n- **Cache storage is independent**: JSON caches use the storage configured\n  under `cache_config.storage`; they do not inherit input\u002Foutput blob settings.\n  `type: file` resolves a relative `base_dir` from the command's working\n  directory. Use `cache_config.storage.type: blob` with an explicit container\n  and credentials to store JSON entries in Azure Blob Storage.\n- **Stale outputs**: Several commands skip existing output files. Use a fresh output directory or delete specific files to force re-evaluation.\n- **Output is in files**: All scores are written to CSV\u002FJSON files. Parse output files, not CLI stdout.\n- **Long-running**: Evaluation with many questions and trials can take hours. Use background execution.\n- **No `config init` for hierarchical\u002Fretrieval**: The `benchmark-qed-setup` skill only supports `autoe_assertion`, `autoe_pairwise`, and `autoe_reference`. For hierarchical, multi-RAG, and retrieval configs, create YAML manually.\n- **Advanced config types**: Use the `benchmark-qed-setup` skill for configuration guidance on advanced config types.\n\n## Azure Blob Storage\n\nAll `autoe` commands support reading config files from Azure Blob Storage using `blob:\u002F\u002F` URIs:\n\n```bash\n# Config file in blob storage\nuvx --from \"git+https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\" benchmark-qed autoe assertion-scores \\\n  blob:\u002F\u002Fmy-container\u002Feval\u002Fsettings.yaml .\u002Feval_output \\\n  --account-url https:\u002F\u002Fmyaccount.blob.core.windows.net\n```\n\nIn addition, `settings.yaml` supports `input_storage` and `output_storage` blocks so the evaluation pipeline can read answers\u002Fassertions from and write results to Azure Blob Storage:\n\n```yaml\n# Read answers and assertions from blob storage\ninput_storage:\n  type: blob\n  container_name: my-datasets\n  connection_string: ${AZURE_STORAGE_CONNECTION_STRING}\n  # Or use managed identity:\n  # account_url: https:\u002F\u002Fmyaccount.blob.core.windows.net\n\n# Write evaluation output to blob storage\noutput_storage:\n  type: blob\n  container_name: my-output\n  connection_string: ${AZURE_STORAGE_CONNECTION_STRING}\n```\n\nWhen using storage blocks, `answer_base_path` and `assertions_path` in the config are resolved relative to the storage container (not the local filesystem).\n\nSee [references\u002Fconfig-reference.md](..\u002Fbenchmark-qed-setup\u002Freferences\u002Fconfig-reference.md) for full `StorageConfig` fields.\n",{"name":5,"description":7},"main","7366a97e7a0b2fefda7b1fbb4cc5d43c791fabaa",".apm\u002Fskills\u002Fbenchmark-qed-autoe",24,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed","https:\u002F\u002Fmicrosoft.github.io\u002Fbenchmark-qed\u002F","https:\u002F\u002Fgithub.com\u002Fmicrosoft","https:\u002F\u002Fopensource.microsoft.com","Open source projects and samples from Microsoft","2026-11-07 01:53:43",{"source":8,"skillName":5,"commands":39,"prompt":49,"zipAvailable":50},[40,43,46],{"tool":41,"command":42},"npx","npx skills add microsoft\u002Fbenchmark-qed --skill benchmark-qed-autoe",{"tool":44,"command":45},"bunx","bunx skills add microsoft\u002Fbenchmark-qed --skill benchmark-qed-autoe",{"tool":47,"command":48},"pnpm","pnpm dlx skills add microsoft\u002Fbenchmark-qed --skill benchmark-qed-autoe","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\u002Fblob\u002Fmain\u002F.apm\u002Fskills\u002Fbenchmark-qed-autoe\u002FSKILL.md",true,[],[53,65],{"skillNo":54,"name":55,"nameZh":6,"description":56,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":57,"categories":58,"keywords":59,"publishedAt":23},"01M69BKZBE6VJ6WD1HX1M7PXG8","benchmark-qed-setup","Initialize and configure benchmark-qed workspaces for RAG benchmarking. Use when: setting up a new benchmarking project, initializing config files for question generation or evaluation, downloading sample datasets, or modifying benchmark-qed settings.yaml configuration. Also use when the user mentions \"benchmark-qed config\", workspace setup, or needs to prepare a benchmarking environment — even if they don't say \"setup\" explicitly.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\u002Fblob\u002Fmain\u002F.apm\u002Fskills\u002Fbenchmark-qed-setup\u002FSKILL.md",[],[60,19,61,62,63,64,21],"setup","benchmark qed setup","config","prepare","benchmarking",{"skillNo":66,"name":67,"nameZh":6,"description":68,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":69,"categories":70,"keywords":71,"publishedAt":23},"01M69BKWP6SD1EWM9AGXW54EYN","benchmark-qed-autoq","Generate benchmark questions and assertions from input data using benchmark-qed. Use when: generating local, global, linked, or activity questions for RAG benchmarking, creating assertions for existing questions, computing assertion statistics, or running the autoq question generation pipeline. Also use when the user wants to create a benchmark question set, build evaluation questions from a dataset, or generate ground-truth assertions — even if they don't say \"autoq\" explicitly.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fbenchmark-qed\u002Fblob\u002Fmain\u002F.apm\u002Fskills\u002Fbenchmark-qed-autoq\u002FSKILL.md",[],[19,72,64,21,73,74],"benchmark qed autoq","create","generate"]