benchmark-qed-autoe

Benchmark-QED Evaluation (autoe)

Evaluate and compare RAG system outputs using LLM-judged scoring, assertion-based evaluation, and retrieval metrics — all with built-in statistical significance testing.

Prerequisites

• Generated questions/assertions from the autoq pipeline (or your own)
• RAG method answer files (JSON, one per method per question set)
• A valid settings.yaml for the evaluation type
• A configured workspace with valid settings.yaml (use the benchmark-qed-setup skill to initialize and configure)
• LLM API key configured

Run all commands with:

uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed <command>

Evaluation Methods Overview

Method	Command	Best for
Pairwise comparison	autoe pairwise-scores	Comparing two RAG methods head-to-head
Differential pairwise	autoe differential-pairwise-scores	Reducing length/format bias with extract-and-judge
Reference scoring	autoe reference-scores	Scoring against gold-standard answers
Assertion scoring	autoe assertion-scores	Evaluating with ground-truth assertions (single or multi-RAG)
Hierarchical assertions	autoe hierarchical-assertion-scores	Global + local assertion hierarchies
Retrieval metrics	autoe retrieval-scores	Precision, recall, fidelity of retrieval
Significance tests	autoe assertion-significance	Post-hoc significance on existing scores

Commands

1. Pairwise Scores

Compare RAG methods using LLM-judged pairwise comparisons.

uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoe pairwise-scores <config.yaml> <output_dir> [OPTIONS]

Options:

Option	Default	Description
--alpha	0.05	P-value threshold for significance
--exclude-criteria	[]	Criteria to exclude (repeatable)
--print-model-usage	false	Print LLM token usage
--include-score-id-in-prompt	true	Add a unique ID to provider requests; does not bypass cache_config
--account-url	null	Azure Blob Storage account URL (managed-identity auth). Use when the config path is a blob:// URI.
--connection-string	null	Azure Blob Storage connection string. Use when the config path is a blob:// URI.

Config requires: base (reference method), others (methods to compare), question_sets, criteria, trials (must be even), llm_config, prompt_config. Generated configurations also include cache_config, with persistent SQLite enabled by default.

Default criteria: comprehensiveness, diversity, empowerment, relevance

Output: {question_set}_{base}--{other}.csv, win_rates.csv, winrates_sig_tests.csv

Pairwise caching:

• Keep the generated SQLite cache_config to resume interrupted runs and reuse identical question/criterion/trial judgments.
• Use cache_config: {type: none, storage: null} to request fresh judgments.
• Each trial is cached separately; caching does not collapse repeated trials or counterbalanced answer ordering.
• Existing comparison CSVs are checked before the judgment cache. For a fully fresh comparison, remove/use a new output CSV and disable or relocate the cache.
• Inspect the default cache with benchmark-qed cache inspect .benchmark_qed_cache/pairwise/pairwise.sqlite3.
• --include-score-id-in-prompt may avoid provider-side prompt caching but does not bypass BenchmarkQED's configured pairwise cache.

1a. Differential Pairwise Scores

Use autoe differential-pairwise-scores for the two-stage extract-and-judge method. Generated configs use a separate SQLite cache at .benchmark_qed_cache/differential_pairwise/differential_pairwise.sqlite3.

• Extraction and verdict stages are cached independently.
• Judge failures resume from completed extraction.
• Criteria or judge-prompt changes reuse extraction and recompute verdicts.
• Trials remain separate and counterbalanced.
• Use cache_config: {type: none, storage: null} for fully fresh calls, and also remove/use a new output CSV.

2. Reference Scores

Score generated answers against reference (gold-standard) answers.

uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoe reference-scores <config.yaml> <output_dir> [OPTIONS]

Config requires: reference, generated (list), criteria, score_min/score_max, trials, llm_config. Generated configs include a persistent SQLite cache_config.

Default criteria: correctness, completeness. Default score range: 1–10.

Output: reference_scores-{name}.csv, model_usage.json

Reference caching:

• Each question/criterion/trial is cached separately under the reference namespace.
• Cache identity includes both answers, score range, prompts, model/provider, call arguments, and score-ID mode.
• Rerun after a failure to reuse completed judgments.
• Use cache_config: {type: none, storage: null} for fresh judgments.
• Inspect the default cache with benchmark-qed cache inspect .benchmark_qed_cache/reference/reference.sqlite3.

3. Assertion Scores

Evaluate RAG methods using assertion-based scoring. Auto-detects single-RAG vs multi-RAG config.

uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoe assertion-scores <config.yaml> <output_dir> [OPTIONS]

Options:

Option	Default	Description
--alpha	0.05	Significance threshold (multi-RAG)
--print-model-usage	false	Print LLM token usage
--account-url	null	Azure Blob Storage account URL (managed-identity auth). Use when the config path is a blob:// URI.
--connection-string	null	Azure Blob Storage connection string. Use when the config path is a blob:// URI.

Auto-detection: If the YAML contains a rag_methods key, it runs in multi-RAG mode with automated significance testing. Otherwise, single-RAG mode.

Single-RAG output: assertion_scores.csv, assertion_summary_by_question.csv, eval_summary.json

Multi-RAG output: Per-method scores + significance tests in structured output_dir/

Single-RAG assertion caching:

• Generated configs use SQLite at .benchmark_qed_cache/assertion/assertion.sqlite3.
• Each question/assertion/trial is cached independently; the identity also includes the answer, prompts, model/provider, call arguments, and score-ID mode.
• Rerun after a failure to reuse completed judgments.
• Inspect with benchmark-qed cache inspect .benchmark_qed_cache/assertion/assertion.sqlite3.
• Use cache_config: {type: none, storage: null} for fresh judgments.
• This cache is not used by multi-RAG or hierarchical assertion scoring. Chunk-assertion scoring has a separate assertion/chunk cache.

4. Hierarchical Assertion Scores

Score hierarchical assertions (global assertions with supporting local assertions).

uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoe hierarchical-assertion-scores <config.yaml> <output_dir> [OPTIONS]

Modes: staged (default — evaluate local first, then global) or joint (evaluate together)

Extra field: detect_discovery: true enables detection of novel findings not covered by assertions.

Also auto-detects single vs multi-RAG config (same as assertion-scores).

5. Assertion Significance

Run statistical significance tests on existing assertion scores (no LLM calls).

uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoe assertion-significance <config.yaml>

Config requires: output_dir, rag_methods, question_sets, alpha, correction_method

Correction methods: holm (default, recommended), bonferroni, fdr_bh

6. Hierarchical Assertion Significance

Significance tests on hierarchical assertion scores.

uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoe hierarchical-assertion-significance <config.yaml>

Config requires: scores_dir, rag_methods, scores_filename_template, alpha, correction_method, output_dir

7. Generate Retrieval Reference

Generate cluster relevance reference data for retrieval evaluation (one-off prep step).

uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoe generate-retrieval-reference <config.yaml>

Config requires: llm_config, embedding_config, question source (questions_path or question_sets), text_units_path

Key settings: num_clusters, assessor_type (rationale or bing), semantic_neighbors, centroid_neighbors

8. Retrieval Scores

Evaluate retrieval precision, recall, and fidelity for RAG methods.

uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoe retrieval-scores <config.yaml>

Config requires: rag_methods, question_sets, reference_dir, text_units_path, output_dir

Fidelity metrics: js (Jensen-Shannon divergence) or tvd (total variation distance)

Workflow

Quick Evaluation (Assertion-Based)

• [ ] Step 1: Verify questions and answers exist — list the workspace and confirm a settings.yaml (or config.yaml), question JSON files (typically under output/), and your RAG method answer JSONs are present.
• [ ] Step 2: Initialize eval config — use the benchmark-qed-setup skill to create and configure an assertion evaluation workspace.
• [ ] Step 3: Configure settings.yaml with answer paths and assertion paths
• [ ] Step 4: Run evaluation — uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoe assertion-scores ./eval_workspace/settings.yaml ./eval_output
• [ ] Step 5: Summarize results — read the CSVs in <output_dir> (e.g. assertion_scores.csv, assertion_summary_by_question.csv) and eval_summary.json directly.

Multi-RAG Comparison

For comparing multiple RAG methods, use multi-RAG config format (include rag_methods key in YAML). This gives you automated pairwise significance testing.

Gotchas

• Config auto-detection: assertion-scores and hierarchical-assertion-scores detect single vs multi-RAG based on the rag_methods key in YAML. Ensure your config matches your intent.
• Trials must be even: For pairwise scores, trials must be even (for counterbalancing). Use 4 as default.
• Pairwise cache identity: Questions, answers and labels, criterion, trial, prompts, model/provider, call arguments, and score-ID mode determine reuse. Changing any of them creates a cache miss.
• Blob storage does not relocate SQLite caches: Pairwise cache_config.storage.base_dir remains a local path even when input/output storage uses Azure Blob Storage.
• Cache storage is independent: JSON caches use the storage configured under cache_config.storage; they do not inherit input/output blob settings. type: file resolves a relative base_dir from the command's working directory. Use cache_config.storage.type: blob with an explicit container and credentials to store JSON entries in Azure Blob Storage.
• Stale outputs: Several commands skip existing output files. Use a fresh output directory or delete specific files to force re-evaluation.
• Output is in files: All scores are written to CSV/JSON files. Parse output files, not CLI stdout.
• Long-running: Evaluation with many questions and trials can take hours. Use background execution.
• No config init for hierarchical/retrieval: The benchmark-qed-setup skill only supports autoe_assertion, autoe_pairwise, and autoe_reference. For hierarchical, multi-RAG, and retrieval configs, create YAML manually.
• Advanced config types: Use the benchmark-qed-setup skill for configuration guidance on advanced config types.

Azure Blob Storage

All autoe commands support reading config files from Azure Blob Storage using blob:// URIs:

# Config file in blob storage
uvx --from "git+https://github.com/microsoft/benchmark-qed" benchmark-qed autoe assertion-scores \
  blob://my-container/eval/settings.yaml ./eval_output \
  --account-url https://myaccount.blob.core.windows.net

In addition, settings.yaml supports input_storage and output_storage blocks so the evaluation pipeline can read answers/assertions from and write results to Azure Blob Storage:

# Read answers and assertions from blob storage
input_storage:
  type: blob
  container_name: my-datasets
  connection_string: ${AZURE_STORAGE_CONNECTION_STRING}
  # Or use managed identity:
  # account_url: https://myaccount.blob.core.windows.net

# Write evaluation output to blob storage
output_storage:
  type: blob
  container_name: my-output
  connection_string: ${AZURE_STORAGE_CONNECTION_STRING}

When using storage blocks, answer_base_path and assertions_path in the config are resolved relative to the storage container (not the local filesystem).

See references/config-reference.md for full StorageConfig fields.