rag-eval
Filesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). Not for prod monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.
- License 许可
- Apache-2.0
- 最近更新
- 2026-10-08
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- rag-eval
- license
- Apache-2.0
- version
- 2.6.0
- metadata
- { "tags": [ "nvidia", "blueprint", "rag", "evaluation", "ragas", "benchmarking", "nvidia-rag-blueprint" ], "author": "NVIDIA RAG <foundational-rag-dev@exchange.nvidia.com>", "domain": "ai-ml", "languages": [ "python", "shell" ], "frameworks": [ "ragas", "fastapi" ], "github-url": "https://github.com/NVIDIA-AI-Blueprints/rag", "argument-hint": "RAGAS eval | evaluate_rag | train.json | corpus | results json | error triage | uv run --project scripts/eval | enable_reranker | query_rewriting | temperature | skip_ingestion", "endpoint-openapi-schemas": [ "docs/api_reference/openapi_schema_rag_server.json", "docs/api_reference/openapi_schema_ingestor_server.json" ] }
- description
- Filesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). Not for prod monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.
- allowed-tools
- Read Grep Glob Bash(ls *) Bash(python3 *) Bash(uv *) Write Edit
- compatibility
- Repository checkout with uv; Python 3.11+; run from repo root; uv sync --project scripts/eval (eval deps live in scripts/eval/pyproject.toml); network to RAG, ingestor, and vdb endpoints; NVIDIA_API_KEY for RAGAS; optional RAG_EVAL_JUDGE_MODEL (default mistralai/mixtral-8x22b-instruct-v0.1).
02 资源文件 9
- 参考文档BENCHMARK.md2.8 KB
- 参考文档eval/h100.json2.8 KB
- 参考文档eval/nvidia_hosted.json1.9 KB
- 参考文档references/benchmark-execution.md8.1 KB
- 参考文档references/dataset-and-conversion.md8.2 KB
- 参考文档references/evaluate-rag-cli.md4.2 KB
- 参考文档references/result-analysis.md2.8 KB
- 参考文档skill-card.md2.6 KB
- 其他skill.oms.sig5.7 KB
03 SKILL.md 正文
On-disk RAG evaluation (corpus/ + train.json)
Purpose
Guide agents through NVIDIA RAG Blueprint filesystem benchmarks: preparing corpus/ and train.json, running scripts/eval/evaluate_rag.py, tuning retrieval and generation flags for quality comparisons, interpreting RAGAS JSON outputs, and triaging failures (HTTP/stream errors, empty contexts, collection mismatch, judge API).
For latency, throughput, and load testing, use the rag-perf skill (scripts/rag-perf, docs/performance-benchmarking.md) — not this skill.
When not to use
Do not use this skill for: deploying or repairing services (use rag-blueprint); evaluating APIs without the corpus/ + train.json layout; general ML experimentation unrelated to this evaluator; production monitoring/alerting; or latency/throughput benchmarking (use rag-perf).
Prerequisites
- Repo cloned; run commands from repo root (imports and paths assume this).
- Python 3.11+ and uv; eval deps:
uv sync --project scripts/eval. - Reachable RAG server and ingestor (defaults often
localhost:8081/8082). NVIDIA_API_KEYfor RAGAS (see credential hygiene); optionalRAG_EVAL_JUDGE_MODEL.- Dataset roots passed to
--dataset-pathseach containcorpus/andtrain.json.
Instructions
- Prepare data — Ensure each dataset directory matches the layout and
train.jsonrules inreferences/dataset-and-conversion.md. When sources arrive as public links (sites or dataset pages), materialize documents undercorpus/—prefer PDF for multimodal content so images stay embedded; convert CSV/JSONL/etc. using the patterns there. - Run eval —
uv run --project scripts/eval python scripts/eval/evaluate_rag.pywith--dataset-paths,--host, and--port. Seereferences/benchmark-execution.mdfor command examples, outputs, and errors. Usereferences/evaluate-rag-cli.mdfor flag-level detail. - Tune quality — Adjust
--top_k/--vdb_top_k, reranker and query-rewriting toggles, and generation overrides (--temperature,--top-p,--max-tokens) as documented inreferences/benchmark-execution.mdwhen comparing retrieval/generation configs for RAGAS scores. - Analyze results — Use
references/result-analysis.mdfor scripts; scanrag_*_evaluation_summary.jsonfor headline RAGAS metrics. - Triage errors — Use the error signal table and the Troubleshooting section below.
Examples
Set API key without putting secrets in shell history (preferred patterns): load from a gitignored env file or secrets manager; avoid committing .env; rotate keys if exposed. Details: references/benchmark-execution.md#credential-hygiene-nvidia_api_key.
Minimal eval (key already in environment):
uv sync --project scripts/eval
uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
--dataset-paths /path/to/my_dataset \
--host localhost \
--port 8081
Pretty-print summary JSON:
python3 -m json.tool results/my_dataset/rag_my_dataset_evaluation_summary.json
More examples (skip ingestion, quality sweeps): references/benchmark-execution.md.
Limitations
- Evaluator behavior is fixed to the filesystem contract and
evaluate_rag.py; it does not substitute for custom offline judges or non-RAG benchmarks. - Vector DB / embedding choices follow deployed ingestor and RAG env — not overridden by this CLI alone.
- Scores depend on retrieval quality, judge model availability, and
NVIDIA_API_KEY; empty contexts yield partial RAGAS metrics (see references). - Large procedural detail lives under
references/to keep routing concise; read those files when the user needs step-by-step conversion, full flags, or error tables.
Troubleshooting
| Error / signal | Likely cause | What to do |
|---|---|---|
Immediate exit mentioning NVIDIA_API_KEY |
Missing or invalid key | Set key via secure channel; see credential hygiene in references/benchmark-execution.md. |
train.json must be a JSON array |
Wrong JSON shape | Top-level array of objects; validate per references/dataset-and-conversion.md. |
Fewer rows in evaluation_data.json than train.json |
Per-query failures | Check stderr: network or stream JSON errors; see error table in benchmark-execution. |
Empty generated_contexts everywhere |
Retrieval gap | Verify collection, ingestion, top_k / vdb_top_k, and ingestor_server_url without /v1 suffix. |
| Ingestor 404 on upload | Bad ingestor base URL | Pass http://host:port only — code appends /v1/. |
Full signal table: references/benchmark-execution.md#common-error-cases-and-signals.
Gotchas
- Run from repo root: paths and imports in
scripts/eval/evaluate_rag.pyassume this; a wrong directory silently breaks imports. --ingestor_server_url: passhttp://host:portwithout/v1—the code appends/v1/automatically. Including/v1causes 404s on ingestor calls.- Vector DB / embedding settings: not set by this CLI; configure via the deployed ingestor and RAG server env vars (e.g.
APP_VECTORSTORE_URL, embedding model). --model/--llm_endpoint: forwarded verbatim only when explicitly set; omit to keep the server's configured LLM.- Stale collections: a previous run's ingested data persists unless you use
--force_ingestion. Use--collectionwith a unique name when comparing quality across isolated runs. - Empty context metrics: if all
generated_contextsare empty, RAGAS scores onlynv_accuracyand leaves the other two metrics blank—this is not a silent success.
Source of truth
| Piece | Location |
|---|---|
| Driver | scripts/eval/evaluate_rag.py (CORPUS_DIRECTORY = corpus, EVAL_DATA = train.json) |
| Human README (always in-repo) | scripts/eval/README.md |
| Full CLI (flags, defaults) | scripts/eval/evaluate_rag.py --help; references/evaluate-rag-cli.md |
| Dataset / conversion | references/dataset-and-conversion.md |
| Runs, outputs, errors | references/benchmark-execution.md |
| Result analysis scripts | references/result-analysis.md |
| Latency / throughput | rag-perf skill, docs/performance-benchmarking.md |
Agent playbook
- Run eval —
uv sync --project scripts/evalthenuv run --project scripts/eval python scripts/eval/evaluate_rag.pywith required--dataset-paths,--host, and--port(and envNVIDIA_API_KEY). Argument--ingestor_server_urlis optional (defaults tohttp://localhost:8082); pass it only when overriding the ingestor endpoint. - Quality tuning — See
references/benchmark-execution.md:--top_k/--vdb_top_k, reranker and query-rewriting toggles,--temperature,--top-p,--max-tokens. - Data conversion — Follow
references/dataset-and-conversion.md. - Analyze results —
references/result-analysis.md; quick scan:python3 -m json.tool results/<dataset>/rag_<dataset>_evaluation_summary.json. - Error triage —
references/benchmark-execution.md#common-error-cases-and-signals.
04 同仓库更多 Skills
zkrouter
Self-hosted OpenAI-compatible LLM router — smart tier-based routing, ZK-stamped routing decisions, mode overrides, free OpenRouter at install via the Birth bot. Save 60-80% on inference costs by routing to the cheapest capable model across 12+ OpenRouter models.
zk
For custom ZK Solana programs and privacy-preserving applications to prevent double spending. Guide to integrate rent-free nullifier PDAs for double-spend prevention.
zk-omni-messaging
Zero-knowledge omnichain messaging between Robinhood Chain and Solana using CheshireZkOmniMessenger (msgType 4) with nullifier anti-replay and the zk-omni-relayer service.
zerion-wallet-data
Use when the user asks to check a wallet's portfolio, token balances, DeFi positions, PnL, transaction history, or NFTs. Provides interpreted on-chain data across 40+ EVM chains and Solana via x402 pay-per-request or API key — no raw RPC decoding needed.
youtube-clipper
YouTube 视频智能剪辑工具。下载视频和字幕,AI 分析生成精细章节(几分钟级别), 用户选择片段后自动剪辑、翻译字幕为中英双语、烧录字幕到视频,并生成总结文案。 使用场景:当用户需要剪辑 YouTube 视频、生成短视频片段、制作双语字幕版本时。 关键词:视频剪辑、YouTube、字幕翻译、双语字幕、视频下载、clip video
yield-optimization
Multi-chain yield optimization via yield.xyz (StakeKit). Discover 2,988+ yield opportunities across 75+ blockchains, build deposit/withdrawal transactions with shell scripts, and sign them with a MoonPay wallet. Use when the user wants to earn yield, find the best lending or staking rate, enter/exit a position, or check portfolio balances.