AGENT SKILL

rag-eval

Filesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). Not for prod monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.

License 许可
Apache-2.0
所属仓库
themusebook/skills
最近更新
2026-11-02
Stars
★ 0
Forks
1

01 前言 Front Matter

name
rag-eval
license
Apache-2.0
version
2.6.0
metadata
{ "tags": [ "nvidia", "blueprint", "rag", "evaluation", "ragas", "benchmarking", "nvidia-rag-blueprint" ], "author": "NVIDIA RAG <foundational-rag-dev@exchange.nvidia.com>", "domain": "ai-ml", "languages": [ "python", "shell" ], "frameworks": [ "ragas", "fastapi" ], "github-url": "https://github.com/NVIDIA-AI-Blueprints/rag", "argument-hint": "RAGAS eval | evaluate_rag | train.json | corpus | results json | error triage | uv run --project scripts/eval | enable_reranker | query_rewriting | temperature | skip_ingestion", "endpoint-openapi-schemas": [ "docs/api_reference/openapi_schema_rag_server.json", "docs/api_reference/openapi_schema_ingestor_server.json" ] }
description
Filesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). Not for prod monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.
allowed-tools
Read Grep Glob Bash(ls *) Bash(python3 *) Bash(uv *) Write Edit
compatibility
Repository checkout with uv; Python 3.11+; run from repo root; uv sync --project scripts/eval (eval deps live in scripts/eval/pyproject.toml); network to RAG, ingestor, and vdb endpoints; NVIDIA_API_KEY for RAGAS; optional RAG_EVAL_JUDGE_MODEL (default mistralai/mixtral-8x22b-instruct-v0.1).

02 资源文件 9

03 SKILL.md 正文

On-disk RAG evaluation (corpus/ + train.json)

Purpose

Guide agents through NVIDIA RAG Blueprint filesystem benchmarks: preparing corpus/ and train.json, running scripts/eval/evaluate_rag.py, tuning retrieval and generation flags for quality comparisons, interpreting RAGAS JSON outputs, and triaging failures (HTTP/stream errors, empty contexts, collection mismatch, judge API).

For latency, throughput, and load testing, use the rag-perf skill (scripts/rag-perf, docs/performance-benchmarking.md) — not this skill.

When not to use

Do not use this skill for: deploying or repairing services (use rag-blueprint); evaluating APIs without the corpus/ + train.json layout; general ML experimentation unrelated to this evaluator; production monitoring/alerting; or latency/throughput benchmarking (use rag-perf).

Prerequisites

  • Repo cloned; run commands from repo root (imports and paths assume this).
  • Python 3.11+ and uv; eval deps: uv sync --project scripts/eval.
  • Reachable RAG server and ingestor (defaults often localhost:8081 / 8082).
  • NVIDIA_API_KEY for RAGAS (see credential hygiene); optional RAG_EVAL_JUDGE_MODEL.
  • Dataset roots passed to --dataset-paths each contain corpus/ and train.json.

Instructions

  1. Prepare data — Ensure each dataset directory matches the layout and train.json rules in references/dataset-and-conversion.md. When sources arrive as public links (sites or dataset pages), materialize documents under corpus/—prefer PDF for multimodal content so images stay embedded; convert CSV/JSONL/etc. using the patterns there.
  2. Run eval — uv run --project scripts/eval python scripts/eval/evaluate_rag.py with --dataset-paths, --host, and --port. See references/benchmark-execution.md for command examples, outputs, and errors. Use references/evaluate-rag-cli.md for flag-level detail.
  3. Tune quality — Adjust --top_k / --vdb_top_k, reranker and query-rewriting toggles, and generation overrides (--temperature, --top-p, --max-tokens) as documented in references/benchmark-execution.md when comparing retrieval/generation configs for RAGAS scores.
  4. Analyze results — Use references/result-analysis.md for scripts; scan rag_*_evaluation_summary.json for headline RAGAS metrics.
  5. Triage errors — Use the error signal table and the Troubleshooting section below.

Examples

Set API key without putting secrets in shell history (preferred patterns): load from a gitignored env file or secrets manager; avoid committing .env; rotate keys if exposed. Details: references/benchmark-execution.md#credential-hygiene-nvidia_api_key.

Minimal eval (key already in environment):

uv sync --project scripts/eval
uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
  --dataset-paths /path/to/my_dataset \
  --host localhost \
  --port 8081

Pretty-print summary JSON:

python3 -m json.tool results/my_dataset/rag_my_dataset_evaluation_summary.json

More examples (skip ingestion, quality sweeps): references/benchmark-execution.md.

Limitations

  • Evaluator behavior is fixed to the filesystem contract and evaluate_rag.py; it does not substitute for custom offline judges or non-RAG benchmarks.
  • Vector DB / embedding choices follow deployed ingestor and RAG env — not overridden by this CLI alone.
  • Scores depend on retrieval quality, judge model availability, and NVIDIA_API_KEY; empty contexts yield partial RAGAS metrics (see references).
  • Large procedural detail lives under references/ to keep routing concise; read those files when the user needs step-by-step conversion, full flags, or error tables.

Troubleshooting

Error / signal Likely cause What to do
Immediate exit mentioning NVIDIA_API_KEY Missing or invalid key Set key via secure channel; see credential hygiene in references/benchmark-execution.md.
train.json must be a JSON array Wrong JSON shape Top-level array of objects; validate per references/dataset-and-conversion.md.
Fewer rows in evaluation_data.json than train.json Per-query failures Check stderr: network or stream JSON errors; see error table in benchmark-execution.
Empty generated_contexts everywhere Retrieval gap Verify collection, ingestion, top_k / vdb_top_k, and ingestor_server_url without /v1 suffix.
Ingestor 404 on upload Bad ingestor base URL Pass http://host:port only — code appends /v1/.

Full signal table: references/benchmark-execution.md#common-error-cases-and-signals.

Gotchas

  • Run from repo root: paths and imports in scripts/eval/evaluate_rag.py assume this; a wrong directory silently breaks imports.
  • --ingestor_server_url: pass http://host:port without /v1—the code appends /v1/ automatically. Including /v1 causes 404s on ingestor calls.
  • Vector DB / embedding settings: not set by this CLI; configure via the deployed ingestor and RAG server env vars (e.g. APP_VECTORSTORE_URL, embedding model).
  • --model / --llm_endpoint: forwarded verbatim only when explicitly set; omit to keep the server's configured LLM.
  • Stale collections: a previous run's ingested data persists unless you use --force_ingestion. Use --collection with a unique name when comparing quality across isolated runs.
  • Empty context metrics: if all generated_contexts are empty, RAGAS scores only nv_accuracy and leaves the other two metrics blank—this is not a silent success.

Source of truth

Piece Location
Driver scripts/eval/evaluate_rag.py (CORPUS_DIRECTORY = corpus, EVAL_DATA = train.json)
Human README (always in-repo) scripts/eval/README.md
Full CLI (flags, defaults) scripts/eval/evaluate_rag.py --help; references/evaluate-rag-cli.md
Dataset / conversion references/dataset-and-conversion.md
Runs, outputs, errors references/benchmark-execution.md
Result analysis scripts references/result-analysis.md
Latency / throughput rag-perf skill, docs/performance-benchmarking.md

Agent playbook

  1. Run eval — uv sync --project scripts/eval then uv run --project scripts/eval python scripts/eval/evaluate_rag.py with required --dataset-paths, --host, and --port (and env NVIDIA_API_KEY). Argument --ingestor_server_url is optional (defaults to http://localhost:8082); pass it only when overriding the ingestor endpoint.
  2. Quality tuning — See references/benchmark-execution.md: --top_k/--vdb_top_k, reranker and query-rewriting toggles, --temperature, --top-p, --max-tokens.
  3. Data conversion — Follow references/dataset-and-conversion.md.
  4. Analyze results — references/result-analysis.md; quick scan: python3 -m json.tool results/<dataset>/rag_<dataset>_evaluation_summary.json.
  5. Error triage — references/benchmark-execution.md#common-error-cases-and-signals.

04 同仓库更多 Skills

themusebook
themusebook/skills

youtube-clipper

YouTube 视频智能剪辑工具。下载视频和字幕,AI 分析生成精细章节(几分钟级别), 用户选择片段后自动剪辑、翻译字幕为中英双语、烧录字幕到视频,并生成总结文案。 使用场景:当用户需要剪辑 YouTube 视频、生成短视频片段、制作双语字幕版本时。 关键词:视频剪辑、YouTube、字幕翻译、双语字幕、视频下载、clip video

★ 0
themusebook
themusebook/skills

viem-integration

Integrate EVM blockchains using viem. Use when user says "read blockchain data", "send transaction", "interact with smart contract", "connect to Ethereum", "use viem", "use wagmi", "wallet integration", "viem setup", or mentions blockchain/EVM development with TypeScript.

★ 0
themusebook
themusebook/skills

v4-security-foundations

Security-first Uniswap v4 hook development. Use when user mentions "v4 hooks", "hook security", "PoolManager", "beforeSwap", "afterSwap", or asks about V4 hook best practices, vulnerabilities, or audit requirements.

★ 0
themusebook
themusebook/skills

v4-sdk-integration

App-layer SDK guide for building swap and liquidity experiences directly with the Uniswap v4 SDK. Use when user asks about "v4 sdk", "uniswap v4", "v4 swap", "v4 liquidity", "PoolManager", "V4Planner", "StateView", "PositionManager", "pool state", "v4 position", "uniswap sdk", or when building swap/liquidity UX directly with SDKs rather than via the Trading API.

★ 0
themusebook
themusebook/skills

v4-hook-generator

Generate Uniswap v4 hook contracts via OpenZeppelin MCP. Use when building custom swap logic, async swaps, hook-owned liquidity, custom curves, dynamic fees, MEV protection, limit orders, or oracle hooks.

★ 0
themusebook
themusebook/skills

testing

For testing with Light Protocol programs and clients on localnet, devnet, and mainnet validation.

★ 0