AGENT SKILL

rag-perf

NVIDIA RAG Blueprint performance-tuning guidance for profiling retrieval stacks, comparing bottlenecks, and validating latency or throughput improvements.

License 许可
Apache-2.0
所属仓库
bg-szy/TOP-SKILLS
最近更新
2026-11-15
Stars
★ 6
Forks
0

01 前言 Front Matter

name
rag-perf
tags
[ "nvidia", "rag", "performance", "latency", "throughput", "profiling" ]
license
Apache-2.0
version
2.0
description
NVIDIA RAG Blueprint performance-tuning guidance for profiling retrieval stacks, comparing bottlenecks, and validating latency or throughput improvements.
last_updated
1788825600000
compatibility
Guidance imported from the NVIDIA RAG Blueprint performance skill for current benchmark and profiling workflows.

02 资源文件 4

03 SKILL.md 正文

RAG-Perf — config-driven perf benchmark CLI

Purpose

Drive a deployed NVIDIA RAG Blueprint server with a YAML config, run a server-side profiling pass (per-stage timing, citation quality, bottleneck inference) and an optional aiperf load test (TTFT / E2E / token & request throughput / error rate), and write a unified report. The CLI is intentionally minimal: rag-perf -c <config> plus --help / --version. Behaviour is fully config-driven; field variations belong in YAML.

Scope

  • Accuracy / RAGAS scoring of answer quality → use the rag-eval skill.
  • Deploying, repairing, or configuring services (compose, helm, NIM env vars) → use the rag-blueprint skill.
  • Production monitoring / alerting — rag-perf is a one-shot benchmark tool.
  • Runtime requirement: a deployed RAG server reachable on the network.

Prerequisites

  • Repo cloned; run commands from the repo root (config paths in the presets are repo-root-relative).
  • Python 3.11+ and uv on PATH.
  • Install rag-perf into its own uv-managed venv: uv sync --project scripts/rag-perf.
  • For unit tests: install dev extras as well — uv sync --project scripts/rag-perf --extra dev (otherwise pytest-asyncio is missing and async tests error out at collection time).
  • A reachable RAG server (default http://localhost:8081). For the aiperf phase, the bundled nvidia_rag endpoint plugin must be installed — pip install -e ./scripts/rag-perf registers it via the aiperf.plugins entry point.
  • For synthetic queries: an OpenAI-compatible chat-completions endpoint reachable at synthetic.llm_url (default http://localhost:8999/v1/chat/completions).
  • rag-perf itself runs without NVIDIA_API_KEY (unlike rag-eval). The synthetic LLM endpoint may require its own auth — that's the deployment's concern.

Instructions

  1. Pick a preset. The three under scripts/rag-perf/configs/ are:

    • quick_profile.yaml — profile-only, ~30 s. Skips load test. For fast iteration on retrieval / reranker tuning.
    • single_run.yaml — one concurrency level, profiling + aiperf, ~2 min. Regression checks.
    • sweep.yaml — multi-axis sweep. load.concurrency, rag.vdb_top_k, rag.reranker_top_k are all int | list[int]; any of them as a list becomes a sweep axis (Cartesian product).
  2. Edit the preset. Required: replace rag.collection_names: ["<collection_name>"] with a real collection on the deployed ingestor server. Verify the collection exists via GET /v1/collections on the ingestor. The placeholder <collection_name> validates fine but every request will fail at retrieval. Use a copied YAML preset for variants; the CLI surface is intentionally config-only.

  3. Run. From repo root:

    uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/single_run.yaml
    

    Same form for the other presets. The CLI accepts only -c / --config (required), --help, --version.

  4. Read stdout. Every invocation prints, in order: a startup banner, a one-line summary, the fully resolved config as YAML (so the run is reproducible from terminal output), per-grid-point progress with the shlex-joined aiperf command in copy-pastable form, a rich per-point summary table (stage breakdown with bars, citation quality, bottleneck, load-test block), and finally a side-by-side comparison table auto-labelled by whichever axis varied. See references/output-and-analysis.md.

  5. Inspect artifacts. Layout depends on run shape — flat for single-point + iterations=1, nested under iter_<i>/<point>/... otherwise. See references/output-and-analysis.md for the full directory tree, file purposes, and how to parse results.json / results.csv / report.md.

  6. Summarise for the user. When reporting back, follow the playbook in references/output-and-analysis.md#summarising-results-to-the-user: pick the canonical result file for the run shape, build a headline table (concurrency × top-k axes × TTFT × throughput × bottleneck × citation quality), compute scaling efficiency on sweeps, always flag zero citations / non-zero error rate / suspect llm_ttft_ms / small-sample p99, and propose a concrete next-experiment YAML.

  7. Tune. Schema is fully documented in docs/performance-benchmarking.md and the deeper-dive references below. Common knobs: turn aiperf.enabled: false for profile-only mode, increase load.iterations for variance estimation, set load.sleep_between_points_s: 60 for overnight Cartesian sweeps.

Examples

Profile-only (quickest signal on retrieval / reranker tuning):

uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/quick_profile.yaml

Output: rag-perf-results/quick_profile/run_<ts>/{profile_report.md, profile_results.json, profiling/}. The aiperf_rag_on/ directory is omitted. Filenames are profile_* because aiperf.enabled: false.

Single benchmark point with full report:

uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/single_run.yaml

Output: flat run_<ts>/{report.md, results.json, results.csv, profiling/, aiperf_rag_on/}.

Concurrency sweep:

uv run --project scripts/rag-perf rag-perf -c scripts/rag-perf/configs/sweep.yaml

Output: nested run_<ts>/iter_1/<CR:_VDB-K:_RERANKER-K:_…>/{profiling,aiperf_rag_on}/ per point, plus aggregate report.md / results.json / results.csv at the run root.

Run unit tests:

uv sync --project scripts/rag-perf --extra dev   # one-time, installs pytest-asyncio
uv run --project scripts/rag-perf python -m pytest tests/unit/test_rag_perf/

Limitations

  • The CLI is config-only: author or copy YAML to vary a parameter.
  • load.concurrency / rag.vdb_top_k / rag.reranker_top_k accept int | list[int]; the validator requires unique list values because each value names a unique point dir.
  • input.file and input.synthetic follow an XOR rule — both set fails validation. When neither is set, synthetic auto-fills with defaults so a bare config still validates.
  • File-based input format is inferred from extension only (.jsonl or .csv); other extensions are rejected.
  • Synthetic generation streams each query to disk as it completes (failure-resilient) but fails fast on the first LLM error — partial JSONL is preserved. Re-run after fixing the endpoint.
  • Reasoning models (Nemotron Omni, Qwen-Reasoning) require synthetic.disable_thinking: true (the default). Without it the model exhausts the token budget on chain-of-thought and content returns empty — the generator now raises with a clear message instead of substituting reasoning_content for the answer.
  • aiperf-specific knobs outside the YAML surface (request rate distribution, GPU telemetry config, etc.) require editing AiperfRunner._base_aiperf_cmd in scripts/rag-perf/rag_perf/runner.py.
  • Procedural detail lives under references/ to keep this file concise.

Troubleshooting

Error / signal Likely cause What to do
Configuration errors in <yaml>: • input — ... XOR rule Both input.file and input.synthetic set Pick one. The XOR validator runs at YAML load time.
input.file must end in .jsonl or .csv Extension other than .jsonl / .csv Rename or convert.
load.concurrency has duplicate values e.g. [2, 2, 4] Each concurrency maps to a unique point dir; dedupe.
warmup_requests must be >= 1 YAML had warmup_requests: 0 aiperf rejects warmup=0; minimum is 1.
LLM returned empty content (reasoning_content was populated — model exhausted its budget on chain-of-thought; raise min_query_tokens or set synthetic.disable_thinking=true). Reasoning model used CoT and ran out of tokens Set synthetic.disable_thinking: true (the default) or raise min_query_tokens.
✗ All N profiling requests failed across M point(s). + exit 1 Bad URL, server down, wrong collection Verify target.url, rag.collection_names (the <collection_name> placeholder will hit this).
Per-iteration ⚠ N profiling requests failed warning, run continues Some requests timed out / errored mid-run Check rag-server logs, raise target.timeout_s, drop concurrency.
RuntimeError: Random synthetic query generation failed at query N: ... LLM endpoint rejected a request mid-generation Partial JSONL is at synthetic.jsonl_output_path; fix endpoint and re-run with reduced num_queries, or point input.file at the partial file.
Citation count (mean): 0 and Citation relevance score: N/A for a non-empty deployment Collection mismatch between rag.collection_names and what's actually ingested Run curl -s http://<ingestor>:8082/v1/collections to list real collections.
Tests error with ModuleNotFoundError: No module named 'pytest_asyncio' Dev extras missing uv sync --project scripts/rag-perf --extra dev.
CI: ModuleNotFoundError: No module named 'ruamel' from tests/unit/test_rag_perf/ rag-perf package missing from CI venv Add uv pip install -e ./scripts/rag-perf after the top-level install in the unit-tests job.

Gotchas

  • Run from repo root. Preset configs reference scripts/rag-perf/examples/queries.jsonl and scripts/rag-perf/prompts/default_prompts.yaml with repo-root-relative paths. Running from inside scripts/rag-perf/ will fail those file lookups.
  • CLI is config-only. Edit the YAML or copy a preset for URL, concurrency, collection, and similar fields.
  • Always edit rag.collection_names before the first run. The presets ship with ["<collection_name>"] as a deliberate placeholder. Validation passes, retrieval fails silently for every request — manifests as Citation count (mean): 0 everywhere.
  • load.concurrency_list, rag.vdb_top_k_list, rag.reranker_top_k_list are read-only properties that normalise scalar-or-list to a list. Use them when reasoning about the grid; the underlying YAML field is whatever the user wrote.
  • aiperf.enabled: false changes filenames. The top-level outputs become profile_report.md / profile_results.json / profile_results.csv. The aggregate sweep table also suppresses load-test rows and the "Optimal throughput" footer.
  • Resolved-config dump is verbose (50+ lines) — expected. It's what makes terminal output a self-contained reproducer; don't filter it out in scripts.
  • The aiperf shell command is logged before each subprocess. Look for \n $ python -m aiperf profile -m ... --endpoint-type nvidia_rag ... in stdout — copy-paste runnable for reproducing a single point outside rag-perf.
  • --endpoint-type nvidia_rag comes from the bundled plugin at scripts/rag-perf/rag_perf/plugin/nvidia_rag.py. It teaches aiperf about the RAG /v1/generate request shape and parses citations + per-stage metrics out of the SSE stream. If aiperf can't resolve nvidia_rag, rag-perf needs editable installation in the venv — re-run uv sync --project scripts/rag-perf (or uv pip install -e ./scripts/rag-perf).
  • Sweep-mode point-name collision. When two points differ only in concurrency (e.g. [1, 4] × single vdb_top_k), the dir name encodes everything: CR:1_ISL:50_OSL:512_VDB-K:20_RERANKER-K:4_Model:.... Cluster / GPU / experiment_name (output.cluster, output.gpu, output.experiment_name) are appended too — useful for diff-friendly artifact paths across machines.
  • load.iterations > 1 repeats the entire grid. Each repetition writes to its own iter_<i>/. Aggregate CSV row count = n_points × iterations.

Source of truth

Piece Location
Driver scripts/rag-perf/rag_perf/cli.py (main is the single Click command)
Schema scripts/rag-perf/rag_perf/config.py (RunConfig and sub-models)
Orchestrator scripts/rag-perf/rag_perf/runner.py (BenchmarkRunner.run, RagProfiler, AiperfRunner)
aiperf plugin scripts/rag-perf/rag_perf/plugin/nvidia_rag.py
User-facing doc docs/performance-benchmarking.md
Presets scripts/rag-perf/configs/{quick_profile,single_run,sweep}.yaml
Sample queries scripts/rag-perf/examples/queries.jsonl
Synthetic prompts scripts/rag-perf/prompts/default_prompts.yaml
Config schema details references/config-schema.md
Synthetic-query generation references/synthetic-generation.md
Output layout & metric semantics references/output-and-analysis.md

Agent playbook

  1. Sync deps: uv sync --project scripts/rag-perf (one-time per checkout).
  2. Pick & customise a preset: copy scripts/rag-perf/configs/<preset>.yaml if you want a variant; always set rag.collection_names to a real collection.
  3. Run: uv run --project scripts/rag-perf rag-perf -c <config> from repo root.
  4. Read the per-point + aggregate tables on stdout. Bottleneck inference is in the per-point profiling section; comparison across points is the final aggregate table.
  5. Parse artifacts under output.dir/run_<ts>/ — see references/output-and-analysis.md. For multi-point runs, results.csv has one row per (point × iteration).
  6. Summarise for the user using the playbook in references/output-and-analysis.md#summarising-results-to-the-user — headline table, scaling-efficiency math for sweeps, mandatory flags for zero citations / non-zero errors / suspect llm_ttft_ms / low sample size, and a concrete next-experiment YAML.
  7. Tune retrieval / reranker: flip to quick_profile.yaml or aiperf.enabled: false for fast iteration, then return to single_run.yaml / sweep.yaml when characterising under load.
  8. Triage failures: see Troubleshooting above and references/output-and-analysis.md for empty-citation / bottleneck=N/A patterns.

<!-- MCP:START -->

<!-- PORTABILITY:START -->

Cross-Client Portability

This skill is written to stay usable across GitHub Copilot, Claude Code, and Codex.

  • GitHub Copilot: keep the folder in a Copilot-visible skill path or wrap the workflow in project instructions when folder discovery is unavailable.
  • Claude Code: keep the folder in a local skills directory or a compatible plugin source.
  • Codex: install or sync the folder into $CODEX_HOME/skills/rag-perf and restart Codex after major changes.

<!-- PORTABILITY:END -->

MCP Availability And Fallback

Preferred MCP Server: None required

  • Fallback prompt: "Use the RAG-Perf — config-driven perf benchmark CLI skill without MCP. Rely on its local instructions, bundled resources, standard shell or editor tools, and direct verification. Show the evidence used before concluding."
  • Do not claim an MCP operation was used when the active host does not expose it.
  • Treat local files, tests, rendered outputs, logs, or screenshots as the fallback evidence path.

<!-- MCP:END -->

Anti-Patterns

  • Optimizing before recording a baseline: Without a starting point, there is no trustworthy performance story.
  • Using toy traffic to justify production tuning: Tiny prompts or empty corpora hide the real bottleneck.
  • Accepting lower answer quality as an untracked side effect of a latency win.

Verification Protocol

Before claiming "skill applied successfully":

  1. Pass/fail: The performance investigation starts from a named baseline and a concrete bottleneck hypothesis.
  2. Pass/fail: The workload shape, corpus size, and concurrency assumptions match the path being optimized.
  3. Pass/fail: Any latency or throughput claim is paired with a correctness or quality guardrail.
  4. Pressure-test scenario: Re-run the workflow on a retrieval stack that speeds up only because caching masked a stale index.
  5. Success metric: The user gets a reproducible benchmark path and a tuning change that improves the intended metric without hidden regressions.

Related Skills

  • devops-tooling: Use it when the performance work also needs container, CI, or infrastructure diagnostics.
  • cloud-design-patterns: Use it when bottlenecks reveal larger scaling or architecture tradeoffs.
  • documentation-verification: Use it when benchmark claims must be published or preserved in repo docs.

04 同仓库更多 Skills

bg-szy
bg-szy/TOP-SKILLS

serpmaven

Scrape Google search results (SERP) to JSON with SERP Maven (serpmaven) - a real Chrome browser on a residential proxy with sticky sessions. Run it as a CLI, a Python library, a local HTTP search API (`serve`), an MCP server or a Docker image; set the proxy login, workers, pages and cost tracking. Use when the user wants Google results, rank tracking, a self-hosted SERP API, or live Google search for an AI agent.

★ 6
bg-szy
bg-szy/TOP-SKILLS

revnu-landing-page-audit

Find out why a landing page isn't turning visitors into signups, demos or sales. Audits one page against what the ad, post or search that sends people there promised, then checks the first screen, proof, call to action, friction and mobile. Returns at most 7 prioritized fixes, plus what is fine and what could not be checked; rewrites specific lines and fixes the page in the repo when asked. Never promises a conversion lift. Works without a Revnu account. Use when the user says or asks: "audit my landing page", "why isn't this page converting", "review my homepage", "my ads get clicks but no signups", "people bounce from the pricing page", "CRO review", "improve conversion rate", "is my hero section clear", "roast my landing page", "rewrite my headline", "what's wrong with this page", "landing page feedback", "make the signup page convert better". Not for SEO or AI-search audits, page speed engineering on its own, or building a new page from scratch.

★ 6
bg-szy
bg-szy/TOP-SKILLS

revnu-growth-context

Write down once who buys, what they're stuck on, what it costs, what proof you have and what you've already tried, so every growth task starts from facts instead of a questionnaire. Creates or updates .agents/revnu-growth.md from the repo, the live site and the user, and imports an existing product-marketing.md. Use when the user says "set up our marketing context", "who is our ideal customer", "write our ICP", "describe the product for marketing", "positioning doc", "what's our brand voice", "remember this about our customers", "log what we tried", "what have we learned from marketing", "we changed pricing, update the context", "update our growth notes", "summarize the business for the agent", "import my product-marketing.md", "start a growth doc", or before a first marketing, SEO, outreach or launch task when no context file exists. Works without a Revnu account.

★ 6
bg-szy
bg-szy/TOP-SKILLS

revnu-get-listed

Get onto the third-party pages AI answers and search cite when buyers ask for the best tool in your category: "best X" roundups, directories, review sites, GitHub awesome lists, community threads and comparison pages. Finds which ones are cited, ranks them by fit and effort, and drafts each submission, PR or pitch by the target's own rules. Use when the user says "get us listed", "which directories should we submit to", "how do we get into best-of lists", "why does ChatGPT recommend competitors and not us", "get mentioned in AI answers", "submit to awesome lists", "get on G2", "where are our competitors listed", "pitch listicle authors", "find backlink opportunities", "launch directories", "get into roundups", "Product Hunt alternatives", "where should we be listed", "get cited by Perplexity". Never submits, pays or posts without a yes. Works without a Revnu account.

★ 6
bg-szy
bg-szy/TOP-SKILLS

revnu-ai-readiness-audit

Audit whether your website gives ChatGPT, Perplexity, Claude, Gemini and Google's AI Overviews something worth quoting, and fix the pages that don't. Use when the user says or asks "is my site ready for AI search", "audit my site for AI answers", "AEO audit", "GEO audit", "why doesn't ChatGPT quote our pricing", "make our pages citable by AI", "do I need llms.txt", "does schema help with ChatGPT", "check our structured data", "is our content visible without JavaScript", "AI SEO audit", "optimize our homepage for AI answers", "LLM SEO check", "why does AI get our product wrong", "fix our site for AI Overviews". Checks fetchability and indexability, answers in the raw HTML, entity facts, schema (including JS-injected markup), headings, dates and llms.txt, then gives at most 10 prioritised fixes and edits the repo when asked. Works without a Revnu account.

★ 6
bg-szy
bg-szy/TOP-SKILLS

proxy-troubleshooting

Diagnose residential proxy failures, with NodeMaven's gateway as the reference - 407 with correct credentials, 406, 410, 500, a connection that hangs, ERR_TUNNEL_CONNECTION_FAILED in a browser, Playwright or Puppeteer ignoring proxy credentials, a sticky session that changes IP, geo-targeting that is silently ignored, passwords with special characters, and unexpected traffic from headless Chrome. Use when a proxy connection fails, returns the wrong location, or costs more traffic than expected.

★ 6