[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6PCGDXBKD0QJCCN4FE3WH6J":3,"skill-same:01M6PCGDXBKD0QJCCN4FE3WH6J":77},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":38,"linkOnly":39,"markdownContent":40,"markdownBody":41,"frontMatter":42,"refName":27,"commitSha":52,"skillDirectory":53,"forksCount":13,"repositoryHtmlUrl":54,"repositoryHomepageUrl":6,"ownerHtmlUrl":55,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":56,"install":57,"resources":70},"01M6PCGDXBKD0QJCCN4FE3WH6J","rag-observability-evals",null,"Monitor and evaluate RAG systems with retrieval quality metrics, groundedness checks, hallucination detection, and continuous regression testing.","duc-duy-1612\u002FOnline-Delivery-Systems-Web","duc-duy-1612","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F192231384?v=4","01M6NSDKQQDKVTHZ8DZMZ1VYY6","01M6NSDKW8DDVTQHBT1HBDDJPT",0,"MIT","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Frag-observability-evals\u002FSKILL.md",[],[18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37],"rag observability evals","rag","observability","evals","regression","relevant","evaluate rag","scripts","critical","main","community","only","testing","access","devops-security","Hallucination Detection","kubectl","ci","git","security","2026-11-10 10:12:30",false,"---\nname: rag-observability-evals\ndescription: Monitor and evaluate RAG systems with retrieval quality metrics, groundedness\n  checks, hallucination detection, and continuous regression testing.\ncategory: devops\nrisk: critical\nsource: https:\u002F\u002Fgithub.com\u002FBagelHole\u002FDevOps-Security-Agent-Skills\nsource_repo: BagelHole\u002FDevOps-Security-Agent-Skills\nsource_type: community\ndate_added: '2026-09-20'\nlicense: MIT\nlicense_source: https:\u002F\u002Fgithub.com\u002FBagelHole\u002FDevOps-Security-Agent-Skills\u002Fblob\u002Fmain\u002FLICENSE\ncompatibility: Requires the relevant platform CLIs (kubectl, helm, terraform, git,\n  CI runners) and authorized access to the target environment. Docs-only; helper scripts\n  and templates not bundled.\nmetadata:\n  author: devops-skills\n  version: '1.0'\n---\n\n# RAG Observability and Evaluations\n\nRun retrieval-augmented generation like a measurable production system, not a black box.\n\n## Prerequisites\n\n- RAG pipeline with instrumented retrieval and generation stages\n- Python 3.10+ with evaluation libraries (ragas, langchain, openai)\n- Prometheus endpoint for custom metrics export\n- Benchmark dataset with gold-standard question\u002Fanswer\u002Fsource triples\n- OpenTelemetry SDK integrated into the RAG service\n\n## What to Measure\n\n### Retrieval Quality\n- Recall@k and MRR for top-k chunks\n- Citation coverage and source freshness\n- Embedding drift and index staleness\n\n### Generation Quality\n- Groundedness score (answer supported by retrieved context)\n- Hallucination rate by route\u002Fuse case\n- Instruction adherence and format validity\n\n### Reliability and Cost\n- p50\u002Fp95 latency split by retrieval vs generation\n- Token usage per stage\n- Cache hit rate and cost per successful answer\n\n## RAGAS Evaluation Script\n\n```python\n# rag_eval.py\n\"\"\"Evaluate RAG pipeline quality using RAGAS metrics.\"\"\"\nfrom ragas import evaluate\nfrom ragas.metrics import (\n    faithfulness,\n    answer_relevancy,\n    context_precision,\n    context_recall,\n    context_entity_recall,\n    answer_similarity,\n)\nfrom datasets import Dataset\nimport json\nimport sys\n\ndef load_eval_dataset(path: str) -> Dataset:\n    \"\"\"Load evaluation dataset with required columns.\"\"\"\n    with open(path) as f:\n        data = json.load(f)\n\n    return Dataset.from_dict({\n        \"question\": [d[\"question\"] for d in data],\n        \"answer\": [d[\"generated_answer\"] for d in data],\n        \"contexts\": [d[\"retrieved_contexts\"] for d in data],\n        \"ground_truth\": [d[\"reference_answer\"] for d in data],\n    })\n\ndef run_evaluation(dataset_path: str, output_path: str):\n    \"\"\"Run full RAGAS evaluation suite.\"\"\"\n    dataset = load_eval_dataset(dataset_path)\n\n    metrics = [\n        faithfulness,\n        answer_relevancy,\n        context_precision,\n        context_recall,\n        context_entity_recall,\n        answer_similarity,\n    ]\n\n    results = evaluate(dataset, metrics=metrics)\n\n    # Print summary\n    print(\"=== RAG Evaluation Results ===\")\n    for metric_name, score in results.items():\n        print(f\"  {metric_name}: {score:.4f}\")\n\n    # Save detailed results\n    with open(output_path, \"w\") as f:\n        json.dump({\n            \"summary\": {k: float(v) for k, v in results.items()},\n            \"dataset_size\": len(dataset),\n        }, f, indent=2)\n\n    return results\n\nif __name__ == \"__main__\":\n    run_evaluation(sys.argv[1], sys.argv[2])\n```\n\n## Groundedness Scoring\n\n```python\n# groundedness.py\n\"\"\"Score whether generated answers are grounded in retrieved context.\"\"\"\nfrom openai import OpenAI\nimport json\nfrom typing import List\n\nclient = OpenAI()\n\nGROUNDEDNESS_PROMPT = \"\"\"You are evaluating whether an AI answer is fully grounded\nin the provided context documents. Score each claim in the answer.\n\nContext documents:\n{contexts}\n\nAnswer to evaluate:\n{answer}\n\nFor each distinct claim in the answer, determine:\n1. SUPPORTED - the claim is directly supported by the context\n2. PARTIALLY_SUPPORTED - the claim is partially supported\n3. NOT_SUPPORTED - the claim has no support in the context\n\nReturn JSON:\n{{\n  \"claims\": [\n    {{\"claim\": \"...\", \"verdict\": \"SUPPORTED|PARTIALLY_SUPPORTED|NOT_SUPPORTED\", \"evidence\": \"...\"}}\n  ],\n  \"groundedness_score\": \u003Cfloat 0-1>,\n  \"unsupported_claims\": [\"...\"]\n}}\n\"\"\"\n\ndef score_groundedness(answer: str, contexts: List[str]) -> dict:\n    \"\"\"Score groundedness of a single answer against its contexts.\"\"\"\n    context_text = \"\\n---\\n\".join(\n        f\"[Document {i+1}]: {c}\" for i, c in enumerate(contexts)\n    )\n\n    response = client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=[{\n            \"role\": \"user\",\n            \"content\": GROUNDEDNESS_PROMPT.format(\n                contexts=context_text, answer=answer\n            ),\n        }],\n        response_format={\"type\": \"json_object\"},\n        temperature=0,\n    )\n\n    return json.loads(response.choices[0].message.content)\n\ndef batch_groundedness(eval_data: list) -> dict:\n    \"\"\"Score groundedness for a batch of QA pairs.\"\"\"\n    scores = []\n    unsupported_count = 0\n    total_claims = 0\n\n    for item in eval_data:\n        result = score_groundedness(\n            item[\"generated_answer\"],\n            item[\"retrieved_contexts\"],\n        )\n        scores.append(result[\"groundedness_score\"])\n        unsupported_count += len(result[\"unsupported_claims\"])\n        total_claims += len(result[\"claims\"])\n\n    avg_score = sum(scores) \u002F len(scores) if scores else 0\n    return {\n        \"average_groundedness\": avg_score,\n        \"total_claims\": total_claims,\n        \"unsupported_claims\": unsupported_count,\n        \"unsupported_rate\": unsupported_count \u002F total_claims if total_claims else 0,\n        \"sample_count\": len(eval_data),\n    }\n```\n\n## Retrieval Quality Metrics\n\n```python\n# retrieval_metrics.py\n\"\"\"Compute retrieval quality metrics for RAG evaluation.\"\"\"\nfrom typing import List, Set\nimport numpy as np\n\ndef recall_at_k(\n    retrieved_ids: List[str],\n    relevant_ids: Set[str],\n    k: int\n) -> float:\n    \"\"\"Compute Recall@K for a single query.\"\"\"\n    top_k = set(retrieved_ids[:k])\n    if not relevant_ids:\n        return 0.0\n    return len(top_k & relevant_ids) \u002F len(relevant_ids)\n\ndef mrr(\n    retrieved_ids: List[str],\n    relevant_ids: Set[str]\n) -> float:\n    \"\"\"Compute Mean Reciprocal Rank for a single query.\"\"\"\n    for i, doc_id in enumerate(retrieved_ids):\n        if doc_id in relevant_ids:\n            return 1.0 \u002F (i + 1)\n    return 0.0\n\ndef ndcg_at_k(\n    retrieved_ids: List[str],\n    relevant_ids: Set[str],\n    k: int\n) -> float:\n    \"\"\"Compute NDCG@K for a single query.\"\"\"\n    dcg = 0.0\n    for i, doc_id in enumerate(retrieved_ids[:k]):\n        if doc_id in relevant_ids:\n            dcg += 1.0 \u002F np.log2(i + 2)\n\n    ideal_dcg = sum(1.0 \u002F np.log2(i + 2) for i in range(min(len(relevant_ids), k)))\n    return dcg \u002F ideal_dcg if ideal_dcg > 0 else 0.0\n\ndef compute_retrieval_metrics(\n    queries: list,\n    k_values: list = [1, 3, 5, 10]\n) -> dict:\n    \"\"\"Compute aggregate retrieval metrics across all queries.\"\"\"\n    results = {}\n    for k in k_values:\n        recalls = [\n            recall_at_k(q[\"retrieved_ids\"], set(q[\"relevant_ids\"]), k)\n            for q in queries\n        ]\n        mrrs = [mrr(q[\"retrieved_ids\"], set(q[\"relevant_ids\"])) for q in queries]\n        ndcgs = [\n            ndcg_at_k(q[\"retrieved_ids\"], set(q[\"relevant_ids\"]), k)\n            for q in queries\n        ]\n        results[f\"recall@{k}\"] = np.mean(recalls)\n        results[f\"ndcg@{k}\"] = np.mean(ndcgs)\n\n    results[\"mrr\"] = np.mean(mrrs)\n    return results\n```\n\n## Prometheus Metrics Export\n\n```python\n# rag_metrics_exporter.py\n\"\"\"Export RAG quality metrics to Prometheus.\"\"\"\nfrom prometheus_client import Histogram, Counter, Gauge, start_http_server\nimport time\n\n# Latency histograms by stage\nRETRIEVAL_LATENCY = Histogram(\n    \"rag_retrieval_duration_seconds\",\n    \"Time spent in retrieval stage\",\n    [\"index_name\", \"retriever_type\"],\n    buckets=[0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0],\n)\n\nGENERATION_LATENCY = Histogram(\n    \"rag_generation_duration_seconds\",\n    \"Time spent in generation stage\",\n    [\"model\", \"route\"],\n    buckets=[0.5, 1.0, 2.0, 5.0, 10.0, 30.0],\n)\n\nRERANKING_LATENCY = Histogram(\n    \"rag_reranking_duration_seconds\",\n    \"Time spent in reranking stage\",\n    [\"reranker_model\"],\n    buckets=[0.05, 0.1, 0.25, 0.5, 1.0],\n)\n\n# Quality gauges (updated from offline evals)\nGROUNDEDNESS_SCORE = Gauge(\n    \"rag_groundedness_score\",\n    \"Latest groundedness evaluation score\",\n    [\"route\", \"model\"],\n)\n\nFAITHFULNESS_SCORE = Gauge(\n    \"rag_faithfulness_score\",\n    \"Latest faithfulness evaluation score\",\n    [\"route\", \"model\"],\n)\n\nCONTEXT_PRECISION = Gauge(\n    \"rag_context_precision_score\",\n    \"Latest context precision score\",\n    [\"route\", \"index_name\"],\n)\n\nRECALL_AT_K = Gauge(\n    \"rag_recall_at_k\",\n    \"Recall@K for retrieval\",\n    [\"k\", \"index_name\"],\n)\n\n# Operational counters\nREQUESTS_TOTAL = Counter(\n    \"rag_requests_total\",\n    \"Total RAG requests\",\n    [\"route\", \"status\"],\n)\n\nHALLUCINATION_DETECTED = Counter(\n    \"rag_hallucination_detected_total\",\n    \"Detected hallucinations\",\n    [\"route\", \"severity\"],\n)\n\nFALLBACK_TRIGGERED = Counter(\n    \"rag_fallback_triggered_total\",\n    \"Times RAG fell back to abstain\u002Fdefault\",\n    [\"route\", \"reason\"],\n)\n\nTOKENS_USED = Counter(\n    \"rag_tokens_used_total\",\n    \"Tokens consumed by stage\",\n    [\"stage\", \"model\"],\n)\n\nCACHE_HITS = Counter(\n    \"rag_cache_hits_total\",\n    \"Semantic cache hits\",\n    [\"cache_type\"],\n)\n\n# Index health\nINDEX_STALENESS_SECONDS = Gauge(\n    \"rag_index_staleness_seconds\",\n    \"Seconds since last index update\",\n    [\"index_name\"],\n)\n\nINDEX_DOCUMENT_COUNT = Gauge(\n    \"rag_index_document_count\",\n    \"Number of documents in index\",\n    [\"index_name\"],\n)\n\ndef start_metrics_server(port: int = 9090):\n    \"\"\"Start Prometheus metrics HTTP server.\"\"\"\n    start_http_server(port)\n    print(f\"RAG metrics server running on :{port}\u002Fmetrics\")\n```\n\n## Evaluation Pipeline\n\n1. Curate a benchmark set with gold answers and source docs.\n2. Run nightly offline evals for every retriever\u002Fmodel configuration.\n3. Execute online shadow evals on sampled production traffic.\n4. Gate releases on minimum quality + safety + latency thresholds.\n\n```yaml\n# eval-pipeline-cron.yaml\napiVersion: batch\u002Fv1\nkind: CronJob\nmetadata:\n  name: rag-nightly-eval\n  namespace: ai-evals\nspec:\n  schedule: \"0 2 * * *\"\n  jobTemplate:\n    spec:\n      template:\n        spec:\n          containers:\n            - name: eval-runner\n              image: registry.internal\u002Frag-eval:latest\n              command:\n                - python\n                - -m\n                - rag_eval\n                - --dataset=\u002Fdata\u002Fbenchmark_v3.json\n                - --output=\u002Fresults\u002Fnightly-$(date +%Y%m%d).json\n                - --push-metrics\n                - --fail-on-regression\n              env:\n                - name: PROMETHEUS_PUSHGATEWAY\n                  value: \"http:\u002F\u002Fpushgateway:9091\"\n                - name: MLFLOW_TRACKING_URI\n                  value: \"http:\u002F\u002Fmlflow:5000\"\n              volumeMounts:\n                - name: eval-data\n                  mountPath: \u002Fdata\n                - name: results\n                  mountPath: \u002Fresults\n          volumes:\n            - name: eval-data\n              persistentVolumeClaim:\n                claimName: eval-benchmark-data\n            - name: results\n              persistentVolumeClaim:\n                claimName: eval-results\n          restartPolicy: OnFailure\n```\n\n\n## Contents\n\n- [Alerting Strategy](references\u002Fdetails.md)\n- [Practical Guardrails](references\u002Fdetails.md)\n- [Incident Triage Checklist](references\u002Fdetails.md)\n- [Troubleshooting](references\u002Fdetails.md)\n- [Related Skills](references\u002Fdetails.md)\n\n## When to Use This Skill\n\n- Deploying a RAG system to production and need quality monitoring\n- Setting up automated evaluation pipelines for retrieval and generation\n- Debugging hallucination or relevance regressions\n- Building dashboards for RAG-specific golden signals\n- Establishing quality gates for RAG pipeline changes\n\n## Limitations\n\n- Guidance executes against real environments: confirm target, blast radius, and rollback plan before applying anything.\n- Never deploy to production without explicit approval. Docs-only import: upstream scripts and templates not bundled.\n\n### Example\n\n```bash\ngit status && git diff --stat\nkubectl diff -f manifest.yaml\n```\n\n> Adapted from [BagelHole\u002FDevOps-Security-Agent-Skills](https:\u002F\u002Fgithub.com\u002FBagelHole\u002FDevOps-Security-Agent-Skills) (MIT); frontmatter, When to Use\u002FLimitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled.\n","# RAG Observability and Evaluations\n\nRun retrieval-augmented generation like a measurable production system, not a black box.\n\n## Prerequisites\n\n- RAG pipeline with instrumented retrieval and generation stages\n- Python 3.10+ with evaluation libraries (ragas, langchain, openai)\n- Prometheus endpoint for custom metrics export\n- Benchmark dataset with gold-standard question\u002Fanswer\u002Fsource triples\n- OpenTelemetry SDK integrated into the RAG service\n\n## What to Measure\n\n### Retrieval Quality\n- Recall@k and MRR for top-k chunks\n- Citation coverage and source freshness\n- Embedding drift and index staleness\n\n### Generation Quality\n- Groundedness score (answer supported by retrieved context)\n- Hallucination rate by route\u002Fuse case\n- Instruction adherence and format validity\n\n### Reliability and Cost\n- p50\u002Fp95 latency split by retrieval vs generation\n- Token usage per stage\n- Cache hit rate and cost per successful answer\n\n## RAGAS Evaluation Script\n\n```python\n# rag_eval.py\n\"\"\"Evaluate RAG pipeline quality using RAGAS metrics.\"\"\"\nfrom ragas import evaluate\nfrom ragas.metrics import (\n    faithfulness,\n    answer_relevancy,\n    context_precision,\n    context_recall,\n    context_entity_recall,\n    answer_similarity,\n)\nfrom datasets import Dataset\nimport json\nimport sys\n\ndef load_eval_dataset(path: str) -> Dataset:\n    \"\"\"Load evaluation dataset with required columns.\"\"\"\n    with open(path) as f:\n        data = json.load(f)\n\n    return Dataset.from_dict({\n        \"question\": [d[\"question\"] for d in data],\n        \"answer\": [d[\"generated_answer\"] for d in data],\n        \"contexts\": [d[\"retrieved_contexts\"] for d in data],\n        \"ground_truth\": [d[\"reference_answer\"] for d in data],\n    })\n\ndef run_evaluation(dataset_path: str, output_path: str):\n    \"\"\"Run full RAGAS evaluation suite.\"\"\"\n    dataset = load_eval_dataset(dataset_path)\n\n    metrics = [\n        faithfulness,\n        answer_relevancy,\n        context_precision,\n        context_recall,\n        context_entity_recall,\n        answer_similarity,\n    ]\n\n    results = evaluate(dataset, metrics=metrics)\n\n    # Print summary\n    print(\"=== RAG Evaluation Results ===\")\n    for metric_name, score in results.items():\n        print(f\"  {metric_name}: {score:.4f}\")\n\n    # Save detailed results\n    with open(output_path, \"w\") as f:\n        json.dump({\n            \"summary\": {k: float(v) for k, v in results.items()},\n            \"dataset_size\": len(dataset),\n        }, f, indent=2)\n\n    return results\n\nif __name__ == \"__main__\":\n    run_evaluation(sys.argv[1], sys.argv[2])\n```\n\n## Groundedness Scoring\n\n```python\n# groundedness.py\n\"\"\"Score whether generated answers are grounded in retrieved context.\"\"\"\nfrom openai import OpenAI\nimport json\nfrom typing import List\n\nclient = OpenAI()\n\nGROUNDEDNESS_PROMPT = \"\"\"You are evaluating whether an AI answer is fully grounded\nin the provided context documents. Score each claim in the answer.\n\nContext documents:\n{contexts}\n\nAnswer to evaluate:\n{answer}\n\nFor each distinct claim in the answer, determine:\n1. SUPPORTED - the claim is directly supported by the context\n2. PARTIALLY_SUPPORTED - the claim is partially supported\n3. NOT_SUPPORTED - the claim has no support in the context\n\nReturn JSON:\n{{\n  \"claims\": [\n    {{\"claim\": \"...\", \"verdict\": \"SUPPORTED|PARTIALLY_SUPPORTED|NOT_SUPPORTED\", \"evidence\": \"...\"}}\n  ],\n  \"groundedness_score\": \u003Cfloat 0-1>,\n  \"unsupported_claims\": [\"...\"]\n}}\n\"\"\"\n\ndef score_groundedness(answer: str, contexts: List[str]) -> dict:\n    \"\"\"Score groundedness of a single answer against its contexts.\"\"\"\n    context_text = \"\\n---\\n\".join(\n        f\"[Document {i+1}]: {c}\" for i, c in enumerate(contexts)\n    )\n\n    response = client.chat.completions.create(\n        model=\"gpt-4o\",\n        messages=[{\n            \"role\": \"user\",\n            \"content\": GROUNDEDNESS_PROMPT.format(\n                contexts=context_text, answer=answer\n            ),\n        }],\n        response_format={\"type\": \"json_object\"},\n        temperature=0,\n    )\n\n    return json.loads(response.choices[0].message.content)\n\ndef batch_groundedness(eval_data: list) -> dict:\n    \"\"\"Score groundedness for a batch of QA pairs.\"\"\"\n    scores = []\n    unsupported_count = 0\n    total_claims = 0\n\n    for item in eval_data:\n        result = score_groundedness(\n            item[\"generated_answer\"],\n            item[\"retrieved_contexts\"],\n        )\n        scores.append(result[\"groundedness_score\"])\n        unsupported_count += len(result[\"unsupported_claims\"])\n        total_claims += len(result[\"claims\"])\n\n    avg_score = sum(scores) \u002F len(scores) if scores else 0\n    return {\n        \"average_groundedness\": avg_score,\n        \"total_claims\": total_claims,\n        \"unsupported_claims\": unsupported_count,\n        \"unsupported_rate\": unsupported_count \u002F total_claims if total_claims else 0,\n        \"sample_count\": len(eval_data),\n    }\n```\n\n## Retrieval Quality Metrics\n\n```python\n# retrieval_metrics.py\n\"\"\"Compute retrieval quality metrics for RAG evaluation.\"\"\"\nfrom typing import List, Set\nimport numpy as np\n\ndef recall_at_k(\n    retrieved_ids: List[str],\n    relevant_ids: Set[str],\n    k: int\n) -> float:\n    \"\"\"Compute Recall@K for a single query.\"\"\"\n    top_k = set(retrieved_ids[:k])\n    if not relevant_ids:\n        return 0.0\n    return len(top_k & relevant_ids) \u002F len(relevant_ids)\n\ndef mrr(\n    retrieved_ids: List[str],\n    relevant_ids: Set[str]\n) -> float:\n    \"\"\"Compute Mean Reciprocal Rank for a single query.\"\"\"\n    for i, doc_id in enumerate(retrieved_ids):\n        if doc_id in relevant_ids:\n            return 1.0 \u002F (i + 1)\n    return 0.0\n\ndef ndcg_at_k(\n    retrieved_ids: List[str],\n    relevant_ids: Set[str],\n    k: int\n) -> float:\n    \"\"\"Compute NDCG@K for a single query.\"\"\"\n    dcg = 0.0\n    for i, doc_id in enumerate(retrieved_ids[:k]):\n        if doc_id in relevant_ids:\n            dcg += 1.0 \u002F np.log2(i + 2)\n\n    ideal_dcg = sum(1.0 \u002F np.log2(i + 2) for i in range(min(len(relevant_ids), k)))\n    return dcg \u002F ideal_dcg if ideal_dcg > 0 else 0.0\n\ndef compute_retrieval_metrics(\n    queries: list,\n    k_values: list = [1, 3, 5, 10]\n) -> dict:\n    \"\"\"Compute aggregate retrieval metrics across all queries.\"\"\"\n    results = {}\n    for k in k_values:\n        recalls = [\n            recall_at_k(q[\"retrieved_ids\"], set(q[\"relevant_ids\"]), k)\n            for q in queries\n        ]\n        mrrs = [mrr(q[\"retrieved_ids\"], set(q[\"relevant_ids\"])) for q in queries]\n        ndcgs = [\n            ndcg_at_k(q[\"retrieved_ids\"], set(q[\"relevant_ids\"]), k)\n            for q in queries\n        ]\n        results[f\"recall@{k}\"] = np.mean(recalls)\n        results[f\"ndcg@{k}\"] = np.mean(ndcgs)\n\n    results[\"mrr\"] = np.mean(mrrs)\n    return results\n```\n\n## Prometheus Metrics Export\n\n```python\n# rag_metrics_exporter.py\n\"\"\"Export RAG quality metrics to Prometheus.\"\"\"\nfrom prometheus_client import Histogram, Counter, Gauge, start_http_server\nimport time\n\n# Latency histograms by stage\nRETRIEVAL_LATENCY = Histogram(\n    \"rag_retrieval_duration_seconds\",\n    \"Time spent in retrieval stage\",\n    [\"index_name\", \"retriever_type\"],\n    buckets=[0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0],\n)\n\nGENERATION_LATENCY = Histogram(\n    \"rag_generation_duration_seconds\",\n    \"Time spent in generation stage\",\n    [\"model\", \"route\"],\n    buckets=[0.5, 1.0, 2.0, 5.0, 10.0, 30.0],\n)\n\nRERANKING_LATENCY = Histogram(\n    \"rag_reranking_duration_seconds\",\n    \"Time spent in reranking stage\",\n    [\"reranker_model\"],\n    buckets=[0.05, 0.1, 0.25, 0.5, 1.0],\n)\n\n# Quality gauges (updated from offline evals)\nGROUNDEDNESS_SCORE = Gauge(\n    \"rag_groundedness_score\",\n    \"Latest groundedness evaluation score\",\n    [\"route\", \"model\"],\n)\n\nFAITHFULNESS_SCORE = Gauge(\n    \"rag_faithfulness_score\",\n    \"Latest faithfulness evaluation score\",\n    [\"route\", \"model\"],\n)\n\nCONTEXT_PRECISION = Gauge(\n    \"rag_context_precision_score\",\n    \"Latest context precision score\",\n    [\"route\", \"index_name\"],\n)\n\nRECALL_AT_K = Gauge(\n    \"rag_recall_at_k\",\n    \"Recall@K for retrieval\",\n    [\"k\", \"index_name\"],\n)\n\n# Operational counters\nREQUESTS_TOTAL = Counter(\n    \"rag_requests_total\",\n    \"Total RAG requests\",\n    [\"route\", \"status\"],\n)\n\nHALLUCINATION_DETECTED = Counter(\n    \"rag_hallucination_detected_total\",\n    \"Detected hallucinations\",\n    [\"route\", \"severity\"],\n)\n\nFALLBACK_TRIGGERED = Counter(\n    \"rag_fallback_triggered_total\",\n    \"Times RAG fell back to abstain\u002Fdefault\",\n    [\"route\", \"reason\"],\n)\n\nTOKENS_USED = Counter(\n    \"rag_tokens_used_total\",\n    \"Tokens consumed by stage\",\n    [\"stage\", \"model\"],\n)\n\nCACHE_HITS = Counter(\n    \"rag_cache_hits_total\",\n    \"Semantic cache hits\",\n    [\"cache_type\"],\n)\n\n# Index health\nINDEX_STALENESS_SECONDS = Gauge(\n    \"rag_index_staleness_seconds\",\n    \"Seconds since last index update\",\n    [\"index_name\"],\n)\n\nINDEX_DOCUMENT_COUNT = Gauge(\n    \"rag_index_document_count\",\n    \"Number of documents in index\",\n    [\"index_name\"],\n)\n\ndef start_metrics_server(port: int = 9090):\n    \"\"\"Start Prometheus metrics HTTP server.\"\"\"\n    start_http_server(port)\n    print(f\"RAG metrics server running on :{port}\u002Fmetrics\")\n```\n\n## Evaluation Pipeline\n\n1. Curate a benchmark set with gold answers and source docs.\n2. Run nightly offline evals for every retriever\u002Fmodel configuration.\n3. Execute online shadow evals on sampled production traffic.\n4. Gate releases on minimum quality + safety + latency thresholds.\n\n```yaml\n# eval-pipeline-cron.yaml\napiVersion: batch\u002Fv1\nkind: CronJob\nmetadata:\n  name: rag-nightly-eval\n  namespace: ai-evals\nspec:\n  schedule: \"0 2 * * *\"\n  jobTemplate:\n    spec:\n      template:\n        spec:\n          containers:\n            - name: eval-runner\n              image: registry.internal\u002Frag-eval:latest\n              command:\n                - python\n                - -m\n                - rag_eval\n                - --dataset=\u002Fdata\u002Fbenchmark_v3.json\n                - --output=\u002Fresults\u002Fnightly-$(date +%Y%m%d).json\n                - --push-metrics\n                - --fail-on-regression\n              env:\n                - name: PROMETHEUS_PUSHGATEWAY\n                  value: \"http:\u002F\u002Fpushgateway:9091\"\n                - name: MLFLOW_TRACKING_URI\n                  value: \"http:\u002F\u002Fmlflow:5000\"\n              volumeMounts:\n                - name: eval-data\n                  mountPath: \u002Fdata\n                - name: results\n                  mountPath: \u002Fresults\n          volumes:\n            - name: eval-data\n              persistentVolumeClaim:\n                claimName: eval-benchmark-data\n            - name: results\n              persistentVolumeClaim:\n                claimName: eval-results\n          restartPolicy: OnFailure\n```\n\n\n## Contents\n\n- [Alerting Strategy](references\u002Fdetails.md)\n- [Practical Guardrails](references\u002Fdetails.md)\n- [Incident Triage Checklist](references\u002Fdetails.md)\n- [Troubleshooting](references\u002Fdetails.md)\n- [Related Skills](references\u002Fdetails.md)\n\n## When to Use This Skill\n\n- Deploying a RAG system to production and need quality monitoring\n- Setting up automated evaluation pipelines for retrieval and generation\n- Debugging hallucination or relevance regressions\n- Building dashboards for RAG-specific golden signals\n- Establishing quality gates for RAG pipeline changes\n\n## Limitations\n\n- Guidance executes against real environments: confirm target, blast radius, and rollback plan before applying anything.\n- Never deploy to production without explicit approval. Docs-only import: upstream scripts and templates not bundled.\n\n### Example\n\n```bash\ngit status && git diff --stat\nkubectl diff -f manifest.yaml\n```\n\n> Adapted from [BagelHole\u002FDevOps-Security-Agent-Skills](https:\u002F\u002Fgithub.com\u002FBagelHole\u002FDevOps-Security-Agent-Skills) (MIT); frontmatter, When to Use\u002FLimitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled.\n",{"name":5,"risk":26,"source":43,"license":14,"category":44,"metadata":45,"date_added":48,"description":7,"source_repo":49,"source_type":28,"compatibility":50,"license_source":51},"https:\u002F\u002Fgithub.com\u002FBagelHole\u002FDevOps-Security-Agent-Skills","devops",{"author":46,"version":47},"devops-skills","1.0","2026-09-20","BagelHole\u002FDevOps-Security-Agent-Skills","Requires the relevant platform CLIs (kubectl, helm, terraform, git, CI runners) and authorized access to the target environment. Docs-only; helper scripts and templates not bundled.","https:\u002F\u002Fgithub.com\u002FBagelHole\u002FDevOps-Security-Agent-Skills\u002Fblob\u002Fmain\u002FLICENSE","f3312ba749561b6159f477dbfc9313c875d40a56",".agents\u002Fskills_archive\u002Frag-observability-evals","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612","2026-10-25 23:50:58",{"source":8,"skillName":5,"commands":58,"prompt":68,"zipAvailable":69},[59,62,65],{"tool":60,"command":61},"npx","npx skills add duc-duy-1612\u002FOnline-Delivery-Systems-Web --skill rag-observability-evals",{"tool":63,"command":64},"bunx","bunx skills add duc-duy-1612\u002FOnline-Delivery-Systems-Web --skill rag-observability-evals",{"tool":66,"command":67},"pnpm","pnpm dlx skills add duc-duy-1612\u002FOnline-Delivery-Systems-Web --skill rag-observability-evals","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Frag-observability-evals\u002FSKILL.md",true,[71],{"path":72,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":75,"sourceHtmlUrl":76},"references\u002Fdetails.md","REFERENCE","参考文档",3301,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Frag-observability-evals\u002Freferences\u002Fdetails.md",[78,101,125,150,178,202],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":89,"publishedAt":38},"01M6Q0FA5FZ410Z33Z3TFQ4DBA","zero-trust","Implement zero-trust network architecture. Configure identity-based access, micro-segmentation, and continuous verification. Use when implementing modern security architectures.","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fzero-trust\u002FSKILL.md",[84],{"categoryNo":85,"categoryName":86,"taxonomyType":87,"taxonomyTypeDesc":88},"01JBSEEDCAT000000000000011","运维与部署","SCENARIO","使用场景",[90,91,92,93,94,95,80,96,32,31,97,29,28,27,98,26,99,25,23,100],"ZeRO","trust","zero trust","architecture","assessment","verification","segmentation","configure","active","scope","implementing",{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":38},"01M6Q0F1YZJW3KX5FBKEVZ8CV4","zcode-delegate","Delegate coding tasks to the Z.AI ZCode CLI only when the user explicitly requests it, while the orchestrator retains review and landing responsibility.","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fzcode-delegate\u002FSKILL.md",[],[108,109,110,111,112,113,114,115,116,117,118,119,120,29,121,28,122,123,26,124],"delegate","zcode","zcode delegate","review","cli","orchestration","tasks","run","Bash","model","node","read","orchestrator","files","desktop app","app","agent-orchestration",{"skillNo":126,"name":127,"nameZh":6,"description":128,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":129,"categories":130,"keywords":131,"publishedAt":38},"01M6Q0ETK41VGNMYEXW0T00AJN","youtube-transcript","Fetch YouTube transcripts through DeepAPI or local fallback tooling and save clean text output.","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fyoutube-transcript\u002FSKILL.md",[],[132,133,134,135,136,137,138,139,140,141,142,143,144,28,27,145,146,147,148,149],"transcript","youtube transcript","youtube","safe","github","codex","research","license","https","text","fetch","tooling","clean","local","output","deepapi","youtube transcripts","blob",{"skillNo":151,"name":152,"nameZh":6,"description":153,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":154,"categories":155,"keywords":167,"publishedAt":38},"01M6Q0EPPH2PVMDW40017F8WX0","youtube-transcript-skills","Fetch YouTube video transcripts, search videos\u002Fchannels, browse channels, and extract playlists via the getyoutubetranscript.com API - free tier, no card required.","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fyoutube-transcript-skills\u002FSKILL.md",[156,159,162],{"categoryNo":157,"categoryName":158,"taxonomyType":87,"taxonomyTypeDesc":88},"01JBSEEDCAT000000000000010","后端与接口",{"categoryNo":160,"categoryName":161,"taxonomyType":87,"taxonomyTypeDesc":88},"01JBSEEDCAT000000000000016","图像与多媒体",{"categoryNo":163,"categoryName":164,"taxonomyType":165,"taxonomyTypeDesc":166},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[132,133,134,168,135,169,136,137,139,170,171,172,173,142,174,28,27,175,176,177],"youtube transcript skills","gemini","search","api","Video","browse","api-integration","free","card","channels",{"skillNo":179,"name":180,"nameZh":6,"description":181,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":182,"categories":183,"keywords":188,"publishedAt":38},"01M6Q0EAB96PX83W5PTE8A0R8D","youtube-notetaker","Turn YouTube talks into local study notes with slides, transcripts, editable annotations, and a markdown-backed viewer.","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fyoutube-notetaker\u002FSKILL.md",[184,187],{"categoryNo":185,"categoryName":186,"taxonomyType":87,"taxonomyTypeDesc":88},"01JBSEEDCAT000000000000003","演示文稿",{"categoryNo":160,"categoryName":161,"taxonomyType":87,"taxonomyTypeDesc":88},[134,189,190,112,135,191,192,193,194,139,172,195,196,197,198,27,199,145,200,201],"youtube notetaker","notetaker","workflow","official","notes","code","markdown","slides","annotations","README","study","dair-academy","viewer",{"skillNo":203,"name":204,"nameZh":6,"description":205,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":206,"categories":207,"keywords":209,"publishedAt":38},"01M6Q0E55F0GW5MX48V8HXS8B5","youtube-full","Fetch YouTube transcripts, search videos, browse channels, and extract playlists via TranscriptAPI — no yt-dlp, no Google API key, works from any cloud server.","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fyoutube-full\u002FSKILL.md",[208],{"categoryNo":157,"categoryName":158,"taxonomyType":87,"taxonomyTypeDesc":88},[210,134,211,212,135,169,170,213,214,171,172,173,142,215,174,28,27,175,216,148],"youtube full","full","google","server","dlp","credits","automatic"]