[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M792B06D0D844VJWZHGYY75V":3,"skill-same:01M792B06D0D844VJWZHGYY75V":63},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":40,"commitSha":41,"skillDirectory":42,"forksCount":43,"repositoryHtmlUrl":44,"repositoryHomepageUrl":6,"ownerHtmlUrl":45,"ownerBlogUrl":46,"ownerDescription":47,"updatedAt":48,"install":49,"resources":62},"01M792B06D0D844VJWZHGYY75V","langchain-eval-harness",null,"Build reproducible evaluation pipelines for LangChain 1.0 chains and LangGraph 1.0\nagents — golden datasets, LangSmith evaluate(), ragas RAG metrics, deepeval\nLLM-as-judge, agent trajectory analysis, and CI gating on quality regressions.\nUse when setting up quality measurement for a new chain, diagnosing regression\nafter a model switch, or building an evaluation gate for a pull request.\nTrigger with \"langchain eval\", \"langsmith evaluate\", \"ragas\", \"llm-as-judge\",\n\"agent trajectory eval\", \"eval regression gate\".","jukeyman\u002Fjukeyman-skills","jukeyman","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F142960420?v=4","01M78FR4Y3DHMMCBCC0GXXNB3F","RICK JEFFERSON","01M78FR4Z8RVQRAD31Y5J7V6EB",1,"MIT","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fsaas-packs--langchain-py-pack--langchain-eval-harness\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: langchain-eval-harness\ndescription: |\n  Build reproducible evaluation pipelines for LangChain 1.0 chains and LangGraph 1.0\n  agents — golden datasets, LangSmith evaluate(), ragas RAG metrics, deepeval\n  LLM-as-judge, agent trajectory analysis, and CI gating on quality regressions.\n  Use when setting up quality measurement for a new chain, diagnosing regression\n  after a model switch, or building an evaluation gate for a pull request.\n  Trigger with \"langchain eval\", \"langsmith evaluate\", \"ragas\", \"llm-as-judge\",\n  \"agent trajectory eval\", \"eval regression gate\".\nallowed-tools: Read, Write, Edit, Bash(python:*), Bash(pip:*), Bash(pytest:*)\nversion: 2.0.0\nlicense: MIT\nauthor: Jeremy Longshore \u003Cjeremy@intentsolutions.io>\ntags: [saas, langchain, langgraph, python, langchain-1.0, evaluation, langsmith, ragas, deepeval, research]\ncompatible-with: claude-code, codex\n---\n\n# LangChain Eval Harness (Python)\n\n## Overview\n\nA team swapped `gpt-4o` for `claude-sonnet-4-6` to save money and a week later CS\nnoticed answer quality dropped on 15% of refund tickets — the regression was\ninvisible in code review and invisible in CI because no golden set existed.\n\nFix: a versioned golden set, a stacked eval pipeline (LangSmith +\nragas + deepeval + custom trajectory), and a PR-blocking regression gate\nwith paired Wilcoxon significance. The tooling exists; the patterns for\nwiring it into a statistically honest loop are scattered across five doc sites.\n\nBuild a 100-example JSONL golden set, wire LangSmith `evaluate()` with a\ncustom correctness evaluator, add a ragas quartet (faithfulness, answer\nrelevance, context precision\u002Frecall) for RAG, add deepeval LLM-as-judge\nwith N=3 judge quorum, score LangGraph trajectories on coverage\u002Fprecision\u002F\norder, and gate PRs on a 2% aggregate drop or 5% per-example drop. Pin:\n`langchain-core 1.0.x`, `langgraph 1.0.x`, `langsmith>=0.2`, `ragas>=0.2`,\n`deepeval>=2.0`. Pain-catalog anchors: P01, P11, P12, P22, P33.\n\n## Prerequisites\n\n- Python 3.10+\n- `langchain-core >= 1.0, \u003C 2.0`, `langgraph >= 1.0, \u003C 2.0` for the system under eval\n- `pip install langsmith>=0.2 ragas>=0.2 deepeval>=2.0 scipy`\n- LangSmith account + `LANGSMITH_API_KEY` (free tier is sufficient for dataset versioning)\n- Provider API keys for the judge LLM: `OPENAI_API_KEY` and\u002For `ANTHROPIC_API_KEY`\n\n## Instructions\n\n### Step 1 — Build a versioned golden set\n\nFormat: JSONL, one example per line, with a `dataset_version` tag. Minimum 20\nexamples to start; grow to 100 for PR gating, 200+ for absolute-metric claims.\n\n```python\n# evals\u002Fgolden_set\u002Fv2026.04.jsonl\n{\"id\": \"gs-0001\", \"input\": \"Refund policy for SKU ABC-42?\", \"expected\": \"30 days with receipt\", \"contexts\": [\"policy_v3.md\"], \"tags\": [\"refund\"], \"difficulty\": \"easy\", \"dataset_version\": \"2026.04\"}\n{\"id\": \"gs-0002\", \"input\": \"Return policy for opened software?\", \"expected\": \"No, opened software is final sale\", \"contexts\": [\"policy_v3.md#returns\"], \"tags\": [\"refund\"], \"difficulty\": \"medium\", \"dataset_version\": \"2026.04\"}\n```\n\nSample from real traffic (redacted), not imagination. Stratify by tag and\ndifficulty (aim for 30% hard). Two annotators per example, disagreements\nreconciled — reconciliation rate under 90% means your task definition is\nambiguous. Treat the file as immutable within a version; bump the version\nto refresh. See [Golden Set Curation](references\u002Fgolden-set-curation.md) for\nsourcing strategy, annotation tool options, and the refresh cadence.\n\n### Step 2 — Wire LangSmith `evaluate()` with a custom evaluator\n\n```python\nfrom langsmith import Client\nfrom langsmith.evaluation import evaluate, EvaluationResult\nfrom langchain_anthropic import ChatAnthropic\n\nclient = Client()\nDATASET_VERSION = \"2026.04\"\n\n# One-time: upload golden set as a versioned dataset\ndef upload_golden_set(jsonl_path, dataset_name):\n    examples = [json.loads(line) for line in open(jsonl_path)]\n    client.create_dataset(dataset_name)\n    client.create_examples(\n        inputs=[{\"input\": e[\"input\"]} for e in examples],\n        outputs=[{\"expected\": e[\"expected\"]} for e in examples],\n        metadata=[{\"id\": e[\"id\"], \"tags\": e[\"tags\"]} for e in examples],\n        dataset_name=dataset_name,\n    )\n\nchain = ChatAnthropic(model=\"claude-sonnet-4-6\", temperature=0, timeout=30)\n\ndef target(inputs):\n    return {\"answer\": chain.invoke(inputs[\"input\"]).content}\n\ndef correctness(outputs, reference_outputs):\n    \"\"\"Deterministic exact-match floor — baseline, not ceiling.\"\"\"\n    match = outputs[\"answer\"].strip().lower() == reference_outputs[\"expected\"].strip().lower()\n    return EvaluationResult(key=\"exact_match\", score=float(match))\n\nresults = evaluate(\n    target,\n    data=f\"golden-set-v{DATASET_VERSION}\",\n    evaluators=[correctness],\n    experiment_prefix=\"refund-bot-v3\",\n    max_concurrency=10,   # Avoid 429s on judge LLM (P22)\n)\n```\n\nFree-form outputs need semantic scoring (ragas, deepeval, or LLM-as-judge — Step 4).\n\n### Step 3 — Add ragas metrics for RAG pipelines\n\nFor a RAG chain returning `{answer, contexts}`, ragas scores four standard\ndimensions. The default judge is `gpt-4o-mini`; override to pin model +\ncost:\n\n```python\nfrom ragas import evaluate as ragas_evaluate\nfrom ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall\nfrom langchain_openai import ChatOpenAI\nfrom langchain_openai import OpenAIEmbeddings\nfrom datasets import Dataset\n\njudge = ChatOpenAI(model=\"gpt-4o-mini\", temperature=0)\nembed = OpenAIEmbeddings(model=\"text-embedding-3-small\")\n\n# Prepare rows — ragas wants HuggingFace Dataset shape\nrows = []\nfor ex in golden_examples:\n    result = rag_chain.invoke({\"question\": ex[\"input\"]})\n    rows.append({\n        \"question\": ex[\"input\"],\n        \"answer\": result[\"answer\"],\n        \"contexts\": [d.page_content for d in result[\"source_documents\"]],\n        \"ground_truth\": ex[\"expected\"],\n    })\n\nragas_results = ragas_evaluate(\n    Dataset.from_list(rows),\n    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],\n    llm=judge,\n    embeddings=embed,\n)\n# ragas_results is a dict of per-metric means; call .to_pandas() for per-row\n```\n\nDo not use ragas on non-RAG chains — `context_precision` against an empty\ncontext list returns 0 and looks like a regression. See\n[Framework Comparison](references\u002Fframework-comparison.md) for when each\ntool fits.\n\n### Step 4 — Add deepeval LLM-as-judge for free-form outputs\n\ndeepeval is pytest-shaped — each example is an `LLMTestCase` asserting against\nmetrics. Run N=3 judge invocations per example and take the median to tame\nLLM-as-judge variance (±5-15% across runs; single-run scores are not CI-ready):\n\n```python\nimport statistics\nfrom deepeval.metrics import GEval\nfrom deepeval.test_case import LLMTestCase, LLMTestCaseParams\n\ndef eval_with_quorum(test_case, metric, n=3):\n    scores = []\n    for _ in range(n):\n        metric.measure(test_case)\n        scores.append(metric.score)\n    return statistics.median(scores), statistics.stdev(scores) if n > 1 else 0.0\n\ncorrectness = GEval(\n    name=\"Correctness\",\n    criteria=\"Does the actual output match the expected output in meaning?\",\n    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],\n    model=\"gpt-4o-mini\",\n)\n\nfor ex in golden_examples:\n    result = chain.invoke({\"input\": ex[\"input\"]})\n    case = LLMTestCase(input=ex[\"input\"], actual_output=result, expected_output=ex[\"expected\"])\n    median, sd = eval_with_quorum(case, correctness, n=3)\n    if sd > 0.2:  # judge disagreeing with itself — flag, don't gate\n        flag_for_review(ex[\"id\"], median, sd)\n```\n\n### Step 5 — LangGraph agent trajectory eval\n\nFor agents, final-answer correctness misses the process. Score the tool-call\nsequence on three axes — coverage (did required tools run?), precision\n(were extra tools used?), and order (Kendall's tau on shared tools):\n\n```python\nfrom langchain_core.messages import AIMessage\n\ndef extract_trajectory(final_state: dict) -> list[dict]:\n    return [\n        {\"tool\": tc[\"name\"], \"args\": tc[\"args\"]}\n        for msg in final_state[\"messages\"] if isinstance(msg, AIMessage)\n        for tc in (msg.tool_calls or [])\n    ]\n\ndef trajectory_score(expected: list[str], actual: list[str]) -> dict:\n    e_set, a_set = set(expected), set(actual)\n    coverage = len(e_set & a_set) \u002F len(e_set) if e_set else 1.0\n    precision = len(e_set & a_set) \u002F len(a_set) if a_set else 0.0\n    shared = [t for t in actual if t in e_set]\n    order = _kendall_tau(expected, shared) if len(shared) >= 2 else 1.0\n    return {\"coverage\": coverage, \"precision\": precision, \"order\": order}\n\n# Composite: 0.5 * coverage + 0.3 * precision + 0.2 * order\n```\n\nSet `temperature=0` for the agent during eval — `temperature > 0` produces\ndifferent trajectories across runs (P11) and makes paired comparison\nstatistically invalid. See [Agent Trajectory Eval](references\u002Fagent-trajectory-eval.md)\nfor args-level matching, efficiency\u002Fsafety scoring, and the LLM-as-judge\nfallback for non-deterministic trajectories.\n\n### Step 6 — Gate PRs on regression\n\nA PR touching prompts, chain code, or model config runs the eval suite on\nPR branch and `main`, then blocks merge on any of: aggregate mean drop > **2.0%**,\nany single-example drop > **5.0%**, or paired Wilcoxon signed-rank p \u003C 0.05\nwith negative mean delta.\n\n```python\nfrom scipy.stats import wilcoxon\n\ndef paired_regression_check(baseline, candidate, alpha=0.05):\n    \"\"\"Wilcoxon — right test when metric distribution is non-normal (most LLM metrics).\"\"\"\n    n = len(baseline)\n    if n \u003C 50:\n        return {\"verdict\": \"too_small_n\", \"n\": n}\n    diffs = [c - b for b, c in zip(baseline, candidate)]\n    _, p = wilcoxon(diffs, alternative=\"less\")\n    return {\"n\": n, \"mean_delta\": sum(diffs) \u002F n, \"p_value\": float(p),\n            \"regression\": p \u003C alpha and sum(diffs) \u003C 0}\n```\n\nAt n=100 and α=0.05 this detects a ~3-5% true regression at ~80% power. See\n[CI Integration](references\u002Fci-integration.md) for the GitHub Actions workflow,\nPR-comment delta table, bootstrap CI, and spend\u002Frate-limit safety rails.\n\n## Output\n\n- JSONL golden set at `evals\u002Fgolden_set\u002Fv2026.04.jsonl` with an immutable version tag\n- LangSmith dataset uploaded and versioned; experiment runs linked to traces\n- Ragas scores (faithfulness, answer relevance, context precision\u002Frecall) on RAG chains\n- Deepeval `LLMTestCase` assertions in pytest, with median-of-3 judge quorum\n- LangGraph trajectory scores (coverage, precision, order) with composite summary\n- GitHub Actions workflow gating PRs on 2% aggregate \u002F 5% per-example \u002F Wilcoxon p \u003C 0.05\n- PR-comment delta table posted on every eval run\n\n## Framework selection at a glance\n\n| Use case | LangSmith | ragas | deepeval | Custom |\n|---|---|---|---|---|\n| RAG metrics (faithfulness, context recall) | — | **Primary** | Fallback | — |\n| Pytest-style assertion in CI | Secondary | — | **Primary** | — |\n| Trace capture + dataset versioning | **Primary** | Complementary | Complementary | — |\n| Agent trajectory (tool-call sequence) | Secondary (traces) | — | — | **Primary** |\n| Exact match \u002F JSON schema \u002F structured output | — | — | — | **Primary** |\n| Free-form paraphrase scoring | Via custom evaluator | — | **Primary (G-Eval)** | — |\n\nMost real pipelines stack two or three. The anti-pattern is running all four\non every example — you pay $10-30 per run for signal you are not using. See\n[Framework Comparison](references\u002Fframework-comparison.md) for the full\ndecision tree and dependency weight comparison.\n\n## Error Handling\n\n| Error \u002F Failure mode | Cause | Fix |\n|---|---|---|\n| `TimeoutError` on eval runs > 20 min | Long agent trajectories on slow models; 100 examples × 30s each exceeds default GH Actions job timeout | Cap `max_concurrency=10`, use `asyncio.gather` with `asyncio.Semaphore`, split eval into sharded jobs |\n| Judge disagreement (stdev > 0.2 on [0,1] scale across N=3 runs) | LLM-as-judge variance on ambiguous examples | Flag example for manual review; do not use that row's score for gating |\n| `ValidationError: missing 'contexts'` in ragas | Chain does not return retrieved docs | Modify chain to surface `source_documents`, or switch to non-RAG evaluator |\n| Wilcoxon p-value is NaN | All paired diffs are 0 (identical outputs) | Expected when the PR did not change behavior — no regression, skip the stat test |\n| LangSmith 429 rate limit during upload | > 50 examples\u002Fsec to `create_examples` | Batch with `client.create_examples(..., batch_size=20)` and sleep between batches |\n| Spend overrun ($50+ per run) | Judge calls scaling with `N_examples × N_metrics × N_judge_runs` | Use `gpt-4o-mini` not `gpt-4o` for judge; cache per `(dataset_version, chain_version)` |\n| `AttributeError: 'list' has no attribute 'lower'` in custom evaluator | Claude `AIMessage.content` is `list[dict]` not `str` (P02 — see langchain-model-inference) | Use `msg.text()` or iterate content blocks |\n| Trajectory comparison drifts week-over-week on unchanged agent | `temperature > 0` non-determinism (P11) | Set `temperature=0` for all eval runs; pin `seed` where supported |\n\n## Examples\n\n### Setting up eval for a new RAG chain\n\nStart with 20 production-sampled golden examples, wire up `ragas_evaluate`\nwith four metrics, record scores to `evals\u002Fbaselines\u002F` as the reference,\nand promote to LangSmith dataset versioning once two engineers annotate in\nparallel. See [Golden Set Curation](references\u002Fgolden-set-curation.md).\n\n### Diagnosing regression after a model swap\n\nRun the main-branch chain on the golden set, then swap the model and rerun.\nDiff per-example scores sorted by delta — the top-10 regressions usually\ncluster by tag (long contexts, one-shot lookups). Report paired Wilcoxon\nand per-tag breakdown before deciding to ship. See [CI Integration](references\u002Fci-integration.md).\n\n### Evaluating a LangGraph tool-calling agent\n\nRecord expected tool-call sequences for 50 tasks, capture actual trajectories\nvia `extract_trajectory`, and score on coverage\u002Fprecision\u002Forder. Composite\ndrops indicate a policy change — diff sequences to find the drift. See\n[Agent Trajectory Eval](references\u002Fagent-trajectory-eval.md).\n\n## Resources\n\n- [LangSmith evaluation tutorial](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation\u002Ftutorials\u002Fevaluation)\n- [LangSmith `evaluate()` reference](https:\u002F\u002Fdocs.smith.langchain.com\u002Freference\u002Fpython\u002Fsdk_reference\u002Flangsmith.evaluation)\n- [ragas metrics overview](https:\u002F\u002Fdocs.ragas.io\u002Fen\u002Flatest\u002Fconcepts\u002Fmetrics\u002Foverview\u002F)\n- [deepeval metrics reference](https:\u002F\u002Fdocs.confident-ai.com\u002Fdocs\u002Fmetrics-introduction)\n- [scipy Wilcoxon signed-rank](https:\u002F\u002Fdocs.scipy.org\u002Fdoc\u002Fscipy\u002Freference\u002Fgenerated\u002Fscipy.stats.wilcoxon.html)\n- [G-Eval variance paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2303.16634)\n- Pack pain catalog: `docs\u002Fpain-catalog.md` (entries P01, P11, P12, P22, P33)\n","# LangChain Eval Harness (Python)\n\n## Overview\n\nA team swapped `gpt-4o` for `claude-sonnet-4-6` to save money and a week later CS\nnoticed answer quality dropped on 15% of refund tickets — the regression was\ninvisible in code review and invisible in CI because no golden set existed.\n\nFix: a versioned golden set, a stacked eval pipeline (LangSmith +\nragas + deepeval + custom trajectory), and a PR-blocking regression gate\nwith paired Wilcoxon significance. The tooling exists; the patterns for\nwiring it into a statistically honest loop are scattered across five doc sites.\n\nBuild a 100-example JSONL golden set, wire LangSmith `evaluate()` with a\ncustom correctness evaluator, add a ragas quartet (faithfulness, answer\nrelevance, context precision\u002Frecall) for RAG, add deepeval LLM-as-judge\nwith N=3 judge quorum, score LangGraph trajectories on coverage\u002Fprecision\u002F\norder, and gate PRs on a 2% aggregate drop or 5% per-example drop. Pin:\n`langchain-core 1.0.x`, `langgraph 1.0.x`, `langsmith>=0.2`, `ragas>=0.2`,\n`deepeval>=2.0`. Pain-catalog anchors: P01, P11, P12, P22, P33.\n\n## Prerequisites\n\n- Python 3.10+\n- `langchain-core >= 1.0, \u003C 2.0`, `langgraph >= 1.0, \u003C 2.0` for the system under eval\n- `pip install langsmith>=0.2 ragas>=0.2 deepeval>=2.0 scipy`\n- LangSmith account + `LANGSMITH_API_KEY` (free tier is sufficient for dataset versioning)\n- Provider API keys for the judge LLM: `OPENAI_API_KEY` and\u002For `ANTHROPIC_API_KEY`\n\n## Instructions\n\n### Step 1 — Build a versioned golden set\n\nFormat: JSONL, one example per line, with a `dataset_version` tag. Minimum 20\nexamples to start; grow to 100 for PR gating, 200+ for absolute-metric claims.\n\n```python\n# evals\u002Fgolden_set\u002Fv2026.04.jsonl\n{\"id\": \"gs-0001\", \"input\": \"Refund policy for SKU ABC-42?\", \"expected\": \"30 days with receipt\", \"contexts\": [\"policy_v3.md\"], \"tags\": [\"refund\"], \"difficulty\": \"easy\", \"dataset_version\": \"2026.04\"}\n{\"id\": \"gs-0002\", \"input\": \"Return policy for opened software?\", \"expected\": \"No, opened software is final sale\", \"contexts\": [\"policy_v3.md#returns\"], \"tags\": [\"refund\"], \"difficulty\": \"medium\", \"dataset_version\": \"2026.04\"}\n```\n\nSample from real traffic (redacted), not imagination. Stratify by tag and\ndifficulty (aim for 30% hard). Two annotators per example, disagreements\nreconciled — reconciliation rate under 90% means your task definition is\nambiguous. Treat the file as immutable within a version; bump the version\nto refresh. See [Golden Set Curation](references\u002Fgolden-set-curation.md) for\nsourcing strategy, annotation tool options, and the refresh cadence.\n\n### Step 2 — Wire LangSmith `evaluate()` with a custom evaluator\n\n```python\nfrom langsmith import Client\nfrom langsmith.evaluation import evaluate, EvaluationResult\nfrom langchain_anthropic import ChatAnthropic\n\nclient = Client()\nDATASET_VERSION = \"2026.04\"\n\n# One-time: upload golden set as a versioned dataset\ndef upload_golden_set(jsonl_path, dataset_name):\n    examples = [json.loads(line) for line in open(jsonl_path)]\n    client.create_dataset(dataset_name)\n    client.create_examples(\n        inputs=[{\"input\": e[\"input\"]} for e in examples],\n        outputs=[{\"expected\": e[\"expected\"]} for e in examples],\n        metadata=[{\"id\": e[\"id\"], \"tags\": e[\"tags\"]} for e in examples],\n        dataset_name=dataset_name,\n    )\n\nchain = ChatAnthropic(model=\"claude-sonnet-4-6\", temperature=0, timeout=30)\n\ndef target(inputs):\n    return {\"answer\": chain.invoke(inputs[\"input\"]).content}\n\ndef correctness(outputs, reference_outputs):\n    \"\"\"Deterministic exact-match floor — baseline, not ceiling.\"\"\"\n    match = outputs[\"answer\"].strip().lower() == reference_outputs[\"expected\"].strip().lower()\n    return EvaluationResult(key=\"exact_match\", score=float(match))\n\nresults = evaluate(\n    target,\n    data=f\"golden-set-v{DATASET_VERSION}\",\n    evaluators=[correctness],\n    experiment_prefix=\"refund-bot-v3\",\n    max_concurrency=10,   # Avoid 429s on judge LLM (P22)\n)\n```\n\nFree-form outputs need semantic scoring (ragas, deepeval, or LLM-as-judge — Step 4).\n\n### Step 3 — Add ragas metrics for RAG pipelines\n\nFor a RAG chain returning `{answer, contexts}`, ragas scores four standard\ndimensions. The default judge is `gpt-4o-mini`; override to pin model +\ncost:\n\n```python\nfrom ragas import evaluate as ragas_evaluate\nfrom ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall\nfrom langchain_openai import ChatOpenAI\nfrom langchain_openai import OpenAIEmbeddings\nfrom datasets import Dataset\n\njudge = ChatOpenAI(model=\"gpt-4o-mini\", temperature=0)\nembed = OpenAIEmbeddings(model=\"text-embedding-3-small\")\n\n# Prepare rows — ragas wants HuggingFace Dataset shape\nrows = []\nfor ex in golden_examples:\n    result = rag_chain.invoke({\"question\": ex[\"input\"]})\n    rows.append({\n        \"question\": ex[\"input\"],\n        \"answer\": result[\"answer\"],\n        \"contexts\": [d.page_content for d in result[\"source_documents\"]],\n        \"ground_truth\": ex[\"expected\"],\n    })\n\nragas_results = ragas_evaluate(\n    Dataset.from_list(rows),\n    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],\n    llm=judge,\n    embeddings=embed,\n)\n# ragas_results is a dict of per-metric means; call .to_pandas() for per-row\n```\n\nDo not use ragas on non-RAG chains — `context_precision` against an empty\ncontext list returns 0 and looks like a regression. See\n[Framework Comparison](references\u002Fframework-comparison.md) for when each\ntool fits.\n\n### Step 4 — Add deepeval LLM-as-judge for free-form outputs\n\ndeepeval is pytest-shaped — each example is an `LLMTestCase` asserting against\nmetrics. Run N=3 judge invocations per example and take the median to tame\nLLM-as-judge variance (±5-15% across runs; single-run scores are not CI-ready):\n\n```python\nimport statistics\nfrom deepeval.metrics import GEval\nfrom deepeval.test_case import LLMTestCase, LLMTestCaseParams\n\ndef eval_with_quorum(test_case, metric, n=3):\n    scores = []\n    for _ in range(n):\n        metric.measure(test_case)\n        scores.append(metric.score)\n    return statistics.median(scores), statistics.stdev(scores) if n > 1 else 0.0\n\ncorrectness = GEval(\n    name=\"Correctness\",\n    criteria=\"Does the actual output match the expected output in meaning?\",\n    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],\n    model=\"gpt-4o-mini\",\n)\n\nfor ex in golden_examples:\n    result = chain.invoke({\"input\": ex[\"input\"]})\n    case = LLMTestCase(input=ex[\"input\"], actual_output=result, expected_output=ex[\"expected\"])\n    median, sd = eval_with_quorum(case, correctness, n=3)\n    if sd > 0.2:  # judge disagreeing with itself — flag, don't gate\n        flag_for_review(ex[\"id\"], median, sd)\n```\n\n### Step 5 — LangGraph agent trajectory eval\n\nFor agents, final-answer correctness misses the process. Score the tool-call\nsequence on three axes — coverage (did required tools run?), precision\n(were extra tools used?), and order (Kendall's tau on shared tools):\n\n```python\nfrom langchain_core.messages import AIMessage\n\ndef extract_trajectory(final_state: dict) -> list[dict]:\n    return [\n        {\"tool\": tc[\"name\"], \"args\": tc[\"args\"]}\n        for msg in final_state[\"messages\"] if isinstance(msg, AIMessage)\n        for tc in (msg.tool_calls or [])\n    ]\n\ndef trajectory_score(expected: list[str], actual: list[str]) -> dict:\n    e_set, a_set = set(expected), set(actual)\n    coverage = len(e_set & a_set) \u002F len(e_set) if e_set else 1.0\n    precision = len(e_set & a_set) \u002F len(a_set) if a_set else 0.0\n    shared = [t for t in actual if t in e_set]\n    order = _kendall_tau(expected, shared) if len(shared) >= 2 else 1.0\n    return {\"coverage\": coverage, \"precision\": precision, \"order\": order}\n\n# Composite: 0.5 * coverage + 0.3 * precision + 0.2 * order\n```\n\nSet `temperature=0` for the agent during eval — `temperature > 0` produces\ndifferent trajectories across runs (P11) and makes paired comparison\nstatistically invalid. See [Agent Trajectory Eval](references\u002Fagent-trajectory-eval.md)\nfor args-level matching, efficiency\u002Fsafety scoring, and the LLM-as-judge\nfallback for non-deterministic trajectories.\n\n### Step 6 — Gate PRs on regression\n\nA PR touching prompts, chain code, or model config runs the eval suite on\nPR branch and `main`, then blocks merge on any of: aggregate mean drop > **2.0%**,\nany single-example drop > **5.0%**, or paired Wilcoxon signed-rank p \u003C 0.05\nwith negative mean delta.\n\n```python\nfrom scipy.stats import wilcoxon\n\ndef paired_regression_check(baseline, candidate, alpha=0.05):\n    \"\"\"Wilcoxon — right test when metric distribution is non-normal (most LLM metrics).\"\"\"\n    n = len(baseline)\n    if n \u003C 50:\n        return {\"verdict\": \"too_small_n\", \"n\": n}\n    diffs = [c - b for b, c in zip(baseline, candidate)]\n    _, p = wilcoxon(diffs, alternative=\"less\")\n    return {\"n\": n, \"mean_delta\": sum(diffs) \u002F n, \"p_value\": float(p),\n            \"regression\": p \u003C alpha and sum(diffs) \u003C 0}\n```\n\nAt n=100 and α=0.05 this detects a ~3-5% true regression at ~80% power. See\n[CI Integration](references\u002Fci-integration.md) for the GitHub Actions workflow,\nPR-comment delta table, bootstrap CI, and spend\u002Frate-limit safety rails.\n\n## Output\n\n- JSONL golden set at `evals\u002Fgolden_set\u002Fv2026.04.jsonl` with an immutable version tag\n- LangSmith dataset uploaded and versioned; experiment runs linked to traces\n- Ragas scores (faithfulness, answer relevance, context precision\u002Frecall) on RAG chains\n- Deepeval `LLMTestCase` assertions in pytest, with median-of-3 judge quorum\n- LangGraph trajectory scores (coverage, precision, order) with composite summary\n- GitHub Actions workflow gating PRs on 2% aggregate \u002F 5% per-example \u002F Wilcoxon p \u003C 0.05\n- PR-comment delta table posted on every eval run\n\n## Framework selection at a glance\n\n| Use case | LangSmith | ragas | deepeval | Custom |\n|---|---|---|---|---|\n| RAG metrics (faithfulness, context recall) | — | **Primary** | Fallback | — |\n| Pytest-style assertion in CI | Secondary | — | **Primary** | — |\n| Trace capture + dataset versioning | **Primary** | Complementary | Complementary | — |\n| Agent trajectory (tool-call sequence) | Secondary (traces) | — | — | **Primary** |\n| Exact match \u002F JSON schema \u002F structured output | — | — | — | **Primary** |\n| Free-form paraphrase scoring | Via custom evaluator | — | **Primary (G-Eval)** | — |\n\nMost real pipelines stack two or three. The anti-pattern is running all four\non every example — you pay $10-30 per run for signal you are not using. See\n[Framework Comparison](references\u002Fframework-comparison.md) for the full\ndecision tree and dependency weight comparison.\n\n## Error Handling\n\n| Error \u002F Failure mode | Cause | Fix |\n|---|---|---|\n| `TimeoutError` on eval runs > 20 min | Long agent trajectories on slow models; 100 examples × 30s each exceeds default GH Actions job timeout | Cap `max_concurrency=10`, use `asyncio.gather` with `asyncio.Semaphore`, split eval into sharded jobs |\n| Judge disagreement (stdev > 0.2 on [0,1] scale across N=3 runs) | LLM-as-judge variance on ambiguous examples | Flag example for manual review; do not use that row's score for gating |\n| `ValidationError: missing 'contexts'` in ragas | Chain does not return retrieved docs | Modify chain to surface `source_documents`, or switch to non-RAG evaluator |\n| Wilcoxon p-value is NaN | All paired diffs are 0 (identical outputs) | Expected when the PR did not change behavior — no regression, skip the stat test |\n| LangSmith 429 rate limit during upload | > 50 examples\u002Fsec to `create_examples` | Batch with `client.create_examples(..., batch_size=20)` and sleep between batches |\n| Spend overrun ($50+ per run) | Judge calls scaling with `N_examples × N_metrics × N_judge_runs` | Use `gpt-4o-mini` not `gpt-4o` for judge; cache per `(dataset_version, chain_version)` |\n| `AttributeError: 'list' has no attribute 'lower'` in custom evaluator | Claude `AIMessage.content` is `list[dict]` not `str` (P02 — see langchain-model-inference) | Use `msg.text()` or iterate content blocks |\n| Trajectory comparison drifts week-over-week on unchanged agent | `temperature > 0` non-determinism (P11) | Set `temperature=0` for all eval runs; pin `seed` where supported |\n\n## Examples\n\n### Setting up eval for a new RAG chain\n\nStart with 20 production-sampled golden examples, wire up `ragas_evaluate`\nwith four metrics, record scores to `evals\u002Fbaselines\u002F` as the reference,\nand promote to LangSmith dataset versioning once two engineers annotate in\nparallel. See [Golden Set Curation](references\u002Fgolden-set-curation.md).\n\n### Diagnosing regression after a model swap\n\nRun the main-branch chain on the golden set, then swap the model and rerun.\nDiff per-example scores sorted by delta — the top-10 regressions usually\ncluster by tag (long contexts, one-shot lookups). Report paired Wilcoxon\nand per-tag breakdown before deciding to ship. See [CI Integration](references\u002Fci-integration.md).\n\n### Evaluating a LangGraph tool-calling agent\n\nRecord expected tool-call sequences for 50 tasks, capture actual trajectories\nvia `extract_trajectory`, and score on coverage\u002Fprecision\u002Forder. Composite\ndrops indicate a policy change — diff sequences to find the drift. See\n[Agent Trajectory Eval](references\u002Fagent-trajectory-eval.md).\n\n## Resources\n\n- [LangSmith evaluation tutorial](https:\u002F\u002Fdocs.smith.langchain.com\u002Fevaluation\u002Ftutorials\u002Fevaluation)\n- [LangSmith `evaluate()` reference](https:\u002F\u002Fdocs.smith.langchain.com\u002Freference\u002Fpython\u002Fsdk_reference\u002Flangsmith.evaluation)\n- [ragas metrics overview](https:\u002F\u002Fdocs.ragas.io\u002Fen\u002Flatest\u002Fconcepts\u002Fmetrics\u002Foverview\u002F)\n- [deepeval metrics reference](https:\u002F\u002Fdocs.confident-ai.com\u002Fdocs\u002Fmetrics-introduction)\n- [scipy Wilcoxon signed-rank](https:\u002F\u002Fdocs.scipy.org\u002Fdoc\u002Fscipy\u002Freference\u002Fgenerated\u002Fscipy.stats.wilcoxon.html)\n- [G-Eval variance paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2303.16634)\n- Pack pain catalog: `docs\u002Fpain-catalog.md` (entries P01, P11, P12, P22, P33)\n",{"name":5,"tags":24,"author":35,"license":15,"version":36,"description":37,"allowed-tools":38,"compatible-with":39},[25,26,27,28,29,30,31,32,33,34],"saas","langchain","langgraph","python","langchain-1.0","evaluation","langsmith","ragas","deepeval","research","Jeremy Longshore \u003Cjeremy@intentsolutions.io>","2.0.0","Build reproducible evaluation pipelines for LangChain 1.0 chains and LangGraph 1.0\nagents — golden datasets, LangSmith evaluate(), ragas RAG metrics, deepeval\nLLM-as-judge, agent trajectory analysis, and CI gating on quality regressions.\nUse when setting up quality measurement for a new chain, diagnosing regression\nafter a model switch, or building an evaluation gate for a pull request.\nTrigger with \"langchain eval\", \"langsmith evaluate\", \"ragas\", \"llm-as-judge\",\n\"agent trajectory eval\", \"eval regression gate\".\n","Read, Write, Edit, Bash(python:*), Bash(pip:*), Bash(pytest:*)","claude-code, codex","main","99825ad3f3bcc6597ddcc7294a14e6e922f2b67b","skills\u002Fsaas-packs--langchain-py-pack--langchain-eval-harness",0,"https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills","https:\u002F\u002Fgithub.com\u002Fjukeyman","https:\u002F\u002FRJBUSINESSSOLUTIONS.ORG","Credit and Finance Expert | Developer\r\n\r\nI'm Rick Jefferson, a seasoned credit and finance expert with over a decade of experience in financial literacy,","2026-07-13 07:03:29",{"source":8,"skillName":5,"commands":50,"prompt":60,"zipAvailable":61},[51,54,57],{"tool":52,"command":53},"npx","npx skills add jukeyman\u002Fjukeyman-skills --skill langchain-eval-harness",{"tool":55,"command":56},"bunx","bunx skills add jukeyman\u002Fjukeyman-skills --skill langchain-eval-harness",{"tool":58,"command":59},"pnpm","pnpm dlx skills add jukeyman\u002Fjukeyman-skills --skill langchain-eval-harness","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fsaas-packs--langchain-py-pack--langchain-eval-harness\u002FSKILL.md",true,[],[64,71,78,85,92,100],{"skillNo":65,"name":66,"nameZh":6,"description":67,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":68,"categories":69,"keywords":70,"publishedAt":19},"01M79ANS2SM329NEP5ZCX106RR","zustand-store-creator","Create zustand store creator operations. Auto-activating skill for Frontend Development.\nTriggers on: zustand store creator, zustand store creator\nPart of the Frontend Development skill category. Use when working with zustand store creator functionality. Trigger with phrases like \"zustand store creator\", \"zustand creator\", \"zustand\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fzustand-store-creator\u002FSKILL.md",[],[],{"skillNo":72,"name":73,"nameZh":6,"description":74,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":77,"publishedAt":19},"01M79AMZKP4YDEZ7EHG1Y7PP74","zero-trust-config-helper","Configure with zero trust config helper operations. Auto-activating skill for Security Advanced.\nTriggers on: zero trust config helper, zero trust config helper\nPart of the Security Advanced skill category. Use when configuring systems or services. Trigger with phrases like \"zero trust config helper\", \"zero helper\", \"zero\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fzero-trust-config-helper\u002FSKILL.md",[],[],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":19},"01M79AMJSBCEJQNBTYNAHYN5HM","zapier-integration-helper","Assist with zapier integration helper operations. Auto-activating skill for Business Automation.\nTriggers on: zapier integration helper, zapier integration helper\nPart of the Business Automation skill category. Use when working with APIs or building integrations. Trigger with phrases like \"zapier integration helper\", \"zapier helper\", \"zapier\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fzapier-integration-helper\u002FSKILL.md",[],[],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":19},"01M79AHJZGVMARS13VFQYRR1MP","yaml-config-validator","Validate yaml config validator operations. Auto-activating skill for DevOps Basics.\nTriggers on: yaml config validator, yaml config validator\nPart of the DevOps Basics skill category. Use when configuring systems or services. Trigger with phrases like \"yaml config validator\", \"yaml validator\", \"yaml\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fyaml-config-validator\u002FSKILL.md",[],[],{"skillNo":93,"name":94,"nameZh":6,"description":95,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":96,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":19},"01M79AHGDZRN0020RDQSQDPAM8","xvary-stock-research","Thesis-driven equity analysis from public SEC EDGAR and market data; \u002Fanalyze, \u002Fscore, \u002Fcompare workflows with bundled Python tools (Claude Code, Cursor, Codex).","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fxvary-stock-research\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fxvary-stock-research\u002FSKILL.md",[],[],{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":19},"01M79AHCYJ9FV9RMHQ24HDRN33","xss-vulnerability-scanner","Scan xss vulnerability scanner operations. Auto-activating skill for Security Fundamentals.\nTriggers on: xss vulnerability scanner, xss vulnerability scanner\nPart of the Security Fundamentals skill category. Use when working with xss vulnerability scanner functionality. Trigger with phrases like \"xss vulnerability scanner\", \"xss scanner\", \"xss\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fxss-vulnerability-scanner\u002FSKILL.md",[],[]]