AGENT SKILL

DeepEval LLM Evaluation

Test LLM applications with DeepEval, pytest-style unit tests for LLM outputs using G-Eval, answer relevancy, faithfulness, hallucination and custom metrics, with CI quality gates and dataset-driven regression runs.

License 许可
MIT
最近更新
2026-11-04
Stars
★ 0
Forks
0

01 前言 Front Matter

name
DeepEval LLM Evaluation
tags
[ "deepeval", "llm-evals", "g-eval", "hallucination", "answer-relevancy", "faithfulness", "llm-testing", "pytest", "ci-gates" ]
agents
[ "claude-code", "cursor", "github-copilot", "windsurf", "codex", "aider", "continue", "cline", "zed", "bolt", "gemini-cli", "amp" ]
author
thetestingacademy
domains
[ "ai", "llm", "api" ]
license
MIT
version
1.0.0
languages
[ "python" ]
frameworks
[ "deepeval", "pytest" ]
description
Test LLM applications with DeepEval, pytest-style unit tests for LLM outputs using G-Eval, answer relevancy, faithfulness, hallucination and custom metrics, with CI quality gates and dataset-driven regression runs.
testingTypes
[ "llm-evals", "unit", "regression" ]

03 SKILL.md 正文

DeepEval LLM Evaluation Skill

You are an expert AI quality engineer specializing in DeepEval. When the user asks you to test, evaluate, or gate LLM application outputs, follow these instructions.

Core Principles

  1. Evals are unit tests. Write them pytest-style, run them in CI, fail builds on regressions. No dashboard-only quality.
  2. Metric per failure mode. Pick metrics for the failures that matter (hallucination, irrelevance, unfaithfulness to context), not every metric available.
  3. Thresholds are contracts. Every metric gets an explicit threshold agreed with the team; a metric without a threshold is a vibe.
  4. Datasets over ad-hoc prompts. Evaluate against a versioned golden dataset, grow it from production failures.
  5. LLM-as-judge needs spot checks. Periodically hand-verify judge scores; recalibrate criteria when the judge drifts from human judgment.

Setup

pip install deepeval
# judge model key (defaults to OpenAI; other providers configurable)
export OPENAI_API_KEY=sk-...
deepeval login   # optional: Confident AI dashboard for run history

Project Structure

llm-app/
├── evals/
│   ├── conftest.py            # fixtures: app client, dataset loader
│   ├── datasets/
│   │   └── golden_v3.jsonl    # versioned eval cases
│   ├── test_correctness.py    # G-Eval correctness suite
│   ├── test_rag_quality.py    # faithfulness + relevancy for RAG
│   └── test_safety.py         # hallucination, bias, toxicity
└── .github/workflows/evals.yml

Writing Eval Tests

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    AnswerRelevancyMetric,
    FaithfulnessMetric,
    HallucinationMetric,
    GEval,
)
from deepeval.test_case import LLMTestCaseParams

def make_case(query: str) -> LLMTestCase:
    response = my_app.answer(query)          # your application under test
    return LLMTestCase(
        input=query,
        actual_output=response.text,
        retrieval_context=response.chunks,   # required for faithfulness
    )

def test_answer_relevancy():
    case = make_case("What is your refund policy for annual plans?")
    assert_test(case, [AnswerRelevancyMetric(threshold=0.8)])

def test_faithfulness_to_context():
    case = make_case("How long does shipping take to Germany?")
    assert_test(case, [FaithfulnessMetric(threshold=0.9)])

# G-Eval: custom criteria in natural language, scored by a judge model
correctness = GEval(
    name="Correctness",
    criteria="Determine whether the actual output states the same policy facts as the expected output. Penalize invented numbers or dates.",
    evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
    threshold=0.7,
)

def test_policy_correctness():
    case = LLMTestCase(
        input="Can I cancel within 30 days?",
        actual_output=my_app.answer("Can I cancel within 30 days?").text,
        expected_output="Yes, full refund within 30 days of purchase.",
    )
    assert_test(case, [correctness])

Run: deepeval test run evals/ -n 8 (parallel) or plain pytest evals/.

Dataset-Driven Regression

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.add_test_cases_from_json_file(
    file_path="evals/datasets/golden_v3.jsonl",
    input_key_name="input",
    expected_output_key_name="expected",
)

@pytest.mark.parametrize("case", dataset.test_cases)
def test_golden(case):
    case.actual_output = my_app.answer(case.input).text
    assert_test(case, [AnswerRelevancyMetric(threshold=0.8), correctness])

Rules for the golden set: 30 to 200 cases per feature; every production incident adds a case; version the file and reference the version in eval reports; never edit expected outputs to make a failing run pass without review.

Metric Selection Guide

Failure mode Metric Typical threshold
Answer ignores the question AnswerRelevancyMetric 0.7 to 0.85
Answer contradicts retrieved docs FaithfulnessMetric 0.85 to 0.95
Invented facts vs provided context HallucinationMetric <= 0.1 (lower is better)
Domain-specific correctness GEval with written criteria 0.7 start, calibrate
Retrieval brought wrong chunks ContextualRelevancyMetric 0.7
Tone, format, policy compliance GEval per rule per rule

CI Quality Gate

# .github/workflows/evals.yml
- name: Run LLM evals
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
  run: deepeval test run evals/ --display-mode failing

Gate policy: block merge on any metric below threshold for P0 flows; run the full golden set nightly (judge calls cost money, so PR runs can use a 20-case smoke slice); alert on aggregate score drops week over week even when above threshold.

Common Mistakes

  • Testing with one hand-written prompt instead of a dataset; you are testing the demo, not the product
  • FaithfulnessMetric without passing retrieval_context; the metric needs the chunks
  • Thresholds copied from docs instead of calibrated against 20 human-labeled examples
  • Letting judge model version float; pin it, bump deliberately, re-baseline
  • Ignoring cost: cache app responses so re-runs only re-judge, and slice suites for PR vs nightly

Checklist

  • [ ] Metrics mapped to real failure modes, each with a threshold
  • [ ] Golden dataset versioned, grown from production failures
  • [ ] PR smoke slice + nightly full run wired in CI
  • [ ] Judge model pinned; monthly human spot-check of 10 judge scores
  • [ ] Eval results visible to the team (CI output or dashboard), regressions block release

04 同仓库更多 Skills

GrupoANDevelopment-m
GrupoANDevelopment-m/alberto-ai

nemoclaw-user-reference

Describes the NemoClaw integration layer and blueprint architecture and how they orchestrate compatible agent sandboxes. Use when looking up architecture, agent integration, plugin structure, or blueprint design. Trigger keywords - nemoclaw architecture, nemoclaw agent architecture, nemoclaw plugin blueprint structure, nemoclaw vs openshell, which cli, nemoclaw cli, openshell cli, sandbox commands, nemoclaw cli commands, nemoclaw command reference, nemoclaw network policy, sandbox egress control operator approval, nemoclaw troubleshooting, nemoclaw debug sandbox issues.

★ 0
GrupoANDevelopment-m
GrupoANDevelopment-m/alberto-ai

nemoclaw-user-overview

Explains how OpenClaw, OpenShell, and NemoClaw form the ecosystem, NemoClaw's position in the stack, what NemoClaw adds beyond the community sandbox, and when to prefer NemoClaw versus integrating OpenShell and OpenClaw directly. Use when users ask about the relationship between OpenClaw, OpenShell, and NemoClaw, or when to use NemoClaw versus OpenShell. Trigger keywords - nemoclaw ecosystem, openclaw openshell, nemoclaw vs openshell, sandboxed openclaw, how nemoclaw works, nemoclaw sandbox lifecycle blueprint, nemoclaw overview, openclaw always-on assistants, nvidia openshell, nvidia nemotron, nemoclaw release notes, nemoclaw changelog.

★ 0
GrupoANDevelopment-m
GrupoANDevelopment-m/alberto-ai

nemoclaw-user-monitor-sandbox

Inspects sandbox health, traces agent behavior, and diagnoses problems. Use when monitoring a running sandbox, debugging agent issues, or checking sandbox logs. Trigger keywords - monitor nemoclaw sandbox, debug nemoclaw agent issues.

★ 0
GrupoANDevelopment-m
GrupoANDevelopment-m/alberto-ai

nemoclaw-user-manage-sandboxes

Explains operational tasks after the quickstart: listing sandboxes, status and health checks, logs, diagnostics, port forwards, multiple sandboxes, credential reset, rebuilds, network presets, upgrades, and uninstall. Trigger keywords - manage nemoclaw sandboxes, nemoclaw status, nemoclaw list, nemoclaw dashboard port, nemoclaw rebuild, nemoclaw upgrade sandboxes, nemoclaw uninstall, sandbox mutability, sandbox runtime configuration, sandbox rebuild, nemoclaw backup, nemoclaw restore, workspace backup, openshell sandbox download upload, nemoclaw messaging channels, nemoclaw telegram, nemoclaw discord, nemoclaw slack, nemoclaw wechat, nemoclaw whatsapp, openshell channel messaging, nemoclaw workspace files, soul.md, user.md, identity.md, agents.md, sandbox persistence.

★ 0
GrupoANDevelopment-m
GrupoANDevelopment-m/alberto-ai

nemoclaw-user-manage-policy

Adds, removes, or modifies allowed endpoints in the sandbox policy. Use when customizing network policy, changing egress rules, or configuring sandbox endpoint access. Trigger keywords - customize nemoclaw network policy, sandbox egress policy configuration, nemoclaw integration policy examples, post-install policy setup, openshell approval workflow, policy preset, nemoclaw approve network requests, sandbox egress approval tui.

★ 0
GrupoANDevelopment-m
GrupoANDevelopment-m/alberto-ai

nemoclaw-user-get-started

Installs NemoClaw, launches a sandbox, and runs the first agent prompt. Use when onboarding, installing, or launching a NemoClaw sandbox for the first time. Trigger keywords - nemoclaw quickstart, install nemoclaw openclaw sandbox, nemohermes quickstart, hermes agent nemoclaw, run hermes openshell sandbox, nemoclaw prerequisites, nemoclaw supported platforms, nemoclaw hardware software, nemoclaw windows wsl2 setup, nemoclaw install windows docker desktop.

★ 0