[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7D8T49AFDD22Y3ABV6B0B2F":3,"skill-same:01M7D8T49AFDD22Y3ABV6B0B2F":80},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":58,"commitSha":59,"skillDirectory":60,"forksCount":14,"repositoryHtmlUrl":61,"repositoryHomepageUrl":6,"ownerHtmlUrl":62,"ownerBlogUrl":63,"ownerDescription":64,"updatedAt":65,"install":66,"resources":79},"01M7D8T49AFDD22Y3ABV6B0B2F","Langfuse LLM Observability Testing",null,"Instrument LLM apps with Langfuse tracing, then use traces, scores, and datasets to test in production, run evaluations on real traffic, catch regressions, and close the loop from incident to golden dataset.","GrupoANDevelopment-m\u002Falberto-ai","GrupoANDevelopment-m","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F257455179?v=4","01M7D69X2N2KEVN0SJF6WGM6SH","Antônio  Ndombe","01M7D69X3P5JTT8H8S904N3888",0,"MIT","https:\u002F\u002Fgithub.com\u002FGrupoANDevelopment-m\u002Falberto-ai\u002Fblob\u002Fmain\u002Fskills\u002Fqa\u002Flangfuse-llm-observability\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: Langfuse LLM Observability Testing\ndescription: Instrument LLM apps with Langfuse tracing, then use traces, scores, and datasets to test in production, run evaluations on real traffic, catch regressions, and close the loop from incident to golden dataset.\nversion: 1.0.0\nauthor: thetestingacademy\nlicense: MIT\ntags: [langfuse, llm-observability, tracing, llm-evals, scores, datasets, production-testing, regression]\ntestingTypes: [llm-evals, integration, regression]\nframeworks: [langfuse]\nlanguages: [python, typescript]\ndomains: [ai, llm, backend]\nagents: [claude-code, cursor, github-copilot, windsurf, codex, aider, continue, cline, zed, bolt, gemini-cli, amp]\n---\n\n# Langfuse LLM Observability Testing Skill\n\nYou are an expert AI quality engineer specializing in Langfuse. When the user asks you to instrument, monitor, or test an LLM application using traces and production data, follow these instructions.\n\n## Core Principles\n\n1. **You cannot test what you cannot see.** Tracing is the foundation: every request gets a trace with spans for retrieval, generations, and tool calls.\n2. **Production is the best test set.** Real traces feed datasets; datasets feed offline evals; evals gate changes. That loop is the whole practice.\n3. **Score everything three ways.** Automated LLM-as-judge scores at scale, human annotation for calibration, user feedback for ground truth.\n4. **Sessions and users over single calls.** Multi-turn quality problems only appear at session level.\n5. **Costs and latency are quality metrics.** Track them per trace; a regression in tokens-per-answer is a regression.\n\n## Setup\n\n```bash\npip install langfuse            # python\nnpm install langfuse            # typescript\nexport LANGFUSE_PUBLIC_KEY=pk-...\nexport LANGFUSE_SECRET_KEY=sk-...\nexport LANGFUSE_HOST=https:\u002F\u002Fcloud.langfuse.com   # or self-hosted URL\n```\n\n## Instrumentation (Python)\n\n```python\nfrom langfuse import Langfuse, observe\n\nlangfuse = Langfuse()\n\n@observe()                       # creates a trace per call\ndef answer(user_id: str, session_id: str, query: str):\n    langfuse.update_current_trace(user_id=user_id, session_id=session_id,\n                                  tags=[\"support-bot\", \"prod\"])\n    chunks = retrieve(query)     # decorate with @observe() too: becomes a span\n    reply = generate(query, chunks)   # generations auto-capture model, tokens, cost\n    return reply\n```\n\nDecorate retrieval, reranking, generation, and tool calls separately; a flat trace cannot localize failures. Wrappers\u002Fintegrations exist for OpenAI, LangChain, LlamaIndex, and the Vercel AI SDK; prefer them over manual spans.\n\n## Scoring: the Test Signal\n\n```python\n# 1. User feedback from the app (thumbs up\u002Fdown)\nlangfuse.create_score(trace_id=trace_id, name=\"user-feedback\", value=0, comment=\"wrong policy quoted\")\n\n# 2. Automated LLM-as-judge on sampled traces (configure evaluators in the UI\n#    or run your own job):\nfrom my_judges import faithfulness_judge\nfor trace in fetch_traces(tags=[\"support-bot\"], sample=0.1):\n    score = faithfulness_judge(trace.input, trace.output, trace.metadata[\"contexts\"])\n    langfuse.create_score(trace_id=trace.id, name=\"faithfulness\", value=score)\n\n# 3. Human annotation queues in the UI for calibration batches\n```\n\nAlerting policy: dashboard the 7-day moving average per score name; investigate any sustained drop even inside \"acceptable\" range, since judge drift and product drift look identical until triaged.\n\n## From Traces to Regression Tests\n\nThe core testing workflow: bad trace -> dataset item -> offline eval -> CI gate.\n\n```python\n# 1. Curate: add a failing production trace to a dataset\nlangfuse.create_dataset(name=\"support-golden\")\nlangfuse.create_dataset_item(\n    dataset_name=\"support-golden\",\n    input={\"query\": \"Can I get a refund after 45 days?\"},\n    expected_output=\"No; refund window is 30 days. Offer credit options.\",\n    source_trace_id=bad_trace.id,        # provenance\n)\n\n# 2. Experiment: run a candidate change against the dataset\ndataset = langfuse.get_dataset(\"support-golden\")\nfor item in dataset.items:\n    with item.run(run_name=\"prompt-v9\") as root:\n        output = my_app.answer_candidate(item.input[\"query\"])\n        root.update(output=output)\n        root.score(name=\"correctness\", value=judge(output, item.expected_output))\n\n# 3. Compare runs in the UI (prompt-v8 vs prompt-v9) or via API in CI\n```\n\nCI gate pattern: nightly job runs the current build against the dataset, pushes scores as a run, and fails if aggregate correctness drops below threshold or below the previous run by more than 2 points.\n\n## Prompt Management as Change Control\n\nStore prompts in Langfuse prompt management with labels (production, staging). Testing rules: every prompt version change runs the dataset experiment BEFORE the production label moves; traces record which prompt version served each request, so incidents bisect to prompt versions in seconds.\n\n## Monitoring Views That Catch Bugs\n\n| View | Bug class it catches |\n|---|---|\n| Traces filtered by low user-feedback | Real failures, source for dataset items |\n| Score trend by prompt\u002Fmodel version | Regressions from \"harmless\" prompt edits |\n| Cost per trace over time | Token explosions from context stuffing |\n| Latency percentiles per span | Slow retrieval hiding behind fast generation |\n| Sessions with high turn count | Loops, users re-asking because answers fail |\n\n## Common Mistakes\n\n- Tracing only the final generation; without retrieval spans you cannot tell bad context from bad generation\n- No user\u002Fsession IDs on traces; multi-turn and per-cohort analysis becomes impossible later\n- Judge scores never calibrated against human labels; drifting judges silently redefine quality\n- Datasets built once and frozen; the loop only works if bad traces keep flowing in\n- Shipping prompt edits without a dataset experiment because \"it is just wording\"\n\n## Checklist\n\n- [ ] Traces with spans for retrieval, generation, tools; user_id, session_id, tags set\n- [ ] Three score streams live: user feedback, sampled LLM-judge, human calibration\n- [ ] Golden dataset in Langfuse, grown weekly from bad traces, with provenance\n- [ ] Experiments run on every prompt\u002Fmodel change before production label moves\n- [ ] Nightly CI experiment with threshold + delta gates; cost and latency dashboards reviewed\n","# Langfuse LLM Observability Testing Skill\n\nYou are an expert AI quality engineer specializing in Langfuse. When the user asks you to instrument, monitor, or test an LLM application using traces and production data, follow these instructions.\n\n## Core Principles\n\n1. **You cannot test what you cannot see.** Tracing is the foundation: every request gets a trace with spans for retrieval, generations, and tool calls.\n2. **Production is the best test set.** Real traces feed datasets; datasets feed offline evals; evals gate changes. That loop is the whole practice.\n3. **Score everything three ways.** Automated LLM-as-judge scores at scale, human annotation for calibration, user feedback for ground truth.\n4. **Sessions and users over single calls.** Multi-turn quality problems only appear at session level.\n5. **Costs and latency are quality metrics.** Track them per trace; a regression in tokens-per-answer is a regression.\n\n## Setup\n\n```bash\npip install langfuse            # python\nnpm install langfuse            # typescript\nexport LANGFUSE_PUBLIC_KEY=pk-...\nexport LANGFUSE_SECRET_KEY=sk-...\nexport LANGFUSE_HOST=https:\u002F\u002Fcloud.langfuse.com   # or self-hosted URL\n```\n\n## Instrumentation (Python)\n\n```python\nfrom langfuse import Langfuse, observe\n\nlangfuse = Langfuse()\n\n@observe()                       # creates a trace per call\ndef answer(user_id: str, session_id: str, query: str):\n    langfuse.update_current_trace(user_id=user_id, session_id=session_id,\n                                  tags=[\"support-bot\", \"prod\"])\n    chunks = retrieve(query)     # decorate with @observe() too: becomes a span\n    reply = generate(query, chunks)   # generations auto-capture model, tokens, cost\n    return reply\n```\n\nDecorate retrieval, reranking, generation, and tool calls separately; a flat trace cannot localize failures. Wrappers\u002Fintegrations exist for OpenAI, LangChain, LlamaIndex, and the Vercel AI SDK; prefer them over manual spans.\n\n## Scoring: the Test Signal\n\n```python\n# 1. User feedback from the app (thumbs up\u002Fdown)\nlangfuse.create_score(trace_id=trace_id, name=\"user-feedback\", value=0, comment=\"wrong policy quoted\")\n\n# 2. Automated LLM-as-judge on sampled traces (configure evaluators in the UI\n#    or run your own job):\nfrom my_judges import faithfulness_judge\nfor trace in fetch_traces(tags=[\"support-bot\"], sample=0.1):\n    score = faithfulness_judge(trace.input, trace.output, trace.metadata[\"contexts\"])\n    langfuse.create_score(trace_id=trace.id, name=\"faithfulness\", value=score)\n\n# 3. Human annotation queues in the UI for calibration batches\n```\n\nAlerting policy: dashboard the 7-day moving average per score name; investigate any sustained drop even inside \"acceptable\" range, since judge drift and product drift look identical until triaged.\n\n## From Traces to Regression Tests\n\nThe core testing workflow: bad trace -> dataset item -> offline eval -> CI gate.\n\n```python\n# 1. Curate: add a failing production trace to a dataset\nlangfuse.create_dataset(name=\"support-golden\")\nlangfuse.create_dataset_item(\n    dataset_name=\"support-golden\",\n    input={\"query\": \"Can I get a refund after 45 days?\"},\n    expected_output=\"No; refund window is 30 days. Offer credit options.\",\n    source_trace_id=bad_trace.id,        # provenance\n)\n\n# 2. Experiment: run a candidate change against the dataset\ndataset = langfuse.get_dataset(\"support-golden\")\nfor item in dataset.items:\n    with item.run(run_name=\"prompt-v9\") as root:\n        output = my_app.answer_candidate(item.input[\"query\"])\n        root.update(output=output)\n        root.score(name=\"correctness\", value=judge(output, item.expected_output))\n\n# 3. Compare runs in the UI (prompt-v8 vs prompt-v9) or via API in CI\n```\n\nCI gate pattern: nightly job runs the current build against the dataset, pushes scores as a run, and fails if aggregate correctness drops below threshold or below the previous run by more than 2 points.\n\n## Prompt Management as Change Control\n\nStore prompts in Langfuse prompt management with labels (production, staging). Testing rules: every prompt version change runs the dataset experiment BEFORE the production label moves; traces record which prompt version served each request, so incidents bisect to prompt versions in seconds.\n\n## Monitoring Views That Catch Bugs\n\n| View | Bug class it catches |\n|---|---|\n| Traces filtered by low user-feedback | Real failures, source for dataset items |\n| Score trend by prompt\u002Fmodel version | Regressions from \"harmless\" prompt edits |\n| Cost per trace over time | Token explosions from context stuffing |\n| Latency percentiles per span | Slow retrieval hiding behind fast generation |\n| Sessions with high turn count | Loops, users re-asking because answers fail |\n\n## Common Mistakes\n\n- Tracing only the final generation; without retrieval spans you cannot tell bad context from bad generation\n- No user\u002Fsession IDs on traces; multi-turn and per-cohort analysis becomes impossible later\n- Judge scores never calibrated against human labels; drifting judges silently redefine quality\n- Datasets built once and frozen; the loop only works if bad traces keep flowing in\n- Shipping prompt edits without a dataset experiment because \"it is just wording\"\n\n## Checklist\n\n- [ ] Traces with spans for retrieval, generation, tools; user_id, session_id, tags set\n- [ ] Three score streams live: user feedback, sampled LLM-judge, human calibration\n- [ ] Golden dataset in Langfuse, grown weekly from bad traces, with provenance\n- [ ] Experiments run on every prompt\u002Fmodel change before production label moves\n- [ ] Nightly CI experiment with threshold + delta gates; cost and latency dashboards reviewed\n",{"name":5,"tags":24,"agents":33,"author":46,"domains":47,"license":15,"version":51,"languages":52,"frameworks":55,"description":7,"testingTypes":56},[25,26,27,28,29,30,31,32],"langfuse","llm-observability","tracing","llm-evals","scores","datasets","production-testing","regression",[34,35,36,37,38,39,40,41,42,43,44,45],"claude-code","cursor","github-copilot","windsurf","codex","aider","continue","cline","zed","bolt","gemini-cli","amp","thetestingacademy",[48,49,50],"ai","llm","backend","1.0.0",[53,54],"python","typescript",[25],[28,57,32],"integration","main","eac438ea9216a7cb8a570cab803eb01eeafc6793","skills\u002Fqa\u002Flangfuse-llm-observability","https:\u002F\u002Fgithub.com\u002FGrupoANDevelopment-m\u002Falberto-ai","https:\u002F\u002Fgithub.com\u002FGrupoANDevelopment-m","https:\u002F\u002Fid-preview--d5bef06d-6c9e-4563-a3d4-a855b9d61610.lovable.app\u002F","Eu sou Antonio B. B. Ndombe \r\nEu sou um Pesquisador e desenvolvedor de sistema autonomia cognitivos, fundador de sistemas ODCs ( organismos digital conscientes)","2026-11-04 06:53:39",{"source":8,"skillName":5,"commands":67,"prompt":77,"zipAvailable":78},[68,71,74],{"tool":69,"command":70},"npx","npx skills add GrupoANDevelopment-m\u002Falberto-ai --skill \"Langfuse LLM Observability Testing\"",{"tool":72,"command":73},"bunx","bunx skills add GrupoANDevelopment-m\u002Falberto-ai --skill \"Langfuse LLM Observability Testing\"",{"tool":75,"command":76},"pnpm","pnpm dlx skills add GrupoANDevelopment-m\u002Falberto-ai --skill \"Langfuse LLM Observability Testing\"","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FGrupoANDevelopment-m\u002Falberto-ai\u002Fblob\u002Fmain\u002Fskills\u002Fqa\u002Flangfuse-llm-observability\u002FSKILL.md",true,[],[81,89,96,103,110,117],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":85,"licenseFileUrl":6,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":19},"01M7DASC6FSCJP49JZBSK0NY1Z","nemoclaw-user-reference","Describes the NemoClaw integration layer and blueprint architecture and how they orchestrate compatible agent sandboxes. Use when looking up architecture, agent integration, plugin structure, or blueprint design. Trigger keywords - nemoclaw architecture, nemoclaw agent architecture, nemoclaw plugin blueprint structure, nemoclaw vs openshell, which cli, nemoclaw cli, openshell cli, sandbox commands, nemoclaw cli commands, nemoclaw command reference, nemoclaw network policy, sandbox egress control operator approval, nemoclaw troubleshooting, nemoclaw debug sandbox issues.","Apache-2.0","https:\u002F\u002Fgithub.com\u002FGrupoANDevelopment-m\u002Falberto-ai\u002Fblob\u002Fmain\u002Fupstream\u002Fnemoclaw\u002Fskills\u002Fnemoclaw-user-reference\u002FSKILL.md",[],[],{"skillNo":90,"name":91,"nameZh":6,"description":92,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":85,"licenseFileUrl":6,"sourceHtmlUrl":93,"categories":94,"keywords":95,"publishedAt":19},"01M7DAS5S79VK33XFGVAM2TFWF","nemoclaw-user-overview","Explains how OpenClaw, OpenShell, and NemoClaw form the ecosystem, NemoClaw's position in the stack, what NemoClaw adds beyond the community sandbox, and when to prefer NemoClaw versus integrating OpenShell and OpenClaw directly. Use when users ask about the relationship between OpenClaw, OpenShell, and NemoClaw, or when to use NemoClaw versus OpenShell. Trigger keywords - nemoclaw ecosystem, openclaw openshell, nemoclaw vs openshell, sandboxed openclaw, how nemoclaw works, nemoclaw sandbox lifecycle blueprint, nemoclaw overview, openclaw always-on assistants, nvidia openshell, nvidia nemotron, nemoclaw release notes, nemoclaw changelog.","https:\u002F\u002Fgithub.com\u002FGrupoANDevelopment-m\u002Falberto-ai\u002Fblob\u002Fmain\u002Fupstream\u002Fnemoclaw\u002Fskills\u002Fnemoclaw-user-overview\u002FSKILL.md",[],[],{"skillNo":97,"name":98,"nameZh":6,"description":99,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":85,"licenseFileUrl":6,"sourceHtmlUrl":100,"categories":101,"keywords":102,"publishedAt":19},"01M7DAS0PQEMET5MNGZ1ABSZVS","nemoclaw-user-monitor-sandbox","Inspects sandbox health, traces agent behavior, and diagnoses problems. Use when monitoring a running sandbox, debugging agent issues, or checking sandbox logs. Trigger keywords - monitor nemoclaw sandbox, debug nemoclaw agent issues.","https:\u002F\u002Fgithub.com\u002FGrupoANDevelopment-m\u002Falberto-ai\u002Fblob\u002Fmain\u002Fupstream\u002Fnemoclaw\u002Fskills\u002Fnemoclaw-user-monitor-sandbox\u002FSKILL.md",[],[],{"skillNo":104,"name":105,"nameZh":6,"description":106,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":85,"licenseFileUrl":6,"sourceHtmlUrl":107,"categories":108,"keywords":109,"publishedAt":19},"01M7DARQYGF7E9A69VH6S3CGN9","nemoclaw-user-manage-sandboxes","Explains operational tasks after the quickstart: listing sandboxes, status and health checks, logs, diagnostics, port forwards, multiple sandboxes, credential reset, rebuilds, network presets, upgrades, and uninstall. Trigger keywords - manage nemoclaw sandboxes, nemoclaw status, nemoclaw list, nemoclaw dashboard port, nemoclaw rebuild, nemoclaw upgrade sandboxes, nemoclaw uninstall, sandbox mutability, sandbox runtime configuration, sandbox rebuild, nemoclaw backup, nemoclaw restore, workspace backup, openshell sandbox download upload, nemoclaw messaging channels, nemoclaw telegram, nemoclaw discord, nemoclaw slack, nemoclaw wechat, nemoclaw whatsapp, openshell channel messaging, nemoclaw workspace files, soul.md, user.md, identity.md, agents.md, sandbox persistence.","https:\u002F\u002Fgithub.com\u002FGrupoANDevelopment-m\u002Falberto-ai\u002Fblob\u002Fmain\u002Fupstream\u002Fnemoclaw\u002Fskills\u002Fnemoclaw-user-manage-sandboxes\u002FSKILL.md",[],[],{"skillNo":111,"name":112,"nameZh":6,"description":113,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":85,"licenseFileUrl":6,"sourceHtmlUrl":114,"categories":115,"keywords":116,"publishedAt":19},"01M7DARH8BRTG51TRCAB7YWDJ4","nemoclaw-user-manage-policy","Adds, removes, or modifies allowed endpoints in the sandbox policy. Use when customizing network policy, changing egress rules, or configuring sandbox endpoint access. Trigger keywords - customize nemoclaw network policy, sandbox egress policy configuration, nemoclaw integration policy examples, post-install policy setup, openshell approval workflow, policy preset, nemoclaw approve network requests, sandbox egress approval tui.","https:\u002F\u002Fgithub.com\u002FGrupoANDevelopment-m\u002Falberto-ai\u002Fblob\u002Fmain\u002Fupstream\u002Fnemoclaw\u002Fskills\u002Fnemoclaw-user-manage-policy\u002FSKILL.md",[],[],{"skillNo":118,"name":119,"nameZh":6,"description":120,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":85,"licenseFileUrl":6,"sourceHtmlUrl":121,"categories":122,"keywords":123,"publishedAt":19},"01M7DARAT62M2VR37590VCZ5NS","nemoclaw-user-get-started","Installs NemoClaw, launches a sandbox, and runs the first agent prompt. Use when onboarding, installing, or launching a NemoClaw sandbox for the first time. Trigger keywords - nemoclaw quickstart, install nemoclaw openclaw sandbox, nemohermes quickstart, hermes agent nemoclaw, run hermes openshell sandbox, nemoclaw prerequisites, nemoclaw supported platforms, nemoclaw hardware software, nemoclaw windows wsl2 setup, nemoclaw install windows docker desktop.","https:\u002F\u002Fgithub.com\u002FGrupoANDevelopment-m\u002Falberto-ai\u002Fblob\u002Fmain\u002Fupstream\u002Fnemoclaw\u002Fskills\u002Fnemoclaw-user-get-started\u002FSKILL.md",[],[]]