[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M752FST7C9Y2T1TC5K37V44N":3,"skill-same:01M752FST7C9Y2T1TC5K37V44N":49},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":29,"commitSha":30,"skillDirectory":5,"forksCount":14,"repositoryHtmlUrl":31,"repositoryHomepageUrl":6,"ownerHtmlUrl":32,"ownerBlogUrl":33,"ownerDescription":6,"updatedAt":34,"install":35,"resources":48},"01M752FST7C9Y2T1TC5K37V44N","langfuse-core-workflow-b",null,"Execute Langfuse secondary workflow: Evaluation and scoring.\nUse when implementing LLM evaluation, adding user feedback,\nor setting up automated quality scoring for AI outputs.\nTrigger with phrases like \"langfuse evaluation\", \"langfuse scoring\",\n\"rate llm outputs\", \"langfuse feedback\", \"evaluate ai responses\".","SamuelChien\u002Fmega-skills-collection","SamuelChien","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F4652114?v=4","01M74KK34WVHA0P5ZYSJRDDFMG","Samuel Chien","01M74KK35FQN1N9VZ9EPTH3P2S",0,"MIT","https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection\u002Fblob\u002Fmain\u002Flangfuse-core-workflow-b\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: langfuse-core-workflow-b\ndescription: |\n  Execute Langfuse secondary workflow: Evaluation and scoring.\n  Use when implementing LLM evaluation, adding user feedback,\n  or setting up automated quality scoring for AI outputs.\n  Trigger with phrases like \"langfuse evaluation\", \"langfuse scoring\",\n  \"rate llm outputs\", \"langfuse feedback\", \"evaluate ai responses\".\nallowed-tools: Read, Write, Edit, Bash(npm:*), Grep\nversion: 1.0.0\nlicense: MIT\nauthor: Jeremy Longshore \u003Cjeremy@intentsolutions.io>\ncompatible-with: claude-code, codex, openclaw\n---\n# Langfuse Core Workflow B: Evaluation & Scoring\n\n## Overview\nImplement LLM output evaluation and scoring with Langfuse. Covers manual user feedback collection, automated evaluation functions, A\u002FB testing prompts with score comparison, and building evaluation datasets.\n\n## Prerequisites\n- Langfuse instance (Cloud or self-hosted)\n- Langfuse SDK configured with API keys\n- Traces already being collected (see workflow A)\n- Understanding of evaluation metrics\n\n## Instructions\n\n### Step 1: Manual User Feedback Scoring\n```typescript\nimport { Langfuse } from 'langfuse';\n\nconst langfuse = new Langfuse();\n\n\u002F\u002F Score a specific trace with user feedback\nasync function submitUserFeedback(\n  traceId: string,\n  rating: 'positive' | 'negative',\n  comment?: string\n) {\n  langfuse.score({\n    traceId,\n    name: 'user-feedback',\n    value: rating === 'positive' ? 1 : 0,\n    comment,\n  });\n\n  await langfuse.flushAsync();\n}\n\n\u002F\u002F Score with granular rating (1-5 stars)\nasync function submitStarRating(\n  traceId: string,\n  stars: number,\n  observationId?: string\n) {\n  langfuse.score({\n    traceId,\n    observationId, \u002F\u002F Optional: score a specific generation\n    name: 'star-rating',\n    value: stars \u002F 5, \u002F\u002F Normalize to 0-1\n    comment: `${stars}\u002F5 stars`,\n  });\n}\n```\n\n### Step 2: Automated Evaluation Functions\n```typescript\n\u002F\u002F Evaluate response quality automatically\nfunction evaluateResponse(response: string, expectedTopics: string[]): number {\n  const topicsFound = expectedTopics.filter(topic =>\n    response.toLowerCase().includes(topic.toLowerCase())\n  );\n  return topicsFound.length \u002F expectedTopics.length;\n}\n\nfunction evaluateLength(response: string, minWords: number, maxWords: number): number {\n  const wordCount = response.split(\u002F\\s+\u002F).length;\n  if (wordCount \u003C minWords) return wordCount \u002F minWords;\n  if (wordCount > maxWords) return maxWords \u002F wordCount;\n  return 1.0;\n}\n\n\u002F\u002F Score traces automatically after generation\nasync function autoScore(\n  traceId: string,\n  generationId: string,\n  response: string,\n  expectedTopics: string[]\n) {\n  const topicScore = evaluateResponse(response, expectedTopics);\n  const lengthScore = evaluateLength(response, 50, 500);  # HTTP 500 Internal Server Error\n\n  langfuse.score({\n    traceId,\n    observationId: generationId,\n    name: 'topic-coverage',\n    value: topicScore,\n  });\n\n  langfuse.score({\n    traceId,\n    observationId: generationId,\n    name: 'length-quality',\n    value: lengthScore,\n  });\n\n  await langfuse.flushAsync();\n}\n```\n\n### Step 3: LLM-as-Judge Evaluation\n```typescript\nimport OpenAI from 'openai';\n\nconst openai = new OpenAI();\n\nasync function llmJudge(\n  question: string,\n  response: string,\n  criteria: string\n): Promise\u003C{ score: number; reasoning: string }> {\n  const result = await openai.chat.completions.create({\n    model: 'gpt-4o-mini',\n    temperature: 0,\n    messages: [\n      {\n        role: 'system',\n        content: `You are an AI evaluator. Score the response on a scale of 0-10 based on: ${criteria}.\nReturn JSON: {\"score\": \u003Cnumber>, \"reasoning\": \"\u003Cexplanation>\"}`,\n      },\n      {\n        role: 'user',\n        content: `Question: ${question}\\nResponse: ${response}`,\n      },\n    ],\n    response_format: { type: 'json_object' },\n  });\n\n  return JSON.parse(result.choices[0].message.content || '{}');\n}\n\n\u002F\u002F Use with Langfuse scoring\nasync function judgeAndScore(traceId: string, question: string, response: string) {\n  const evaluation = await llmJudge(question, response, 'accuracy, helpfulness, and clarity');\n\n  langfuse.score({\n    traceId,\n    name: 'llm-judge',\n    value: evaluation.score \u002F 10, \u002F\u002F Normalize to 0-1\n    comment: evaluation.reasoning,\n  });\n}\n```\n\n### Step 4: A\u002FB Prompt Comparison\n```typescript\nasync function comparePrompts(\n  testCases: Array\u003C{ input: string; expectedTopics: string[] }>,\n  promptA: string,\n  promptB: string\n) {\n  const results = { promptA: { scores: [] as number[] }, promptB: { scores: [] as number[] } };\n\n  for (const testCase of testCases) {\n    \u002F\u002F Test Prompt A\n    const traceA = langfuse.trace({ name: 'ab-test-prompt-a' });\n    const genA = traceA.generation({\n      name: 'generate',\n      input: testCase.input,\n      model: 'gpt-4o-mini',\n      metadata: { promptVersion: 'A' },\n    });\n\n    const responseA = await callLLM(promptA, testCase.input);\n    genA.end({ output: responseA });\n\n    const scoreA = evaluateResponse(responseA, testCase.expectedTopics);\n    traceA.score({ name: 'topic-coverage', value: scoreA });\n    results.promptA.scores.push(scoreA);\n\n    \u002F\u002F Test Prompt B\n    const traceB = langfuse.trace({ name: 'ab-test-prompt-b' });\n    const responseB = await callLLM(promptB, testCase.input);\n    const scoreB = evaluateResponse(responseB, testCase.expectedTopics);\n    traceB.score({ name: 'topic-coverage', value: scoreB });\n    results.promptB.scores.push(scoreB);\n  }\n\n  const avgA = results.promptA.scores.reduce((a, b) => a + b, 0) \u002F results.promptA.scores.length;\n  const avgB = results.promptB.scores.reduce((a, b) => a + b, 0) \u002F results.promptB.scores.length;\n\n  await langfuse.flushAsync();\n  return { promptA: avgA.toFixed(3), promptB: avgB.toFixed(3), winner: avgA > avgB ? 'A' : 'B' };\n}\n```\n\n## Error Handling\n| Issue | Cause | Solution |\n|-------|-------|----------|\n| Scores not appearing | Flush not called | Always call `flushAsync` after scoring |\n| Score out of range | Value not normalized | Normalize all scores to 0-1 range |\n| LLM judge inconsistent | High temperature | Set temperature to 0 for evaluations |\n| Missing trace ID | Score submitted without trace | Ensure trace exists before scoring |\n\n## Examples\n\n### Quick Feedback Widget Integration\n```typescript\n\u002F\u002F API endpoint for frontend feedback\napp.post('\u002Fapi\u002Ffeedback', async (req, res) => {\n  const { traceId, rating, comment } = req.body;\n  await submitUserFeedback(traceId, rating, comment);\n  res.json({ success: true });\n});\n```\n\n## Resources\n- [Langfuse Scores](https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fscores)\n- [Langfuse Evaluation](https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fscores\u002Fmodel-based-evals)\n- [Langfuse Datasets](https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fdatasets)\n\n## Output\n\n- Configuration files or code changes applied to the project\n- Validation report confirming correct implementation\n- Summary of changes made and their rationale","# Langfuse Core Workflow B: Evaluation & Scoring\n\n## Overview\nImplement LLM output evaluation and scoring with Langfuse. Covers manual user feedback collection, automated evaluation functions, A\u002FB testing prompts with score comparison, and building evaluation datasets.\n\n## Prerequisites\n- Langfuse instance (Cloud or self-hosted)\n- Langfuse SDK configured with API keys\n- Traces already being collected (see workflow A)\n- Understanding of evaluation metrics\n\n## Instructions\n\n### Step 1: Manual User Feedback Scoring\n```typescript\nimport { Langfuse } from 'langfuse';\n\nconst langfuse = new Langfuse();\n\n\u002F\u002F Score a specific trace with user feedback\nasync function submitUserFeedback(\n  traceId: string,\n  rating: 'positive' | 'negative',\n  comment?: string\n) {\n  langfuse.score({\n    traceId,\n    name: 'user-feedback',\n    value: rating === 'positive' ? 1 : 0,\n    comment,\n  });\n\n  await langfuse.flushAsync();\n}\n\n\u002F\u002F Score with granular rating (1-5 stars)\nasync function submitStarRating(\n  traceId: string,\n  stars: number,\n  observationId?: string\n) {\n  langfuse.score({\n    traceId,\n    observationId, \u002F\u002F Optional: score a specific generation\n    name: 'star-rating',\n    value: stars \u002F 5, \u002F\u002F Normalize to 0-1\n    comment: `${stars}\u002F5 stars`,\n  });\n}\n```\n\n### Step 2: Automated Evaluation Functions\n```typescript\n\u002F\u002F Evaluate response quality automatically\nfunction evaluateResponse(response: string, expectedTopics: string[]): number {\n  const topicsFound = expectedTopics.filter(topic =>\n    response.toLowerCase().includes(topic.toLowerCase())\n  );\n  return topicsFound.length \u002F expectedTopics.length;\n}\n\nfunction evaluateLength(response: string, minWords: number, maxWords: number): number {\n  const wordCount = response.split(\u002F\\s+\u002F).length;\n  if (wordCount \u003C minWords) return wordCount \u002F minWords;\n  if (wordCount > maxWords) return maxWords \u002F wordCount;\n  return 1.0;\n}\n\n\u002F\u002F Score traces automatically after generation\nasync function autoScore(\n  traceId: string,\n  generationId: string,\n  response: string,\n  expectedTopics: string[]\n) {\n  const topicScore = evaluateResponse(response, expectedTopics);\n  const lengthScore = evaluateLength(response, 50, 500);  # HTTP 500 Internal Server Error\n\n  langfuse.score({\n    traceId,\n    observationId: generationId,\n    name: 'topic-coverage',\n    value: topicScore,\n  });\n\n  langfuse.score({\n    traceId,\n    observationId: generationId,\n    name: 'length-quality',\n    value: lengthScore,\n  });\n\n  await langfuse.flushAsync();\n}\n```\n\n### Step 3: LLM-as-Judge Evaluation\n```typescript\nimport OpenAI from 'openai';\n\nconst openai = new OpenAI();\n\nasync function llmJudge(\n  question: string,\n  response: string,\n  criteria: string\n): Promise\u003C{ score: number; reasoning: string }> {\n  const result = await openai.chat.completions.create({\n    model: 'gpt-4o-mini',\n    temperature: 0,\n    messages: [\n      {\n        role: 'system',\n        content: `You are an AI evaluator. Score the response on a scale of 0-10 based on: ${criteria}.\nReturn JSON: {\"score\": \u003Cnumber>, \"reasoning\": \"\u003Cexplanation>\"}`,\n      },\n      {\n        role: 'user',\n        content: `Question: ${question}\\nResponse: ${response}`,\n      },\n    ],\n    response_format: { type: 'json_object' },\n  });\n\n  return JSON.parse(result.choices[0].message.content || '{}');\n}\n\n\u002F\u002F Use with Langfuse scoring\nasync function judgeAndScore(traceId: string, question: string, response: string) {\n  const evaluation = await llmJudge(question, response, 'accuracy, helpfulness, and clarity');\n\n  langfuse.score({\n    traceId,\n    name: 'llm-judge',\n    value: evaluation.score \u002F 10, \u002F\u002F Normalize to 0-1\n    comment: evaluation.reasoning,\n  });\n}\n```\n\n### Step 4: A\u002FB Prompt Comparison\n```typescript\nasync function comparePrompts(\n  testCases: Array\u003C{ input: string; expectedTopics: string[] }>,\n  promptA: string,\n  promptB: string\n) {\n  const results = { promptA: { scores: [] as number[] }, promptB: { scores: [] as number[] } };\n\n  for (const testCase of testCases) {\n    \u002F\u002F Test Prompt A\n    const traceA = langfuse.trace({ name: 'ab-test-prompt-a' });\n    const genA = traceA.generation({\n      name: 'generate',\n      input: testCase.input,\n      model: 'gpt-4o-mini',\n      metadata: { promptVersion: 'A' },\n    });\n\n    const responseA = await callLLM(promptA, testCase.input);\n    genA.end({ output: responseA });\n\n    const scoreA = evaluateResponse(responseA, testCase.expectedTopics);\n    traceA.score({ name: 'topic-coverage', value: scoreA });\n    results.promptA.scores.push(scoreA);\n\n    \u002F\u002F Test Prompt B\n    const traceB = langfuse.trace({ name: 'ab-test-prompt-b' });\n    const responseB = await callLLM(promptB, testCase.input);\n    const scoreB = evaluateResponse(responseB, testCase.expectedTopics);\n    traceB.score({ name: 'topic-coverage', value: scoreB });\n    results.promptB.scores.push(scoreB);\n  }\n\n  const avgA = results.promptA.scores.reduce((a, b) => a + b, 0) \u002F results.promptA.scores.length;\n  const avgB = results.promptB.scores.reduce((a, b) => a + b, 0) \u002F results.promptB.scores.length;\n\n  await langfuse.flushAsync();\n  return { promptA: avgA.toFixed(3), promptB: avgB.toFixed(3), winner: avgA > avgB ? 'A' : 'B' };\n}\n```\n\n## Error Handling\n| Issue | Cause | Solution |\n|-------|-------|----------|\n| Scores not appearing | Flush not called | Always call `flushAsync` after scoring |\n| Score out of range | Value not normalized | Normalize all scores to 0-1 range |\n| LLM judge inconsistent | High temperature | Set temperature to 0 for evaluations |\n| Missing trace ID | Score submitted without trace | Ensure trace exists before scoring |\n\n## Examples\n\n### Quick Feedback Widget Integration\n```typescript\n\u002F\u002F API endpoint for frontend feedback\napp.post('\u002Fapi\u002Ffeedback', async (req, res) => {\n  const { traceId, rating, comment } = req.body;\n  await submitUserFeedback(traceId, rating, comment);\n  res.json({ success: true });\n});\n```\n\n## Resources\n- [Langfuse Scores](https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fscores)\n- [Langfuse Evaluation](https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fscores\u002Fmodel-based-evals)\n- [Langfuse Datasets](https:\u002F\u002Flangfuse.com\u002Fdocs\u002Fdatasets)\n\n## Output\n\n- Configuration files or code changes applied to the project\n- Validation report confirming correct implementation\n- Summary of changes made and their rationale",{"name":5,"author":24,"license":15,"version":25,"description":26,"allowed-tools":27,"compatible-with":28},"Jeremy Longshore \u003Cjeremy@intentsolutions.io>","1.0.0","Execute Langfuse secondary workflow: Evaluation and scoring.\nUse when implementing LLM evaluation, adding user feedback,\nor setting up automated quality scoring for AI outputs.\nTrigger with phrases like \"langfuse evaluation\", \"langfuse scoring\",\n\"rate llm outputs\", \"langfuse feedback\", \"evaluate ai responses\".\n","Read, Write, Edit, Bash(npm:*), Grep","claude-code, codex, openclaw","main","c2b1fdccc779e99ac814e55fb69ca8d71215dd0f","https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection","https:\u002F\u002Fgithub.com\u002FSamuelChien","http:\u002F\u002Fsamuelchien.com\u002F","2026-05-26 08:27:14",{"source":8,"skillName":5,"commands":36,"prompt":46,"zipAvailable":47},[37,40,43],{"tool":38,"command":39},"npx","npx skills add SamuelChien\u002Fmega-skills-collection --skill langfuse-core-workflow-b",{"tool":41,"command":42},"bunx","bunx skills add SamuelChien\u002Fmega-skills-collection --skill langfuse-core-workflow-b",{"tool":44,"command":45},"pnpm","pnpm dlx skills add SamuelChien\u002Fmega-skills-collection --skill langfuse-core-workflow-b","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection\u002Fblob\u002Fmain\u002Flangfuse-core-workflow-b\u002FSKILL.md",true,[],[50,57,64,71,78,86],{"skillNo":51,"name":52,"nameZh":6,"description":53,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":54,"categories":55,"keywords":56,"publishedAt":19},"01M76R4WZQZAXGGXMQCA0FZ716","zustand-store-ts","Create Zustand stores with TypeScript, subscribeWithSelector middleware, and proper state\u002Faction separation. Use when building React state management, creating global stores, or implementing reactive state patterns with Zustand.","https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection\u002Fblob\u002Fmain\u002Fzustand-store-ts\u002FSKILL.md",[],[],{"skillNo":58,"name":59,"nameZh":6,"description":60,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":61,"categories":62,"keywords":63,"publishedAt":19},"01M76R4T9KENDH7FENGZQHBW4G","zustand-store-creator","Zustand Store Creator - Auto-activating skill for Frontend Development.\nTriggers on: zustand store creator, zustand store creator\nPart of the Frontend Development skill category.","https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection\u002Fblob\u002Fmain\u002Fzustand-store-creator\u002FSKILL.md",[],[],{"skillNo":65,"name":66,"nameZh":6,"description":67,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":68,"categories":69,"keywords":70,"publishedAt":19},"01M76R3YWN84B7SANJ52X799ZE","zero-trust-config-helper","Zero Trust Config Helper - Auto-activating skill for Security Advanced.\nTriggers on: zero trust config helper, zero trust config helper\nPart of the Security Advanced skill category.","https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection\u002Fblob\u002Fmain\u002Fzero-trust-config-helper\u002FSKILL.md",[],[],{"skillNo":72,"name":73,"nameZh":6,"description":74,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":77,"publishedAt":19},"01M76R3HNSV2AEYSKHM915GJ58","zapier-integration-helper","Zapier Integration Helper - Auto-activating skill for Business Automation.\nTriggers on: zapier integration helper, zapier integration helper\nPart of the Business Automation skill category.","https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection\u002Fblob\u002Fmain\u002Fzapier-integration-helper\u002FSKILL.md",[],[],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":82,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":19},"01M76R3DJSKAT4FA7DV4BF3GCE","zai-cli","Execute z.AI CLI providing vision, search, reader, and GitHub exploration via CLI and MCP.\nUse when user needs image\u002Fvideo analysis, OCR, UI-to-code conversion, error diagnosis,\nreal-time web search, web page to markdown extraction, or GitHub code exploration.\nTrigger with phrases like \"analyze this image\", \"search the web for\", \"read this page\",\n\"explore this repo\", or \"use zai\". Requires Z_AI_API_KEY.","Apache-2.0","https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection\u002Fblob\u002Fmain\u002Fzai-cli\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":19},"01M76R2KAM699XE9AJVAWE4H2E","yt-research","Research competitor YouTube channels, niches, and trending topics for your content strategy.\nUse this skill whenever the user says \"research channels\", \"analyze competitors\", \"find trending topics\",\n\"niche analysis\", \"competitive research\", \"what are other creators doing\", \"scrape YouTube channels\",\nor wants to understand the competitive landscape for a specific tool or topic area. Use when working with yt research. Trigger with 'yt', 'research'.","https:\u002F\u002Fgithub.com\u002FSamuelChien\u002Fmega-skills-collection\u002Fblob\u002Fmain\u002Fyt-research\u002FSKILL.md",[],[]]