[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BWSWEWTV72EVCAH06RPR8M":3,"skill-same:01M7BWSWEWTV72EVCAH06RPR8M":57},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":28,"commitSha":29,"skillDirectory":30,"forksCount":31,"repositoryHtmlUrl":32,"repositoryHomepageUrl":6,"ownerHtmlUrl":33,"ownerBlogUrl":34,"ownerDescription":35,"updatedAt":36,"install":37,"resources":50},"01M7BWSWEWTV72EVCAH06RPR8M","online-evals",null,"Attach judges to config variations for automatic LLM-as-a-judge evaluation. Create custom judges, configure sampling rates, and monitor quality scores.","CognitionAI\u002Fdevin-marketplace","CognitionAI","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F163493330?v=4","01M7BVSRD98NC4FG8T3EZS6V5M","Cognition","01M7BVSRDX111C2XPWJ0E6MY26",29,"MIT","https:\u002F\u002Fgithub.com\u002FCognitionAI\u002Fdevin-marketplace\u002Fblob\u002Fmain\u002Fplugins\u002Flaunchdarkly\u002Fskills\u002Fagentcontrol\u002Fonline-evals\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: online-evals\ndescription: Attach judges to config variations for automatic LLM-as-a-judge evaluation. Create custom judges, configure sampling rates, and monitor quality scores.\ncompatibility: Requires LaunchDarkly API access token with ai-configs:write permission. SDK versions Python v0.20.0+ or Node.js v0.20.0+ for automatic metric recording and the consolidated `track_judge_result` \u002F `trackJudgeResult` API.\nmetadata:\n  author: launchdarkly\n  version: \"0.1.0\"\n---\n\n# Config Online Evaluations\n\nAttach judges to config variations for automatic quality scoring using LLM-as-a-judge methodology. Judges evaluate responses and return scores between 0.0 and 1.0.\n\n## Prerequisites\n\n- LaunchDarkly account with AgentControl enabled\n- API access token with write permissions\n- Existing config with variations (use `configs-create` skill)\n- For automatic metric recording and the consolidated judge-result API: Python AI SDK v0.20.0+ or Node.js AI SDK v0.20.0+\n\n## API Key Detection\n\n1. **Check environment variables** - `LAUNCHDARKLY_API_KEY`, `LAUNCHDARKLY_API_TOKEN`, `LD_API_KEY`\n2. **Check MCP config** - Claude: `~\u002F.claude\u002Fconfig.json` -> `mcpServers.launchdarkly.env.LAUNCHDARKLY_API_KEY`\n3. **Prompt user** - Only if detection fails\n\n## Core Concepts\n\n### What Are Judges?\n\nJudges are specialized configs in **judge mode** that evaluate responses from other configs. They use an LLM to score outputs and return structured results:\n\n```json\n{\n  \"score\": 0.85,\n  \"reasoning\": \"Answered correctly with one minor omission\"\n}\n```\n\n### Built-in Judges\n\nLaunchDarkly provides three pre-configured judges:\n\n| Judge | Metric Key | Measures |\n|-------|-----------|----------|\n| Accuracy | `$ld:ai:judge:accuracy` | How correct and grounded the response is |\n| Relevance | `$ld:ai:judge:relevance` | How well it addresses the user request |\n| Toxicity | `$ld:ai:judge:toxicity` | Harmful or unsafe phrasing (lower = safer) |\n\n### Completion Mode Only\n\nJudges can only be attached to **completion mode** configs in the UI. For agent mode or custom pipelines, use programmatic evaluation via the SDK.\n\n### Restrictions\n\n- Cannot attach judges to judges (no recursion)\n- Cannot attach multiple judges with the same metric key to a single variation\n- Cannot view\u002Fedit model parameters or tools on judge variations\n\n## Workflow\n\n### Step 1: Create Custom Judges (Optional)\n\nFor domain-specific evaluation, create judge configs:\n\n```bash\n# Create judge config\ncurl -X POST \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\" \\\n  -H \"Authorization: {api_token}\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -H \"LD-API-Version: beta\" \\\n  -d '{\n    \"key\": \"security-judge\",\n    \"name\": \"Security Judge\",\n    \"mode\": \"judge\",\n    \"evaluationMetricKey\": \"security\",\n    \"isInverted\": false\n  }'\n```\n\n> **Note:** Set `isInverted: true` for metrics like toxicity where 0.0 is better.\n\nThen add a variation with the evaluation prompt:\n\n```bash\ncurl -X POST \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\u002Fsecurity-judge\u002Fvariations\" \\\n  -H \"Authorization: {api_token}\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -H \"LD-API-Version: beta\" \\\n  -d '{\n    \"key\": \"default\",\n    \"name\": \"Default\",\n    \"messages\": [\n      {\n        \"role\": \"system\",\n        \"content\": \"You are a security auditor. Score from 0.0 to 1.0:\\n- 1.0: No security issues\\n- 0.7-0.9: Minor issues\\n- 0.4-0.6: Moderate issues\\n- 0.1-0.3: Serious vulnerabilities\\n- 0.0: Critical vulnerabilities\\n\\nCheck for: SQL injection, XSS, hardcoded secrets, command injection.\"\n      }\n    ],\n    \"modelConfigKey\": \"OpenAI.gpt-4o-mini\",\n    \"model\": {\n      \"parameters\": {\n        \"temperature\": 0.3\n      }\n    }\n  }'\n```\n\n### Step 2: Attach Judges to Variations\n\nUse the variation PATCH endpoint:\n\n```bash\ncurl -X PATCH \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\u002F{configKey}\u002Fvariations\u002F{variationKey}\" \\\n  -H \"Authorization: {api_token}\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -H \"LD-API-Version: beta\" \\\n  -d '{\n    \"judgeConfiguration\": {\n      \"judges\": [\n        {\"judgeConfigKey\": \"security-judge\", \"samplingRate\": 1.0},\n        {\"judgeConfigKey\": \"api-contract-judge\", \"samplingRate\": 0.5}\n      ]\n    }\n  }'\n```\n\n> **Important:** The `judges` array **replaces all existing** judge attachments. An empty array removes all judges.\n\n### Step 3: Set Fallthrough on Judges\n\nEach judge config needs its fallthrough set to the enabled variation. Configs default to the \"disabled\" variation (index 0).\n\n> **Note:** `turnTargetingOn` does not work for configs. Use `updateFallthroughVariationOrRollout` instead.\n\n```bash\n# First get the variation ID for \"Default\" from GET targeting response\ncurl -X PATCH \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\u002Fsecurity-judge\u002Ftargeting\" \\\n  -H \"Authorization: {api_token}\" \\\n  -H \"Content-Type: application\u002Fjson; domain-model=launchdarkly.semanticpatch\" \\\n  -H \"LD-API-Version: beta\" \\\n  -d '{\n    \"environmentKey\": \"production\",\n    \"instructions\": [{\n      \"kind\": \"updateFallthroughVariationOrRollout\",\n      \"variationId\": \"your-default-variation-uuid\"\n    }]\n  }'\n```\n\n## Python Implementation\n\n```python\nimport requests\nimport os\nfrom typing import Optional\n\nclass AIConfigJudges:\n    \"\"\"Manager for config judge attachments\"\"\"\n\n    def __init__(self, api_token: str, project_key: str):\n        self.api_token = api_token\n        self.project_key = project_key\n        self.base_url = \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\"\n        self.headers = {\n            \"Authorization\": api_token,\n            \"Content-Type\": \"application\u002Fjson\",\n            \"LD-API-Version\": \"beta\"\n        }\n\n    def attach_judges(self, config_key: str, variation_key: str,\n                      judges: list[dict]) -> dict:\n        \"\"\"\n        Attach judges to a variation.\n\n        Args:\n            config_key: config key\n            variation_key: Variation key\n            judges: List of {\"judgeConfigKey\": str, \"samplingRate\": float}\n        \"\"\"\n        url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\u002F{config_key}\u002Fvariations\u002F{variation_key}\"\n\n        response = requests.patch(url, headers=self.headers, json={\n            \"judgeConfiguration\": {\"judges\": judges}\n        })\n\n        if response.status_code == 200:\n            print(f\"[OK] Attached {len(judges)} judges to {config_key}\u002F{variation_key}\")\n            return response.json()\n        print(f\"[ERROR] {response.status_code}: {response.text}\")\n        return {}\n\n    def create_judge(self, key: str, name: str, metric_key: str,\n                     system_prompt: str, model: str = \"OpenAI.gpt-4o-mini\",\n                     is_inverted: bool = False) -> dict:\n        \"\"\"\n        Create a judge config.\n\n        Args:\n            key: Judge config key\n            name: Display name\n            metric_key: Metric key for scoring (appears as $ld:ai:judge:{metric_key})\n            system_prompt: Evaluation instructions\n            is_inverted: True if lower scores are better (e.g., toxicity)\n        \"\"\"\n        # Create config\n        config_url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\"\n        response = requests.post(config_url, headers=self.headers, json={\n            \"key\": key,\n            \"name\": name,\n            \"mode\": \"judge\",\n            \"evaluationMetricKey\": metric_key,\n            \"isInverted\": is_inverted\n        })\n\n        if response.status_code not in [200, 201]:\n            print(f\"[ERROR] Creating config: {response.text}\")\n            return {}\n\n        # Create variation\n        var_url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\u002F{key}\u002Fvariations\"\n        response = requests.post(var_url, headers=self.headers, json={\n            \"key\": \"default\",\n            \"name\": \"Default\",\n            \"messages\": [{\"role\": \"system\", \"content\": system_prompt}],\n            \"modelConfigKey\": model,\n            \"model\": {\"parameters\": {\"temperature\": 0.3}}\n        })\n\n        if response.status_code in [200, 201]:\n            print(f\"[OK] Created judge: {key}\")\n            return response.json()\n        print(f\"[ERROR] Creating variation: {response.text}\")\n        return {}\n\n    def set_fallthrough(self, config_key: str, environment: str,\n                        variation_key: str = \"default\") -> bool:\n        \"\"\"\n        Set fallthrough to enable a judge config.\n\n        Note: turnTargetingOn doesn't work for configs. Instead, set the\n        fallthrough from disabled (index 0) to the enabled variation.\n        \"\"\"\n        # Get variation ID\n        url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\u002F{config_key}\u002Ftargeting\"\n        response = requests.get(url, headers=self.headers)\n\n        if response.status_code != 200:\n            print(f\"[ERROR] {response.status_code}: {response.text}\")\n            return False\n\n        targeting = response.json()\n        variation_id = None\n        for var in targeting.get(\"variations\", []):\n            if var.get(\"key\") == variation_key or var.get(\"name\") == variation_key:\n                variation_id = var.get(\"_id\")\n                break\n\n        if not variation_id:\n            print(f\"[ERROR] Variation '{variation_key}' not found\")\n            return False\n\n        # Set fallthrough\n        response = requests.patch(url, headers={\n            **self.headers,\n            \"Content-Type\": \"application\u002Fjson; domain-model=launchdarkly.semanticpatch\"\n        }, json={\n            \"environmentKey\": environment,\n            \"instructions\": [{\n                \"kind\": \"updateFallthroughVariationOrRollout\",\n                \"variationId\": variation_id\n            }]\n        })\n\n        if response.status_code == 200:\n            print(f\"[OK] Fallthrough set for {config_key}\")\n            return True\n        print(f\"[ERROR] {response.status_code}: {response.text}\")\n        return False\n```\n\n## SDK: Automatic Evaluation\n\nWhen using `create_model()` + `run()`, attached judges evaluate automatically:\n\n```python\nimport os\nimport json\nimport asyncio\nimport ldclient\nfrom ldclient import Context\nfrom ldclient.config import Config\nfrom ldai import LDAIClient, AICompletionConfigDefault\n\nsdk_key = os.getenv('LAUNCHDARKLY_SDK_KEY')\nai_config_key = os.getenv('LAUNCHDARKLY_AI_CONFIG_KEY', 'sample-ai-config')\n\nasync def async_main():\n    ldclient.set_config(Config(sdk_key))\n    aiclient = LDAIClient(ldclient.get())\n\n    context = (\n        Context.builder('example-user-key')\n        .kind('user')\n        .name('Sandy')\n        .build()\n    )\n\n    default_value = AICompletionConfigDefault(enabled=False)\n\n    # create_model() initializes with judges from Config\n    model = await aiclient.create_model(ai_config_key, context, default_value, {})\n\n    if not model:\n        print(f\"agent configuration not enabled for: {ai_config_key}\")\n        return\n\n    user_input = 'How can LaunchDarkly help me?'\n\n    # run() automatically evaluates with attached judges\n    result = await model.run(user_input)\n    print(\"Response:\", result.content)\n\n    # Await evaluation results\n    if result.evaluations and len(result.evaluations) > 0:\n        eval_results = await asyncio.gather(*result.evaluations)\n        results_to_display = [\n            r.to_dict() if r is not None else \"not evaluated\"\n            for r in eval_results\n        ]\n        print(\"Judge results:\")\n        print(json.dumps(results_to_display, indent=2, default=str))\n\n    # Always flush events before closing — trailing events are at risk of being\n    # lost otherwise, in short-lived scripts and long-running services alike.\n    ldclient.get().flush()\n    ldclient.get().close()\n```\n\n## SDK: Direct Judge Evaluation\n\nFor agent mode or custom pipelines, evaluate input\u002Foutput pairs directly:\n\n```python\nimport os\nimport json\nimport asyncio\nimport ldclient\nfrom ldclient import Context\nfrom ldclient.config import Config\nfrom ldai import LDAIClient, AIJudgeConfigDefault\n\nsdk_key = os.getenv('LAUNCHDARKLY_SDK_KEY')\njudge_key = os.getenv('LAUNCHDARKLY_AI_JUDGE_KEY', 'sample-ai-judge-accuracy')\n\nasync def async_main():\n    ldclient.set_config(Config(sdk_key))\n    aiclient = LDAIClient(ldclient.get())\n\n    context = (\n        Context.builder('example-user-key')\n        .kind('user')\n        .name('Sandy')\n        .build()\n    )\n\n    judge_default_value = AIJudgeConfigDefault(enabled=False)\n\n    # Get judge configuration from LaunchDarkly\n    judge = aiclient.create_judge(judge_key, context, judge_default_value)\n\n    if not judge:\n        print(f\"agent judge configuration not enabled for key: {judge_key}\")\n        return\n\n    input_text = 'You are a helpful assistant. How can you help me?'\n    output_text = 'I can answer any question you have.'\n\n    # Evaluate the input\u002Foutput pair — returns a JudgeResult.\n    judge_result = await judge.evaluate(input_text, output_text)\n\n    if not judge_result.sampled:\n        print(\"Judge evaluation was skipped (sample rate or configuration issue)\")\n        return\n\n    # Track the consolidated result on the Config tracker if needed:\n    # tracker = ai_config.create_tracker()\n    # tracker.track_judge_result(judge_result)\n\n    print(\"Judge Result:\")\n    print(json.dumps(judge_result.to_dict(), default=str))\n\n    # Always flush events before closing — trailing events are at risk of being\n    # lost otherwise, in short-lived scripts and long-running services alike.\n    ldclient.get().flush()\n    ldclient.get().close()\n```\n\n> **Note:** Direct evaluation does not automatically record metrics. Obtain a tracker via `ai_config.create_tracker()` \u002F `aiConfig.createTracker()` and call `tracker.track_judge_result(result)` \u002F `tracker.trackJudgeResult(result)` to record scores for the config you're evaluating.\n\n## Sampling Rates\n\nEach evaluated response sends an additional request to your model provider, increasing token usage and costs. Start with a lower sampling percentage and increase only if you need more evaluation coverage.\n\nYou can adjust sampling rates at any time from the Judges section of a variation, or disable a judge by setting its sampling to 0%.\n\n## Viewing Results\n\n1. Navigate to **configs** > select your config\n2. Click **Monitoring** tab\n3. Select **Evaluator metrics** from dropdown\n4. View scores by variation and time range\n\nResults appear within 1-2 minutes of evaluation.\n\n## Use in Guardrails and Experiments\n\nEvaluation metrics integrate with:\n- **Guarded rollouts**: Pause\u002Frevert when scores fall below threshold\n- **Experiments**: Compare variations using evaluation metrics as goals\n\n## Error Handling\n\n| Status | Cause | Solution |\n|--------|-------|----------|\n| 404 | Config\u002Fvariation not found | Verify keys exist |\n| 400 | Invalid judge config | Check judgeConfigKey exists |\n| 403 | Insufficient permissions | Check API token permissions |\n| 422 | Duplicate metric key | Cannot attach multiple judges with same metric key |\n\n## Next Steps\n\nAfter attaching judges:\n1. **Set fallthrough** on judge configs to an enabled variation (required)\n2. **Monitor results** in Monitoring tab\n3. **Adjust sampling** based on cost\u002Fcoverage needs\n4. **Set up guarded rollouts** for automatic regression detection\n\n## Related Skills\n\n- `configs-create` - Create configs and judges\n- `configs-targeting` - Configure targeting rules\n- `configs-variations` - Manage variations\n\n## References\n\n- [Online Evaluations](https:\u002F\u002Fdocs.launchdarkly.com\u002Fhome\u002Fai-configs\u002Fonline-evaluations.md)\n- [Custom Judges](https:\u002F\u002Fdocs.launchdarkly.com\u002Fhome\u002Fai-configs\u002Fcustom-judges.md)\n\n**Python SDK examples:**\n- [create_judge_example.py](https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fhello-python-ai\u002Fblob\u002Fmain\u002Ffeatures\u002Fcreate_judge\u002Fcreate_judge_example.py) - Evaluate input\u002Foutput pairs directly via `create_judge` + `evaluate`\n- [create_model_example.py](https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fhello-python-ai\u002Fblob\u002Fmain\u002Ffeatures\u002Fcreate_model\u002Fcreate_model_example.py) - Automatic evaluation with `create_model` + `run` (attached judges fire during the run)\n\n**Node.js SDK examples:**\n- [features\u002Fcreate-judge](https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fjs-core\u002Fblob\u002Fmain\u002Fpackages\u002Fsdk\u002Fserver-ai\u002Fexamples\u002Ffeatures\u002Fcreate-judge\u002Fsrc\u002Findex.ts) - Evaluate input\u002Foutput pairs directly via `createJudge` + `evaluate`\n- [features\u002Fcreate-model](https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fjs-core\u002Fblob\u002Fmain\u002Fpackages\u002Fsdk\u002Fserver-ai\u002Fexamples\u002Ffeatures\u002Fcreate-model\u002Fsrc\u002Findex.ts) - Automatic evaluation with `createModel` + `run` (attached judges fire during the run)\n","# Config Online Evaluations\n\nAttach judges to config variations for automatic quality scoring using LLM-as-a-judge methodology. Judges evaluate responses and return scores between 0.0 and 1.0.\n\n## Prerequisites\n\n- LaunchDarkly account with AgentControl enabled\n- API access token with write permissions\n- Existing config with variations (use `configs-create` skill)\n- For automatic metric recording and the consolidated judge-result API: Python AI SDK v0.20.0+ or Node.js AI SDK v0.20.0+\n\n## API Key Detection\n\n1. **Check environment variables** - `LAUNCHDARKLY_API_KEY`, `LAUNCHDARKLY_API_TOKEN`, `LD_API_KEY`\n2. **Check MCP config** - Claude: `~\u002F.claude\u002Fconfig.json` -> `mcpServers.launchdarkly.env.LAUNCHDARKLY_API_KEY`\n3. **Prompt user** - Only if detection fails\n\n## Core Concepts\n\n### What Are Judges?\n\nJudges are specialized configs in **judge mode** that evaluate responses from other configs. They use an LLM to score outputs and return structured results:\n\n```json\n{\n  \"score\": 0.85,\n  \"reasoning\": \"Answered correctly with one minor omission\"\n}\n```\n\n### Built-in Judges\n\nLaunchDarkly provides three pre-configured judges:\n\n| Judge | Metric Key | Measures |\n|-------|-----------|----------|\n| Accuracy | `$ld:ai:judge:accuracy` | How correct and grounded the response is |\n| Relevance | `$ld:ai:judge:relevance` | How well it addresses the user request |\n| Toxicity | `$ld:ai:judge:toxicity` | Harmful or unsafe phrasing (lower = safer) |\n\n### Completion Mode Only\n\nJudges can only be attached to **completion mode** configs in the UI. For agent mode or custom pipelines, use programmatic evaluation via the SDK.\n\n### Restrictions\n\n- Cannot attach judges to judges (no recursion)\n- Cannot attach multiple judges with the same metric key to a single variation\n- Cannot view\u002Fedit model parameters or tools on judge variations\n\n## Workflow\n\n### Step 1: Create Custom Judges (Optional)\n\nFor domain-specific evaluation, create judge configs:\n\n```bash\n# Create judge config\ncurl -X POST \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\" \\\n  -H \"Authorization: {api_token}\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -H \"LD-API-Version: beta\" \\\n  -d '{\n    \"key\": \"security-judge\",\n    \"name\": \"Security Judge\",\n    \"mode\": \"judge\",\n    \"evaluationMetricKey\": \"security\",\n    \"isInverted\": false\n  }'\n```\n\n> **Note:** Set `isInverted: true` for metrics like toxicity where 0.0 is better.\n\nThen add a variation with the evaluation prompt:\n\n```bash\ncurl -X POST \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\u002Fsecurity-judge\u002Fvariations\" \\\n  -H \"Authorization: {api_token}\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -H \"LD-API-Version: beta\" \\\n  -d '{\n    \"key\": \"default\",\n    \"name\": \"Default\",\n    \"messages\": [\n      {\n        \"role\": \"system\",\n        \"content\": \"You are a security auditor. Score from 0.0 to 1.0:\\n- 1.0: No security issues\\n- 0.7-0.9: Minor issues\\n- 0.4-0.6: Moderate issues\\n- 0.1-0.3: Serious vulnerabilities\\n- 0.0: Critical vulnerabilities\\n\\nCheck for: SQL injection, XSS, hardcoded secrets, command injection.\"\n      }\n    ],\n    \"modelConfigKey\": \"OpenAI.gpt-4o-mini\",\n    \"model\": {\n      \"parameters\": {\n        \"temperature\": 0.3\n      }\n    }\n  }'\n```\n\n### Step 2: Attach Judges to Variations\n\nUse the variation PATCH endpoint:\n\n```bash\ncurl -X PATCH \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\u002F{configKey}\u002Fvariations\u002F{variationKey}\" \\\n  -H \"Authorization: {api_token}\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -H \"LD-API-Version: beta\" \\\n  -d '{\n    \"judgeConfiguration\": {\n      \"judges\": [\n        {\"judgeConfigKey\": \"security-judge\", \"samplingRate\": 1.0},\n        {\"judgeConfigKey\": \"api-contract-judge\", \"samplingRate\": 0.5}\n      ]\n    }\n  }'\n```\n\n> **Important:** The `judges` array **replaces all existing** judge attachments. An empty array removes all judges.\n\n### Step 3: Set Fallthrough on Judges\n\nEach judge config needs its fallthrough set to the enabled variation. Configs default to the \"disabled\" variation (index 0).\n\n> **Note:** `turnTargetingOn` does not work for configs. Use `updateFallthroughVariationOrRollout` instead.\n\n```bash\n# First get the variation ID for \"Default\" from GET targeting response\ncurl -X PATCH \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\u002Fprojects\u002F{projectKey}\u002Fai-configs\u002Fsecurity-judge\u002Ftargeting\" \\\n  -H \"Authorization: {api_token}\" \\\n  -H \"Content-Type: application\u002Fjson; domain-model=launchdarkly.semanticpatch\" \\\n  -H \"LD-API-Version: beta\" \\\n  -d '{\n    \"environmentKey\": \"production\",\n    \"instructions\": [{\n      \"kind\": \"updateFallthroughVariationOrRollout\",\n      \"variationId\": \"your-default-variation-uuid\"\n    }]\n  }'\n```\n\n## Python Implementation\n\n```python\nimport requests\nimport os\nfrom typing import Optional\n\nclass AIConfigJudges:\n    \"\"\"Manager for config judge attachments\"\"\"\n\n    def __init__(self, api_token: str, project_key: str):\n        self.api_token = api_token\n        self.project_key = project_key\n        self.base_url = \"https:\u002F\u002Fapp.launchdarkly.com\u002Fapi\u002Fv2\"\n        self.headers = {\n            \"Authorization\": api_token,\n            \"Content-Type\": \"application\u002Fjson\",\n            \"LD-API-Version\": \"beta\"\n        }\n\n    def attach_judges(self, config_key: str, variation_key: str,\n                      judges: list[dict]) -> dict:\n        \"\"\"\n        Attach judges to a variation.\n\n        Args:\n            config_key: config key\n            variation_key: Variation key\n            judges: List of {\"judgeConfigKey\": str, \"samplingRate\": float}\n        \"\"\"\n        url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\u002F{config_key}\u002Fvariations\u002F{variation_key}\"\n\n        response = requests.patch(url, headers=self.headers, json={\n            \"judgeConfiguration\": {\"judges\": judges}\n        })\n\n        if response.status_code == 200:\n            print(f\"[OK] Attached {len(judges)} judges to {config_key}\u002F{variation_key}\")\n            return response.json()\n        print(f\"[ERROR] {response.status_code}: {response.text}\")\n        return {}\n\n    def create_judge(self, key: str, name: str, metric_key: str,\n                     system_prompt: str, model: str = \"OpenAI.gpt-4o-mini\",\n                     is_inverted: bool = False) -> dict:\n        \"\"\"\n        Create a judge config.\n\n        Args:\n            key: Judge config key\n            name: Display name\n            metric_key: Metric key for scoring (appears as $ld:ai:judge:{metric_key})\n            system_prompt: Evaluation instructions\n            is_inverted: True if lower scores are better (e.g., toxicity)\n        \"\"\"\n        # Create config\n        config_url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\"\n        response = requests.post(config_url, headers=self.headers, json={\n            \"key\": key,\n            \"name\": name,\n            \"mode\": \"judge\",\n            \"evaluationMetricKey\": metric_key,\n            \"isInverted\": is_inverted\n        })\n\n        if response.status_code not in [200, 201]:\n            print(f\"[ERROR] Creating config: {response.text}\")\n            return {}\n\n        # Create variation\n        var_url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\u002F{key}\u002Fvariations\"\n        response = requests.post(var_url, headers=self.headers, json={\n            \"key\": \"default\",\n            \"name\": \"Default\",\n            \"messages\": [{\"role\": \"system\", \"content\": system_prompt}],\n            \"modelConfigKey\": model,\n            \"model\": {\"parameters\": {\"temperature\": 0.3}}\n        })\n\n        if response.status_code in [200, 201]:\n            print(f\"[OK] Created judge: {key}\")\n            return response.json()\n        print(f\"[ERROR] Creating variation: {response.text}\")\n        return {}\n\n    def set_fallthrough(self, config_key: str, environment: str,\n                        variation_key: str = \"default\") -> bool:\n        \"\"\"\n        Set fallthrough to enable a judge config.\n\n        Note: turnTargetingOn doesn't work for configs. Instead, set the\n        fallthrough from disabled (index 0) to the enabled variation.\n        \"\"\"\n        # Get variation ID\n        url = f\"{self.base_url}\u002Fprojects\u002F{self.project_key}\u002Fai-configs\u002F{config_key}\u002Ftargeting\"\n        response = requests.get(url, headers=self.headers)\n\n        if response.status_code != 200:\n            print(f\"[ERROR] {response.status_code}: {response.text}\")\n            return False\n\n        targeting = response.json()\n        variation_id = None\n        for var in targeting.get(\"variations\", []):\n            if var.get(\"key\") == variation_key or var.get(\"name\") == variation_key:\n                variation_id = var.get(\"_id\")\n                break\n\n        if not variation_id:\n            print(f\"[ERROR] Variation '{variation_key}' not found\")\n            return False\n\n        # Set fallthrough\n        response = requests.patch(url, headers={\n            **self.headers,\n            \"Content-Type\": \"application\u002Fjson; domain-model=launchdarkly.semanticpatch\"\n        }, json={\n            \"environmentKey\": environment,\n            \"instructions\": [{\n                \"kind\": \"updateFallthroughVariationOrRollout\",\n                \"variationId\": variation_id\n            }]\n        })\n\n        if response.status_code == 200:\n            print(f\"[OK] Fallthrough set for {config_key}\")\n            return True\n        print(f\"[ERROR] {response.status_code}: {response.text}\")\n        return False\n```\n\n## SDK: Automatic Evaluation\n\nWhen using `create_model()` + `run()`, attached judges evaluate automatically:\n\n```python\nimport os\nimport json\nimport asyncio\nimport ldclient\nfrom ldclient import Context\nfrom ldclient.config import Config\nfrom ldai import LDAIClient, AICompletionConfigDefault\n\nsdk_key = os.getenv('LAUNCHDARKLY_SDK_KEY')\nai_config_key = os.getenv('LAUNCHDARKLY_AI_CONFIG_KEY', 'sample-ai-config')\n\nasync def async_main():\n    ldclient.set_config(Config(sdk_key))\n    aiclient = LDAIClient(ldclient.get())\n\n    context = (\n        Context.builder('example-user-key')\n        .kind('user')\n        .name('Sandy')\n        .build()\n    )\n\n    default_value = AICompletionConfigDefault(enabled=False)\n\n    # create_model() initializes with judges from Config\n    model = await aiclient.create_model(ai_config_key, context, default_value, {})\n\n    if not model:\n        print(f\"agent configuration not enabled for: {ai_config_key}\")\n        return\n\n    user_input = 'How can LaunchDarkly help me?'\n\n    # run() automatically evaluates with attached judges\n    result = await model.run(user_input)\n    print(\"Response:\", result.content)\n\n    # Await evaluation results\n    if result.evaluations and len(result.evaluations) > 0:\n        eval_results = await asyncio.gather(*result.evaluations)\n        results_to_display = [\n            r.to_dict() if r is not None else \"not evaluated\"\n            for r in eval_results\n        ]\n        print(\"Judge results:\")\n        print(json.dumps(results_to_display, indent=2, default=str))\n\n    # Always flush events before closing — trailing events are at risk of being\n    # lost otherwise, in short-lived scripts and long-running services alike.\n    ldclient.get().flush()\n    ldclient.get().close()\n```\n\n## SDK: Direct Judge Evaluation\n\nFor agent mode or custom pipelines, evaluate input\u002Foutput pairs directly:\n\n```python\nimport os\nimport json\nimport asyncio\nimport ldclient\nfrom ldclient import Context\nfrom ldclient.config import Config\nfrom ldai import LDAIClient, AIJudgeConfigDefault\n\nsdk_key = os.getenv('LAUNCHDARKLY_SDK_KEY')\njudge_key = os.getenv('LAUNCHDARKLY_AI_JUDGE_KEY', 'sample-ai-judge-accuracy')\n\nasync def async_main():\n    ldclient.set_config(Config(sdk_key))\n    aiclient = LDAIClient(ldclient.get())\n\n    context = (\n        Context.builder('example-user-key')\n        .kind('user')\n        .name('Sandy')\n        .build()\n    )\n\n    judge_default_value = AIJudgeConfigDefault(enabled=False)\n\n    # Get judge configuration from LaunchDarkly\n    judge = aiclient.create_judge(judge_key, context, judge_default_value)\n\n    if not judge:\n        print(f\"agent judge configuration not enabled for key: {judge_key}\")\n        return\n\n    input_text = 'You are a helpful assistant. How can you help me?'\n    output_text = 'I can answer any question you have.'\n\n    # Evaluate the input\u002Foutput pair — returns a JudgeResult.\n    judge_result = await judge.evaluate(input_text, output_text)\n\n    if not judge_result.sampled:\n        print(\"Judge evaluation was skipped (sample rate or configuration issue)\")\n        return\n\n    # Track the consolidated result on the Config tracker if needed:\n    # tracker = ai_config.create_tracker()\n    # tracker.track_judge_result(judge_result)\n\n    print(\"Judge Result:\")\n    print(json.dumps(judge_result.to_dict(), default=str))\n\n    # Always flush events before closing — trailing events are at risk of being\n    # lost otherwise, in short-lived scripts and long-running services alike.\n    ldclient.get().flush()\n    ldclient.get().close()\n```\n\n> **Note:** Direct evaluation does not automatically record metrics. Obtain a tracker via `ai_config.create_tracker()` \u002F `aiConfig.createTracker()` and call `tracker.track_judge_result(result)` \u002F `tracker.trackJudgeResult(result)` to record scores for the config you're evaluating.\n\n## Sampling Rates\n\nEach evaluated response sends an additional request to your model provider, increasing token usage and costs. Start with a lower sampling percentage and increase only if you need more evaluation coverage.\n\nYou can adjust sampling rates at any time from the Judges section of a variation, or disable a judge by setting its sampling to 0%.\n\n## Viewing Results\n\n1. Navigate to **configs** > select your config\n2. Click **Monitoring** tab\n3. Select **Evaluator metrics** from dropdown\n4. View scores by variation and time range\n\nResults appear within 1-2 minutes of evaluation.\n\n## Use in Guardrails and Experiments\n\nEvaluation metrics integrate with:\n- **Guarded rollouts**: Pause\u002Frevert when scores fall below threshold\n- **Experiments**: Compare variations using evaluation metrics as goals\n\n## Error Handling\n\n| Status | Cause | Solution |\n|--------|-------|----------|\n| 404 | Config\u002Fvariation not found | Verify keys exist |\n| 400 | Invalid judge config | Check judgeConfigKey exists |\n| 403 | Insufficient permissions | Check API token permissions |\n| 422 | Duplicate metric key | Cannot attach multiple judges with same metric key |\n\n## Next Steps\n\nAfter attaching judges:\n1. **Set fallthrough** on judge configs to an enabled variation (required)\n2. **Monitor results** in Monitoring tab\n3. **Adjust sampling** based on cost\u002Fcoverage needs\n4. **Set up guarded rollouts** for automatic regression detection\n\n## Related Skills\n\n- `configs-create` - Create configs and judges\n- `configs-targeting` - Configure targeting rules\n- `configs-variations` - Manage variations\n\n## References\n\n- [Online Evaluations](https:\u002F\u002Fdocs.launchdarkly.com\u002Fhome\u002Fai-configs\u002Fonline-evaluations.md)\n- [Custom Judges](https:\u002F\u002Fdocs.launchdarkly.com\u002Fhome\u002Fai-configs\u002Fcustom-judges.md)\n\n**Python SDK examples:**\n- [create_judge_example.py](https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fhello-python-ai\u002Fblob\u002Fmain\u002Ffeatures\u002Fcreate_judge\u002Fcreate_judge_example.py) - Evaluate input\u002Foutput pairs directly via `create_judge` + `evaluate`\n- [create_model_example.py](https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fhello-python-ai\u002Fblob\u002Fmain\u002Ffeatures\u002Fcreate_model\u002Fcreate_model_example.py) - Automatic evaluation with `create_model` + `run` (attached judges fire during the run)\n\n**Node.js SDK examples:**\n- [features\u002Fcreate-judge](https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fjs-core\u002Fblob\u002Fmain\u002Fpackages\u002Fsdk\u002Fserver-ai\u002Fexamples\u002Ffeatures\u002Fcreate-judge\u002Fsrc\u002Findex.ts) - Evaluate input\u002Foutput pairs directly via `createJudge` + `evaluate`\n- [features\u002Fcreate-model](https:\u002F\u002Fgithub.com\u002Flaunchdarkly\u002Fjs-core\u002Fblob\u002Fmain\u002Fpackages\u002Fsdk\u002Fserver-ai\u002Fexamples\u002Ffeatures\u002Fcreate-model\u002Fsrc\u002Findex.ts) - Automatic evaluation with `createModel` + `run` (attached judges fire during the run)\n",{"name":5,"metadata":24,"description":7,"compatibility":27},{"author":25,"version":26},"launchdarkly","0.1.0","Requires LaunchDarkly API access token with ai-configs:write permission. SDK versions Python v0.20.0+ or Node.js v0.20.0+ for automatic metric recording and the consolidated `track_judge_result` \u002F `trackJudgeResult` API.","main","9836d94e88404aabb682086405c2aa9cba7657ac","plugins\u002Flaunchdarkly\u002Fskills\u002Fagentcontrol\u002Fonline-evals",156,"https:\u002F\u002Fgithub.com\u002FCognitionAI\u002Fdevin-marketplace","https:\u002F\u002Fgithub.com\u002FCognitionAI","https:\u002F\u002Fcognition.ai\u002F","Makers of Devin, the AI software engineer.","2026-11-13 07:08:23",{"source":8,"skillName":5,"commands":38,"prompt":48,"zipAvailable":49},[39,42,45],{"tool":40,"command":41},"npx","npx skills add CognitionAI\u002Fdevin-marketplace --skill online-evals",{"tool":43,"command":44},"bunx","bunx skills add CognitionAI\u002Fdevin-marketplace --skill online-evals",{"tool":46,"command":47},"pnpm","pnpm dlx skills add CognitionAI\u002Fdevin-marketplace --skill online-evals","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FCognitionAI\u002Fdevin-marketplace\u002Fblob\u002Fmain\u002Fplugins\u002Flaunchdarkly\u002Fskills\u002Fagentcontrol\u002Fonline-evals\u002FSKILL.md",true,[51],{"path":52,"resourceType":53,"resourceTypeDesc":54,"sizeBytes":55,"sourceHtmlUrl":56},"README.md","REFERENCE","参考文档",1290,"https:\u002F\u002Fgithub.com\u002FCognitionAI\u002Fdevin-marketplace\u002Fblob\u002Fmain\u002Fplugins\u002Flaunchdarkly\u002Fskills\u002Fagentcontrol\u002Fonline-evals\u002FREADME.md",[58,64,71,77,84,91],{"skillNo":59,"name":60,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":61,"categories":62,"keywords":63,"publishedAt":19},"01M7BYC4AJGM98P2G9JH7QS24Z","business-data","https:\u002F\u002Fgithub.com\u002FCognitionAI\u002Fdevin-marketplace\u002Fblob\u002Fmain\u002Fplugins\u002Fwindsor-ai\u002Fskills\u002Fbusiness-data\u002FSKILL.md",[],[],{"skillNo":65,"name":66,"nameZh":6,"description":67,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":68,"categories":69,"keywords":70,"publishedAt":19},"01M7BYBZ0CD4CBCPZHRZ551BNK","workflow","Creates durable, resumable workflows using Vercel's Workflow SDK. Use when building workflows that need to survive restarts, pause for external events, retry on failure, or coordinate multi-step operations over time. Triggers on mentions of \"workflow\", \"durable functions\", \"resumable\", \"workflow sdk\", \"queue\", \"event\", \"push\", \"subscribe\", or step-based orchestration.","https:\u002F\u002Fgithub.com\u002FCognitionAI\u002Fdevin-marketplace\u002Fblob\u002Fmain\u002Fplugins\u002Fvercel\u002Fskills\u002Fworkflow\u002Fupstream\u002FSKILL.md",[],[],{"skillNo":72,"name":66,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M7BYB1XSZHKHD8JZTQJ62JN0","Vercel Workflow SDK expert guidance. Use when building durable workflows, long-running tasks, API routes or agents that need pause\u002Fresume, retries, step-based execution, or crash-safe orchestration with Vercel Workflow.","https:\u002F\u002Fgithub.com\u002FCognitionAI\u002Fdevin-marketplace\u002Fblob\u002Fmain\u002Fplugins\u002Fvercel\u002Fskills\u002Fworkflow\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":19},"01M7BYASGMK935J33E21N5WTW0","verification","Full-story verification — infers what the user is building, then verifies the complete flow end-to-end: browser → API → data → response. Triggers on dev server start and 'why isn't this working' signals.","https:\u002F\u002Fgithub.com\u002FCognitionAI\u002Fdevin-marketplace\u002Fblob\u002Fmain\u002Fplugins\u002Fvercel\u002Fskills\u002Fverification\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":19},"01M7BY9WJERKSKEH9HK6E71GW2","vercel-storage","Vercel storage expert guidance — Blob, Global Config (formerly Edge Config), and Marketplace storage (Neon Postgres, Upstash Redis). Use when choosing, configuring, or using data storage with Vercel applications.","https:\u002F\u002Fgithub.com\u002FCognitionAI\u002Fdevin-marketplace\u002Fblob\u002Fmain\u002Fplugins\u002Fvercel\u002Fskills\u002Fvercel-storage\u002FSKILL.md",[],[],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":19},"01M7BY9N034894YV2J9DCRZHYS","vercel-services","Configure and troubleshoot Vercel Services for multiple frontends and backends in one project. Use when composing a polyglot or multi-service application on one Vercel deployment; defining the `services` key, service-targeted rewrites, or service bindings in `vercel.json`; or running all services with `vercel dev`.","https:\u002F\u002Fgithub.com\u002FCognitionAI\u002Fdevin-marketplace\u002Fblob\u002Fmain\u002Fplugins\u002Fvercel\u002Fskills\u002Fvercel-services\u002FSKILL.md",[],[]]