[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M757M56NFSDHNFKQR1BWA953":3,"skill-same:01M757M56NFSDHNFKQR1BWA953":53},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":29,"commitSha":30,"skillDirectory":31,"forksCount":32,"repositoryHtmlUrl":33,"repositoryHomepageUrl":34,"ownerHtmlUrl":35,"ownerBlogUrl":36,"ownerDescription":37,"updatedAt":38,"install":39,"resources":52},"01M757M56NFSDHNFKQR1BWA953","llm-observability-and-cost",null,"Reference-grade guide to production LLM observability and cost attribution — distributed traces and spans for multi-step agents, OpenTelemetry GenAI semantic conventions, the metrics\u002Fpercentiles that matter, drift detection with online evals, and per-feature\u002Ftenant\u002Fuser cost attribution via span metadata. Real tooling (LangSmith, Langfuse, Phoenix, Helicone, OTel), tables, do\u002Fdon't, and the failure modes that leave you blind.","jpoindexter\u002FVanta","jpoindexter","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F2729617?v=4","01M72VRRYY1CP7RPF3B7GHH0CR","01M757GWDBT6RD147ZZ79N4MEX",4,"MIT","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fai-engineering-skills\u002Fllm-observability-and-cost\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: llm-observability-and-cost\ndescription: Reference-grade guide to production LLM observability and cost attribution — distributed traces and spans for multi-step agents, OpenTelemetry GenAI semantic conventions, the metrics\u002Fpercentiles that matter, drift detection with online evals, and per-feature\u002Ftenant\u002Fuser cost attribution via span metadata. Real tooling (LangSmith, Langfuse, Phoenix, Helicone, OTel), tables, do\u002Fdon't, and the failure modes that leave you blind.\ntags: [ai-engineering, observability, tracing, cost, monitoring]\n---\n# LLM Observability & Cost Attribution\n\nYou cannot debug, optimize, or cost what you cannot see. Traditional APM assumes deterministic, cheap, fast functions. LLM systems break all three assumptions: the same input yields different outputs, a single user request fans out into a dozen model and tool calls, every call costs real money proportional to tokens, and output quality silently drifts as models, prompts, and data change underneath you. Observability for LLM systems is therefore a first-class engineering discipline, not a dashboard you bolt on later — it is the only way to answer \"why did this request fail,\" \"which feature is burning our budget,\" and \"did quality regress after the deploy.\"\n\nHow to use this skill: instrument every LLM call as a span inside a request-level trace (§2), attach the metadata tags that make cost and quality attributable (§5), emit the metrics in §3, watch for drift in §4, and debug from traces in §6. Treat the failure modes in §9 as a pre-ship checklist.\n\n---\n\n## 1. Why LLM observability is its own discipline\n\n- **Non-determinism.** Temperature, sampling, and model updates mean the same prompt produces different responses. You cannot reproduce a bug from inputs alone — you need the *exact* request\u002Fresponse that was captured, including the model version that served it.\n- **Multi-step agents.** A modern request is a tree: planner → retrieval → tool call → sub-agent → synthesis. A failure three levels deep surfaces as a vague top-level error. Without nested traces you see the symptom, never the cause.\n- **Cost is variable and large.** Spend scales with tokens, not requests. A single runaway agent loop or a bloated context window can cost 100× a normal request. You must measure cost per call to control it.\n- **Quality drifts silently.** No exception is thrown when answers get worse. A provider ships a new model snapshot, your RAG corpus shifts, a prompt edit regresses one intent — and nothing alerts unless you measure quality online.\n\n**Concretely, what blindness costs you.** A support agent starts giving wrong answers after a Tuesday deploy. Without traces: you know error rate is flat (it didn't *error*, it answered confidently and wrongly), you can't reproduce the complaint (different output every run), you can't tell if it was the prompt edit, a model snapshot rollover, or stale retrieval, and you have no idea the bad path is also the most expensive one because it retries internally. With traces + tags + online evals: the faithfulness eval dropped at 14:02, every failing trace shows an empty retrieval span, the prompt-version stamp points at the deploy, and the cost-by-feature chart already flagged the spike. Same incident, minutes vs. days.\n\n> The discipline in one line: capture every call with enough structure (trace shape) and enough metadata (tags) that you can later ask *why* it failed and *what* it cost — broken down by the dimensions your business cares about.\n\n---\n\n## 2. Traces & spans — the backbone\n\nA **trace** is one end-to-end request (a user turn, an API call, a scheduled job). A **span** is one unit of work inside it. Model the request as a tree of nested spans so latency, cost, and errors roll up naturally.\n\n```\ntrace: chat-request (user=u_42, tenant=acme, feature=support-copilot, session=s_9)\n├─ span: agent.plan            (llm)      120 tok in \u002F 40 out   180ms\n├─ span: retrieval.search      (vector)   k=8, 0.85 top score   45ms\n├─ span: tool.get_order_status (tool)     args={id:...} ok      210ms\n├─ span: agent.synthesize      (llm)      1340 tok in \u002F 280 out 1.9s\n│   └─ span: subagent.summarize (llm)     900 tok in \u002F 120 out  900ms\n└─ span: guardrail.check       (eval)     pass                  30ms\n```\n\n**Capture on every span:** model + version, prompt (messages\u002Ftemplate + resolved variables), response\u002Fcompletion, token counts (input \u002F output \u002F **cached** separately — cached reads are priced lower), latency (TTFT and total), computed cost, tool args + results, error type, and the metadata block (§5). For retrieval spans also capture the query, the k documents returned, and their scores; for tool spans, the raw arguments and return value.\n\n**Nesting & correlation.** Propagate a single `trace_id` across the whole agent run and parent each span to its caller. When a sub-agent or a separate service is invoked, pass the trace context through (HTTP header \u002F message field) so the child spans attach to the same trace. Without this you get orphaned fragments and cannot follow one request.\n\n**Instrumentation pattern.** Wrap each unit of work in a span context — a decorator\u002F`with` block — that opens the span, records inputs, runs the call, records outputs + usage + cost on the way out, and closes it. The pattern:\n\n```\nwith tracer.span(\"agent.synthesize\", metadata=ctx) as span:\n    resp = client.chat(model, messages)\n    span.set(input_tokens=resp.usage.in, output_tokens=resp.usage.out,\n             cached_tokens=resp.usage.cached, response_model=resp.model)\n    span.set(cost=price(model, resp.usage))   # computed, not guessed\n    return resp                                # exceptions auto-record error_type\n```\n\nMost SDKs (Langfuse, LangSmith, OpenInference\u002FPhoenix) ship decorators that do this; the contract is the same regardless of vendor — open, record, close, and never let an exception escape un-recorded.\n\n### OpenTelemetry GenAI semantic conventions\n\nOpenTelemetry defines **GenAI semantic conventions** so spans are portable across vendors. These conventions are still marked *experimental\u002Fincubating* — pin a version and expect attribute names to evolve. The stable, well-known set:\n\n| Attribute | Meaning |\n|---|---|\n| `gen_ai.system` | provider (`openai`, `anthropic`, `vertex_ai`…) |\n| `gen_ai.operation.name` | `chat`, `embeddings`, `text_completion` |\n| `gen_ai.request.model` | requested model id |\n| `gen_ai.response.model` | model that actually served (snapshot\u002Fversion) |\n| `gen_ai.usage.input_tokens` | prompt tokens |\n| `gen_ai.usage.output_tokens` | completion tokens |\n| `gen_ai.request.temperature` \u002F `.max_tokens` | sampling params |\n\nSpan name follows the pattern `{operation} {model}` (e.g. `chat gpt-4o`). Prompt\u002Fresponse content is recorded as span events (or attributes, gated behind a content-capture flag because of PII — see §7). Emitting OTel-native spans means any OTel backend (Phoenix, Langfuse, Grafana, Datadog) can ingest them without bespoke glue. **Do** capture both `request.model` and `response.model` — provider snapshots silently change behavior and you need the served version to reproduce.\n\n---\n\n## 3. Metrics — what to chart\n\nMetrics are aggregates over spans. Charts and alerts run on these; traces are for drill-down.\n\n| Metric | Why | Watch |\n|---|---|---|\n| Latency p50 \u002F p95 \u002F p99 | tail latency is the user's reality | **never** average — averages hide the tail |\n| TTFT (time to first token) | streaming UX; perceived speed | p95 |\n| TPOT \u002F inter-token latency | streaming smoothness | spikes mid-stream |\n| Token usage (in \u002F out \u002F cached) | drives cost; bloat detector | rising input = context creep |\n| Throughput (req\u002Fs, tok\u002Fs) | capacity & saturation | vs. provider rate limits |\n| Error rate **by type** | timeouts ≠ refusals ≠ malformed | break down, don't sum |\n| Cache hit rate | prompt-cache \u002F semantic-cache savings | falling rate = rising cost |\n| Retrieval quality | hit@k, MRR, score distribution | for RAG paths |\n| Online eval scores | quality you can chart | per intent\u002Ffeature |\n| Tool success rate | agent reliability | per tool |\n\n**Error taxonomy is load-bearing.** A single \"error rate\" number is useless. Separate: `timeout`, `rate_limit` (429), `context_length_exceeded`, `content_filter`\u002F`refusal`, `malformed_output` (JSON\u002Fschema parse fail), `tool_error`, and `upstream_5xx`. Each demands a different fix — retries help timeouts and rate limits, prompt changes help refusals and malformed output, a token budget fixes `context_length_exceeded`. **Do** alert on `malformed_output` rate; a jump usually means a prompt or model change broke structured output.\n\n**Percentiles, not averages — always.** Latency distributions are heavily right-skewed: a p50 of 800ms can hide a p99 of 12s where your most engaged users live. Chart p50\u002Fp95\u002Fp99 side by side and alert on the *tail*. The same applies to tokens and cost — a p99 token count reveals the runaway-context outliers that an average buries. Compute percentiles in the field at p75+ load, not from a lab benchmark of ten requests.\n\n**Retrieval quality (RAG paths).** For any retrieval-augmented call, treat the retrieval span as a measurable subsystem: track **hit@k** (did a known-relevant doc appear in top-k), **MRR** (how high), and the **score distribution** of returned chunks. A falling top-score or a rising fraction of low-score retrievals is your earliest signal that answers are about to degrade — it precedes the eval-score drop because bad context produces confident-but-wrong output. Also chart the *fraction of calls that retrieved nothing usable* (all chunks below threshold); those are guaranteed hallucination candidates.\n\n---\n\n## 4. Drift detection — catching silent regressions\n\nNothing throws when quality degrades. You detect drift by measuring continuously and alerting on deltas.\n\n- **Input drift.** The distribution of incoming requests shifts (new topics, longer inputs, a new client). Track input length, language, and intent-classification distribution over time; a shift often precedes a quality drop because your prompts\u002Fretrieval were tuned for the old distribution.\n- **Output-quality drift.** Run **online evals** on a sample of live traffic: cheap deterministic checks on everything (schema valid, length, refusal markers, toxicity) plus an **LLM-as-judge** on a sampled subset (helpfulness, faithfulness\u002Fgroundedness, relevance). Run judges asynchronously off the trace stream so they never add user-facing latency; write the score back onto the trace so cost and quality live together. Chart the scores per intent\u002Ffeature; alert when a rolling window drops below threshold. Calibrate the judge against a human-labeled set first — an uncalibrated judge drifts too, and a judge that disagrees with humans gives you false confidence.\n- **Embedding \u002F retrieval drift.** Monitor the score distribution of retrieved chunks and hit@k. Falling top scores mean the query distribution moved away from the indexed corpus, or the corpus went stale — RAG silently degrades here.\n- **Model-version change effects.** When the provider rolls a new snapshot (or you change models), watch evals and cost\u002Flatency across the cutover. Annotate dashboards with deploys and model changes so a regression lines up with its cause.\n\n**Do** keep a small fixed regression set replayed on every prompt\u002Fmodel change and gate the deploy on it. **Don't** rely only on offline evals — they don't see the live input distribution, so production drift sails past them.\n\n---\n\n## 5. Cost attribution — the discipline, not just $\u002Fmodel\n\nKnowing \"GPT-4o cost us $4,000 this month\" is nearly useless. You cannot act on a model-level number — you don't know *which feature, customer, or code path* drove it. The actual discipline is **attributing every dollar to the dimensions your business decides on.** It rests on one mechanism: **metadata tags propagated onto every span.**\n\n**The mechanism, end to end:**\n\n1. **Tag every call.** At the entry point, stamp a metadata block and propagate it onto every span in the trace:\n   ```\n   metadata = {\n     request_id, session_id, trace_id,\n     user_id, tenant_id,              # who\n     feature, workflow, step,         # what product surface\n     environment, app_version,        # where\n   }\n   ```\n2. **Compute per-call cost from tokens.** Cost is not stored by the provider per request you can query cheaply — compute it: `cost = input_tokens × price_in + cached_tokens × price_cached + output_tokens × price_out`, using a per-model price table you keep current. **Cached input is priced differently** (often ~10–25% of full input) — count it separately or your numbers will be wrong on cache-heavy workloads.\n3. **Roll up by tag.** Because cost lives on tagged spans, every breakdown is a group-by: by `feature`, by `tenant_id`, by `workflow`, by `user_id`, by `model`. The same data answers all of them.\n\nA live spend guardrail reads the same tags it would later aggregate on:\n\n```\ndef before_call(ctx, est_tokens):\n    if spend_today(ctx.tenant_id) > tenant_budget(ctx.tenant_id):\n        return Degrade(model=\"cheaper-model\")     # or Refuse(...)\n    if est_tokens > MAX_TOKENS_PER_CALL or ctx.step > MAX_AGENT_STEPS:\n        return Refuse(\"token\u002Fstep ceiling hit\")\n    return Allow()\n```\n\nThe guardrail and the chargeback report are the *same* `tenant_id` tag used at two moments — one to prevent overspend, one to bill it.\n\n**What the tags unlock:**\n\n- **Unit economics.** Cost per task, per session, per active user, per resolved ticket. This is the number that tells you whether the product is viable — not the monthly invoice.\n- **Cost per *successful* outcome.** Join cost with a success\u002Feval signal on the same trace: `cost ÷ (% traces that succeeded)`. A feature that's cheap per call but fails half the time is expensive per outcome. This metric catches what raw cost hides.\n- **Expensive-path discovery.** Group-by `workflow`\u002F`step` to find the spans that dominate spend — usually a giant context window, an agent that loops, or an over-eager retrieval `k`. Optimize the top 3, ignore the long tail.\n- **Budget alerts & guardrails.** Alert when a tenant or feature exceeds a daily\u002Fmonthly budget. Enforce *guardrails* in-line: per-request token ceilings, max agent steps, and a per-tenant rate\u002Fspend limit that degrades to a cheaper model or refuses rather than blowing the budget.\n- **Showback \u002F chargeback.** Multi-tenant SaaS: roll cost up by `tenant_id` for internal visibility (showback) or actual billing (chargeback). Impossible without per-call tenant tags — this is the single most common thing teams wish they'd instrumented from day one.\n\n| Dimension | Group-by tag | Answers |\n|---|---|---|\n| Feature | `feature` | which surface to optimize or price |\n| Customer | `tenant_id` | margin per account; chargeback |\n| User journey | `session_id` | cost of one end-to-end task |\n| Code path | `workflow` \u002F `step` | where the spend concentrates |\n| Model | `gen_ai.request.model` | mix shift; cheaper-model candidates |\n\n**Worked unit-economics example.** A \"summarize document\" feature: 2,000 traces\u002Fday, avg 6k input + 800 output tokens, success rate 80%. Per-trace cost = `6000×$in + 800×$out`. The invoice says the model cost $X\u002Fmonth — uninformative. The tagged rollup says: $0.024 per *attempted* summary, but $0.030 per *successful* one (you pay for the 20% that fail too), and `workflow=batch-reprocess` is 3% of traffic but 22% of spend because it re-summarizes on every edit. Now you can act: cache summaries to kill the reprocess path, and either fix or fail-fast the 20% so you stop paying for doomed calls. None of that is visible from $\u002Fmodel.\n\n**Do** tag at the boundary and propagate — retrofitting tags onto historical data is impossible. **Don't** settle for model-level cost: it tells you *what* you spent, never *why*, so you can't act on it. **Don't** trust the provider's dashboard total for attribution — it has no idea which feature or tenant a call belonged to; only your tags do.\n\n---\n\n## 6. Debugging with traces\n\nTraces turn \"it failed sometimes\" into a reproducible artifact.\n\n- **Find the failing span.** Open the trace, scan top-down for the first error or anomalous span (wrong tool args, empty retrieval, a refusal). The tree localizes the fault instead of you guessing across services.\n- **Replay.** Re-run a captured trace against a candidate prompt\u002Fmodel with the *exact* recorded inputs. This is the only reliable repro for a non-deterministic system — you froze the inputs at capture time.\n- **Diff.** Compare two traces (good vs. bad, before vs. after a prompt change): diff prompts, retrieved context, and outputs to isolate what changed. Diffing the resolved prompt often reveals a template variable that came through empty.\n- **Prompt versioning.** Version every prompt\u002Ftemplate and stamp the version onto the span. When a regression appears, you can see exactly which prompt version served it and bisect to the change. Tie eval scores to prompt versions to compare them quantitatively before rollout.\n\n**Worked example.** Complaint: \"the agent recommends out-of-stock items.\" (1) Filter traces by `feature=product-rec` + low faithfulness score → 40 bad traces. (2) Open one: the `retrieval.search` span returns 8 docs but the top score is 0.31 (normally >0.7) — retrieval is the failing span, not the LLM. (3) Diff against a good trace: the bad query embedded an empty `category` variable, so the template resolved to a malformed query. (4) The `prompt_version` stamp shows it changed in last night's deploy, which renamed `category` to `product_category` in the data but not the template. (5) Replay the 40 traces against the fixed template → faithfulness back to baseline. The trace tree localized a *retrieval* bug that looked like an *LLM* bug.\n\n---\n\n## 7. Logging, PII & redaction\n\nPrompts and responses routinely contain PII, secrets, and proprietary data — capturing them verbatim creates a compliance and breach liability.\n\n- **Gate content capture.** Make full prompt\u002Fresponse logging a flag, not a default. Redact PII (emails, names, phone\u002Fcard\u002FSSN patterns) at capture time *before* it leaves the process — a regex\u002FNER scrubber on prompts and completions. For high-sensitivity tenants, log only token counts, metadata, and a content hash — never the body. The hash still lets you detect duplicate inputs and verify a replay without storing the text.\n- **Never log secrets.** Strip API keys, auth headers, bearer tokens, and tool credentials from tool-span args and retrieval results before capture. A trace viewer is a place engineers paste links and screenshots — assume anything captured will eventually be seen by someone who shouldn't see it. Maintain a deny-list of arg keys (`authorization`, `api_key`, `password`, `token`) that are always masked.\n- **Retention & access.** Set short retention on content (e.g. 7–30 days), longer on metrics\u002Fmetadata (months — they're not sensitive and you need trends). Lock content access behind role\u002Ftenant boundaries. Honor deletion\u002FDSAR requests by keying captures on `user_id`\u002F`tenant_id` so you can purge one subject's traces without touching aggregates.\n- **Sampling helps privacy too.** The less raw content you persist (§8), the smaller your exposure surface. Sample content aggressively; keep metrics on everything.\n\n---\n\n## 8. Sampling for volume\n\nAt scale you cannot afford to capture full content on 100% of traffic.\n\n- **Tail-based sampling.** Keep all **errored, slow, and high-cost** traces (the ones you'll debug); sample the boring successful ones (e.g. 1–10%). Decide *after* the trace completes so you don't discard the failure you needed.\n- **Always-on metrics, sampled content.** Compute metrics and cost on **every** request (they're cheap aggregates), but persist full prompts\u002Fresponses only for the sampled or flagged set. **Don't** sample cost — partial cost data makes attribution lie.\n- **Per-tenant overrides.** Sample more aggressively on high-volume tenants, keep 100% for a customer you're actively debugging.\n\n---\n\n## 9. Tooling landscape\n\nThe load-bearing distinction is *how* a tool captures data: a **proxy\u002Fgateway** (swap your base URL, zero code change) vs. **SDK\u002Fdecorator instrumentation** (wrap your code) vs. **OTel-native** (emit standard spans).\n\n| Tool | Capture model | Notes |\n|---|---|---|\n| **Helicone** | Proxy \u002F gateway | Base-URL swap; fast to adopt, cost\u002Flatency\u002Fcaching out of the box; less deep on custom spans |\n| **Langfuse** | SDK \u002F decorator | Open-source, self-hostable; traces, evals, prompt mgmt, cost; OTel-compatible ingestion |\n| **LangSmith** | SDK \u002F decorator | Commercial (LangChain); deep tracing, datasets, evals; best inside LangChain\u002FLangGraph |\n| **Phoenix (Arize)** | OTel-native | OpenInference\u002FOTel spans, eval- and RAG-focused; self-host friendly |\n| **OTel + backend** | OTel-native | Vendor-neutral GenAI semconv into Datadog\u002FGrafana\u002FHoneycomb; most portable, most assembly |\n\n**Choosing:** want zero code change and cost-first → proxy (Helicone). Want self-host + evals + prompt management → Langfuse. Deep in LangChain → LangSmith. Standardizing on OTel across services → Phoenix or raw OTel into your existing backend. **Do** prefer a tool that speaks OTel so you're not locked in; **don't** build a bespoke logging schema you'll have to migrate later.\n\n---\n\n## 10. Failure modes\n\n| Failure mode | Symptom | Fix |\n|---|---|---|\n| **No per-call metadata** | can't attribute cost or debug a request | tag every span at the boundary (§5) |\n| **Only model-level cost** | can't find the expensive feature\u002Ftenant | group-by `feature`\u002F`tenant_id` from tags |\n| **No online evals** | quality drifts silently, users complain first | sample + LLM-judge live traffic (§4) |\n| **Averages, no percentiles** | tail latency invisible; p99 users suffer | chart p50\u002Fp95\u002Fp99 + TTFT (§3) |\n| **One lumped error rate** | can't tell timeout from refusal from bad JSON | error taxonomy (§3) |\n| **Logging secrets\u002FPII verbatim** | compliance & breach exposure | gate content capture, redact (§7) |\n| **No trace correlation** | agent failure shows as vague top-level error | propagate `trace_id`, nest spans (§2) |\n| **Cached tokens priced as full** | cost numbers wrong on cache-heavy paths | count cached tokens separately (§5) |\n| **Sampling cost data** | attribution lies | metrics\u002Fcost on 100%, sample content only (§8) |\n| **No prompt versioning** | regression can't be tied to a change | stamp prompt version on span (§6) |\n| **No retrieval metrics** | RAG degrades invisibly until users complain | track hit@k + score distribution (§3) |\n| **Async judge in the hot path** | online evals add user-facing latency | run judges off the trace stream (§4) |\n\n## 11. Operating cadence\n\nObservability is a practice, not a one-time setup. The minimum loop that keeps a production LLM system honest:\n\n- **Per deploy:** replay the fixed regression set, diff evals and cost vs. the previous version, annotate the dashboard with the deploy and any model-snapshot change. Gate the release on the regression set.\n- **Daily:** scan p95\u002Fp99 latency, error-rate-by-type, and cost-per-feature for anomalies; review a sample of low-eval-score traces by hand (the judge isn't perfect — humans catch what it misses).\n- **Weekly:** review unit economics (cost per successful outcome) per feature\u002Ftenant, check budget burn against limits, and re-rank the expensive paths to optimize next.\n- **On every alert:** start from the trace, not the metric. The metric tells you *that* something moved; the trace tells you *why*.\n\nThe through-line: **structure** (nested traces) plus **metadata** (tags) plus **continuous measurement** (metrics + online evals) is what turns a non-deterministic, expensive, drifting system into one you can debug, cost, and trust. Instrument it before you need it — you can't retrofit a tag onto a request that already happened, and the trace you didn't capture is the one you'll wish you had.\n","# LLM Observability & Cost Attribution\n\nYou cannot debug, optimize, or cost what you cannot see. Traditional APM assumes deterministic, cheap, fast functions. LLM systems break all three assumptions: the same input yields different outputs, a single user request fans out into a dozen model and tool calls, every call costs real money proportional to tokens, and output quality silently drifts as models, prompts, and data change underneath you. Observability for LLM systems is therefore a first-class engineering discipline, not a dashboard you bolt on later — it is the only way to answer \"why did this request fail,\" \"which feature is burning our budget,\" and \"did quality regress after the deploy.\"\n\nHow to use this skill: instrument every LLM call as a span inside a request-level trace (§2), attach the metadata tags that make cost and quality attributable (§5), emit the metrics in §3, watch for drift in §4, and debug from traces in §6. Treat the failure modes in §9 as a pre-ship checklist.\n\n---\n\n## 1. Why LLM observability is its own discipline\n\n- **Non-determinism.** Temperature, sampling, and model updates mean the same prompt produces different responses. You cannot reproduce a bug from inputs alone — you need the *exact* request\u002Fresponse that was captured, including the model version that served it.\n- **Multi-step agents.** A modern request is a tree: planner → retrieval → tool call → sub-agent → synthesis. A failure three levels deep surfaces as a vague top-level error. Without nested traces you see the symptom, never the cause.\n- **Cost is variable and large.** Spend scales with tokens, not requests. A single runaway agent loop or a bloated context window can cost 100× a normal request. You must measure cost per call to control it.\n- **Quality drifts silently.** No exception is thrown when answers get worse. A provider ships a new model snapshot, your RAG corpus shifts, a prompt edit regresses one intent — and nothing alerts unless you measure quality online.\n\n**Concretely, what blindness costs you.** A support agent starts giving wrong answers after a Tuesday deploy. Without traces: you know error rate is flat (it didn't *error*, it answered confidently and wrongly), you can't reproduce the complaint (different output every run), you can't tell if it was the prompt edit, a model snapshot rollover, or stale retrieval, and you have no idea the bad path is also the most expensive one because it retries internally. With traces + tags + online evals: the faithfulness eval dropped at 14:02, every failing trace shows an empty retrieval span, the prompt-version stamp points at the deploy, and the cost-by-feature chart already flagged the spike. Same incident, minutes vs. days.\n\n> The discipline in one line: capture every call with enough structure (trace shape) and enough metadata (tags) that you can later ask *why* it failed and *what* it cost — broken down by the dimensions your business cares about.\n\n---\n\n## 2. Traces & spans — the backbone\n\nA **trace** is one end-to-end request (a user turn, an API call, a scheduled job). A **span** is one unit of work inside it. Model the request as a tree of nested spans so latency, cost, and errors roll up naturally.\n\n```\ntrace: chat-request (user=u_42, tenant=acme, feature=support-copilot, session=s_9)\n├─ span: agent.plan            (llm)      120 tok in \u002F 40 out   180ms\n├─ span: retrieval.search      (vector)   k=8, 0.85 top score   45ms\n├─ span: tool.get_order_status (tool)     args={id:...} ok      210ms\n├─ span: agent.synthesize      (llm)      1340 tok in \u002F 280 out 1.9s\n│   └─ span: subagent.summarize (llm)     900 tok in \u002F 120 out  900ms\n└─ span: guardrail.check       (eval)     pass                  30ms\n```\n\n**Capture on every span:** model + version, prompt (messages\u002Ftemplate + resolved variables), response\u002Fcompletion, token counts (input \u002F output \u002F **cached** separately — cached reads are priced lower), latency (TTFT and total), computed cost, tool args + results, error type, and the metadata block (§5). For retrieval spans also capture the query, the k documents returned, and their scores; for tool spans, the raw arguments and return value.\n\n**Nesting & correlation.** Propagate a single `trace_id` across the whole agent run and parent each span to its caller. When a sub-agent or a separate service is invoked, pass the trace context through (HTTP header \u002F message field) so the child spans attach to the same trace. Without this you get orphaned fragments and cannot follow one request.\n\n**Instrumentation pattern.** Wrap each unit of work in a span context — a decorator\u002F`with` block — that opens the span, records inputs, runs the call, records outputs + usage + cost on the way out, and closes it. The pattern:\n\n```\nwith tracer.span(\"agent.synthesize\", metadata=ctx) as span:\n    resp = client.chat(model, messages)\n    span.set(input_tokens=resp.usage.in, output_tokens=resp.usage.out,\n             cached_tokens=resp.usage.cached, response_model=resp.model)\n    span.set(cost=price(model, resp.usage))   # computed, not guessed\n    return resp                                # exceptions auto-record error_type\n```\n\nMost SDKs (Langfuse, LangSmith, OpenInference\u002FPhoenix) ship decorators that do this; the contract is the same regardless of vendor — open, record, close, and never let an exception escape un-recorded.\n\n### OpenTelemetry GenAI semantic conventions\n\nOpenTelemetry defines **GenAI semantic conventions** so spans are portable across vendors. These conventions are still marked *experimental\u002Fincubating* — pin a version and expect attribute names to evolve. The stable, well-known set:\n\n| Attribute | Meaning |\n|---|---|\n| `gen_ai.system` | provider (`openai`, `anthropic`, `vertex_ai`…) |\n| `gen_ai.operation.name` | `chat`, `embeddings`, `text_completion` |\n| `gen_ai.request.model` | requested model id |\n| `gen_ai.response.model` | model that actually served (snapshot\u002Fversion) |\n| `gen_ai.usage.input_tokens` | prompt tokens |\n| `gen_ai.usage.output_tokens` | completion tokens |\n| `gen_ai.request.temperature` \u002F `.max_tokens` | sampling params |\n\nSpan name follows the pattern `{operation} {model}` (e.g. `chat gpt-4o`). Prompt\u002Fresponse content is recorded as span events (or attributes, gated behind a content-capture flag because of PII — see §7). Emitting OTel-native spans means any OTel backend (Phoenix, Langfuse, Grafana, Datadog) can ingest them without bespoke glue. **Do** capture both `request.model` and `response.model` — provider snapshots silently change behavior and you need the served version to reproduce.\n\n---\n\n## 3. Metrics — what to chart\n\nMetrics are aggregates over spans. Charts and alerts run on these; traces are for drill-down.\n\n| Metric | Why | Watch |\n|---|---|---|\n| Latency p50 \u002F p95 \u002F p99 | tail latency is the user's reality | **never** average — averages hide the tail |\n| TTFT (time to first token) | streaming UX; perceived speed | p95 |\n| TPOT \u002F inter-token latency | streaming smoothness | spikes mid-stream |\n| Token usage (in \u002F out \u002F cached) | drives cost; bloat detector | rising input = context creep |\n| Throughput (req\u002Fs, tok\u002Fs) | capacity & saturation | vs. provider rate limits |\n| Error rate **by type** | timeouts ≠ refusals ≠ malformed | break down, don't sum |\n| Cache hit rate | prompt-cache \u002F semantic-cache savings | falling rate = rising cost |\n| Retrieval quality | hit@k, MRR, score distribution | for RAG paths |\n| Online eval scores | quality you can chart | per intent\u002Ffeature |\n| Tool success rate | agent reliability | per tool |\n\n**Error taxonomy is load-bearing.** A single \"error rate\" number is useless. Separate: `timeout`, `rate_limit` (429), `context_length_exceeded`, `content_filter`\u002F`refusal`, `malformed_output` (JSON\u002Fschema parse fail), `tool_error`, and `upstream_5xx`. Each demands a different fix — retries help timeouts and rate limits, prompt changes help refusals and malformed output, a token budget fixes `context_length_exceeded`. **Do** alert on `malformed_output` rate; a jump usually means a prompt or model change broke structured output.\n\n**Percentiles, not averages — always.** Latency distributions are heavily right-skewed: a p50 of 800ms can hide a p99 of 12s where your most engaged users live. Chart p50\u002Fp95\u002Fp99 side by side and alert on the *tail*. The same applies to tokens and cost — a p99 token count reveals the runaway-context outliers that an average buries. Compute percentiles in the field at p75+ load, not from a lab benchmark of ten requests.\n\n**Retrieval quality (RAG paths).** For any retrieval-augmented call, treat the retrieval span as a measurable subsystem: track **hit@k** (did a known-relevant doc appear in top-k), **MRR** (how high), and the **score distribution** of returned chunks. A falling top-score or a rising fraction of low-score retrievals is your earliest signal that answers are about to degrade — it precedes the eval-score drop because bad context produces confident-but-wrong output. Also chart the *fraction of calls that retrieved nothing usable* (all chunks below threshold); those are guaranteed hallucination candidates.\n\n---\n\n## 4. Drift detection — catching silent regressions\n\nNothing throws when quality degrades. You detect drift by measuring continuously and alerting on deltas.\n\n- **Input drift.** The distribution of incoming requests shifts (new topics, longer inputs, a new client). Track input length, language, and intent-classification distribution over time; a shift often precedes a quality drop because your prompts\u002Fretrieval were tuned for the old distribution.\n- **Output-quality drift.** Run **online evals** on a sample of live traffic: cheap deterministic checks on everything (schema valid, length, refusal markers, toxicity) plus an **LLM-as-judge** on a sampled subset (helpfulness, faithfulness\u002Fgroundedness, relevance). Run judges asynchronously off the trace stream so they never add user-facing latency; write the score back onto the trace so cost and quality live together. Chart the scores per intent\u002Ffeature; alert when a rolling window drops below threshold. Calibrate the judge against a human-labeled set first — an uncalibrated judge drifts too, and a judge that disagrees with humans gives you false confidence.\n- **Embedding \u002F retrieval drift.** Monitor the score distribution of retrieved chunks and hit@k. Falling top scores mean the query distribution moved away from the indexed corpus, or the corpus went stale — RAG silently degrades here.\n- **Model-version change effects.** When the provider rolls a new snapshot (or you change models), watch evals and cost\u002Flatency across the cutover. Annotate dashboards with deploys and model changes so a regression lines up with its cause.\n\n**Do** keep a small fixed regression set replayed on every prompt\u002Fmodel change and gate the deploy on it. **Don't** rely only on offline evals — they don't see the live input distribution, so production drift sails past them.\n\n---\n\n## 5. Cost attribution — the discipline, not just $\u002Fmodel\n\nKnowing \"GPT-4o cost us $4,000 this month\" is nearly useless. You cannot act on a model-level number — you don't know *which feature, customer, or code path* drove it. The actual discipline is **attributing every dollar to the dimensions your business decides on.** It rests on one mechanism: **metadata tags propagated onto every span.**\n\n**The mechanism, end to end:**\n\n1. **Tag every call.** At the entry point, stamp a metadata block and propagate it onto every span in the trace:\n   ```\n   metadata = {\n     request_id, session_id, trace_id,\n     user_id, tenant_id,              # who\n     feature, workflow, step,         # what product surface\n     environment, app_version,        # where\n   }\n   ```\n2. **Compute per-call cost from tokens.** Cost is not stored by the provider per request you can query cheaply — compute it: `cost = input_tokens × price_in + cached_tokens × price_cached + output_tokens × price_out`, using a per-model price table you keep current. **Cached input is priced differently** (often ~10–25% of full input) — count it separately or your numbers will be wrong on cache-heavy workloads.\n3. **Roll up by tag.** Because cost lives on tagged spans, every breakdown is a group-by: by `feature`, by `tenant_id`, by `workflow`, by `user_id`, by `model`. The same data answers all of them.\n\nA live spend guardrail reads the same tags it would later aggregate on:\n\n```\ndef before_call(ctx, est_tokens):\n    if spend_today(ctx.tenant_id) > tenant_budget(ctx.tenant_id):\n        return Degrade(model=\"cheaper-model\")     # or Refuse(...)\n    if est_tokens > MAX_TOKENS_PER_CALL or ctx.step > MAX_AGENT_STEPS:\n        return Refuse(\"token\u002Fstep ceiling hit\")\n    return Allow()\n```\n\nThe guardrail and the chargeback report are the *same* `tenant_id` tag used at two moments — one to prevent overspend, one to bill it.\n\n**What the tags unlock:**\n\n- **Unit economics.** Cost per task, per session, per active user, per resolved ticket. This is the number that tells you whether the product is viable — not the monthly invoice.\n- **Cost per *successful* outcome.** Join cost with a success\u002Feval signal on the same trace: `cost ÷ (% traces that succeeded)`. A feature that's cheap per call but fails half the time is expensive per outcome. This metric catches what raw cost hides.\n- **Expensive-path discovery.** Group-by `workflow`\u002F`step` to find the spans that dominate spend — usually a giant context window, an agent that loops, or an over-eager retrieval `k`. Optimize the top 3, ignore the long tail.\n- **Budget alerts & guardrails.** Alert when a tenant or feature exceeds a daily\u002Fmonthly budget. Enforce *guardrails* in-line: per-request token ceilings, max agent steps, and a per-tenant rate\u002Fspend limit that degrades to a cheaper model or refuses rather than blowing the budget.\n- **Showback \u002F chargeback.** Multi-tenant SaaS: roll cost up by `tenant_id` for internal visibility (showback) or actual billing (chargeback). Impossible without per-call tenant tags — this is the single most common thing teams wish they'd instrumented from day one.\n\n| Dimension | Group-by tag | Answers |\n|---|---|---|\n| Feature | `feature` | which surface to optimize or price |\n| Customer | `tenant_id` | margin per account; chargeback |\n| User journey | `session_id` | cost of one end-to-end task |\n| Code path | `workflow` \u002F `step` | where the spend concentrates |\n| Model | `gen_ai.request.model` | mix shift; cheaper-model candidates |\n\n**Worked unit-economics example.** A \"summarize document\" feature: 2,000 traces\u002Fday, avg 6k input + 800 output tokens, success rate 80%. Per-trace cost = `6000×$in + 800×$out`. The invoice says the model cost $X\u002Fmonth — uninformative. The tagged rollup says: $0.024 per *attempted* summary, but $0.030 per *successful* one (you pay for the 20% that fail too), and `workflow=batch-reprocess` is 3% of traffic but 22% of spend because it re-summarizes on every edit. Now you can act: cache summaries to kill the reprocess path, and either fix or fail-fast the 20% so you stop paying for doomed calls. None of that is visible from $\u002Fmodel.\n\n**Do** tag at the boundary and propagate — retrofitting tags onto historical data is impossible. **Don't** settle for model-level cost: it tells you *what* you spent, never *why*, so you can't act on it. **Don't** trust the provider's dashboard total for attribution — it has no idea which feature or tenant a call belonged to; only your tags do.\n\n---\n\n## 6. Debugging with traces\n\nTraces turn \"it failed sometimes\" into a reproducible artifact.\n\n- **Find the failing span.** Open the trace, scan top-down for the first error or anomalous span (wrong tool args, empty retrieval, a refusal). The tree localizes the fault instead of you guessing across services.\n- **Replay.** Re-run a captured trace against a candidate prompt\u002Fmodel with the *exact* recorded inputs. This is the only reliable repro for a non-deterministic system — you froze the inputs at capture time.\n- **Diff.** Compare two traces (good vs. bad, before vs. after a prompt change): diff prompts, retrieved context, and outputs to isolate what changed. Diffing the resolved prompt often reveals a template variable that came through empty.\n- **Prompt versioning.** Version every prompt\u002Ftemplate and stamp the version onto the span. When a regression appears, you can see exactly which prompt version served it and bisect to the change. Tie eval scores to prompt versions to compare them quantitatively before rollout.\n\n**Worked example.** Complaint: \"the agent recommends out-of-stock items.\" (1) Filter traces by `feature=product-rec` + low faithfulness score → 40 bad traces. (2) Open one: the `retrieval.search` span returns 8 docs but the top score is 0.31 (normally >0.7) — retrieval is the failing span, not the LLM. (3) Diff against a good trace: the bad query embedded an empty `category` variable, so the template resolved to a malformed query. (4) The `prompt_version` stamp shows it changed in last night's deploy, which renamed `category` to `product_category` in the data but not the template. (5) Replay the 40 traces against the fixed template → faithfulness back to baseline. The trace tree localized a *retrieval* bug that looked like an *LLM* bug.\n\n---\n\n## 7. Logging, PII & redaction\n\nPrompts and responses routinely contain PII, secrets, and proprietary data — capturing them verbatim creates a compliance and breach liability.\n\n- **Gate content capture.** Make full prompt\u002Fresponse logging a flag, not a default. Redact PII (emails, names, phone\u002Fcard\u002FSSN patterns) at capture time *before* it leaves the process — a regex\u002FNER scrubber on prompts and completions. For high-sensitivity tenants, log only token counts, metadata, and a content hash — never the body. The hash still lets you detect duplicate inputs and verify a replay without storing the text.\n- **Never log secrets.** Strip API keys, auth headers, bearer tokens, and tool credentials from tool-span args and retrieval results before capture. A trace viewer is a place engineers paste links and screenshots — assume anything captured will eventually be seen by someone who shouldn't see it. Maintain a deny-list of arg keys (`authorization`, `api_key`, `password`, `token`) that are always masked.\n- **Retention & access.** Set short retention on content (e.g. 7–30 days), longer on metrics\u002Fmetadata (months — they're not sensitive and you need trends). Lock content access behind role\u002Ftenant boundaries. Honor deletion\u002FDSAR requests by keying captures on `user_id`\u002F`tenant_id` so you can purge one subject's traces without touching aggregates.\n- **Sampling helps privacy too.** The less raw content you persist (§8), the smaller your exposure surface. Sample content aggressively; keep metrics on everything.\n\n---\n\n## 8. Sampling for volume\n\nAt scale you cannot afford to capture full content on 100% of traffic.\n\n- **Tail-based sampling.** Keep all **errored, slow, and high-cost** traces (the ones you'll debug); sample the boring successful ones (e.g. 1–10%). Decide *after* the trace completes so you don't discard the failure you needed.\n- **Always-on metrics, sampled content.** Compute metrics and cost on **every** request (they're cheap aggregates), but persist full prompts\u002Fresponses only for the sampled or flagged set. **Don't** sample cost — partial cost data makes attribution lie.\n- **Per-tenant overrides.** Sample more aggressively on high-volume tenants, keep 100% for a customer you're actively debugging.\n\n---\n\n## 9. Tooling landscape\n\nThe load-bearing distinction is *how* a tool captures data: a **proxy\u002Fgateway** (swap your base URL, zero code change) vs. **SDK\u002Fdecorator instrumentation** (wrap your code) vs. **OTel-native** (emit standard spans).\n\n| Tool | Capture model | Notes |\n|---|---|---|\n| **Helicone** | Proxy \u002F gateway | Base-URL swap; fast to adopt, cost\u002Flatency\u002Fcaching out of the box; less deep on custom spans |\n| **Langfuse** | SDK \u002F decorator | Open-source, self-hostable; traces, evals, prompt mgmt, cost; OTel-compatible ingestion |\n| **LangSmith** | SDK \u002F decorator | Commercial (LangChain); deep tracing, datasets, evals; best inside LangChain\u002FLangGraph |\n| **Phoenix (Arize)** | OTel-native | OpenInference\u002FOTel spans, eval- and RAG-focused; self-host friendly |\n| **OTel + backend** | OTel-native | Vendor-neutral GenAI semconv into Datadog\u002FGrafana\u002FHoneycomb; most portable, most assembly |\n\n**Choosing:** want zero code change and cost-first → proxy (Helicone). Want self-host + evals + prompt management → Langfuse. Deep in LangChain → LangSmith. Standardizing on OTel across services → Phoenix or raw OTel into your existing backend. **Do** prefer a tool that speaks OTel so you're not locked in; **don't** build a bespoke logging schema you'll have to migrate later.\n\n---\n\n## 10. Failure modes\n\n| Failure mode | Symptom | Fix |\n|---|---|---|\n| **No per-call metadata** | can't attribute cost or debug a request | tag every span at the boundary (§5) |\n| **Only model-level cost** | can't find the expensive feature\u002Ftenant | group-by `feature`\u002F`tenant_id` from tags |\n| **No online evals** | quality drifts silently, users complain first | sample + LLM-judge live traffic (§4) |\n| **Averages, no percentiles** | tail latency invisible; p99 users suffer | chart p50\u002Fp95\u002Fp99 + TTFT (§3) |\n| **One lumped error rate** | can't tell timeout from refusal from bad JSON | error taxonomy (§3) |\n| **Logging secrets\u002FPII verbatim** | compliance & breach exposure | gate content capture, redact (§7) |\n| **No trace correlation** | agent failure shows as vague top-level error | propagate `trace_id`, nest spans (§2) |\n| **Cached tokens priced as full** | cost numbers wrong on cache-heavy paths | count cached tokens separately (§5) |\n| **Sampling cost data** | attribution lies | metrics\u002Fcost on 100%, sample content only (§8) |\n| **No prompt versioning** | regression can't be tied to a change | stamp prompt version on span (§6) |\n| **No retrieval metrics** | RAG degrades invisibly until users complain | track hit@k + score distribution (§3) |\n| **Async judge in the hot path** | online evals add user-facing latency | run judges off the trace stream (§4) |\n\n## 11. Operating cadence\n\nObservability is a practice, not a one-time setup. The minimum loop that keeps a production LLM system honest:\n\n- **Per deploy:** replay the fixed regression set, diff evals and cost vs. the previous version, annotate the dashboard with the deploy and any model-snapshot change. Gate the release on the regression set.\n- **Daily:** scan p95\u002Fp99 latency, error-rate-by-type, and cost-per-feature for anomalies; review a sample of low-eval-score traces by hand (the judge isn't perfect — humans catch what it misses).\n- **Weekly:** review unit economics (cost per successful outcome) per feature\u002Ftenant, check budget burn against limits, and re-rank the expensive paths to optimize next.\n- **On every alert:** start from the trace, not the metric. The metric tells you *that* something moved; the trace tells you *why*.\n\nThe through-line: **structure** (nested traces) plus **metadata** (tags) plus **continuous measurement** (metrics + online evals) is what turns a non-deterministic, expensive, drifting system into one you can debug, cost, and trust. Instrument it before you need it — you can't retrofit a tag onto a request that already happened, and the trace you didn't capture is the one you'll wish you had.\n",{"name":5,"tags":23,"description":7},[24,25,26,27,28],"ai-engineering","observability","tracing","cost","monitoring","main","4911ae44bbb35beef4511ba298475ba5a82b7e1c","ai-engineering-skills\u002Fllm-observability-and-cost",0,"https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta","https:\u002F\u002Fdocs.vanta.theft.studio","https:\u002F\u002Fgithub.com\u002Fjpoindexter","jason.theft.studio","Just some guy trying to build stuff","2026-11-12 17:11:35",{"source":8,"skillName":5,"commands":40,"prompt":50,"zipAvailable":51},[41,44,47],{"tool":42,"command":43},"npx","npx skills add jpoindexter\u002FVanta --skill llm-observability-and-cost",{"tool":45,"command":46},"bunx","bunx skills add jpoindexter\u002FVanta --skill llm-observability-and-cost",{"tool":48,"command":49},"pnpm","pnpm dlx skills add jpoindexter\u002FVanta --skill llm-observability-and-cost","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fai-engineering-skills\u002Fllm-observability-and-cost\u002FSKILL.md",true,[],[54,61,68,75,82,88],{"skillNo":55,"name":56,"nameZh":6,"description":57,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":58,"categories":59,"keywords":60,"publishedAt":18},"01M7582PEM7XM8B98VFRDV0C0J","write-changelog","Turn a range of commits into a user-facing changelog","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fsrc\u002Fmigrate\u002F__fixtures__\u002Fopenclaw\u002Fskills\u002Fwrite-changelog\u002FSKILL.md",[],[],{"skillNo":62,"name":63,"nameZh":6,"description":64,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":65,"categories":66,"keywords":67,"publishedAt":18},"01M7582KTQJMP5Y7GS83MS1C7Z","debug-flaky-test","Reproduce and isolate a flaky test before fixing it","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fsrc\u002Fmigrate\u002F__fixtures__\u002Fopenclaw\u002Fskills\u002Fdebug-flaky-test\u002FSKILL.md",[],[],{"skillNo":69,"name":70,"nameZh":6,"description":71,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":72,"categories":73,"keywords":74,"publishedAt":18},"01M7582HG7XFM4Z5NGG1YFECT8","triage-incident","First 10 minutes of an incident — stabilize, then diagnose","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fsrc\u002Fmigrate\u002F__fixtures__\u002Fhermes\u002Fskills\u002Ftriage-incident\u002FSKILL.md",[],[],{"skillNo":76,"name":77,"nameZh":6,"description":78,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":81,"publishedAt":18},"01M7582DQABK7A96XHMB770KQZ","writing-plans","Write implementation plans: bite-sized tasks, paths, code.","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fskills-library\u002Fwriting-plans\u002FSKILL.md",[],[],{"skillNo":83,"name":84,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":18},"01M7582BBK8H0HGSXKK08FCXJ1","vanta-port-adapter","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fskills-library\u002Fvanta-port-adapter\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":18},"01M75828P8PX5XSBMJ2BVVDA9Y","test-driven-development","TDD: enforce RED-GREEN-REFACTOR, tests before code.","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fskills-library\u002Ftest-driven-development\u002FSKILL.md",[],[]]