[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M792DWVBACRPGGZ8ZERTZATM":3,"skill-same:01M792DWVBACRPGGZ8ZERTZATM":62},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":39,"commitSha":40,"skillDirectory":41,"forksCount":42,"repositoryHtmlUrl":43,"repositoryHomepageUrl":6,"ownerHtmlUrl":44,"ownerBlogUrl":45,"ownerDescription":46,"updatedAt":47,"install":48,"resources":61},"01M792DWVBACRPGGZ8ZERTZATM","langchain-otel-observability",null,"Wire LangChain 1.0 \u002F LangGraph 1.0 traces into an OpenTelemetry-native backend\n(Jaeger, Honeycomb, Grafana Tempo, Datadog) with LLM-specific SLOs, safe\nprompt-content policy, and subgraph-aware span propagation. Use when LangSmith\nis not the right fit (existing OTEL stack, compliance, multi-cloud) or\nalongside LangSmith for deep-system traces.\nTrigger with \"langchain OTEL\", \"langchain opentelemetry\", \"langchain jaeger\",\n\"langchain honeycomb\", \"langchain SLO\", \"LLM span\", \"langchain tempo\",\n\"langchain datadog tracing\".","jukeyman\u002Fjukeyman-skills","jukeyman","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F142960420?v=4","01M78FR4Y3DHMMCBCC0GXXNB3F","RICK JEFFERSON","01M78FR4Z8RVQRAD31Y5J7V6EB",1,"MIT","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fsaas-packs--langchain-py-pack--langchain-otel-observability\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: langchain-otel-observability\ndescription: |\n  Wire LangChain 1.0 \u002F LangGraph 1.0 traces into an OpenTelemetry-native backend\n  (Jaeger, Honeycomb, Grafana Tempo, Datadog) with LLM-specific SLOs, safe\n  prompt-content policy, and subgraph-aware span propagation. Use when LangSmith\n  is not the right fit (existing OTEL stack, compliance, multi-cloud) or\n  alongside LangSmith for deep-system traces.\n  Trigger with \"langchain OTEL\", \"langchain opentelemetry\", \"langchain jaeger\",\n  \"langchain honeycomb\", \"langchain SLO\", \"LLM span\", \"langchain tempo\",\n  \"langchain datadog tracing\".\nallowed-tools: Read, Write, Edit, Bash(python:*), Bash(pip:*)\nversion: 2.0.0\nlicense: MIT\nauthor: Jeremy Longshore \u003Cjeremy@intentsolutions.io>\ntags: [saas, langchain, langgraph, python, langchain-1.0, observability, opentelemetry, jaeger, honeycomb]\ncompatible-with: claude-code, codex\n---\n\n# LangChain OTEL Observability (Python)\n\n## Overview\n\nAn engineer wires OpenTelemetry expecting to see prompts and responses in\nHoneycomb. The traces land — but only timing, model name, and token counts\nappear. The prompt body is blank. This is **not** a bug: it's the OTEL GenAI\nsemantic-conventions privacy-safe default (P27), where\n`OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` is off. The instinct is to\nflip it on and move on. On a multi-tenant workload that flip is a leak — the\nnext engineer to search traces for Tenant A sees Tenant B's PII in the results,\nbecause redaction was supposed to happen upstream and never did.\n\nA second trap lives inside LangGraph. A `BaseCallbackHandler` attached to the\nparent runnable never fires on inner agent tool calls, because LangGraph\ncreates a child runtime per subgraph and callbacks do not inherit (P28). Spans\ninside subgraphs appear orphaned in the waterfall — or they do not appear at\nall — and SLO dashboards under-count latency on the exact calls that matter\nmost: the nested agent loops.\n\nThis skill wires LangChain 1.0 \u002F LangGraph 1.0 into an OTEL-native backend\n(Jaeger, Honeycomb, Grafana Tempo, Datadog) with a correct content-capture\npolicy, subgraph-aware span propagation, and five LLM-specific SLOs (p95 \u002F p99\nlatency, error rate, cost-per-request, TTFT) with burn-rate alerts. Pin:\n`langchain-core 1.0.x`, `langgraph 1.0.x`,\n`opentelemetry-instrumentation-langchain >= 0.33`, OTEL GenAI semconv as of\n2026-04. Pain-catalog anchors: P27, P28 (and cross-references P04, P34, P37).\n\n## Prerequisites\n\n- Python 3.10+\n- `langchain-core >= 1.0, \u003C 2.0`, `langgraph >= 1.0, \u003C 2.0`\n- An OTEL-native backend picked: Jaeger (dev), Honeycomb \u002F Tempo \u002F Datadog (prod)\n- For multi-tenant: upstream redaction middleware already in place (see\n  `langchain-security-basics` and `langchain-middleware-patterns`)\n- Access to set env vars at deploy time (`OTLP_ENDPOINT`, API keys)\n\n## Instructions\n\n### Step 1 — Install the SDK and instrumentor, configure the exporter\n\n```bash\npip install \\\n  opentelemetry-api \\\n  opentelemetry-sdk \\\n  opentelemetry-exporter-otlp-proto-http \\\n  \"opentelemetry-instrumentation-langchain>=0.33\"\n```\n\n```python\nimport os\nfrom opentelemetry import trace\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\nfrom opentelemetry.sdk.resources import Resource\nfrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter\nfrom opentelemetry.instrumentation.langchain import LangchainInstrumentor\n\nresource = Resource.create({\n    \"service.name\": \"my-langchain-app\",\n    \"service.version\": \"1.0.0\",\n    \"deployment.environment\": os.getenv(\"ENV\", \"dev\"),\n})\nprovider = TracerProvider(resource=resource)\nprovider.add_span_processor(BatchSpanProcessor(\n    OTLPSpanExporter(\n        endpoint=os.environ[\"OTLP_ENDPOINT\"],       # per-backend; see matrix\n        headers=_parse_headers(os.getenv(\"OTLP_HEADERS\", \"\")),\n    ),\n    max_queue_size=2048,        # spans buffered before drop; raise for high volume\n    max_export_batch_size=512,  # batched export keeps per-span overhead under 1ms\n))\ntrace.set_tracer_provider(provider)\n\nLangchainInstrumentor().instrument()   # emits gen_ai.* attrs on every run\n```\n\n`BatchSpanProcessor` keeps per-span overhead well under 1 ms. Use\n`SimpleSpanProcessor` only in local dev — it blocks the call path per span.\n\nPer-backend `OTLP_ENDPOINT` and header config lives in\n[Backend Setup Matrix](references\u002Fbackend-setup-matrix.md) — Jaeger,\nHoneycomb, Grafana Tempo, Datadog.\n\n### Step 2 — Verify the GenAI attribute schema\n\nTrigger one call and inspect what landed in the backend. LangChain 1.0 emits\nthese `gen_ai.*` attributes natively on every chat-model span:\n\n| Attribute | Example |\n|-----------|---------|\n| `gen_ai.system` | `anthropic` |\n| `gen_ai.request.model` | `claude-sonnet-4-6` |\n| `gen_ai.request.temperature` | `0.0` |\n| `gen_ai.usage.input_tokens` | `1234` |\n| `gen_ai.usage.output_tokens` | `567` |\n| `gen_ai.response.finish_reasons` | `[\"stop\"]` |\n\nMissing anything? Likely a stale instrumentor version or an outdated provider\npackage. The full emitted-vs-custom matrix plus LangGraph's span taxonomy\n(`LangGraph.invoke` → `LangGraph.node.*` → `LangGraph.subgraph.*`) is in\n[GenAI Semantic Conventions](references\u002Fgenai-semantic-conventions.md).\n\n### Step 3 — Decide on prompt-content capture (critical — do not skip)\n\nThe engineer's instinct is to flip the capture flag to see prompts. Before\nflipping it, classify the workload into one of these buckets:\n\n| Workload | Flag | Notes |\n|----------|------|-------|\n| Dev \u002F staging with synthetic inputs | `true` | Fine. Do not copy these traces to prod. |\n| Single-tenant internal tool | `true` | Fine if RBAC on backend is tight. |\n| Single-tenant product, signed compliance artifacts | `true` | BAA \u002F DPIA in place; retention policy matches log retention. |\n| Multi-tenant SaaS, **no upstream redaction** | **`false`** | Hard no. Fix redaction first. |\n| Multi-tenant SaaS, **with upstream redaction** | `true` | Safe — the span sees the already-redacted text. |\n| Healthcare \u002F finance \u002F legal without legal sign-off | **`false`** | Hard no. |\n\n```bash\n# trusted single-tenant ONLY\nexport OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true\nexport TRACELOOP_TRACE_CONTENT=true   # OpenLLMetry alias; set both to be safe\n```\n\nLeave unset (default) anywhere else. To capture bodies in a multi-tenant\nsystem, wire redaction middleware upstream of the model call first — see\n[Prompt Content Policy](references\u002Fprompt-content-policy.md) and cross-reference\npack siblings `langchain-security-basics` (PII redaction middleware pattern,\nP34) and `langchain-middleware-patterns` (middleware order: redact → cache →\nmodel, P24). **Failure pattern P27** — prompts missing from traces because\ncapture was never opted in — is the #1 first-day OTEL complaint; make the\ndecision explicit instead of surprise-flipping the flag in prod.\n\n### Step 4 — Propagate callbacks through subgraphs (P28)\n\nLangGraph creates a child runtime per subgraph. Callbacks bound at the parent\ndefinition time do **not** inherit:\n\n```python\n# WRONG — subagent spans orphaned or missing (P28)\nagent = create_react_agent(model=llm, tools=tools).with_config(\n    callbacks=[my_handler]  # bound at definition time; children do not see it\n)\nagent.invoke({\"messages\": [...]})\n\n# RIGHT — pass callbacks at invocation via config; they propagate down\nagent.invoke(\n    {\"messages\": [...]},\n    config={\"callbacks\": [my_handler]}  # invocation-time; inherited by children\n)\n```\n\nThe same rule applies to custom attribute handlers (e.g. the\n`CostAttributeHandler` in the semantic-conventions reference that stamps\n`gen_ai.usage.cost_usd` on each model span). Attach via\n`config[\"callbacks\"]`, never via `.with_config()`. **Failure pattern P28\nsymptom:** SLO dashboards show low latency because the slow nested spans are\nmissing entirely, not because the nested calls are fast.\n\n### Step 5 — Define LLM SLOs and dashboards\n\nFive SLIs matter from day one. All five derive from `gen_ai.*` span attributes\n— no second pipeline required:\n\n| SLI | Target example | Why |\n|-----|----------------|-----|\n| **p95 latency** (top-level chat) | `\u003C 5 s` for chat UI | Provider variance dominates |\n| **p99 latency** | `\u003C 15 s` | Tail matters on chat; agents with tools live here |\n| **Error rate** | `\u003C 0.5%` | Includes 429s + `finish_reason IN (\"length\",\"content_filter\")` |\n| **Cost per request** (p95) | `\u003C $0.05` | Catches `haiku`→`opus` regressions |\n| **TTFT p95** (streaming) | `\u003C 2 s` | Perceived latency, not total duration |\n\nConcrete Honeycomb \u002F PromQL \u002F Datadog queries for each SLI, plus multi-window\nmulti-burn-rate alerts (14.4× \u002F 1h fast burn, 6× \u002F 6h slow burn), are in\n[LLM SLO Dashboards](references\u002Fllm-slo-dashboards.md).\n\n### Step 6 — Tune sampling\n\nDefaults are wrong for two ends of the volume spectrum:\n\n```python\nfrom opentelemetry.sdk.trace.sampling import TraceIdRatioBased\n\n# Low\u002Fmedium volume — keep everything for debuggability\n# (\u003C ~100 req\u002Fs) — SDK default 100% is fine\n\n# High volume — head-sample, but carve out errors + slow spans via tail sampling\n# at the OTEL Collector (see references\u002Fllm-slo-dashboards.md)\nprovider = TracerProvider(\n    resource=resource,\n    sampler=TraceIdRatioBased(0.10),  # 10% head sample\n)\n```\n\n**Watch out:** head sampling at 10% means 90% of p99 outliers are discarded\nbefore they reach the backend — p99 metrics become noisy and biased toward\nthe median. For tail-latency SLOs, move sampling to a Collector with\n`tailsamplingprocessor` so errors and slow spans (`latency > 5000ms`) are\nalways kept while the rest is probabilistically sampled at 10%. Typical trace\noverhead with `BatchSpanProcessor` at the 512-span batch size: **under 1 ms\nper span**; recommended sampling rate for high-volume production is **1-10%**.\n\n## Output\n\n- OTEL exporter wired to a chosen backend (Jaeger \u002F Honeycomb \u002F Tempo \u002F Datadog)\n- `opentelemetry-instrumentation-langchain` emitting `gen_ai.*` attrs on every\n  LangChain and LangGraph span\n- Explicit prompt-content capture decision recorded against a workload bucket,\n  with the multi-tenant guardrail enforced upstream\n- Callbacks propagated via `config[\"callbacks\"]` at invocation time so\n  subgraph spans nest correctly under their parent node\n- Five LLM SLOs (p95 \u002F p99 latency, error rate, cost-per-request, TTFT) with\n  dashboards and MWMBR burn-rate alerts\n- Sampling strategy matched to workload volume and SLO precision needs\n\n## Error Handling\n\n| Symptom | Cause | Fix |\n|---------|-------|-----|\n| Traces land but prompt and completion bodies are empty | `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` unset (P27 — privacy-safe default) | Set to `true` **only** for the workload buckets in Step 3; for multi-tenant, wire upstream redaction first |\n| Subgraph \u002F tool-call spans orphaned or missing | Callbacks bound via `.with_config()` at definition time (P28) | Pass via `config[\"callbacks\"]` at invocation time so children inherit |\n| `gen_ai.usage.cache_read_input_tokens` resets every call | Per-call usage, aggregation is your job (P04) | Custom callback summing across calls keyed by `session.id`; see `langchain-model-inference` |\n| p99 dashboard looks noisy and median-biased | 10% head sampling drops outliers before backend | Move to Collector `tailsamplingprocessor` — always keep errors and `latency > 5000ms` |\n| Traces never appear | `OTLPSpanExporter` endpoint wrong protocol (gRPC on 4317 vs HTTP on 4318) | Verify with `curl -v $OTLP_ENDPOINT`; swap to the `proto-grpc` exporter package if your backend expects gRPC |\n| Cost attribute missing from spans | LangChain 1.0 does not emit `gen_ai.usage.cost_usd` natively | Add a `BaseCallbackHandler` that computes from tokens × pricing; see semantic-conventions reference |\n| PR review flags `sk-...` in trace attributes | Secrets in prompts captured via `gen_ai.prompt.content` (P37-adjacent) | Upstream redactor must strip API-key patterns before model call; audit via 0.1% sampler |\n| Exporter dropping spans silently | Queue overflow at high volume | Increase `max_queue_size` to 4096+; add Collector between SDK and backend |\n\n## Examples\n\n### Running Jaeger locally for dev-loop tracing\n\nSpin up Jaeger in Docker, point the SDK at `http:\u002F\u002Flocalhost:4318\u002Fv1\u002Ftraces`,\nleave content capture on (it's dev, inputs are synthetic). You get a generic\nspan waterfall — no LLM-specific UX, but good for verifying the instrumentor\nemits what you expect before paying for a SaaS backend.\n\nSee [Backend Setup Matrix](references\u002Fbackend-setup-matrix.md) for the\n`docker run` command and SDK config.\n\n### Investigating an agent latency incident in Honeycomb\n\nHoneycomb's BubbleUp over `gen_ai.request.model`, `gen_ai.usage.input_tokens`,\nand tool call count is the fastest path from \"p95 spiked at 14:00\" to \"one\nspecific tool took 20 s because the vectorstore was slow.\" Requires\ncontent-capture-off by default so you can turn the team loose on search\nwithout PII-leak worries.\n\nSee [LLM SLO Dashboards](references\u002Fllm-slo-dashboards.md) for the exact\nHoneycomb query shape.\n\n### Dual-exporting during a LangSmith → Tempo migration\n\nRegister two `BatchSpanProcessor`s — one to LangSmith's OTLP endpoint, one to\nTempo. Run both for two weeks, compare waterfalls, cut over. LangSmith handles\nLLM-specific analytics; Tempo handles unified trace search across LLM and\nnon-LLM services in your Grafana stack.\n\nSee [Backend Setup Matrix](references\u002Fbackend-setup-matrix.md) dual-export\nsection.\n\n## Resources\n\n- [OTEL GenAI semantic conventions](https:\u002F\u002Fopentelemetry.io\u002Fdocs\u002Fspecs\u002Fsemconv\u002Fgen-ai\u002F)\n- [OTEL Python SDK](https:\u002F\u002Fopentelemetry.io\u002Fdocs\u002Flanguages\u002Fpython\u002F)\n- [OpenLLMetry LangChain instrumentation](https:\u002F\u002Fgithub.com\u002Ftraceloop\u002Fopenllmetry\u002Ftree\u002Fmain\u002Fpackages\u002Fopentelemetry-instrumentation-langchain)\n- [Honeycomb OTLP ingest](https:\u002F\u002Fdocs.honeycomb.io\u002Fgetting-data-in\u002Fotlp\u002F)\n- [Grafana Tempo](https:\u002F\u002Fgrafana.com\u002Fdocs\u002Ftempo\u002Flatest\u002F)\n- [Datadog OTLP](https:\u002F\u002Fdocs.datadoghq.com\u002Fopentelemetry\u002F)\n- [Google SRE — Alerting on SLOs](https:\u002F\u002Fsre.google\u002Fworkbook\u002Falerting-on-slos\u002F)\n- Pack cross-references: `langchain-security-basics` (redaction, P34),\n  `langchain-middleware-patterns` (order: redact → cache → model, P24),\n  `langchain-model-inference` (cost callback pattern, P04)\n- Pack pain catalog: `docs\u002Fpain-catalog.md` — P27 (content-capture default),\n  P28 (subgraph callback propagation), P04 (cache token aggregation),\n  P34 (prompt injection), P37 (secrets in env \u002F prompts)\n","# LangChain OTEL Observability (Python)\n\n## Overview\n\nAn engineer wires OpenTelemetry expecting to see prompts and responses in\nHoneycomb. The traces land — but only timing, model name, and token counts\nappear. The prompt body is blank. This is **not** a bug: it's the OTEL GenAI\nsemantic-conventions privacy-safe default (P27), where\n`OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` is off. The instinct is to\nflip it on and move on. On a multi-tenant workload that flip is a leak — the\nnext engineer to search traces for Tenant A sees Tenant B's PII in the results,\nbecause redaction was supposed to happen upstream and never did.\n\nA second trap lives inside LangGraph. A `BaseCallbackHandler` attached to the\nparent runnable never fires on inner agent tool calls, because LangGraph\ncreates a child runtime per subgraph and callbacks do not inherit (P28). Spans\ninside subgraphs appear orphaned in the waterfall — or they do not appear at\nall — and SLO dashboards under-count latency on the exact calls that matter\nmost: the nested agent loops.\n\nThis skill wires LangChain 1.0 \u002F LangGraph 1.0 into an OTEL-native backend\n(Jaeger, Honeycomb, Grafana Tempo, Datadog) with a correct content-capture\npolicy, subgraph-aware span propagation, and five LLM-specific SLOs (p95 \u002F p99\nlatency, error rate, cost-per-request, TTFT) with burn-rate alerts. Pin:\n`langchain-core 1.0.x`, `langgraph 1.0.x`,\n`opentelemetry-instrumentation-langchain >= 0.33`, OTEL GenAI semconv as of\n2026-04. Pain-catalog anchors: P27, P28 (and cross-references P04, P34, P37).\n\n## Prerequisites\n\n- Python 3.10+\n- `langchain-core >= 1.0, \u003C 2.0`, `langgraph >= 1.0, \u003C 2.0`\n- An OTEL-native backend picked: Jaeger (dev), Honeycomb \u002F Tempo \u002F Datadog (prod)\n- For multi-tenant: upstream redaction middleware already in place (see\n  `langchain-security-basics` and `langchain-middleware-patterns`)\n- Access to set env vars at deploy time (`OTLP_ENDPOINT`, API keys)\n\n## Instructions\n\n### Step 1 — Install the SDK and instrumentor, configure the exporter\n\n```bash\npip install \\\n  opentelemetry-api \\\n  opentelemetry-sdk \\\n  opentelemetry-exporter-otlp-proto-http \\\n  \"opentelemetry-instrumentation-langchain>=0.33\"\n```\n\n```python\nimport os\nfrom opentelemetry import trace\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\nfrom opentelemetry.sdk.resources import Resource\nfrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter\nfrom opentelemetry.instrumentation.langchain import LangchainInstrumentor\n\nresource = Resource.create({\n    \"service.name\": \"my-langchain-app\",\n    \"service.version\": \"1.0.0\",\n    \"deployment.environment\": os.getenv(\"ENV\", \"dev\"),\n})\nprovider = TracerProvider(resource=resource)\nprovider.add_span_processor(BatchSpanProcessor(\n    OTLPSpanExporter(\n        endpoint=os.environ[\"OTLP_ENDPOINT\"],       # per-backend; see matrix\n        headers=_parse_headers(os.getenv(\"OTLP_HEADERS\", \"\")),\n    ),\n    max_queue_size=2048,        # spans buffered before drop; raise for high volume\n    max_export_batch_size=512,  # batched export keeps per-span overhead under 1ms\n))\ntrace.set_tracer_provider(provider)\n\nLangchainInstrumentor().instrument()   # emits gen_ai.* attrs on every run\n```\n\n`BatchSpanProcessor` keeps per-span overhead well under 1 ms. Use\n`SimpleSpanProcessor` only in local dev — it blocks the call path per span.\n\nPer-backend `OTLP_ENDPOINT` and header config lives in\n[Backend Setup Matrix](references\u002Fbackend-setup-matrix.md) — Jaeger,\nHoneycomb, Grafana Tempo, Datadog.\n\n### Step 2 — Verify the GenAI attribute schema\n\nTrigger one call and inspect what landed in the backend. LangChain 1.0 emits\nthese `gen_ai.*` attributes natively on every chat-model span:\n\n| Attribute | Example |\n|-----------|---------|\n| `gen_ai.system` | `anthropic` |\n| `gen_ai.request.model` | `claude-sonnet-4-6` |\n| `gen_ai.request.temperature` | `0.0` |\n| `gen_ai.usage.input_tokens` | `1234` |\n| `gen_ai.usage.output_tokens` | `567` |\n| `gen_ai.response.finish_reasons` | `[\"stop\"]` |\n\nMissing anything? Likely a stale instrumentor version or an outdated provider\npackage. The full emitted-vs-custom matrix plus LangGraph's span taxonomy\n(`LangGraph.invoke` → `LangGraph.node.*` → `LangGraph.subgraph.*`) is in\n[GenAI Semantic Conventions](references\u002Fgenai-semantic-conventions.md).\n\n### Step 3 — Decide on prompt-content capture (critical — do not skip)\n\nThe engineer's instinct is to flip the capture flag to see prompts. Before\nflipping it, classify the workload into one of these buckets:\n\n| Workload | Flag | Notes |\n|----------|------|-------|\n| Dev \u002F staging with synthetic inputs | `true` | Fine. Do not copy these traces to prod. |\n| Single-tenant internal tool | `true` | Fine if RBAC on backend is tight. |\n| Single-tenant product, signed compliance artifacts | `true` | BAA \u002F DPIA in place; retention policy matches log retention. |\n| Multi-tenant SaaS, **no upstream redaction** | **`false`** | Hard no. Fix redaction first. |\n| Multi-tenant SaaS, **with upstream redaction** | `true` | Safe — the span sees the already-redacted text. |\n| Healthcare \u002F finance \u002F legal without legal sign-off | **`false`** | Hard no. |\n\n```bash\n# trusted single-tenant ONLY\nexport OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true\nexport TRACELOOP_TRACE_CONTENT=true   # OpenLLMetry alias; set both to be safe\n```\n\nLeave unset (default) anywhere else. To capture bodies in a multi-tenant\nsystem, wire redaction middleware upstream of the model call first — see\n[Prompt Content Policy](references\u002Fprompt-content-policy.md) and cross-reference\npack siblings `langchain-security-basics` (PII redaction middleware pattern,\nP34) and `langchain-middleware-patterns` (middleware order: redact → cache →\nmodel, P24). **Failure pattern P27** — prompts missing from traces because\ncapture was never opted in — is the #1 first-day OTEL complaint; make the\ndecision explicit instead of surprise-flipping the flag in prod.\n\n### Step 4 — Propagate callbacks through subgraphs (P28)\n\nLangGraph creates a child runtime per subgraph. Callbacks bound at the parent\ndefinition time do **not** inherit:\n\n```python\n# WRONG — subagent spans orphaned or missing (P28)\nagent = create_react_agent(model=llm, tools=tools).with_config(\n    callbacks=[my_handler]  # bound at definition time; children do not see it\n)\nagent.invoke({\"messages\": [...]})\n\n# RIGHT — pass callbacks at invocation via config; they propagate down\nagent.invoke(\n    {\"messages\": [...]},\n    config={\"callbacks\": [my_handler]}  # invocation-time; inherited by children\n)\n```\n\nThe same rule applies to custom attribute handlers (e.g. the\n`CostAttributeHandler` in the semantic-conventions reference that stamps\n`gen_ai.usage.cost_usd` on each model span). Attach via\n`config[\"callbacks\"]`, never via `.with_config()`. **Failure pattern P28\nsymptom:** SLO dashboards show low latency because the slow nested spans are\nmissing entirely, not because the nested calls are fast.\n\n### Step 5 — Define LLM SLOs and dashboards\n\nFive SLIs matter from day one. All five derive from `gen_ai.*` span attributes\n— no second pipeline required:\n\n| SLI | Target example | Why |\n|-----|----------------|-----|\n| **p95 latency** (top-level chat) | `\u003C 5 s` for chat UI | Provider variance dominates |\n| **p99 latency** | `\u003C 15 s` | Tail matters on chat; agents with tools live here |\n| **Error rate** | `\u003C 0.5%` | Includes 429s + `finish_reason IN (\"length\",\"content_filter\")` |\n| **Cost per request** (p95) | `\u003C $0.05` | Catches `haiku`→`opus` regressions |\n| **TTFT p95** (streaming) | `\u003C 2 s` | Perceived latency, not total duration |\n\nConcrete Honeycomb \u002F PromQL \u002F Datadog queries for each SLI, plus multi-window\nmulti-burn-rate alerts (14.4× \u002F 1h fast burn, 6× \u002F 6h slow burn), are in\n[LLM SLO Dashboards](references\u002Fllm-slo-dashboards.md).\n\n### Step 6 — Tune sampling\n\nDefaults are wrong for two ends of the volume spectrum:\n\n```python\nfrom opentelemetry.sdk.trace.sampling import TraceIdRatioBased\n\n# Low\u002Fmedium volume — keep everything for debuggability\n# (\u003C ~100 req\u002Fs) — SDK default 100% is fine\n\n# High volume — head-sample, but carve out errors + slow spans via tail sampling\n# at the OTEL Collector (see references\u002Fllm-slo-dashboards.md)\nprovider = TracerProvider(\n    resource=resource,\n    sampler=TraceIdRatioBased(0.10),  # 10% head sample\n)\n```\n\n**Watch out:** head sampling at 10% means 90% of p99 outliers are discarded\nbefore they reach the backend — p99 metrics become noisy and biased toward\nthe median. For tail-latency SLOs, move sampling to a Collector with\n`tailsamplingprocessor` so errors and slow spans (`latency > 5000ms`) are\nalways kept while the rest is probabilistically sampled at 10%. Typical trace\noverhead with `BatchSpanProcessor` at the 512-span batch size: **under 1 ms\nper span**; recommended sampling rate for high-volume production is **1-10%**.\n\n## Output\n\n- OTEL exporter wired to a chosen backend (Jaeger \u002F Honeycomb \u002F Tempo \u002F Datadog)\n- `opentelemetry-instrumentation-langchain` emitting `gen_ai.*` attrs on every\n  LangChain and LangGraph span\n- Explicit prompt-content capture decision recorded against a workload bucket,\n  with the multi-tenant guardrail enforced upstream\n- Callbacks propagated via `config[\"callbacks\"]` at invocation time so\n  subgraph spans nest correctly under their parent node\n- Five LLM SLOs (p95 \u002F p99 latency, error rate, cost-per-request, TTFT) with\n  dashboards and MWMBR burn-rate alerts\n- Sampling strategy matched to workload volume and SLO precision needs\n\n## Error Handling\n\n| Symptom | Cause | Fix |\n|---------|-------|-----|\n| Traces land but prompt and completion bodies are empty | `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` unset (P27 — privacy-safe default) | Set to `true` **only** for the workload buckets in Step 3; for multi-tenant, wire upstream redaction first |\n| Subgraph \u002F tool-call spans orphaned or missing | Callbacks bound via `.with_config()` at definition time (P28) | Pass via `config[\"callbacks\"]` at invocation time so children inherit |\n| `gen_ai.usage.cache_read_input_tokens` resets every call | Per-call usage, aggregation is your job (P04) | Custom callback summing across calls keyed by `session.id`; see `langchain-model-inference` |\n| p99 dashboard looks noisy and median-biased | 10% head sampling drops outliers before backend | Move to Collector `tailsamplingprocessor` — always keep errors and `latency > 5000ms` |\n| Traces never appear | `OTLPSpanExporter` endpoint wrong protocol (gRPC on 4317 vs HTTP on 4318) | Verify with `curl -v $OTLP_ENDPOINT`; swap to the `proto-grpc` exporter package if your backend expects gRPC |\n| Cost attribute missing from spans | LangChain 1.0 does not emit `gen_ai.usage.cost_usd` natively | Add a `BaseCallbackHandler` that computes from tokens × pricing; see semantic-conventions reference |\n| PR review flags `sk-...` in trace attributes | Secrets in prompts captured via `gen_ai.prompt.content` (P37-adjacent) | Upstream redactor must strip API-key patterns before model call; audit via 0.1% sampler |\n| Exporter dropping spans silently | Queue overflow at high volume | Increase `max_queue_size` to 4096+; add Collector between SDK and backend |\n\n## Examples\n\n### Running Jaeger locally for dev-loop tracing\n\nSpin up Jaeger in Docker, point the SDK at `http:\u002F\u002Flocalhost:4318\u002Fv1\u002Ftraces`,\nleave content capture on (it's dev, inputs are synthetic). You get a generic\nspan waterfall — no LLM-specific UX, but good for verifying the instrumentor\nemits what you expect before paying for a SaaS backend.\n\nSee [Backend Setup Matrix](references\u002Fbackend-setup-matrix.md) for the\n`docker run` command and SDK config.\n\n### Investigating an agent latency incident in Honeycomb\n\nHoneycomb's BubbleUp over `gen_ai.request.model`, `gen_ai.usage.input_tokens`,\nand tool call count is the fastest path from \"p95 spiked at 14:00\" to \"one\nspecific tool took 20 s because the vectorstore was slow.\" Requires\ncontent-capture-off by default so you can turn the team loose on search\nwithout PII-leak worries.\n\nSee [LLM SLO Dashboards](references\u002Fllm-slo-dashboards.md) for the exact\nHoneycomb query shape.\n\n### Dual-exporting during a LangSmith → Tempo migration\n\nRegister two `BatchSpanProcessor`s — one to LangSmith's OTLP endpoint, one to\nTempo. Run both for two weeks, compare waterfalls, cut over. LangSmith handles\nLLM-specific analytics; Tempo handles unified trace search across LLM and\nnon-LLM services in your Grafana stack.\n\nSee [Backend Setup Matrix](references\u002Fbackend-setup-matrix.md) dual-export\nsection.\n\n## Resources\n\n- [OTEL GenAI semantic conventions](https:\u002F\u002Fopentelemetry.io\u002Fdocs\u002Fspecs\u002Fsemconv\u002Fgen-ai\u002F)\n- [OTEL Python SDK](https:\u002F\u002Fopentelemetry.io\u002Fdocs\u002Flanguages\u002Fpython\u002F)\n- [OpenLLMetry LangChain instrumentation](https:\u002F\u002Fgithub.com\u002Ftraceloop\u002Fopenllmetry\u002Ftree\u002Fmain\u002Fpackages\u002Fopentelemetry-instrumentation-langchain)\n- [Honeycomb OTLP ingest](https:\u002F\u002Fdocs.honeycomb.io\u002Fgetting-data-in\u002Fotlp\u002F)\n- [Grafana Tempo](https:\u002F\u002Fgrafana.com\u002Fdocs\u002Ftempo\u002Flatest\u002F)\n- [Datadog OTLP](https:\u002F\u002Fdocs.datadoghq.com\u002Fopentelemetry\u002F)\n- [Google SRE — Alerting on SLOs](https:\u002F\u002Fsre.google\u002Fworkbook\u002Falerting-on-slos\u002F)\n- Pack cross-references: `langchain-security-basics` (redaction, P34),\n  `langchain-middleware-patterns` (order: redact → cache → model, P24),\n  `langchain-model-inference` (cost callback pattern, P04)\n- Pack pain catalog: `docs\u002Fpain-catalog.md` — P27 (content-capture default),\n  P28 (subgraph callback propagation), P04 (cache token aggregation),\n  P34 (prompt injection), P37 (secrets in env \u002F prompts)\n",{"name":5,"tags":24,"author":34,"license":15,"version":35,"description":36,"allowed-tools":37,"compatible-with":38},[25,26,27,28,29,30,31,32,33],"saas","langchain","langgraph","python","langchain-1.0","observability","opentelemetry","jaeger","honeycomb","Jeremy Longshore \u003Cjeremy@intentsolutions.io>","2.0.0","Wire LangChain 1.0 \u002F LangGraph 1.0 traces into an OpenTelemetry-native backend\n(Jaeger, Honeycomb, Grafana Tempo, Datadog) with LLM-specific SLOs, safe\nprompt-content policy, and subgraph-aware span propagation. Use when LangSmith\nis not the right fit (existing OTEL stack, compliance, multi-cloud) or\nalongside LangSmith for deep-system traces.\nTrigger with \"langchain OTEL\", \"langchain opentelemetry\", \"langchain jaeger\",\n\"langchain honeycomb\", \"langchain SLO\", \"LLM span\", \"langchain tempo\",\n\"langchain datadog tracing\".\n","Read, Write, Edit, Bash(python:*), Bash(pip:*)","claude-code, codex","main","99825ad3f3bcc6597ddcc7294a14e6e922f2b67b","skills\u002Fsaas-packs--langchain-py-pack--langchain-otel-observability",0,"https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills","https:\u002F\u002Fgithub.com\u002Fjukeyman","https:\u002F\u002FRJBUSINESSSOLUTIONS.ORG","Credit and Finance Expert | Developer\r\n\r\nI'm Rick Jefferson, a seasoned credit and finance expert with over a decade of experience in financial literacy,","2026-07-13 07:03:29",{"source":8,"skillName":5,"commands":49,"prompt":59,"zipAvailable":60},[50,53,56],{"tool":51,"command":52},"npx","npx skills add jukeyman\u002Fjukeyman-skills --skill langchain-otel-observability",{"tool":54,"command":55},"bunx","bunx skills add jukeyman\u002Fjukeyman-skills --skill langchain-otel-observability",{"tool":57,"command":58},"pnpm","pnpm dlx skills add jukeyman\u002Fjukeyman-skills --skill langchain-otel-observability","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fsaas-packs--langchain-py-pack--langchain-otel-observability\u002FSKILL.md",true,[],[63,70,77,84,91,99],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":19},"01M79ANS2SM329NEP5ZCX106RR","zustand-store-creator","Create zustand store creator operations. Auto-activating skill for Frontend Development.\nTriggers on: zustand store creator, zustand store creator\nPart of the Frontend Development skill category. Use when working with zustand store creator functionality. Trigger with phrases like \"zustand store creator\", \"zustand creator\", \"zustand\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fzustand-store-creator\u002FSKILL.md",[],[],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M79AMZKP4YDEZ7EHG1Y7PP74","zero-trust-config-helper","Configure with zero trust config helper operations. Auto-activating skill for Security Advanced.\nTriggers on: zero trust config helper, zero trust config helper\nPart of the Security Advanced skill category. Use when configuring systems or services. Trigger with phrases like \"zero trust config helper\", \"zero helper\", \"zero\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fzero-trust-config-helper\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":19},"01M79AMJSBCEJQNBTYNAHYN5HM","zapier-integration-helper","Assist with zapier integration helper operations. Auto-activating skill for Business Automation.\nTriggers on: zapier integration helper, zapier integration helper\nPart of the Business Automation skill category. Use when working with APIs or building integrations. Trigger with phrases like \"zapier integration helper\", \"zapier helper\", \"zapier\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fzapier-integration-helper\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":19},"01M79AHJZGVMARS13VFQYRR1MP","yaml-config-validator","Validate yaml config validator operations. Auto-activating skill for DevOps Basics.\nTriggers on: yaml config validator, yaml config validator\nPart of the DevOps Basics skill category. Use when configuring systems or services. Trigger with phrases like \"yaml config validator\", \"yaml validator\", \"yaml\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fyaml-config-validator\u002FSKILL.md",[],[],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":95,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":19},"01M79AHGDZRN0020RDQSQDPAM8","xvary-stock-research","Thesis-driven equity analysis from public SEC EDGAR and market data; \u002Fanalyze, \u002Fscore, \u002Fcompare workflows with bundled Python tools (Claude Code, Cursor, Codex).","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fxvary-stock-research\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fxvary-stock-research\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":19},"01M79AHCYJ9FV9RMHQ24HDRN33","xss-vulnerability-scanner","Scan xss vulnerability scanner operations. Auto-activating skill for Security Fundamentals.\nTriggers on: xss vulnerability scanner, xss vulnerability scanner\nPart of the Security Fundamentals skill category. Use when working with xss vulnerability scanner functionality. Trigger with phrases like \"xss vulnerability scanner\", \"xss scanner\", \"xss\".","https:\u002F\u002Fgithub.com\u002Fjukeyman\u002Fjukeyman-skills\u002Fblob\u002Fmain\u002Fskills\u002Fxss-vulnerability-scanner\u002FSKILL.md",[],[]]