langchain-observability

LangChain Observability

Contents

• Overview
• Prerequisites
• Instructions
• Output
• Error Handling
• Examples
• Resources

Overview

Set up comprehensive observability for LangChain applications with LangSmith, OpenTelemetry, Prometheus, and structured logging.

Prerequisites

• LangChain application in staging/production
• LangSmith account (optional but recommended)
• Prometheus/Grafana infrastructure
• OpenTelemetry collector (optional)

Instructions

Step 1: Enable LangSmith Tracing

Set LANGCHAIN_TRACING_V2=true and configure LANGCHAIN_API_KEY and LANGCHAIN_PROJECT environment variables. All chains are automatically traced.

Step 2: Add Prometheus Metrics

Create a PrometheusCallback handler that tracks langchain_llm_requests_total, langchain_llm_latency_seconds, and langchain_llm_tokens_total counters/histograms.

Step 3: Integrate OpenTelemetry

Use OTLPSpanExporter with a custom OpenTelemetryCallback to add spans for chain and LLM operations with parent-child relationships.

Step 4: Configure Structured Logging

Use structlog with a StructuredLoggingCallback to emit JSON logs for all LLM start/end/error events.

Step 5: Set Up Grafana Dashboard

Create panels for request rate, P95 latency, token usage, and error rate using Prometheus queries.

Step 6: Configure Alerting Rules

Define Prometheus alerts for high error rate (>5%), high latency (P95 >5s), and token budget exceeded.

See detailed implementation for complete callback code, dashboard JSON, and alert rules.

Output

• LangSmith tracing enabled
• Prometheus metrics exported
• OpenTelemetry spans
• Structured logging
• Grafana dashboard and alerts

Error Handling

Issue	Cause	Solution
Missing metrics	Callback not attached	Pass callback to LLM constructor
Trace gaps	Missing context propagation	Check parent span handling
Alert storms	Thresholds too sensitive	Tune for duration and thresholds

Examples

Basic usage: Apply langchain observability to a standard project setup with default configuration options.

Advanced scenario: Customize langchain observability for production environments with multiple constraints and team-specific requirements.

Resources

• LangSmith Documentation
• OpenTelemetry Python
• Prometheus Python Client

Next Steps

Use langchain-incident-runbook for incident response procedures.