[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6F1CP8GKGQ8TH12TW9MAS9R":3,"skill-same:01M6F1CP8GKGQ8TH12TW9MAS9R":75},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":41,"linkOnly":42,"markdownContent":43,"markdownBody":44,"frontMatter":45,"refName":46,"commitSha":47,"skillDirectory":48,"forksCount":49,"repositoryHtmlUrl":50,"repositoryHomepageUrl":51,"ownerHtmlUrl":52,"ownerBlogUrl":6,"ownerDescription":53,"updatedAt":54,"install":55,"resources":68},"01M6F1CP8GKGQ8TH12TW9MAS9R","distributed-tracing",null,"Implement distributed tracing with Jaeger and Tempo to track requests across microservices and identify performance bottlenecks. Use when debugging microservices, analyzing request flows, or implementing observability for distributed systems.","ComeOnOliver\u002Fskillshub","ComeOnOliver","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F43215937?v=4","01M6E5JNFECJAAVFE0W7SMWWD2","Haoran Shu","01M6E5JNNDQS7AJZDYFQ8TPZCK",67,"MIT","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fwshobson\u002Fdistributed-tracing\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000011","运维与部署","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40],"distributed","distributed tracing","tracing","analyzing","with","flows","tempo","jaeger","systems","distributed systems","across","use","performance","microservices","observability","debugging","implement","2026-11-10 10:12:30",false,"---\nname: distributed-tracing\ndescription: Implement distributed tracing with Jaeger and Tempo to track requests across microservices and identify performance bottlenecks. Use when debugging microservices, analyzing request flows, or implementing observability for distributed systems.\n---\n\n# Distributed Tracing\n\nImplement distributed tracing with Jaeger and Tempo for request flow visibility across microservices.\n\n## Purpose\n\nTrack requests across distributed systems to understand latency, dependencies, and failure points.\n\n## When to Use\n\n- Debug latency issues\n- Understand service dependencies\n- Identify bottlenecks\n- Trace error propagation\n- Analyze request paths\n\n## Distributed Tracing Concepts\n\n### Trace Structure\n```\nTrace (Request ID: abc123)\n  ↓\nSpan (frontend) [100ms]\n  ↓\nSpan (api-gateway) [80ms]\n  ├→ Span (auth-service) [10ms]\n  └→ Span (user-service) [60ms]\n      └→ Span (database) [40ms]\n```\n\n### Key Components\n- **Trace** - End-to-end request journey\n- **Span** - Single operation within a trace\n- **Context** - Metadata propagated between services\n- **Tags** - Key-value pairs for filtering\n- **Logs** - Timestamped events within a span\n\n## Jaeger Setup\n\n### Kubernetes Deployment\n\n```bash\n# Deploy Jaeger Operator\nkubectl create namespace observability\nkubectl create -f https:\u002F\u002Fgithub.com\u002Fjaegertracing\u002Fjaeger-operator\u002Freleases\u002Fdownload\u002Fv1.51.0\u002Fjaeger-operator.yaml -n observability\n\n# Deploy Jaeger instance\nkubectl apply -f - \u003C\u003CEOF\napiVersion: jaegertracing.io\u002Fv1\nkind: Jaeger\nmetadata:\n  name: jaeger\n  namespace: observability\nspec:\n  strategy: production\n  storage:\n    type: elasticsearch\n    options:\n      es:\n        server-urls: http:\u002F\u002Felasticsearch:9200\n  ingress:\n    enabled: true\nEOF\n```\n\n### Docker Compose\n\n```yaml\nversion: '3.8'\nservices:\n  jaeger:\n    image: jaegertracing\u002Fall-in-one:latest\n    ports:\n      - \"5775:5775\u002Fudp\"\n      - \"6831:6831\u002Fudp\"\n      - \"6832:6832\u002Fudp\"\n      - \"5778:5778\"\n      - \"16686:16686\"  # UI\n      - \"14268:14268\"  # Collector\n      - \"14250:14250\"  # gRPC\n      - \"9411:9411\"    # Zipkin\n    environment:\n      - COLLECTOR_ZIPKIN_HOST_PORT=:9411\n```\n\n**Reference:** See `references\u002Fjaeger-setup.md`\n\n## Application Instrumentation\n\n### OpenTelemetry (Recommended)\n\n#### Python (Flask)\n```python\nfrom opentelemetry import trace\nfrom opentelemetry.exporter.jaeger.thrift import JaegerExporter\nfrom opentelemetry.sdk.resources import SERVICE_NAME, Resource\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\nfrom opentelemetry.instrumentation.flask import FlaskInstrumentor\nfrom flask import Flask\n\n# Initialize tracer\nresource = Resource(attributes={SERVICE_NAME: \"my-service\"})\nprovider = TracerProvider(resource=resource)\nprocessor = BatchSpanProcessor(JaegerExporter(\n    agent_host_name=\"jaeger\",\n    agent_port=6831,\n))\nprovider.add_span_processor(processor)\ntrace.set_tracer_provider(provider)\n\n# Instrument Flask\napp = Flask(__name__)\nFlaskInstrumentor().instrument_app(app)\n\n@app.route('\u002Fapi\u002Fusers')\ndef get_users():\n    tracer = trace.get_tracer(__name__)\n\n    with tracer.start_as_current_span(\"get_users\") as span:\n        span.set_attribute(\"user.count\", 100)\n        # Business logic\n        users = fetch_users_from_db()\n        return {\"users\": users}\n\ndef fetch_users_from_db():\n    tracer = trace.get_tracer(__name__)\n\n    with tracer.start_as_current_span(\"database_query\") as span:\n        span.set_attribute(\"db.system\", \"postgresql\")\n        span.set_attribute(\"db.statement\", \"SELECT * FROM users\")\n        # Database query\n        return query_database()\n```\n\n#### Node.js (Express)\n```javascript\nconst { NodeTracerProvider } = require('@opentelemetry\u002Fsdk-trace-node');\nconst { JaegerExporter } = require('@opentelemetry\u002Fexporter-jaeger');\nconst { BatchSpanProcessor } = require('@opentelemetry\u002Fsdk-trace-base');\nconst { registerInstrumentations } = require('@opentelemetry\u002Finstrumentation');\nconst { HttpInstrumentation } = require('@opentelemetry\u002Finstrumentation-http');\nconst { ExpressInstrumentation } = require('@opentelemetry\u002Finstrumentation-express');\n\n\u002F\u002F Initialize tracer\nconst provider = new NodeTracerProvider({\n  resource: { attributes: { 'service.name': 'my-service' } }\n});\n\nconst exporter = new JaegerExporter({\n  endpoint: 'http:\u002F\u002Fjaeger:14268\u002Fapi\u002Ftraces'\n});\n\nprovider.addSpanProcessor(new BatchSpanProcessor(exporter));\nprovider.register();\n\n\u002F\u002F Instrument libraries\nregisterInstrumentations({\n  instrumentations: [\n    new HttpInstrumentation(),\n    new ExpressInstrumentation(),\n  ],\n});\n\nconst express = require('express');\nconst app = express();\n\napp.get('\u002Fapi\u002Fusers', async (req, res) => {\n  const tracer = trace.getTracer('my-service');\n  const span = tracer.startSpan('get_users');\n\n  try {\n    const users = await fetchUsers();\n    span.setAttributes({ 'user.count': users.length });\n    res.json({ users });\n  } finally {\n    span.end();\n  }\n});\n```\n\n#### Go\n```go\npackage main\n\nimport (\n    \"context\"\n    \"go.opentelemetry.io\u002Fotel\"\n    \"go.opentelemetry.io\u002Fotel\u002Fexporters\u002Fjaeger\"\n    \"go.opentelemetry.io\u002Fotel\u002Fsdk\u002Fresource\"\n    sdktrace \"go.opentelemetry.io\u002Fotel\u002Fsdk\u002Ftrace\"\n    semconv \"go.opentelemetry.io\u002Fotel\u002Fsemconv\u002Fv1.4.0\"\n)\n\nfunc initTracer() (*sdktrace.TracerProvider, error) {\n    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(\n        jaeger.WithEndpoint(\"http:\u002F\u002Fjaeger:14268\u002Fapi\u002Ftraces\"),\n    ))\n    if err != nil {\n        return nil, err\n    }\n\n    tp := sdktrace.NewTracerProvider(\n        sdktrace.WithBatcher(exporter),\n        sdktrace.WithResource(resource.NewWithAttributes(\n            semconv.SchemaURL,\n            semconv.ServiceNameKey.String(\"my-service\"),\n        )),\n    )\n\n    otel.SetTracerProvider(tp)\n    return tp, nil\n}\n\nfunc getUsers(ctx context.Context) ([]User, error) {\n    tracer := otel.Tracer(\"my-service\")\n    ctx, span := tracer.Start(ctx, \"get_users\")\n    defer span.End()\n\n    span.SetAttributes(attribute.String(\"user.filter\", \"active\"))\n\n    users, err := fetchUsersFromDB(ctx)\n    if err != nil {\n        span.RecordError(err)\n        return nil, err\n    }\n\n    span.SetAttributes(attribute.Int(\"user.count\", len(users)))\n    return users, nil\n}\n```\n\n**Reference:** See `references\u002Finstrumentation.md`\n\n## Context Propagation\n\n### HTTP Headers\n```\ntraceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01\ntracestate: congo=t61rcWkgMzE\n```\n\n### Propagation in HTTP Requests\n\n#### Python\n```python\nfrom opentelemetry.propagate import inject\n\nheaders = {}\ninject(headers)  # Injects trace context\n\nresponse = requests.get('http:\u002F\u002Fdownstream-service\u002Fapi', headers=headers)\n```\n\n#### Node.js\n```javascript\nconst { propagation } = require('@opentelemetry\u002Fapi');\n\nconst headers = {};\npropagation.inject(context.active(), headers);\n\naxios.get('http:\u002F\u002Fdownstream-service\u002Fapi', { headers });\n```\n\n## Tempo Setup (Grafana)\n\n### Kubernetes Deployment\n\n```yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: tempo-config\ndata:\n  tempo.yaml: |\n    server:\n      http_listen_port: 3200\n\n    distributor:\n      receivers:\n        jaeger:\n          protocols:\n            thrift_http:\n            grpc:\n        otlp:\n          protocols:\n            http:\n            grpc:\n\n    storage:\n      trace:\n        backend: s3\n        s3:\n          bucket: tempo-traces\n          endpoint: s3.amazonaws.com\n\n    querier:\n      frontend_worker:\n        frontend_address: tempo-query-frontend:9095\n---\napiVersion: apps\u002Fv1\nkind: Deployment\nmetadata:\n  name: tempo\nspec:\n  replicas: 1\n  template:\n    spec:\n      containers:\n      - name: tempo\n        image: grafana\u002Ftempo:latest\n        args:\n          - -config.file=\u002Fetc\u002Ftempo\u002Ftempo.yaml\n        volumeMounts:\n        - name: config\n          mountPath: \u002Fetc\u002Ftempo\n      volumes:\n      - name: config\n        configMap:\n          name: tempo-config\n```\n\n**Reference:** See `assets\u002Fjaeger-config.yaml.template`\n\n## Sampling Strategies\n\n### Probabilistic Sampling\n```yaml\n# Sample 1% of traces\nsampler:\n  type: probabilistic\n  param: 0.01\n```\n\n### Rate Limiting Sampling\n```yaml\n# Sample max 100 traces per second\nsampler:\n  type: ratelimiting\n  param: 100\n```\n\n### Adaptive Sampling\n```python\nfrom opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased\n\n# Sample based on trace ID (deterministic)\nsampler = ParentBased(root=TraceIdRatioBased(0.01))\n```\n\n## Trace Analysis\n\n### Finding Slow Requests\n\n**Jaeger Query:**\n```\nservice=my-service\nduration > 1s\n```\n\n### Finding Errors\n\n**Jaeger Query:**\n```\nservice=my-service\nerror=true\ntags.http.status_code >= 500\n```\n\n### Service Dependency Graph\n\nJaeger automatically generates service dependency graphs showing:\n- Service relationships\n- Request rates\n- Error rates\n- Average latencies\n\n## Best Practices\n\n1. **Sample appropriately** (1-10% in production)\n2. **Add meaningful tags** (user_id, request_id)\n3. **Propagate context** across all service boundaries\n4. **Log exceptions** in spans\n5. **Use consistent naming** for operations\n6. **Monitor tracing overhead** (\u003C1% CPU impact)\n7. **Set up alerts** for trace errors\n8. **Implement distributed context** (baggage)\n9. **Use span events** for important milestones\n10. **Document instrumentation** standards\n\n## Integration with Logging\n\n### Correlated Logs\n```python\nimport logging\nfrom opentelemetry import trace\n\nlogger = logging.getLogger(__name__)\n\ndef process_request():\n    span = trace.get_current_span()\n    trace_id = span.get_span_context().trace_id\n\n    logger.info(\n        \"Processing request\",\n        extra={\"trace_id\": format(trace_id, '032x')}\n    )\n```\n\n## Troubleshooting\n\n**No traces appearing:**\n- Check collector endpoint\n- Verify network connectivity\n- Check sampling configuration\n- Review application logs\n\n**High latency overhead:**\n- Reduce sampling rate\n- Use batch span processor\n- Check exporter configuration\n\n## Reference Files\n\n- `references\u002Fjaeger-setup.md` - Jaeger installation\n- `references\u002Finstrumentation.md` - Instrumentation patterns\n- `assets\u002Fjaeger-config.yaml.template` - Jaeger configuration\n\n## Related Skills\n\n- `prometheus-configuration` - For metrics\n- `grafana-dashboards` - For visualization\n- `slo-implementation` - For latency SLOs\n","# Distributed Tracing\n\nImplement distributed tracing with Jaeger and Tempo for request flow visibility across microservices.\n\n## Purpose\n\nTrack requests across distributed systems to understand latency, dependencies, and failure points.\n\n## When to Use\n\n- Debug latency issues\n- Understand service dependencies\n- Identify bottlenecks\n- Trace error propagation\n- Analyze request paths\n\n## Distributed Tracing Concepts\n\n### Trace Structure\n```\nTrace (Request ID: abc123)\n  ↓\nSpan (frontend) [100ms]\n  ↓\nSpan (api-gateway) [80ms]\n  ├→ Span (auth-service) [10ms]\n  └→ Span (user-service) [60ms]\n      └→ Span (database) [40ms]\n```\n\n### Key Components\n- **Trace** - End-to-end request journey\n- **Span** - Single operation within a trace\n- **Context** - Metadata propagated between services\n- **Tags** - Key-value pairs for filtering\n- **Logs** - Timestamped events within a span\n\n## Jaeger Setup\n\n### Kubernetes Deployment\n\n```bash\n# Deploy Jaeger Operator\nkubectl create namespace observability\nkubectl create -f https:\u002F\u002Fgithub.com\u002Fjaegertracing\u002Fjaeger-operator\u002Freleases\u002Fdownload\u002Fv1.51.0\u002Fjaeger-operator.yaml -n observability\n\n# Deploy Jaeger instance\nkubectl apply -f - \u003C\u003CEOF\napiVersion: jaegertracing.io\u002Fv1\nkind: Jaeger\nmetadata:\n  name: jaeger\n  namespace: observability\nspec:\n  strategy: production\n  storage:\n    type: elasticsearch\n    options:\n      es:\n        server-urls: http:\u002F\u002Felasticsearch:9200\n  ingress:\n    enabled: true\nEOF\n```\n\n### Docker Compose\n\n```yaml\nversion: '3.8'\nservices:\n  jaeger:\n    image: jaegertracing\u002Fall-in-one:latest\n    ports:\n      - \"5775:5775\u002Fudp\"\n      - \"6831:6831\u002Fudp\"\n      - \"6832:6832\u002Fudp\"\n      - \"5778:5778\"\n      - \"16686:16686\"  # UI\n      - \"14268:14268\"  # Collector\n      - \"14250:14250\"  # gRPC\n      - \"9411:9411\"    # Zipkin\n    environment:\n      - COLLECTOR_ZIPKIN_HOST_PORT=:9411\n```\n\n**Reference:** See `references\u002Fjaeger-setup.md`\n\n## Application Instrumentation\n\n### OpenTelemetry (Recommended)\n\n#### Python (Flask)\n```python\nfrom opentelemetry import trace\nfrom opentelemetry.exporter.jaeger.thrift import JaegerExporter\nfrom opentelemetry.sdk.resources import SERVICE_NAME, Resource\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\nfrom opentelemetry.instrumentation.flask import FlaskInstrumentor\nfrom flask import Flask\n\n# Initialize tracer\nresource = Resource(attributes={SERVICE_NAME: \"my-service\"})\nprovider = TracerProvider(resource=resource)\nprocessor = BatchSpanProcessor(JaegerExporter(\n    agent_host_name=\"jaeger\",\n    agent_port=6831,\n))\nprovider.add_span_processor(processor)\ntrace.set_tracer_provider(provider)\n\n# Instrument Flask\napp = Flask(__name__)\nFlaskInstrumentor().instrument_app(app)\n\n@app.route('\u002Fapi\u002Fusers')\ndef get_users():\n    tracer = trace.get_tracer(__name__)\n\n    with tracer.start_as_current_span(\"get_users\") as span:\n        span.set_attribute(\"user.count\", 100)\n        # Business logic\n        users = fetch_users_from_db()\n        return {\"users\": users}\n\ndef fetch_users_from_db():\n    tracer = trace.get_tracer(__name__)\n\n    with tracer.start_as_current_span(\"database_query\") as span:\n        span.set_attribute(\"db.system\", \"postgresql\")\n        span.set_attribute(\"db.statement\", \"SELECT * FROM users\")\n        # Database query\n        return query_database()\n```\n\n#### Node.js (Express)\n```javascript\nconst { NodeTracerProvider } = require('@opentelemetry\u002Fsdk-trace-node');\nconst { JaegerExporter } = require('@opentelemetry\u002Fexporter-jaeger');\nconst { BatchSpanProcessor } = require('@opentelemetry\u002Fsdk-trace-base');\nconst { registerInstrumentations } = require('@opentelemetry\u002Finstrumentation');\nconst { HttpInstrumentation } = require('@opentelemetry\u002Finstrumentation-http');\nconst { ExpressInstrumentation } = require('@opentelemetry\u002Finstrumentation-express');\n\n\u002F\u002F Initialize tracer\nconst provider = new NodeTracerProvider({\n  resource: { attributes: { 'service.name': 'my-service' } }\n});\n\nconst exporter = new JaegerExporter({\n  endpoint: 'http:\u002F\u002Fjaeger:14268\u002Fapi\u002Ftraces'\n});\n\nprovider.addSpanProcessor(new BatchSpanProcessor(exporter));\nprovider.register();\n\n\u002F\u002F Instrument libraries\nregisterInstrumentations({\n  instrumentations: [\n    new HttpInstrumentation(),\n    new ExpressInstrumentation(),\n  ],\n});\n\nconst express = require('express');\nconst app = express();\n\napp.get('\u002Fapi\u002Fusers', async (req, res) => {\n  const tracer = trace.getTracer('my-service');\n  const span = tracer.startSpan('get_users');\n\n  try {\n    const users = await fetchUsers();\n    span.setAttributes({ 'user.count': users.length });\n    res.json({ users });\n  } finally {\n    span.end();\n  }\n});\n```\n\n#### Go\n```go\npackage main\n\nimport (\n    \"context\"\n    \"go.opentelemetry.io\u002Fotel\"\n    \"go.opentelemetry.io\u002Fotel\u002Fexporters\u002Fjaeger\"\n    \"go.opentelemetry.io\u002Fotel\u002Fsdk\u002Fresource\"\n    sdktrace \"go.opentelemetry.io\u002Fotel\u002Fsdk\u002Ftrace\"\n    semconv \"go.opentelemetry.io\u002Fotel\u002Fsemconv\u002Fv1.4.0\"\n)\n\nfunc initTracer() (*sdktrace.TracerProvider, error) {\n    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(\n        jaeger.WithEndpoint(\"http:\u002F\u002Fjaeger:14268\u002Fapi\u002Ftraces\"),\n    ))\n    if err != nil {\n        return nil, err\n    }\n\n    tp := sdktrace.NewTracerProvider(\n        sdktrace.WithBatcher(exporter),\n        sdktrace.WithResource(resource.NewWithAttributes(\n            semconv.SchemaURL,\n            semconv.ServiceNameKey.String(\"my-service\"),\n        )),\n    )\n\n    otel.SetTracerProvider(tp)\n    return tp, nil\n}\n\nfunc getUsers(ctx context.Context) ([]User, error) {\n    tracer := otel.Tracer(\"my-service\")\n    ctx, span := tracer.Start(ctx, \"get_users\")\n    defer span.End()\n\n    span.SetAttributes(attribute.String(\"user.filter\", \"active\"))\n\n    users, err := fetchUsersFromDB(ctx)\n    if err != nil {\n        span.RecordError(err)\n        return nil, err\n    }\n\n    span.SetAttributes(attribute.Int(\"user.count\", len(users)))\n    return users, nil\n}\n```\n\n**Reference:** See `references\u002Finstrumentation.md`\n\n## Context Propagation\n\n### HTTP Headers\n```\ntraceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01\ntracestate: congo=t61rcWkgMzE\n```\n\n### Propagation in HTTP Requests\n\n#### Python\n```python\nfrom opentelemetry.propagate import inject\n\nheaders = {}\ninject(headers)  # Injects trace context\n\nresponse = requests.get('http:\u002F\u002Fdownstream-service\u002Fapi', headers=headers)\n```\n\n#### Node.js\n```javascript\nconst { propagation } = require('@opentelemetry\u002Fapi');\n\nconst headers = {};\npropagation.inject(context.active(), headers);\n\naxios.get('http:\u002F\u002Fdownstream-service\u002Fapi', { headers });\n```\n\n## Tempo Setup (Grafana)\n\n### Kubernetes Deployment\n\n```yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: tempo-config\ndata:\n  tempo.yaml: |\n    server:\n      http_listen_port: 3200\n\n    distributor:\n      receivers:\n        jaeger:\n          protocols:\n            thrift_http:\n            grpc:\n        otlp:\n          protocols:\n            http:\n            grpc:\n\n    storage:\n      trace:\n        backend: s3\n        s3:\n          bucket: tempo-traces\n          endpoint: s3.amazonaws.com\n\n    querier:\n      frontend_worker:\n        frontend_address: tempo-query-frontend:9095\n---\napiVersion: apps\u002Fv1\nkind: Deployment\nmetadata:\n  name: tempo\nspec:\n  replicas: 1\n  template:\n    spec:\n      containers:\n      - name: tempo\n        image: grafana\u002Ftempo:latest\n        args:\n          - -config.file=\u002Fetc\u002Ftempo\u002Ftempo.yaml\n        volumeMounts:\n        - name: config\n          mountPath: \u002Fetc\u002Ftempo\n      volumes:\n      - name: config\n        configMap:\n          name: tempo-config\n```\n\n**Reference:** See `assets\u002Fjaeger-config.yaml.template`\n\n## Sampling Strategies\n\n### Probabilistic Sampling\n```yaml\n# Sample 1% of traces\nsampler:\n  type: probabilistic\n  param: 0.01\n```\n\n### Rate Limiting Sampling\n```yaml\n# Sample max 100 traces per second\nsampler:\n  type: ratelimiting\n  param: 100\n```\n\n### Adaptive Sampling\n```python\nfrom opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased\n\n# Sample based on trace ID (deterministic)\nsampler = ParentBased(root=TraceIdRatioBased(0.01))\n```\n\n## Trace Analysis\n\n### Finding Slow Requests\n\n**Jaeger Query:**\n```\nservice=my-service\nduration > 1s\n```\n\n### Finding Errors\n\n**Jaeger Query:**\n```\nservice=my-service\nerror=true\ntags.http.status_code >= 500\n```\n\n### Service Dependency Graph\n\nJaeger automatically generates service dependency graphs showing:\n- Service relationships\n- Request rates\n- Error rates\n- Average latencies\n\n## Best Practices\n\n1. **Sample appropriately** (1-10% in production)\n2. **Add meaningful tags** (user_id, request_id)\n3. **Propagate context** across all service boundaries\n4. **Log exceptions** in spans\n5. **Use consistent naming** for operations\n6. **Monitor tracing overhead** (\u003C1% CPU impact)\n7. **Set up alerts** for trace errors\n8. **Implement distributed context** (baggage)\n9. **Use span events** for important milestones\n10. **Document instrumentation** standards\n\n## Integration with Logging\n\n### Correlated Logs\n```python\nimport logging\nfrom opentelemetry import trace\n\nlogger = logging.getLogger(__name__)\n\ndef process_request():\n    span = trace.get_current_span()\n    trace_id = span.get_span_context().trace_id\n\n    logger.info(\n        \"Processing request\",\n        extra={\"trace_id\": format(trace_id, '032x')}\n    )\n```\n\n## Troubleshooting\n\n**No traces appearing:**\n- Check collector endpoint\n- Verify network connectivity\n- Check sampling configuration\n- Review application logs\n\n**High latency overhead:**\n- Reduce sampling rate\n- Use batch span processor\n- Check exporter configuration\n\n## Reference Files\n\n- `references\u002Fjaeger-setup.md` - Jaeger installation\n- `references\u002Finstrumentation.md` - Instrumentation patterns\n- `assets\u002Fjaeger-config.yaml.template` - Jaeger configuration\n\n## Related Skills\n\n- `prometheus-configuration` - For metrics\n- `grafana-dashboards` - For visualization\n- `slo-implementation` - For latency SLOs\n",{"name":5,"description":7},"main","def8531e65114c0fca8fb8551c1871ee0eed705c","skills\u002Faiskillstore\u002Fmarketplace\u002Fwshobson\u002Fdistributed-tracing",24,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub","https:\u002F\u002Fskillshub.wtf","https:\u002F\u002Fgithub.com\u002FComeOnOliver","AI & Blockchain Engineer\r\n","2026-06-24 22:07:19",{"source":8,"skillName":5,"commands":56,"prompt":66,"zipAvailable":67},[57,60,63],{"tool":58,"command":59},"npx","npx skills add ComeOnOliver\u002Fskillshub --skill distributed-tracing",{"tool":61,"command":62},"bunx","bunx skills add ComeOnOliver\u002Fskillshub --skill distributed-tracing",{"tool":64,"command":65},"pnpm","pnpm dlx skills add ComeOnOliver\u002Fskillshub --skill distributed-tracing","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fwshobson\u002Fdistributed-tracing\u002FSKILL.md",true,[69],{"path":70,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":73,"sourceHtmlUrl":74},"skill-report.json","REFERENCE","参考文档",15664,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fwshobson\u002Fdistributed-tracing\u002Fskill-report.json",[76,88,98,107,113,120],{"skillNo":77,"name":78,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":84,"publishedAt":41},"01M6FKJN13GMG4KY57J3QD8GY5","model-hierarchy-skill","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzscole\u002Fmodel-hierarchy-skill\u002Fmodel-hierarchy-skill\u002FSKILL.md",[81],{"categoryNo":82,"categoryName":83,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发",[85,86,87],"model","model hierarchy","model hierarchy skill",{"skillNo":89,"name":90,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":94,"publishedAt":41},"01M6FKJJJ6GCAMR4PPPRD12M5X","zoom-mcp","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-mcp\u002FSKILL.md",[93],{"categoryNo":82,"categoryName":83,"taxonomyType":21,"taxonomyTypeDesc":22},[95,96,97],"MCP","zoom","zoom mcp",{"skillNo":99,"name":100,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":41},"01M6FKJE3KYT3KPEZMEEHYKA6K","zoom-apps-sdk","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-apps-sdk\u002FSKILL.md",[],[104,105,106,96],"sdk","zoom apps sdk","apps",{"skillNo":108,"name":109,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":41},"01M6FKJ9EDH9Q5VWDF8HAJRSP0","windows","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwindows\u002FSKILL.md",[],[109],{"skillNo":114,"name":115,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":116,"categories":117,"keywords":119,"publishedAt":41},"01M6FKJ666BEEFC3AE9GJ72Q93","whiteboard","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwhiteboard\u002FSKILL.md",[118],{"categoryNo":82,"categoryName":83,"taxonomyType":21,"taxonomyTypeDesc":22},[115],{"skillNo":121,"name":122,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":123,"categories":124,"keywords":125,"publishedAt":41},"01M6FKJ21WFCDGTJQ2Q91T9DVW","websockets","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwebsockets\u002FSKILL.md",[],[122]]