[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6HRE5KFVBFGQMV683SP7GPF":3,"skill-same:01M6HRE5KFVBFGQMV683SP7GPF":72},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":22,"publishedAt":46,"linkOnly":47,"markdownContent":48,"markdownBody":49,"frontMatter":50,"refName":51,"commitSha":52,"skillDirectory":53,"forksCount":54,"repositoryHtmlUrl":55,"repositoryHomepageUrl":6,"ownerHtmlUrl":56,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":57,"install":58,"resources":71},"01M6HRE5KFVBFGQMV683SP7GPF","distributed-tracing",null,"Implement distributed tracing with Jaeger and Tempo to track requests across microservices and identify performance bottlenecks. Use when debugging microservices, analyzing request flows, or implementing observability for distributed systems.","welflau\u002Fai-dev-system","welflau","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F7010385?v=4","01M6HPTDGF18RFY07M229P8CTM","01M6HPTDMRMZCJPYBFEA9XVVYT",3,"MIT","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Fobservability-monitoring\u002Fskills\u002Fdistributed-tracing\u002FSKILL.md",[17],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000011","运维与部署","SCENARIO","使用场景",[23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45],"distributed","distributed tracing","tracing","distributed systems","when","request","track","for","implementing","requests","with","flows","tempo","jaeger","systems","performance","across","use","analyzing","microservices","observability","debugging","implement","2026-11-10 10:12:30",false,"---\nname: distributed-tracing\ndescription: Implement distributed tracing with Jaeger and Tempo to track requests across microservices and identify performance bottlenecks. Use when debugging microservices, analyzing request flows, or implementing observability for distributed systems.\n---\n\n# Distributed Tracing\n\nImplement distributed tracing with Jaeger and Tempo for request flow visibility across microservices.\n\n## Purpose\n\nTrack requests across distributed systems to understand latency, dependencies, and failure points.\n\n## When to Use\n\n- Debug latency issues\n- Understand service dependencies\n- Identify bottlenecks\n- Trace error propagation\n- Analyze request paths\n\n## Distributed Tracing Concepts\n\n### Trace Structure\n```\nTrace (Request ID: abc123)\n  ↓\nSpan (frontend) [100ms]\n  ↓\nSpan (api-gateway) [80ms]\n  ├→ Span (auth-service) [10ms]\n  └→ Span (user-service) [60ms]\n      └→ Span (database) [40ms]\n```\n\n### Key Components\n- **Trace** - End-to-end request journey\n- **Span** - Single operation within a trace\n- **Context** - Metadata propagated between services\n- **Tags** - Key-value pairs for filtering\n- **Logs** - Timestamped events within a span\n\n## Jaeger Setup\n\n### Kubernetes Deployment\n\n```bash\n# Deploy Jaeger Operator\nkubectl create namespace observability\nkubectl create -f https:\u002F\u002Fgithub.com\u002Fjaegertracing\u002Fjaeger-operator\u002Freleases\u002Fdownload\u002Fv1.51.0\u002Fjaeger-operator.yaml -n observability\n\n# Deploy Jaeger instance\nkubectl apply -f - \u003C\u003CEOF\napiVersion: jaegertracing.io\u002Fv1\nkind: Jaeger\nmetadata:\n  name: jaeger\n  namespace: observability\nspec:\n  strategy: production\n  storage:\n    type: elasticsearch\n    options:\n      es:\n        server-urls: http:\u002F\u002Felasticsearch:9200\n  ingress:\n    enabled: true\nEOF\n```\n\n### Docker Compose\n\n```yaml\nversion: '3.8'\nservices:\n  jaeger:\n    image: jaegertracing\u002Fall-in-one:latest\n    ports:\n      - \"5775:5775\u002Fudp\"\n      - \"6831:6831\u002Fudp\"\n      - \"6832:6832\u002Fudp\"\n      - \"5778:5778\"\n      - \"16686:16686\"  # UI\n      - \"14268:14268\"  # Collector\n      - \"14250:14250\"  # gRPC\n      - \"9411:9411\"    # Zipkin\n    environment:\n      - COLLECTOR_ZIPKIN_HOST_PORT=:9411\n```\n\n**Reference:** See `references\u002Fjaeger-setup.md`\n\n## Application Instrumentation\n\n### OpenTelemetry (Recommended)\n\n#### Python (Flask)\n```python\nfrom opentelemetry import trace\nfrom opentelemetry.exporter.jaeger.thrift import JaegerExporter\nfrom opentelemetry.sdk.resources import SERVICE_NAME, Resource\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\nfrom opentelemetry.instrumentation.flask import FlaskInstrumentor\nfrom flask import Flask\n\n# Initialize tracer\nresource = Resource(attributes={SERVICE_NAME: \"my-service\"})\nprovider = TracerProvider(resource=resource)\nprocessor = BatchSpanProcessor(JaegerExporter(\n    agent_host_name=\"jaeger\",\n    agent_port=6831,\n))\nprovider.add_span_processor(processor)\ntrace.set_tracer_provider(provider)\n\n# Instrument Flask\napp = Flask(__name__)\nFlaskInstrumentor().instrument_app(app)\n\n@app.route('\u002Fapi\u002Fusers')\ndef get_users():\n    tracer = trace.get_tracer(__name__)\n\n    with tracer.start_as_current_span(\"get_users\") as span:\n        span.set_attribute(\"user.count\", 100)\n        # Business logic\n        users = fetch_users_from_db()\n        return {\"users\": users}\n\ndef fetch_users_from_db():\n    tracer = trace.get_tracer(__name__)\n\n    with tracer.start_as_current_span(\"database_query\") as span:\n        span.set_attribute(\"db.system\", \"postgresql\")\n        span.set_attribute(\"db.statement\", \"SELECT * FROM users\")\n        # Database query\n        return query_database()\n```\n\n#### Node.js (Express)\n```javascript\nconst { NodeTracerProvider } = require('@opentelemetry\u002Fsdk-trace-node');\nconst { JaegerExporter } = require('@opentelemetry\u002Fexporter-jaeger');\nconst { BatchSpanProcessor } = require('@opentelemetry\u002Fsdk-trace-base');\nconst { registerInstrumentations } = require('@opentelemetry\u002Finstrumentation');\nconst { HttpInstrumentation } = require('@opentelemetry\u002Finstrumentation-http');\nconst { ExpressInstrumentation } = require('@opentelemetry\u002Finstrumentation-express');\n\n\u002F\u002F Initialize tracer\nconst provider = new NodeTracerProvider({\n  resource: { attributes: { 'service.name': 'my-service' } }\n});\n\nconst exporter = new JaegerExporter({\n  endpoint: 'http:\u002F\u002Fjaeger:14268\u002Fapi\u002Ftraces'\n});\n\nprovider.addSpanProcessor(new BatchSpanProcessor(exporter));\nprovider.register();\n\n\u002F\u002F Instrument libraries\nregisterInstrumentations({\n  instrumentations: [\n    new HttpInstrumentation(),\n    new ExpressInstrumentation(),\n  ],\n});\n\nconst express = require('express');\nconst app = express();\n\napp.get('\u002Fapi\u002Fusers', async (req, res) => {\n  const tracer = trace.getTracer('my-service');\n  const span = tracer.startSpan('get_users');\n\n  try {\n    const users = await fetchUsers();\n    span.setAttributes({ 'user.count': users.length });\n    res.json({ users });\n  } finally {\n    span.end();\n  }\n});\n```\n\n#### Go\n```go\npackage main\n\nimport (\n    \"context\"\n    \"go.opentelemetry.io\u002Fotel\"\n    \"go.opentelemetry.io\u002Fotel\u002Fexporters\u002Fjaeger\"\n    \"go.opentelemetry.io\u002Fotel\u002Fsdk\u002Fresource\"\n    sdktrace \"go.opentelemetry.io\u002Fotel\u002Fsdk\u002Ftrace\"\n    semconv \"go.opentelemetry.io\u002Fotel\u002Fsemconv\u002Fv1.4.0\"\n)\n\nfunc initTracer() (*sdktrace.TracerProvider, error) {\n    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(\n        jaeger.WithEndpoint(\"http:\u002F\u002Fjaeger:14268\u002Fapi\u002Ftraces\"),\n    ))\n    if err != nil {\n        return nil, err\n    }\n\n    tp := sdktrace.NewTracerProvider(\n        sdktrace.WithBatcher(exporter),\n        sdktrace.WithResource(resource.NewWithAttributes(\n            semconv.SchemaURL,\n            semconv.ServiceNameKey.String(\"my-service\"),\n        )),\n    )\n\n    otel.SetTracerProvider(tp)\n    return tp, nil\n}\n\nfunc getUsers(ctx context.Context) ([]User, error) {\n    tracer := otel.Tracer(\"my-service\")\n    ctx, span := tracer.Start(ctx, \"get_users\")\n    defer span.End()\n\n    span.SetAttributes(attribute.String(\"user.filter\", \"active\"))\n\n    users, err := fetchUsersFromDB(ctx)\n    if err != nil {\n        span.RecordError(err)\n        return nil, err\n    }\n\n    span.SetAttributes(attribute.Int(\"user.count\", len(users)))\n    return users, nil\n}\n```\n\n**Reference:** See `references\u002Finstrumentation.md`\n\n## Context Propagation\n\n### HTTP Headers\n```\ntraceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01\ntracestate: congo=t61rcWkgMzE\n```\n\n### Propagation in HTTP Requests\n\n#### Python\n```python\nfrom opentelemetry.propagate import inject\n\nheaders = {}\ninject(headers)  # Injects trace context\n\nresponse = requests.get('http:\u002F\u002Fdownstream-service\u002Fapi', headers=headers)\n```\n\n#### Node.js\n```javascript\nconst { propagation } = require('@opentelemetry\u002Fapi');\n\nconst headers = {};\npropagation.inject(context.active(), headers);\n\naxios.get('http:\u002F\u002Fdownstream-service\u002Fapi', { headers });\n```\n\n## Tempo Setup (Grafana)\n\n### Kubernetes Deployment\n\n```yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: tempo-config\ndata:\n  tempo.yaml: |\n    server:\n      http_listen_port: 3200\n\n    distributor:\n      receivers:\n        jaeger:\n          protocols:\n            thrift_http:\n            grpc:\n        otlp:\n          protocols:\n            http:\n            grpc:\n\n    storage:\n      trace:\n        backend: s3\n        s3:\n          bucket: tempo-traces\n          endpoint: s3.amazonaws.com\n\n    querier:\n      frontend_worker:\n        frontend_address: tempo-query-frontend:9095\n---\napiVersion: apps\u002Fv1\nkind: Deployment\nmetadata:\n  name: tempo\nspec:\n  replicas: 1\n  template:\n    spec:\n      containers:\n      - name: tempo\n        image: grafana\u002Ftempo:latest\n        args:\n          - -config.file=\u002Fetc\u002Ftempo\u002Ftempo.yaml\n        volumeMounts:\n        - name: config\n          mountPath: \u002Fetc\u002Ftempo\n      volumes:\n      - name: config\n        configMap:\n          name: tempo-config\n```\n\n**Reference:** See `assets\u002Fjaeger-config.yaml.template`\n\n## Sampling Strategies\n\n### Probabilistic Sampling\n```yaml\n# Sample 1% of traces\nsampler:\n  type: probabilistic\n  param: 0.01\n```\n\n### Rate Limiting Sampling\n```yaml\n# Sample max 100 traces per second\nsampler:\n  type: ratelimiting\n  param: 100\n```\n\n### Adaptive Sampling\n```python\nfrom opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased\n\n# Sample based on trace ID (deterministic)\nsampler = ParentBased(root=TraceIdRatioBased(0.01))\n```\n\n## Trace Analysis\n\n### Finding Slow Requests\n\n**Jaeger Query:**\n```\nservice=my-service\nduration > 1s\n```\n\n### Finding Errors\n\n**Jaeger Query:**\n```\nservice=my-service\nerror=true\ntags.http.status_code >= 500\n```\n\n### Service Dependency Graph\n\nJaeger automatically generates service dependency graphs showing:\n- Service relationships\n- Request rates\n- Error rates\n- Average latencies\n\n## Best Practices\n\n1. **Sample appropriately** (1-10% in production)\n2. **Add meaningful tags** (user_id, request_id)\n3. **Propagate context** across all service boundaries\n4. **Log exceptions** in spans\n5. **Use consistent naming** for operations\n6. **Monitor tracing overhead** (\u003C1% CPU impact)\n7. **Set up alerts** for trace errors\n8. **Implement distributed context** (baggage)\n9. **Use span events** for important milestones\n10. **Document instrumentation** standards\n\n## Integration with Logging\n\n### Correlated Logs\n```python\nimport logging\nfrom opentelemetry import trace\n\nlogger = logging.getLogger(__name__)\n\ndef process_request():\n    span = trace.get_current_span()\n    trace_id = span.get_span_context().trace_id\n\n    logger.info(\n        \"Processing request\",\n        extra={\"trace_id\": format(trace_id, '032x')}\n    )\n```\n\n## Troubleshooting\n\n**No traces appearing:**\n- Check collector endpoint\n- Verify network connectivity\n- Check sampling configuration\n- Review application logs\n\n**High latency overhead:**\n- Reduce sampling rate\n- Use batch span processor\n- Check exporter configuration\n\n## Reference Files\n\n- `references\u002Fjaeger-setup.md` - Jaeger installation\n- `references\u002Finstrumentation.md` - Instrumentation patterns\n- `assets\u002Fjaeger-config.yaml.template` - Jaeger configuration\n\n## Related Skills\n\n- `prometheus-configuration` - For metrics\n- `grafana-dashboards` - For visualization\n- `slo-implementation` - For latency SLOs\n","# Distributed Tracing\n\nImplement distributed tracing with Jaeger and Tempo for request flow visibility across microservices.\n\n## Purpose\n\nTrack requests across distributed systems to understand latency, dependencies, and failure points.\n\n## When to Use\n\n- Debug latency issues\n- Understand service dependencies\n- Identify bottlenecks\n- Trace error propagation\n- Analyze request paths\n\n## Distributed Tracing Concepts\n\n### Trace Structure\n```\nTrace (Request ID: abc123)\n  ↓\nSpan (frontend) [100ms]\n  ↓\nSpan (api-gateway) [80ms]\n  ├→ Span (auth-service) [10ms]\n  └→ Span (user-service) [60ms]\n      └→ Span (database) [40ms]\n```\n\n### Key Components\n- **Trace** - End-to-end request journey\n- **Span** - Single operation within a trace\n- **Context** - Metadata propagated between services\n- **Tags** - Key-value pairs for filtering\n- **Logs** - Timestamped events within a span\n\n## Jaeger Setup\n\n### Kubernetes Deployment\n\n```bash\n# Deploy Jaeger Operator\nkubectl create namespace observability\nkubectl create -f https:\u002F\u002Fgithub.com\u002Fjaegertracing\u002Fjaeger-operator\u002Freleases\u002Fdownload\u002Fv1.51.0\u002Fjaeger-operator.yaml -n observability\n\n# Deploy Jaeger instance\nkubectl apply -f - \u003C\u003CEOF\napiVersion: jaegertracing.io\u002Fv1\nkind: Jaeger\nmetadata:\n  name: jaeger\n  namespace: observability\nspec:\n  strategy: production\n  storage:\n    type: elasticsearch\n    options:\n      es:\n        server-urls: http:\u002F\u002Felasticsearch:9200\n  ingress:\n    enabled: true\nEOF\n```\n\n### Docker Compose\n\n```yaml\nversion: '3.8'\nservices:\n  jaeger:\n    image: jaegertracing\u002Fall-in-one:latest\n    ports:\n      - \"5775:5775\u002Fudp\"\n      - \"6831:6831\u002Fudp\"\n      - \"6832:6832\u002Fudp\"\n      - \"5778:5778\"\n      - \"16686:16686\"  # UI\n      - \"14268:14268\"  # Collector\n      - \"14250:14250\"  # gRPC\n      - \"9411:9411\"    # Zipkin\n    environment:\n      - COLLECTOR_ZIPKIN_HOST_PORT=:9411\n```\n\n**Reference:** See `references\u002Fjaeger-setup.md`\n\n## Application Instrumentation\n\n### OpenTelemetry (Recommended)\n\n#### Python (Flask)\n```python\nfrom opentelemetry import trace\nfrom opentelemetry.exporter.jaeger.thrift import JaegerExporter\nfrom opentelemetry.sdk.resources import SERVICE_NAME, Resource\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\nfrom opentelemetry.instrumentation.flask import FlaskInstrumentor\nfrom flask import Flask\n\n# Initialize tracer\nresource = Resource(attributes={SERVICE_NAME: \"my-service\"})\nprovider = TracerProvider(resource=resource)\nprocessor = BatchSpanProcessor(JaegerExporter(\n    agent_host_name=\"jaeger\",\n    agent_port=6831,\n))\nprovider.add_span_processor(processor)\ntrace.set_tracer_provider(provider)\n\n# Instrument Flask\napp = Flask(__name__)\nFlaskInstrumentor().instrument_app(app)\n\n@app.route('\u002Fapi\u002Fusers')\ndef get_users():\n    tracer = trace.get_tracer(__name__)\n\n    with tracer.start_as_current_span(\"get_users\") as span:\n        span.set_attribute(\"user.count\", 100)\n        # Business logic\n        users = fetch_users_from_db()\n        return {\"users\": users}\n\ndef fetch_users_from_db():\n    tracer = trace.get_tracer(__name__)\n\n    with tracer.start_as_current_span(\"database_query\") as span:\n        span.set_attribute(\"db.system\", \"postgresql\")\n        span.set_attribute(\"db.statement\", \"SELECT * FROM users\")\n        # Database query\n        return query_database()\n```\n\n#### Node.js (Express)\n```javascript\nconst { NodeTracerProvider } = require('@opentelemetry\u002Fsdk-trace-node');\nconst { JaegerExporter } = require('@opentelemetry\u002Fexporter-jaeger');\nconst { BatchSpanProcessor } = require('@opentelemetry\u002Fsdk-trace-base');\nconst { registerInstrumentations } = require('@opentelemetry\u002Finstrumentation');\nconst { HttpInstrumentation } = require('@opentelemetry\u002Finstrumentation-http');\nconst { ExpressInstrumentation } = require('@opentelemetry\u002Finstrumentation-express');\n\n\u002F\u002F Initialize tracer\nconst provider = new NodeTracerProvider({\n  resource: { attributes: { 'service.name': 'my-service' } }\n});\n\nconst exporter = new JaegerExporter({\n  endpoint: 'http:\u002F\u002Fjaeger:14268\u002Fapi\u002Ftraces'\n});\n\nprovider.addSpanProcessor(new BatchSpanProcessor(exporter));\nprovider.register();\n\n\u002F\u002F Instrument libraries\nregisterInstrumentations({\n  instrumentations: [\n    new HttpInstrumentation(),\n    new ExpressInstrumentation(),\n  ],\n});\n\nconst express = require('express');\nconst app = express();\n\napp.get('\u002Fapi\u002Fusers', async (req, res) => {\n  const tracer = trace.getTracer('my-service');\n  const span = tracer.startSpan('get_users');\n\n  try {\n    const users = await fetchUsers();\n    span.setAttributes({ 'user.count': users.length });\n    res.json({ users });\n  } finally {\n    span.end();\n  }\n});\n```\n\n#### Go\n```go\npackage main\n\nimport (\n    \"context\"\n    \"go.opentelemetry.io\u002Fotel\"\n    \"go.opentelemetry.io\u002Fotel\u002Fexporters\u002Fjaeger\"\n    \"go.opentelemetry.io\u002Fotel\u002Fsdk\u002Fresource\"\n    sdktrace \"go.opentelemetry.io\u002Fotel\u002Fsdk\u002Ftrace\"\n    semconv \"go.opentelemetry.io\u002Fotel\u002Fsemconv\u002Fv1.4.0\"\n)\n\nfunc initTracer() (*sdktrace.TracerProvider, error) {\n    exporter, err := jaeger.New(jaeger.WithCollectorEndpoint(\n        jaeger.WithEndpoint(\"http:\u002F\u002Fjaeger:14268\u002Fapi\u002Ftraces\"),\n    ))\n    if err != nil {\n        return nil, err\n    }\n\n    tp := sdktrace.NewTracerProvider(\n        sdktrace.WithBatcher(exporter),\n        sdktrace.WithResource(resource.NewWithAttributes(\n            semconv.SchemaURL,\n            semconv.ServiceNameKey.String(\"my-service\"),\n        )),\n    )\n\n    otel.SetTracerProvider(tp)\n    return tp, nil\n}\n\nfunc getUsers(ctx context.Context) ([]User, error) {\n    tracer := otel.Tracer(\"my-service\")\n    ctx, span := tracer.Start(ctx, \"get_users\")\n    defer span.End()\n\n    span.SetAttributes(attribute.String(\"user.filter\", \"active\"))\n\n    users, err := fetchUsersFromDB(ctx)\n    if err != nil {\n        span.RecordError(err)\n        return nil, err\n    }\n\n    span.SetAttributes(attribute.Int(\"user.count\", len(users)))\n    return users, nil\n}\n```\n\n**Reference:** See `references\u002Finstrumentation.md`\n\n## Context Propagation\n\n### HTTP Headers\n```\ntraceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01\ntracestate: congo=t61rcWkgMzE\n```\n\n### Propagation in HTTP Requests\n\n#### Python\n```python\nfrom opentelemetry.propagate import inject\n\nheaders = {}\ninject(headers)  # Injects trace context\n\nresponse = requests.get('http:\u002F\u002Fdownstream-service\u002Fapi', headers=headers)\n```\n\n#### Node.js\n```javascript\nconst { propagation } = require('@opentelemetry\u002Fapi');\n\nconst headers = {};\npropagation.inject(context.active(), headers);\n\naxios.get('http:\u002F\u002Fdownstream-service\u002Fapi', { headers });\n```\n\n## Tempo Setup (Grafana)\n\n### Kubernetes Deployment\n\n```yaml\napiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: tempo-config\ndata:\n  tempo.yaml: |\n    server:\n      http_listen_port: 3200\n\n    distributor:\n      receivers:\n        jaeger:\n          protocols:\n            thrift_http:\n            grpc:\n        otlp:\n          protocols:\n            http:\n            grpc:\n\n    storage:\n      trace:\n        backend: s3\n        s3:\n          bucket: tempo-traces\n          endpoint: s3.amazonaws.com\n\n    querier:\n      frontend_worker:\n        frontend_address: tempo-query-frontend:9095\n---\napiVersion: apps\u002Fv1\nkind: Deployment\nmetadata:\n  name: tempo\nspec:\n  replicas: 1\n  template:\n    spec:\n      containers:\n      - name: tempo\n        image: grafana\u002Ftempo:latest\n        args:\n          - -config.file=\u002Fetc\u002Ftempo\u002Ftempo.yaml\n        volumeMounts:\n        - name: config\n          mountPath: \u002Fetc\u002Ftempo\n      volumes:\n      - name: config\n        configMap:\n          name: tempo-config\n```\n\n**Reference:** See `assets\u002Fjaeger-config.yaml.template`\n\n## Sampling Strategies\n\n### Probabilistic Sampling\n```yaml\n# Sample 1% of traces\nsampler:\n  type: probabilistic\n  param: 0.01\n```\n\n### Rate Limiting Sampling\n```yaml\n# Sample max 100 traces per second\nsampler:\n  type: ratelimiting\n  param: 100\n```\n\n### Adaptive Sampling\n```python\nfrom opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased\n\n# Sample based on trace ID (deterministic)\nsampler = ParentBased(root=TraceIdRatioBased(0.01))\n```\n\n## Trace Analysis\n\n### Finding Slow Requests\n\n**Jaeger Query:**\n```\nservice=my-service\nduration > 1s\n```\n\n### Finding Errors\n\n**Jaeger Query:**\n```\nservice=my-service\nerror=true\ntags.http.status_code >= 500\n```\n\n### Service Dependency Graph\n\nJaeger automatically generates service dependency graphs showing:\n- Service relationships\n- Request rates\n- Error rates\n- Average latencies\n\n## Best Practices\n\n1. **Sample appropriately** (1-10% in production)\n2. **Add meaningful tags** (user_id, request_id)\n3. **Propagate context** across all service boundaries\n4. **Log exceptions** in spans\n5. **Use consistent naming** for operations\n6. **Monitor tracing overhead** (\u003C1% CPU impact)\n7. **Set up alerts** for trace errors\n8. **Implement distributed context** (baggage)\n9. **Use span events** for important milestones\n10. **Document instrumentation** standards\n\n## Integration with Logging\n\n### Correlated Logs\n```python\nimport logging\nfrom opentelemetry import trace\n\nlogger = logging.getLogger(__name__)\n\ndef process_request():\n    span = trace.get_current_span()\n    trace_id = span.get_span_context().trace_id\n\n    logger.info(\n        \"Processing request\",\n        extra={\"trace_id\": format(trace_id, '032x')}\n    )\n```\n\n## Troubleshooting\n\n**No traces appearing:**\n- Check collector endpoint\n- Verify network connectivity\n- Check sampling configuration\n- Review application logs\n\n**High latency overhead:**\n- Reduce sampling rate\n- Use batch span processor\n- Check exporter configuration\n\n## Reference Files\n\n- `references\u002Fjaeger-setup.md` - Jaeger installation\n- `references\u002Finstrumentation.md` - Instrumentation patterns\n- `assets\u002Fjaeger-config.yaml.template` - Jaeger configuration\n\n## Related Skills\n\n- `prometheus-configuration` - For metrics\n- `grafana-dashboards` - For visualization\n- `slo-implementation` - For latency SLOs\n",{"name":5,"description":7},"main","18d0e63d1313ca87098f614aa7c1e2dcf6de79c7","backend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Fobservability-monitoring\u002Fskills\u002Fdistributed-tracing",1,"https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system","https:\u002F\u002Fgithub.com\u002Fwelflau","2026-11-10 11:14:25",{"source":8,"skillName":5,"commands":59,"prompt":69,"zipAvailable":70},[60,63,66],{"tool":61,"command":62},"npx","npx skills add welflau\u002Fai-dev-system --skill distributed-tracing",{"tool":64,"command":65},"bunx","bunx skills add welflau\u002Fai-dev-system --skill distributed-tracing",{"tool":67,"command":68},"pnpm","pnpm dlx skills add welflau\u002Fai-dev-system --skill distributed-tracing","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Fobservability-monitoring\u002Fskills\u002Fdistributed-tracing\u002FSKILL.md",true,[],[73,104,134,157,183,207],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":77,"licenseFileUrl":78,"sourceHtmlUrl":79,"categories":80,"keywords":84,"publishedAt":46},"01M6HSTDP4M3JFXEBK91VFB4PA","web-artifacts-builder","Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn\u002Fui). Use for complex artifacts requiring state management, routing, or shadcn\u002Fui components - not for simple single-file HTML\u002FJSX artifacts.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fplugins\u002Fweb-artifacts-builder\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fplugins\u002Fweb-artifacts-builder\u002FSKILL.md",[81],{"categoryNo":82,"categoryName":83,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000009","前端开发",[85,86,87,88,89,90,91,92,93,94,95,96,97,98,40,99,100,101,102,30,103],"web artifacts builder","web","artifacts","artifacts builder","builder","frontend","css","components","react","routing","state management","state","file","Tailwind","single-file","simple","ui components","multi","single",{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":77,"licenseFileUrl":108,"sourceHtmlUrl":109,"categories":110,"keywords":120,"publishedAt":46},"01M6HSDXH0JSP8R3QTNY9XZ8HN","frontend-design","Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, artifacts, posters, or applications (examples include websites, landing pages, dashboards, React components, HTML\u002FCSS layouts, or when styling\u002Fbeautifying any web UI). Generates creative, polished code and UI design that avoids generic AI aesthetics.","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fplugins\u002Ffrontend-design\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fplugins\u002Ffrontend-design\u002FSKILL.md",[111,112,115],{"categoryNo":82,"categoryName":83,"taxonomyType":20,"taxonomyTypeDesc":21},{"categoryNo":113,"categoryName":114,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000015","视觉设计",{"categoryNo":116,"categoryName":117,"taxonomyType":118,"taxonomyTypeDesc":119},"01JBSEEDCAT000000000000032","设计师","PROFESSION","适用职业",[90,121,122,91,86,92,93,123,124,125,87,126,127,128,40,129,130,131,132,33,133,27],"design","frontend design","react components","create","code","examples","applications","landing","ui design","pages","aesthetics","web ui","generic",{"skillNo":135,"name":136,"nameZh":6,"description":137,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":77,"licenseFileUrl":138,"sourceHtmlUrl":139,"categories":140,"keywords":142,"publishedAt":46},"01M6HSA77QYVP7A72HREGTDN2C","algorithmic-art","Creating algorithmic art using p5.js with seeded randomness and interactive parameter exploration. Use this when users request creating art using code, generative art, algorithmic art, flow fields, or particle systems. Create original algorithmic art rather than copying existing artists' work to avoid copyright violations.","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fplugins\u002Falgorithmic-art\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fplugins\u002Falgorithmic-art\u002FSKILL.md",[141],{"categoryNo":113,"categoryName":114,"taxonomyType":20,"taxonomyTypeDesc":21},[143,144,145,124,125,146,147,148,149,40,150,151,37,152,33,153,28,154,155,156,27],"algorithmic art","Art","algorithmic","flow","js","work","interactive","generative art","exploration","users","Existing","generative","copyright","fields",{"skillNo":158,"name":159,"nameZh":6,"description":160,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":77,"licenseFileUrl":161,"sourceHtmlUrl":162,"categories":163,"keywords":165,"publishedAt":46},"01M6HS9HE0X390Y6RQKTY1YNGJ","theme-factory","Toolkit for styling artifacts with a theme. These artifacts can be slides, docs, reportings, HTML landing pages, etc. There are 10 pre-set themes with colors\u002Ffonts that you can apply to any artifact that has been creating, or can generate a new theme on-the-fly.","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Ftheme-factory\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Ftheme-factory\u002FSKILL.md",[164],{"categoryNo":113,"categoryName":114,"taxonomyType":20,"taxonomyTypeDesc":21},[166,167,168,169,170,171,87,172,173,174,128,175,176,130,177,33,178,179,180,30,181,182],"theme factory","factory","theme","docs","toolkit","generate","SET","fly","slides","styling","artifact","themes","you","fonts","can","apply","new",{"skillNo":184,"name":185,"nameZh":6,"description":186,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":187,"categories":188,"keywords":192,"publishedAt":46},"01M6HS74GASM6FJ77F83EY679E","browsing","Use when you need direct browser control - teaches Chrome DevTools Protocol for controlling existing browser sessions, multi-tab management, form automation, and content extraction via use_browser MCP tool","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Fsuperpowers-chrome\u002Fskills\u002Fbrowsing\u002FSKILL.md",[189],{"categoryNo":190,"categoryName":191,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000023","Skill与智能体开发",[185,193,194,195,196,197,198,199,200,201,202,40,203,204,205,206,102,178,30,153],"MCP","automation","chrome","sessions","protocol","browser","devtools","chrome devtools","tool","direct","browser control","control","via","form",{"skillNo":208,"name":209,"nameZh":6,"description":210,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":77,"licenseFileUrl":211,"sourceHtmlUrl":212,"categories":213,"keywords":220,"publishedAt":46},"01M6HS6BZT2CKDNQ6YF20DXN4T","slack-gif-creator","Knowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like \"make me a GIF of X doing Y for Slack.\"","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Fslack-gif-creator\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fwelflau\u002Fai-dev-system\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fmarketplace\u002Fcodebuddy-plugins-official\u002Fexternal_plugins\u002Fslack-gif-creator\u002FSKILL.md",[214,217],{"categoryNo":215,"categoryName":216,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000005","邮件与沟通",{"categoryNo":218,"categoryName":219,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000016","图像与多媒体",[221,222,223,224,225,226,227,228,229,40,230,152,231,232,233,30,28,27],"slack gif creator","slack","gif","creator","validation","Tools","constraints","Animation","knowledge","me","make","Utilities","concepts"]