[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M737W02ERMATZ15V2NFX469Z":3,"skill-same:01M737W02ERMATZ15V2NFX469Z":65},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":37,"commitSha":38,"skillDirectory":39,"forksCount":40,"repositoryHtmlUrl":41,"repositoryHomepageUrl":6,"ownerHtmlUrl":42,"ownerBlogUrl":43,"ownerDescription":6,"updatedAt":44,"install":45,"resources":58},"01M737W02ERMATZ15V2NFX469Z","correlate-observability-signals",null,"Unify metrics, logs, and traces for cohesive debugging. Implement exemplars for log-to-trace linking, build unified dashboards using RED\u002FUSE methods, and enable rapid root cause analysis across observability signals. Use when investigating complex incidents spanning multiple systems, reducing mean time to resolution, implementing distributed tracing, or moving from siloed tools to a unified observability platform.","roohe\u002Fagentic-super-skills","roohe","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F99054084?v=4","01M734WCDFP6RZVT5CNM00QR26","01M734WCE4BD0T4NZS5PCV9V5F",10,"MIT","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fcorrelate-observability-signals\u002FSKILL.md",[],[],"2026-11-11 15:11:14",false,"---\nname: correlate-observability-signals\nlocale: caveman-lite\nsource_locale: en\nsource_commit: 82c77053\ntranslator: \"Julius Brussee homage — caveman\"\ntranslation_date: \"2026-04-19\"\ndescription: >\n  Unify metrics, logs, and traces for cohesive debugging. Implement exemplars\n  for log-to-trace linking, build unified dashboards using RED\u002FUSE methods,\n  and enable rapid root cause analysis across observability signals. Use when\n  investigating complex incidents spanning multiple systems, reducing mean time\n  to resolution, implementing distributed tracing, or moving from siloed tools\n  to a unified observability platform.\nlicense: MIT\nallowed-tools: Read Write Edit Bash Grep Glob\nmetadata:\n  author: Philipp Thoss\n  version: \"1.0\"\n  domain: observability\n  complexity: advanced\n  language: multi\n  tags: correlation, exemplars, red-method, use-method, unified-observability\n---\n\n# Correlate Observability Signals\n\nConnect metrics, logs, and traces for unified debugging across the three pillars of observability.\n\n## When to Use\n\n- Investigating complex incidents that span multiple systems\n- Reducing MTTR (mean time to resolution)\n- Building unified observability dashboards\n- Implementing distributed tracing\n- Moving from siloed tools to unified observability\n\n## Inputs\n\n- **Required**: Prometheus (metrics)\n- **Required**: Log aggregation system (Loki, Elasticsearch, CloudWatch)\n- **Required**: Distributed tracing backend (Tempo, Jaeger, Zipkin)\n- **Optional**: Grafana for unified visualization\n- **Optional**: OpenTelemetry instrumentation\n\n## Procedure\n\n> See [Extended Examples](references\u002FEXAMPLES.md) for complete configuration files and templates.\n\n\n### Step 1: Implement Trace Context Propagation\n\nAdd trace IDs to all logs and metrics using OpenTelemetry:\n\n```go\n\u002F\u002F Go example: Propagate trace context to logs\npackage main\n\nimport (\n    \"context\"\n    \"log\"\n\n    \"go.opentelemetry.io\u002Fotel\"\n    \"go.opentelemetry.io\u002Fotel\u002Ftrace\"\n)\n\nfunc handleRequest(ctx context.Context, userID string) {\n    \u002F\u002F Extract trace context\n    span := trace.SpanFromContext(ctx)\n    traceID := span.SpanContext().TraceID().String()\n\n    \u002F\u002F Include trace ID in structured logs\n    log.Printf(\"trace_id=%s user_id=%s action=process_request\", traceID, userID)\n\n    \u002F\u002F Business logic here\n    processData(ctx, userID)\n}\n\nfunc processData(ctx context.Context, userID string) {\n    tracer := otel.Tracer(\"my-service\")\n    ctx, span := tracer.Start(ctx, \"processData\")\n    defer span.End()\n\n    traceID := span.SpanContext().TraceID().String()\n    log.Printf(\"trace_id=%s user_id=%s action=process_data\", traceID, userID)\n\n    \u002F\u002F More work\n}\n```\n\nPython example:\n\n```python\n# Python: Flask with OpenTelemetry\nfrom flask import Flask, request\nfrom opentelemetry import trace\nfrom opentelemetry.instrumentation.flask import FlaskInstrumentor\nimport logging\n\napp = Flask(__name__)\nFlaskInstrumentor().instrument_app(app)\n\nlogging.basicConfig(\n    format='%(asctime)s trace_id=%(otelTraceID)s span_id=%(otelSpanID)s %(message)s',\n    level=logging.INFO\n)\n\n@app.route('\u002Fapi\u002Fusers\u002F\u003Cuser_id>')\ndef get_user(user_id):\n    span = trace.get_current_span()\n    trace_id = format(span.get_span_context().trace_id, '032x')\n\n    logging.info(f\"Fetching user {user_id}\", extra={\n        'otelTraceID': trace_id,\n        'otelSpanID': format(span.get_span_context().span_id, '016x')\n    })\n\n    # Business logic\n    return {\"user_id\": user_id}\n```\n\n**Got:** All logs include `trace_id` field, enabling log-to-trace correlation.\n\n**If fail:** If trace IDs missing, check OpenTelemetry SDK initialization and context propagation.\n\n### Step 2: Configure Exemplars in Prometheus\n\nExemplars link metrics to traces:\n\n```yaml\n# prometheus.yml\nglobal:\n  scrape_interval: 15s\n  # Enable exemplar storage\n  exemplars:\n    max_exemplars: 100000  # Per TSDB block\n\nscrape_configs:\n  - job_name: 'api-service'\n    static_configs:\n      - targets: ['api-service:8080']\n    # Scrape exemplars\n    metric_relabel_configs:\n      - source_labels: [__name__]\n        regex: 'http_request_duration_seconds.*'\n        action: keep\n```\n\nInstrument application to emit exemplars:\n\n```go\n\u002F\u002F Go: Emit exemplars with Prometheus histogram\npackage main\n\nimport (\n    \"github.com\u002Fprometheus\u002Fclient_golang\u002Fprometheus\"\n    \"github.com\u002Fprometheus\u002Fclient_golang\u002Fprometheus\u002Fpromauto\"\n    \"go.opentelemetry.io\u002Fotel\u002Ftrace\"\n)\n\nvar httpDuration = promauto.NewHistogramVec(\n    prometheus.HistogramOpts{\n        Name:    \"http_request_duration_seconds\",\n        Help:    \"HTTP request duration\",\n        Buckets: prometheus.DefBuckets,\n    },\n    []string{\"method\", \"endpoint\", \"status\"},\n)\n\nfunc recordRequest(ctx context.Context, method, endpoint, status string, duration float64) {\n    \u002F\u002F Get trace ID for exemplar\n    span := trace.SpanFromContext(ctx)\n    traceID := span.SpanContext().TraceID().String()\n\n    \u002F\u002F Record metric with exemplar\n    observer := httpDuration.WithLabelValues(method, endpoint, status)\n    observer.(prometheus.ExemplarObserver).ObserveWithExemplar(\n        duration,\n        prometheus.Labels{\"trace_id\": traceID},\n    )\n}\n```\n\nQuery exemplars in Prometheus:\n\n```promql\n# Histogram with exemplars\nhistogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))\n```\n\nIn Grafana, exemplars appear as dots on histogram graphs that link to traces.\n\n**Got:** Grafana shows exemplars in metric graphs, clicking opens corresponding trace.\n\n**If fail:** Verify Prometheus version ≥2.26 (exemplar support), check Grafana data source config enables exemplars.\n\n### Step 3: Build Unified Dashboard with RED Method\n\nRED Method: Rate, Errors, Duration (for services)\n\n```json\n{\n  \"dashboard\": {\n    \"title\": \"API Service - RED Dashboard\",\n    \"panels\": [\n      {\n        \"title\": \"Request Rate (req\u002Fs)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"sum(rate(http_requests_total{job=\\\"api-service\\\"}[5m])) by (endpoint)\",\n            \"legendFormat\": \"{{ endpoint }}\"\n          }\n        ],\n        \"exemplars\": true\n      },\n      {\n        \"title\": \"Error Rate (%)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"sum(rate(http_requests_total{job=\\\"api-service\\\", status=~\\\"5..\\\"}[5m])) \u002F sum(rate(http_requests_total{job=\\\"api-service\\\"}[5m])) * 100\",\n            \"legendFormat\": \"Error %\"\n          }\n        ],\n        \"exemplars\": true\n      },\n      {\n        \"title\": \"Request Duration (p50, p95, p99)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"histogram_quantile(0.50, rate(http_request_duration_seconds_bucket{job=\\\"api-service\\\"}[5m]))\",\n            \"legendFormat\": \"p50\"\n          },\n          {\n            \"expr\": \"histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job=\\\"api-service\\\"}[5m]))\",\n            \"legendFormat\": \"p95\"\n          },\n          {\n            \"expr\": \"histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{job=\\\"api-service\\\"}[5m]))\",\n            \"legendFormat\": \"p99\"\n          }\n        ],\n        \"exemplars\": true\n      },\n      {\n        \"title\": \"Correlated Logs\",\n        \"type\": \"logs\",\n        \"datasource\": \"Loki\",\n        \"targets\": [\n          {\n            \"expr\": \"{job=\\\"api-service\\\"} |= \\\"error\\\"\"\n          }\n        ],\n        \"options\": {\n          \"showTime\": true,\n          \"enableLogDetails\": true\n        }\n      }\n    ]\n  }\n}\n```\n\n**Got:** Single dashboard showing rate, errors, duration + correlated logs.\n\n**If fail:** If panels show \"No Data\", verify metric names match your instrumentation.\n\n### Step 4: Implement USE Method for Resources\n\nUSE Method: Utilization, Saturation, Errors (for resources like CPU, memory, disk)\n\n```json\n{\n  \"dashboard\": {\n    \"title\": \"Node Resources - USE Dashboard\",\n    \"panels\": [\n      {\n        \"title\": \"CPU Utilization (%)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"100 - (avg(rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[5m])) * 100)\",\n            \"legendFormat\": \"CPU Usage %\"\n          }\n        ]\n      },\n      {\n        \"title\": \"CPU Saturation (Load Average)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"node_load1\",\n            \"legendFormat\": \"1min load\"\n          },\n          {\n            \"expr\": \"node_load5\",\n            \"legendFormat\": \"5min load\"\n          },\n          {\n            \"expr\": \"count(node_cpu_seconds_total{mode=\\\"idle\\\"})\",\n            \"legendFormat\": \"CPU cores (threshold)\"\n          }\n        ]\n      },\n      {\n        \"title\": \"Memory Utilization (%)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) \u002F node_memory_MemTotal_bytes * 100\",\n            \"legendFormat\": \"Memory Usage %\"\n          }\n        ]\n      },\n      {\n        \"title\": \"Memory Saturation (Page Faults)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"rate(node_vmstat_pgmajfault[5m])\",\n            \"legendFormat\": \"Major page faults\u002Fs\"\n          }\n        ]\n      },\n      {\n        \"title\": \"Disk Utilization (%)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"(node_filesystem_size_bytes - node_filesystem_free_bytes) \u002F node_filesystem_size_bytes * 100\",\n            \"legendFormat\": \"{{ device }}\"\n          }\n        ]\n      },\n      {\n        \"title\": \"Disk Saturation (IO Wait %)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"rate(node_cpu_seconds_total{mode=\\\"iowait\\\"}[5m]) * 100\",\n            \"legendFormat\": \"IO Wait %\"\n          }\n        ]\n      }\n    ]\n  }\n}\n```\n\n**Got:** Dashboard showing resource health across all USE dimensions.\n\n**If fail:** Ensure node_exporter is running and scraping system metrics.\n\n### Step 5: Link Logs to Traces in Loki\n\nConfigure Loki to extract trace IDs:\n\n```yaml\n# loki-config.yml\nschema_config:\n  configs:\n    - from: 2024-01-01\n      store: boltdb-shipper\n      object_store: s3\n      schema: v11\n      index:\n        prefix: index_\n        period: 24h\n\n# Derived fields for trace linking\nquery_config:\n  derived_fields:\n    - name: TraceID\n      source: trace_id\n      url: 'https:\u002F\u002Ftempo.company.com\u002Ftrace\u002F${__value.raw}'\n      urlDisplayLabel: 'View Trace'\n```\n\nIn Grafana, configure Loki data source:\n\n```json\n{\n  \"name\": \"Loki\",\n  \"type\": \"loki\",\n  \"url\": \"http:\u002F\u002Floki:3100\",\n  \"jsonData\": {\n    \"derivedFields\": [\n      {\n        \"datasourceUid\": \"tempo-uid\",\n        \"matcherRegex\": \"trace_id=(\\\\w+)\",\n        \"name\": \"TraceID\",\n        \"url\": \"$${__value.raw}\"\n      }\n    ]\n  }\n}\n```\n\n**Got:** Clicking trace ID in Loki logs opens corresponding trace in Tempo.\n\n**If fail:** Verify regex matches your log format, check Tempo data source UID.\n\n### Step 6: Create Unified Incident View\n\nBuild a dashboard that brings all signals together:\n\n```json\n{\n  \"dashboard\": {\n    \"title\": \"Incident Investigation\",\n    \"templating\": {\n      \"list\": [\n        {\n# ... (see EXAMPLES.md for complete configuration)\n```\n\nWorkflow during incident:\n\n1. Alert fires for high error rate\n2. On-call engineer opens Grafana dashboard\n3. Identifies spike in error rate at specific time\n4. Clicks exemplar dot on duration histogram → opens trace\n5. Trace shows slow database query\n6. Clicks \"View Logs\" on span → opens logs for that trace\n7. Logs reveal specific SQL query causing timeout\n8. Root cause identified in \u003C2 minutes\n\n**Got:** Single pane of glass for debugging, jumping between metrics\u002Flogs\u002Ftraces.\n\n**If fail:** If links don't work, check data source configurations and trace ID propagation.\n\n## Validation\n\n- [ ] Trace IDs present in all application logs\n- [ ] Prometheus scraping exemplars\n- [ ] Grafana dashboards show exemplar dots on histograms\n- [ ] Clicking exemplar opens corresponding trace in Tempo\u002FJaeger\n- [ ] Loki logs have \"View Trace\" links that work\n- [ ] RED dashboard created for key services\n- [ ] USE dashboard created for infrastructure\n- [ ] Unified incident dashboard tested during GameDay\n\n## Pitfalls\n\n- **Inconsistent trace ID format**: OpenTelemetry uses 32-char hex, Jaeger uses 16-char. Choose one.\n- **Missing context propagation**: If trace IDs don't flow across services, distributed tracing breaks. Use OpenTelemetry auto-instrumentation.\n- **Exemplar overload**: Too many exemplars (>100k) can slow Prometheus. Sample high-volume metrics.\n- **Clock skew**: Traces span multiple services. Ensure NTP is configured; clock drift causes trace ordering issues.\n- **Data retention mismatch**: If traces expire before metrics, correlation breaks. Align retention policies.\n\n## Related Skills\n\n- `setup-prometheus-monitoring` - metrics foundation for correlation\n- `configure-log-aggregation` - logs foundation for correlation\n- `instrument-distributed-tracing` - traces foundation for correlation\n- `build-grafana-dashboards` - unified visualization layer\n","# Correlate Observability Signals\n\nConnect metrics, logs, and traces for unified debugging across the three pillars of observability.\n\n## When to Use\n\n- Investigating complex incidents that span multiple systems\n- Reducing MTTR (mean time to resolution)\n- Building unified observability dashboards\n- Implementing distributed tracing\n- Moving from siloed tools to unified observability\n\n## Inputs\n\n- **Required**: Prometheus (metrics)\n- **Required**: Log aggregation system (Loki, Elasticsearch, CloudWatch)\n- **Required**: Distributed tracing backend (Tempo, Jaeger, Zipkin)\n- **Optional**: Grafana for unified visualization\n- **Optional**: OpenTelemetry instrumentation\n\n## Procedure\n\n> See [Extended Examples](references\u002FEXAMPLES.md) for complete configuration files and templates.\n\n\n### Step 1: Implement Trace Context Propagation\n\nAdd trace IDs to all logs and metrics using OpenTelemetry:\n\n```go\n\u002F\u002F Go example: Propagate trace context to logs\npackage main\n\nimport (\n    \"context\"\n    \"log\"\n\n    \"go.opentelemetry.io\u002Fotel\"\n    \"go.opentelemetry.io\u002Fotel\u002Ftrace\"\n)\n\nfunc handleRequest(ctx context.Context, userID string) {\n    \u002F\u002F Extract trace context\n    span := trace.SpanFromContext(ctx)\n    traceID := span.SpanContext().TraceID().String()\n\n    \u002F\u002F Include trace ID in structured logs\n    log.Printf(\"trace_id=%s user_id=%s action=process_request\", traceID, userID)\n\n    \u002F\u002F Business logic here\n    processData(ctx, userID)\n}\n\nfunc processData(ctx context.Context, userID string) {\n    tracer := otel.Tracer(\"my-service\")\n    ctx, span := tracer.Start(ctx, \"processData\")\n    defer span.End()\n\n    traceID := span.SpanContext().TraceID().String()\n    log.Printf(\"trace_id=%s user_id=%s action=process_data\", traceID, userID)\n\n    \u002F\u002F More work\n}\n```\n\nPython example:\n\n```python\n# Python: Flask with OpenTelemetry\nfrom flask import Flask, request\nfrom opentelemetry import trace\nfrom opentelemetry.instrumentation.flask import FlaskInstrumentor\nimport logging\n\napp = Flask(__name__)\nFlaskInstrumentor().instrument_app(app)\n\nlogging.basicConfig(\n    format='%(asctime)s trace_id=%(otelTraceID)s span_id=%(otelSpanID)s %(message)s',\n    level=logging.INFO\n)\n\n@app.route('\u002Fapi\u002Fusers\u002F\u003Cuser_id>')\ndef get_user(user_id):\n    span = trace.get_current_span()\n    trace_id = format(span.get_span_context().trace_id, '032x')\n\n    logging.info(f\"Fetching user {user_id}\", extra={\n        'otelTraceID': trace_id,\n        'otelSpanID': format(span.get_span_context().span_id, '016x')\n    })\n\n    # Business logic\n    return {\"user_id\": user_id}\n```\n\n**Got:** All logs include `trace_id` field, enabling log-to-trace correlation.\n\n**If fail:** If trace IDs missing, check OpenTelemetry SDK initialization and context propagation.\n\n### Step 2: Configure Exemplars in Prometheus\n\nExemplars link metrics to traces:\n\n```yaml\n# prometheus.yml\nglobal:\n  scrape_interval: 15s\n  # Enable exemplar storage\n  exemplars:\n    max_exemplars: 100000  # Per TSDB block\n\nscrape_configs:\n  - job_name: 'api-service'\n    static_configs:\n      - targets: ['api-service:8080']\n    # Scrape exemplars\n    metric_relabel_configs:\n      - source_labels: [__name__]\n        regex: 'http_request_duration_seconds.*'\n        action: keep\n```\n\nInstrument application to emit exemplars:\n\n```go\n\u002F\u002F Go: Emit exemplars with Prometheus histogram\npackage main\n\nimport (\n    \"github.com\u002Fprometheus\u002Fclient_golang\u002Fprometheus\"\n    \"github.com\u002Fprometheus\u002Fclient_golang\u002Fprometheus\u002Fpromauto\"\n    \"go.opentelemetry.io\u002Fotel\u002Ftrace\"\n)\n\nvar httpDuration = promauto.NewHistogramVec(\n    prometheus.HistogramOpts{\n        Name:    \"http_request_duration_seconds\",\n        Help:    \"HTTP request duration\",\n        Buckets: prometheus.DefBuckets,\n    },\n    []string{\"method\", \"endpoint\", \"status\"},\n)\n\nfunc recordRequest(ctx context.Context, method, endpoint, status string, duration float64) {\n    \u002F\u002F Get trace ID for exemplar\n    span := trace.SpanFromContext(ctx)\n    traceID := span.SpanContext().TraceID().String()\n\n    \u002F\u002F Record metric with exemplar\n    observer := httpDuration.WithLabelValues(method, endpoint, status)\n    observer.(prometheus.ExemplarObserver).ObserveWithExemplar(\n        duration,\n        prometheus.Labels{\"trace_id\": traceID},\n    )\n}\n```\n\nQuery exemplars in Prometheus:\n\n```promql\n# Histogram with exemplars\nhistogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))\n```\n\nIn Grafana, exemplars appear as dots on histogram graphs that link to traces.\n\n**Got:** Grafana shows exemplars in metric graphs, clicking opens corresponding trace.\n\n**If fail:** Verify Prometheus version ≥2.26 (exemplar support), check Grafana data source config enables exemplars.\n\n### Step 3: Build Unified Dashboard with RED Method\n\nRED Method: Rate, Errors, Duration (for services)\n\n```json\n{\n  \"dashboard\": {\n    \"title\": \"API Service - RED Dashboard\",\n    \"panels\": [\n      {\n        \"title\": \"Request Rate (req\u002Fs)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"sum(rate(http_requests_total{job=\\\"api-service\\\"}[5m])) by (endpoint)\",\n            \"legendFormat\": \"{{ endpoint }}\"\n          }\n        ],\n        \"exemplars\": true\n      },\n      {\n        \"title\": \"Error Rate (%)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"sum(rate(http_requests_total{job=\\\"api-service\\\", status=~\\\"5..\\\"}[5m])) \u002F sum(rate(http_requests_total{job=\\\"api-service\\\"}[5m])) * 100\",\n            \"legendFormat\": \"Error %\"\n          }\n        ],\n        \"exemplars\": true\n      },\n      {\n        \"title\": \"Request Duration (p50, p95, p99)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"histogram_quantile(0.50, rate(http_request_duration_seconds_bucket{job=\\\"api-service\\\"}[5m]))\",\n            \"legendFormat\": \"p50\"\n          },\n          {\n            \"expr\": \"histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job=\\\"api-service\\\"}[5m]))\",\n            \"legendFormat\": \"p95\"\n          },\n          {\n            \"expr\": \"histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{job=\\\"api-service\\\"}[5m]))\",\n            \"legendFormat\": \"p99\"\n          }\n        ],\n        \"exemplars\": true\n      },\n      {\n        \"title\": \"Correlated Logs\",\n        \"type\": \"logs\",\n        \"datasource\": \"Loki\",\n        \"targets\": [\n          {\n            \"expr\": \"{job=\\\"api-service\\\"} |= \\\"error\\\"\"\n          }\n        ],\n        \"options\": {\n          \"showTime\": true,\n          \"enableLogDetails\": true\n        }\n      }\n    ]\n  }\n}\n```\n\n**Got:** Single dashboard showing rate, errors, duration + correlated logs.\n\n**If fail:** If panels show \"No Data\", verify metric names match your instrumentation.\n\n### Step 4: Implement USE Method for Resources\n\nUSE Method: Utilization, Saturation, Errors (for resources like CPU, memory, disk)\n\n```json\n{\n  \"dashboard\": {\n    \"title\": \"Node Resources - USE Dashboard\",\n    \"panels\": [\n      {\n        \"title\": \"CPU Utilization (%)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"100 - (avg(rate(node_cpu_seconds_total{mode=\\\"idle\\\"}[5m])) * 100)\",\n            \"legendFormat\": \"CPU Usage %\"\n          }\n        ]\n      },\n      {\n        \"title\": \"CPU Saturation (Load Average)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"node_load1\",\n            \"legendFormat\": \"1min load\"\n          },\n          {\n            \"expr\": \"node_load5\",\n            \"legendFormat\": \"5min load\"\n          },\n          {\n            \"expr\": \"count(node_cpu_seconds_total{mode=\\\"idle\\\"})\",\n            \"legendFormat\": \"CPU cores (threshold)\"\n          }\n        ]\n      },\n      {\n        \"title\": \"Memory Utilization (%)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) \u002F node_memory_MemTotal_bytes * 100\",\n            \"legendFormat\": \"Memory Usage %\"\n          }\n        ]\n      },\n      {\n        \"title\": \"Memory Saturation (Page Faults)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"rate(node_vmstat_pgmajfault[5m])\",\n            \"legendFormat\": \"Major page faults\u002Fs\"\n          }\n        ]\n      },\n      {\n        \"title\": \"Disk Utilization (%)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"(node_filesystem_size_bytes - node_filesystem_free_bytes) \u002F node_filesystem_size_bytes * 100\",\n            \"legendFormat\": \"{{ device }}\"\n          }\n        ]\n      },\n      {\n        \"title\": \"Disk Saturation (IO Wait %)\",\n        \"type\": \"graph\",\n        \"targets\": [\n          {\n            \"expr\": \"rate(node_cpu_seconds_total{mode=\\\"iowait\\\"}[5m]) * 100\",\n            \"legendFormat\": \"IO Wait %\"\n          }\n        ]\n      }\n    ]\n  }\n}\n```\n\n**Got:** Dashboard showing resource health across all USE dimensions.\n\n**If fail:** Ensure node_exporter is running and scraping system metrics.\n\n### Step 5: Link Logs to Traces in Loki\n\nConfigure Loki to extract trace IDs:\n\n```yaml\n# loki-config.yml\nschema_config:\n  configs:\n    - from: 2024-01-01\n      store: boltdb-shipper\n      object_store: s3\n      schema: v11\n      index:\n        prefix: index_\n        period: 24h\n\n# Derived fields for trace linking\nquery_config:\n  derived_fields:\n    - name: TraceID\n      source: trace_id\n      url: 'https:\u002F\u002Ftempo.company.com\u002Ftrace\u002F${__value.raw}'\n      urlDisplayLabel: 'View Trace'\n```\n\nIn Grafana, configure Loki data source:\n\n```json\n{\n  \"name\": \"Loki\",\n  \"type\": \"loki\",\n  \"url\": \"http:\u002F\u002Floki:3100\",\n  \"jsonData\": {\n    \"derivedFields\": [\n      {\n        \"datasourceUid\": \"tempo-uid\",\n        \"matcherRegex\": \"trace_id=(\\\\w+)\",\n        \"name\": \"TraceID\",\n        \"url\": \"$${__value.raw}\"\n      }\n    ]\n  }\n}\n```\n\n**Got:** Clicking trace ID in Loki logs opens corresponding trace in Tempo.\n\n**If fail:** Verify regex matches your log format, check Tempo data source UID.\n\n### Step 6: Create Unified Incident View\n\nBuild a dashboard that brings all signals together:\n\n```json\n{\n  \"dashboard\": {\n    \"title\": \"Incident Investigation\",\n    \"templating\": {\n      \"list\": [\n        {\n# ... (see EXAMPLES.md for complete configuration)\n```\n\nWorkflow during incident:\n\n1. Alert fires for high error rate\n2. On-call engineer opens Grafana dashboard\n3. Identifies spike in error rate at specific time\n4. Clicks exemplar dot on duration histogram → opens trace\n5. Trace shows slow database query\n6. Clicks \"View Logs\" on span → opens logs for that trace\n7. Logs reveal specific SQL query causing timeout\n8. Root cause identified in \u003C2 minutes\n\n**Got:** Single pane of glass for debugging, jumping between metrics\u002Flogs\u002Ftraces.\n\n**If fail:** If links don't work, check data source configurations and trace ID propagation.\n\n## Validation\n\n- [ ] Trace IDs present in all application logs\n- [ ] Prometheus scraping exemplars\n- [ ] Grafana dashboards show exemplar dots on histograms\n- [ ] Clicking exemplar opens corresponding trace in Tempo\u002FJaeger\n- [ ] Loki logs have \"View Trace\" links that work\n- [ ] RED dashboard created for key services\n- [ ] USE dashboard created for infrastructure\n- [ ] Unified incident dashboard tested during GameDay\n\n## Pitfalls\n\n- **Inconsistent trace ID format**: OpenTelemetry uses 32-char hex, Jaeger uses 16-char. Choose one.\n- **Missing context propagation**: If trace IDs don't flow across services, distributed tracing breaks. Use OpenTelemetry auto-instrumentation.\n- **Exemplar overload**: Too many exemplars (>100k) can slow Prometheus. Sample high-volume metrics.\n- **Clock skew**: Traces span multiple services. Ensure NTP is configured; clock drift causes trace ordering issues.\n- **Data retention mismatch**: If traces expire before metrics, correlation breaks. Align retention policies.\n\n## Related Skills\n\n- `setup-prometheus-monitoring` - metrics foundation for correlation\n- `configure-log-aggregation` - logs foundation for correlation\n- `instrument-distributed-tracing` - traces foundation for correlation\n- `build-grafana-dashboards` - unified visualization layer\n",{"name":5,"locale":23,"license":14,"metadata":24,"translator":31,"description":32,"allowed-tools":33,"source_commit":34,"source_locale":35,"translation_date":36},"caveman-lite",{"tags":25,"author":26,"domain":27,"version":28,"language":29,"complexity":30},"correlation, exemplars, red-method, use-method, unified-observability","Philipp Thoss","observability","1.0","multi","advanced","Julius Brussee homage — caveman","Unify metrics, logs, and traces for cohesive debugging. Implement exemplars for log-to-trace linking, build unified dashboards using RED\u002FUSE methods, and enable rapid root cause analysis across observability signals. Use when investigating complex incidents spanning multiple systems, reducing mean time to resolution, implementing distributed tracing, or moving from siloed tools to a unified observability platform.\n","Read Write Edit Bash Grep Glob","82c77053","en","2026-04-19","master","f483e54f3ad5f202a77d45f2e57f8d970b71195f","skills_library\u002Fcorrelate-observability-signals",2,"https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills","https:\u002F\u002Fgithub.com\u002Froohe","yoou.link","2026-05-19 15:56:07",{"source":8,"skillName":5,"commands":46,"prompt":56,"zipAvailable":57},[47,50,53],{"tool":48,"command":49},"npx","npx skills add roohe\u002Fagentic-super-skills --skill correlate-observability-signals",{"tool":51,"command":52},"bunx","bunx skills add roohe\u002Fagentic-super-skills --skill correlate-observability-signals",{"tool":54,"command":55},"pnpm","pnpm dlx skills add roohe\u002Fagentic-super-skills --skill correlate-observability-signals","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fcorrelate-observability-signals\u002FSKILL.md",true,[59],{"path":60,"resourceType":61,"resourceTypeDesc":62,"sizeBytes":63,"sourceHtmlUrl":64},"references\u002FEXAMPLES.md","REFERENCE","参考文档",8827,"https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fcorrelate-observability-signals\u002Freferences\u002FEXAMPLES.md",[66,73,80,87,94,101],{"skillNo":67,"name":68,"nameZh":6,"description":69,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":70,"categories":71,"keywords":72,"publishedAt":18},"01M73D865CZME2FCMKA4PQM348","zarr-python","Chunked N-D arrays for cloud storage. Compressed arrays, parallel I\u002FO, S3\u002FGCS integration, NumPy\u002FDask\u002FXarray compatible, for large-scale scientific computing pipelines.","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fzarr-python\u002FSKILL.md",[],[],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":77,"categories":78,"keywords":79,"publishedAt":18},"01M73D7Z64J4ABWZJD7P44XBBF","youtube-downloader","Download and process YouTube content for research. Use when: downloading competitor videos for analysis; extracting audio for podcasts; getting transcripts for content repurposing; archiving webinars; research content curation","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fyoutube-downloader\u002FSKILL.md",[],[],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":18},"01M73D7K8778TGZV3GJFR3703D","yc-pitch-deck","Create a compelling startup pitch deck that follows the structure proven to raise billions from top investors. Master the YC and Sequoia formats that get founders funded. Use when: **Fundraising** to create investor pitch decks; **YC application** to structure your narrative; **Demo Day prep** to craft your 2-minute pitch; **Angel investors** to communicate your opportunity; **Partnership pitches** to structure compelling asks","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fyc-pitch-deck\u002FSKILL.md",[],[],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":18},"01M73D70KJSE0E8Z28YMNNRT04","x-twitter-scraper","Use when the user needs X (Twitter) data or confirmation-gated X actions through Xquik: tweet search, user lookup, follower extraction, media download, monitoring, webhooks, MCP, SDKs, posting, likes, DMs, and profile updates. Requires a Xquik API key. Never ask for X login material.","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fx-twitter-scraper\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":18},"01M73D6ED01GV077R7K4NZNFZJ","write-vignette","Create R package vignettes using R Markdown or Quarto. Covers vignette setup, YAML configuration, code chunk options, building and testing, and CRAN requirements for vignettes. Use when adding a Getting Started tutorial, documenting complex workflows spanning multiple functions, creating domain-specific guides, or when CRAN submission requires user-facing documentation beyond function help pages.","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fwrite-vignette\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":18},"01M73D69YJR0BK3NFHE0CYG3EF","write-validation-documentation","Write IQ\u002FOQ\u002FPQ validation documentation for computerized systems in regulated environments. Covers protocols, reports, test scripts, deviation handling, and approval workflows. Use when validating R or other software for regulated use, preparing for a regulatory audit, documenting qualification of computing environments, or creating and updating validation protocols and reports for new or re-qualified systems.","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fwrite-validation-documentation\u002FSKILL.md",[],[]]