[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M737Q4JSPM0NF2WMKZQJPW2P":3,"skill-same:01M737Q4JSPM0NF2WMKZQJPW2P":65},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":37,"commitSha":38,"skillDirectory":39,"forksCount":40,"repositoryHtmlUrl":41,"repositoryHomepageUrl":6,"ownerHtmlUrl":42,"ownerBlogUrl":43,"ownerDescription":6,"updatedAt":44,"install":45,"resources":58},"01M737Q4JSPM0NF2WMKZQJPW2P","configure-log-aggregation",null,"Set up centralized log aggregation with Loki and Promtail (or ELK stack), including log parsing, label extraction, retention policies, and integration with metrics for correlation. Use when consolidating logs from multiple services into a searchable system, replacing local log files with centralized queryable storage, correlating logs with metrics and traces, implementing structured logging with label extraction, or troubleshooting production incidents requiring cross-service log analysis.","roohe\u002Fagentic-super-skills","roohe","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F99054084?v=4","01M734WCDFP6RZVT5CNM00QR26","01M734WCE4BD0T4NZS5PCV9V5F",10,"MIT","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fconfigure-log-aggregation\u002FSKILL.md",[],[],"2026-11-11 15:11:14",false,"---\nname: configure-log-aggregation\nlocale: caveman-lite\nsource_locale: en\nsource_commit: 82c77053\ntranslator: \"Julius Brussee homage — caveman\"\ntranslation_date: \"2026-04-19\"\ndescription: >\n  Set up centralized log aggregation with Loki and Promtail (or ELK stack), including\n  log parsing, label extraction, retention policies, and integration with metrics for\n  correlation. Use when consolidating logs from multiple services into a searchable\n  system, replacing local log files with centralized queryable storage, correlating logs\n  with metrics and traces, implementing structured logging with label extraction, or\n  troubleshooting production incidents requiring cross-service log analysis.\nlicense: MIT\nallowed-tools: Read Write Edit Bash Grep Glob\nmetadata:\n  author: Philipp Thoss\n  version: \"1.0\"\n  domain: observability\n  complexity: intermediate\n  language: multi\n  tags: loki, promtail, logging, elk, log-aggregation\n---\n\n# Configure Log Aggregation\n\nImplement centralized log collection, parsing, and querying with Loki\u002FPromtail or ELK stack for operational visibility.\n\n## When to Use\n\n- Consolidating logs from multiple services or hosts into a searchable system\n- Replacing local log files with centralized, queryable log storage\n- Correlating logs with metrics and traces for full observability\n- Implementing structured logging with label extraction from unstructured logs\n- Setting retention policies for log data based on storage and compliance needs\n- Troubleshooting production incidents requiring log analysis across services\n\n## Inputs\n\n- **Required**: Log sources (application logs, system logs, container logs)\n- **Required**: Log format patterns (JSON, plaintext, syslog, etc.)\n- **Optional**: Label extraction rules for structured querying\n- **Optional**: Retention and compression policies\n- **Optional**: Existing log shipper configuration (Fluentd, Filebeat, Promtail)\n\n## Procedure\n\n> See [Extended Examples](references\u002FEXAMPLES.md) for complete configuration files and templates.\n\n\n### Step 1: Choose Log Aggregation Stack\n\nSelect between Loki (Prometheus-style) or ELK (Elasticsearch-based) based on requirements.\n\n**Loki advantages**:\n- Lightweight, designed for Kubernetes and cloud-native environments\n- Label-based indexing (like Prometheus) for low storage overhead\n- Native integration with Grafana for unified dashboards\n- Horizontal scalability with object storage (S3, GCS)\n- Lower resource consumption compared to Elasticsearch\n\n**ELK advantages**:\n- Full-text search across all log content (not only labels)\n- Rich query DSL and aggregations\n- Mature ecosystem with beats, logstash plugins\n- Better for compliance\u002Faudit logs requiring deep historical search\n\nFor this guide, we'll focus on **Loki + Promtail** (recommended for most modern setups).\n\nDecision criteria:\n```markdown\nUse Loki if:\n- You want label-based queries similar to Prometheus\n- Storage costs are a concern (Loki indexes only labels)\n- You already use Grafana for metrics\n- Kubernetes\u002Fcontainer-native deployment\n\nUse ELK if:\n- You need full-text search across all log content\n- You have complex log parsing and enrichment requirements\n- You require advanced analytics and aggregations\n- Legacy systems with existing Logstash pipelines\n```\n\n**Got:** Clear choice made based on requirements, team downloads appropriate installation artifacts.\n\n**If fail:**\n- Benchmark storage requirements: Loki ~10x less than Elasticsearch for same logs\n- Evaluate query patterns: full-text search needs vs label filtering\n- Consider operational overhead: ELK requires more tuning and resources\n\n### Step 2: Deploy Loki\n\nInstall and configure Loki with appropriate storage backend.\n\n**Docker Compose deployment** (`docker-compose.yml`):\n\n```yaml\nversion: '3.8'\n\nservices:\n  loki:\n    image: grafana\u002Floki:2.9.0\n    ports:\n      - \"3100:3100\"\n    volumes:\n      - .\u002Floki-config.yml:\u002Fetc\u002Floki\u002Flocal-config.yaml\n      - loki-data:\u002Floki\n    command: -config.file=\u002Fetc\u002Floki\u002Flocal-config.yaml\n    restart: unless-stopped\n\n  promtail:\n    image: grafana\u002Fpromtail:2.9.0\n    volumes:\n      - .\u002Fpromtail-config.yml:\u002Fetc\u002Fpromtail\u002Fconfig.yml\n      - \u002Fvar\u002Flog:\u002Fvar\u002Flog:ro\n      - \u002Fvar\u002Flib\u002Fdocker\u002Fcontainers:\u002Fvar\u002Flib\u002Fdocker\u002Fcontainers:ro\n    command: -config.file=\u002Fetc\u002Fpromtail\u002Fconfig.yml\n    restart: unless-stopped\n    depends_on:\n      - loki\n\nvolumes:\n  loki-data:\n```\n\n**Loki configuration** (`loki-config.yml`):\n\n```yaml\nauth_enabled: false\n\nserver:\n  http_listen_port: 3100\n  grpc_listen_port: 9096\n\n# ... (see EXAMPLES.md for complete configuration)\n```\n\nFor **production** with S3 storage:\n\n```yaml\nstorage_config:\n  aws:\n    s3: s3:\u002F\u002Fus-east-1\u002Fmy-loki-bucket\n    s3forcepathstyle: true\n  boltdb_shipper:\n    active_index_directory: \u002Floki\u002Findex\n    cache_location: \u002Floki\u002Fcache\n    shared_store: s3\n```\n\n**Got:** Loki starts successfully, health check passes at `http:\u002F\u002Flocalhost:3100\u002Fready`, logs stored according to retention policy.\n\n**If fail:**\n- Check Loki logs: `docker logs loki`\n- Verify storage directories exist and are writable\n- Test config syntax: `docker run grafana\u002Floki:2.9.0 -config.file=\u002Fetc\u002Floki\u002Flocal-config.yaml -verify-config`\n- Ensure retention settings don't exceed disk capacity\n- For S3: verify IAM permissions and bucket access\n\n### Step 3: Configure Promtail for Log Shipping\n\nSet up Promtail to scrape logs and forward to Loki with label extraction.\n\n**Promtail configuration** (`promtail-config.yml`):\n\n```yaml\nserver:\n  http_listen_port: 9080\n  grpc_listen_port: 0\n\npositions:\n  filename: \u002Ftmp\u002Fpositions.yaml\n# ... (see EXAMPLES.md for complete configuration)\n```\n\nKey Promtail concepts:\n- **Scrape configs**: Define log sources and how to discover them\n- **Pipeline stages**: Transform and label logs before sending to Loki\n- **Relabel configs**: Dynamic labeling based on metadata\n- **Positions file**: Tracks read offsets to avoid re-processing logs\n\n**Got:** Promtail scrapes configured log files, labels applied correctly, logs visible in Loki via LogQL queries.\n\n**If fail:**\n- Check Promtail logs: `docker logs promtail`\n- Verify file paths are accessible: `docker exec promtail ls \u002Fvar\u002Flog`\n- Test regex patterns independently with sample log lines\n- Monitor Promtail metrics: `curl http:\u002F\u002Flocalhost:9080\u002Fmetrics | grep promtail`\n- Check positions file for progress: `cat \u002Ftmp\u002Fpositions.yaml`\n\n### Step 4: Query Logs with LogQL\n\nLearn LogQL syntax for filtering and aggregating logs.\n\n**Basic queries**:\n\n```logql\n# All logs from a job\n{job=\"app\"}\n\n# Logs with specific label values\n{job=\"app\", level=\"error\"}\n\n# Regex filter on log line content\n{job=\"app\"} |~ \"authentication failed\"\n\n# Case-insensitive regex\n{job=\"app\"} |~ \"(?i)error\"\n\n# Line filter (doesn't parse, just includes\u002Fexcludes)\n{job=\"app\"} |= \"user\"  # Contains \"user\"\n{job=\"app\"} != \"debug\" # Doesn't contain \"debug\"\n```\n\n**Parsing and filtering**:\n\n```logql\n# JSON parsing\n{job=\"app\"} | json | level=\"error\"\n\n# Regex parsing with named groups\n{job=\"app\"} | regexp \"user_id=(?P\u003Cuser_id>\\\\d+)\" | user_id=\"12345\"\n\n# Logfmt parsing (key=value format)\n{job=\"app\"} | logfmt | level=\"error\", service=\"auth\"\n\n# Pattern parsing\n{job=\"nginx\"} | pattern `\u003Cip> - \u003Cuser> [\u003Ctimestamp>] \"\u003Cmethod> \u003Cpath> \u003Cprotocol>\" \u003Cstatus> \u003Csize>` | status >= 500\n```\n\n**Aggregations** (metrics from logs):\n\n```logql\n# Count log lines per level\nsum by (level) (count_over_time({job=\"app\"}[5m]))\n\n# Rate of error logs\nrate({job=\"app\", level=\"error\"}[5m])\n\n# Bytes processed per service\nsum by (service) (bytes_over_time({job=\"app\"}[1h]))\n\n# Average request duration from logs\navg_over_time({job=\"app\"} | json | unwrap duration [5m])\n\n# Top 10 error messages\ntopk(10, sum by (message) (count_over_time({level=\"error\"} [1h])))\n```\n\n**Filtering by extracted fields**:\n\n```logql\n# Find specific trace in logs\n{job=\"app\"} | json | trace_id=\"abc123def456\"\n\n# HTTP 5xx errors from nginx\n{job=\"nginx\"} | pattern `\u003C_> \"\u003C_> \u003C_> \u003C_>\" \u003Cstatus> \u003C_>` | status >= 500\n\n# Failed authentication attempts\n{job=\"app\"} | json | message=~\"authentication failed\" | user_id != \"\"\n```\n\nCreate Grafana explore queries or dashboard panels using these patterns.\n\n**Got:** Queries return expected log lines, filtering works correctly, aggregations produce metrics from logs.\n\n**If fail:**\n- Use Grafana Explore to debug queries interactively\n- Check label names: `curl http:\u002F\u002Flocalhost:3100\u002Floki\u002Fapi\u002Fv1\u002Flabels`\n- Verify label values: `curl http:\u002F\u002Flocalhost:3100\u002Floki\u002Fapi\u002Fv1\u002Flabel\u002F{label_name}\u002Fvalues`\n- Simplify query: start with basic label selector, add filters incrementally\n- Check time range: logs might not exist in selected window\n\n### Step 5: Integrate Logs with Metrics and Traces\n\nCorrelate logs with Prometheus metrics and distributed traces for unified observability.\n\n**Add trace IDs to logs** (application instrumentation):\n\n```python\n# Python with OpenTelemetry\nimport logging\nfrom opentelemetry import trace\n\nlogger = logging.getLogger(__name__)\n\ndef handle_request():\n    span = trace.get_current_span()\n    trace_id = span.get_span_context().trace_id\n\n    logger.info(\n        \"Processing request\",\n        extra={\"trace_id\": format(trace_id, \"032x\")}\n    )\n```\n\n```go\n\u002F\u002F Go with OpenTelemetry\nimport (\n    \"go.opentelemetry.io\u002Fotel\u002Ftrace\"\n    \"go.uber.org\u002Fzap\"\n)\n\nfunc handleRequest(ctx context.Context) {\n    span := trace.SpanFromContext(ctx)\n    traceID := span.SpanContext().TraceID().String()\n\n    logger.Info(\"Processing request\",\n        zap.String(\"trace_id\", traceID),\n    )\n}\n```\n\n**Configure Grafana data links** from metrics to logs:\n\nIn Prometheus panel field config:\n\n```json\n{\n  \"fieldConfig\": {\n    \"defaults\": {\n      \"links\": [\n        {\n          \"title\": \"View Logs\",\n          \"url\": \"\u002Fexplore?left={\\\"datasource\\\":\\\"Loki\\\",\\\"queries\\\":[{\\\"refId\\\":\\\"A\\\",\\\"expr\\\":\\\"{job=\\\\\\\"app\\\\\\\",instance=\\\\\\\"${__field.labels.instance}\\\\\\\"} |= `${__field.labels.trace_id}`\\\"}],\\\"range\\\":{\\\"from\\\":\\\"${__from}\\\",\\\"to\\\":\\\"${__to}\\\"}}\",\n          \"targetBlank\": false\n        }\n      ]\n    }\n  }\n}\n```\n\n**Configure Grafana data links** from logs to traces:\n\nIn Loki datasource config:\n\n```yaml\ndatasources:\n  - name: Loki\n    type: loki\n    url: http:\u002F\u002Floki:3100\n    jsonData:\n      derivedFields:\n        - datasourceName: Tempo\n          matcherRegex: \"trace_id=(\\\\w+)\"\n          name: TraceID\n          url: \"$${__value.raw}\"\n```\n\n**Correlate logs in Grafana Explore**:\n1. Query metrics in Prometheus\n2. Click on data point\n3. Select \"View Logs\" from context menu\n4. Loki query auto-populated with relevant labels and time range\n5. Click trace ID in logs\n6. Tempo trace view opens with full distributed trace\n\n**Got:** Clicking metrics opens related logs, trace IDs in logs link to trace viewer, single pane for metrics\u002Flogs\u002Ftraces navigation.\n\n**If fail:**\n- Verify trace ID format matches regex in derived fields\n- Check that trace_id label extracted by Promtail pipeline\n- Ensure Tempo datasource configured in Grafana\n- Test URL encoding for complex filter expressions\n- Validate data link URLs in incognito\u002Fprivate browser window\n\n### Step 6: Set Up Log Retention and Compaction\n\nConfigure retention policies and compaction to manage storage costs.\n\n**Retention by stream** (in Loki config):\n\n```yaml\nlimits_config:\n  retention_period: 720h  # Global default: 30 days\n\n  # Per-tenant retention (requires multi-tenancy enabled)\n  per_tenant_override_config: \u002Fetc\u002Floki\u002Foverrides.yaml\n\n# overrides.yaml\noverrides:\n  production:\n    retention_period: 2160h  # 90 days for production\n  staging:\n    retention_period: 360h   # 15 days for staging\n  development:\n    retention_period: 168h   # 7 days for dev\n```\n\n**Retention by stream labels** (requires compactor):\n\n```yaml\ncompactor:\n  working_directory: \u002Floki\u002Fcompactor\n  shared_store: filesystem\n  compaction_interval: 10m\n  retention_enabled: true\n  retention_delete_delay: 2h\n# ... (see EXAMPLES.md for complete configuration)\n```\n\nPriority determines which rule applies when multiple match (lower number = higher priority).\n\n**Compression settings**:\n\n```yaml\nchunk_store_config:\n  chunk_cache_config:\n    enable_fifocache: true\n    fifocache:\n      max_size_bytes: 1GB\n      ttl: 24h\n# ... (see EXAMPLES.md for complete configuration)\n```\n\n**Monitor retention**:\n\n```bash\n# Check chunk stats\ncurl http:\u002F\u002Flocalhost:3100\u002Floki\u002Fapi\u002Fv1\u002Fstatus\u002Fchunks | jq\n\n# Check compactor metrics\ncurl http:\u002F\u002Flocalhost:3100\u002Fmetrics | grep loki_compactor\n\n# Verify deleted chunks\ncurl http:\u002F\u002Flocalhost:3100\u002Fmetrics | grep loki_boltdb_shipper_retention_deleted\n```\n\n**Got:** Old logs automatically deleted per retention policy, storage usage stabilizes, compaction reduces index size.\n\n**If fail:**\n- Enable compactor in Loki config if retention not working\n- Check compactor logs: `docker logs loki | grep compactor`\n- Verify retention_enabled: true and retention_deletes_enabled: true\n- Monitor disk usage: `du -sh \u002Floki\u002F`\n- For S3: check bucket lifecycle policies don't conflict with Loki retention\n\n## Validation\n\n- [ ] Loki API health check returns 200: `curl http:\u002F\u002Flocalhost:3100\u002Fready`\n- [ ] Promtail successfully scraping logs from all configured sources\n- [ ] Labels extracted correctly from log lines (visible in Grafana Explore)\n- [ ] LogQL queries return expected results with proper filtering\n- [ ] Log retention policy enforced (old logs deleted after retention period)\n- [ ] Logs accessible from Grafana dashboards and Explore view\n- [ ] Trace IDs from logs link to Tempo trace viewer\n- [ ] Metrics panels have data links to relevant logs\n- [ ] Compaction running and reducing storage overhead\n- [ ] Storage usage within allocated disk\u002FS3 budget\n\n## Pitfalls\n\n- **High cardinality labels**: Using unbounded label values (user IDs, request IDs) causes index explosion. Use fixed labels (level, service, env) and put variables in log lines.\n- **Missing log parsing**: Sending raw logs without label extraction limits query capabilities. Always parse structured logs (JSON, logfmt) or use regex for unstructured.\n- **Incorrect time parsing**: Mismatched timestamp formats cause logs to be out of order or rejected. Test timestamp parsing with sample logs.\n- **Retention not working**: Compactor must be enabled for retention to delete old data. Check `retention_enabled: true` and `retention_deletes_enabled: true`.\n- **Ingestion rate limits**: Default limits (10MB\u002Fs) may be too low for high-volume systems. Adjust `ingestion_rate_mb` and `ingestion_burst_size_mb`.\n- **Query timeouts**: Broad queries over long time ranges can timeout. Use more specific label selectors and shorter time windows.\n- **Log duplication**: Multiple Promtail instances scraping same logs create duplicates. Use unique labels or positions file coordination.\n\n## Related Skills\n\n- `correlate-observability-signals` - Unified debugging across metrics, logs, and traces using trace IDs\n- `build-grafana-dashboards` - Visualize log-derived metrics and create log panels in dashboards\n- `setup-prometheus-monitoring` - Metrics provide context for when to query logs during incidents\n- `instrument-distributed-tracing` - Add trace IDs to logs for correlation with distributed traces\n","# Configure Log Aggregation\n\nImplement centralized log collection, parsing, and querying with Loki\u002FPromtail or ELK stack for operational visibility.\n\n## When to Use\n\n- Consolidating logs from multiple services or hosts into a searchable system\n- Replacing local log files with centralized, queryable log storage\n- Correlating logs with metrics and traces for full observability\n- Implementing structured logging with label extraction from unstructured logs\n- Setting retention policies for log data based on storage and compliance needs\n- Troubleshooting production incidents requiring log analysis across services\n\n## Inputs\n\n- **Required**: Log sources (application logs, system logs, container logs)\n- **Required**: Log format patterns (JSON, plaintext, syslog, etc.)\n- **Optional**: Label extraction rules for structured querying\n- **Optional**: Retention and compression policies\n- **Optional**: Existing log shipper configuration (Fluentd, Filebeat, Promtail)\n\n## Procedure\n\n> See [Extended Examples](references\u002FEXAMPLES.md) for complete configuration files and templates.\n\n\n### Step 1: Choose Log Aggregation Stack\n\nSelect between Loki (Prometheus-style) or ELK (Elasticsearch-based) based on requirements.\n\n**Loki advantages**:\n- Lightweight, designed for Kubernetes and cloud-native environments\n- Label-based indexing (like Prometheus) for low storage overhead\n- Native integration with Grafana for unified dashboards\n- Horizontal scalability with object storage (S3, GCS)\n- Lower resource consumption compared to Elasticsearch\n\n**ELK advantages**:\n- Full-text search across all log content (not only labels)\n- Rich query DSL and aggregations\n- Mature ecosystem with beats, logstash plugins\n- Better for compliance\u002Faudit logs requiring deep historical search\n\nFor this guide, we'll focus on **Loki + Promtail** (recommended for most modern setups).\n\nDecision criteria:\n```markdown\nUse Loki if:\n- You want label-based queries similar to Prometheus\n- Storage costs are a concern (Loki indexes only labels)\n- You already use Grafana for metrics\n- Kubernetes\u002Fcontainer-native deployment\n\nUse ELK if:\n- You need full-text search across all log content\n- You have complex log parsing and enrichment requirements\n- You require advanced analytics and aggregations\n- Legacy systems with existing Logstash pipelines\n```\n\n**Got:** Clear choice made based on requirements, team downloads appropriate installation artifacts.\n\n**If fail:**\n- Benchmark storage requirements: Loki ~10x less than Elasticsearch for same logs\n- Evaluate query patterns: full-text search needs vs label filtering\n- Consider operational overhead: ELK requires more tuning and resources\n\n### Step 2: Deploy Loki\n\nInstall and configure Loki with appropriate storage backend.\n\n**Docker Compose deployment** (`docker-compose.yml`):\n\n```yaml\nversion: '3.8'\n\nservices:\n  loki:\n    image: grafana\u002Floki:2.9.0\n    ports:\n      - \"3100:3100\"\n    volumes:\n      - .\u002Floki-config.yml:\u002Fetc\u002Floki\u002Flocal-config.yaml\n      - loki-data:\u002Floki\n    command: -config.file=\u002Fetc\u002Floki\u002Flocal-config.yaml\n    restart: unless-stopped\n\n  promtail:\n    image: grafana\u002Fpromtail:2.9.0\n    volumes:\n      - .\u002Fpromtail-config.yml:\u002Fetc\u002Fpromtail\u002Fconfig.yml\n      - \u002Fvar\u002Flog:\u002Fvar\u002Flog:ro\n      - \u002Fvar\u002Flib\u002Fdocker\u002Fcontainers:\u002Fvar\u002Flib\u002Fdocker\u002Fcontainers:ro\n    command: -config.file=\u002Fetc\u002Fpromtail\u002Fconfig.yml\n    restart: unless-stopped\n    depends_on:\n      - loki\n\nvolumes:\n  loki-data:\n```\n\n**Loki configuration** (`loki-config.yml`):\n\n```yaml\nauth_enabled: false\n\nserver:\n  http_listen_port: 3100\n  grpc_listen_port: 9096\n\n# ... (see EXAMPLES.md for complete configuration)\n```\n\nFor **production** with S3 storage:\n\n```yaml\nstorage_config:\n  aws:\n    s3: s3:\u002F\u002Fus-east-1\u002Fmy-loki-bucket\n    s3forcepathstyle: true\n  boltdb_shipper:\n    active_index_directory: \u002Floki\u002Findex\n    cache_location: \u002Floki\u002Fcache\n    shared_store: s3\n```\n\n**Got:** Loki starts successfully, health check passes at `http:\u002F\u002Flocalhost:3100\u002Fready`, logs stored according to retention policy.\n\n**If fail:**\n- Check Loki logs: `docker logs loki`\n- Verify storage directories exist and are writable\n- Test config syntax: `docker run grafana\u002Floki:2.9.0 -config.file=\u002Fetc\u002Floki\u002Flocal-config.yaml -verify-config`\n- Ensure retention settings don't exceed disk capacity\n- For S3: verify IAM permissions and bucket access\n\n### Step 3: Configure Promtail for Log Shipping\n\nSet up Promtail to scrape logs and forward to Loki with label extraction.\n\n**Promtail configuration** (`promtail-config.yml`):\n\n```yaml\nserver:\n  http_listen_port: 9080\n  grpc_listen_port: 0\n\npositions:\n  filename: \u002Ftmp\u002Fpositions.yaml\n# ... (see EXAMPLES.md for complete configuration)\n```\n\nKey Promtail concepts:\n- **Scrape configs**: Define log sources and how to discover them\n- **Pipeline stages**: Transform and label logs before sending to Loki\n- **Relabel configs**: Dynamic labeling based on metadata\n- **Positions file**: Tracks read offsets to avoid re-processing logs\n\n**Got:** Promtail scrapes configured log files, labels applied correctly, logs visible in Loki via LogQL queries.\n\n**If fail:**\n- Check Promtail logs: `docker logs promtail`\n- Verify file paths are accessible: `docker exec promtail ls \u002Fvar\u002Flog`\n- Test regex patterns independently with sample log lines\n- Monitor Promtail metrics: `curl http:\u002F\u002Flocalhost:9080\u002Fmetrics | grep promtail`\n- Check positions file for progress: `cat \u002Ftmp\u002Fpositions.yaml`\n\n### Step 4: Query Logs with LogQL\n\nLearn LogQL syntax for filtering and aggregating logs.\n\n**Basic queries**:\n\n```logql\n# All logs from a job\n{job=\"app\"}\n\n# Logs with specific label values\n{job=\"app\", level=\"error\"}\n\n# Regex filter on log line content\n{job=\"app\"} |~ \"authentication failed\"\n\n# Case-insensitive regex\n{job=\"app\"} |~ \"(?i)error\"\n\n# Line filter (doesn't parse, just includes\u002Fexcludes)\n{job=\"app\"} |= \"user\"  # Contains \"user\"\n{job=\"app\"} != \"debug\" # Doesn't contain \"debug\"\n```\n\n**Parsing and filtering**:\n\n```logql\n# JSON parsing\n{job=\"app\"} | json | level=\"error\"\n\n# Regex parsing with named groups\n{job=\"app\"} | regexp \"user_id=(?P\u003Cuser_id>\\\\d+)\" | user_id=\"12345\"\n\n# Logfmt parsing (key=value format)\n{job=\"app\"} | logfmt | level=\"error\", service=\"auth\"\n\n# Pattern parsing\n{job=\"nginx\"} | pattern `\u003Cip> - \u003Cuser> [\u003Ctimestamp>] \"\u003Cmethod> \u003Cpath> \u003Cprotocol>\" \u003Cstatus> \u003Csize>` | status >= 500\n```\n\n**Aggregations** (metrics from logs):\n\n```logql\n# Count log lines per level\nsum by (level) (count_over_time({job=\"app\"}[5m]))\n\n# Rate of error logs\nrate({job=\"app\", level=\"error\"}[5m])\n\n# Bytes processed per service\nsum by (service) (bytes_over_time({job=\"app\"}[1h]))\n\n# Average request duration from logs\navg_over_time({job=\"app\"} | json | unwrap duration [5m])\n\n# Top 10 error messages\ntopk(10, sum by (message) (count_over_time({level=\"error\"} [1h])))\n```\n\n**Filtering by extracted fields**:\n\n```logql\n# Find specific trace in logs\n{job=\"app\"} | json | trace_id=\"abc123def456\"\n\n# HTTP 5xx errors from nginx\n{job=\"nginx\"} | pattern `\u003C_> \"\u003C_> \u003C_> \u003C_>\" \u003Cstatus> \u003C_>` | status >= 500\n\n# Failed authentication attempts\n{job=\"app\"} | json | message=~\"authentication failed\" | user_id != \"\"\n```\n\nCreate Grafana explore queries or dashboard panels using these patterns.\n\n**Got:** Queries return expected log lines, filtering works correctly, aggregations produce metrics from logs.\n\n**If fail:**\n- Use Grafana Explore to debug queries interactively\n- Check label names: `curl http:\u002F\u002Flocalhost:3100\u002Floki\u002Fapi\u002Fv1\u002Flabels`\n- Verify label values: `curl http:\u002F\u002Flocalhost:3100\u002Floki\u002Fapi\u002Fv1\u002Flabel\u002F{label_name}\u002Fvalues`\n- Simplify query: start with basic label selector, add filters incrementally\n- Check time range: logs might not exist in selected window\n\n### Step 5: Integrate Logs with Metrics and Traces\n\nCorrelate logs with Prometheus metrics and distributed traces for unified observability.\n\n**Add trace IDs to logs** (application instrumentation):\n\n```python\n# Python with OpenTelemetry\nimport logging\nfrom opentelemetry import trace\n\nlogger = logging.getLogger(__name__)\n\ndef handle_request():\n    span = trace.get_current_span()\n    trace_id = span.get_span_context().trace_id\n\n    logger.info(\n        \"Processing request\",\n        extra={\"trace_id\": format(trace_id, \"032x\")}\n    )\n```\n\n```go\n\u002F\u002F Go with OpenTelemetry\nimport (\n    \"go.opentelemetry.io\u002Fotel\u002Ftrace\"\n    \"go.uber.org\u002Fzap\"\n)\n\nfunc handleRequest(ctx context.Context) {\n    span := trace.SpanFromContext(ctx)\n    traceID := span.SpanContext().TraceID().String()\n\n    logger.Info(\"Processing request\",\n        zap.String(\"trace_id\", traceID),\n    )\n}\n```\n\n**Configure Grafana data links** from metrics to logs:\n\nIn Prometheus panel field config:\n\n```json\n{\n  \"fieldConfig\": {\n    \"defaults\": {\n      \"links\": [\n        {\n          \"title\": \"View Logs\",\n          \"url\": \"\u002Fexplore?left={\\\"datasource\\\":\\\"Loki\\\",\\\"queries\\\":[{\\\"refId\\\":\\\"A\\\",\\\"expr\\\":\\\"{job=\\\\\\\"app\\\\\\\",instance=\\\\\\\"${__field.labels.instance}\\\\\\\"} |= `${__field.labels.trace_id}`\\\"}],\\\"range\\\":{\\\"from\\\":\\\"${__from}\\\",\\\"to\\\":\\\"${__to}\\\"}}\",\n          \"targetBlank\": false\n        }\n      ]\n    }\n  }\n}\n```\n\n**Configure Grafana data links** from logs to traces:\n\nIn Loki datasource config:\n\n```yaml\ndatasources:\n  - name: Loki\n    type: loki\n    url: http:\u002F\u002Floki:3100\n    jsonData:\n      derivedFields:\n        - datasourceName: Tempo\n          matcherRegex: \"trace_id=(\\\\w+)\"\n          name: TraceID\n          url: \"$${__value.raw}\"\n```\n\n**Correlate logs in Grafana Explore**:\n1. Query metrics in Prometheus\n2. Click on data point\n3. Select \"View Logs\" from context menu\n4. Loki query auto-populated with relevant labels and time range\n5. Click trace ID in logs\n6. Tempo trace view opens with full distributed trace\n\n**Got:** Clicking metrics opens related logs, trace IDs in logs link to trace viewer, single pane for metrics\u002Flogs\u002Ftraces navigation.\n\n**If fail:**\n- Verify trace ID format matches regex in derived fields\n- Check that trace_id label extracted by Promtail pipeline\n- Ensure Tempo datasource configured in Grafana\n- Test URL encoding for complex filter expressions\n- Validate data link URLs in incognito\u002Fprivate browser window\n\n### Step 6: Set Up Log Retention and Compaction\n\nConfigure retention policies and compaction to manage storage costs.\n\n**Retention by stream** (in Loki config):\n\n```yaml\nlimits_config:\n  retention_period: 720h  # Global default: 30 days\n\n  # Per-tenant retention (requires multi-tenancy enabled)\n  per_tenant_override_config: \u002Fetc\u002Floki\u002Foverrides.yaml\n\n# overrides.yaml\noverrides:\n  production:\n    retention_period: 2160h  # 90 days for production\n  staging:\n    retention_period: 360h   # 15 days for staging\n  development:\n    retention_period: 168h   # 7 days for dev\n```\n\n**Retention by stream labels** (requires compactor):\n\n```yaml\ncompactor:\n  working_directory: \u002Floki\u002Fcompactor\n  shared_store: filesystem\n  compaction_interval: 10m\n  retention_enabled: true\n  retention_delete_delay: 2h\n# ... (see EXAMPLES.md for complete configuration)\n```\n\nPriority determines which rule applies when multiple match (lower number = higher priority).\n\n**Compression settings**:\n\n```yaml\nchunk_store_config:\n  chunk_cache_config:\n    enable_fifocache: true\n    fifocache:\n      max_size_bytes: 1GB\n      ttl: 24h\n# ... (see EXAMPLES.md for complete configuration)\n```\n\n**Monitor retention**:\n\n```bash\n# Check chunk stats\ncurl http:\u002F\u002Flocalhost:3100\u002Floki\u002Fapi\u002Fv1\u002Fstatus\u002Fchunks | jq\n\n# Check compactor metrics\ncurl http:\u002F\u002Flocalhost:3100\u002Fmetrics | grep loki_compactor\n\n# Verify deleted chunks\ncurl http:\u002F\u002Flocalhost:3100\u002Fmetrics | grep loki_boltdb_shipper_retention_deleted\n```\n\n**Got:** Old logs automatically deleted per retention policy, storage usage stabilizes, compaction reduces index size.\n\n**If fail:**\n- Enable compactor in Loki config if retention not working\n- Check compactor logs: `docker logs loki | grep compactor`\n- Verify retention_enabled: true and retention_deletes_enabled: true\n- Monitor disk usage: `du -sh \u002Floki\u002F`\n- For S3: check bucket lifecycle policies don't conflict with Loki retention\n\n## Validation\n\n- [ ] Loki API health check returns 200: `curl http:\u002F\u002Flocalhost:3100\u002Fready`\n- [ ] Promtail successfully scraping logs from all configured sources\n- [ ] Labels extracted correctly from log lines (visible in Grafana Explore)\n- [ ] LogQL queries return expected results with proper filtering\n- [ ] Log retention policy enforced (old logs deleted after retention period)\n- [ ] Logs accessible from Grafana dashboards and Explore view\n- [ ] Trace IDs from logs link to Tempo trace viewer\n- [ ] Metrics panels have data links to relevant logs\n- [ ] Compaction running and reducing storage overhead\n- [ ] Storage usage within allocated disk\u002FS3 budget\n\n## Pitfalls\n\n- **High cardinality labels**: Using unbounded label values (user IDs, request IDs) causes index explosion. Use fixed labels (level, service, env) and put variables in log lines.\n- **Missing log parsing**: Sending raw logs without label extraction limits query capabilities. Always parse structured logs (JSON, logfmt) or use regex for unstructured.\n- **Incorrect time parsing**: Mismatched timestamp formats cause logs to be out of order or rejected. Test timestamp parsing with sample logs.\n- **Retention not working**: Compactor must be enabled for retention to delete old data. Check `retention_enabled: true` and `retention_deletes_enabled: true`.\n- **Ingestion rate limits**: Default limits (10MB\u002Fs) may be too low for high-volume systems. Adjust `ingestion_rate_mb` and `ingestion_burst_size_mb`.\n- **Query timeouts**: Broad queries over long time ranges can timeout. Use more specific label selectors and shorter time windows.\n- **Log duplication**: Multiple Promtail instances scraping same logs create duplicates. Use unique labels or positions file coordination.\n\n## Related Skills\n\n- `correlate-observability-signals` - Unified debugging across metrics, logs, and traces using trace IDs\n- `build-grafana-dashboards` - Visualize log-derived metrics and create log panels in dashboards\n- `setup-prometheus-monitoring` - Metrics provide context for when to query logs during incidents\n- `instrument-distributed-tracing` - Add trace IDs to logs for correlation with distributed traces\n",{"name":5,"locale":23,"license":14,"metadata":24,"translator":31,"description":32,"allowed-tools":33,"source_commit":34,"source_locale":35,"translation_date":36},"caveman-lite",{"tags":25,"author":26,"domain":27,"version":28,"language":29,"complexity":30},"loki, promtail, logging, elk, log-aggregation","Philipp Thoss","observability","1.0","multi","intermediate","Julius Brussee homage — caveman","Set up centralized log aggregation with Loki and Promtail (or ELK stack), including log parsing, label extraction, retention policies, and integration with metrics for correlation. Use when consolidating logs from multiple services into a searchable system, replacing local log files with centralized queryable storage, correlating logs with metrics and traces, implementing structured logging with label extraction, or troubleshooting production incidents requiring cross-service log analysis.\n","Read Write Edit Bash Grep Glob","82c77053","en","2026-04-19","master","f483e54f3ad5f202a77d45f2e57f8d970b71195f","skills_library\u002Fconfigure-log-aggregation",2,"https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills","https:\u002F\u002Fgithub.com\u002Froohe","yoou.link","2026-05-19 15:56:07",{"source":8,"skillName":5,"commands":46,"prompt":56,"zipAvailable":57},[47,50,53],{"tool":48,"command":49},"npx","npx skills add roohe\u002Fagentic-super-skills --skill configure-log-aggregation",{"tool":51,"command":52},"bunx","bunx skills add roohe\u002Fagentic-super-skills --skill configure-log-aggregation",{"tool":54,"command":55},"pnpm","pnpm dlx skills add roohe\u002Fagentic-super-skills --skill configure-log-aggregation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fconfigure-log-aggregation\u002FSKILL.md",true,[59],{"path":60,"resourceType":61,"resourceTypeDesc":62,"sizeBytes":63,"sourceHtmlUrl":64},"references\u002FEXAMPLES.md","REFERENCE","参考文档",6826,"https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fconfigure-log-aggregation\u002Freferences\u002FEXAMPLES.md",[66,73,80,87,94,101],{"skillNo":67,"name":68,"nameZh":6,"description":69,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":70,"categories":71,"keywords":72,"publishedAt":18},"01M73D865CZME2FCMKA4PQM348","zarr-python","Chunked N-D arrays for cloud storage. Compressed arrays, parallel I\u002FO, S3\u002FGCS integration, NumPy\u002FDask\u002FXarray compatible, for large-scale scientific computing pipelines.","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fzarr-python\u002FSKILL.md",[],[],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":77,"categories":78,"keywords":79,"publishedAt":18},"01M73D7Z64J4ABWZJD7P44XBBF","youtube-downloader","Download and process YouTube content for research. Use when: downloading competitor videos for analysis; extracting audio for podcasts; getting transcripts for content repurposing; archiving webinars; research content curation","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fyoutube-downloader\u002FSKILL.md",[],[],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":18},"01M73D7K8778TGZV3GJFR3703D","yc-pitch-deck","Create a compelling startup pitch deck that follows the structure proven to raise billions from top investors. Master the YC and Sequoia formats that get founders funded. Use when: **Fundraising** to create investor pitch decks; **YC application** to structure your narrative; **Demo Day prep** to craft your 2-minute pitch; **Angel investors** to communicate your opportunity; **Partnership pitches** to structure compelling asks","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fyc-pitch-deck\u002FSKILL.md",[],[],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":18},"01M73D70KJSE0E8Z28YMNNRT04","x-twitter-scraper","Use when the user needs X (Twitter) data or confirmation-gated X actions through Xquik: tweet search, user lookup, follower extraction, media download, monitoring, webhooks, MCP, SDKs, posting, likes, DMs, and profile updates. Requires a Xquik API key. Never ask for X login material.","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fx-twitter-scraper\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":18},"01M73D6ED01GV077R7K4NZNFZJ","write-vignette","Create R package vignettes using R Markdown or Quarto. Covers vignette setup, YAML configuration, code chunk options, building and testing, and CRAN requirements for vignettes. Use when adding a Getting Started tutorial, documenting complex workflows spanning multiple functions, creating domain-specific guides, or when CRAN submission requires user-facing documentation beyond function help pages.","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fwrite-vignette\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":18},"01M73D69YJR0BK3NFHE0CYG3EF","write-validation-documentation","Write IQ\u002FOQ\u002FPQ validation documentation for computerized systems in regulated environments. Covers protocols, reports, test scripts, deviation handling, and approval workflows. Use when validating R or other software for regulated use, preparing for a regulatory audit, documenting qualification of computing environments, or creating and updating validation protocols and reports for new or re-qualified systems.","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fwrite-validation-documentation\u002FSKILL.md",[],[]]