setting-up-log-aggregation

Prerequisites

Before using this skill, ensure:

• Target infrastructure is identified (Kubernetes, Docker, VMs)
• Storage requirements are calculated based on log volume
• Network connectivity between log sources and aggregation platform
• Authentication mechanism is defined (LDAP, OAuth, basic auth)
• Resource allocation planned (CPU, memory, disk)

Instructions

1. Select Platform: Choose ELK, Loki, Grafana Loki, or Splunk
2. Configure Ingestion: Set up log shippers (Filebeat, Promtail, Fluentd)
3. Define Storage: Configure retention policies and index lifecycle
4. Set Up Processing: Create parsing rules and field extractions
5. Deploy Visualization: Configure Kibana/Grafana dashboards
6. Implement Security: Enable authentication, encryption, and RBAC
7. Test Pipeline: Verify logs flow from sources to visualization

Output

ELK Stack (Docker Compose):

# {baseDir}/elk/docker-compose.yml
version: '3.8'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
    volumes:
      - es-data:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"

  logstash:
    image: docker.elastic.co/logstash/logstash:8.11.0
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    depends_on:
      - elasticsearch

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

Loki Configuration:

# {baseDir}/loki/loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  chunk_idle_period: 5m
  chunk_retain_period: 30s

schema_config:
  configs:
    - from: 2024-01-01
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

Error Handling

Out of Memory

• Error: "Elasticsearch heap space exhausted"
• Solution: Increase heap size in elasticsearch.yml or add more nodes

Connection Refused

• Error: "Cannot connect to Elasticsearch"
• Solution: Verify network connectivity and firewall rules

Index Creation Failed

• Error: "Failed to create index"
• Solution: Check disk space and index template configuration

Log Parsing Errors

• Error: "Failed to parse log line"
• Solution: Review grok patterns or JSON parsing configuration

Resources

• ELK Stack guide: https://www.elastic.co/guide/
• Loki documentation: https://grafana.com/docs/loki/
• Example configurations in {baseDir}/log-aggregation-examples/