monitoring-sysadmin

What I do

• Set up comprehensive monitoring systems
• Configure meaningful alerts
• Create actionable dashboards
• Analyze metrics and trends
• Implement observability
• Establish SLOs and SLIs

When to use me

When tracking system health, debugging issues, or establishing operational excellence.

The Three Pillars

Metrics (Prometheus, Datadog)

• Quantitative measurements
• Time-series data
• Counters, gauges, histograms
• Aggregations

Logs (ELK, Loki, Splunk)

• Detailed event records
• Structured/unstructured
• Log levels (DEBUG, INFO, WARN, ERROR)
• Correlation IDs

Traces (Jaeger, Zipkin)

• Request flow across services
• Latency breakdown
• Distributed tracing
• Span context

Metrics Types

Golden Signals

• Latency: Response time
• Traffic: Requests per second
• Errors: Error rate
• Saturation: Resource utilization

RED Metrics (Rate, Errors, Duration)

• For request-driven services

USE Metrics (Utilization, Saturation, Errors)

• For resource-driven services

Alerting

Best Practices

• Alert on symptoms, not causes
• Tune thresholds
• Avoid alert fatigue
• Include context
• Runbooks for each alert

Severity Levels

• Critical: Immediate action needed
• Warning: Attention needed
• Info: For awareness

Dashboards

Design Principles

• Show relevant data
• Include context
• Time ranges
• Drill-down capability
• Shareable links

Common Dashboards

• Service overview
• Infrastructure
• Application performance
• Business metrics
• Incident response

Tools

Monitoring

• Prometheus: Metrics, alerting
• Datadog: Full-stack
• CloudWatch: AWS
• Stackdriver: GCP

Visualization

• Grafana: Dashboards
• Kibana: Logs

Logging

• ELK Stack: Elasticsearch, Logstash, Kibana
• Loki: Promtail + Loki + Grafana
• Splunk: Enterprise

Tracing

• Jaeger: Distributed tracing
• Zipkin: Twitter's tracer
• AWS X-Ray: Cloud

SRE Concepts

SLI (Service Level Indicator)

• Metric measuring service level
• Request latency
• Error rate
• Availability

SLO (Service Level Objective)

• Target SLI value
• "99.9% of requests < 200ms"

SLA (Service Level Agreement)

• Customer-facing commitment
• SLO + consequences