AGENT SKILL

prometheus-grafana-metrics

Design application metrics, Prometheus scraping topologies, PromQL queries, alerting rules, and Grafana dashboards as code. Triggers when implementing RED / USE metric patterns, writing custom Prometheus exporters (Python / Go), writing alerting rules, calculating SLO burn rates in PromQL, or generating Grafana dashboard JSON configurations.

License 许可
MIT
最近更新
2026-10-17
Stars
★ 8
Forks
1

01 前言 Front Matter

name
prometheus-grafana-metrics
metadata
{ "category": "Observability Monitoring and Telemetry" }
description
Design application metrics, Prometheus scraping topologies, PromQL queries, alerting rules, and Grafana dashboards as code. Triggers when implementing RED / USE metric patterns, writing custom Prometheus exporters (Python / Go), writing alerting rules, calculating SLO burn rates in PromQL, or generating Grafana dashboard JSON configurations.
compatibility
Prometheus (>= 2.45.0), Grafana (>= 10.0), Prometheus Operator / Helm, Python / Go Client SDKs

03 SKILL.md 正文

Prometheus & Grafana Metrics as Code

Production patterns for metric instrumentation, PromQL query optimization, Alertmanager routing rules, and Grafana dashboard provisioning.


1. Metrics Methodology Framework

+------------------------------------------------------------------------------------+
| 4 Golden Signals: Latency, Traffic, Errors, Saturation                              |
+------------------------------------------------------------------------------------+
| RED Method (Services & APIs):                                                       |
|   - Rate:     sum(rate(http_requests_total[5m]))                                    |
|   - Errors:   sum(rate(http_requests_total{status=~"5.."}[5m]))                     |
|   - Duration: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
+------------------------------------------------------------------------------------+
| USE Method (Hardware & Resources):                                                  |
|   - Utilization: node_cpu_seconds_total                                            |
|   - Saturation:  node_load1                                                         |
|   - Errors:       node_net_receive_errs_total                                       |
+------------------------------------------------------------------------------------+

2. Custom Application Metric Exporter (Python & Go)

Python Exporter Implementation (metrics_exporter.py)

from prometheus_client import start_http_server, Counter, Histogram, Gauge, Summary
import time
import random

# 1. Metric Definitions following RED pattern
REQUEST_COUNT = Counter(
    "http_requests_total",
    "Total number of HTTP requests processed",
    ["method", "endpoint", "status_code"]
)

LATENCY_HISTOGRAM = Histogram(
    "http_request_duration_seconds",
    "HTTP Request Latency in Seconds",
    ["endpoint"],
    buckets=(0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0) # Tailored latency buckets
)

ACTIVE_CONNECTIONS = Gauge(
    "app_active_connections",
    "Current active database connection pool size"
)

def simulate_work():
    ACTIVE_CONNECTIONS.set(random.randint(5, 20))
    
    endpoint = "/api/v1/orders"
    start_time = time.time()
    
    # Simulate work
    time.sleep(random.uniform(0.02, 0.3))
    duration = time.time() - start_time
    
    status_code = "200" if random.random() > 0.05 else "500"
    
    # Record metrics
    REQUEST_COUNT.labels(method="POST", endpoint=endpoint, status_code=status_code).inc()
    LATENCY_HISTOGRAM.labels(endpoint=endpoint).observe(duration)

if __name__ == "__main__":
    start_http_server(9090)
    print("Prometheus Metrics exporter running on http://localhost:9090/metrics")
    while True:
        simulate_work()

3. Prometheus Alerting Rules & SLO Burn Rates (alert_rules.yaml)

groups:
  - name: API_SLO_Alerts
    rules:
      # 1. High Error Rate Alert (> 5% errors over 5m window)
      - alert: APIHighErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status_code=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) * 100 > 5
        for: 2m
        labels:
          severity: critical
          team: platform-ops
        annotations:
          summary: "API Error Rate exceeds 5%"
          description: "High error rate detected on service {{ $labels.job }}. Current error rate: {{ $value | printf \"%.2f\" }}%."

      # 2. P99 Latency Degradation Alert (> 2.0s over 5m)
      - alert: APIHighP99Latency
        expr: |
          histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, endpoint)) > 2.0
        for: 5m
        labels:
          severity: warning
          team: backend-devs
        annotations:
          summary: "P99 Latency Spike on {{ $labels.endpoint }}"
          description: "99th percentile latency on {{ $labels.endpoint }} is {{ $value | printf \"%.2f\" }}s."

      # 3. Multi-Window Fast Burn Rate SLO Alert (SLO 99.9% availability target)
      - alert: ServiceSLOBurnRateCritical
        expr: |
          (
            sum(rate(http_requests_total{status_code=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          ) > (1 - 0.999) * 14.4
        for: 2m
        labels:
          severity: page
        annotations:
          summary: "Fast Burn Rate: 2% of 30-day Error Budget consumed in 1 hour"

4. PromQL Cheat Sheet for Production

Query Intent PromQL Query Expression
Total Request Throughput sum(rate(http_requests_total[5m])) by (job)
Percentage Error Rate (sum(rate(http_requests_total{status_code=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) * 100
95th Latency Percentile histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
CPU Utilization per Node 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
Memory Available % (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

5. Grafana Dashboard Provisioning as Code (dashboard_provisioning.json)

{
  "annotations": { "list": [] },
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 1,
  "id": null,
  "title": "Production API Services Dashboard",
  "tags": ["production", "prometheus", "red"],
  "timezone": "browser",
  "panels": [
    {
      "title": "HTTP Request Throughput (RPS)",
      "type": "timeseries",
      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
      "targets": [
        {
          "expr": "sum(rate(http_requests_total[2m])) by (endpoint)",
          "legendFormat": "{{endpoint}}"
        }
      ]
    },
    {
      "title": "P99 Latency Quantile",
      "type": "timeseries",
      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 },
      "fieldConfig": {
        "defaults": { "unit": "s" }
      },
      "targets": [
        {
          "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[2m])) by (le, endpoint))",
          "legendFormat": "P99 - {{endpoint}}"
        }
      ]
    }
  ],
  "schemaVersion": 38,
  "version": 1
}

6. Metric Cardinality Best Practices

  1. Avoid High-Cardinality Labels: Never add unique IDs (User ID, IP address, UUID, session tokens) as Prometheus metric label dimensions to prevent TSDB memory exhaustion.
  2. Explicit Histogram Buckets: Define custom exponential or linear histogram buckets tailored to your expected latency SLAs (e.g. 10ms to 2s for web APIs).
  3. Scrape Frequency Alignment: Match alert evaluation intervals (evaluation_interval: 15s) with your scraping interval (scrape_interval: 15s).

04 同仓库更多 Skills

hamzabellouch
hamzabellouch/agent-skills

security-audit

Comprehensive security guidance, source code audit, vulnerability assessment, and penetration test review for codebases, APIs, microservices, CLI tools, libraries, and LLM-backed applications. Triggers on requests to audit a codebase, conduct security reviews, find vulnerabilities or flaws, pen-test source code, or when the user asks in Arabic (فحص أمني، تدقيق الكود أمنياً، مراجعة الثغرات، فحص أمان المشروع). Supports Guidance mode (focused review) and Full Audit mode (deterministic 6-phase harness).

★ 8
hamzabellouch
hamzabellouch/agent-skills

malware-analysis-reversing

Production malware analysis workflows, reverse engineering methodologies, isolated sandbox triage, static/dynamic binary analysis, memory forensics, and YARA detection rule authoring.

★ 8
hamzabellouch
hamzabellouch/agent-skills

application-security-pentesting

Production Application Security (AppSec) standards, OWASP Top 10 vulnerability remediation, SAST/DAST pipeline integration, secure code review, threat modeling, and defensive security posture.

★ 8
hamzabellouch
hamzabellouch/agent-skills

oauth2-oidc-security-flows

Implement enterprise OAuth 2.1 and OpenID Connect (OIDC) authorization code flow with Proof Key for Code Exchange (PKCE), state and nonce verification, refresh token rotation, token revocation, and centralized IdP federation (Keycloak, Auth0, Okta). Trigger when building secure authentication, single sign-on (SSO), or third-party delegated authorization.

★ 8
hamzabellouch
hamzabellouch/agent-skills

jwt-session-management-hardening

Harden JWT-based authentication and session lifecycles using asymmetric signing (RS256/EdDSA), JWKS rotation, strict claims validation, Redis-backed sliding window sessions, revocation lists, and secure HttpOnly cookie storage. Trigger when implementing token validation, session revocation, or hardening API authentication.

★ 8
hamzabellouch
hamzabellouch/agent-skills

soc2-compliance-automation

Best practices for implementing SOC 2 Type II compliance controls, continuous evidence collection, infrastructure policy-as-code, and automated security monitoring. Use when automating Trust Services Criteria (Security, Availability, Confidentiality, Processing Integrity, Privacy), audit logging, IAM least privilege, or CI/CD security gates.

★ 8