AGENT SKILL

prometheus-pro

Prometheus guidance — metrics design, PromQL, alerting rules, exporters, recording rules, and operations.

License 许可
MIT
所属仓库
TanNhatCMS/jarvis
最近更新
2026-11-07
Stars
★ 0
Forks
1

01 前言 Front Matter

name
prometheus-pro
category
development
description
Prometheus guidance — metrics design, PromQL, alerting rules, exporters, recording rules, and operations.

03 SKILL.md 正文

Overview

Prometheus is the standard open-source metrics system: it scrapes HTTP endpoints for time-series metrics, stores them efficiently, queries with PromQL, and alerts via Alertmanager. The pull model, dimensional labels, and powerful query language make it the backbone of Kubernetes observability and much more.

Using Prometheus well is about metric design (what to instrument, how to label), PromQL fluency, and alerting discipline (alert on symptoms, not causes). This skill covers instrumentation patterns, PromQL, recording/alerting rules, and operating Prometheus reliably.

When to use

  • Instrumenting applications (choosing metrics and labels).
  • Writing PromQL queries and dashboards.
  • Designing alerting rules (what to alert on, thresholds).
  • Setting up exporters (node, blackbox, custom).
  • Configuring recording rules for expensive queries.
  • Operating Prometheus (retention, HA, federation, remote storage).
  • Choosing between Prometheus, Datadog, and cloud metrics.

Core concepts

  • Pull model. Prometheus scrapes /metrics endpoints on an interval — targets can die without affecting the server, and there's no agent to deploy per app. Pushgateway exists for batch jobs; it's the exception, not the pattern.
  • Metric types. Counter (monotonically increasing — requests, errors), Gauge (current value — connections, queue depth), Histogram (distributions — latencies, with buckets), Summary (client-computed quantiles — avoid, use histograms). Choosing the wrong type makes queries misleading.
  • Labels are dimensions. http_requests_total{method="GET", status="500"} — labels slice metrics. Keep cardinality bounded: labels with unbounded values (user IDs, request IDs) explode storage and crash servers.
  • Naming conventions. <namespace>_<subsystem>_<unit>_{total,seconds,bytes} — http_request_duration_seconds. Consistent names make PromQL discoverable; document your conventions.
  • RED and USE. RED (Rate, Errors, Duration) for services; USE (Utilization, Saturation, Errors) for resources. These mnemonics structure both instrumentation and dashboards — cover them before exotic metrics.
  • PromQL essentials. rate() for counters (always over a range, never irate for alerting), histogram_quantile() for latency percentiles, avg_over_time/max_over_time for gauges, label matching and aggregation (sum by (service)).
  • Recording rules. Precompute expensive queries (job:http_request_duration:p99) on an interval — dashboards and alerts query the recording, not the raw data. Essential at scale.
  • Alerting rules. Alert on symptoms (error rate, latency, saturation) with for: durations to avoid flapping. Every alert needs a runbook link, a severity, and an owner — alerts without runbooks are just noise.
  • Alertmanager. Deduplication, grouping, silencing, and routing (PagerDuty/Opsgenie/Slack). Group related alerts; route by severity and team; inhibit cascading alerts (don't page for every pod when the node is down).
  • Exporters. node_exporter (host metrics), blackbox_exporter (probing), kube-state-metrics (K8s objects), plus client libraries for app metrics. Instrument apps directly rather than parsing logs for metrics.
  • Service discovery. Kubernetes, Consul, EC2, DNS-based discovery — targets come from the environment, not static configs. Relabeling shapes discovered targets into useful label sets.
  • Retention and storage. Local TSDB with retention by time/size; 15 days default is often too short — size for your investigation needs. Remote write (Thanos, Cortex/Mimir, VictoriaMetrics) for long-term storage and HA.
  • HA. Paired Prometheus servers (each scraping everything) + remote write, or Thanos/Cortex for global query. Single Prometheus is a single point of failure for alerting — plan accordingly.
  • Cardinality. The #1 operational risk: unbounded label values → memory explosion → OOM. Monitor prometheus_tsdb_head_series, set label limits, and review new metrics' cardinality before deploying.
  • Exemplars. Link metrics to traces (trace IDs on histograms) — connecting the RED metrics to the slow traces in one click.

Practical workflow

  1. Instrument RED metrics. Every service exposes request rate, error rate, and duration histogram with bounded labels (route templates, not raw paths).
    REQUESTS = Counter("http_requests_total", "Requests", ["method", "route", "status"])
    LATENCY = Histogram("http_request_duration_seconds", "Latency", ["method", "route"])
    
  2. Deploy exporters. node_exporter on hosts, kube-state-metrics in clusters, blackbox for external probing; app metrics via client libraries on /metrics.
  3. Write recording rules. Precompute SLIs and expensive aggregations; name them clearly (job:metric:aggregation).
    groups:
    - name: slis
      interval: 1m
      rules:
      - record: job:http_requests:error_ratio
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
              / sum by (job) (rate(http_requests_total[5m]))
    
  4. Alert on symptoms. Error-ratio, p99 latency, saturation alerts with for: and runbook annotations; severity tiers (page vs ticket).
    - alert: HighErrorRate
      expr: job:http_requests:error_ratio > 0.05
      for: 5m
      labels: { severity: page }
      annotations: { runbook: https://wiki.example.com/runbooks/high-error-rate }
    
  5. Configure Alertmanager. Grouping by alertname/cluster, routing by severity/team, inhibition for cascades, silences for maintenance — not for ignoring problems.
  6. Build dashboards from SLIs. RED per service, USE per resource, recording-rule-backed panels; dashboards answer "is it broken and where" in under a minute.
  7. Manage cardinality. Review label sets on new metrics; drop high-cardinality labels at scrape time via relabeling; alert on head-series growth.
  8. Plan retention and HA. Size retention for investigation needs; remote-write to long-term storage; paired servers or Thanos for alerting HA.

Common pitfalls

  • Unbounded label cardinality — user IDs or request IDs as labels; OOM. Bound every label's value set.
  • Alerting on causes — paging for CPU spikes instead of user-facing symptoms; alert on SLIs.
  • No for: duration — flapping alerts on transient blips; require sustained breach.
  • Alerts without runbooks — 3am pages with no guidance; every alert links a runbook.
  • irate in alerts — spiky and misleading; rate over sensible windows.
  • Raw paths as labels — /users/123 exploding cardinality; use route templates.
  • Summary instead of histogram — client-side quantiles can't aggregate; histograms with histogram_quantile.
  • Single Prometheus for alerting — SPOF for pages; HA pair or remote-write + redundant alerting.
  • No recording rules at scale — dashboards running expensive queries on every load; precompute.
  • Pushgateway as default — for batch jobs only; services get scraped.
  • Ignoring silences hygiene — permanent silences hiding real problems; expire and review them.
  • Retention too short — 15 days when investigations need months; size deliberately with remote storage.
  • Scraping everything at 15s — unnecessary load; tune intervals per target importance.

04 同仓库更多 Skills

TanNhatCMS
TanNhatCMS/jarvis

web-scrape

Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open/fetch fails on JS-heavy pages, or when you need clean markdown/structured extraction from a site. Requires FIRECRAWL_API_KEY.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

design-taste-frontend

tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML/CSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, "AI-made", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

test-driven-development

Use when implementing any feature or bugfix, before writing implementation code

★ 0
TanNhatCMS
TanNhatCMS/jarvis

systematic-debugging

Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes

★ 0
TanNhatCMS
TanNhatCMS/jarvis

requesting-code-review

Use when completing tasks, implementing major features, or before merging to verify work meets requirements

★ 0
TanNhatCMS
TanNhatCMS/jarvis

brainstorming

You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).

★ 0