prometheus-pro
Prometheus guidance — metrics design, PromQL, alerting rules, exporters, recording rules, and operations.
- License 许可
- MIT
- 最近更新
- 2026-11-07
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- prometheus-pro
- category
- development
- description
- Prometheus guidance — metrics design, PromQL, alerting rules, exporters, recording rules, and operations.
03 SKILL.md 正文
Overview
Prometheus is the standard open-source metrics system: it scrapes HTTP endpoints for time-series metrics, stores them efficiently, queries with PromQL, and alerts via Alertmanager. The pull model, dimensional labels, and powerful query language make it the backbone of Kubernetes observability and much more.
Using Prometheus well is about metric design (what to instrument, how to label), PromQL fluency, and alerting discipline (alert on symptoms, not causes). This skill covers instrumentation patterns, PromQL, recording/alerting rules, and operating Prometheus reliably.
When to use
- Instrumenting applications (choosing metrics and labels).
- Writing PromQL queries and dashboards.
- Designing alerting rules (what to alert on, thresholds).
- Setting up exporters (node, blackbox, custom).
- Configuring recording rules for expensive queries.
- Operating Prometheus (retention, HA, federation, remote storage).
- Choosing between Prometheus, Datadog, and cloud metrics.
Core concepts
- Pull model. Prometheus scrapes
/metricsendpoints on an interval — targets can die without affecting the server, and there's no agent to deploy per app. Pushgateway exists for batch jobs; it's the exception, not the pattern. - Metric types. Counter (monotonically increasing — requests, errors), Gauge (current value — connections, queue depth), Histogram (distributions — latencies, with buckets), Summary (client-computed quantiles — avoid, use histograms). Choosing the wrong type makes queries misleading.
- Labels are dimensions.
http_requests_total{method="GET", status="500"}— labels slice metrics. Keep cardinality bounded: labels with unbounded values (user IDs, request IDs) explode storage and crash servers. - Naming conventions.
<namespace>_<subsystem>_<unit>_{total,seconds,bytes}—http_request_duration_seconds. Consistent names make PromQL discoverable; document your conventions. - RED and USE. RED (Rate, Errors, Duration) for services; USE (Utilization, Saturation, Errors) for resources. These mnemonics structure both instrumentation and dashboards — cover them before exotic metrics.
- PromQL essentials.
rate()for counters (always over a range, neveriratefor alerting),histogram_quantile()for latency percentiles,avg_over_time/max_over_timefor gauges, label matching and aggregation (sum by (service)). - Recording rules. Precompute expensive queries (
job:http_request_duration:p99) on an interval — dashboards and alerts query the recording, not the raw data. Essential at scale. - Alerting rules. Alert on symptoms (error rate, latency, saturation) with
for:durations to avoid flapping. Every alert needs a runbook link, a severity, and an owner — alerts without runbooks are just noise. - Alertmanager. Deduplication, grouping, silencing, and routing (PagerDuty/Opsgenie/Slack). Group related alerts; route by severity and team; inhibit cascading alerts (don't page for every pod when the node is down).
- Exporters. node_exporter (host metrics), blackbox_exporter (probing), kube-state-metrics (K8s objects), plus client libraries for app metrics. Instrument apps directly rather than parsing logs for metrics.
- Service discovery. Kubernetes, Consul, EC2, DNS-based discovery — targets come from the environment, not static configs. Relabeling shapes discovered targets into useful label sets.
- Retention and storage. Local TSDB with retention by time/size; 15 days default is often too short — size for your investigation needs. Remote write (Thanos, Cortex/Mimir, VictoriaMetrics) for long-term storage and HA.
- HA. Paired Prometheus servers (each scraping everything) + remote write, or Thanos/Cortex for global query. Single Prometheus is a single point of failure for alerting — plan accordingly.
- Cardinality. The #1 operational risk: unbounded label values → memory explosion → OOM. Monitor
prometheus_tsdb_head_series, set label limits, and review new metrics' cardinality before deploying. - Exemplars. Link metrics to traces (trace IDs on histograms) — connecting the RED metrics to the slow traces in one click.
Practical workflow
- Instrument RED metrics. Every service exposes request rate, error rate, and duration histogram with bounded labels (route templates, not raw paths).
REQUESTS = Counter("http_requests_total", "Requests", ["method", "route", "status"]) LATENCY = Histogram("http_request_duration_seconds", "Latency", ["method", "route"]) - Deploy exporters. node_exporter on hosts, kube-state-metrics in clusters, blackbox for external probing; app metrics via client libraries on
/metrics. - Write recording rules. Precompute SLIs and expensive aggregations; name them clearly (
job:metric:aggregation).groups: - name: slis interval: 1m rules: - record: job:http_requests:error_ratio expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m])) / sum by (job) (rate(http_requests_total[5m])) - Alert on symptoms. Error-ratio, p99 latency, saturation alerts with
for:and runbook annotations; severity tiers (page vs ticket).- alert: HighErrorRate expr: job:http_requests:error_ratio > 0.05 for: 5m labels: { severity: page } annotations: { runbook: https://wiki.example.com/runbooks/high-error-rate } - Configure Alertmanager. Grouping by alertname/cluster, routing by severity/team, inhibition for cascades, silences for maintenance — not for ignoring problems.
- Build dashboards from SLIs. RED per service, USE per resource, recording-rule-backed panels; dashboards answer "is it broken and where" in under a minute.
- Manage cardinality. Review label sets on new metrics; drop high-cardinality labels at scrape time via relabeling; alert on head-series growth.
- Plan retention and HA. Size retention for investigation needs; remote-write to long-term storage; paired servers or Thanos for alerting HA.
Common pitfalls
- Unbounded label cardinality — user IDs or request IDs as labels; OOM. Bound every label's value set.
- Alerting on causes — paging for CPU spikes instead of user-facing symptoms; alert on SLIs.
- No
for:duration — flapping alerts on transient blips; require sustained breach. - Alerts without runbooks — 3am pages with no guidance; every alert links a runbook.
iratein alerts — spiky and misleading;rateover sensible windows.- Raw paths as labels —
/users/123exploding cardinality; use route templates. - Summary instead of histogram — client-side quantiles can't aggregate; histograms with
histogram_quantile. - Single Prometheus for alerting — SPOF for pages; HA pair or remote-write + redundant alerting.
- No recording rules at scale — dashboards running expensive queries on every load; precompute.
- Pushgateway as default — for batch jobs only; services get scraped.
- Ignoring silences hygiene — permanent silences hiding real problems; expire and review them.
- Retention too short — 15 days when investigations need months; size deliberately with remote storage.
- Scraping everything at 15s — unnecessary load; tune intervals per target importance.
04 同仓库更多 Skills
web-scrape
Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open/fetch fails on JS-heavy pages, or when you need clean markdown/structured extraction from a site. Requires FIRECRAWL_API_KEY.
design-taste-frontend
tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML/CSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, "AI-made", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.
test-driven-development
Use when implementing any feature or bugfix, before writing implementation code
systematic-debugging
Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes
requesting-code-review
Use when completing tasks, implementing major features, or before merging to verify work meets requirements
brainstorming
You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).