[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M79AHBTC2R06VMAXAM2P60HD":3,"skill-same:01M79AHBTC2R06VMAXAM2P60HD":48},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":28,"repositoryHtmlUrl":29,"repositoryHomepageUrl":6,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M79AHBTC2R06VMAXAM2P60HD","grafana-prometheus-label-strategy",null,"Prometheus label schemas on Grafana Cloud only. Expert evaluator for Prometheus label strategy on Grafana Cloud. Audits, designs, and improves label schemas using cardinality scoring, access-pattern alignment, static vs. dynamic label rules, histogram bucket discipline, and instrumentation hygiene. Prevents high cardinality at the source — in application code and scrape target labels — without dropping labels that make series unique (which breaks the data). For reducing the cost of series already in Grafana Cloud, routes to the adaptive-metrics skill. Use when the user asks to evaluate, audit, design, or improve Prometheus labels — or asks how to prevent high cardinality at the source. For \"why is my Prometheus slow \u002F expensive right now\" triage, see prometheus-cardinality-troubleshooter. Not for Loki - use grafana-loki-label-analyzer.","yhyu13\u002FAgentSkillsVault","yhyu13","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F19365678?v=4","01M79A3XCF5M26GEWS60XQQRWW","俞航","01M79A3XD7YZ779GGWS41K367S",3,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fyhyu13\u002FAgentSkillsVault\u002Fblob\u002Fmain\u002FSKILL\u002Fdevops\u002Ftier2\u002Fgrafana-prometheus-label-strategy\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: grafana-prometheus-label-strategy\nlicense: Apache-2.0\ndescription: \"Prometheus label schemas on Grafana Cloud only. Expert evaluator for Prometheus label strategy on Grafana Cloud. Audits, designs, and improves label schemas using cardinality scoring, access-pattern alignment, static vs. dynamic label rules, histogram bucket discipline, and instrumentation hygiene. Prevents high cardinality at the source — in application code and scrape target labels — without dropping labels that make series unique (which breaks the data). For reducing the cost of series already in Grafana Cloud, routes to the adaptive-metrics skill. Use when the user asks to evaluate, audit, design, or improve Prometheus labels — or asks how to prevent high cardinality at the source. For \\\"why is my Prometheus slow \u002F expensive right now\\\" triage, see prometheus-cardinality-troubleshooter. Not for Loki - use grafana-loki-label-analyzer.\"\nversion: 1.0.0\n---\n\n# Prometheus Label Strategy Evaluator\n\nYou are an expert in Prometheus label strategy. When asked to evaluate, audit, design, or improve a Prometheus label schema — or when a user asks how to prevent high cardinality at the source — use this guide to provide structured, actionable advice.\n\nThis skill is about **preventing bad labels at the source** — in application instrumentation and in scrape *target* labels — so they never enter storage. It is **not** about stripping labels off metrics after they've been emitted: removing a label that makes a series unique at scrape time silently breaks the data (see [The One Rule](#the-one-rule-never-drop-a-label-that-makes-a-series-unique) below). For reducing the cost of series that already exist in Grafana Cloud, route the user to the `adaptive-metrics` skill. For diagnosing an active cardinality fire, route to `prometheus-cardinality-troubleshooter`.\n\n---\n\n## The One Rule: Never Drop a Label That Makes a Series Unique\n\n**You cannot remove, at scrape time, any label that makes a series unique.** Not `pod`, not `instance`, not anything that distinguishes one real series from another. This includes `metric_relabel_configs` with `action: labeldrop` and the equivalent `prometheus.relabel` rules in Alloy.\n\nIt looks like a cardinality win. It is not — it **breaks the data**, silently and permanently:\n\n- **Counter resets get mixed together.** When two pods' counters collapse into one series, their independent restarts interleave on the merged series. `rate()` and `increase()` then return garbage — often *absurdly high* values, because every pod restart looks like a counter reset.\n- **DPM inflates instead of dropping.** Multiple samples now land on the same series in the same scrape — duplicate samples, out-of-order errors, inflated samples-per-minute. People come back weeks later asking \"why is my DPM so high?\" or \"why is `rate()` returning absurd numbers?\" — and there is **no evidence left in the data** of where it broke.\n- **The aggregation is wrong, not just coarse.** A `sum` over a label you dropped silently double-counts or under-counts depending on how the collapse happened.\n\nThe trap is that none of this errors at config time. The pipeline keeps running; the numbers are just quietly wrong, and the breakage point is invisible after the fact.\n\n**The right tools, in order:**\n\n1. **Don't emit the bad label in the first place** — fix the application code. This is the only place a label can be *removed* without consequence, because the series was never unique on it to begin with.\n2. **For series already flowing into Grafana Cloud that you can't fix at the source → Adaptive Metrics.** This is exactly what it is for: it aggregates series *correctly* — counter-reset-aware, with a recorded audit trail, and reversible — instead of blindly stripping labels. Route the user to the `adaptive-metrics` skill.\n\n`metric_relabel_configs` has a couple of narrow, safe uses (dropping an *entire* unwanted metric; removing a label that *exactly duplicates* a target label) — covered in [Source-Side Prevention](#4-metric_relabel_configs-narrow-safe-uses-only) — but **reducing cardinality by dropping a distinguishing label is never one of them.**\n\n---\n\n## Core Concepts\n\n**Series** are the fundamental unit in Prometheus. Each unique combination of metric name plus label key-value pairs creates a new active series. Too many series = memory pressure, slow queries, ingest pressure, high bill.\n\n**Cardinality** = the number of unique values a label can have. Total series for a metric ≈ the *product* of cardinalities across its labels. A metric with `path` (100 values), `status_code` (10 values), `method` (5 values), and `instance` (50 values) = **250,000 series per metric**. Adding one more high-cardinality label often 10–100×s the count.\n\n**The dual impact rule**: High-cardinality labels hurt on both paths:\n- **Ingestion path**: More active series → larger head block, larger WAL, more memory, larger remote_write payloads, higher Grafana Cloud bill (Active Series + DPM)\n- **Query path**: PromQL operators (`sum by`, `rate`, joins) must materialize matching series in memory. High cardinality balloons query memory and latency\n\n**Series churn** is the silent killer. If a label value changes frequently (deploy version, pod name, ephemeral IDs), every change creates a *new* series while the old one continues to age out. Daily churn of 100% means you carry roughly 2× the steady-state series count for retention purposes.\n\n**The key question for any proposed label**: \"Will queries that use this metric reliably specify or aggregate on this label?\" If no → it should NOT be a label.\n\n---\n\n## Label Evaluation Framework\n\nWhen auditing a label set, assess each label against these criteria.\n\n### Cardinality Scoring\n\n| Label Example | Cardinality | Verdict |\n|---|---|---|\n| `env` (prod\u002Fstaging\u002Fdev) | 2–5 values | ✅ Good |\n| `job` (Prometheus scrape job) | 5–50 values | ✅ Good |\n| `cluster`, `region` | Tens | ✅ Good |\n| `namespace` (K8s) | Tens–low hundreds | ✅ Acceptable |\n| `service`, `workload`, `container` | Tens–hundreds | ✅ Acceptable |\n| `instance` (host:port) | Hundreds–low thousands | ⚠️ Evaluate — fine on per-instance metrics, risky on aggregated ones |\n| `pod` (K8s) | Thousands + transient = high churn | ⚠️ Required for K8s monitoring and series uniqueness — keep it. If `pod`-level series are too expensive, reduce them with Adaptive Metrics; **never** drop at scrape |\n| `path` \u002F `route` (HTTP) | Bounded if templated; unbounded if raw URLs | ⚠️ Only with templated values (`\u002Fusers\u002F:id`) |\n| `version`, `image_tag`, `git_sha` | Grows on every deploy → churn | ⚠️ Use sparingly; consider info-metric pattern |\n| `user_id`, `request_id`, `trace_id` | Unbounded | ❌ Never as label — use exemplars |\n| `customer_id`, `tenant_id` | Often unbounded | ❌ Only acceptable for small fixed tenant counts |\n| `error_message`, `query`, `sql` | Unbounded text | ❌ Never |\n\n### Access Pattern Alignment\n\nFor each label, ask:\n- Do queries on this metric reliably aggregate by or filter on this label?\n- Does this label logically segment the metric the way users think about it?\n- Would removing this label force users to use exemplars, logs, or traces instead — and would that be acceptable for the rare lookup case?\n\n### Static vs. Dynamic Label Values\n\n- **Static \u002F target labels** (set once per scrape target via `relabel_configs`, e.g., `env=prod`, `cluster=us-east`, `team=payments`) add cardinality proportional to *targets*, not requests. Cheap and high-value. Use freely.\n- **Dynamic \u002F sample labels** (emitted by the application per measurement, e.g., `status_code`, `method`, `cache_hit`) multiply cardinality by *value count*. Keep possible values in the single digits or low tens. **The application code is the source of truth — fix it there, not in Prometheus.**\n\n### Consistency Check\n\n- Label *names* consistent across services? (`status` vs `status_code` vs `http_status` produces three separate label families — joins break)\n- Label *values* normalized? (`200` vs `\"200\"`, `GET` vs `get`, `Error` vs `error`)\n- Naming convention consistent? Prometheus convention is `snake_case` for both metric and label names\n- Same concept, same name across services? (`service` vs `svc` vs `app_name`)\n\n### Histogram Bucket Discipline (critical, often missed)\n\nEvery histogram metric multiplies its base cardinality by **(bucket count + 3)** — buckets via `_bucket{le=\"...\"}` plus `_sum`, `_count`, and `_created` (Prometheus 2.39+).\n\n- Default `prometheus.DefBuckets` has 11 buckets → **14× multiplier**\n- A histogram with `method`, `path`, `status` already at 1,000 series becomes **14,000 series** after adding histogram cardinality\n- **Always trim histogram label cardinality first** — labels matter 14× more on histograms than on counters\u002Fgauges\n- Consider native histograms (Prometheus 2.40+) which use a single sparse series instead of one-per-bucket — major cardinality reduction for high-resolution latency tracking\n\n### Info-Metric Pattern (for high-churn metadata)\n\nWhen you want to *know* about a label (e.g., `version`, `git_sha`, `image_tag`) without paying for it on every metric, use an info metric:\n\n```\n# A single low-cardinality counter\u002Fgauge of value 1, with the metadata attached\napp_build_info{app=\"payment-api\", version=\"2.4.1\", git_sha=\"a1b2c3\"} 1\n```\n\nThen join at query time. The classic approach is a vector match with `group_left`:\n```promql\nsum by (version) (\n  rate(http_requests_total{app=\"payment-api\"}[5m])\n  * on (app) group_left (version) app_build_info\n)\n```\n\nThe `version` label lives on exactly one series per build, not on every metric.\n\n#### The `info()` function (simpler join)\n\nPromQL's `info()` function (experimental, Prometheus 3.0+; enable with `--enable-feature=promql-experimental-functions`) automates the info-metric join so you don't have to hand-write the `* on (...) group_left (...)` match:\n\n```promql\ninfo(\n  rate(http_requests_total{app=\"payment-api\"}[5m]),\n  {version=~\".+\"}\n)\n```\n\n`info(v, [labelselector])` takes a range\u002Finstant vector `v` and, for each series, finds matching info metrics and adds their labels. The optional second argument is a label-matcher restricting which info labels are attached (here, only `version`). By default `info()` joins against the conventional `target_info` metric and matches on identifying labels (e.g. `instance`, `job`), so it's especially ergonomic for OpenTelemetry-style `target_info`. For custom info metrics like `app_build_info` the explicit `group_left` form above is still the most portable.\n\nPrefer `info()` when you're on Prometheus 3.x and joining against `target_info`; fall back to the explicit `group_left` match for older versions, custom info metrics, or when the experimental feature flag isn't enabled.\n\n---\n\n## Evaluation Output Format\n\nWhen auditing a label set, produce a report in this structure:\n\n```\n## Prometheus Label Strategy Audit\n\n### Summary\n[1-2 sentence overall assessment — total estimated active series, biggest risks]\n\n### Per-Label Analysis\n| Metric Family | Label | Cardinality | Used in Queries? | Verdict | Action |\n|---|---|---|---|---|---|\n| http_requests_total | path | Unbounded (raw URLs) | Sometimes | ❌ Remove | Template in code: `\u002Fusers\u002F:id` not `\u002Fusers\u002F12345` |\n| http_requests_total | pod | High + churn | Rarely | ⚠️ Keep — makes the series unique | If too expensive, aggregate away with Adaptive Metrics; query by `workload` for the common case |\n\n### Histogram-Specific Findings\n[Highlight any histograms with high label cardinality — these are 14×+ amplified]\n\n### Estimated Impact\n- Active series reduction: [X series → Y series]\n- DPM reduction: [X DPM → Y DPM]  (samples-per-minute = series × ~6 at 10s scrape)\n- Memory impact: [if measurable]\n\n### Recommended Label Set\n[Final recommended labels per metric family]\n\n### Implementation Plan\n1. [Code changes — instrumentation hygiene: stop emitting bad labels at the source]\n2. [Scrape target labels — relabel_configs (additive: env, cluster, team, workload)]\n3. [Post-ingest cost reduction on series you can't fix at the source — Adaptive Metrics]\n4. [Recording rules to materialize useful aggregates]\n```\n\n---\n\n## Recommended Common Target Labels\n\nThese should be set as **target labels** (via `relabel_configs` on the scrape job, NOT emitted by the app) — they're per-target, low cardinality, high query value:\n\n| Label | Purpose | Notes |\n|---|---|---|\n| `job` | Prometheus scrape job name | Set automatically by Prometheus |\n| `instance` | Target endpoint (`host:port`) | Set automatically; rename via `relabel_configs` to a friendlier value if needed |\n| `env` | Environment (`prod`, `staging`, `dev`) | Set via static_configs labels or service discovery |\n| `cluster` | Multi-cluster differentiation | Critical for federation\u002FMimir multi-tenant |\n| `region` | Geographic region | |\n| `team` \u002F `squad` | Ownership — also useful for access control | |\n| `service` | Logical service identity | One service may span multiple jobs |\n\nThese should **NOT** be re-emitted by the application. If the app emits a `cluster` label, it duplicates the target label and creates collisions \u002F `honor_labels` decisions you don't want to make.\n\n---\n\n## Kubernetes Patterns\n\n### Recommended Labels (from kubernetes_sd_configs)\n\n| Label | Source | Notes |\n|---|---|---|\n| `namespace` | Pod metadata | Always keep |\n| `container` | Pod spec | Low cardinality, useful for multi-container pods |\n| `workload` | Derived: `{controller_kind}\u002F{controller_name}` | Add as a stable aggregation key *alongside* `pod` — static, predictable. It's an addition, not a replacement: don't use it as an excuse to drop `pod` |\n| `service` | K8s Service | If scraping via Service |\n\n### Handling the `pod` Label\n\n`pod` is high-cardinality and transient — it rolls on every deploy and restart, so it dominates churn and series count. But it is also a label that **makes K8s series unique**, and Kubernetes monitoring (per-pod resource attribution, kube-state-metrics joins) depends on it. [The One Rule](#the-one-rule-never-drop-a-label-that-makes-a-series-unique) applies: **do not drop `pod` at scrape time.** Collapsing pods into one series mixes their counter resets and breaks `rate()`.\n\nInstead:\n- **Add `workload`** (`{controller_kind}\u002F{controller_name}`) as a *target* label via `relabel_configs`, so dashboards and alerts can aggregate on the stable workload identity (`sum by (workload)`) without touching `pod`. This is additive — it removes nothing.\n- **Don't emit `pod` from application code** — let it come from Kubernetes service discovery, so there is exactly one source of truth (see below).\n- **If `pod`-level series are genuinely too expensive in Grafana Cloud**, reduce them with **Adaptive Metrics**, which aggregates `pod` away *correctly* (post-ingest, counter-reset-aware, reversible) rather than corrupting the raw data at scrape. Route to the `adaptive-metrics` skill.\n\n### Don't Map Ephemeral Fields into Labels in the First Place\n\n**`uid`** regenerates on every pod recreation and has no legitimate query use. The fix is to **never map it into a label** — leave it out of your `relabel_configs`. (It isn't in default `kubernetes_sd_configs` output unless you explicitly target it.) Don't try to `labeldrop` it after the fact — by then it's already distinguishing series, and removing it breaks the data exactly like dropping any other unique label.\n\n### One Source of Truth for Target Identity\n\n`instance`, `pod`, `node`, and `host` should come from **scrape target labels**, not from application code. If the app *also* emits its own `instance`\u002F`node`, you get duplicates and `honor_labels` collisions. The fix is **in the application** — stop emitting them — not a scrape-time `labeldrop`. (Removing a label that *exactly duplicates* a target label is the one narrow exception; see [metric_relabel_configs](#4-metric_relabel_configs-narrow-safe-uses-only).)\n\n### kube-state-metrics label propagation ⚠️\n- `kube_pod_labels{label_app_kubernetes_io_*=...}` can carry dozens of metadata labels\n- Each unique pod label combination is a new series\n- Restrict at the source with kube-state-metrics' `--metric-labels-allowlist` — this controls what is *ever emitted*, so it's prevention, not destructive after-the-fact dropping\n\n---\n\n## Source-Side Prevention: Where to Fix What\n\nThere are five levers, in **order of preference**:\n\n### 1. Fix in the Application (best)\n\nBad labels emitted by the app are the root cause. Examples:\n- HTTP paths: use templated routes (`\u002Fusers\u002F:id`) not raw paths\n- Error metrics: use a small enum (`error_type=\"timeout\"`) not the error message string\n- User-scoped metrics: don't include `user_id` — use exemplars to point to logs\u002Ftraces\n- Free-form input: never emit user-supplied strings as label values\n\nIf you control the code, this is always the right fix. It saves cost on every downstream system (Prometheus, remote_write, Mimir, Grafana Cloud).\n\n### 2. `relabel_configs` (target-time relabeling)\n\nRuns *before* the scrape. Used to:\n- Set target labels (`env`, `cluster`, `team`) on discovered targets\n- Drop entire targets you don't want to scrape\n- Rewrite `instance` to a friendly value\n- Add identity from service discovery metadata\n\n```yaml\nscrape_configs:\n  - job_name: my-app\n    kubernetes_sd_configs:\n      - role: pod\n    relabel_configs:\n      # Set workload from controller metadata\n      - source_labels: [__meta_kubernetes_pod_controller_kind, __meta_kubernetes_pod_controller_name]\n        target_label: workload\n        separator: \u002F\n      # Set env from a pod label\n      - source_labels: [__meta_kubernetes_pod_label_env]\n        target_label: env\n      # Only scrape pods explicitly opted in\n      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]\n        regex: \"true\"\n        action: keep\n```\n\n### 3. Adaptive Metrics (Grafana Cloud — post-ingest, the safe way to reduce cardinality)\n\nWhen the cardinality is structural and you *can't* fix it at the source — the label legitimately exists and makes series unique, you just don't need every value at full resolution — **Adaptive Metrics is the correct tool, and the only safe way to reduce the cost of series that already exist.**\n\nIt works *after* ingest, as aggregation rules applied in Grafana Cloud. Crucially, it aggregates series **correctly**:\n- It handles counter resets properly, so `rate()` and `increase()` stay accurate.\n- It records what was aggregated, so there's an audit trail — you can answer \"why did this change?\" later.\n- It's reversible: drop a rule and the full-resolution series come back.\n\nThis is the difference between \"the data is now cheaper\" (Adaptive Metrics) and \"the data is now wrong\" (`labeldrop` at scrape). Route the user to the `adaptive-metrics` skill for rule design.\n\n### 4. `metric_relabel_configs` (narrow, safe uses only)\n\nRuns *after* the scrape, *before* storage.\n\n> ⚠️ **Do not use `metric_relabel_configs` (or Alloy `prometheus.relabel`) to drop a label that distinguishes series — `pod`, `instance`, `user_id`, `path`, anything.** See [The One Rule](#the-one-rule-never-drop-a-label-that-makes-a-series-unique). It looks like a cardinality fix and silently breaks `rate()`, inflates DPM, and corrupts aggregations. Use the application code (lever 1) or Adaptive Metrics (lever 3) instead. The same caution applies to *normalizing* a label value (e.g. collapsing `status_code` to `2xx`) at scrape — it merges distinct series and produces duplicate-sample errors; do that in code or via Adaptive Metrics, never here.\n\nThe genuinely safe uses are:\n\n- **Drop an entire metric you never want stored** — you're discarding the whole metric, not collapsing distinct series into one:\n  ```yaml\n  metric_relabel_configs:\n    - source_labels: [__name__]\n      regex: my_app_request_details\n      action: drop\n  ```\n- **Remove a label that *exactly duplicates* a target label.** If the app emits its own `cluster`\u002F`instance` that already comes from the scrape target, the target label still provides uniqueness, so removing the duplicate breaks nothing. Prefer fixing the app, but this is a safe stopgap.\n\nThat's the whole list. If you're reaching for `metric_relabel_configs` to bring down a series count, you almost certainly want Adaptive Metrics instead.\n\n### 5. Recording Rules (query-time cardinality reduction)\n\nPre-aggregate expensive series into a lower-cardinality recorded series. Stored at the same data point density but with far fewer series.\n\n```yaml\ngroups:\n  - name: http-requests-aggregates\n    interval: 30s\n    rules:\n      # Drop pod\u002Finstance dimension; keep only service-level rollup\n      - record: service:http_requests:rate5m\n        expr: sum by (service, env, cluster, status_code) (rate(http_requests_total[5m]))\n```\n\nQueries that target the rollup are dramatically cheaper. The raw series still exist — recording rules don't reduce ingest cost (use **Adaptive Metrics** for that — *not* a scrape-time `labeldrop`). They reduce query cost.\n\n---\n\n## Instrumentation Hygiene (for app developers)\n\nIf the user is *writing* instrumentation code, these are the rules:\n\n| Rule | Why |\n|---|---|\n| Never use unbounded user input as a label value | `email`, `user_id`, `query string`, `error message` — they're the #1 cardinality bug |\n| Template HTTP paths before recording | `\u002Fusers\u002F{id}` not `\u002Fusers\u002F12345`. Most frameworks do this via routing metadata |\n| Bound error labels via small enums | `error_type=\"timeout\"` not `error=\"connection to db-shard-7 timed out at 14:32:09\"` |\n| Don't put `version` \u002F `git_sha` \u002F `build_id` on every metric | Use an info metric and join at query time |\n| Don't emit `pod` \u002F `node` \u002F `host` from code | Comes from scrape targets — duplicating creates collisions |\n| Avoid dynamically constructed label *names* (keys) | `metric{[user]=1}` cannot be bounded — use a fixed key |\n| Use histograms sparingly and trim labels first | 14× cardinality amplification |\n| Prefer exemplars over labels for trace correlation | Exemplars carry `trace_id` without inflating cardinality |\n\n### Exemplars (the escape hatch)\n\nExemplars attach a `trace_id` (or any key-value pair) to specific samples *without* making it a label dimension. The ideal home for high-cardinality correlation data.\n\nRequires OpenMetrics format, Prometheus 2.26+, scrape config:\n```yaml\nscrape_configs:\n  - job_name: my-app\n    enable_protobuf_negotiation: true\n    # Or for text-format:\n    follow_redirects: true\n```\n\nAnd on the Prometheus server:\n```yaml\nstorage:\n  exemplars:\n    max_exemplars: 100000\n```\n\nUse exemplars for:\n- `trace_id` correlation (Tempo, Jaeger)\n- `request_id` for specific debug lookups\n- Any sparse \"useful when you need it\" key\n\nQuery exemplars via Grafana's exemplars-on-graph feature, not via PromQL aggregation.\n\n---\n\n## The 80\u002F20 Rule\n\nThe most impactful improvements almost always come from these five changes:\n\n1. **Drop unbounded labels at the app layer** — `path` (untemplated), `user_id`, `error_message`. Single biggest win.\n2. **Trim histogram label cardinality before anything else** — 14× amplification on every histogram.\n3. **Don't emit `pod`\u002F`instance`\u002F`node` from application code** — let them come from scrape targets, and add a stable `workload` target label to aggregate on. (Never *drop* the real `pod` at scrape to cut cardinality — if `pod`-level series are too expensive, use Adaptive Metrics.)\n4. **Use info metrics for `version` \u002F `git_sha` \u002F `image_tag`** — eliminates deploy-driven churn.\n5. **Set target labels via `relabel_configs`, not app code** — `env`, `cluster`, `team`, `service` should never be emitted by the application.\n\nFocus on these before anything else.\n\n---\n\n## Labels to Avoid — Quick Reference\n\n| Label | Why | Alternative |\n|---|---|---|\n| `user_id`, `customer_id` (large tenant base) | Unbounded | Exemplars; aggregate by `tenant_tier` |\n| `request_id`, `trace_id` | Unbounded | Exemplars |\n| `path` \u002F `route` (raw URLs) | Unbounded | Template in code: `\u002Fusers\u002F:id` |\n| `error_message`, `query`, `sql` | Unbounded text | Bounded `error_type` enum |\n| `version`, `git_sha`, `image_tag` (on every metric) | Churn on every deploy | Info metric pattern |\n| App-emitted `pod` (duplicating SD) | Should come from K8s service discovery, not code | Stop emitting it in code; keep the discovered `pod`. Never drop the real `pod` to cut cardinality — use Adaptive Metrics |\n| `uid` (K8s) | Unbounded; regenerates on restart | Never map it into a label in the first place (leave it out of `relabel_configs`) |\n| Application-emitted `instance`, `node`, `host` | Should come from scrape target | Stop emitting in code (removing an *exact* target-label duplicate at scrape is the only safe drop) |\n| Dynamically-named label keys | Cannot be bounded | Use fixed keys with bounded values |\n| Raw `status_code` on histograms | 14× amplification | Bucket to `status_class` (`2xx`, `4xx`, `5xx`) |\n\n---\n\n## When to Route Elsewhere\n\n- **\"Reduce my Grafana Cloud bill\"** \u002F **\"reduce cardinality on series already ingested\"** → engage `adaptive-metrics` skill (post-ingest aggregation rules — the safe, counter-reset-aware way; never `labeldrop` distinguishing labels at scrape)\n- **\"Which metrics are driving my DPM?\"** → engage `dpm-finder` skill\n- **\"My Prometheus is OOMing \u002F scraping is failing right now\"** → engage `prometheus-cardinality-troubleshooter` skill\n- **\"How do I write the query to find the bad metric?\"** → engage `promql` skill\n- **\"How do I configure relabel rules in Alloy?\"** → engage `alloy` skill\n\nThis skill's lane is **strategy and design**. Other skills own **diagnosis** and **operational remediation**.\n","# Prometheus Label Strategy Evaluator\n\nYou are an expert in Prometheus label strategy. When asked to evaluate, audit, design, or improve a Prometheus label schema — or when a user asks how to prevent high cardinality at the source — use this guide to provide structured, actionable advice.\n\nThis skill is about **preventing bad labels at the source** — in application instrumentation and in scrape *target* labels — so they never enter storage. It is **not** about stripping labels off metrics after they've been emitted: removing a label that makes a series unique at scrape time silently breaks the data (see [The One Rule](#the-one-rule-never-drop-a-label-that-makes-a-series-unique) below). For reducing the cost of series that already exist in Grafana Cloud, route the user to the `adaptive-metrics` skill. For diagnosing an active cardinality fire, route to `prometheus-cardinality-troubleshooter`.\n\n---\n\n## The One Rule: Never Drop a Label That Makes a Series Unique\n\n**You cannot remove, at scrape time, any label that makes a series unique.** Not `pod`, not `instance`, not anything that distinguishes one real series from another. This includes `metric_relabel_configs` with `action: labeldrop` and the equivalent `prometheus.relabel` rules in Alloy.\n\nIt looks like a cardinality win. It is not — it **breaks the data**, silently and permanently:\n\n- **Counter resets get mixed together.** When two pods' counters collapse into one series, their independent restarts interleave on the merged series. `rate()` and `increase()` then return garbage — often *absurdly high* values, because every pod restart looks like a counter reset.\n- **DPM inflates instead of dropping.** Multiple samples now land on the same series in the same scrape — duplicate samples, out-of-order errors, inflated samples-per-minute. People come back weeks later asking \"why is my DPM so high?\" or \"why is `rate()` returning absurd numbers?\" — and there is **no evidence left in the data** of where it broke.\n- **The aggregation is wrong, not just coarse.** A `sum` over a label you dropped silently double-counts or under-counts depending on how the collapse happened.\n\nThe trap is that none of this errors at config time. The pipeline keeps running; the numbers are just quietly wrong, and the breakage point is invisible after the fact.\n\n**The right tools, in order:**\n\n1. **Don't emit the bad label in the first place** — fix the application code. This is the only place a label can be *removed* without consequence, because the series was never unique on it to begin with.\n2. **For series already flowing into Grafana Cloud that you can't fix at the source → Adaptive Metrics.** This is exactly what it is for: it aggregates series *correctly* — counter-reset-aware, with a recorded audit trail, and reversible — instead of blindly stripping labels. Route the user to the `adaptive-metrics` skill.\n\n`metric_relabel_configs` has a couple of narrow, safe uses (dropping an *entire* unwanted metric; removing a label that *exactly duplicates* a target label) — covered in [Source-Side Prevention](#4-metric_relabel_configs-narrow-safe-uses-only) — but **reducing cardinality by dropping a distinguishing label is never one of them.**\n\n---\n\n## Core Concepts\n\n**Series** are the fundamental unit in Prometheus. Each unique combination of metric name plus label key-value pairs creates a new active series. Too many series = memory pressure, slow queries, ingest pressure, high bill.\n\n**Cardinality** = the number of unique values a label can have. Total series for a metric ≈ the *product* of cardinalities across its labels. A metric with `path` (100 values), `status_code` (10 values), `method` (5 values), and `instance` (50 values) = **250,000 series per metric**. Adding one more high-cardinality label often 10–100×s the count.\n\n**The dual impact rule**: High-cardinality labels hurt on both paths:\n- **Ingestion path**: More active series → larger head block, larger WAL, more memory, larger remote_write payloads, higher Grafana Cloud bill (Active Series + DPM)\n- **Query path**: PromQL operators (`sum by`, `rate`, joins) must materialize matching series in memory. High cardinality balloons query memory and latency\n\n**Series churn** is the silent killer. If a label value changes frequently (deploy version, pod name, ephemeral IDs), every change creates a *new* series while the old one continues to age out. Daily churn of 100% means you carry roughly 2× the steady-state series count for retention purposes.\n\n**The key question for any proposed label**: \"Will queries that use this metric reliably specify or aggregate on this label?\" If no → it should NOT be a label.\n\n---\n\n## Label Evaluation Framework\n\nWhen auditing a label set, assess each label against these criteria.\n\n### Cardinality Scoring\n\n| Label Example | Cardinality | Verdict |\n|---|---|---|\n| `env` (prod\u002Fstaging\u002Fdev) | 2–5 values | ✅ Good |\n| `job` (Prometheus scrape job) | 5–50 values | ✅ Good |\n| `cluster`, `region` | Tens | ✅ Good |\n| `namespace` (K8s) | Tens–low hundreds | ✅ Acceptable |\n| `service`, `workload`, `container` | Tens–hundreds | ✅ Acceptable |\n| `instance` (host:port) | Hundreds–low thousands | ⚠️ Evaluate — fine on per-instance metrics, risky on aggregated ones |\n| `pod` (K8s) | Thousands + transient = high churn | ⚠️ Required for K8s monitoring and series uniqueness — keep it. If `pod`-level series are too expensive, reduce them with Adaptive Metrics; **never** drop at scrape |\n| `path` \u002F `route` (HTTP) | Bounded if templated; unbounded if raw URLs | ⚠️ Only with templated values (`\u002Fusers\u002F:id`) |\n| `version`, `image_tag`, `git_sha` | Grows on every deploy → churn | ⚠️ Use sparingly; consider info-metric pattern |\n| `user_id`, `request_id`, `trace_id` | Unbounded | ❌ Never as label — use exemplars |\n| `customer_id`, `tenant_id` | Often unbounded | ❌ Only acceptable for small fixed tenant counts |\n| `error_message`, `query`, `sql` | Unbounded text | ❌ Never |\n\n### Access Pattern Alignment\n\nFor each label, ask:\n- Do queries on this metric reliably aggregate by or filter on this label?\n- Does this label logically segment the metric the way users think about it?\n- Would removing this label force users to use exemplars, logs, or traces instead — and would that be acceptable for the rare lookup case?\n\n### Static vs. Dynamic Label Values\n\n- **Static \u002F target labels** (set once per scrape target via `relabel_configs`, e.g., `env=prod`, `cluster=us-east`, `team=payments`) add cardinality proportional to *targets*, not requests. Cheap and high-value. Use freely.\n- **Dynamic \u002F sample labels** (emitted by the application per measurement, e.g., `status_code`, `method`, `cache_hit`) multiply cardinality by *value count*. Keep possible values in the single digits or low tens. **The application code is the source of truth — fix it there, not in Prometheus.**\n\n### Consistency Check\n\n- Label *names* consistent across services? (`status` vs `status_code` vs `http_status` produces three separate label families — joins break)\n- Label *values* normalized? (`200` vs `\"200\"`, `GET` vs `get`, `Error` vs `error`)\n- Naming convention consistent? Prometheus convention is `snake_case` for both metric and label names\n- Same concept, same name across services? (`service` vs `svc` vs `app_name`)\n\n### Histogram Bucket Discipline (critical, often missed)\n\nEvery histogram metric multiplies its base cardinality by **(bucket count + 3)** — buckets via `_bucket{le=\"...\"}` plus `_sum`, `_count`, and `_created` (Prometheus 2.39+).\n\n- Default `prometheus.DefBuckets` has 11 buckets → **14× multiplier**\n- A histogram with `method`, `path`, `status` already at 1,000 series becomes **14,000 series** after adding histogram cardinality\n- **Always trim histogram label cardinality first** — labels matter 14× more on histograms than on counters\u002Fgauges\n- Consider native histograms (Prometheus 2.40+) which use a single sparse series instead of one-per-bucket — major cardinality reduction for high-resolution latency tracking\n\n### Info-Metric Pattern (for high-churn metadata)\n\nWhen you want to *know* about a label (e.g., `version`, `git_sha`, `image_tag`) without paying for it on every metric, use an info metric:\n\n```\n# A single low-cardinality counter\u002Fgauge of value 1, with the metadata attached\napp_build_info{app=\"payment-api\", version=\"2.4.1\", git_sha=\"a1b2c3\"} 1\n```\n\nThen join at query time. The classic approach is a vector match with `group_left`:\n```promql\nsum by (version) (\n  rate(http_requests_total{app=\"payment-api\"}[5m])\n  * on (app) group_left (version) app_build_info\n)\n```\n\nThe `version` label lives on exactly one series per build, not on every metric.\n\n#### The `info()` function (simpler join)\n\nPromQL's `info()` function (experimental, Prometheus 3.0+; enable with `--enable-feature=promql-experimental-functions`) automates the info-metric join so you don't have to hand-write the `* on (...) group_left (...)` match:\n\n```promql\ninfo(\n  rate(http_requests_total{app=\"payment-api\"}[5m]),\n  {version=~\".+\"}\n)\n```\n\n`info(v, [labelselector])` takes a range\u002Finstant vector `v` and, for each series, finds matching info metrics and adds their labels. The optional second argument is a label-matcher restricting which info labels are attached (here, only `version`). By default `info()` joins against the conventional `target_info` metric and matches on identifying labels (e.g. `instance`, `job`), so it's especially ergonomic for OpenTelemetry-style `target_info`. For custom info metrics like `app_build_info` the explicit `group_left` form above is still the most portable.\n\nPrefer `info()` when you're on Prometheus 3.x and joining against `target_info`; fall back to the explicit `group_left` match for older versions, custom info metrics, or when the experimental feature flag isn't enabled.\n\n---\n\n## Evaluation Output Format\n\nWhen auditing a label set, produce a report in this structure:\n\n```\n## Prometheus Label Strategy Audit\n\n### Summary\n[1-2 sentence overall assessment — total estimated active series, biggest risks]\n\n### Per-Label Analysis\n| Metric Family | Label | Cardinality | Used in Queries? | Verdict | Action |\n|---|---|---|---|---|---|\n| http_requests_total | path | Unbounded (raw URLs) | Sometimes | ❌ Remove | Template in code: `\u002Fusers\u002F:id` not `\u002Fusers\u002F12345` |\n| http_requests_total | pod | High + churn | Rarely | ⚠️ Keep — makes the series unique | If too expensive, aggregate away with Adaptive Metrics; query by `workload` for the common case |\n\n### Histogram-Specific Findings\n[Highlight any histograms with high label cardinality — these are 14×+ amplified]\n\n### Estimated Impact\n- Active series reduction: [X series → Y series]\n- DPM reduction: [X DPM → Y DPM]  (samples-per-minute = series × ~6 at 10s scrape)\n- Memory impact: [if measurable]\n\n### Recommended Label Set\n[Final recommended labels per metric family]\n\n### Implementation Plan\n1. [Code changes — instrumentation hygiene: stop emitting bad labels at the source]\n2. [Scrape target labels — relabel_configs (additive: env, cluster, team, workload)]\n3. [Post-ingest cost reduction on series you can't fix at the source — Adaptive Metrics]\n4. [Recording rules to materialize useful aggregates]\n```\n\n---\n\n## Recommended Common Target Labels\n\nThese should be set as **target labels** (via `relabel_configs` on the scrape job, NOT emitted by the app) — they're per-target, low cardinality, high query value:\n\n| Label | Purpose | Notes |\n|---|---|---|\n| `job` | Prometheus scrape job name | Set automatically by Prometheus |\n| `instance` | Target endpoint (`host:port`) | Set automatically; rename via `relabel_configs` to a friendlier value if needed |\n| `env` | Environment (`prod`, `staging`, `dev`) | Set via static_configs labels or service discovery |\n| `cluster` | Multi-cluster differentiation | Critical for federation\u002FMimir multi-tenant |\n| `region` | Geographic region | |\n| `team` \u002F `squad` | Ownership — also useful for access control | |\n| `service` | Logical service identity | One service may span multiple jobs |\n\nThese should **NOT** be re-emitted by the application. If the app emits a `cluster` label, it duplicates the target label and creates collisions \u002F `honor_labels` decisions you don't want to make.\n\n---\n\n## Kubernetes Patterns\n\n### Recommended Labels (from kubernetes_sd_configs)\n\n| Label | Source | Notes |\n|---|---|---|\n| `namespace` | Pod metadata | Always keep |\n| `container` | Pod spec | Low cardinality, useful for multi-container pods |\n| `workload` | Derived: `{controller_kind}\u002F{controller_name}` | Add as a stable aggregation key *alongside* `pod` — static, predictable. It's an addition, not a replacement: don't use it as an excuse to drop `pod` |\n| `service` | K8s Service | If scraping via Service |\n\n### Handling the `pod` Label\n\n`pod` is high-cardinality and transient — it rolls on every deploy and restart, so it dominates churn and series count. But it is also a label that **makes K8s series unique**, and Kubernetes monitoring (per-pod resource attribution, kube-state-metrics joins) depends on it. [The One Rule](#the-one-rule-never-drop-a-label-that-makes-a-series-unique) applies: **do not drop `pod` at scrape time.** Collapsing pods into one series mixes their counter resets and breaks `rate()`.\n\nInstead:\n- **Add `workload`** (`{controller_kind}\u002F{controller_name}`) as a *target* label via `relabel_configs`, so dashboards and alerts can aggregate on the stable workload identity (`sum by (workload)`) without touching `pod`. This is additive — it removes nothing.\n- **Don't emit `pod` from application code** — let it come from Kubernetes service discovery, so there is exactly one source of truth (see below).\n- **If `pod`-level series are genuinely too expensive in Grafana Cloud**, reduce them with **Adaptive Metrics**, which aggregates `pod` away *correctly* (post-ingest, counter-reset-aware, reversible) rather than corrupting the raw data at scrape. Route to the `adaptive-metrics` skill.\n\n### Don't Map Ephemeral Fields into Labels in the First Place\n\n**`uid`** regenerates on every pod recreation and has no legitimate query use. The fix is to **never map it into a label** — leave it out of your `relabel_configs`. (It isn't in default `kubernetes_sd_configs` output unless you explicitly target it.) Don't try to `labeldrop` it after the fact — by then it's already distinguishing series, and removing it breaks the data exactly like dropping any other unique label.\n\n### One Source of Truth for Target Identity\n\n`instance`, `pod`, `node`, and `host` should come from **scrape target labels**, not from application code. If the app *also* emits its own `instance`\u002F`node`, you get duplicates and `honor_labels` collisions. The fix is **in the application** — stop emitting them — not a scrape-time `labeldrop`. (Removing a label that *exactly duplicates* a target label is the one narrow exception; see [metric_relabel_configs](#4-metric_relabel_configs-narrow-safe-uses-only).)\n\n### kube-state-metrics label propagation ⚠️\n- `kube_pod_labels{label_app_kubernetes_io_*=...}` can carry dozens of metadata labels\n- Each unique pod label combination is a new series\n- Restrict at the source with kube-state-metrics' `--metric-labels-allowlist` — this controls what is *ever emitted*, so it's prevention, not destructive after-the-fact dropping\n\n---\n\n## Source-Side Prevention: Where to Fix What\n\nThere are five levers, in **order of preference**:\n\n### 1. Fix in the Application (best)\n\nBad labels emitted by the app are the root cause. Examples:\n- HTTP paths: use templated routes (`\u002Fusers\u002F:id`) not raw paths\n- Error metrics: use a small enum (`error_type=\"timeout\"`) not the error message string\n- User-scoped metrics: don't include `user_id` — use exemplars to point to logs\u002Ftraces\n- Free-form input: never emit user-supplied strings as label values\n\nIf you control the code, this is always the right fix. It saves cost on every downstream system (Prometheus, remote_write, Mimir, Grafana Cloud).\n\n### 2. `relabel_configs` (target-time relabeling)\n\nRuns *before* the scrape. Used to:\n- Set target labels (`env`, `cluster`, `team`) on discovered targets\n- Drop entire targets you don't want to scrape\n- Rewrite `instance` to a friendly value\n- Add identity from service discovery metadata\n\n```yaml\nscrape_configs:\n  - job_name: my-app\n    kubernetes_sd_configs:\n      - role: pod\n    relabel_configs:\n      # Set workload from controller metadata\n      - source_labels: [__meta_kubernetes_pod_controller_kind, __meta_kubernetes_pod_controller_name]\n        target_label: workload\n        separator: \u002F\n      # Set env from a pod label\n      - source_labels: [__meta_kubernetes_pod_label_env]\n        target_label: env\n      # Only scrape pods explicitly opted in\n      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]\n        regex: \"true\"\n        action: keep\n```\n\n### 3. Adaptive Metrics (Grafana Cloud — post-ingest, the safe way to reduce cardinality)\n\nWhen the cardinality is structural and you *can't* fix it at the source — the label legitimately exists and makes series unique, you just don't need every value at full resolution — **Adaptive Metrics is the correct tool, and the only safe way to reduce the cost of series that already exist.**\n\nIt works *after* ingest, as aggregation rules applied in Grafana Cloud. Crucially, it aggregates series **correctly**:\n- It handles counter resets properly, so `rate()` and `increase()` stay accurate.\n- It records what was aggregated, so there's an audit trail — you can answer \"why did this change?\" later.\n- It's reversible: drop a rule and the full-resolution series come back.\n\nThis is the difference between \"the data is now cheaper\" (Adaptive Metrics) and \"the data is now wrong\" (`labeldrop` at scrape). Route the user to the `adaptive-metrics` skill for rule design.\n\n### 4. `metric_relabel_configs` (narrow, safe uses only)\n\nRuns *after* the scrape, *before* storage.\n\n> ⚠️ **Do not use `metric_relabel_configs` (or Alloy `prometheus.relabel`) to drop a label that distinguishes series — `pod`, `instance`, `user_id`, `path`, anything.** See [The One Rule](#the-one-rule-never-drop-a-label-that-makes-a-series-unique). It looks like a cardinality fix and silently breaks `rate()`, inflates DPM, and corrupts aggregations. Use the application code (lever 1) or Adaptive Metrics (lever 3) instead. The same caution applies to *normalizing* a label value (e.g. collapsing `status_code` to `2xx`) at scrape — it merges distinct series and produces duplicate-sample errors; do that in code or via Adaptive Metrics, never here.\n\nThe genuinely safe uses are:\n\n- **Drop an entire metric you never want stored** — you're discarding the whole metric, not collapsing distinct series into one:\n  ```yaml\n  metric_relabel_configs:\n    - source_labels: [__name__]\n      regex: my_app_request_details\n      action: drop\n  ```\n- **Remove a label that *exactly duplicates* a target label.** If the app emits its own `cluster`\u002F`instance` that already comes from the scrape target, the target label still provides uniqueness, so removing the duplicate breaks nothing. Prefer fixing the app, but this is a safe stopgap.\n\nThat's the whole list. If you're reaching for `metric_relabel_configs` to bring down a series count, you almost certainly want Adaptive Metrics instead.\n\n### 5. Recording Rules (query-time cardinality reduction)\n\nPre-aggregate expensive series into a lower-cardinality recorded series. Stored at the same data point density but with far fewer series.\n\n```yaml\ngroups:\n  - name: http-requests-aggregates\n    interval: 30s\n    rules:\n      # Drop pod\u002Finstance dimension; keep only service-level rollup\n      - record: service:http_requests:rate5m\n        expr: sum by (service, env, cluster, status_code) (rate(http_requests_total[5m]))\n```\n\nQueries that target the rollup are dramatically cheaper. The raw series still exist — recording rules don't reduce ingest cost (use **Adaptive Metrics** for that — *not* a scrape-time `labeldrop`). They reduce query cost.\n\n---\n\n## Instrumentation Hygiene (for app developers)\n\nIf the user is *writing* instrumentation code, these are the rules:\n\n| Rule | Why |\n|---|---|\n| Never use unbounded user input as a label value | `email`, `user_id`, `query string`, `error message` — they're the #1 cardinality bug |\n| Template HTTP paths before recording | `\u002Fusers\u002F{id}` not `\u002Fusers\u002F12345`. Most frameworks do this via routing metadata |\n| Bound error labels via small enums | `error_type=\"timeout\"` not `error=\"connection to db-shard-7 timed out at 14:32:09\"` |\n| Don't put `version` \u002F `git_sha` \u002F `build_id` on every metric | Use an info metric and join at query time |\n| Don't emit `pod` \u002F `node` \u002F `host` from code | Comes from scrape targets — duplicating creates collisions |\n| Avoid dynamically constructed label *names* (keys) | `metric{[user]=1}` cannot be bounded — use a fixed key |\n| Use histograms sparingly and trim labels first | 14× cardinality amplification |\n| Prefer exemplars over labels for trace correlation | Exemplars carry `trace_id` without inflating cardinality |\n\n### Exemplars (the escape hatch)\n\nExemplars attach a `trace_id` (or any key-value pair) to specific samples *without* making it a label dimension. The ideal home for high-cardinality correlation data.\n\nRequires OpenMetrics format, Prometheus 2.26+, scrape config:\n```yaml\nscrape_configs:\n  - job_name: my-app\n    enable_protobuf_negotiation: true\n    # Or for text-format:\n    follow_redirects: true\n```\n\nAnd on the Prometheus server:\n```yaml\nstorage:\n  exemplars:\n    max_exemplars: 100000\n```\n\nUse exemplars for:\n- `trace_id` correlation (Tempo, Jaeger)\n- `request_id` for specific debug lookups\n- Any sparse \"useful when you need it\" key\n\nQuery exemplars via Grafana's exemplars-on-graph feature, not via PromQL aggregation.\n\n---\n\n## The 80\u002F20 Rule\n\nThe most impactful improvements almost always come from these five changes:\n\n1. **Drop unbounded labels at the app layer** — `path` (untemplated), `user_id`, `error_message`. Single biggest win.\n2. **Trim histogram label cardinality before anything else** — 14× amplification on every histogram.\n3. **Don't emit `pod`\u002F`instance`\u002F`node` from application code** — let them come from scrape targets, and add a stable `workload` target label to aggregate on. (Never *drop* the real `pod` at scrape to cut cardinality — if `pod`-level series are too expensive, use Adaptive Metrics.)\n4. **Use info metrics for `version` \u002F `git_sha` \u002F `image_tag`** — eliminates deploy-driven churn.\n5. **Set target labels via `relabel_configs`, not app code** — `env`, `cluster`, `team`, `service` should never be emitted by the application.\n\nFocus on these before anything else.\n\n---\n\n## Labels to Avoid — Quick Reference\n\n| Label | Why | Alternative |\n|---|---|---|\n| `user_id`, `customer_id` (large tenant base) | Unbounded | Exemplars; aggregate by `tenant_tier` |\n| `request_id`, `trace_id` | Unbounded | Exemplars |\n| `path` \u002F `route` (raw URLs) | Unbounded | Template in code: `\u002Fusers\u002F:id` |\n| `error_message`, `query`, `sql` | Unbounded text | Bounded `error_type` enum |\n| `version`, `git_sha`, `image_tag` (on every metric) | Churn on every deploy | Info metric pattern |\n| App-emitted `pod` (duplicating SD) | Should come from K8s service discovery, not code | Stop emitting it in code; keep the discovered `pod`. Never drop the real `pod` to cut cardinality — use Adaptive Metrics |\n| `uid` (K8s) | Unbounded; regenerates on restart | Never map it into a label in the first place (leave it out of `relabel_configs`) |\n| Application-emitted `instance`, `node`, `host` | Should come from scrape target | Stop emitting in code (removing an *exact* target-label duplicate at scrape is the only safe drop) |\n| Dynamically-named label keys | Cannot be bounded | Use fixed keys with bounded values |\n| Raw `status_code` on histograms | 14× amplification | Bucket to `status_class` (`2xx`, `4xx`, `5xx`) |\n\n---\n\n## When to Route Elsewhere\n\n- **\"Reduce my Grafana Cloud bill\"** \u002F **\"reduce cardinality on series already ingested\"** → engage `adaptive-metrics` skill (post-ingest aggregation rules — the safe, counter-reset-aware way; never `labeldrop` distinguishing labels at scrape)\n- **\"Which metrics are driving my DPM?\"** → engage `dpm-finder` skill\n- **\"My Prometheus is OOMing \u002F scraping is failing right now\"** → engage `prometheus-cardinality-troubleshooter` skill\n- **\"How do I write the query to find the bad metric?\"** → engage `promql` skill\n- **\"How do I configure relabel rules in Alloy?\"** → engage `alloy` skill\n\nThis skill's lane is **strategy and design**. Other skills own **diagnosis** and **operational remediation**.\n",{"name":5,"license":15,"version":24,"description":7},"1.0.0","main","3b63fa056f5de7956766fca020e9c71fdf2b0fab","SKILL\u002Fdevops\u002Ftier2\u002Fgrafana-prometheus-label-strategy",0,"https:\u002F\u002Fgithub.com\u002Fyhyu13\u002FAgentSkillsVault","https:\u002F\u002Fgithub.com\u002Fyhyu13","https:\u002F\u002Fyhyu13.github.io","大丈夫自立于天地之间，岂可郁郁久居于人下\r\nA zealot of AI.","2026-11-14 20:53:07",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add yhyu13\u002FAgentSkillsVault --skill grafana-prometheus-label-strategy",{"tool":40,"command":41},"bunx","bunx skills add yhyu13\u002FAgentSkillsVault --skill grafana-prometheus-label-strategy",{"tool":43,"command":44},"pnpm","pnpm dlx skills add yhyu13\u002FAgentSkillsVault --skill grafana-prometheus-label-strategy","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fyhyu13\u002FAgentSkillsVault\u002Fblob\u002Fmain\u002FSKILL\u002Fdevops\u002Ftier2\u002Fgrafana-prometheus-label-strategy\u002FSKILL.md",true,[],[49,57,64,71,78,85],{"skillNo":50,"name":51,"nameZh":6,"description":52,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":53,"licenseFileUrl":6,"sourceHtmlUrl":54,"categories":55,"keywords":56,"publishedAt":19},"01M7DZJW3GMCBHNCC1J37K73T7","feishu-kdocs-cli","Office cloud CLIs for agents: install, authenticate, verify and troubleshoot Feishu `lark-cli` and Kingsoft\u002FWPS Docs `kdocs-cli`, plus the three China office MCP endpoints (Feishu, WPS SkillHub, Tencent Docs). Use when setting up or re-logging-in 飞书 CLI \u002F 金山文档 CLI, when a command fails with missing scope \u002F PATH \u002F token \u002F device-code errors, when a kdocs or lark command returns 400001 or unknown action, or when wiring 飞书 \u002F 金山文档 \u002F 腾讯文档 MCP into Cindy, Claude or another host. Not a per-command reference: domain commands live in the lark-* Skills and the kdocs Skill.","MIT","https:\u002F\u002Fgithub.com\u002Fyhyu13\u002FAgentSkillsVault\u002Fblob\u002Fmain\u002FSKILL\u002Fdevops\u002Ftier0\u002Ffeishu-kdocs-cli\u002FSKILL.md",[],[],{"skillNo":58,"name":59,"nameZh":6,"description":60,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":53,"licenseFileUrl":6,"sourceHtmlUrl":61,"categories":62,"keywords":63,"publishedAt":19},"01M7BE0VQF0T7ACMY3W9V27ZYG","vfx-fx","Cinematic and game VFX in Blender including smoke, fire, particles, trails, and Geometry Nodes FX setups via MCP.","https:\u002F\u002Fgithub.com\u002Fyhyu13\u002FAgentSkillsVault\u002Fblob\u002Fmain\u002FSKILL\u002Fgame-dev\u002Ftier2\u002Fvfx-fx\u002FSKILL.md",[],[],{"skillNo":65,"name":66,"nameZh":6,"description":67,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":53,"licenseFileUrl":6,"sourceHtmlUrl":68,"categories":69,"keywords":70,"publishedAt":19},"01M7BE0Q23SY6QAW24XEADEH0N","uv-workflow","Production UV unwrapping for seam placement, UV packing, texel density, UDIM, lightmap UVs, and modular UV layouts. Use when preparing meshes for texturing and engine export in Blender via MCP.","https:\u002F\u002Fgithub.com\u002Fyhyu13\u002FAgentSkillsVault\u002Fblob\u002Fmain\u002FSKILL\u002Fgame-dev\u002Ftier2\u002Fuv-workflow\u002FSKILL.md",[],[],{"skillNo":72,"name":73,"nameZh":6,"description":74,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":53,"licenseFileUrl":6,"sourceHtmlUrl":75,"categories":76,"keywords":77,"publishedAt":19},"01M7BE0JAKENVJB4Z0SV3Q7QGR","texture-workflow","Production texture workflow for atlases, baking AO\u002Fcurvature\u002Fnormal\u002Froughness maps, decals, and texture memory optimization. Use for game-ready texture pipelines in Blender via MCP.","https:\u002F\u002Fgithub.com\u002Fyhyu13\u002FAgentSkillsVault\u002Fblob\u002Fmain\u002FSKILL\u002Fgame-dev\u002Ftier2\u002Ftexture-workflow\u002FSKILL.md",[],[],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":53,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":19},"01M7BE0CYB7BDJP7HD5WTS2PJB","sculpting","Blender sculpting for organic and hard surface detail using Dyntopo, multiresolution, brushes, alpha workflow, wrinkles, and damage. Use when high-frequency surface detail is needed before retopology in Blender via MCP.","https:\u002F\u002Fgithub.com\u002Fyhyu13\u002FAgentSkillsVault\u002Fblob\u002Fmain\u002FSKILL\u002Fgame-dev\u002Ftier2\u002Fsculpting\u002FSKILL.md",[],[],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":53,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":19},"01M7BE09TBX9NQ5EBHN4JZ9VXM","scene-assembly","Large scene layout in Blender with collections, linking, overrides, streaming-friendly organization, and set assembly via MCP.","https:\u002F\u002Fgithub.com\u002Fyhyu13\u002FAgentSkillsVault\u002Fblob\u002Fmain\u002FSKILL\u002Fgame-dev\u002Ftier2\u002Fscene-assembly\u002FSKILL.md",[],[]]