AGENT SKILL

alerting-irm

Configure Grafana Alerting, Incident Response Management (IRM), and SLOs end-to-end — provisions Grafana-managed and data-source-managed alert rules, contact points (Slack/PagerDuty/email/webhook), notification policies with hierarchical matchers, silences, mute timings, on-call schedules and escalation chains, incident-management integrations, and SLOs with multi-window burn-rate alerts. Use when configuring alerts, debugging notification routing, setting up on-call rotations, declaring or managing incidents, defining SLOs, provisioning alerting via YAML or API, picking matchers for a notification policy, building a PagerDuty/Slack webhook receiver, or troubleshooting why an alert isn't firing — even when the user says "page me on errors", "alert me when X happens", "route this to the platform team", or "set up an SLO" without naming Alerting or IRM.

License 许可
Apache-2.0
最近更新
2026-11-14
Stars
★ 12
Forks
5

01 前言 Front Matter

name
alerting-irm
license
Apache-2.0
description
Configure Grafana Alerting, Incident Response Management (IRM), and SLOs end-to-end — provisions Grafana-managed and data-source-managed alert rules, contact points (Slack/PagerDuty/email/webhook), notification policies with hierarchical matchers, silences, mute timings, on-call schedules and escalation chains, incident-management integrations, and SLOs with multi-window burn-rate alerts. Use when configuring alerts, debugging notification routing, setting up on-call rotations, declaring or managing incidents, defining SLOs, provisioning alerting via YAML or API, picking matchers for a notification policy, building a PagerDuty/Slack webhook receiver, or troubleshooting why an alert isn't firing — even when the user says "page me on errors", "alert me when X happens", "route this to the platform team", or "set up an SLO" without naming Alerting or IRM.

02 资源文件 4

03 SKILL.md 正文

Grafana Alerting & IRM

Docs: https://grafana.com/docs/grafana/latest/alerting.md

Common Workflows

Provisioning a new alert end-to-end

  1. Create contact points (where notifications go):

    curl -X POST https://grafana.example.com/api/v1/provisioning/contact-points \
      -H 'Authorization: Bearer <token>' -H 'Content-Type: application/json' \
      -d @contact-points.json
    

    Verify:

    curl https://grafana.example.com/api/v1/provisioning/contact-points \
      -H 'Authorization: Bearer <token>' | jq '.[].name'
    
  2. Add notification policies (which alerts go where) — see § Notification policies below for the matchers pattern.

  3. Write the alert rule — pick the type:

  4. Verify routing before going live:

    # Force-fire a test alert from the rule's UI, then check Alertmanager's view
    curl https://grafana.example.com/api/alertmanager/grafana/api/v2/alerts \
      -H 'Authorization: Bearer <token>' | jq '.[] | {alertname: .labels.alertname, receiver: .receivers}'
    

    The expected receiver should appear. If the wrong receiver appears, re-check the policy's matchers.

Routing alerts to IRM / on-call

  1. In IRM, create an Integration of type "Grafana Alerting webhook" → copy the integration URL
  2. Add a webhook contact point in Grafana Alerting pointing at that URL (full YAML in references/irm.md § Routing)
  3. Add a notification policy matcher routing the right severity to the new contact point
  4. Verify: trigger a test alert; it should appear in IRM within ~30s. Full debug procedure in references/irm.md § Verifying the IRM integration.

Defining an SLO

  1. Create the SLO via UI or API → Grafana auto-generates recording rules, dashboards, and burn-rate alerts (the generated YAML is in references/slo.md)
  2. Use multi-window burn-rate alerts, not single-window — see references/slo.md § Multi-window burn-rate alerts for why single-window fires on noise
  3. Verify with the 4-step pattern in references/slo.md § Validating SLO config

Contact Points (YAML provisioning)

# provisioning/alerting/contact_points.yaml
apiVersion: 1
contactPoints:
  - orgId: 1
    name: pagerduty-critical
    receivers:
      - uid: pd-receiver
        type: pagerduty
        settings:
          integrationKey: YOUR_PAGERDUTY_KEY
          severity: critical

  - orgId: 1
    name: slack-alerts
    receivers:
      - uid: slack-receiver
        type: slack
        settings:
          url: https://hooks.slack.com/services/YOUR/WEBHOOK/URL
          channel: '#alerts'

For email, webhook, Teams, Telegram, OnCall, and other receiver types, see references/alerting.md § Contact point receiver types.

Notification policies

Hierarchical routing tree with label matchers:

# provisioning/alerting/notification_policies.yaml
apiVersion: 1
policies:
  - orgId: 1
    receiver: default-receiver
    group_by: ['alertname', 'cluster', 'service']
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 12h
    routes:
      # Critical alerts → PagerDuty
      - receiver: pagerduty-critical
        matchers:
          - severity = critical
        group_wait: 10s
        repeat_interval: 4h

      # Platform team → Slack, but page on critical
      - receiver: slack-alerts
        matchers:
          - team = platform
        routes:
          - receiver: pagerduty-critical
            matchers:
              - severity = critical

      # Everything else → email
      - receiver: email-alerts
        matchers:
          - severity =~ "warning|info"

Silences

Suppress notifications for matching alerts without stopping evaluation:

curl -X POST https://grafana.example.com/api/alertmanager/grafana/api/v2/silences \
  -H 'Authorization: Bearer <token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "matchers": [
      {"name": "alertname", "value": "HighErrorRate", "isRegex": false},
      {"name": "env", "value": "staging", "isRegex": false}
    ],
    "startsAt": "2024-01-01T00:00:00Z",
    "endsAt": "2024-01-01T02:00:00Z",
    "comment": "Maintenance window",
    "createdBy": "admin"
  }'

# Verify it was created
curl https://grafana.example.com/api/alertmanager/grafana/api/v2/silences \
  -H 'Authorization: Bearer <token>' | jq '.[] | select(.status.state == "active")'

Alert rule states

State Description
Normal Condition not met
Pending Condition met, waiting for for duration
Firing Condition met for full for duration
NoData Query returned no data
Error Query/evaluation error
Recovering Was firing, condition no longer met

Provisioning directory layout

provisioning/alerting/
├── alert_rules.yaml          # Alert and recording rules
├── contact_points.yaml       # Notification destinations
├── notification_policies.yaml  # Routing tree
├── templates.yaml            # Message templates
└── mute_timings.yaml         # Recurring mute windows

API provisioning (keeps UI editable)

Add X-Disable-Provenance: true to keep resources editable in the UI after API provisioning:

curl -X PUT https://grafana.example.com/api/v1/provisioning/policies \
  -H 'Authorization: Bearer <token>' \
  -H 'X-Disable-Provenance: true' \
  -H 'Content-Type: application/json' \
  -d @policy.json

curl -X POST https://grafana.example.com/api/v1/provisioning/alert-rules \
  -H 'Authorization: Bearer <token>' \
  -H 'X-Disable-Provenance: true' \
  -H 'Content-Type: application/json' \
  -d @rule.json

References

  • references/alerting.md — full alert rule YAML (Grafana-managed / Prometheus / Loki) + notification templates
  • references/slo.md — generated SLO recording rules + multi-window burn-rate alert pattern + validation steps
  • references/irm.md — IRM capabilities, integration sources, Alerting → IRM routing + verification + common failure modes

04 同仓库更多 Skills

stanfish06
stanfish06/skillquarium

zarr-python

Stores and queries chunked N-D scientific arrays with Zarr-Python 3, including codecs, sharding, S3/GCS storage, and NumPy/Dask/Xarray integration. Use for array layout, bounded I/O, format migration, or scientific metadata preservation.

★ 12
stanfish06
stanfish06/skillquarium

yeet

Publish local changes to GitHub by confirming scope, committing intentionally, pushing the branch, and opening a draft PR through the GitHub app from this plugin, with `gh` used only as a fallback where connector coverage is insufficient.

★ 12
stanfish06
stanfish06/skillquarium

xarray

N-dimensional labeled arrays and datasets in Python. Built on top of NumPy and Dask. It introduces labels in the form of dimensions, coordinates, and attributes on top of raw NumPy-like arrays, making data analysis in physical sciences more intuitive and less error-prone. Use for working with multi-dimensional scientific data, NetCDF/GRIB/Zarr files, climate/weather/oceanographic datasets, remote sensing, geospatial imaging, large out-of-memory datasets with Dask, and labeled array operations.

★ 12
stanfish06
stanfish06/skillquarium

writing-mstest-tests

Use when asked to fix, rewrite, update, improve, modernize, show corrected code for, or explain existing MSTest tests or MSTest-specific configuration. Use for "review" when corrected code or edits are wanted, even for one pasted assertion or passing tests with bad failure output. Covers expected/actual labels; generic Boolean, collection, string, numeric, null, identity, exception, hard-cast, and object[] checks; TestContext/lifecycle; timeout/cancellation; OS/CI conditions, retry, cleanup, parallelization, MSTest.Sdk project setup, and MSTESTxxxx. Honor the installed MSTest version. DO NOT USE to design new test cases (code-testing), perform report-only audits, create project files rather than explain MSTest setup, run tests, migrate frameworks, or handle non-MSTest/non-.NET code.

★ 12
stanfish06
stanfish06/skillquarium

wgs-prs

End-to-end WGS to polygenic risk score pipeline. Takes paired-end FASTQ files (or a pre-existing VCF) through nf-core/sarek for variant calling, applies VCF QC (normalisation, hard filtering, Ti/Tv and Het/Hom checks), then computes polygenic risk scores via the PGS Catalog. Fills the FASTQ to VCF gap upstream of the gwas-prs skill.

★ 12
stanfish06
stanfish06/skillquarium

webapp-testing

Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.

★ 12