datadog-pro
Datadog guidance — APM, logs, metrics, monitors, SLOs, dashboards, and cost control on the Datadog platform.
- License 许可
- MIT
- 最近更新
- 2026-11-07
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- datadog-pro
- category
- development
- description
- Datadog guidance — APM, logs, metrics, monitors, SLOs, dashboards, and cost control on the Datadog platform.
03 SKILL.md 正文
Overview
Datadog is the all-in-one observability platform: infrastructure monitoring, APM tracing, log management, RUM, synthetics, and security — unified by tagging. Its strength is correlation with minimal setup: install the agent, get metrics/traces/logs linked automatically. Its weakness is the bill, which scales with hosts, ingested logs, and custom metrics.
Using Datadog well means: tag everything consistently, be deliberate about what's ingested (logs and custom metrics are the cost drivers), and build monitors/SLOs on symptoms. This skill covers the platform's core, cost control, and operational patterns.
When to use
- Setting up Datadog (agents, integrations, APM).
- Designing tagging strategy and dashboards.
- Writing monitors (thresholds, anomaly, composite).
- Defining SLOs and error budgets.
- Controlling Datadog costs (logs, custom metrics, retention).
- Correlating traces, logs, and metrics in investigations.
- Choosing between Datadog and Prometheus/Grafana/ELK.
Core concepts
- Unified tagging. Tags (
env:prod,service:api,version:abc) unify metrics, traces, and logs. A consistent tagging scheme is the highest-leverage Datadog decision — inconsistent tags make correlation impossible. - The Agent. Host/container agent collecting metrics, traces (APM), and logs. DaemonSet in Kubernetes; trace client libraries in apps. One agent version policy — keep them updated.
- APM. Distributed tracing with automatic instrumentation for common frameworks; custom spans for business operations. Trace search and analytics on span tags; link traces to logs via trace IDs.
- Log management. Ingest, parse (pipelines/grok), index selectively. Ingestion is the cost driver — index what you search, archive the rest to object storage (rehydration when needed).
- Custom metrics. The other cost driver — billed per unique metric+tag combination per month. Cardinality discipline (same rule as Prometheus) plus metrics-without-limits (distributions) where appropriate.
- Monitors. Threshold, anomaly, forecast, and composite monitors; multi-alert vs simple-alert (per-group or aggregate). Alert on symptoms; include runbook links and tags for routing.
- SLOs and error budgets. Time-slice or monitor-based SLOs with error-budget alerts — the framework for "how reliable are we" that both engineering and product understand.
- Dashboards and notebooks. Timeboards (shared, versioned-ish) vs screenboards; notebooks for investigations and postmortems. Template variables for reuse; keep golden dashboards curated.
- Synthetics. API tests and browser tests from global locations — proactive monitoring of user journeys, not just backend health. Test the checkout flow, not just
/health. - RUM. Real-user monitoring for frontend performance and errors — Web Vitals, session replay. Privacy implications need review (masking, consent).
- Watchdog. ML-driven anomaly detection surfacing issues without predefined monitors — triage its findings, don't auto-page on them initially.
- Service catalog and Scorecards. Service ownership, docs, and health scorecards — the organizational layer that keeps observability maintained as teams grow.
- Sensitive data. Log scrubbing (obfuscation rules) for PII/secrets — configure before sensitive data flows, not after an incident.
- Cost controls. Usage attribution by team (tags!), log exclusion filters, metrics cardinality management, retention settings, and regular bill reviews with owners per cost center.
- Incident management. Declared incidents with timelines, responders, and retrospectives — the coordination layer that turns alerts into resolved outages.
- Cloud cost management. Cost attribution alongside observability — the same tags showing spend per service, so cost and performance live in one view.
Practical workflow
-
Define the tagging scheme.
env,service,version,teamon everything — agent config, APM, logs. Enforce via admission checks or CI; audit tag coverage. -
Deploy agents consistently. DaemonSet with log collection in K8s; APM enabled; dogstatsd for custom app metrics; keep agent versions current.
-
Control ingestion. Log pipelines parsing only needed fields; exclusion filters dropping debug/health-check noise before indexing; archive everything to storage; custom-metrics cardinality review.
- Rule of thumb: if nobody searches it, don't index it — archive and rehydrate on demand.
-
Build golden dashboards. Per service: RED metrics, saturation, dependencies, deploy markers (via events API from CI). Template variables for env/service.
-
Write monitors on symptoms. Error-rate, latency, and saturation monitors with multi-alert grouping; anomaly detection for seasonal patterns; composite monitors for "really broken" paging.
{ "name": "[prod] High error rate — shop-api", "type": "query alert", "query": "avg(last_5m):sum:trace.http.request.errors{env:prod,service:shop-api}.as_rate() / sum:trace.http.request.hits{env:prod,service:shop-api}.as_rate() > 0.05", "message": "@pagerduty Runbook: https://wiki.example.com/runbooks/high-error-rate" } -
Define SLOs. A few meaningful SLOs per critical service (availability, latency); error-budget burn alerts; review in planning — budgets guide prioritization.
-
Add synthetics for user journeys. API and browser tests for critical flows from relevant locations; alert on failure with the same severity as backend pages.
-
Attribute and control cost. Usage dashboards by team/service; monthly reviews; log rehydration instead of permanent indexing for cold data; right-size retention.
exclusion filter: drop health-check noise before indexing
Query: status:ok service:api @http.url:/health
Action: exclude from index (still archived to object storage)
Common pitfalls
- Inconsistent tagging — correlation impossible; define and enforce the scheme early.
- Indexing everything — log bills exploding; exclusion filters + archive + rehydrate.
- Custom metric cardinality — per-user/per-request tags; the bill scales with unique combinations.
- Monitors on causes — paging for CPU; alert on user-facing symptoms.
- Monitors without runbooks — pages with no guidance; link runbooks.
- Simple-alert when multi-alert needed — one alert for 50 hosts; group appropriately.
- No SLOs — reliability debates without data; define a few meaningful SLOs.
- Watchdog auto-paging — ML findings paging before trust is built; triage first.
- PII in logs — sensitive data ingested unscrubbed; obfuscation rules from day one.
- Stale monitors — alerts for decommissioned services; audit monitors with service lifecycle.
- RUM without privacy review — session replay capturing sensitive input; masking and consent.
- No cost attribution — one bill, no owners; tag-based usage dashboards per team.
- Synthetics only on
/health— green while checkout is broken; test real user journeys. - No exclusion filters — debug and health-check logs indexed at full price; filter noise before indexing.
- Tag sprawl — hundreds of low-value tags multiplying custom-metric costs; audit tag cardinality regularly.
04 同仓库更多 Skills
web-scrape
Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open/fetch fails on JS-heavy pages, or when you need clean markdown/structured extraction from a site. Requires FIRECRAWL_API_KEY.
design-taste-frontend
tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML/CSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, "AI-made", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.
test-driven-development
Use when implementing any feature or bugfix, before writing implementation code
systematic-debugging
Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes
requesting-code-review
Use when completing tasks, implementing major features, or before merging to verify work meets requirements
brainstorming
You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).