AGENT SKILL

observability-designer

Design production-ready observability strategies combining metrics, logs, and traces. Includes SLI/SLO design, golden-signals monitoring, alert optimization. Use when adding observability to a new service, refactoring alerting that is too noisy, or designing an SLO program before scaling production load.

License 许可
MIT
最近更新
2026-08-16
Stars
★ 0
Forks
0

01 前言 Front Matter

name
observability-designer
description
Design production-ready observability strategies combining metrics, logs, and traces. Includes SLI/SLO design, golden-signals monitoring, alert optimization. Use when adding observability to a new service, refactoring alerting that is too noisy, or designing an SLO program before scaling production load.

02 资源文件 12

03 SKILL.md 正文

Observability Designer (POWERFUL)

Category: Engineering
Tier: POWERFUL
Description: Design comprehensive observability strategies for production systems including SLI/SLO frameworks, alerting optimization, and dashboard generation.

Overview

Observability Designer creates production-ready dashboards, alert configurations, and monitoring strategies across the three pillars (metrics, logs, traces).

When NOT to use → slo-architect. For SLO/SLI design with error-budget math, multi-window burn-rate alerting thresholds, and SLO review gates, route to slo-architect — it is the authoritative skill for that half. This skill's slo_designer.py produces a quick scaffold only. This skill's lane: dashboards (dashboard_generator.py) and alert-noise reduction (alert_optimizer.py).

Quick Start

# Dashboard spec (Grafana JSON + docs) for a service
python3 scripts/dashboard_generator.py --service-type api --name payments --criticality critical --role sre --format grafana -o dashboard.json --doc-output dashboard.md

# Analyze an existing alert config for noise, duplicates, and coverage gaps
python3 scripts/alert_optimizer.py --input alerts.json --analyze-only --report alert_report.json
# ...then emit the optimized config once the report is reviewed:
python3 scripts/alert_optimizer.py --input alerts.json --output alerts_optimized.json

# Quick SLO scaffold (hand off to slo-architect for the real error-budget work)
python3 scripts/slo_designer.py --service-type api --criticality high --user-facing true --service-name payments -o slo_scaffold.json

Verification loop: after deploying optimized alerts, track the report's noise metrics for one on-call rotation — if the actionable-alert ratio didn't improve, re-run --analyze-only against the live config and iterate. Import the generated dashboard into Grafana and confirm every golden-signal panel renders with live data before closing the task.

Core Competencies

SLI/SLO/SLA Framework Design

  • Service Level Indicators (SLI): Define measurable signals that indicate service health
  • Service Level Objectives (SLO): Set reliability targets based on user experience
  • Service Level Agreements (SLA): Establish customer-facing commitments with consequences
  • Error Budget Management: Calculate and track error budget consumption
  • Burn Rate Alerting: Multi-window burn rate alerts for proactive SLO protection

Three Pillars of Observability

Metrics

  • Golden Signals: Latency, traffic, errors, and saturation monitoring
  • RED Method: Rate, Errors, and Duration for request-driven services
  • USE Method: Utilization, Saturation, and Errors for resource monitoring
  • Business Metrics: Revenue, user engagement, and feature adoption tracking
  • Infrastructure Metrics: CPU, memory, disk, network, and custom resource metrics

Logs

  • Structured Logging: JSON-based log formats with consistent fields
  • Log Aggregation: Centralized log collection and indexing strategies
  • Log Levels: Appropriate use of DEBUG, INFO, WARN, ERROR, FATAL levels
  • Correlation IDs: Request tracing through distributed systems
  • Log Sampling: Volume management for high-throughput systems

Traces

  • Distributed Tracing: End-to-end request flow visualization
  • Span Design: Meaningful span boundaries and metadata
  • Trace Sampling: Intelligent sampling strategies for performance and cost
  • Service Maps: Automatic dependency discovery through traces
  • Root Cause Analysis: Trace-driven debugging workflows

Dashboard Design Principles

Information Architecture

  • Hierarchy: Overview → Service → Component → Instance drill-down paths
  • Golden Ratio: 80% operational metrics, 20% exploratory metrics
  • Cognitive Load: Maximum 7±2 panels per dashboard screen
  • User Journey: Role-based dashboard personas (SRE, Developer, Executive)

Visualization Best Practices

  • Chart Selection: Time series for trends, heatmaps for distributions, gauges for status
  • Color Theory: Red for critical, amber for warning, green for healthy states
  • Reference Lines: SLO targets, capacity thresholds, and historical baselines
  • Time Ranges: Default to meaningful windows (4h for incidents, 7d for trends)

Panel Design

  • Metric Queries: Efficient Prometheus/InfluxDB queries with proper aggregation
  • Alerting Integration: Visual alert state indicators on relevant panels
  • Interactive Elements: Template variables, drill-down links, and annotation overlays
  • Performance: Sub-second render times through query optimization

Alert Design and Optimization

Alert Classification

  • Severity Levels:
    • Critical: Service down, SLO burn rate high
    • Warning: Approaching thresholds, non-user-facing issues
    • Info: Deployment notifications, capacity planning alerts
  • Actionability: Every alert must have a clear response action
  • Alert Routing: Escalation policies based on severity and team ownership

Alert Fatigue Prevention

  • Signal vs Noise: High precision (few false positives) over high recall
  • Hysteresis: Different thresholds for firing and resolving alerts
  • Suppression: Dependent alert suppression during known outages
  • Grouping: Related alerts grouped into single notifications

Alert Rule Design

  • Threshold Selection: Statistical methods for threshold determination
  • Window Functions: Appropriate averaging windows and percentile calculations
  • Alert Lifecycle: Clear firing conditions and automatic resolution criteria
  • Testing: Alert rule validation against historical data

Runbook Generation and Incident Response

Runbook Structure

  • Alert Context: What the alert means and why it fired
  • Impact Assessment: User-facing vs internal impact evaluation
  • Investigation Steps: Ordered troubleshooting procedures with time estimates
  • Resolution Actions: Common fixes and escalation procedures
  • Post-Incident: Follow-up tasks and prevention measures

Incident Detection Patterns

  • Anomaly Detection: Statistical methods for detecting unusual patterns
  • Composite Alerts: Multi-signal alerts for complex failure modes
  • Predictive Alerts: Capacity and trend-based forward-looking alerts
  • Canary Monitoring: Early detection through progressive deployment monitoring

Golden Signals Framework

Latency Monitoring

  • Request Latency: P50, P95, P99 response time tracking
  • Queue Latency: Time spent waiting in processing queues
  • Network Latency: Inter-service communication delays
  • Database Latency: Query execution and connection pool metrics

Traffic Monitoring

  • Request Rate: Requests per second with burst detection
  • Bandwidth Usage: Network throughput and capacity utilization
  • User Sessions: Active user tracking and session duration
  • Feature Usage: API endpoint and feature adoption metrics

Error Monitoring

  • Error Rate: 4xx and 5xx HTTP response code tracking
  • Error Budget: SLO-based error rate targets and consumption
  • Error Distribution: Error type classification and trending
  • Silent Failures: Detection of processing failures without HTTP errors

Saturation Monitoring

  • Resource Utilization: CPU, memory, disk, and network usage
  • Queue Depth: Processing queue length and wait times
  • Connection Pools: Database and service connection saturation
  • Rate Limiting: API throttling and quota exhaustion tracking

Distributed Tracing Strategies

Trace Architecture

  • Sampling Strategy: Head-based, tail-based, and adaptive sampling
  • Trace Propagation: Context propagation across service boundaries
  • Span Correlation: Parent-child relationship modeling
  • Trace Storage: Retention policies and storage optimization

Service Instrumentation

  • Auto-Instrumentation: Framework-based automatic trace generation
  • Manual Instrumentation: Custom span creation for business logic
  • Baggage Handling: Cross-cutting concern propagation
  • Performance Impact: Instrumentation overhead measurement and optimization

Log Aggregation Patterns

Collection Architecture

  • Agent Deployment: Log shipping agent strategies (push vs pull)
  • Log Routing: Topic-based routing and filtering
  • Parsing Strategies: Structured vs unstructured log handling
  • Schema Evolution: Log format versioning and migration

Storage and Indexing

  • Index Design: Optimized field indexing for common query patterns
  • Retention Policies: Time and volume-based log retention
  • Compression: Log data compression and archival strategies
  • Search Performance: Query optimization and result caching

Cost Optimization for Observability

Data Management

  • Metric Retention: Tiered retention based on metric importance
  • Log Sampling: Intelligent sampling to reduce ingestion costs
  • Trace Sampling: Cost-effective trace collection strategies
  • Data Archival: Cold storage for historical observability data

Resource Optimization

  • Query Efficiency: Optimized metric and log queries
  • Storage Costs: Appropriate storage tiers for different data types
  • Ingestion Rate Limiting: Controlled data ingestion to manage costs
  • Cardinality Management: High-cardinality metric detection and mitigation

Scripts Overview

This skill includes three powerful Python scripts for comprehensive observability design:

1. SLO Designer (slo_designer.py)

Generates complete SLI/SLO frameworks based on service characteristics:

  • Input: Service description JSON (type, criticality, dependencies)
  • Output: SLI definitions, SLO targets, error budgets, burn rate alerts, SLA recommendations
  • Features: Multi-window burn rate calculations, error budget policies, alert rule generation

2. Alert Optimizer (alert_optimizer.py)

Analyzes and optimizes existing alert configurations:

  • Input: Alert configuration JSON with rules, thresholds, and routing
  • Output: Optimization report and improved alert configuration
  • Features: Noise detection, coverage gaps, duplicate identification, threshold optimization

3. Dashboard Generator (dashboard_generator.py)

Creates comprehensive dashboard specifications:

  • Input: Service/system description JSON
  • Output: Grafana-compatible dashboard JSON and documentation
  • Features: Golden signals coverage, RED/USE methods, drill-down paths, role-based views

Integration Patterns

Monitoring Stack Integration

  • Prometheus: Metric collection and alerting rule generation
  • Grafana: Dashboard creation and visualization configuration
  • Elasticsearch/Kibana: Log analysis and dashboard integration
  • Jaeger/Zipkin: Distributed tracing configuration and analysis

CI/CD Integration

  • Pipeline Monitoring: Build, test, and deployment observability
  • Deployment Correlation: Release impact tracking and rollback triggers
  • Feature Flag Monitoring: A/B test and feature rollout observability
  • Performance Regression: Automated performance monitoring in pipelines

Incident Management Integration

  • PagerDuty/VictorOps: Alert routing and escalation policies
  • Slack/Teams: Notification and collaboration integration
  • JIRA/ServiceNow: Incident tracking and resolution workflows
  • Post-Mortem: Automated incident analysis and improvement tracking

Advanced Patterns

Multi-Cloud Observability

  • Cross-Cloud Metrics: Unified metrics across AWS, GCP, Azure
  • Network Observability: Inter-cloud connectivity monitoring
  • Cost Attribution: Cloud resource cost tracking and optimization
  • Compliance Monitoring: Security and compliance posture tracking

Microservices Observability

  • Service Mesh Integration: Istio/Linkerd observability configuration
  • API Gateway Monitoring: Request routing and rate limiting observability
  • Container Orchestration: Kubernetes cluster and workload monitoring
  • Service Discovery: Dynamic service monitoring and health checks

Machine Learning Observability

  • Model Performance: Accuracy, drift, and bias monitoring
  • Feature Store Monitoring: Feature quality and freshness tracking
  • Pipeline Observability: ML pipeline execution and performance monitoring
  • A/B Test Analysis: Statistical significance and business impact measurement

Best Practices

Organizational Alignment

  • SLO Setting: Collaborative target setting between product and engineering
  • Alert Ownership: Clear escalation paths and team responsibilities
  • Dashboard Governance: Centralized dashboard management and standards
  • Training Programs: Team education on observability tools and practices

Technical Excellence

  • Infrastructure as Code: Observability configuration version control
  • Testing Strategy: Alert rule testing and dashboard validation
  • Performance Monitoring: Observability system performance tracking
  • Security Considerations: Access control and data privacy in observability

Continuous Improvement

  • Metrics Review: Regular SLI/SLO effectiveness assessment
  • Alert Tuning: Ongoing alert threshold and routing optimization
  • Dashboard Evolution: User feedback-driven dashboard improvements
  • Tool Evaluation: Regular assessment of observability tool effectiveness

04 同仓库更多 Skills

GongYuanCaiJi
GongYuanCaiJi/dsh-claude-skills

syllabus

Generates a curated supplementary reading list from any course syllabus using Consensus academic search. Grill-me intake (syllabus input format + course audience + year range) plus a grouping forcing-options checkpoint before any search runs — so the reading list matches the course's level and recency need. Parses the syllabus to extract topics and learning outcomes, searches Consensus for recent peer-reviewed papers per topic, and produces a professionally formatted .docx with clickable Consensus links, plain-language summaries calibrated to audience level, and Bloom-higher-order discussion questions tied to course learning goals. Use when the user uploads a syllabus, course outline, or curriculum document and wants supplementary readings (e.g., 'create a reading list from this syllabus', 'find recent papers for my course') — even casual mentions with a syllabus attached should trigger this skill.

★ 0
GongYuanCaiJi
GongYuanCaiJi/dsh-claude-skills

research

Default entry point for any research request — a hybrid router that classifies the question deterministically and either delegates to a specialist research skill (pulse for trends/sentiment, grants for NIH funding, litreview for academic literature, syllabus for course reading, patent for prior-art + IP landscape, dossier for entity research) or runs its own plan-decompose-multi-source-search-synthesize-cite fallback workflow when no specialist matches. Always surfaces the routing decision so users can override. Use when the user makes any research request that doesn't obviously match a more-specific specialist skill (e.g., "research [topic]", "look into [topic]", "what do we know about [topic]", "investigate [topic]", "find me information on [topic]", "do some research on [topic]", "I need to understand [topic]"). Output is a markdown briefing (default) or .docx document (on request) with full citations and an audit log.

★ 0
GongYuanCaiJi
GongYuanCaiJi/dsh-claude-skills

pulse

Multi-source recency research skill that takes the pulse of any topic across Reddit, Hacker News, the open web, and optionally X/Twitter within a configurable recent window (default 30 days). Forcing intake clarifies topic specificity, angle (trend/sentiment/problems/opportunities/comparison), time window, and platform scope before searching. Returns a synthesized briefing with citations, engagement metrics, and cross-platform pattern analysis. Use when the user requests multi-source recency intelligence on a topic (e.g., 'pulse on [topic]', 'what's happening with [topic]', 'what are people saying about [topic]', 'current conversation about [topic]', 'take the pulse of [topic]', 'trending: [topic]', 'find me info on [topic]'), and for competitor research, trend discovery, tool comparisons, and audience sentiment analysis.

★ 0
GongYuanCaiJi
GongYuanCaiJi/dsh-claude-skills

patent

Patent prior-art and landscape intelligence skill — not generic patent help. Commits to one of five sub-use-cases via forcing intake (novelty search / freedom-to-operate / competitive landscape / acquisition diligence / litigation prior-art) before any search runs. Searches Google Patents, Espacenet, USPTO, and optionally Lens.org for citation-graph signals. Output is an editable Word document (.docx) with verdict, ranked closest art (claim-text extracted), CPC-class-aware landscape, family-resolved hits, geographic coverage, FTO flags where applicable, strategy recommendations, and full audit log. Use when the user asks for patent searching or analysis (e.g., 'prior art search for [invention]', 'freedom to operate analysis for [product]'). Produces search signal, not legal advice — always recommends consulting a patent attorney before filing or licensing decisions. Trademark, copyright, and trade-secret questions are out of scope.

★ 0
GongYuanCaiJi
GongYuanCaiJi/dsh-claude-skills

notebooklm

Browser automation skill for controlling Google's NotebookLM. Use when the user wants anything done in NotebookLM (e.g., 'open NotebookLM', 'check my [name] notebook', 'ask my notebook about X', 'add [source] to NotebookLM', 'generate a Video Overview from my notebook', 'use NotebookLM Studio'). Handles reading and querying notebooks, adding sources (URLs, text, files, YouTube links, synthesized content), generating Studio outputs (Audio/Video Overviews, Mind Maps, Reports incl. Briefing Doc/Study Guide/FAQ, Flashcards, Quiz, slide decks, infographics — discover the exact set from the live Studio panel; the UI evolves fast), and creating new notebooks. Requires browser automation environment — fails gracefully when unavailable.

★ 0
GongYuanCaiJi
GongYuanCaiJi/dsh-claude-skills

litreview

Academic literature orientation skill that searches papers via free keyless APIs (PubMed E-utilities + OpenAlex) by default — with the Consensus MCP as an optional enhancement lane when connected — builds a strategic search plan using PICO (default) or SPIDER / Decomposition / hybrid as fallbacks, and synthesizes findings into a formatted Word (.docx) research guide. Grill-me intake (research question specificity + framework hint + tentative depth) before the recon search; a second forcing checkpoint after Phase 2 confirms framework + sub-areas + depth before searches consume budget. Configurable depth (5/10/20 queries) controls coverage vs. speed. Output is a 'launching pad' — an orientation guide that lets a researcher dive in confidently, not a finished review. Use when the user starts literature-oriented research (e.g., 'litreview on [topic]', 'literature review on [topic]', 'I'm starting a literature review on X', 'I'm writing a paper on X', 'help me research X', 'I'm doing research on X', 'can you help me research X'). Do NOT use for single one-off paper searches wanting a quick list — that's a plain PubMed/OpenAlex (or Consensus) query.

★ 0