[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M73D5SWRJ5AAWTQZY683YC57":3,"skill-same:01M73D5SWRJ5AAWTQZY683YC57":65},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":37,"commitSha":38,"skillDirectory":39,"forksCount":40,"repositoryHtmlUrl":41,"repositoryHomepageUrl":6,"ownerHtmlUrl":42,"ownerBlogUrl":43,"ownerDescription":6,"updatedAt":44,"install":45,"resources":58},"01M73D5SWRJ5AAWTQZY683YC57","write-incident-runbook",null,"Create structured incident runbooks with diagnostic steps, resolution procedures, escalation paths, and communication templates for effective incident response. Use when documenting response procedures for recurring alerts, standardizing incident response across an on-call rotation, reducing MTTR with clear diagnostic steps, creating training materials for new team members, or linking alert annotations directly to resolution procedures.","roohe\u002Fagentic-super-skills","roohe","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F99054084?v=4","01M734WCDFP6RZVT5CNM00QR26","01M734WCE4BD0T4NZS5PCV9V5F",10,"MIT","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fwrite-incident-runbook\u002FSKILL.md",[],[],"2026-11-11 15:11:14",false,"---\nname: write-incident-runbook\nlocale: caveman-lite\nsource_locale: en\nsource_commit: 82c77053\ntranslator: \"Julius Brussee homage — caveman\"\ntranslation_date: \"2026-05-03\"\ndescription: >\n  Create structured incident runbooks with diagnostic steps, resolution procedures, escalation\n  paths, and communication templates for effective incident response. Use when documenting\n  response procedures for recurring alerts, standardizing incident response across an on-call\n  rotation, reducing MTTR with clear diagnostic steps, creating training materials for new\n  team members, or linking alert annotations directly to resolution procedures.\nlicense: MIT\nallowed-tools: Read Write Edit Bash Grep Glob\nmetadata:\n  author: Philipp Thoss\n  version: \"1.0\"\n  domain: observability\n  complexity: basic\n  language: multi\n  tags: runbook, incident-response, diagnostics, escalation, documentation\n---\n\n# Write Incident Runbook\n\nCreate actionable runbooks that guide responders through incident diagnosis and resolution.\n\n## When to Use\n\n- Documenting response procedures for recurring alerts or incidents\n- Standardizing incident response across on-call rotation members\n- Reducing mean time to resolution (MTTR) with clear diagnostic steps\n- Creating training materials for new team members on incident handling\n- Establishing escalation paths and communication protocols\n- Migrating tribal knowledge to written documentation\n- Linking alerts to resolution procedures (alert annotations)\n\n## Inputs\n\n- **Required**: Incident or alert name\u002Fdescription\n- **Required**: Historical incident data and resolution patterns\n- **Optional**: Diagnostic queries (Prometheus, logs, traces)\n- **Optional**: Escalation contacts and communication channels\n- **Optional**: Previous incident post-mortems\n\n## Procedure\n\n### Step 1: Choose Runbook Template Structure\n\n> See [Extended Examples](references\u002FEXAMPLES.md#step-1-runbook-template-examples) for complete template files.\n\nSelect an appropriate template based on incident type and complexity.\n\n**Basic runbook template structure**:\n```markdown\n# [Alert\u002FIncident Name] Runbook\n## Overview | Severity | Symptoms\n## Diagnostic Steps | Resolution Steps\n## Escalation | Communication | Prevention | Related\n```\n\n**Advanced SRE runbook template** (excerpt):\n```markdown\n# [Service Name] - [Incident Type] Runbook\n\n## Metadata\n- Service, Owner, Severity, On-Call, Last Updated\n\n## Diagnostic Phase\n### Quick Health Check (\u003C 5 min): Dashboard, error rate, deployments\n### Detailed Investigation (5-20 min): Metrics, logs, traces, failure patterns\n# ... (see EXAMPLES.md for complete template)\n```\n\nKey template components:\n- **Metadata**: Service ownership, severity, on-call rotation\n- **Diagnostic Phase**: Quick checks → detailed investigation → failure patterns\n- **Resolution Phase**: Immediate mitigation → root cause fix → verification\n- **Escalation**: Criteria and contact paths\n- **Communication**: Internal\u002Fexternal templates\n- **Prevention**: Short\u002Flong-term actions\n\n**Got:** Template selected matches incident complexity, sections appropriate for service type.\n\n**If fail:**\n- Start with basic template, iterate based on incident patterns\n- Review industry examples (Google SRE books, vendor runbooks)\n- Adapt template based on team feedback after first use\n\n### Step 2: Document Diagnostic Procedures\n\n> See [Extended Examples](references\u002FEXAMPLES.md#step-2-complete-diagnostic-procedures) for complete diagnostic queries and decision trees.\n\nCreate step-by-step investigation procedures with specific queries.\n\n**Six-step diagnostic checklist**:\n\n1. **Verify Service Health**: Health endpoint checks and uptime metrics\n   ```bash\n   curl -I https:\u002F\u002Fapi.example.com\u002Fhealth  # Expected: HTTP 200 OK\n   ```\n   ```promql\n   up{job=\"api-service\"}  # Expected: 1 for all instances\n   ```\n\n2. **Check Error Rate**: Current error percentage and breakdown by endpoint\n   ```promql\n   sum(rate(http_requests_total{status=~\"5..\"}[5m]))\n   \u002F sum(rate(http_requests_total[5m])) * 100  # Expected: \u003C 1%\n   ```\n\n3. **Analyze Logs**: Recent errors and top error messages from Loki\n   ```logql\n   {job=\"api-service\"} |= \"error\" | json | level=\"error\"\n   ```\n\n4. **Check Resource Utilization**: CPU, memory, and connection pool status\n   ```promql\n   avg(rate(container_cpu_usage_seconds_total{pod=~\"api-service.*\"}[5m])) * 100\n   # Expected: \u003C 70%\n   ```\n\n5. **Review Recent Changes**: Deployments, git commits, infrastructure changes\n\n6. **Examine Dependencies**: Downstream service health, database\u002FAPI latency\n\n**Failure pattern decision tree** (excerpt):\n- Service down? → Check all pods\u002Finstances\n- Error rate elevated? → Check specific error types (5xx, gateway, database, timeouts)\n- When did it start? → After deployment (rollback), gradual (resource leak), sudden (traffic\u002Fdependency)\n\n**Got:** Diagnostic procedures are specific, include expected vs actual values, guide responder through investigation.\n\n**If fail:**\n- Test queries in actual monitoring system before documenting\n- Include screenshots of dashboards for visual reference\n- Add \"Common mistakes\" section for frequently missed steps\n- Iterate based on feedback from incident responders\n\n### Step 3: Define Resolution Procedures\n\n> See [Extended Examples](references\u002FEXAMPLES.md#step-3-complete-resolution-procedures) for all 5 resolution options with full commands and rollback procedures.\n\nDocument step-by-step remediation with rollback options.\n\n**Five resolution options** (brief summary):\n\n1. **Rollback Deployment** (fastest): For post-deployment errors\n   ```bash\n   kubectl rollout undo deployment\u002Fapi-service\n   ```\n   Verify → Monitor → Confirm resolution (error rate \u003C 1%, latency normal, no alerts)\n\n2. **Scale Up Resources**: For high CPU\u002Fmemory, connection pool exhaustion\n   ```bash\n   kubectl scale deployment\u002Fapi-service --replicas=$((current * 3\u002F2))\n   ```\n\n3. **Restart Service**: For memory leaks, stuck connections, cache corruption\n   ```bash\n   kubectl rollout restart deployment\u002Fapi-service\n   ```\n\n4. **Feature Flag \u002F Circuit Breaker**: For specific feature errors or external dependency failures\n   ```bash\n   kubectl set env deployment\u002Fapi-service FEATURE_NAME=false\n   ```\n\n5. **Database Remediation**: For database connections, slow queries, pool exhaustion\n   ```sql\n   -- Kill long-running queries, restart connection pool, increase pool size\n   ```\n\n**Universal verification checklist**:\n- [ ] Error rate \u003C 1%\n- [ ] Latency P99 \u003C threshold\n- [ ] Throughput at baseline\n- [ ] Resource usage healthy (CPU \u003C 70%, Memory \u003C 80%)\n- [ ] Dependencies healthy\n- [ ] User-facing tests pass\n- [ ] No active alerts\n\n**Rollback procedure**: If resolution worsens situation → pause\u002Fcancel → revert → reassess\n\n**Got:** Resolution steps are clear, include verification checks, provide rollback options for each action.\n\n**If fail:**\n- Add more granular steps for complex procedures\n- Include screenshots or diagrams for multi-step processes\n- Document command outputs (expected vs actual)\n- Create separate runbook for complex resolution procedures\n\n### Step 4: Establish Escalation Paths\n\n> See [Extended Examples](references\u002FEXAMPLES.md#step-4-complete-escalation-guidelines) for full escalation levels and contact directory template.\n\nDefine when and how to escalate incidents.\n\n**When to escalate immediately**:\n- Customer-facing outage > 15 minutes\n- SLO error budget > 10% depleted\n- Data loss\u002Fcorruption or security breach suspected\n- Unable to identify root cause within 20 minutes\n- Mitigation attempts fail or worsen situation\n\n**Five escalation levels**:\n1. **Primary On-Call** (5 min response): Deploy fixes, rollback, scale (up to 30 min solo)\n2. **Secondary On-Call** (auto after 15 min): Additional investigation support\n3. **Team Lead** (architectural decisions): Database changes, vendor escalation, incidents > 1 hour\n4. **Incident Commander** (cross-team coord): Multiple teams, customer comms, incidents > 2 hours\n5. **Executive** (C-level): Major impact (>50% users), SLA breach, media\u002FPR, outages > 4 hours\n\n**Escalation process**:\n1. Notify target with: current status, impact, actions taken, help needed, dashboard link\n2. Handoff if needed: share timeline, actions, access, remain available\n3. Don't go silent: update every 15 min, ask questions, provide feedback\n\n**Contact directory**: Maintain table with role, Slack, phone, PagerDuty for:\n- Platform\u002FDatabase\u002FSecurity\u002FNetwork teams\n- Incident Commander\n- External vendors (AWS, database vendor, CDN provider)\n\n**Got:** Clear criteria for escalation, contact information readily accessible, escalation paths aligned with organizational structure.\n\n**If fail:**\n- Validate contact information is current (test quarterly)\n- Add decision tree for when to escalate\n- Include examples of escalation messages\n- Document response time expectations for each level\n\n### Step 5: Create Communication Templates\n\n> See [Extended Examples](references\u002FEXAMPLES.md#step-5-complete-communication-templates) for all internal and external templates with full formatting.\n\nProvide pre-written messages for incident updates.\n\n**Internal templates** (Slack #incident-response):\n\n1. **Initial Declaration**:\n   ```\n   🚨 INCIDENT: [Title] | Severity: [Critical\u002FHigh\u002FMedium]\n   Impact: [users\u002Fservices] | Owner: @username | Dashboard: [link]\n   Quick Summary: [1-2 sentences] | Next update: 15 min\n   ```\n\n2. **Progress Update** (every 15-30 min):\n   ```\n   📊 UPDATE #N | Status: [Investigating\u002FMitigating\u002FMonitoring]\n   Actions: [what we tried and outcomes]\n   Theory: [what we think is happening]\n   Next: [planned actions]\n   ```\n\n3. **Mitigation Complete**:\n   ```\n   ✅ MITIGATION | Metrics: Error [before→after], Latency [before→after]\n   Root Cause: [brief or \"investigating\"] | Monitoring 30min before resolved\n   ```\n\n4. **Resolution**:\n   ```\n   🎉 RESOLVED | Duration: [time] | Root Cause + Impact + Follow-up actions\n   ```\n\n5. **False Alarm**: No impact, no follow-up needed\n\n**External templates** (status page):\n- **Initial**: Investigating, started time, next update in 15 min\n- **Progress**: Identified cause (customer-friendly), implementing fix, estimated resolution\n- **Resolution**: Resolved time, root cause (simple), duration, prevention measures\n\n**Customer email template**: Timeline, impact description, resolution, prevention, compensation (if applicable)\n\n**Got:** Templates save time during incidents, ensure consistent communication, reduce cognitive load on responders.\n\n**If fail:**\n- Customize templates to match company communication style\n- Pre-fill templates with common incident types\n- Create Slack workflow\u002Fbot to populate templates automatically\n- Review templates during incident retrospectives\n\n### Step 6: Link Runbook to Monitoring\n\n> See [Extended Examples](references\u002FEXAMPLES.md#step-6-alert-integration-examples) for complete Prometheus alert configuration and Grafana dashboard JSON.\n\nIntegrate runbook with alerts and dashboards.\n\n**Add runbook links to Prometheus alerts**:\n```yaml\n- alert: HighErrorRate\n  annotations:\n    runbook_url: \"https:\u002F\u002Fwiki.example.com\u002Frunbooks\u002Fhigh-error-rate\"\n    dashboard_url: \"https:\u002F\u002Fgrafana.example.com\u002Fd\u002Fservice-overview\"\n    incident_channel: \"#incident-platform\"\n```\n\n**Embed quick diagnostic links in runbook**:\n- Service Overview Dashboard\n- Error Rate Last 1h (Prometheus direct link)\n- Recent Error Logs (Loki\u002FGrafana Explore)\n- Recent Deployments (GitHub\u002FCI)\n- PagerDuty Incidents\n\n**Create Grafana dashboard panel** with runbook links (markdown panel listing all incident runbooks with on-call and escalation info)\n\n**Got:** Responders can access runbooks directly from alerts or dashboards, diagnostic queries pre-filled, one-click access to relevant tools.\n\n**If fail:**\n- Verify runbook URLs are accessible without VPN\u002Flogin\n- Use URL shorteners for complex Grafana\u002FPrometheus links\n- Test links quarterly to ensure they don't break\n- Create browser bookmarks for frequently used runbooks\n\n## Validation\n\n- [ ] Runbook follows consistent template structure\n- [ ] Diagnostic procedures include specific queries and expected values\n- [ ] Resolution steps are actionable with clear commands\n- [ ] Escalation criteria and contacts are current\n- [ ] Communication templates provided for internal and external audiences\n- [ ] Runbook linked from monitoring alerts and dashboards\n- [ ] Runbook tested during incident simulation or actual incident\n- [ ] Feedback from responders incorporated into runbook\n- [ ] Revision history tracked with dates and authors\n- [ ] Runbook accessible without authentication (or cached offline)\n\n## Pitfalls\n\n- **Too generic**: Runbooks with vague steps like \"check the logs\" without specific queries are not actionable. Be specific.\n- **Outdated information**: Runbooks referencing old systems or commands become useless. Review quarterly.\n- **No verification steps**: Resolution without verification leads to false positives. Always include \"how to confirm it's fixed.\"\n- **Missing rollback procedures**: Every action should have a rollback plan. Don't trap responders in worse state.\n- **Assuming knowledge**: Runbooks for experts only exclude junior engineers. Write for the least experienced person on rotation.\n- **No ownership**: Runbooks without owners become stale. Assign team\u002Fperson responsible for updates.\n- **Hidden behind auth**: Runbooks inaccessible during VPN\u002FSSO issues are useless during crisis. Cache copies or use public wiki.\n\n## Related Skills\n\n- `configure-alerting-rules` - Link runbooks to alert annotations for immediate access during incidents\n- `build-grafana-dashboards` - Embed runbook links in dashboards and diagnostic panels\n- `setup-prometheus-monitoring` - Include diagnostic queries from Prometheus in runbook procedures\n- `define-slo-sli-sla` - Reference SLO impact in incident severity classification\n","# Write Incident Runbook\n\nCreate actionable runbooks that guide responders through incident diagnosis and resolution.\n\n## When to Use\n\n- Documenting response procedures for recurring alerts or incidents\n- Standardizing incident response across on-call rotation members\n- Reducing mean time to resolution (MTTR) with clear diagnostic steps\n- Creating training materials for new team members on incident handling\n- Establishing escalation paths and communication protocols\n- Migrating tribal knowledge to written documentation\n- Linking alerts to resolution procedures (alert annotations)\n\n## Inputs\n\n- **Required**: Incident or alert name\u002Fdescription\n- **Required**: Historical incident data and resolution patterns\n- **Optional**: Diagnostic queries (Prometheus, logs, traces)\n- **Optional**: Escalation contacts and communication channels\n- **Optional**: Previous incident post-mortems\n\n## Procedure\n\n### Step 1: Choose Runbook Template Structure\n\n> See [Extended Examples](references\u002FEXAMPLES.md#step-1-runbook-template-examples) for complete template files.\n\nSelect an appropriate template based on incident type and complexity.\n\n**Basic runbook template structure**:\n```markdown\n# [Alert\u002FIncident Name] Runbook\n## Overview | Severity | Symptoms\n## Diagnostic Steps | Resolution Steps\n## Escalation | Communication | Prevention | Related\n```\n\n**Advanced SRE runbook template** (excerpt):\n```markdown\n# [Service Name] - [Incident Type] Runbook\n\n## Metadata\n- Service, Owner, Severity, On-Call, Last Updated\n\n## Diagnostic Phase\n### Quick Health Check (\u003C 5 min): Dashboard, error rate, deployments\n### Detailed Investigation (5-20 min): Metrics, logs, traces, failure patterns\n# ... (see EXAMPLES.md for complete template)\n```\n\nKey template components:\n- **Metadata**: Service ownership, severity, on-call rotation\n- **Diagnostic Phase**: Quick checks → detailed investigation → failure patterns\n- **Resolution Phase**: Immediate mitigation → root cause fix → verification\n- **Escalation**: Criteria and contact paths\n- **Communication**: Internal\u002Fexternal templates\n- **Prevention**: Short\u002Flong-term actions\n\n**Got:** Template selected matches incident complexity, sections appropriate for service type.\n\n**If fail:**\n- Start with basic template, iterate based on incident patterns\n- Review industry examples (Google SRE books, vendor runbooks)\n- Adapt template based on team feedback after first use\n\n### Step 2: Document Diagnostic Procedures\n\n> See [Extended Examples](references\u002FEXAMPLES.md#step-2-complete-diagnostic-procedures) for complete diagnostic queries and decision trees.\n\nCreate step-by-step investigation procedures with specific queries.\n\n**Six-step diagnostic checklist**:\n\n1. **Verify Service Health**: Health endpoint checks and uptime metrics\n   ```bash\n   curl -I https:\u002F\u002Fapi.example.com\u002Fhealth  # Expected: HTTP 200 OK\n   ```\n   ```promql\n   up{job=\"api-service\"}  # Expected: 1 for all instances\n   ```\n\n2. **Check Error Rate**: Current error percentage and breakdown by endpoint\n   ```promql\n   sum(rate(http_requests_total{status=~\"5..\"}[5m]))\n   \u002F sum(rate(http_requests_total[5m])) * 100  # Expected: \u003C 1%\n   ```\n\n3. **Analyze Logs**: Recent errors and top error messages from Loki\n   ```logql\n   {job=\"api-service\"} |= \"error\" | json | level=\"error\"\n   ```\n\n4. **Check Resource Utilization**: CPU, memory, and connection pool status\n   ```promql\n   avg(rate(container_cpu_usage_seconds_total{pod=~\"api-service.*\"}[5m])) * 100\n   # Expected: \u003C 70%\n   ```\n\n5. **Review Recent Changes**: Deployments, git commits, infrastructure changes\n\n6. **Examine Dependencies**: Downstream service health, database\u002FAPI latency\n\n**Failure pattern decision tree** (excerpt):\n- Service down? → Check all pods\u002Finstances\n- Error rate elevated? → Check specific error types (5xx, gateway, database, timeouts)\n- When did it start? → After deployment (rollback), gradual (resource leak), sudden (traffic\u002Fdependency)\n\n**Got:** Diagnostic procedures are specific, include expected vs actual values, guide responder through investigation.\n\n**If fail:**\n- Test queries in actual monitoring system before documenting\n- Include screenshots of dashboards for visual reference\n- Add \"Common mistakes\" section for frequently missed steps\n- Iterate based on feedback from incident responders\n\n### Step 3: Define Resolution Procedures\n\n> See [Extended Examples](references\u002FEXAMPLES.md#step-3-complete-resolution-procedures) for all 5 resolution options with full commands and rollback procedures.\n\nDocument step-by-step remediation with rollback options.\n\n**Five resolution options** (brief summary):\n\n1. **Rollback Deployment** (fastest): For post-deployment errors\n   ```bash\n   kubectl rollout undo deployment\u002Fapi-service\n   ```\n   Verify → Monitor → Confirm resolution (error rate \u003C 1%, latency normal, no alerts)\n\n2. **Scale Up Resources**: For high CPU\u002Fmemory, connection pool exhaustion\n   ```bash\n   kubectl scale deployment\u002Fapi-service --replicas=$((current * 3\u002F2))\n   ```\n\n3. **Restart Service**: For memory leaks, stuck connections, cache corruption\n   ```bash\n   kubectl rollout restart deployment\u002Fapi-service\n   ```\n\n4. **Feature Flag \u002F Circuit Breaker**: For specific feature errors or external dependency failures\n   ```bash\n   kubectl set env deployment\u002Fapi-service FEATURE_NAME=false\n   ```\n\n5. **Database Remediation**: For database connections, slow queries, pool exhaustion\n   ```sql\n   -- Kill long-running queries, restart connection pool, increase pool size\n   ```\n\n**Universal verification checklist**:\n- [ ] Error rate \u003C 1%\n- [ ] Latency P99 \u003C threshold\n- [ ] Throughput at baseline\n- [ ] Resource usage healthy (CPU \u003C 70%, Memory \u003C 80%)\n- [ ] Dependencies healthy\n- [ ] User-facing tests pass\n- [ ] No active alerts\n\n**Rollback procedure**: If resolution worsens situation → pause\u002Fcancel → revert → reassess\n\n**Got:** Resolution steps are clear, include verification checks, provide rollback options for each action.\n\n**If fail:**\n- Add more granular steps for complex procedures\n- Include screenshots or diagrams for multi-step processes\n- Document command outputs (expected vs actual)\n- Create separate runbook for complex resolution procedures\n\n### Step 4: Establish Escalation Paths\n\n> See [Extended Examples](references\u002FEXAMPLES.md#step-4-complete-escalation-guidelines) for full escalation levels and contact directory template.\n\nDefine when and how to escalate incidents.\n\n**When to escalate immediately**:\n- Customer-facing outage > 15 minutes\n- SLO error budget > 10% depleted\n- Data loss\u002Fcorruption or security breach suspected\n- Unable to identify root cause within 20 minutes\n- Mitigation attempts fail or worsen situation\n\n**Five escalation levels**:\n1. **Primary On-Call** (5 min response): Deploy fixes, rollback, scale (up to 30 min solo)\n2. **Secondary On-Call** (auto after 15 min): Additional investigation support\n3. **Team Lead** (architectural decisions): Database changes, vendor escalation, incidents > 1 hour\n4. **Incident Commander** (cross-team coord): Multiple teams, customer comms, incidents > 2 hours\n5. **Executive** (C-level): Major impact (>50% users), SLA breach, media\u002FPR, outages > 4 hours\n\n**Escalation process**:\n1. Notify target with: current status, impact, actions taken, help needed, dashboard link\n2. Handoff if needed: share timeline, actions, access, remain available\n3. Don't go silent: update every 15 min, ask questions, provide feedback\n\n**Contact directory**: Maintain table with role, Slack, phone, PagerDuty for:\n- Platform\u002FDatabase\u002FSecurity\u002FNetwork teams\n- Incident Commander\n- External vendors (AWS, database vendor, CDN provider)\n\n**Got:** Clear criteria for escalation, contact information readily accessible, escalation paths aligned with organizational structure.\n\n**If fail:**\n- Validate contact information is current (test quarterly)\n- Add decision tree for when to escalate\n- Include examples of escalation messages\n- Document response time expectations for each level\n\n### Step 5: Create Communication Templates\n\n> See [Extended Examples](references\u002FEXAMPLES.md#step-5-complete-communication-templates) for all internal and external templates with full formatting.\n\nProvide pre-written messages for incident updates.\n\n**Internal templates** (Slack #incident-response):\n\n1. **Initial Declaration**:\n   ```\n   🚨 INCIDENT: [Title] | Severity: [Critical\u002FHigh\u002FMedium]\n   Impact: [users\u002Fservices] | Owner: @username | Dashboard: [link]\n   Quick Summary: [1-2 sentences] | Next update: 15 min\n   ```\n\n2. **Progress Update** (every 15-30 min):\n   ```\n   📊 UPDATE #N | Status: [Investigating\u002FMitigating\u002FMonitoring]\n   Actions: [what we tried and outcomes]\n   Theory: [what we think is happening]\n   Next: [planned actions]\n   ```\n\n3. **Mitigation Complete**:\n   ```\n   ✅ MITIGATION | Metrics: Error [before→after], Latency [before→after]\n   Root Cause: [brief or \"investigating\"] | Monitoring 30min before resolved\n   ```\n\n4. **Resolution**:\n   ```\n   🎉 RESOLVED | Duration: [time] | Root Cause + Impact + Follow-up actions\n   ```\n\n5. **False Alarm**: No impact, no follow-up needed\n\n**External templates** (status page):\n- **Initial**: Investigating, started time, next update in 15 min\n- **Progress**: Identified cause (customer-friendly), implementing fix, estimated resolution\n- **Resolution**: Resolved time, root cause (simple), duration, prevention measures\n\n**Customer email template**: Timeline, impact description, resolution, prevention, compensation (if applicable)\n\n**Got:** Templates save time during incidents, ensure consistent communication, reduce cognitive load on responders.\n\n**If fail:**\n- Customize templates to match company communication style\n- Pre-fill templates with common incident types\n- Create Slack workflow\u002Fbot to populate templates automatically\n- Review templates during incident retrospectives\n\n### Step 6: Link Runbook to Monitoring\n\n> See [Extended Examples](references\u002FEXAMPLES.md#step-6-alert-integration-examples) for complete Prometheus alert configuration and Grafana dashboard JSON.\n\nIntegrate runbook with alerts and dashboards.\n\n**Add runbook links to Prometheus alerts**:\n```yaml\n- alert: HighErrorRate\n  annotations:\n    runbook_url: \"https:\u002F\u002Fwiki.example.com\u002Frunbooks\u002Fhigh-error-rate\"\n    dashboard_url: \"https:\u002F\u002Fgrafana.example.com\u002Fd\u002Fservice-overview\"\n    incident_channel: \"#incident-platform\"\n```\n\n**Embed quick diagnostic links in runbook**:\n- Service Overview Dashboard\n- Error Rate Last 1h (Prometheus direct link)\n- Recent Error Logs (Loki\u002FGrafana Explore)\n- Recent Deployments (GitHub\u002FCI)\n- PagerDuty Incidents\n\n**Create Grafana dashboard panel** with runbook links (markdown panel listing all incident runbooks with on-call and escalation info)\n\n**Got:** Responders can access runbooks directly from alerts or dashboards, diagnostic queries pre-filled, one-click access to relevant tools.\n\n**If fail:**\n- Verify runbook URLs are accessible without VPN\u002Flogin\n- Use URL shorteners for complex Grafana\u002FPrometheus links\n- Test links quarterly to ensure they don't break\n- Create browser bookmarks for frequently used runbooks\n\n## Validation\n\n- [ ] Runbook follows consistent template structure\n- [ ] Diagnostic procedures include specific queries and expected values\n- [ ] Resolution steps are actionable with clear commands\n- [ ] Escalation criteria and contacts are current\n- [ ] Communication templates provided for internal and external audiences\n- [ ] Runbook linked from monitoring alerts and dashboards\n- [ ] Runbook tested during incident simulation or actual incident\n- [ ] Feedback from responders incorporated into runbook\n- [ ] Revision history tracked with dates and authors\n- [ ] Runbook accessible without authentication (or cached offline)\n\n## Pitfalls\n\n- **Too generic**: Runbooks with vague steps like \"check the logs\" without specific queries are not actionable. Be specific.\n- **Outdated information**: Runbooks referencing old systems or commands become useless. Review quarterly.\n- **No verification steps**: Resolution without verification leads to false positives. Always include \"how to confirm it's fixed.\"\n- **Missing rollback procedures**: Every action should have a rollback plan. Don't trap responders in worse state.\n- **Assuming knowledge**: Runbooks for experts only exclude junior engineers. Write for the least experienced person on rotation.\n- **No ownership**: Runbooks without owners become stale. Assign team\u002Fperson responsible for updates.\n- **Hidden behind auth**: Runbooks inaccessible during VPN\u002FSSO issues are useless during crisis. Cache copies or use public wiki.\n\n## Related Skills\n\n- `configure-alerting-rules` - Link runbooks to alert annotations for immediate access during incidents\n- `build-grafana-dashboards` - Embed runbook links in dashboards and diagnostic panels\n- `setup-prometheus-monitoring` - Include diagnostic queries from Prometheus in runbook procedures\n- `define-slo-sli-sla` - Reference SLO impact in incident severity classification\n",{"name":5,"locale":23,"license":14,"metadata":24,"translator":31,"description":32,"allowed-tools":33,"source_commit":34,"source_locale":35,"translation_date":36},"caveman-lite",{"tags":25,"author":26,"domain":27,"version":28,"language":29,"complexity":30},"runbook, incident-response, diagnostics, escalation, documentation","Philipp Thoss","observability","1.0","multi","basic","Julius Brussee homage — caveman","Create structured incident runbooks with diagnostic steps, resolution procedures, escalation paths, and communication templates for effective incident response. Use when documenting response procedures for recurring alerts, standardizing incident response across an on-call rotation, reducing MTTR with clear diagnostic steps, creating training materials for new team members, or linking alert annotations directly to resolution procedures.\n","Read Write Edit Bash Grep Glob","82c77053","en","2026-05-03","master","f483e54f3ad5f202a77d45f2e57f8d970b71195f","skills_library\u002Fwrite-incident-runbook",2,"https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills","https:\u002F\u002Fgithub.com\u002Froohe","yoou.link","2026-05-19 15:56:07",{"source":8,"skillName":5,"commands":46,"prompt":56,"zipAvailable":57},[47,50,53],{"tool":48,"command":49},"npx","npx skills add roohe\u002Fagentic-super-skills --skill write-incident-runbook",{"tool":51,"command":52},"bunx","bunx skills add roohe\u002Fagentic-super-skills --skill write-incident-runbook",{"tool":54,"command":55},"pnpm","pnpm dlx skills add roohe\u002Fagentic-super-skills --skill write-incident-runbook","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fwrite-incident-runbook\u002FSKILL.md",true,[59],{"path":60,"resourceType":61,"resourceTypeDesc":62,"sizeBytes":63,"sourceHtmlUrl":64},"references\u002FEXAMPLES.md","REFERENCE","参考文档",24883,"https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fwrite-incident-runbook\u002Freferences\u002FEXAMPLES.md",[66,73,80,87,94,101],{"skillNo":67,"name":68,"nameZh":6,"description":69,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":70,"categories":71,"keywords":72,"publishedAt":18},"01M73D865CZME2FCMKA4PQM348","zarr-python","Chunked N-D arrays for cloud storage. Compressed arrays, parallel I\u002FO, S3\u002FGCS integration, NumPy\u002FDask\u002FXarray compatible, for large-scale scientific computing pipelines.","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fzarr-python\u002FSKILL.md",[],[],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":77,"categories":78,"keywords":79,"publishedAt":18},"01M73D7Z64J4ABWZJD7P44XBBF","youtube-downloader","Download and process YouTube content for research. Use when: downloading competitor videos for analysis; extracting audio for podcasts; getting transcripts for content repurposing; archiving webinars; research content curation","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fyoutube-downloader\u002FSKILL.md",[],[],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":18},"01M73D7K8778TGZV3GJFR3703D","yc-pitch-deck","Create a compelling startup pitch deck that follows the structure proven to raise billions from top investors. Master the YC and Sequoia formats that get founders funded. Use when: **Fundraising** to create investor pitch decks; **YC application** to structure your narrative; **Demo Day prep** to craft your 2-minute pitch; **Angel investors** to communicate your opportunity; **Partnership pitches** to structure compelling asks","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fyc-pitch-deck\u002FSKILL.md",[],[],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":18},"01M73D70KJSE0E8Z28YMNNRT04","x-twitter-scraper","Use when the user needs X (Twitter) data or confirmation-gated X actions through Xquik: tweet search, user lookup, follower extraction, media download, monitoring, webhooks, MCP, SDKs, posting, likes, DMs, and profile updates. Requires a Xquik API key. Never ask for X login material.","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fx-twitter-scraper\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":18},"01M73D6ED01GV077R7K4NZNFZJ","write-vignette","Create R package vignettes using R Markdown or Quarto. Covers vignette setup, YAML configuration, code chunk options, building and testing, and CRAN requirements for vignettes. Use when adding a Getting Started tutorial, documenting complex workflows spanning multiple functions, creating domain-specific guides, or when CRAN submission requires user-facing documentation beyond function help pages.","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fwrite-vignette\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":18},"01M73D69YJR0BK3NFHE0CYG3EF","write-validation-documentation","Write IQ\u002FOQ\u002FPQ validation documentation for computerized systems in regulated environments. Covers protocols, reports, test scripts, deviation handling, and approval workflows. Use when validating R or other software for regulated use, preparing for a regulatory audit, documenting qualification of computing environments, or creating and updating validation protocols and reports for new or re-qualified systems.","https:\u002F\u002Fgithub.com\u002Froohe\u002Fagentic-super-skills\u002Fblob\u002Fmaster\u002Fskills_library\u002Fwrite-validation-documentation\u002FSKILL.md",[],[]]