[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M78BFP6BHCRFHHF45S7S8MVP":3,"skill-same:01M78BFP6BHCRFHHF45S7S8MVP":54},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":23,"commitSha":24,"skillDirectory":25,"forksCount":26,"repositoryHtmlUrl":27,"repositoryHomepageUrl":6,"ownerHtmlUrl":28,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":29,"install":30,"resources":43},"01M78BFP6BHCRFHHF45S7S8MVP","observability-monitoring",null,"Design, audit, and troubleshoot production monitoring and observability using user-impact checks, layered telemetry, USE\u002FRED, SLI\u002FSLO\u002FSLA, error budgets, cardinality controls, actionable alerting, burn-rate response, and postmortems. Use when asked about monitoring, наблюдаемость, алерты, Prometheus, Grafana, OpenTelemetry, logs, traces, profiles, service health, or incident evidence. Do not use for generic dashboard styling, frontend-only UI work, or unrelated code review.","hermes-jarvis-bot\u002Fhermes-agent-config-kit","hermes-jarvis-bot","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F288284026?v=4","01M78AV3BBD4M67J3YWSQRKP1Q","01M78AV3C2SKH53E0PZMXAKTTS",1,"MIT","https:\u002F\u002Fgithub.com\u002Fhermes-jarvis-bot\u002Fhermes-agent-config-kit\u002Fblob\u002Fmain\u002Fupstream\u002Fclaude-code-config\u002Fsnapshot\u002Fskills\u002Foperational\u002Fobservability-monitoring\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: observability-monitoring\ndescription: \"Design, audit, and troubleshoot production monitoring and observability using user-impact checks, layered telemetry, USE\u002FRED, SLI\u002FSLO\u002FSLA, error budgets, cardinality controls, actionable alerting, burn-rate response, and postmortems. Use when asked about monitoring, наблюдаемость, алерты, Prometheus, Grafana, OpenTelemetry, logs, traces, profiles, service health, or incident evidence. Do not use for generic dashboard styling, frontend-only UI work, or unrelated code review.\"\n---\n\n# Observability Monitoring\n\nUse this skill to turn vague \"is it working?\" questions into evidence-backed monitoring, alerting, and incident workflows. Start from user or business impact, then move down through the system layers and choose the signal that can prove the current hypothesis.\n\n## Operating rule\n\nDo not treat a green dashboard as proof of health. A monitoring claim is complete only when it names:\n\n1. the observed scope and time window;\n2. the user, business, or operator outcome being protected;\n3. the signal and exact query\u002Fprobe that supports the claim;\n4. the threshold or SLO that defines bad;\n5. the next human action and its runbook\u002Fevidence link.\n\nKeep code review, live runtime proof, UI\u002Frender proof, and release readiness as separate verdicts.\n\nInvestigation is read-only by default. A restart, alert suppression, metric-schema\u002Flabel change, sampling or retention change, or vendor reconfiguration is a production mutation: involve the responsible owner or incident authority, preserve the relevant evidence first, capture the exact config\u002Fcommand diff, state the rollback condition, and verify the user probe plus SLI after the change.\n\n## Workflow\n\n### 1. Freeze scope and collect live facts\n\nBefore changing a monitor, alert, host, or service:\n\n- Read the repository `AGENTS.md`, relevant rules, runbooks, and deployment docs.\n- Establish the actual checkout\u002Fbranch, deployment, process, host, port, proxy\u002Ftunnel, and data source.\n- Record the observation time, environment, query\u002Fprobe, and whether the result is current or historical.\n- Inspect the source code\u002Fconfig that emits or consumes the signal before changing it.\n- Never infer health from an old screenshot, a stale handoff, a single PID, or a dashboard with no successful user probe.\n\nFor infrastructure fixes, document the traffic path (DNS, proxy, tunnel, ingress, service, backend) before touching a surprising value such as `127.0.0.1`, a non-default port, or a disabled check.\n\n### 2. Define the outcome before the metric\n\nWrite the protected outcome in concrete terms:\n\n- **User:** can open the page, submit the request, download the result, or complete the workflow.\n- **Business:** orders, successful jobs, conversions, revenue, or another domain event continue to occur.\n- **Operator:** an on-call engineer receives a page early enough to act and can identify the next step.\n\nAdd at least one black-box or synthetic check for the user-facing path. Add real-user telemetry when the experience can vary by browser, geography, device, or network. Technical resource health is not a substitute for a user or business signal.\n\n### 3. Map the monitored layers\n\nInspect the layers from bottom to top and state which ones are in scope:\n\n1. **Hardware\u002Finfrastructure:** CPU, memory, disk capacity and latency, network, temperature, power, GPU, host availability.\n2. **Host\u002FOS:** load, processes, file descriptors, swap, service state, kernel\u002Fresource pressure.\n3. **Network:** reachability, packet loss, latency, port state, DNS, TLS, ingress\u002Fproxy health.\n4. **Application\u002FAPM:** request rate, errors, latency, saturation, dependency calls, queue depth.\n5. **Databases and queues:** throughput, slow queries, connection pools, replication lag, backlog, consumer health.\n6. **Containers\u002Forchestration:** desired versus ready replicas, restarts, scheduling, evictions, resource limits, ephemeral identity.\n7. **Business:** successful workflows, jobs, orders, revenue, conversion, or domain-specific zero-activity checks.\n\nDo not stop at the infrastructure layer when the business outcome is failing. A system with green CPU and memory can still have a broken payment, form, queue consumer, or model job.\n\n### 4. Choose the right signal\n\nUse the signal that answers the question instead of collecting everything indiscriminately:\n\n| Question | Primary signal | Practical method |\n|---|---|---|\n| Is a resource busy, queued, or failing? | Metrics | USE: utilization, saturation, errors |\n| Is a service serving users correctly? | Metrics | RED: rate, errors, duration |\n| What happened at a specific time? | Structured logs | Search by timestamp, service, severity, request\u002Ftrace ID |\n| Where did a distributed request slow or fail? | Traces | Follow the trace across services and spans |\n| Why is code or a process consuming resources? | Profiles | Inspect sampled stacks, CPU, memory, locks, or I\u002FO |\n| Do users actually succeed? | Synthetic\u002FRUM\u002Fbusiness probes | Run the workflow and inspect the domain result |\n\nMetrics answer aggregate **how much\u002Fhow often**. Logs answer **what happened**. Traces answer **where in the path**. Profiles answer **which code or process consumed the resource**. Correlate them with stable resource identity, timestamps, and trace\u002Frequest context.\n\n### 5. Design metrics without cardinality accidents\n\nTreat every unique metric-name plus label set as a time series. Before adding a label, estimate its possible values and lifetime.\n\n- Good metric dimensions are bounded and useful for aggregation: method, status class\u002Fcode, service, region, queue, job type, or environment.\n- Keep high-cardinality or per-event values in logs\u002Ftraces: user ID, request ID, email, raw URL with parameters, exception text, or arbitrary object IDs.\n- Prefer route templates over raw URLs.\n- Set explicit cardinality limits or views where the telemetry SDK supports them, and monitor overflow\u002Fdropped-attribute indicators.\n- Prefer histograms for fleet-wide aggregatable latency SLOs. Client-side summaries\u002Fquantiles generally cannot be aggregated across instances without changing their meaning; use them only when that limitation is understood. Never hide tail latency behind an average.\n\nIf a query needs a unique ID to be useful, that ID belongs in a trace or structured log field, not in a metric label.\n\n### 6. Define SLI, SLO, SLA, and error budget\n\nUse the terms precisely:\n\n- **SLI:** a measured indicator, such as successful requests divided by valid requests or p95 latency for a workflow.\n- **SLO:** the internal target and time window for that indicator.\n- **SLA:** an external agreement with consequences if the target is missed.\n- **Error budget:** the allowed unreliability implied by the SLO during the window.\n\nChoose the SLI from the user-facing outcome, not from whichever metric is easiest to collect. Define the valid-request population, exclusions, time window, aggregation, and owner. Treat an SLO as a control loop: remaining budget informs release pace, risk, testing, and reliability work.\n\n### 7. Build actionable alerts\n\nAlert on symptoms that indicate user or service pain. Use dashboards and drill-downs to investigate causes.\n\nEvery paging alert must include:\n\n- symptom and affected scope;\n- exact query, threshold, duration, and evaluation window;\n- severity and owner;\n- a link to the dashboard\u002Fquery and runbook;\n- concrete first actions, rollback criteria, and escalation path;\n- a test or smoke procedure proving the alert route works.\n\nIf no immediate action exists, make it a dashboard annotation, ticket, or recording rule instead of a page. Use a pending duration (`for`) to suppress short blips. If using hysteresis, define separate fire and clear thresholds; if using multi-window evaluation, define each window and its condition. Do not confuse either control with Slack\u002Fchat routing. Treat \"more than two incidents per shift\" as a review heuristic, not a universal law: frequent pages mean the system or alert policy needs repair.\n\nPrefer multi-window or burn-rate alerts for SLOs when the backend supports them. A burn rate near 1 spends the budget at the planned rate; a materially higher rate requires faster response. Verify the math against the actual SLO window and alert implementation.\n\n### Portable examples\n\nAdapt metric and label names to the live schema; these are patterns, not copy-paste production queries:\n\n```promql\n# RED error ratio for one service over five minutes.\nsum(rate(http_requests_total{service=\"checkout\",code=~\"5..\"}[5m]))\n\u002F\nsum(rate(http_requests_total{service=\"checkout\"}[5m]))\n\n# Aggregatable fleet-wide p95 from a histogram.\nhistogram_quantile(0.95,\n  sum by (le, service) (\n    rate(http_request_duration_seconds_bucket{service=\"checkout\"}[5m])\n  )\n)\n```\n\nUse a synthetic\u002Fdomain probe that asserts the real outcome, not only HTTP reachability. Prefer a read-only endpoint or a sandbox\u002Ftest tenant. If a write path is unavoidable, require a dry-run or idempotency key, a bounded fixture, explicit cleanup, and an owner-approved change window: `submit safe test request -> assert expected status and domain result\u002Fjob ID -> record latency and trace ID -> clean up`. Never send an arbitrary representative request into production.\n\nA burn rate is `observed error ratio \u002F allowed error ratio`, where `allowed error ratio = 1 - SLO`. Worked example: for an SLO of 99.9% (`allowed = 0.001`), a 5-minute observed error ratio of 1.0% (`0.010`) gives burn rate `10`; if the local paging policy is the illustrative `>=10 for 5m AND >=5 for 1h`, page, otherwise route it according to the lower-severity policy. The query pattern is:\n\n```promql\n(\n  sum(rate(http_request_errors_total{service=\"checkout\"}[5m]))\n  \u002F\n  sum(rate(http_requests_total{service=\"checkout\"}[5m]))\n) \u002F 0.001\n```\n\nThe thresholds and windows are examples only; derive and test them from the real SLO, traffic population, exclusions, and paging budget.\n\n### 8. Triage in evidence order\n\nFor a live incident, follow this sequence and keep a short timeline. Apply the change-control boundary above before any mutation:\n\n1. Confirm the user\u002Fbusiness symptom with a fresh probe.\n2. Bound blast radius: service, region, tenant, job type, version, and start time.\n3. Check RED for the affected service and USE for saturated dependencies\u002Fresources.\n4. Correlate logs by time, service, severity, and trace\u002Frequest ID.\n5. Follow a failing or slow trace to the first bad span\u002Fdependency.\n6. Use a profile only when resource\u002Fcode attribution remains unclear or the issue is intermittent.\n7. Compare deploys, config changes, feature flags, capacity, and external dependencies.\n8. Apply the smallest reversible mitigation with an explicit rollback condition.\n9. Re-run the user probe and the relevant SLI query; confirm recovery over a meaningful window.\n10. Record the evidence, unresolved uncertainty, and follow-up owner.\n\nDo not restart or reconfigure a service merely because a check is red. First establish whether the check is stale, misrouted, a false positive, or a real symptom, and preserve the evidence needed to explain the decision.\n\n### 9. Close the loop\n\nAfter recovery:\n\n- Verify the original symptom is gone, not only that the process is alive.\n- Record detection time, acknowledgement, mitigation, recovery, and customer impact.\n- Review alert quality: actionable, correctly routed, deduplicated, and linked to evidence.\n- Write a blameless postmortem for material incidents: timeline, impact, contributing\u002Fsystemic causes, what went well, what failed, and tracked prevention items.\n- Update the runbook, monitor, test, or architecture so the same failure is easier to detect and diagnose next time.\n\n## Vendor mapping\n\nUse the actual stack discovered in the repository\u002Fruntime. Common roles from the source video are:\n\n- Prometheus or VictoriaMetrics: scrape and store metrics;\n- Grafana: visualize and query multiple sources;\n- Loki or Elasticsearch\u002FOpenSearch: logs;\n- Jaeger or Tempo: traces;\n- OpenTelemetry: vendor-neutral collection and export of telemetry signals;\n- Mimir or Thanos: longer-term or larger-scale metric storage;\n- Zabbix\u002FNagios: host and infrastructure checks.\n\nHistorical names in the video (ping, syslog, SNMP, MRTG\u002FRRD, Nagios, and Cacti; 00:55-03:17) explain the evolution of monitoring and are not default deployment recommendations.\n\nThese are roles, not defaults. Do not install, replace, or reconfigure a vendor component without checking the live architecture, documentation, compatibility, retention, cost, authorization, and rollback path.\n\n## Minimal runbook template\n\nUse this shape for every new page:\n\n```text\nName:\nUser symptom:\nSLI\u002Fquery:\nTrigger and duration:\nScope labels:\nSeverity\u002Fowner:\nDashboard and raw query:\nFirst safe action:\nRollback condition:\nVerification probe:\nEscalation:\nKnown false positives:\nLast tested:\n```\n\n## Gotchas\n\n- A host can be healthy while the business workflow is broken.\n- Averages hide tail latency; inspect percentiles and failed requests.\n- A metric label creates a new series for every unique combination; raw URLs and IDs can exhaust memory.\n- A trace without stable service\u002Fresource context is hard to correlate; a log without timestamps or IDs is weak evidence.\n- A page without a concrete action trains the operator to ignore pages.\n- A synthetics-only check misses real-user variance; RUM-only telemetry can discover pain too late.\n- Profiles are an attribution signal, not a replacement for service-level metrics, logs, or traces.\n- An SLO target is not an SLA promise unless an external contract says so.\n- Auto-generated captions and copied vendor names can be wrong; verify terms before putting them into code or runbooks. See `references\u002Fsource-notes.md`.\n\n## Troubleshooting\n\n| Symptom | Likely cause | Fix |\n|---|---|---|\n| Dashboard is green but users fail | Missing black-box\u002Fbusiness SLI, wrong route, or stale data source | Run a fresh user probe, verify timestamps\u002Flabels, add the missing outcome signal |\n| One deploy creates a huge series spike | High-cardinality label or unbounded route\u002FID | Remove the label from metrics, normalize routes, move detail to logs\u002Ftraces, cap SDK cardinality |\n| Alert fires but nobody acts | Alert is a cause guess, too sensitive, unowned, or lacks a runbook | Alert on a user symptom, add owner\u002Faction\u002Fquery\u002Frunbook, add `for`; if using hysteresis define separate fire\u002Fclear thresholds, and if using multi-window define each condition; test delivery |\n| Error rate rises but cause is unclear | Logs are unstructured or traces are not correlated | Add structured fields and trace IDs, sample a failing trace, inspect dependency spans |\n| Service is slow but RED looks normal | Low traffic, bad aggregation, or resource contention outside the app | Check synthetic latency, USE for host\u002Fqueue\u002Fstorage, tail percentiles, and profiles |\n| Dashboard is blank or stale | Target\u002Fscrape\u002Fexporter\u002Fcollector\u002Fbackend failure, retention gap, or clock skew | Check last-sample time, target and collector health, dropped\u002Fingestion counters, retention, and time synchronization; do not interpret blank as zero |\n| A query shows zero but the system may be uninstrumented | The series is absent, filtered out, or genuinely zero | Test absent-series semantics, inspect raw labels and last sample, and add an explicit freshness\u002Ftelemetry-health check |\n| Trace is incomplete | Sampling, context propagation, collector drops, or backend retention | Check trace ID propagation, sampling decision, collector drop counters, backend ingestion, and clock synchronization |\n| Page storm during one incident | No grouping\u002Fdeduplication or too many low-value alerts | Group by incident\u002Fservice, keep one page for the symptom, demote diagnostic signals |\n| Burn-rate alert disagrees with the dashboard | Different SLI population, window, exclusions, or recording rule | Reconcile the numerator\u002Fdenominator, window, and query; test against known scenarios |\n| Restart appears to fix it but it returns | Mitigation hid a systemic cause or destroyed evidence | Capture logs\u002Fmetrics first, record the restart, identify the recurring trigger, add prevention |\n\n## Source and current-practice notes\n\nThe reusable concepts in this skill were extracted from the supplied video and cross-checked against current primary documentation. See `references\u002Fsource-notes.md` for the source URL, timestamp map, caption corrections, and official references.\n","# Observability Monitoring\n\nUse this skill to turn vague \"is it working?\" questions into evidence-backed monitoring, alerting, and incident workflows. Start from user or business impact, then move down through the system layers and choose the signal that can prove the current hypothesis.\n\n## Operating rule\n\nDo not treat a green dashboard as proof of health. A monitoring claim is complete only when it names:\n\n1. the observed scope and time window;\n2. the user, business, or operator outcome being protected;\n3. the signal and exact query\u002Fprobe that supports the claim;\n4. the threshold or SLO that defines bad;\n5. the next human action and its runbook\u002Fevidence link.\n\nKeep code review, live runtime proof, UI\u002Frender proof, and release readiness as separate verdicts.\n\nInvestigation is read-only by default. A restart, alert suppression, metric-schema\u002Flabel change, sampling or retention change, or vendor reconfiguration is a production mutation: involve the responsible owner or incident authority, preserve the relevant evidence first, capture the exact config\u002Fcommand diff, state the rollback condition, and verify the user probe plus SLI after the change.\n\n## Workflow\n\n### 1. Freeze scope and collect live facts\n\nBefore changing a monitor, alert, host, or service:\n\n- Read the repository `AGENTS.md`, relevant rules, runbooks, and deployment docs.\n- Establish the actual checkout\u002Fbranch, deployment, process, host, port, proxy\u002Ftunnel, and data source.\n- Record the observation time, environment, query\u002Fprobe, and whether the result is current or historical.\n- Inspect the source code\u002Fconfig that emits or consumes the signal before changing it.\n- Never infer health from an old screenshot, a stale handoff, a single PID, or a dashboard with no successful user probe.\n\nFor infrastructure fixes, document the traffic path (DNS, proxy, tunnel, ingress, service, backend) before touching a surprising value such as `127.0.0.1`, a non-default port, or a disabled check.\n\n### 2. Define the outcome before the metric\n\nWrite the protected outcome in concrete terms:\n\n- **User:** can open the page, submit the request, download the result, or complete the workflow.\n- **Business:** orders, successful jobs, conversions, revenue, or another domain event continue to occur.\n- **Operator:** an on-call engineer receives a page early enough to act and can identify the next step.\n\nAdd at least one black-box or synthetic check for the user-facing path. Add real-user telemetry when the experience can vary by browser, geography, device, or network. Technical resource health is not a substitute for a user or business signal.\n\n### 3. Map the monitored layers\n\nInspect the layers from bottom to top and state which ones are in scope:\n\n1. **Hardware\u002Finfrastructure:** CPU, memory, disk capacity and latency, network, temperature, power, GPU, host availability.\n2. **Host\u002FOS:** load, processes, file descriptors, swap, service state, kernel\u002Fresource pressure.\n3. **Network:** reachability, packet loss, latency, port state, DNS, TLS, ingress\u002Fproxy health.\n4. **Application\u002FAPM:** request rate, errors, latency, saturation, dependency calls, queue depth.\n5. **Databases and queues:** throughput, slow queries, connection pools, replication lag, backlog, consumer health.\n6. **Containers\u002Forchestration:** desired versus ready replicas, restarts, scheduling, evictions, resource limits, ephemeral identity.\n7. **Business:** successful workflows, jobs, orders, revenue, conversion, or domain-specific zero-activity checks.\n\nDo not stop at the infrastructure layer when the business outcome is failing. A system with green CPU and memory can still have a broken payment, form, queue consumer, or model job.\n\n### 4. Choose the right signal\n\nUse the signal that answers the question instead of collecting everything indiscriminately:\n\n| Question | Primary signal | Practical method |\n|---|---|---|\n| Is a resource busy, queued, or failing? | Metrics | USE: utilization, saturation, errors |\n| Is a service serving users correctly? | Metrics | RED: rate, errors, duration |\n| What happened at a specific time? | Structured logs | Search by timestamp, service, severity, request\u002Ftrace ID |\n| Where did a distributed request slow or fail? | Traces | Follow the trace across services and spans |\n| Why is code or a process consuming resources? | Profiles | Inspect sampled stacks, CPU, memory, locks, or I\u002FO |\n| Do users actually succeed? | Synthetic\u002FRUM\u002Fbusiness probes | Run the workflow and inspect the domain result |\n\nMetrics answer aggregate **how much\u002Fhow often**. Logs answer **what happened**. Traces answer **where in the path**. Profiles answer **which code or process consumed the resource**. Correlate them with stable resource identity, timestamps, and trace\u002Frequest context.\n\n### 5. Design metrics without cardinality accidents\n\nTreat every unique metric-name plus label set as a time series. Before adding a label, estimate its possible values and lifetime.\n\n- Good metric dimensions are bounded and useful for aggregation: method, status class\u002Fcode, service, region, queue, job type, or environment.\n- Keep high-cardinality or per-event values in logs\u002Ftraces: user ID, request ID, email, raw URL with parameters, exception text, or arbitrary object IDs.\n- Prefer route templates over raw URLs.\n- Set explicit cardinality limits or views where the telemetry SDK supports them, and monitor overflow\u002Fdropped-attribute indicators.\n- Prefer histograms for fleet-wide aggregatable latency SLOs. Client-side summaries\u002Fquantiles generally cannot be aggregated across instances without changing their meaning; use them only when that limitation is understood. Never hide tail latency behind an average.\n\nIf a query needs a unique ID to be useful, that ID belongs in a trace or structured log field, not in a metric label.\n\n### 6. Define SLI, SLO, SLA, and error budget\n\nUse the terms precisely:\n\n- **SLI:** a measured indicator, such as successful requests divided by valid requests or p95 latency for a workflow.\n- **SLO:** the internal target and time window for that indicator.\n- **SLA:** an external agreement with consequences if the target is missed.\n- **Error budget:** the allowed unreliability implied by the SLO during the window.\n\nChoose the SLI from the user-facing outcome, not from whichever metric is easiest to collect. Define the valid-request population, exclusions, time window, aggregation, and owner. Treat an SLO as a control loop: remaining budget informs release pace, risk, testing, and reliability work.\n\n### 7. Build actionable alerts\n\nAlert on symptoms that indicate user or service pain. Use dashboards and drill-downs to investigate causes.\n\nEvery paging alert must include:\n\n- symptom and affected scope;\n- exact query, threshold, duration, and evaluation window;\n- severity and owner;\n- a link to the dashboard\u002Fquery and runbook;\n- concrete first actions, rollback criteria, and escalation path;\n- a test or smoke procedure proving the alert route works.\n\nIf no immediate action exists, make it a dashboard annotation, ticket, or recording rule instead of a page. Use a pending duration (`for`) to suppress short blips. If using hysteresis, define separate fire and clear thresholds; if using multi-window evaluation, define each window and its condition. Do not confuse either control with Slack\u002Fchat routing. Treat \"more than two incidents per shift\" as a review heuristic, not a universal law: frequent pages mean the system or alert policy needs repair.\n\nPrefer multi-window or burn-rate alerts for SLOs when the backend supports them. A burn rate near 1 spends the budget at the planned rate; a materially higher rate requires faster response. Verify the math against the actual SLO window and alert implementation.\n\n### Portable examples\n\nAdapt metric and label names to the live schema; these are patterns, not copy-paste production queries:\n\n```promql\n# RED error ratio for one service over five minutes.\nsum(rate(http_requests_total{service=\"checkout\",code=~\"5..\"}[5m]))\n\u002F\nsum(rate(http_requests_total{service=\"checkout\"}[5m]))\n\n# Aggregatable fleet-wide p95 from a histogram.\nhistogram_quantile(0.95,\n  sum by (le, service) (\n    rate(http_request_duration_seconds_bucket{service=\"checkout\"}[5m])\n  )\n)\n```\n\nUse a synthetic\u002Fdomain probe that asserts the real outcome, not only HTTP reachability. Prefer a read-only endpoint or a sandbox\u002Ftest tenant. If a write path is unavoidable, require a dry-run or idempotency key, a bounded fixture, explicit cleanup, and an owner-approved change window: `submit safe test request -> assert expected status and domain result\u002Fjob ID -> record latency and trace ID -> clean up`. Never send an arbitrary representative request into production.\n\nA burn rate is `observed error ratio \u002F allowed error ratio`, where `allowed error ratio = 1 - SLO`. Worked example: for an SLO of 99.9% (`allowed = 0.001`), a 5-minute observed error ratio of 1.0% (`0.010`) gives burn rate `10`; if the local paging policy is the illustrative `>=10 for 5m AND >=5 for 1h`, page, otherwise route it according to the lower-severity policy. The query pattern is:\n\n```promql\n(\n  sum(rate(http_request_errors_total{service=\"checkout\"}[5m]))\n  \u002F\n  sum(rate(http_requests_total{service=\"checkout\"}[5m]))\n) \u002F 0.001\n```\n\nThe thresholds and windows are examples only; derive and test them from the real SLO, traffic population, exclusions, and paging budget.\n\n### 8. Triage in evidence order\n\nFor a live incident, follow this sequence and keep a short timeline. Apply the change-control boundary above before any mutation:\n\n1. Confirm the user\u002Fbusiness symptom with a fresh probe.\n2. Bound blast radius: service, region, tenant, job type, version, and start time.\n3. Check RED for the affected service and USE for saturated dependencies\u002Fresources.\n4. Correlate logs by time, service, severity, and trace\u002Frequest ID.\n5. Follow a failing or slow trace to the first bad span\u002Fdependency.\n6. Use a profile only when resource\u002Fcode attribution remains unclear or the issue is intermittent.\n7. Compare deploys, config changes, feature flags, capacity, and external dependencies.\n8. Apply the smallest reversible mitigation with an explicit rollback condition.\n9. Re-run the user probe and the relevant SLI query; confirm recovery over a meaningful window.\n10. Record the evidence, unresolved uncertainty, and follow-up owner.\n\nDo not restart or reconfigure a service merely because a check is red. First establish whether the check is stale, misrouted, a false positive, or a real symptom, and preserve the evidence needed to explain the decision.\n\n### 9. Close the loop\n\nAfter recovery:\n\n- Verify the original symptom is gone, not only that the process is alive.\n- Record detection time, acknowledgement, mitigation, recovery, and customer impact.\n- Review alert quality: actionable, correctly routed, deduplicated, and linked to evidence.\n- Write a blameless postmortem for material incidents: timeline, impact, contributing\u002Fsystemic causes, what went well, what failed, and tracked prevention items.\n- Update the runbook, monitor, test, or architecture so the same failure is easier to detect and diagnose next time.\n\n## Vendor mapping\n\nUse the actual stack discovered in the repository\u002Fruntime. Common roles from the source video are:\n\n- Prometheus or VictoriaMetrics: scrape and store metrics;\n- Grafana: visualize and query multiple sources;\n- Loki or Elasticsearch\u002FOpenSearch: logs;\n- Jaeger or Tempo: traces;\n- OpenTelemetry: vendor-neutral collection and export of telemetry signals;\n- Mimir or Thanos: longer-term or larger-scale metric storage;\n- Zabbix\u002FNagios: host and infrastructure checks.\n\nHistorical names in the video (ping, syslog, SNMP, MRTG\u002FRRD, Nagios, and Cacti; 00:55-03:17) explain the evolution of monitoring and are not default deployment recommendations.\n\nThese are roles, not defaults. Do not install, replace, or reconfigure a vendor component without checking the live architecture, documentation, compatibility, retention, cost, authorization, and rollback path.\n\n## Minimal runbook template\n\nUse this shape for every new page:\n\n```text\nName:\nUser symptom:\nSLI\u002Fquery:\nTrigger and duration:\nScope labels:\nSeverity\u002Fowner:\nDashboard and raw query:\nFirst safe action:\nRollback condition:\nVerification probe:\nEscalation:\nKnown false positives:\nLast tested:\n```\n\n## Gotchas\n\n- A host can be healthy while the business workflow is broken.\n- Averages hide tail latency; inspect percentiles and failed requests.\n- A metric label creates a new series for every unique combination; raw URLs and IDs can exhaust memory.\n- A trace without stable service\u002Fresource context is hard to correlate; a log without timestamps or IDs is weak evidence.\n- A page without a concrete action trains the operator to ignore pages.\n- A synthetics-only check misses real-user variance; RUM-only telemetry can discover pain too late.\n- Profiles are an attribution signal, not a replacement for service-level metrics, logs, or traces.\n- An SLO target is not an SLA promise unless an external contract says so.\n- Auto-generated captions and copied vendor names can be wrong; verify terms before putting them into code or runbooks. See `references\u002Fsource-notes.md`.\n\n## Troubleshooting\n\n| Symptom | Likely cause | Fix |\n|---|---|---|\n| Dashboard is green but users fail | Missing black-box\u002Fbusiness SLI, wrong route, or stale data source | Run a fresh user probe, verify timestamps\u002Flabels, add the missing outcome signal |\n| One deploy creates a huge series spike | High-cardinality label or unbounded route\u002FID | Remove the label from metrics, normalize routes, move detail to logs\u002Ftraces, cap SDK cardinality |\n| Alert fires but nobody acts | Alert is a cause guess, too sensitive, unowned, or lacks a runbook | Alert on a user symptom, add owner\u002Faction\u002Fquery\u002Frunbook, add `for`; if using hysteresis define separate fire\u002Fclear thresholds, and if using multi-window define each condition; test delivery |\n| Error rate rises but cause is unclear | Logs are unstructured or traces are not correlated | Add structured fields and trace IDs, sample a failing trace, inspect dependency spans |\n| Service is slow but RED looks normal | Low traffic, bad aggregation, or resource contention outside the app | Check synthetic latency, USE for host\u002Fqueue\u002Fstorage, tail percentiles, and profiles |\n| Dashboard is blank or stale | Target\u002Fscrape\u002Fexporter\u002Fcollector\u002Fbackend failure, retention gap, or clock skew | Check last-sample time, target and collector health, dropped\u002Fingestion counters, retention, and time synchronization; do not interpret blank as zero |\n| A query shows zero but the system may be uninstrumented | The series is absent, filtered out, or genuinely zero | Test absent-series semantics, inspect raw labels and last sample, and add an explicit freshness\u002Ftelemetry-health check |\n| Trace is incomplete | Sampling, context propagation, collector drops, or backend retention | Check trace ID propagation, sampling decision, collector drop counters, backend ingestion, and clock synchronization |\n| Page storm during one incident | No grouping\u002Fdeduplication or too many low-value alerts | Group by incident\u002Fservice, keep one page for the symptom, demote diagnostic signals |\n| Burn-rate alert disagrees with the dashboard | Different SLI population, window, exclusions, or recording rule | Reconcile the numerator\u002Fdenominator, window, and query; test against known scenarios |\n| Restart appears to fix it but it returns | Mitigation hid a systemic cause or destroyed evidence | Capture logs\u002Fmetrics first, record the restart, identify the recurring trigger, add prevention |\n\n## Source and current-practice notes\n\nThe reusable concepts in this skill were extracted from the supplied video and cross-checked against current primary documentation. See `references\u002Fsource-notes.md` for the source URL, timestamp map, caption corrections, and official references.\n",{"name":5,"description":7},"main","07d02526f424fc814f8350d6b0b728e81576603d","upstream\u002Fclaude-code-config\u002Fsnapshot\u002Fskills\u002Foperational\u002Fobservability-monitoring",0,"https:\u002F\u002Fgithub.com\u002Fhermes-jarvis-bot\u002Fhermes-agent-config-kit","https:\u002F\u002Fgithub.com\u002Fhermes-jarvis-bot","2026-11-14 18:25:27",{"source":8,"skillName":5,"commands":31,"prompt":41,"zipAvailable":42},[32,35,38],{"tool":33,"command":34},"npx","npx skills add hermes-jarvis-bot\u002Fhermes-agent-config-kit --skill observability-monitoring",{"tool":36,"command":37},"bunx","bunx skills add hermes-jarvis-bot\u002Fhermes-agent-config-kit --skill observability-monitoring",{"tool":39,"command":40},"pnpm","pnpm dlx skills add hermes-jarvis-bot\u002Fhermes-agent-config-kit --skill observability-monitoring","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhermes-jarvis-bot\u002Fhermes-agent-config-kit\u002Fblob\u002Fmain\u002Fupstream\u002Fclaude-code-config\u002Fsnapshot\u002Fskills\u002Foperational\u002Fobservability-monitoring\u002FSKILL.md",true,[44,50],{"path":45,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":48,"sourceHtmlUrl":49},"agents\u002Fopenai.yaml","REFERENCE","参考文档",291,"https:\u002F\u002Fgithub.com\u002Fhermes-jarvis-bot\u002Fhermes-agent-config-kit\u002Fblob\u002Fmain\u002Fupstream\u002Fclaude-code-config\u002Fsnapshot\u002Fskills\u002Foperational\u002Fobservability-monitoring\u002Fagents\u002Fopenai.yaml",{"path":51,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":52,"sourceHtmlUrl":53},"references\u002Fsource-notes.md",3829,"https:\u002F\u002Fgithub.com\u002Fhermes-jarvis-bot\u002Fhermes-agent-config-kit\u002Fblob\u002Fmain\u002Fupstream\u002Fclaude-code-config\u002Fsnapshot\u002Fskills\u002Foperational\u002Fobservability-monitoring\u002Freferences\u002Fsource-notes.md",[55,61,68,75,82,89],{"skillNo":56,"name":57,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":58,"categories":59,"keywords":60,"publishedAt":18},"01M78BHH6VNT0MKZXGMP2F50X1","owner-voice","https:\u002F\u002Fgithub.com\u002Fhermes-jarvis-bot\u002Fhermes-agent-config-kit\u002Fblob\u002Fmain\u002Fupstream\u002Fclaude-code-config\u002Fsnapshot\u002Fskills\u002Fwriting\u002Fowner-voice\u002FSKILL.md",[],[],{"skillNo":62,"name":63,"nameZh":6,"description":64,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":65,"categories":66,"keywords":67,"publishedAt":18},"01M78BHD34XJHAK6EMZ7J1M80X","humanize-russian","Редактировать русский текст для ясности и естественного звучания, сохраняя факты,\nтермины, источники и жанр. Использовать когда: убрать канцелярит, сделать текст\nживее, отредактировать русский черновик или статью перед публикацией.\nНе определяет авторство и не обещает обход детекторов. Для английского —\nhumanize-english; для структуры и доказательств — article-structure-review.","https:\u002F\u002Fgithub.com\u002Fhermes-jarvis-bot\u002Fhermes-agent-config-kit\u002Fblob\u002Fmain\u002Fupstream\u002Fclaude-code-config\u002Fsnapshot\u002Fskills\u002Fwriting\u002Fhumanize-russian\u002FSKILL.md",[],[],{"skillNo":69,"name":70,"nameZh":6,"description":71,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":72,"categories":73,"keywords":74,"publishedAt":18},"01M78BH9D09HCA7ECZMPEEAE5Y","humanize-english","Edit English drafts for clear, natural language while preserving facts, uncertainty,\ntechnical meaning, citations and genre. Use when: humanize this text, remove AI-style\nfiller, improve English prose, or edit an English article before publication.\nNot an authorship detector or a promise to bypass one. For Russian use\nhumanize-russian; for thesis\u002Fevidence and structure use article-structure-review.","https:\u002F\u002Fgithub.com\u002Fhermes-jarvis-bot\u002Fhermes-agent-config-kit\u002Fblob\u002Fmain\u002Fupstream\u002Fclaude-code-config\u002Fsnapshot\u002Fskills\u002Fwriting\u002Fhumanize-english\u002FSKILL.md",[],[],{"skillNo":76,"name":77,"nameZh":6,"description":78,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":81,"publishedAt":18},"01M78BH44Q25YMA16MC6FKZ2WY","caveman","Ultra-compressed reply mode. Cuts output tokens ~65-75% by answering in terse \"caveman\" style while keeping full technical accuracy, and shapes what survives: action first, state restated every turn, concrete time estimates, visible wins. Intensity levels: lite, full (default), ultra, wenyan-lite, wenyan-full, wenyan-ultra. Use when user says \"caveman\", \"talk like caveman\", \"use caveman\", \"less tokens\", \"be brief\", or Russian \"пещерный режим\", \"говори как пещерный\", \"кратко\", \"меньше токенов\", \"экономь токены\" — or invokes \u002Fcaveman. Preserves the user's language (Russian in → Russian caveman out). Off only: \"stop caveman\" \u002F \"normal mode\" \u002F \"обычный режим\". Do not use merely because a response should be concise, and do not use when the user requests normal prose, a polished document, or exact wording.","https:\u002F\u002Fgithub.com\u002Fhermes-jarvis-bot\u002Fhermes-agent-config-kit\u002Fblob\u002Fmain\u002Fupstream\u002Fclaude-code-config\u002Fsnapshot\u002Fskills\u002Fwriting\u002Fcaveman\u002FSKILL.md",[],[],{"skillNo":83,"name":84,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":18},"01M78BGY0JTMTW3J3YE5RQTHCK","article-structure-review","Проверить структуру технической статьи после черновика: задача читателя, жанр,\nопора значимых утверждений на доказательства и честные ограничения.\nUse after the first draft, before publication. Исправлять структуру, когда это\nвходит в запрос; не создавать обязательный конвейер из других навыков.\nДля локальной языковой редактуры — humanize-english или humanize-russian.","https:\u002F\u002Fgithub.com\u002Fhermes-jarvis-bot\u002Fhermes-agent-config-kit\u002Fblob\u002Fmain\u002Fupstream\u002Fclaude-code-config\u002Fsnapshot\u002Fskills\u002Fwriting\u002Farticle-structure-review\u002FSKILL.md",[],[],{"skillNo":90,"name":91,"nameZh":6,"description":92,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":93,"categories":94,"keywords":95,"publishedAt":18},"01M78BGRJYZGK1TNTCHWGY32W3","video-post-production","Video post-production rules: audio mastering, color, captions, platform export. Use when: 'add music', 'add voiceover', 'export for tiktok', 'add captions', 'color grade', 'audio levels', 'master audio', 'export settings', 'platform requirements'. Covers FFmpeg patterns, audio chain, subtitle standards, and platform-specific export configs. Do NOT use to generate or animate the video frames\u002Fscenes in code (use remotion-production-guide), to write the script (use video-narrative-arc), or to evaluate script quality (use script-evaluator); this is the finishing stage applied to an already-rendered video.","https:\u002F\u002Fgithub.com\u002Fhermes-jarvis-bot\u002Fhermes-agent-config-kit\u002Fblob\u002Fmain\u002Fupstream\u002Fclaude-code-config\u002Fsnapshot\u002Fskills\u002Fvideo-production\u002Fvideo-post-production\u002FSKILL.md",[],[]]