backend-tail-latency
Investigate backend P99/P999 tail latency — coordinated GC, lock contention, queue depth, slow downstream, retry storms; surface ONE dominant cause.
- License 许可
- MIT
- 最近更新
- 2026-10-29
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- backend-tail-latency
- description
- Investigate backend P99/P999 tail latency — coordinated GC, lock contention, queue depth, slow downstream, retry storms; surface ONE dominant cause.
03 SKILL.md 正文
backend-tail-latency
P50 latency is what the median user sees; P99 is what your worst-day user sees. Tail problems hide in coordinated phenomena, not single slow lines.
Mode: analytical — judgment inside a fixed frame. Keep the output shape; every verdict cites its evidence; unknowns are marked [TBD] or [VERIFY], never filled in.
When to use
- P99 / P999 latency is materially worse than P50 and you don't know why.
- A service is meeting P50 SLO but failing P99 SLO.
Inputs
- The service + the affected route(s).
- Recent P50, P95, P99 percentiles and the timeframe of the regression.
- Available signals: flamegraphs, trace samples, GC logs, DB slow-query logs, dependency latencies.
Procedure
Walk these in order; each is a common tail-latency root cause. Stop when one is the dominant signal (not when one is "plausible").
- Coordinated GC / runtime pauses:
- Look at GC pause histograms. Tail-of-tail (P99.9) is usually GC if pauses > 50ms.
- Action: tune heap; switch GC algo (G1 / ZGC / Shenandoah); reduce allocation in hot path.
- Lock contention:
- Profile shows threads parked. Per-thread CPU is low but latency is high.
- Action: shorten critical section; introduce read-write split; pool partitioning.
- Queue depth ahead of you:
- Worker pool / DB connection pool / external service queue.
- Look at queue depth + wait time histograms. P99 wait often > P99 service.
- Action: scale pool; cap upstream concurrency; add backpressure.
- Slow downstream:
- Trace samples show the long span isn't your code — it's a downstream call.
- Look at downstream P99 latency for the affected timeframe; compare to baseline.
- Action: timeout aggressively; hedged requests (only for idempotent reads, or writes protected by an idempotency key — hedging a non-idempotent call duplicates it); fall back; add cache.
- Retry storms / cascade:
- 5xx rate downstream spikes; your tail balloons because retries pile on.
- Action: token-bucket retry budget; circuit breaker; back off exponentially.
- Cold path / cold cache:
- First request after deploy / autoscaler spin-up is slow; warm requests fast.
- Action: pre-warm on deploy; smaller batch sizes; concurrent prefetch.
- Single hot key / hot partition:
- One key in a cache / DB partition absorbs disproportionate traffic.
- Action: read-through with local LRU; partition by user; rate-limit per key.
Output
Tail-latency investigation: <service>/<route>
Current state:
P50: 60 ms P95: 180 ms P99: 1,400 ms P99.9: 4,200 ms
Signals reviewed:
- GC pause histogram (max 80 ms; not the dominant signal)
- Lock contention via async-profiler (parked threads 8% of CPU)
- DB connection pool (P99 wait 22 ms; not it)
- Stripe-client P99 (3,800 ms during peak — match!)
- Retry storm (no — Stripe-client retries off)
Dominant cause:
Downstream Stripe-client P99 is 3,800 ms. Our P99 is 1,400 ms because we
apply a 1s timeout, dropping the worst Stripe-client cases. Our P99.9
catches the un-timeout-able subset.
First fix (single change):
Move the Stripe refund call off the request path: enqueue it (with an
idempotency key) and return 202 + status polling. The user-facing P99
then depends on our enqueue time, not Stripe's tail.
(Not hedging: a refund POST is not idempotent without the key — a hedge
would issue two refunds — and doubling calls to a slow dependency deepens
its overload.)
Expected effect: measure in staging — P99/P99.9 for the route before and
after; don't quote a percentage before you have it.
Validation:
Confirm the gain in staging before rolling to prod canary. Watch
Stripe-client error rate doesn't materially rise.
Quality bar
- One dominant cause named, with evidence.
- One first fix recommended, with the expected effect and how to validate it.
- Other signals reviewed are listed (so reviewer can argue with the elimination).
Anti-patterns to avoid
- "Might be GC, might be locks, might be downstream" — narrow it down.
- Recommending 3 fixes at once. Pick the highest-leverage one; validate; iterate.
- Hedging or retrying a non-idempotent call (payments, sends, creates) to cut latency.
- Tuning GC because GC is famous without checking the GC pause histogram first.
- Adding cache without first checking the actual hit rate ceiling.
Related skills
polymath-performance:app-profiling— when the dominant cause is CPU, allocation, or lock cost inside the service.polymath-performance:high-load-io-design— when it is pool/queue sizing, connection reuse, or missing backpressure.polymath-database:db-performance-triage— when the slow span is the database.
04 同仓库更多 Skills
runbook
Write or refresh an operational runbook for an alert/symptom; structured as TL;DR + Pre-checks + Steps + Verification + Rollback + Escalation.
rfc
Draft a lightweight design RFC to docs/rfcs/<slug>.md; Summary, Motivation, Detailed design, Drawbacks, Alternatives, Unresolved questions. A proposal to debate, not a recorded decision (adr).
editorial-pass
Review and tighten prose in a Markdown artifact for a target tone (terse, formal, friendly) without dropping information; preserve quotations and load-bearing constraints.
architecture-review-facilitation
Run an architecture review as a process — entry criteria, reviewer checklist, decision log, and follow-up actions — pairing the design artifact with a red-team pass.
architecture-doc
Write a C4-style architecture document; System Context + Containers (+ Components when interesting) with mermaid diagrams. Describes structure, not a decision (adr) or proposal (rfc).
adr
Write an Architecture Decision Record (Michael Nygard format) to docs/adrs/NNNN-<slug>.md; frontmatter validated by the ADR artifact schema. Records a made decision, not a proposal to debate (rfc).