[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M74GEFB9HM02TRK7DDACMYCB":3,"skill-same:01M74GEFB9HM02TRK7DDACMYCB":44},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":14,"repositoryHtmlUrl":27,"repositoryHomepageUrl":6,"ownerHtmlUrl":28,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":29,"install":30,"resources":43},"01M74GEFB9HM02TRK7DDACMYCB","backend-tail-latency",null,"Investigate backend P99\u002FP999 tail latency — coordinated GC, lock contention, queue depth, slow downstream, retry storms; surface ONE dominant cause.","MohammadBafkar\u002Fpolymath","MohammadBafkar","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F34940462?v=4","01M74G540TQFZWK6KVAPKFVGNR","Mohammad Bafkar","01M74G541HG19QP9V5DQCJVKYE",0,"MIT","https:\u002F\u002Fgithub.com\u002FMohammadBafkar\u002Fpolymath\u002Fblob\u002Fmain\u002Fplugins\u002Fpolymath-performance\u002Fskills\u002Fbackend-tail-latency\u002FSKILL.md",[],[],"2026-11-11 15:11:14",false,"---\nname: backend-tail-latency\ndescription: Investigate backend P99\u002FP999 tail latency — coordinated GC, lock contention, queue depth, slow downstream, retry storms; surface ONE dominant cause.\n---\n\n# backend-tail-latency\n\n> P50 latency is what the median user sees; P99 is what your worst-day user sees. Tail problems hide in coordinated phenomena, not single slow lines.\n\n**Mode:** analytical — judgment inside a fixed frame. Keep the output shape; every verdict cites its evidence; unknowns are marked `[TBD]` or `[VERIFY]`, never filled in.\n\n## When to use\n\n- P99 \u002F P999 latency is materially worse than P50 and you don't know why.\n- A service is meeting P50 SLO but failing P99 SLO.\n\n## Inputs\n\n- The service + the affected route(s).\n- Recent P50, P95, P99 percentiles and the timeframe of the regression.\n- Available signals: flamegraphs, trace samples, GC logs, DB slow-query logs, dependency latencies.\n\n## Procedure\n\nWalk these in order; each is a common tail-latency root cause. Stop when one is the dominant signal (not when one is \"plausible\").\n\n1. **Coordinated GC \u002F runtime pauses**:\n   - Look at GC pause histograms. Tail-of-tail (P99.9) is usually GC if pauses > 50ms.\n   - Action: tune heap; switch GC algo (G1 \u002F ZGC \u002F Shenandoah); reduce allocation in hot path.\n2. **Lock contention**:\n   - Profile shows threads parked. Per-thread CPU is low but latency is high.\n   - Action: shorten critical section; introduce read-write split; pool partitioning.\n3. **Queue depth ahead of you**:\n   - Worker pool \u002F DB connection pool \u002F external service queue.\n   - Look at queue depth + wait time histograms. P99 wait often > P99 service.\n   - Action: scale pool; cap upstream concurrency; add backpressure.\n4. **Slow downstream**:\n   - Trace samples show the long span isn't your code — it's a downstream call.\n   - Look at downstream P99 latency for the affected timeframe; compare to baseline.\n   - Action: timeout aggressively; hedged requests (only for idempotent reads, or writes protected by an idempotency key — hedging a non-idempotent call duplicates it); fall back; add cache.\n5. **Retry storms \u002F cascade**:\n   - 5xx rate downstream spikes; your tail balloons because retries pile on.\n   - Action: token-bucket retry budget; circuit breaker; back off exponentially.\n6. **Cold path \u002F cold cache**:\n   - First request after deploy \u002F autoscaler spin-up is slow; warm requests fast.\n   - Action: pre-warm on deploy; smaller batch sizes; concurrent prefetch.\n7. **Single hot key \u002F hot partition**:\n   - One key in a cache \u002F DB partition absorbs disproportionate traffic.\n   - Action: read-through with local LRU; partition by user; rate-limit per key.\n\n## Output\n\n```text\nTail-latency investigation: \u003Cservice>\u002F\u003Croute>\n\nCurrent state:\n  P50: 60 ms   P95: 180 ms   P99: 1,400 ms   P99.9: 4,200 ms\n\nSignals reviewed:\n  - GC pause histogram         (max 80 ms; not the dominant signal)\n  - Lock contention via async-profiler  (parked threads 8% of CPU)\n  - DB connection pool         (P99 wait 22 ms; not it)\n  - Stripe-client P99          (3,800 ms during peak — match!)\n  - Retry storm                (no — Stripe-client retries off)\n\nDominant cause:\n  Downstream Stripe-client P99 is 3,800 ms. Our P99 is 1,400 ms because we\n  apply a 1s timeout, dropping the worst Stripe-client cases. Our P99.9\n  catches the un-timeout-able subset.\n\nFirst fix (single change):\n  Move the Stripe refund call off the request path: enqueue it (with an\n  idempotency key) and return 202 + status polling. The user-facing P99\n  then depends on our enqueue time, not Stripe's tail.\n  (Not hedging: a refund POST is not idempotent without the key — a hedge\n  would issue two refunds — and doubling calls to a slow dependency deepens\n  its overload.)\n  Expected effect: measure in staging — P99\u002FP99.9 for the route before and\n  after; don't quote a percentage before you have it.\n\nValidation:\n  Confirm the gain in staging before rolling to prod canary. Watch\n  Stripe-client error rate doesn't materially rise.\n```\n\n## Quality bar\n\n- One dominant cause named, with evidence.\n- One first fix recommended, with the expected effect and how to validate it.\n- Other signals reviewed are listed (so reviewer can argue with the elimination).\n\n## Anti-patterns to avoid\n\n- \"Might be GC, might be locks, might be downstream\" — narrow it down.\n- Recommending 3 fixes at once. Pick the highest-leverage one; validate; iterate.\n- Hedging or retrying a non-idempotent call (payments, sends, creates) to cut latency.\n- Tuning GC because GC is famous without checking the GC pause histogram first.\n- Adding cache without first checking the actual hit rate ceiling.\n\n## Related skills\n\n- `polymath-performance:app-profiling` — when the dominant cause is CPU, allocation, or lock cost inside the service.\n- `polymath-performance:high-load-io-design` — when it is pool\u002Fqueue sizing, connection reuse, or missing backpressure.\n- `polymath-database:db-performance-triage` — when the slow span is the database.\n","# backend-tail-latency\n\n> P50 latency is what the median user sees; P99 is what your worst-day user sees. Tail problems hide in coordinated phenomena, not single slow lines.\n\n**Mode:** analytical — judgment inside a fixed frame. Keep the output shape; every verdict cites its evidence; unknowns are marked `[TBD]` or `[VERIFY]`, never filled in.\n\n## When to use\n\n- P99 \u002F P999 latency is materially worse than P50 and you don't know why.\n- A service is meeting P50 SLO but failing P99 SLO.\n\n## Inputs\n\n- The service + the affected route(s).\n- Recent P50, P95, P99 percentiles and the timeframe of the regression.\n- Available signals: flamegraphs, trace samples, GC logs, DB slow-query logs, dependency latencies.\n\n## Procedure\n\nWalk these in order; each is a common tail-latency root cause. Stop when one is the dominant signal (not when one is \"plausible\").\n\n1. **Coordinated GC \u002F runtime pauses**:\n   - Look at GC pause histograms. Tail-of-tail (P99.9) is usually GC if pauses > 50ms.\n   - Action: tune heap; switch GC algo (G1 \u002F ZGC \u002F Shenandoah); reduce allocation in hot path.\n2. **Lock contention**:\n   - Profile shows threads parked. Per-thread CPU is low but latency is high.\n   - Action: shorten critical section; introduce read-write split; pool partitioning.\n3. **Queue depth ahead of you**:\n   - Worker pool \u002F DB connection pool \u002F external service queue.\n   - Look at queue depth + wait time histograms. P99 wait often > P99 service.\n   - Action: scale pool; cap upstream concurrency; add backpressure.\n4. **Slow downstream**:\n   - Trace samples show the long span isn't your code — it's a downstream call.\n   - Look at downstream P99 latency for the affected timeframe; compare to baseline.\n   - Action: timeout aggressively; hedged requests (only for idempotent reads, or writes protected by an idempotency key — hedging a non-idempotent call duplicates it); fall back; add cache.\n5. **Retry storms \u002F cascade**:\n   - 5xx rate downstream spikes; your tail balloons because retries pile on.\n   - Action: token-bucket retry budget; circuit breaker; back off exponentially.\n6. **Cold path \u002F cold cache**:\n   - First request after deploy \u002F autoscaler spin-up is slow; warm requests fast.\n   - Action: pre-warm on deploy; smaller batch sizes; concurrent prefetch.\n7. **Single hot key \u002F hot partition**:\n   - One key in a cache \u002F DB partition absorbs disproportionate traffic.\n   - Action: read-through with local LRU; partition by user; rate-limit per key.\n\n## Output\n\n```text\nTail-latency investigation: \u003Cservice>\u002F\u003Croute>\n\nCurrent state:\n  P50: 60 ms   P95: 180 ms   P99: 1,400 ms   P99.9: 4,200 ms\n\nSignals reviewed:\n  - GC pause histogram         (max 80 ms; not the dominant signal)\n  - Lock contention via async-profiler  (parked threads 8% of CPU)\n  - DB connection pool         (P99 wait 22 ms; not it)\n  - Stripe-client P99          (3,800 ms during peak — match!)\n  - Retry storm                (no — Stripe-client retries off)\n\nDominant cause:\n  Downstream Stripe-client P99 is 3,800 ms. Our P99 is 1,400 ms because we\n  apply a 1s timeout, dropping the worst Stripe-client cases. Our P99.9\n  catches the un-timeout-able subset.\n\nFirst fix (single change):\n  Move the Stripe refund call off the request path: enqueue it (with an\n  idempotency key) and return 202 + status polling. The user-facing P99\n  then depends on our enqueue time, not Stripe's tail.\n  (Not hedging: a refund POST is not idempotent without the key — a hedge\n  would issue two refunds — and doubling calls to a slow dependency deepens\n  its overload.)\n  Expected effect: measure in staging — P99\u002FP99.9 for the route before and\n  after; don't quote a percentage before you have it.\n\nValidation:\n  Confirm the gain in staging before rolling to prod canary. Watch\n  Stripe-client error rate doesn't materially rise.\n```\n\n## Quality bar\n\n- One dominant cause named, with evidence.\n- One first fix recommended, with the expected effect and how to validate it.\n- Other signals reviewed are listed (so reviewer can argue with the elimination).\n\n## Anti-patterns to avoid\n\n- \"Might be GC, might be locks, might be downstream\" — narrow it down.\n- Recommending 3 fixes at once. Pick the highest-leverage one; validate; iterate.\n- Hedging or retrying a non-idempotent call (payments, sends, creates) to cut latency.\n- Tuning GC because GC is famous without checking the GC pause histogram first.\n- Adding cache without first checking the actual hit rate ceiling.\n\n## Related skills\n\n- `polymath-performance:app-profiling` — when the dominant cause is CPU, allocation, or lock cost inside the service.\n- `polymath-performance:high-load-io-design` — when it is pool\u002Fqueue sizing, connection reuse, or missing backpressure.\n- `polymath-database:db-performance-triage` — when the slow span is the database.\n",{"name":5,"description":7},"main","23f6e21841a39580c9da2b5ec941f39dda17cd86","plugins\u002Fpolymath-performance\u002Fskills\u002Fbackend-tail-latency","https:\u002F\u002Fgithub.com\u002FMohammadBafkar\u002Fpolymath","https:\u002F\u002Fgithub.com\u002FMohammadBafkar","2026-10-29 18:34:59",{"source":8,"skillName":5,"commands":31,"prompt":41,"zipAvailable":42},[32,35,38],{"tool":33,"command":34},"npx","npx skills add MohammadBafkar\u002Fpolymath --skill backend-tail-latency",{"tool":36,"command":37},"bunx","bunx skills add MohammadBafkar\u002Fpolymath --skill backend-tail-latency",{"tool":39,"command":40},"pnpm","pnpm dlx skills add MohammadBafkar\u002Fpolymath --skill backend-tail-latency","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FMohammadBafkar\u002Fpolymath\u002Fblob\u002Fmain\u002Fplugins\u002Fpolymath-performance\u002Fskills\u002Fbackend-tail-latency\u002FSKILL.md",true,[],[45,52,59,66,73,80],{"skillNo":46,"name":47,"nameZh":6,"description":48,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":49,"categories":50,"keywords":51,"publishedAt":19},"01M74GN87EBRHM9N87M7AP1HDD","runbook","Write or refresh an operational runbook for an alert\u002Fsymptom; structured as TL;DR + Pre-checks + Steps + Verification + Rollback + Escalation.","https:\u002F\u002Fgithub.com\u002FMohammadBafkar\u002Fpolymath\u002Fblob\u002Fmain\u002Fplugins\u002Fpolymath-writing\u002Fskills\u002Frunbook\u002FSKILL.md",[],[],{"skillNo":53,"name":54,"nameZh":6,"description":55,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":56,"categories":57,"keywords":58,"publishedAt":19},"01M74GN5GPKEC1JEKP2DFQNEQJ","rfc","Draft a lightweight design RFC to docs\u002Frfcs\u002F\u003Cslug>.md; Summary, Motivation, Detailed design, Drawbacks, Alternatives, Unresolved questions. A proposal to debate, not a recorded decision (adr).","https:\u002F\u002Fgithub.com\u002FMohammadBafkar\u002Fpolymath\u002Fblob\u002Fmain\u002Fplugins\u002Fpolymath-writing\u002Fskills\u002Frfc\u002FSKILL.md",[],[],{"skillNo":60,"name":61,"nameZh":6,"description":62,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":63,"categories":64,"keywords":65,"publishedAt":19},"01M74GN2F4CWPMV4X09TG4E304","editorial-pass","Review and tighten prose in a Markdown artifact for a target tone (terse, formal, friendly) without dropping information; preserve quotations and load-bearing constraints.","https:\u002F\u002Fgithub.com\u002FMohammadBafkar\u002Fpolymath\u002Fblob\u002Fmain\u002Fplugins\u002Fpolymath-writing\u002Fskills\u002Feditorial-pass\u002FSKILL.md",[],[],{"skillNo":67,"name":68,"nameZh":6,"description":69,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":70,"categories":71,"keywords":72,"publishedAt":19},"01M74GMZWGYMZ37C4DYGT6P23G","architecture-review-facilitation","Run an architecture review as a process — entry criteria, reviewer checklist, decision log, and follow-up actions — pairing the design artifact with a red-team pass.","https:\u002F\u002Fgithub.com\u002FMohammadBafkar\u002Fpolymath\u002Fblob\u002Fmain\u002Fplugins\u002Fpolymath-writing\u002Fskills\u002Farchitecture-review-facilitation\u002FSKILL.md",[],[],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":77,"categories":78,"keywords":79,"publishedAt":19},"01M74GMXVX0YX3ETDXS527XDGM","architecture-doc","Write a C4-style architecture document; System Context + Containers (+ Components when interesting) with mermaid diagrams. Describes structure, not a decision (adr) or proposal (rfc).","https:\u002F\u002Fgithub.com\u002FMohammadBafkar\u002Fpolymath\u002Fblob\u002Fmain\u002Fplugins\u002Fpolymath-writing\u002Fskills\u002Farchitecture-doc\u002FSKILL.md",[],[],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":19},"01M74GMVG1WSMCK066S3KE6V5K","adr","Write an Architecture Decision Record (Michael Nygard format) to docs\u002Fadrs\u002FNNNN-\u003Cslug>.md; frontmatter validated by the ADR artifact schema. Records a made decision, not a proposal to debate (rfc).","https:\u002F\u002Fgithub.com\u002FMohammadBafkar\u002Fpolymath\u002Fblob\u002Fmain\u002Fplugins\u002Fpolymath-writing\u002Fskills\u002Fadr\u002FSKILL.md",[],[]]