llm-cost-latency
Cut LLM cost and latency — model routing/cascade, prompt caching, batching, streaming, context trimming, output-token control, and a per-request cost/latency budget.
- License 许可
- MIT
- 最近更新
- 2026-10-29
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- llm-cost-latency
- description
- Cut LLM cost and latency — model routing/cascade, prompt caching, batching, streaming, context trimming, output-token control, and a per-request cost/latency budget.
02 资源文件 1
- 参考文档routing.yaml391 B
03 SKILL.md 正文
llm-cost-latency
LLM spend is dominated by tokens × model tier × call count. Most savings come from sending fewer tokens to a smaller model less often — measure the request first, then cut the biggest line.
Mode: analytical — judgment inside a fixed frame. Keep the output shape; every verdict cites its evidence; unknowns are marked [TBD] or [VERIFY], never filled in.
When to use
- An LLM feature is too slow or too expensive, or you are sizing one before launch.
- You need a per-request cost/latency budget and a path to hit it.
- A workflow invokes
polymath-ai:llm-cost-latency.
Not this when: this is a non-LLM service's latency/cost — that is polymath-performance:backend-tail-latency / perf-budget. For prompt wording quality, polymath-ai:prompt-engineer.
Inputs
- The current model, prompt sizes (system + context + output), call pattern, and measured cost/latency.
- The quality bar the cheaper path must still clear.
Procedure
- Account the request: break cost into input tokens, output tokens, and model tier; break latency into time-to-first-token, generation time, and tool/round-trips. Optimize the largest contributor, not the easiest.
- Right-size the model: route easy cases to a smaller/cheaper model and escalate only hard ones (a cascade). Reserve the top tier for tasks that measurably need it — verify with
polymath-ai:eval-planthat the cheaper model still passes. - Cut input tokens: trim/compress context, retrieve less but better (tune
rag-design), drop stale transcript, and reuse a stable system prompt to exploit prompt caching (order prefix-stable content first; check the provider's minimum cacheable length, cache TTL, and the write premium vs read discount, then measure the cache-hit rate). In agent loops, input cost grows with every turn because the history is re-sent — caching and summarization matter most there. - Cut output tokens: ask for the minimal format (structured/short), cap max output, and stop sequences — output tokens are priced several times higher than input and dominate generation latency.
- Cut call count: batch independent requests (asynchronous batch APIs trade latency for a large discount on non-interactive work), collapse multi-call chains where one call suffices, and cache deterministic results.
- Hide latency: stream tokens to first paint; parallelize independent tool calls; precompute where the input is known ahead of time.
- Set the budget + tripwire: a per-request cost and p95 latency target, with an alert when exceeded — the same discipline as a perf budget.
Output
LLM cost/latency plan: support-reply drafting
Per request (measured, 1,000-request sample):
input 6,200 tok (system 1,800 · retrieved 3,900 · history 500) × $3.00/M = $0.0186
output 450 tok × $15.00/M = $0.0068
total $0.0254/request · p95 latency 7.8 s (TTFT 1.1 s, generation 6.7 s)
largest line: retrieved context (61% of input cost)
Cuts (expected)
1. retrieve top-4 not top-10 chunks (eval: answer quality 91% → 90%, within noise) −$0.0070
2. prompt caching on the 1,800-token system prefix (hit rate 85% measured in staging) −$0.0041
3. cap output at 300 tokens + "3 short paragraphs" format −$0.0023, −2.2 s p95
4. route "FAQ-match" requests (38%) to the small tier after eval-plan passes on that slice
Streaming: on — first paint at TTFT
Budget + tripwire: ≤ $0.015/request and p95 ≤ 5 s; alert at 120% of either for 1 h
Prices are the provider's current list prices, cited with the date checked; an unknown price is [TBD: price], never remembered.
Quality bar
- The optimization targets the measured largest cost/latency contributor, not a guess.
- Any model downgrade is gated by an eval that shows quality holds.
- Prompt caching is exploited via prefix-stable prompt ordering where the provider supports it.
- A per-request budget + tripwire is named.
04 同仓库更多 Skills
runbook
Write or refresh an operational runbook for an alert/symptom; structured as TL;DR + Pre-checks + Steps + Verification + Rollback + Escalation.
rfc
Draft a lightweight design RFC to docs/rfcs/<slug>.md; Summary, Motivation, Detailed design, Drawbacks, Alternatives, Unresolved questions. A proposal to debate, not a recorded decision (adr).
editorial-pass
Review and tighten prose in a Markdown artifact for a target tone (terse, formal, friendly) without dropping information; preserve quotations and load-bearing constraints.
architecture-review-facilitation
Run an architecture review as a process — entry criteria, reviewer checklist, decision log, and follow-up actions — pairing the design artifact with a red-team pass.
architecture-doc
Write a C4-style architecture document; System Context + Containers (+ Components when interesting) with mermaid diagrams. Describes structure, not a decision (adr) or proposal (rfc).
adr
Write an Architecture Decision Record (Michael Nygard format) to docs/adrs/NNNN-<slug>.md; frontmatter validated by the ADR artifact schema. Records a made decision, not a proposal to debate (rfc).