[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M757KHEZ7TFBYJW2HASTRZX5":3,"skill-same:01M757KHEZ7TFBYJW2HASTRZX5":52},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":28,"commitSha":29,"skillDirectory":30,"forksCount":31,"repositoryHtmlUrl":32,"repositoryHomepageUrl":33,"ownerHtmlUrl":34,"ownerBlogUrl":35,"ownerDescription":36,"updatedAt":37,"install":38,"resources":51},"01M757KHEZ7TFBYJW2HASTRZX5","inference-caching-and-kv",null,"Reference-grade guide to caching in LLM inference — provider prompt caching (Anthropic cache_control breakpoints, OpenAI automatic prefix caching, Gemini implicit\u002Fexplicit), semantic caching, and KV-cache internals & management (PagedAttention\u002FvLLM, RadixAttention\u002FSGLang, eviction, quantized KV, memory pressure, multi-tenant safety). Concrete numbers, formulas, failure modes.","jpoindexter\u002FVanta","jpoindexter","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F2729617?v=4","01M72VRRYY1CP7RPF3B7GHH0CR","01M757GWDBT6RD147ZZ79N4MEX",4,"MIT","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fai-engineering-skills\u002Finference-caching-and-kv\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: inference-caching-and-kv\ndescription: Reference-grade guide to caching in LLM inference — provider prompt caching (Anthropic cache_control breakpoints, OpenAI automatic prefix caching, Gemini implicit\u002Fexplicit), semantic caching, and KV-cache internals & management (PagedAttention\u002FvLLM, RadixAttention\u002FSGLang, eviction, quantized KV, memory pressure, multi-tenant safety). Concrete numbers, formulas, failure modes.\ntags: [ai-engineering, caching, kv-cache, inference]\n---\n\n# Inference Caching & KV Cache Management\n\nThree distinct caches get confused under one word. They live at different layers,\nhave different correctness properties, and fail differently.\n\n| Cache | Layer | Match | Returns | Risk if wrong |\n|-------|-------|-------|---------|---------------|\n| **Prompt cache** | Provider \u002F serving engine | Exact token-prefix | Skips *prefill* compute, regenerates output | None (cheaper, identical output) |\n| **Semantic cache** | Your app, in front of the API | Embedding similarity | A *stored prior response* | Serves a **wrong answer** |\n| **KV cache** | GPU, inside the model | Per-token K\u002FV tensors | Enables O(1) decode | OOM \u002F preemption |\n\nRule of thumb: prompt cache is **exact + cheap + safe** — on always. Semantic\ncache is **fuzzy + risky** — use surgically. KV cache is **not optional** (it's\nhow decode works) and at scale dominates GPU memory.\n\n---\n\n## 1. Prompt caching (provider-level)\n\n### How it works\nThe model's *prefill* phase computes K\u002FV for every prompt token before the first\noutput token, at cost O(prompt_length). Prompt caching stores that computed K\u002FV\nkeyed by an **exact token prefix**; on a hit the engine skips prefill for the\ncached span and resumes from the first divergent token. Matching is on **exact\ntokens from position 0** — a prefix match, not fuzzy or substring. One changed\nbyte invalidates everything from that byte onward. Prefill is the dominant cost\nfor RAG\u002Fagent\u002Flong-context workloads where the prompt dwarfs the completion, so\ncaching it turns repeated large prefixes into near-free reads.\n\n**Anthropic** (explicit, `cache_control` breakpoints):\n- Cache **write** ≈ **1.25×** base input price (5-min TTL) or **2×** (1-hr TTL).\n- Cache **read** ≈ **0.1×** base input price (~90% discount).\n- Up to **4 breakpoints**; each marks the end of a cacheable prefix. The engine\n  also checks shorter prefixes, so you don't need a breakpoint per turn.\n- TTL options: **5 minutes** (default) and **1 hour**. TTL refreshes on each hit.\n- Minimum cacheable size: **1024 tokens** (Sonnet\u002FOpus) or **2048** (Haiku).\n\nBreak-even: a write costs 0.25× extra (5-min), each read saves 0.9×, so a cached\nprefix pays for itself after **~1 reuse within the TTL** (0.25 \u003C 0.9). A 50-call\nagent loop reusing a 20k-token system+tools prefix pays 1.25× once + 0.1× × 49\ninstead of 1.0× × 50 — roughly an **~85% input-cost cut** on that prefix.\n\n**OpenAI** (automatic prefix caching):\n- No API flag. Prompts **≥1024 tokens** are auto-cached; matched in **128-token\n  increments** (1024, 1152, 1280, …).\n- Cached input tokens billed at **~0.5×** (50% off, model-dependent); some newer\n  models advertise deeper discounts. No separate write surcharge.\n- TTL: evicted after **5–10 min** idle, up to ~1 hr off-peak. Route the same\n  prompt family to the same backend with a stable `prompt_cache_key` (or\n  legacy `user` field) to raise hit rate.\n- `usage.prompt_tokens_details.cached_tokens` reports the hit size.\n\n**Gemini**:\n- **Implicit caching** (auto, on by default for 2.5 models, ~0.25× on the cached\n  portion, needs the common prefix at the *start* of the request) plus **explicit\n  caching** (`CachedContent` — you set TTL, get a guaranteed discount + a storage\n  fee per token-hour).\n\n### Stable-prefix design (the one thing that makes or breaks hit rate)\nOrder the prompt **most-stable → most-volatile**:\n\n```\n[ system instructions ]   ← never changes        ┐\n[ tool \u002F function defs ]   ← changes per deploy   │ cache this prefix\n[ static context: docs,    ← changes per session  │ (breakpoint here)\n  schemas, few-shot ]                             ┘\n------------------------------------------------- ← cache breakpoint\n[ retrieved chunks ]       ← per query   ┐ keep AFTER the breakpoint\n[ conversation history ]   ← per turn    │ so the stable prefix\n[ user's current message ] ← per call    ┘ survives\n```\n\n```python\n# Anthropic: cache the big static prefix, leave volatile tail uncached\nmessages = [{\n  \"role\": \"user\",\n  \"content\": [\n    {\"type\": \"text\", \"text\": SYSTEM_AND_TOOLS},            # stable\n    {\"type\": \"text\", \"text\": STATIC_DOCS,\n     \"cache_control\": {\"type\": \"ephemeral\", \"ttl\": \"1h\"}}, # breakpoint\n    {\"type\": \"text\", \"text\": user_query},                  # volatile, uncached\n  ],\n}]\n```\n\n**Cache hit rate** is the metric. Track:\n`cache_read_tokens \u002F (cache_read + cache_write + uncached_input)`. Healthy\nagent\u002FRAG loops hit **60–90%+**. A sudden drop to ~0% means the prefix went\nunstable (see failure modes).\n\n### What silently breaks a prefix\n- A timestamp, request ID, UUID, or `Date.now()` injected into the system prompt.\n- Reordering tools, or non-deterministic JSON key order in tool schemas.\n- Trimming\u002Fwhitespace differences, locale-formatted numbers, or a model\u002Fversion\n  swap (cache is per exact model).\n- Putting retrieved\u002Fvolatile content **before** the static block.\n- Per-user personalization spliced into the *front* instead of the tail.\n\n---\n\n## 2. Semantic caching\n\nEmbed the incoming query → nearest-neighbour search in a vector store → if a\nstored entry is within a similarity threshold, **return its cached response**\nwithout calling the model.\n\n```python\ndef semantic_lookup(query, threshold=0.92):\n    q = embed(query)\n    hit, score = vector_store.nearest(q)           # cosine similarity\n    if score >= threshold and not is_stale(hit):\n        return hit.response                        # skip the LLM entirely\n    resp = call_llm(query)\n    vector_store.upsert(embed(query), resp, ts=now())\n    return resp\n```\n\n### When it helps\n- High-repetition, low-variance intents: FAQ bots, support deflection, classifier\n  prompts, doc Q&A where many users ask the same thing different ways.\n- Workloads with a **long tail of paraphrases** of a small set of questions.\n\n### When it hurts (don't use it)\n- Personalized, stateful, time-sensitive, or computational answers (\"what's *my*\n  balance\", \"current price\", \"summarize *this* doc\"). Two queries that *look*\n  similar can require different answers.\n- Anything where a wrong-but-plausible answer is costly.\n\n### Thresholds & the false-hit risk\n- Cosine **0.95+** is conservative (fewer hits, few false hits); **0.85–0.90**\n  is aggressive (more hits, real false-hit risk). Tune on labelled pairs, not\n  vibes — measure precision of \"same intent\" at each threshold.\n- **False hit** = two semantically *different* queries score above threshold and\n  the cache serves the wrong stored answer. This is the defining hazard:\n  embeddings collapse negation (\"can I cancel\" vs \"can I *not* cancel\"), numbers,\n  and entities. Mitigate: raise threshold, add a cheap LLM\u002Frule re-rank gate on\n  borderline hits, key on normalized intent + critical entities, never cache\n  responses containing user-specific data.\n- **Staleness**: cached answers go wrong when the underlying truth changes\n  (price, policy, docs). Attach a TTL and a content-version tag; invalidate by\n  version bump or event, not just time. Always cache *with* an embedding model\n  version — re-embedding on model change is required.\n\n### Prompt-cache vs semantic-cache\n| | Prompt cache | Semantic cache |\n|---|---|---|\n| Match | Exact prefix | Fuzzy (embedding) |\n| Returns | Recomputed correct output | Stored prior output |\n| Correctness | Always correct | Can serve wrong answer |\n| Saves | Prefill compute \u002F input cost | Whole inference call |\n| Owner | Provider\u002Fengine | You |\n| Default stance | **On always** | **Off until justified** |\n\nThey compose: semantic cache catches whole-query repeats; prompt cache makes the\n*misses* cheaper. Layer semantic in front, prompt cache underneath.\n\n---\n\n## 3. KV cache internals\n\n### What it is\nDuring decode, each new token attends to **all previous tokens'** keys and values.\nRecomputing K\u002FV for the whole sequence every step is O(n²) total. Instead the\nengine **caches K and V per token per layer** once, so each decode step only\nprojects the one new token and attends against the cached set — **O(1)** in\npast-token work. The KV cache is what makes autoregressive decode tractable, and\nit grows by one token's worth of K\u002FV every step.\n\n### Memory cost — the formula\n```\nkv_bytes = 2 · num_layers · num_kv_heads · head_dim · seq_len · batch · dtype_bytes\n           │\n           └─ the 2 is for K *and* V\n```\n`num_kv_heads` is the **GQA\u002FMQA** head count (often ≪ attention heads — this is a\nprimary KV-memory lever). `dtype_bytes`: FP16\u002FBF16 = 2, FP8\u002FINT8 = 1.\n\n**Worked example — Llama-3-8B**, 32 layers, 8 KV heads (GQA), head_dim 128, FP16:\n- Per token: `2 · 32 · 8 · 128 · 2 = 131,072 B ≈ 128 KiB\u002Ftoken`.\n- 8k-token sequence: `128 KiB · 8192 ≈ 1.0 GiB`. Batch of 32: **~32 GiB of KV** —\n  on top of ~16 GiB of FP16 weights. On an 80 GB A100\u002FH100 you are KV-bound long\n  before weight-bound.\n\n**Llama-3-70B** (80 layers, 8 KV heads, head_dim 128, FP16): ~320 KiB\u002Ftoken → a\nsingle 32k-context request needs **~10 GiB of KV by itself**. KV cache, not\nweights, sets the throughput ceiling at long context \u002F high batch.\n\nTakeaway: **KV scales linearly with seq_len × batch**; weights are fixed. Past a\npoint, KV dominates total memory.\n\n---\n\n## 4. KV cache management\n\n### PagedAttention (vLLM)\nNaive serving pre-allocates a contiguous KV buffer sized to `max_seq_len` per\nrequest → internal fragmentation and reserved-but-unused memory (often 60–80%\nwaste). PagedAttention borrows OS virtual memory: KV is split into **fixed-size\nblocks** (e.g. 16 tokens), stored **non-contiguously**, with a block table mapping\nlogical → physical. Result: near-zero fragmentation, on-demand allocation, and\n**copy-on-write block sharing** for identical prefixes (parallel samples, beam\nsearch, shared system prompts). vLLM reports up to ~2–4× throughput vs naive\nallocation, mostly from packing more concurrent sequences into the same VRAM.\n\n### Prefix sharing \u002F RadixAttention (SGLang)\nCaches prefixes in a **radix tree** keyed by token sequence, so any request\nsharing a prefix (system prompt, few-shot block, multi-turn history,\ntree-of-thought branches) **reuses** the computed KV instead of re-running\nprefill (LRU eviction on the tree). Automatic cross-request KV reuse — the\nserver-side analogue of provider prompt caching. Huge for agent fan-out;\nhit rates of 50–90% are common. vLLM's `--enable-prefix-caching` does the same\nvia hashed block reuse.\n\n### Eviction & scheduling\nKV blocks are a finite pool. When full:\n- **LRU \u002F priority eviction** of cached (non-active) prefix blocks first.\n- **Preemption** of *running* sequences when even active KV won't fit:\n  - **Recompute**: drop the sequence's KV, re-run prefill later. Cheap memory,\n    costs compute. Default for short prompts.\n  - **Swap\u002Foffload**: move KV to **CPU RAM** (or disk\u002FNVMe) and page it back.\n    Saves recompute, costs PCIe bandwidth (offload ~tens of GB\u002Fs vs HBM ~TB\u002Fs).\n    Better for long prompts where recompute is expensive.\n- Continuous batching adds\u002Fremoves sequences each step; preemption keeps the batch within the KV budget rather than OOM-crashing.\n\n### Quantized KV cache (FP8 \u002F INT8)\nStore K\u002FV at **1 byte\u002Felem** instead of 2 → **~2× longer context or batch** in the\nsame VRAM. FP8 KV typically costs \u003C1% quality; INT8 needs per-channel\u002Fper-token\nscales to stay clean. (`--kv-cache-dtype fp8` in vLLM.)\n\n### Sliding-window \u002F streaming\nModels with **sliding-window attention** (e.g. Mistral) attend only to the last W tokens, so KV is **bounded at W** — O(W) memory, not O(n). **StreamingLLM** keeps a few \"attention-sink\" initial tokens + a recent window to run effectively unbounded streams without OOM (dropping the true middle) — for long-running chats\u002Flogs where old context is expendable.\n\n### KV reuse across requests\nPersisting KV beyond one request (RadixAttention, vLLM prefix cache, or external\nstores like LMCache offloading to CPU\u002Fdisk) turns repeated prefixes into reads —\nprovider prompt caching implemented in your own serving layer when you self-host.\n\n---\n\n## 5. Memory pressure at scale\n\nThe core tradeoff on a fixed GPU:\n```\nVRAM ≈ weights + activations + KV(batch, context)     (KV is the variable term)\n```\nSince `KV ∝ batch · context`, you trade them against each other:\n\n| Lever | Effect | Cost |\n|-------|--------|------|\n| ↑ batch size | ↑ throughput (tok\u002Fs aggregate) | ↑ KV → OOM risk, ↑ per-request latency |\n| ↑ context length | longer prompts\u002Foutputs | ↑ KV per seq → fewer concurrent seqs |\n| ↓ KV dtype (FP8) | ~2× capacity | small quality hit |\n| ↓ num_kv_heads (GQA\u002FMQA) | big KV cut | model-architecture decision |\n| sliding window | bounded KV | loses long-range context |\n\n**OOM** happens when active + cached KV exceeds the pool — usually triggered by a\nburst of long-context requests, not steady state. Defend with: a hard\n`max_num_seqs` \u002F `max_num_batched_tokens` cap, `gpu_memory_utilization` headroom\n(e.g. 0.90, not 0.98), admission control \u002F queueing, and preemption (swap or\nrecompute) so the server degrades to higher latency instead of crashing.\n\n**Capacity planning** (back-of-envelope):\n```\nkv_budget   = (VRAM · util) − weights − activation_overhead\nmax_tokens  = kv_budget \u002F bytes_per_token        # from §3 formula\nmax_concurrent ≈ max_tokens \u002F avg_seq_len\n```\nPlan for **p95 sequence length**, not mean — long tails are what OOM you. Leave\n~10–15% headroom for activation spikes and fragmentation.\n\n---\n\n## 6. Cache safety & multi-tenancy\n\n| Cache | Leakage vector | Isolation |\n|-------|----------------|-----------|\n| KV prefix sharing | Tenant B reads tenant A's reused prefix KV → **content\u002Ftiming leak** | Namespace the cache key by tenant; don't share blocks across tenants for sensitive prefixes |\n| Semantic cache | Cached response contains tenant A's PII, served to tenant B on a fuzzy match | **Per-tenant cache partition**; never cache PII-bearing responses |\n| Provider prompt cache | Same-org reuse only (providers scope to API key\u002Forg); a cache *timing* side-channel can reveal \"someone cached this prefix\" | Treat presence of a hit as low-sensitivity; don't put secrets in shared system prompts |\n\nRules:\n- **Partition every shared cache by tenant\u002Fuser** when content is user-specific.\n  A global prefix cache is safe *only* for genuinely shared content (the system\n  prompt, public docs).\n- **Never store secrets\u002FPII in a shared prefix** that other tenants' requests\n  could match against.\n- Semantic caches must key on `(tenant_id, normalized_query)` and exclude any\n  response embedding user data — or scope the vector index per tenant.\n\n---\n\n## 7. Failure modes (and the fix)\n\n- **Unstable prefix → ~0% hit rate.** A UUID\u002Ftimestamp\u002Freordered-tool crept into\n  the cached prefix. Symptom: cache_read_tokens collapse, input cost spikes.\n  → Move all volatile content after the breakpoint; make tool\u002FJSON order\n  deterministic; diff two consecutive prompts byte-for-byte.\n- **Semantic false hit → wrong answer served.** Threshold too low; negation or\n  entity difference collapsed by the embedding. → Raise threshold, add a\n  re-rank\u002Fverify gate, key on critical entities, exclude personalized intents.\n- **Semantic staleness → confidently outdated answer.** Truth changed; cache\n  didn't. → Version-tag + event-driven invalidation, not TTL alone.\n- **KV OOM under long context.** A burst of long prompts exceeds the KV pool.\n  → Cap max_num_seqs \u002F batched_tokens, enable preemption (swap\u002Frecompute), FP8\n  KV, lower gpu_memory_utilization, admission-queue.\n- **Cross-tenant cache contamination.** Shared prefix\u002Fsemantic cache leaks one\n  tenant's data to another. → Partition keys by tenant; never cache PII in a\n  shared namespace.\n- **Cache thrash.** Working set > cache → constant evict\u002Frefetch, worse than no\n  cache. → Size cache to the working set, pin hot prefixes, or shorten the prefix.\n- **TTL expiry mid-loop.** A slow agent step lets the prefix expire → surprise\n  cache-write surcharge. → Use the 1-hr TTL for long runs; hits refresh the TTL.\n\n---\n\n## 8. Do \u002F Don't\n\n**Do**\n- Turn on provider prompt caching for every repeated-prefix workload — it's free\n  correctness-wise and pays off after one reuse.\n- Order prompts stable→volatile; put the cache breakpoint after the static block.\n- Track cache hit rate as a first-class metric; alert on a sudden drop.\n- Use FP8 KV + PagedAttention + prefix caching when self-hosting at scale.\n- Partition shared caches by tenant; plan KV for p95 length with headroom.\n\n**Don't**\n- Don't inject timestamps\u002FUUIDs\u002Fnon-deterministic ordering into a cached prefix.\n- Don't semantic-cache personalized, time-sensitive, or computational answers.\n- Don't set the semantic threshold by feel — measure precision on labelled pairs.\n- Don't run gpu_memory_utilization at 0.98 and expect no OOM under load.\n- Don't share KV blocks or cached responses across tenants for sensitive content.\n","# Inference Caching & KV Cache Management\n\nThree distinct caches get confused under one word. They live at different layers,\nhave different correctness properties, and fail differently.\n\n| Cache | Layer | Match | Returns | Risk if wrong |\n|-------|-------|-------|---------|---------------|\n| **Prompt cache** | Provider \u002F serving engine | Exact token-prefix | Skips *prefill* compute, regenerates output | None (cheaper, identical output) |\n| **Semantic cache** | Your app, in front of the API | Embedding similarity | A *stored prior response* | Serves a **wrong answer** |\n| **KV cache** | GPU, inside the model | Per-token K\u002FV tensors | Enables O(1) decode | OOM \u002F preemption |\n\nRule of thumb: prompt cache is **exact + cheap + safe** — on always. Semantic\ncache is **fuzzy + risky** — use surgically. KV cache is **not optional** (it's\nhow decode works) and at scale dominates GPU memory.\n\n---\n\n## 1. Prompt caching (provider-level)\n\n### How it works\nThe model's *prefill* phase computes K\u002FV for every prompt token before the first\noutput token, at cost O(prompt_length). Prompt caching stores that computed K\u002FV\nkeyed by an **exact token prefix**; on a hit the engine skips prefill for the\ncached span and resumes from the first divergent token. Matching is on **exact\ntokens from position 0** — a prefix match, not fuzzy or substring. One changed\nbyte invalidates everything from that byte onward. Prefill is the dominant cost\nfor RAG\u002Fagent\u002Flong-context workloads where the prompt dwarfs the completion, so\ncaching it turns repeated large prefixes into near-free reads.\n\n**Anthropic** (explicit, `cache_control` breakpoints):\n- Cache **write** ≈ **1.25×** base input price (5-min TTL) or **2×** (1-hr TTL).\n- Cache **read** ≈ **0.1×** base input price (~90% discount).\n- Up to **4 breakpoints**; each marks the end of a cacheable prefix. The engine\n  also checks shorter prefixes, so you don't need a breakpoint per turn.\n- TTL options: **5 minutes** (default) and **1 hour**. TTL refreshes on each hit.\n- Minimum cacheable size: **1024 tokens** (Sonnet\u002FOpus) or **2048** (Haiku).\n\nBreak-even: a write costs 0.25× extra (5-min), each read saves 0.9×, so a cached\nprefix pays for itself after **~1 reuse within the TTL** (0.25 \u003C 0.9). A 50-call\nagent loop reusing a 20k-token system+tools prefix pays 1.25× once + 0.1× × 49\ninstead of 1.0× × 50 — roughly an **~85% input-cost cut** on that prefix.\n\n**OpenAI** (automatic prefix caching):\n- No API flag. Prompts **≥1024 tokens** are auto-cached; matched in **128-token\n  increments** (1024, 1152, 1280, …).\n- Cached input tokens billed at **~0.5×** (50% off, model-dependent); some newer\n  models advertise deeper discounts. No separate write surcharge.\n- TTL: evicted after **5–10 min** idle, up to ~1 hr off-peak. Route the same\n  prompt family to the same backend with a stable `prompt_cache_key` (or\n  legacy `user` field) to raise hit rate.\n- `usage.prompt_tokens_details.cached_tokens` reports the hit size.\n\n**Gemini**:\n- **Implicit caching** (auto, on by default for 2.5 models, ~0.25× on the cached\n  portion, needs the common prefix at the *start* of the request) plus **explicit\n  caching** (`CachedContent` — you set TTL, get a guaranteed discount + a storage\n  fee per token-hour).\n\n### Stable-prefix design (the one thing that makes or breaks hit rate)\nOrder the prompt **most-stable → most-volatile**:\n\n```\n[ system instructions ]   ← never changes        ┐\n[ tool \u002F function defs ]   ← changes per deploy   │ cache this prefix\n[ static context: docs,    ← changes per session  │ (breakpoint here)\n  schemas, few-shot ]                             ┘\n------------------------------------------------- ← cache breakpoint\n[ retrieved chunks ]       ← per query   ┐ keep AFTER the breakpoint\n[ conversation history ]   ← per turn    │ so the stable prefix\n[ user's current message ] ← per call    ┘ survives\n```\n\n```python\n# Anthropic: cache the big static prefix, leave volatile tail uncached\nmessages = [{\n  \"role\": \"user\",\n  \"content\": [\n    {\"type\": \"text\", \"text\": SYSTEM_AND_TOOLS},            # stable\n    {\"type\": \"text\", \"text\": STATIC_DOCS,\n     \"cache_control\": {\"type\": \"ephemeral\", \"ttl\": \"1h\"}}, # breakpoint\n    {\"type\": \"text\", \"text\": user_query},                  # volatile, uncached\n  ],\n}]\n```\n\n**Cache hit rate** is the metric. Track:\n`cache_read_tokens \u002F (cache_read + cache_write + uncached_input)`. Healthy\nagent\u002FRAG loops hit **60–90%+**. A sudden drop to ~0% means the prefix went\nunstable (see failure modes).\n\n### What silently breaks a prefix\n- A timestamp, request ID, UUID, or `Date.now()` injected into the system prompt.\n- Reordering tools, or non-deterministic JSON key order in tool schemas.\n- Trimming\u002Fwhitespace differences, locale-formatted numbers, or a model\u002Fversion\n  swap (cache is per exact model).\n- Putting retrieved\u002Fvolatile content **before** the static block.\n- Per-user personalization spliced into the *front* instead of the tail.\n\n---\n\n## 2. Semantic caching\n\nEmbed the incoming query → nearest-neighbour search in a vector store → if a\nstored entry is within a similarity threshold, **return its cached response**\nwithout calling the model.\n\n```python\ndef semantic_lookup(query, threshold=0.92):\n    q = embed(query)\n    hit, score = vector_store.nearest(q)           # cosine similarity\n    if score >= threshold and not is_stale(hit):\n        return hit.response                        # skip the LLM entirely\n    resp = call_llm(query)\n    vector_store.upsert(embed(query), resp, ts=now())\n    return resp\n```\n\n### When it helps\n- High-repetition, low-variance intents: FAQ bots, support deflection, classifier\n  prompts, doc Q&A where many users ask the same thing different ways.\n- Workloads with a **long tail of paraphrases** of a small set of questions.\n\n### When it hurts (don't use it)\n- Personalized, stateful, time-sensitive, or computational answers (\"what's *my*\n  balance\", \"current price\", \"summarize *this* doc\"). Two queries that *look*\n  similar can require different answers.\n- Anything where a wrong-but-plausible answer is costly.\n\n### Thresholds & the false-hit risk\n- Cosine **0.95+** is conservative (fewer hits, few false hits); **0.85–0.90**\n  is aggressive (more hits, real false-hit risk). Tune on labelled pairs, not\n  vibes — measure precision of \"same intent\" at each threshold.\n- **False hit** = two semantically *different* queries score above threshold and\n  the cache serves the wrong stored answer. This is the defining hazard:\n  embeddings collapse negation (\"can I cancel\" vs \"can I *not* cancel\"), numbers,\n  and entities. Mitigate: raise threshold, add a cheap LLM\u002Frule re-rank gate on\n  borderline hits, key on normalized intent + critical entities, never cache\n  responses containing user-specific data.\n- **Staleness**: cached answers go wrong when the underlying truth changes\n  (price, policy, docs). Attach a TTL and a content-version tag; invalidate by\n  version bump or event, not just time. Always cache *with* an embedding model\n  version — re-embedding on model change is required.\n\n### Prompt-cache vs semantic-cache\n| | Prompt cache | Semantic cache |\n|---|---|---|\n| Match | Exact prefix | Fuzzy (embedding) |\n| Returns | Recomputed correct output | Stored prior output |\n| Correctness | Always correct | Can serve wrong answer |\n| Saves | Prefill compute \u002F input cost | Whole inference call |\n| Owner | Provider\u002Fengine | You |\n| Default stance | **On always** | **Off until justified** |\n\nThey compose: semantic cache catches whole-query repeats; prompt cache makes the\n*misses* cheaper. Layer semantic in front, prompt cache underneath.\n\n---\n\n## 3. KV cache internals\n\n### What it is\nDuring decode, each new token attends to **all previous tokens'** keys and values.\nRecomputing K\u002FV for the whole sequence every step is O(n²) total. Instead the\nengine **caches K and V per token per layer** once, so each decode step only\nprojects the one new token and attends against the cached set — **O(1)** in\npast-token work. The KV cache is what makes autoregressive decode tractable, and\nit grows by one token's worth of K\u002FV every step.\n\n### Memory cost — the formula\n```\nkv_bytes = 2 · num_layers · num_kv_heads · head_dim · seq_len · batch · dtype_bytes\n           │\n           └─ the 2 is for K *and* V\n```\n`num_kv_heads` is the **GQA\u002FMQA** head count (often ≪ attention heads — this is a\nprimary KV-memory lever). `dtype_bytes`: FP16\u002FBF16 = 2, FP8\u002FINT8 = 1.\n\n**Worked example — Llama-3-8B**, 32 layers, 8 KV heads (GQA), head_dim 128, FP16:\n- Per token: `2 · 32 · 8 · 128 · 2 = 131,072 B ≈ 128 KiB\u002Ftoken`.\n- 8k-token sequence: `128 KiB · 8192 ≈ 1.0 GiB`. Batch of 32: **~32 GiB of KV** —\n  on top of ~16 GiB of FP16 weights. On an 80 GB A100\u002FH100 you are KV-bound long\n  before weight-bound.\n\n**Llama-3-70B** (80 layers, 8 KV heads, head_dim 128, FP16): ~320 KiB\u002Ftoken → a\nsingle 32k-context request needs **~10 GiB of KV by itself**. KV cache, not\nweights, sets the throughput ceiling at long context \u002F high batch.\n\nTakeaway: **KV scales linearly with seq_len × batch**; weights are fixed. Past a\npoint, KV dominates total memory.\n\n---\n\n## 4. KV cache management\n\n### PagedAttention (vLLM)\nNaive serving pre-allocates a contiguous KV buffer sized to `max_seq_len` per\nrequest → internal fragmentation and reserved-but-unused memory (often 60–80%\nwaste). PagedAttention borrows OS virtual memory: KV is split into **fixed-size\nblocks** (e.g. 16 tokens), stored **non-contiguously**, with a block table mapping\nlogical → physical. Result: near-zero fragmentation, on-demand allocation, and\n**copy-on-write block sharing** for identical prefixes (parallel samples, beam\nsearch, shared system prompts). vLLM reports up to ~2–4× throughput vs naive\nallocation, mostly from packing more concurrent sequences into the same VRAM.\n\n### Prefix sharing \u002F RadixAttention (SGLang)\nCaches prefixes in a **radix tree** keyed by token sequence, so any request\nsharing a prefix (system prompt, few-shot block, multi-turn history,\ntree-of-thought branches) **reuses** the computed KV instead of re-running\nprefill (LRU eviction on the tree). Automatic cross-request KV reuse — the\nserver-side analogue of provider prompt caching. Huge for agent fan-out;\nhit rates of 50–90% are common. vLLM's `--enable-prefix-caching` does the same\nvia hashed block reuse.\n\n### Eviction & scheduling\nKV blocks are a finite pool. When full:\n- **LRU \u002F priority eviction** of cached (non-active) prefix blocks first.\n- **Preemption** of *running* sequences when even active KV won't fit:\n  - **Recompute**: drop the sequence's KV, re-run prefill later. Cheap memory,\n    costs compute. Default for short prompts.\n  - **Swap\u002Foffload**: move KV to **CPU RAM** (or disk\u002FNVMe) and page it back.\n    Saves recompute, costs PCIe bandwidth (offload ~tens of GB\u002Fs vs HBM ~TB\u002Fs).\n    Better for long prompts where recompute is expensive.\n- Continuous batching adds\u002Fremoves sequences each step; preemption keeps the batch within the KV budget rather than OOM-crashing.\n\n### Quantized KV cache (FP8 \u002F INT8)\nStore K\u002FV at **1 byte\u002Felem** instead of 2 → **~2× longer context or batch** in the\nsame VRAM. FP8 KV typically costs \u003C1% quality; INT8 needs per-channel\u002Fper-token\nscales to stay clean. (`--kv-cache-dtype fp8` in vLLM.)\n\n### Sliding-window \u002F streaming\nModels with **sliding-window attention** (e.g. Mistral) attend only to the last W tokens, so KV is **bounded at W** — O(W) memory, not O(n). **StreamingLLM** keeps a few \"attention-sink\" initial tokens + a recent window to run effectively unbounded streams without OOM (dropping the true middle) — for long-running chats\u002Flogs where old context is expendable.\n\n### KV reuse across requests\nPersisting KV beyond one request (RadixAttention, vLLM prefix cache, or external\nstores like LMCache offloading to CPU\u002Fdisk) turns repeated prefixes into reads —\nprovider prompt caching implemented in your own serving layer when you self-host.\n\n---\n\n## 5. Memory pressure at scale\n\nThe core tradeoff on a fixed GPU:\n```\nVRAM ≈ weights + activations + KV(batch, context)     (KV is the variable term)\n```\nSince `KV ∝ batch · context`, you trade them against each other:\n\n| Lever | Effect | Cost |\n|-------|--------|------|\n| ↑ batch size | ↑ throughput (tok\u002Fs aggregate) | ↑ KV → OOM risk, ↑ per-request latency |\n| ↑ context length | longer prompts\u002Foutputs | ↑ KV per seq → fewer concurrent seqs |\n| ↓ KV dtype (FP8) | ~2× capacity | small quality hit |\n| ↓ num_kv_heads (GQA\u002FMQA) | big KV cut | model-architecture decision |\n| sliding window | bounded KV | loses long-range context |\n\n**OOM** happens when active + cached KV exceeds the pool — usually triggered by a\nburst of long-context requests, not steady state. Defend with: a hard\n`max_num_seqs` \u002F `max_num_batched_tokens` cap, `gpu_memory_utilization` headroom\n(e.g. 0.90, not 0.98), admission control \u002F queueing, and preemption (swap or\nrecompute) so the server degrades to higher latency instead of crashing.\n\n**Capacity planning** (back-of-envelope):\n```\nkv_budget   = (VRAM · util) − weights − activation_overhead\nmax_tokens  = kv_budget \u002F bytes_per_token        # from §3 formula\nmax_concurrent ≈ max_tokens \u002F avg_seq_len\n```\nPlan for **p95 sequence length**, not mean — long tails are what OOM you. Leave\n~10–15% headroom for activation spikes and fragmentation.\n\n---\n\n## 6. Cache safety & multi-tenancy\n\n| Cache | Leakage vector | Isolation |\n|-------|----------------|-----------|\n| KV prefix sharing | Tenant B reads tenant A's reused prefix KV → **content\u002Ftiming leak** | Namespace the cache key by tenant; don't share blocks across tenants for sensitive prefixes |\n| Semantic cache | Cached response contains tenant A's PII, served to tenant B on a fuzzy match | **Per-tenant cache partition**; never cache PII-bearing responses |\n| Provider prompt cache | Same-org reuse only (providers scope to API key\u002Forg); a cache *timing* side-channel can reveal \"someone cached this prefix\" | Treat presence of a hit as low-sensitivity; don't put secrets in shared system prompts |\n\nRules:\n- **Partition every shared cache by tenant\u002Fuser** when content is user-specific.\n  A global prefix cache is safe *only* for genuinely shared content (the system\n  prompt, public docs).\n- **Never store secrets\u002FPII in a shared prefix** that other tenants' requests\n  could match against.\n- Semantic caches must key on `(tenant_id, normalized_query)` and exclude any\n  response embedding user data — or scope the vector index per tenant.\n\n---\n\n## 7. Failure modes (and the fix)\n\n- **Unstable prefix → ~0% hit rate.** A UUID\u002Ftimestamp\u002Freordered-tool crept into\n  the cached prefix. Symptom: cache_read_tokens collapse, input cost spikes.\n  → Move all volatile content after the breakpoint; make tool\u002FJSON order\n  deterministic; diff two consecutive prompts byte-for-byte.\n- **Semantic false hit → wrong answer served.** Threshold too low; negation or\n  entity difference collapsed by the embedding. → Raise threshold, add a\n  re-rank\u002Fverify gate, key on critical entities, exclude personalized intents.\n- **Semantic staleness → confidently outdated answer.** Truth changed; cache\n  didn't. → Version-tag + event-driven invalidation, not TTL alone.\n- **KV OOM under long context.** A burst of long prompts exceeds the KV pool.\n  → Cap max_num_seqs \u002F batched_tokens, enable preemption (swap\u002Frecompute), FP8\n  KV, lower gpu_memory_utilization, admission-queue.\n- **Cross-tenant cache contamination.** Shared prefix\u002Fsemantic cache leaks one\n  tenant's data to another. → Partition keys by tenant; never cache PII in a\n  shared namespace.\n- **Cache thrash.** Working set > cache → constant evict\u002Frefetch, worse than no\n  cache. → Size cache to the working set, pin hot prefixes, or shorten the prefix.\n- **TTL expiry mid-loop.** A slow agent step lets the prefix expire → surprise\n  cache-write surcharge. → Use the 1-hr TTL for long runs; hits refresh the TTL.\n\n---\n\n## 8. Do \u002F Don't\n\n**Do**\n- Turn on provider prompt caching for every repeated-prefix workload — it's free\n  correctness-wise and pays off after one reuse.\n- Order prompts stable→volatile; put the cache breakpoint after the static block.\n- Track cache hit rate as a first-class metric; alert on a sudden drop.\n- Use FP8 KV + PagedAttention + prefix caching when self-hosting at scale.\n- Partition shared caches by tenant; plan KV for p95 length with headroom.\n\n**Don't**\n- Don't inject timestamps\u002FUUIDs\u002Fnon-deterministic ordering into a cached prefix.\n- Don't semantic-cache personalized, time-sensitive, or computational answers.\n- Don't set the semantic threshold by feel — measure precision on labelled pairs.\n- Don't run gpu_memory_utilization at 0.98 and expect no OOM under load.\n- Don't share KV blocks or cached responses across tenants for sensitive content.\n",{"name":5,"tags":23,"description":7},[24,25,26,27],"ai-engineering","caching","kv-cache","inference","main","4911ae44bbb35beef4511ba298475ba5a82b7e1c","ai-engineering-skills\u002Finference-caching-and-kv",0,"https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta","https:\u002F\u002Fdocs.vanta.theft.studio","https:\u002F\u002Fgithub.com\u002Fjpoindexter","jason.theft.studio","Just some guy trying to build stuff","2026-11-12 17:11:35",{"source":8,"skillName":5,"commands":39,"prompt":49,"zipAvailable":50},[40,43,46],{"tool":41,"command":42},"npx","npx skills add jpoindexter\u002FVanta --skill inference-caching-and-kv",{"tool":44,"command":45},"bunx","bunx skills add jpoindexter\u002FVanta --skill inference-caching-and-kv",{"tool":47,"command":48},"pnpm","pnpm dlx skills add jpoindexter\u002FVanta --skill inference-caching-and-kv","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fai-engineering-skills\u002Finference-caching-and-kv\u002FSKILL.md",true,[],[53,60,67,74,81,87],{"skillNo":54,"name":55,"nameZh":6,"description":56,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":57,"categories":58,"keywords":59,"publishedAt":18},"01M7582PEM7XM8B98VFRDV0C0J","write-changelog","Turn a range of commits into a user-facing changelog","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fsrc\u002Fmigrate\u002F__fixtures__\u002Fopenclaw\u002Fskills\u002Fwrite-changelog\u002FSKILL.md",[],[],{"skillNo":61,"name":62,"nameZh":6,"description":63,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":64,"categories":65,"keywords":66,"publishedAt":18},"01M7582KTQJMP5Y7GS83MS1C7Z","debug-flaky-test","Reproduce and isolate a flaky test before fixing it","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fsrc\u002Fmigrate\u002F__fixtures__\u002Fopenclaw\u002Fskills\u002Fdebug-flaky-test\u002FSKILL.md",[],[],{"skillNo":68,"name":69,"nameZh":6,"description":70,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":71,"categories":72,"keywords":73,"publishedAt":18},"01M7582HG7XFM4Z5NGG1YFECT8","triage-incident","First 10 minutes of an incident — stabilize, then diagnose","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fsrc\u002Fmigrate\u002F__fixtures__\u002Fhermes\u002Fskills\u002Ftriage-incident\u002FSKILL.md",[],[],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":18},"01M7582DQABK7A96XHMB770KQZ","writing-plans","Write implementation plans: bite-sized tasks, paths, code.","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fskills-library\u002Fwriting-plans\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":18},"01M7582BBK8H0HGSXKK08FCXJ1","vanta-port-adapter","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fskills-library\u002Fvanta-port-adapter\u002FSKILL.md",[],[],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":18},"01M75828P8PX5XSBMJ2BVVDA9Y","test-driven-development","TDD: enforce RED-GREEN-REFACTOR, tests before code.","https:\u002F\u002Fgithub.com\u002Fjpoindexter\u002FVanta\u002Fblob\u002Fmain\u002Fvanta-ts\u002Fskills-library\u002Ftest-driven-development\u002FSKILL.md",[],[]]