disaggregated-prefill-decode
暂无简介
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 13
- Forks
- 2
03 SKILL.md 正文
/disaggregated-prefill-decode
When to Use
- Inference spend >$500K/year on colocated serving — disaggregation can save 30–40%
- Mixed workloads with both long-prompt (prefill-heavy) and long-output (decode-heavy) requests
- MoE models (DeepSeek-R1, Mixtral) where colocation wastes H100 compute on memory-bound decode
- Already on Kubernetes and want CNCF-native orchestration (llm-d)
Do NOT use for
- Short prompts + short outputs (<512 tokens each) — NIXL transfer tax exceeds savings
- Small-scale serving (<10 GPUs) — orchestration overhead not justified
- Development / testing environments — complexity far outweighs benefit
Why the bottlenecks differ
Prefill — full forward pass over input prompt → matrix multiplications dominate
→ COMPUTE-BOUND — H100 FP8 ~2,000 TFLOPS
→ batch many tokens in one forward → high GPU utilization
Decode — one token per step, reading full weights each time
→ MEMORY-BANDWIDTH-BOUND — HBM3 ~3 TB/s
→ only amortizes at high concurrency
Colocation problem:
You buy H100 compute for decode (memory bound → compute wasted)
You buy H100 HBM for prefill (compute bound → memory wasted)
20–40% GPU time wasted on the wrong resource at typical workloads.
Architecture diagram
┌──────────────┐
Request → │ Router │ ─────────────────────────┐
└──────┬───────┘ │
│ (prompt only) │
▼ │
┌──────────────┐ KV cache via NIXL ┌──▼───────────┐
│ Prefill pool │ ──────────────────────► │ Decode pool │
│ (compute) │ │ (memory) │
└──────────────┘ └──────┬───────┘
│ tokens
▼
Client
NIXL: NVIDIA inter-node transport
- Uses RDMA/InfiniBand when available
- TCP fallback otherwise
- Typical KV transfer latency: 20–80 ms for 4K-token prompt on 70B FP8
→ Short prompts (<512 tokens): transfer tax > compute savings → don't disaggregate
NVIDIA Dynamo (GTC 2025 GA)
# Dynamo sits ABOVE vLLM / SGLang / TRT-LLM as an orchestrator
# Rust core, Python extensibility
# Install
pip install nvidia-dynamo
# Launch with disaggregated P/D
dynamo serve \
--model deepseek-ai/DeepSeek-R1 \
--prefill-replicas 2 \
--decode-replicas 4 \
--kv-transfer nixl \
--sla-planner auto
# Dynamo Planner Profiler auto-measures workload and sets P:D ratio
# Published results (developer.nvidia.com, 2025-06):
# DeepSeek-R1 MoE on GB200 NVL72 + Dynamo: ~6x throughput in medium-latency regime
# GB300 NVL72 + Dynamo: up to 50x MoE throughput vs Hopper (product page, undated)
llm-d (Red Hat + AWS, Kubernetes-native)
# llm-d separates prefill / decode / router as independent K8s Services
# Per-role HPA based on correct signals for each role
# Prefill Deployment — scale on queue depth
apiVersion: apps/v1
kind: Deployment
metadata:
name: prefill-server
spec:
replicas: 2
template:
spec:
containers:
- name: prefill
image: llmd/vllm-prefill:latest
env:
- name: ROLE
value: prefill
- name: NIXL_ENDPOINT
value: decode-service:8001
---
# Decode Deployment — scale on KV utilization
apiVersion: apps/v1
kind: Deployment
metadata:
name: decode-server
spec:
replicas: 4
template:
spec:
containers:
- name: decode
image: llmd/vllm-decode:latest
env:
- name: ROLE
value: decode
---
# HPA for prefill (queue depth signal)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: prefill-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: prefill-server
minReplicas: 1
maxReplicas: 8
metrics:
- type: External
external:
metric:
name: llmd_prefill_queue_depth
target:
type: AverageValue
averageValue: "10"
When NOT to disaggregate
def should_disaggregate(
avg_prompt_tokens: int,
avg_output_tokens: int,
monthly_gpu_spend: float,
) -> tuple[bool, str]:
if avg_prompt_tokens < 512 and avg_output_tokens < 512:
return False, "Short prompts + short outputs: NIXL transfer tax exceeds savings"
if monthly_gpu_spend < 40_000:
return False, "Orchestration complexity not justified under ~$40K/month"
return True, f"Expected savings: 30–40% on ${monthly_gpu_spend:,.0f}/month"
Dynamo vs llm-d: choose based on context
Dynamo → already using NVIDIA stack (TRT-LLM, vLLM)
want managed P:D ratio tuning (SLA Planner auto-configures)
comfortable with NVIDIA licensing + support
llm-d → committed to Kubernetes / CNCF ecosystem
need GitOps-friendly declarative config (YAML deployments)
want scale-to-zero + LoRA routing + hierarchical KV offloading (v0.5)
Red Hat/AWS support preferred
Anti-Fake-Pass Checklist
❌ Short prompts (<512 tokens): NIXL transfer latency dominates → no throughput gain
❌ Prefill:decode ratio not tuned to workload → wrong pool sizes → one is idle, one is bottlenecked
❌ RDMA not available → TCP fallback → higher latency → reduces the savings window
❌ Same GPU type for both pools → defeats the purpose; prefill wants compute, decode wants memory bandwidth
❌ No KV cache compression before NIXL transfer → 70B 4K-prompt KV cache can be 4–8 GB per request
❌ Citing "30x" as a guaranteed number — it's a community aggregate on specific Blackwell+Dynamo+R1 stacks
04 同仓库更多 Skills
openai--sentry--sentry
Use when the user asks to inspect Sentry issues or events, summarize recent production errors, or pull basic Sentry health data via the Sentry API; perform read-only queries with the bundled script and require `SENTRY_AUTH_TOKEN`.
ito-basket-compare
Compare Itô prediction-market baskets against a user's knowledge base, portfolio notes, financial context, watchlist, or research thesis. Use for read-only basket comparison and gap analysis without investment advice or live trading.
implementing-network-traffic-baselining
Build network traffic baselines from NetFlow/IPFIX data using Python pandas for statistical analysis, z-score anomaly detection, and hourly/daily traffic pattern profiling
detecting-evasion-techniques-in-endpoint-logs
Detects defense evasion techniques used by adversaries in endpoint logs including log tampering, timestomping, process injection, and security tool disabling. Use when investigating suspicious endpoint behavior, building detection rules for evasion tactics, or conducting threat hunting for stealthy adversary activity. Activates for requests involving evasion detection, defense evasion analysis, log tampering detection, or MITRE ATT&CK TA0005.
implementing-microsegmentation-with-guardicore
Implementing microsegmentation using Akamai Guardicore Segmentation to map application dependencies, create granular network policies, visualize east-west traffic flows, and enforce least-privilege communication between workloads across data centers and cloud.
ramdisk-tmpfs-patterns
暂无简介