AGENT SKILL

disaggregated-prefill-decode

暂无简介

License 许可
Apache-2.0
最近更新
2026-11-14
Stars
★ 13
Forks
2

03 SKILL.md 正文

/disaggregated-prefill-decode

When to Use

  • Inference spend >$500K/year on colocated serving — disaggregation can save 30–40%
  • Mixed workloads with both long-prompt (prefill-heavy) and long-output (decode-heavy) requests
  • MoE models (DeepSeek-R1, Mixtral) where colocation wastes H100 compute on memory-bound decode
  • Already on Kubernetes and want CNCF-native orchestration (llm-d)

Do NOT use for

  • Short prompts + short outputs (<512 tokens each) — NIXL transfer tax exceeds savings
  • Small-scale serving (<10 GPUs) — orchestration overhead not justified
  • Development / testing environments — complexity far outweighs benefit

Why the bottlenecks differ

Prefill  — full forward pass over input prompt → matrix multiplications dominate
           → COMPUTE-BOUND — H100 FP8 ~2,000 TFLOPS
           → batch many tokens in one forward → high GPU utilization

Decode   — one token per step, reading full weights each time
           → MEMORY-BANDWIDTH-BOUND — HBM3 ~3 TB/s
           → only amortizes at high concurrency

Colocation problem:
  You buy H100 compute for decode (memory bound → compute wasted)
  You buy H100 HBM for prefill (compute bound → memory wasted)
  20–40% GPU time wasted on the wrong resource at typical workloads.

Architecture diagram

            ┌──────────────┐
  Request → │    Router    │ ─────────────────────────┐
            └──────┬───────┘                          │
                   │ (prompt only)                    │
                   ▼                                  │
            ┌──────────────┐   KV cache via NIXL   ┌──▼───────────┐
            │ Prefill pool │ ──────────────────────► │ Decode pool  │
            │ (compute)    │                         │ (memory)     │
            └──────────────┘                         └──────┬───────┘
                                                            │ tokens
                                                            ▼
                                                         Client

NIXL: NVIDIA inter-node transport
  - Uses RDMA/InfiniBand when available
  - TCP fallback otherwise
  - Typical KV transfer latency: 20–80 ms for 4K-token prompt on 70B FP8
  → Short prompts (<512 tokens): transfer tax > compute savings → don't disaggregate

NVIDIA Dynamo (GTC 2025 GA)

# Dynamo sits ABOVE vLLM / SGLang / TRT-LLM as an orchestrator
# Rust core, Python extensibility

# Install
pip install nvidia-dynamo

# Launch with disaggregated P/D
dynamo serve \
  --model deepseek-ai/DeepSeek-R1 \
  --prefill-replicas 2 \
  --decode-replicas 4 \
  --kv-transfer nixl \
  --sla-planner auto

# Dynamo Planner Profiler auto-measures workload and sets P:D ratio
# Published results (developer.nvidia.com, 2025-06):
#   DeepSeek-R1 MoE on GB200 NVL72 + Dynamo: ~6x throughput in medium-latency regime
#   GB300 NVL72 + Dynamo: up to 50x MoE throughput vs Hopper (product page, undated)

llm-d (Red Hat + AWS, Kubernetes-native)

# llm-d separates prefill / decode / router as independent K8s Services
# Per-role HPA based on correct signals for each role

# Prefill Deployment — scale on queue depth
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prefill-server
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: prefill
          image: llmd/vllm-prefill:latest
          env:
            - name: ROLE
              value: prefill
            - name: NIXL_ENDPOINT
              value: decode-service:8001

---
# Decode Deployment — scale on KV utilization
apiVersion: apps/v1
kind: Deployment
metadata:
  name: decode-server
spec:
  replicas: 4
  template:
    spec:
      containers:
        - name: decode
          image: llmd/vllm-decode:latest
          env:
            - name: ROLE
              value: decode

---
# HPA for prefill (queue depth signal)
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: prefill-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: prefill-server
  minReplicas: 1
  maxReplicas: 8
  metrics:
    - type: External
      external:
        metric:
          name: llmd_prefill_queue_depth
        target:
          type: AverageValue
          averageValue: "10"

When NOT to disaggregate

def should_disaggregate(
    avg_prompt_tokens: int,
    avg_output_tokens: int,
    monthly_gpu_spend: float,
) -> tuple[bool, str]:
    if avg_prompt_tokens < 512 and avg_output_tokens < 512:
        return False, "Short prompts + short outputs: NIXL transfer tax exceeds savings"
    if monthly_gpu_spend < 40_000:
        return False, "Orchestration complexity not justified under ~$40K/month"
    return True, f"Expected savings: 30–40% on ${monthly_gpu_spend:,.0f}/month"

Dynamo vs llm-d: choose based on context

Dynamo  → already using NVIDIA stack (TRT-LLM, vLLM)
           want managed P:D ratio tuning (SLA Planner auto-configures)
           comfortable with NVIDIA licensing + support

llm-d   → committed to Kubernetes / CNCF ecosystem
           need GitOps-friendly declarative config (YAML deployments)
           want scale-to-zero + LoRA routing + hierarchical KV offloading (v0.5)
           Red Hat/AWS support preferred

Anti-Fake-Pass Checklist

❌ Short prompts (<512 tokens): NIXL transfer latency dominates → no throughput gain
❌ Prefill:decode ratio not tuned to workload → wrong pool sizes → one is idle, one is bottlenecked
❌ RDMA not available → TCP fallback → higher latency → reduces the savings window
❌ Same GPU type for both pools → defeats the purpose; prefill wants compute, decode wants memory bandwidth
❌ No KV cache compression before NIXL transfer → 70B 4K-prompt KV cache can be 4–8 GB per request
❌ Citing "30x" as a guaranteed number — it's a community aggregate on specific Blackwell+Dynamo+R1 stacks

04 同仓库更多 Skills

Horonomos-laps
Horonomos-laps/Yana-AI

openai--sentry--sentry

Use when the user asks to inspect Sentry issues or events, summarize recent production errors, or pull basic Sentry health data via the Sentry API; perform read-only queries with the bundled script and require `SENTRY_AUTH_TOKEN`.

★ 13
Horonomos-laps
Horonomos-laps/Yana-AI

ito-basket-compare

Compare Itô prediction-market baskets against a user's knowledge base, portfolio notes, financial context, watchlist, or research thesis. Use for read-only basket comparison and gap analysis without investment advice or live trading.

★ 13
Horonomos-laps
Horonomos-laps/Yana-AI

implementing-network-traffic-baselining

Build network traffic baselines from NetFlow/IPFIX data using Python pandas for statistical analysis, z-score anomaly detection, and hourly/daily traffic pattern profiling

★ 13
Horonomos-laps
Horonomos-laps/Yana-AI

detecting-evasion-techniques-in-endpoint-logs

Detects defense evasion techniques used by adversaries in endpoint logs including log tampering, timestomping, process injection, and security tool disabling. Use when investigating suspicious endpoint behavior, building detection rules for evasion tactics, or conducting threat hunting for stealthy adversary activity. Activates for requests involving evasion detection, defense evasion analysis, log tampering detection, or MITRE ATT&CK TA0005.

★ 13
Horonomos-laps
Horonomos-laps/Yana-AI

implementing-microsegmentation-with-guardicore

Implementing microsegmentation using Akamai Guardicore Segmentation to map application dependencies, create granular network policies, visualize east-west traffic flows, and enforce least-privilege communication between workloads across data centers and cloud.

★ 13
Horonomos-laps
Horonomos-laps/Yana-AI

ramdisk-tmpfs-patterns

暂无简介

★ 13