AGENT SKILL

vllm-serving

Serve LLMs at scale with high-throughput inference servers — continuous batching, paged attention, tensor parallelism, and production operations. Use when self-hosting models for many users or high QPS.

License 许可
MIT
所属仓库
TanNhatCMS/jarvis
最近更新
2026-11-07
Stars
★ 0
Forks
1

01 前言 Front Matter

name
vllm-serving
category
ai-research
description
Serve LLMs at scale with high-throughput inference servers — continuous batching, paged attention, tensor parallelism, and production operations. Use when self-hosting models for many users or high QPS.

03 SKILL.md 正文

High-Throughput LLM Serving

Serving LLMs efficiently is a systems problem: GPU memory is finite, requests arrive unpredictably, and naive serving wastes most of the hardware. Modern inference servers use continuous batching and paged memory management to multiply throughput — this skill covers operating them.

Overview

The key techniques: continuous batching (requests join and leave the batch dynamically instead of waiting for fixed batches), paged KV-cache management (memory allocated in blocks, eliminating fragmentation), and tensor parallelism (splitting large models across GPUs). Together they raise throughput several-fold over naive serving. Your job: configure for your workload, monitor the right metrics, and operate reliably.

When to use

  • Self-hosting open models for production traffic.
  • Batch inference over large datasets where throughput is cost.
  • Low-latency interactive serving with concurrent users.
  • Choosing between serving frameworks or sizing GPU infrastructure.

Core concepts

  • Continuous batching: the scheduler packs requests at the token level — new requests join mid-generation, finished ones leave. Utilization stays high under variable load.
  • Paged KV cache: the attention key-value cache managed in fixed blocks, like OS virtual memory. Kills fragmentation; enables larger batches and longer contexts.
  • Tensor parallelism: splitting model layers across multiple GPUs for models that don't fit on one. Needed for large models; adds communication overhead.
  • Quantization for serving: weight quantization (FP8/INT8/INT4) shrinks memory, allowing bigger batches or smaller GPUs. Pair with the server's supported formats.
  • Scheduling policies: prioritizing by arrival, deadline, or prefix-sharing. Prefix caching (shared system prompts) saves recomputation across requests.
  • Metrics that matter: time-to-first-token (TTFT), inter-token latency, tokens/sec/GPU, queue wait time, and GPU memory utilization. Optimize for your workload's sensitivity.

Practical workflow

  1. Size the deployment: model size + quantization → GPU count and memory headroom for KV cache.
  2. Configure: max batch tokens, max sequences, KV cache allocation — start from recommended defaults for your GPU, then tune.
  3. Load-test with realistic traffic: concurrent users, real prompt-length distribution. Measure TTFT, latency, throughput.
  4. Tune the knobs that matter: batch sizes for throughput, parallelism for large models, prefix caching for shared prompts.
  5. Set up monitoring and autoscaling: queue depth, latency percentiles, error rates — alert before users notice.
  6. Plan operations: rolling updates, model versioning, request logging (redacted), and capacity headroom for spikes.
Serving checklist:
[ ] GPU sizing: model + KV cache headroom calculated
[ ] Quantization chosen and quality-validated
[ ] Load-tested with realistic traffic mix
[ ] TTFT + inter-token latency within targets
[ ] Monitoring: queue depth, latency p95/p99, errors
[ ] Autoscaling + rolling update plan

Common pitfalls

  • Under-provisioned KV cache: long contexts + many concurrent requests exhaust memory. Size for your actual context lengths.
  • Benchmarking with toy prompts: uniform short prompts hide real-world variance. Test with your distribution.
  • Ignoring TTFT: optimizing tokens/sec while users wait seconds for the first token. Match metrics to UX.
  • No quantization: serving FP16 when INT8/FP8 would double capacity at negligible quality cost. Validate and quantize.
  • Single-GPU thinking: large models need tensor parallelism; misconfigured parallelism wastes GPUs.
  • No backpressure: accepting unlimited requests until latency explodes. Queue with limits; shed or degrade gracefully.

04 同仓库更多 Skills

TanNhatCMS
TanNhatCMS/jarvis

web-scrape

Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open/fetch fails on JS-heavy pages, or when you need clean markdown/structured extraction from a site. Requires FIRECRAWL_API_KEY.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

design-taste-frontend

tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML/CSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, "AI-made", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

test-driven-development

Use when implementing any feature or bugfix, before writing implementation code

★ 0
TanNhatCMS
TanNhatCMS/jarvis

systematic-debugging

Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes

★ 0
TanNhatCMS
TanNhatCMS/jarvis

requesting-code-review

Use when completing tasks, implementing major features, or before merging to verify work meets requirements

★ 0
TanNhatCMS
TanNhatCMS/jarvis

brainstorming

You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).

★ 0