sglang-guide
Serve LLMs fast with SGLang — structured generation, radix attention caching, and high-throughput inference runtimes.
- License 许可
- MIT
- 最近更新
- 2026-11-07
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- sglang-guide
- category
- ai-research
- description
- Serve LLMs fast with SGLang — structured generation, radix attention caching, and high-throughput inference runtimes.
03 SKILL.md 正文
Overview
SGLang is a serving framework for large language models that combines a fast runtime with a structured-generation frontend language. On the backend, its headline innovation is RadixAttention — a radix-tree-based KV cache that reuses computation across requests sharing prefixes (system prompts, few-shot examples, conversation history), dramatically raising throughput for workloads with common prefixes. On the frontend, SGLang's language lets you express constrained generation (choices, regex, JSON schemas) and multi-step programs that run efficiently against the runtime.
Use SGLang when you're self-hosting open models and throughput matters: many concurrent requests, long shared prefixes, or structured outputs at scale. It's an inference-engine choice (like vLLM or TensorRT-LLM) with the extra benefit of a programming model for structured generation built in.
The mental model: SGLang treats your serving workload as programs with shared structure, and exploits that structure — prefix reuse in the cache, constraint awareness in the scheduler — to serve more requests per GPU.
When to use
- Self-hosting open models (Llama, Qwen, DeepSeek, etc.) with high request concurrency.
- Workloads with large shared prefixes: same system prompt across requests, few-shot templates, multi-turn conversations, agent tool loops.
- Structured generation at scale: JSON outputs, constrained choices, regex formats — expressed in SGLang's frontend language.
- Multi-step LLM programs (generate → verify → regenerate) where the runtime can optimize the whole program.
- Research on serving efficiency, caching, or constrained decoding.
- Cost-per-token optimization for open-model deployments.
Core concepts
- RadixAttention / prefix caching: the KV cache is organized as a radix tree, so requests sharing a prefix reuse its computed key-values. The win scales with prefix length and sharing — measure hit rates on your workload.
- Structured generation DSL: express outputs as
genwith choices, regexes, or JSON schemas, composed into programs with control flow. Constraints execute against the runtime efficiently (no naive retry loops). - Continuous batching: requests join and leave the batch dynamically as they finish, keeping the GPU saturated. Standard in modern servers; SGLang's implementation is tuned around its cache design.
- Cache-aware scheduling: the scheduler considers cache locality when ordering requests — grouping requests with shared prefixes to maximize reuse. This is where the radix design pays off beyond raw caching.
- Multi-step programs: the frontend language supports branching, loops, and
parallel generation (
fork) within one program, with the runtime optimizing across steps (e.g., shared prefixes between branches). - OpenAI-compatible API: serve with a familiar
/v1interface so existing clients work unchanged; the SGLang-specific features are available through its own client/language. - Quantization support: run quantized models to fit larger models or higher concurrency per GPU. Match quantization to your quality bar with evals.
- Disaggregated / distributed serving: options for splitting prefill and decode across instances and for tensor/pipeline parallelism on multi-GPU setups.
Practical workflow
- Profile your workload's prefix sharing. Estimate how much of each request is shared prefix (system prompt, examples, history). High sharing → SGLang's cache design wins big; fully independent short prompts → less advantage.
- Deploy the server. Container-based deployment with your model weights; configure parallelism (tensor parallel across GPUs) and quantization to fit your hardware and latency targets.
- Benchmark throughput and latency. Measure tokens/sec, time-to-first-token, and inter-token latency at your target concurrency. Tune max batch/token budgets against your SLOs.
- Express structured outputs in the DSL. Replace "output JSON" prompts with
schema-constrained
gencalls; replace retry loops with constrained generation. Measure the quality and cost delta. - Monitor cache hit rates. Track prefix cache hits — a low hit rate means your workload doesn't share structure, or your prefix construction is inconsistent (e.g., nondeterministic prompt assembly defeating the cache).
- Stabilize prompt construction. Build prompts deterministically — same logical prefix should produce byte-identical prefix strings, or the cache can't match them.
- Load-test before committing. Soak-test at peak concurrency; watch for latency cliffs, OOMs, and cache eviction churn under sustained load.
Checklist for an SGLang deployment:
- Prefix sharing quantified; cache hit rates monitored.
- Throughput/latency benchmarked at target concurrency.
- Structured outputs expressed as constraints, not prompts.
- Prompt construction deterministic for cache matching.
- Quantization quality validated against the full-precision baseline.
Common pitfalls
- Nondeterministic prefixes. Timestamps, random example ordering, or user IDs embedded early in the prompt defeat prefix caching silently. Keep shared content byte-identical and put variable content late.
- Expecting wins on unshared workloads. If every request is a unique short prompt, RadixAttention adds little — SGLang is still a fine server, but the headline benefit needs shared structure.
- Ignoring time-to-first-token. Throughput optimizations can trade TTFT for tokens/sec. If your UX needs fast first tokens (chat), tune for it explicitly.
- Constraint overuse. Constraining every generation adds overhead; use constraints where structure is required, plain generation elsewhere.
- Undersized GPU memory planning. KV cache sizing interacts with batch size and sequence length. Plan memory for your p99 sequence lengths, not the mean.
- No evals on quantized models. Quantization can subtly degrade quality on your specific task. Evaluate the quantized model on your task before production.
- Treating the DSL as required. You can use SGLang purely as an OpenAI-compatible server. Adopt the frontend language where it helps; don't rewrite working clients unnecessarily.
- Skipping soak tests. Serving looks fine at 10 concurrent requests and falls over at 200. Load-test at realistic peaks plus headroom.
04 同仓库更多 Skills
web-scrape
Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open/fetch fails on JS-heavy pages, or when you need clean markdown/structured extraction from a site. Requires FIRECRAWL_API_KEY.
design-taste-frontend
tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML/CSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, "AI-made", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.
test-driven-development
Use when implementing any feature or bugfix, before writing implementation code
systematic-debugging
Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes
requesting-code-review
Use when completing tasks, implementing major features, or before merging to verify work meets requirements
brainstorming
You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).