AGENT SKILL

sglang-guide

Serve LLMs fast with SGLang — structured generation, radix attention caching, and high-throughput inference runtimes.

License 许可
MIT
所属仓库
TanNhatCMS/jarvis
最近更新
2026-11-07
Stars
★ 0
Forks
1

01 前言 Front Matter

name
sglang-guide
category
ai-research
description
Serve LLMs fast with SGLang — structured generation, radix attention caching, and high-throughput inference runtimes.

03 SKILL.md 正文

Overview

SGLang is a serving framework for large language models that combines a fast runtime with a structured-generation frontend language. On the backend, its headline innovation is RadixAttention — a radix-tree-based KV cache that reuses computation across requests sharing prefixes (system prompts, few-shot examples, conversation history), dramatically raising throughput for workloads with common prefixes. On the frontend, SGLang's language lets you express constrained generation (choices, regex, JSON schemas) and multi-step programs that run efficiently against the runtime.

Use SGLang when you're self-hosting open models and throughput matters: many concurrent requests, long shared prefixes, or structured outputs at scale. It's an inference-engine choice (like vLLM or TensorRT-LLM) with the extra benefit of a programming model for structured generation built in.

The mental model: SGLang treats your serving workload as programs with shared structure, and exploits that structure — prefix reuse in the cache, constraint awareness in the scheduler — to serve more requests per GPU.

When to use

  • Self-hosting open models (Llama, Qwen, DeepSeek, etc.) with high request concurrency.
  • Workloads with large shared prefixes: same system prompt across requests, few-shot templates, multi-turn conversations, agent tool loops.
  • Structured generation at scale: JSON outputs, constrained choices, regex formats — expressed in SGLang's frontend language.
  • Multi-step LLM programs (generate → verify → regenerate) where the runtime can optimize the whole program.
  • Research on serving efficiency, caching, or constrained decoding.
  • Cost-per-token optimization for open-model deployments.

Core concepts

  • RadixAttention / prefix caching: the KV cache is organized as a radix tree, so requests sharing a prefix reuse its computed key-values. The win scales with prefix length and sharing — measure hit rates on your workload.
  • Structured generation DSL: express outputs as gen with choices, regexes, or JSON schemas, composed into programs with control flow. Constraints execute against the runtime efficiently (no naive retry loops).
  • Continuous batching: requests join and leave the batch dynamically as they finish, keeping the GPU saturated. Standard in modern servers; SGLang's implementation is tuned around its cache design.
  • Cache-aware scheduling: the scheduler considers cache locality when ordering requests — grouping requests with shared prefixes to maximize reuse. This is where the radix design pays off beyond raw caching.
  • Multi-step programs: the frontend language supports branching, loops, and parallel generation (fork) within one program, with the runtime optimizing across steps (e.g., shared prefixes between branches).
  • OpenAI-compatible API: serve with a familiar /v1 interface so existing clients work unchanged; the SGLang-specific features are available through its own client/language.
  • Quantization support: run quantized models to fit larger models or higher concurrency per GPU. Match quantization to your quality bar with evals.
  • Disaggregated / distributed serving: options for splitting prefill and decode across instances and for tensor/pipeline parallelism on multi-GPU setups.

Practical workflow

  1. Profile your workload's prefix sharing. Estimate how much of each request is shared prefix (system prompt, examples, history). High sharing → SGLang's cache design wins big; fully independent short prompts → less advantage.
  2. Deploy the server. Container-based deployment with your model weights; configure parallelism (tensor parallel across GPUs) and quantization to fit your hardware and latency targets.
  3. Benchmark throughput and latency. Measure tokens/sec, time-to-first-token, and inter-token latency at your target concurrency. Tune max batch/token budgets against your SLOs.
  4. Express structured outputs in the DSL. Replace "output JSON" prompts with schema-constrained gen calls; replace retry loops with constrained generation. Measure the quality and cost delta.
  5. Monitor cache hit rates. Track prefix cache hits — a low hit rate means your workload doesn't share structure, or your prefix construction is inconsistent (e.g., nondeterministic prompt assembly defeating the cache).
  6. Stabilize prompt construction. Build prompts deterministically — same logical prefix should produce byte-identical prefix strings, or the cache can't match them.
  7. Load-test before committing. Soak-test at peak concurrency; watch for latency cliffs, OOMs, and cache eviction churn under sustained load.

Checklist for an SGLang deployment:

  • Prefix sharing quantified; cache hit rates monitored.
  • Throughput/latency benchmarked at target concurrency.
  • Structured outputs expressed as constraints, not prompts.
  • Prompt construction deterministic for cache matching.
  • Quantization quality validated against the full-precision baseline.

Common pitfalls

  • Nondeterministic prefixes. Timestamps, random example ordering, or user IDs embedded early in the prompt defeat prefix caching silently. Keep shared content byte-identical and put variable content late.
  • Expecting wins on unshared workloads. If every request is a unique short prompt, RadixAttention adds little — SGLang is still a fine server, but the headline benefit needs shared structure.
  • Ignoring time-to-first-token. Throughput optimizations can trade TTFT for tokens/sec. If your UX needs fast first tokens (chat), tune for it explicitly.
  • Constraint overuse. Constraining every generation adds overhead; use constraints where structure is required, plain generation elsewhere.
  • Undersized GPU memory planning. KV cache sizing interacts with batch size and sequence length. Plan memory for your p99 sequence lengths, not the mean.
  • No evals on quantized models. Quantization can subtly degrade quality on your specific task. Evaluate the quantized model on your task before production.
  • Treating the DSL as required. You can use SGLang purely as an OpenAI-compatible server. Adopt the frontend language where it helps; don't rewrite working clients unnecessarily.
  • Skipping soak tests. Serving looks fine at 10 concurrent requests and falls over at 200. Load-test at realistic peaks plus headroom.

04 同仓库更多 Skills

TanNhatCMS
TanNhatCMS/jarvis

web-scrape

Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open/fetch fails on JS-heavy pages, or when you need clean markdown/structured extraction from a site. Requires FIRECRAWL_API_KEY.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

design-taste-frontend

tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML/CSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, "AI-made", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

test-driven-development

Use when implementing any feature or bugfix, before writing implementation code

★ 0
TanNhatCMS
TanNhatCMS/jarvis

systematic-debugging

Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes

★ 0
TanNhatCMS
TanNhatCMS/jarvis

requesting-code-review

Use when completing tasks, implementing major features, or before merging to verify work meets requirements

★ 0
TanNhatCMS
TanNhatCMS/jarvis

brainstorming

You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).

★ 0