[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7EVXYK1JJ39RY6Z0RJS2BZG":3,"skill-same:01M7EVXYK1JJ39RY6Z0RJS2BZG":47},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":28,"repositoryHtmlUrl":29,"repositoryHomepageUrl":6,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":6,"updatedAt":32,"install":33,"resources":46},"01M7EVXYK1JJ39RY6Z0RJS2BZG","sglang-guide",null,"Serve LLMs fast with SGLang — structured generation, radix attention caching, and high-throughput inference runtimes.","TanNhatCMS\u002Fjarvis","TanNhatCMS","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F113796420?v=4","01M753MV17KG5CD3152G3WQEQY","KietNT","01M7EKEDYVYQ6PWVJJMZ911VMX",0,"MIT","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fams-sglang-guide\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: sglang-guide\ndescription: Serve LLMs fast with SGLang — structured generation, radix attention caching, and high-throughput inference runtimes.\ncategory: ai-research\n---\n\n## Overview\n\nSGLang is a serving framework for large language models that combines a fast\nruntime with a structured-generation frontend language. On the backend, its\nheadline innovation is RadixAttention — a radix-tree-based KV cache that reuses\ncomputation across requests sharing prefixes (system prompts, few-shot examples,\nconversation history), dramatically raising throughput for workloads with common\nprefixes. On the frontend, SGLang's language lets you express constrained\ngeneration (choices, regex, JSON schemas) and multi-step programs that run\nefficiently against the runtime.\n\nUse SGLang when you're self-hosting open models and throughput matters: many\nconcurrent requests, long shared prefixes, or structured outputs at scale. It's\nan inference-engine choice (like vLLM or TensorRT-LLM) with the extra benefit of\na programming model for structured generation built in.\n\nThe mental model: SGLang treats your serving workload as programs with shared\nstructure, and exploits that structure — prefix reuse in the cache, constraint\nawareness in the scheduler — to serve more requests per GPU.\n\n## When to use\n\n- Self-hosting open models (Llama, Qwen, DeepSeek, etc.) with high request\n  concurrency.\n- Workloads with large shared prefixes: same system prompt across requests,\n  few-shot templates, multi-turn conversations, agent tool loops.\n- Structured generation at scale: JSON outputs, constrained choices, regex\n  formats — expressed in SGLang's frontend language.\n- Multi-step LLM programs (generate → verify → regenerate) where the runtime can\n  optimize the whole program.\n- Research on serving efficiency, caching, or constrained decoding.\n- Cost-per-token optimization for open-model deployments.\n\n## Core concepts\n\n- **RadixAttention \u002F prefix caching**: the KV cache is organized as a radix tree,\n  so requests sharing a prefix reuse its computed key-values. The win scales with\n  prefix length and sharing — measure hit rates on your workload.\n- **Structured generation DSL**: express outputs as `gen` with choices, regexes,\n  or JSON schemas, composed into programs with control flow. Constraints execute\n  against the runtime efficiently (no naive retry loops).\n- **Continuous batching**: requests join and leave the batch dynamically as they\n  finish, keeping the GPU saturated. Standard in modern servers; SGLang's\n  implementation is tuned around its cache design.\n- **Cache-aware scheduling**: the scheduler considers cache locality when ordering\n  requests — grouping requests with shared prefixes to maximize reuse. This is\n  where the radix design pays off beyond raw caching.\n- **Multi-step programs**: the frontend language supports branching, loops, and\n  parallel generation (`fork`) within one program, with the runtime optimizing\n  across steps (e.g., shared prefixes between branches).\n- **OpenAI-compatible API**: serve with a familiar `\u002Fv1` interface so existing\n  clients work unchanged; the SGLang-specific features are available through its\n  own client\u002Flanguage.\n- **Quantization support**: run quantized models to fit larger models or higher\n  concurrency per GPU. Match quantization to your quality bar with evals.\n- **Disaggregated \u002F distributed serving**: options for splitting prefill and\n  decode across instances and for tensor\u002Fpipeline parallelism on multi-GPU\n  setups.\n\n## Practical workflow\n\n1. **Profile your workload's prefix sharing.** Estimate how much of each request\n   is shared prefix (system prompt, examples, history). High sharing → SGLang's\n   cache design wins big; fully independent short prompts → less advantage.\n2. **Deploy the server.** Container-based deployment with your model weights;\n   configure parallelism (tensor parallel across GPUs) and quantization to fit\n   your hardware and latency targets.\n3. **Benchmark throughput and latency.** Measure tokens\u002Fsec, time-to-first-token,\n   and inter-token latency at your target concurrency. Tune max batch\u002Ftoken\n   budgets against your SLOs.\n4. **Express structured outputs in the DSL.** Replace \"output JSON\" prompts with\n   schema-constrained `gen` calls; replace retry loops with constrained\n   generation. Measure the quality and cost delta.\n5. **Monitor cache hit rates.** Track prefix cache hits — a low hit rate means\n   your workload doesn't share structure, or your prefix construction is\n   inconsistent (e.g., nondeterministic prompt assembly defeating the cache).\n6. **Stabilize prompt construction.** Build prompts deterministically — same\n   logical prefix should produce byte-identical prefix strings, or the cache\n   can't match them.\n7. **Load-test before committing.** Soak-test at peak concurrency; watch for\n   latency cliffs, OOMs, and cache eviction churn under sustained load.\n\nChecklist for an SGLang deployment:\n- Prefix sharing quantified; cache hit rates monitored.\n- Throughput\u002Flatency benchmarked at target concurrency.\n- Structured outputs expressed as constraints, not prompts.\n- Prompt construction deterministic for cache matching.\n- Quantization quality validated against the full-precision baseline.\n\n## Common pitfalls\n\n- **Nondeterministic prefixes.** Timestamps, random example ordering, or user IDs\n  embedded early in the prompt defeat prefix caching silently. Keep shared\n  content byte-identical and put variable content late.\n- **Expecting wins on unshared workloads.** If every request is a unique short\n  prompt, RadixAttention adds little — SGLang is still a fine server, but the\n  headline benefit needs shared structure.\n- **Ignoring time-to-first-token.** Throughput optimizations can trade TTFT for\n  tokens\u002Fsec. If your UX needs fast first tokens (chat), tune for it explicitly.\n- **Constraint overuse.** Constraining every generation adds overhead; use\n  constraints where structure is required, plain generation elsewhere.\n- **Undersized GPU memory planning.** KV cache sizing interacts with batch size\n  and sequence length. Plan memory for your p99 sequence lengths, not the mean.\n- **No evals on quantized models.** Quantization can subtly degrade quality on\n  your specific task. Evaluate the quantized model on your task before\n  production.\n- **Treating the DSL as required.** You can use SGLang purely as an\n  OpenAI-compatible server. Adopt the frontend language where it helps; don't\n  rewrite working clients unnecessarily.\n- **Skipping soak tests.** Serving looks fine at 10 concurrent requests and falls\n  over at 200. Load-test at realistic peaks plus headroom.\n","## Overview\n\nSGLang is a serving framework for large language models that combines a fast\nruntime with a structured-generation frontend language. On the backend, its\nheadline innovation is RadixAttention — a radix-tree-based KV cache that reuses\ncomputation across requests sharing prefixes (system prompts, few-shot examples,\nconversation history), dramatically raising throughput for workloads with common\nprefixes. On the frontend, SGLang's language lets you express constrained\ngeneration (choices, regex, JSON schemas) and multi-step programs that run\nefficiently against the runtime.\n\nUse SGLang when you're self-hosting open models and throughput matters: many\nconcurrent requests, long shared prefixes, or structured outputs at scale. It's\nan inference-engine choice (like vLLM or TensorRT-LLM) with the extra benefit of\na programming model for structured generation built in.\n\nThe mental model: SGLang treats your serving workload as programs with shared\nstructure, and exploits that structure — prefix reuse in the cache, constraint\nawareness in the scheduler — to serve more requests per GPU.\n\n## When to use\n\n- Self-hosting open models (Llama, Qwen, DeepSeek, etc.) with high request\n  concurrency.\n- Workloads with large shared prefixes: same system prompt across requests,\n  few-shot templates, multi-turn conversations, agent tool loops.\n- Structured generation at scale: JSON outputs, constrained choices, regex\n  formats — expressed in SGLang's frontend language.\n- Multi-step LLM programs (generate → verify → regenerate) where the runtime can\n  optimize the whole program.\n- Research on serving efficiency, caching, or constrained decoding.\n- Cost-per-token optimization for open-model deployments.\n\n## Core concepts\n\n- **RadixAttention \u002F prefix caching**: the KV cache is organized as a radix tree,\n  so requests sharing a prefix reuse its computed key-values. The win scales with\n  prefix length and sharing — measure hit rates on your workload.\n- **Structured generation DSL**: express outputs as `gen` with choices, regexes,\n  or JSON schemas, composed into programs with control flow. Constraints execute\n  against the runtime efficiently (no naive retry loops).\n- **Continuous batching**: requests join and leave the batch dynamically as they\n  finish, keeping the GPU saturated. Standard in modern servers; SGLang's\n  implementation is tuned around its cache design.\n- **Cache-aware scheduling**: the scheduler considers cache locality when ordering\n  requests — grouping requests with shared prefixes to maximize reuse. This is\n  where the radix design pays off beyond raw caching.\n- **Multi-step programs**: the frontend language supports branching, loops, and\n  parallel generation (`fork`) within one program, with the runtime optimizing\n  across steps (e.g., shared prefixes between branches).\n- **OpenAI-compatible API**: serve with a familiar `\u002Fv1` interface so existing\n  clients work unchanged; the SGLang-specific features are available through its\n  own client\u002Flanguage.\n- **Quantization support**: run quantized models to fit larger models or higher\n  concurrency per GPU. Match quantization to your quality bar with evals.\n- **Disaggregated \u002F distributed serving**: options for splitting prefill and\n  decode across instances and for tensor\u002Fpipeline parallelism on multi-GPU\n  setups.\n\n## Practical workflow\n\n1. **Profile your workload's prefix sharing.** Estimate how much of each request\n   is shared prefix (system prompt, examples, history). High sharing → SGLang's\n   cache design wins big; fully independent short prompts → less advantage.\n2. **Deploy the server.** Container-based deployment with your model weights;\n   configure parallelism (tensor parallel across GPUs) and quantization to fit\n   your hardware and latency targets.\n3. **Benchmark throughput and latency.** Measure tokens\u002Fsec, time-to-first-token,\n   and inter-token latency at your target concurrency. Tune max batch\u002Ftoken\n   budgets against your SLOs.\n4. **Express structured outputs in the DSL.** Replace \"output JSON\" prompts with\n   schema-constrained `gen` calls; replace retry loops with constrained\n   generation. Measure the quality and cost delta.\n5. **Monitor cache hit rates.** Track prefix cache hits — a low hit rate means\n   your workload doesn't share structure, or your prefix construction is\n   inconsistent (e.g., nondeterministic prompt assembly defeating the cache).\n6. **Stabilize prompt construction.** Build prompts deterministically — same\n   logical prefix should produce byte-identical prefix strings, or the cache\n   can't match them.\n7. **Load-test before committing.** Soak-test at peak concurrency; watch for\n   latency cliffs, OOMs, and cache eviction churn under sustained load.\n\nChecklist for an SGLang deployment:\n- Prefix sharing quantified; cache hit rates monitored.\n- Throughput\u002Flatency benchmarked at target concurrency.\n- Structured outputs expressed as constraints, not prompts.\n- Prompt construction deterministic for cache matching.\n- Quantization quality validated against the full-precision baseline.\n\n## Common pitfalls\n\n- **Nondeterministic prefixes.** Timestamps, random example ordering, or user IDs\n  embedded early in the prompt defeat prefix caching silently. Keep shared\n  content byte-identical and put variable content late.\n- **Expecting wins on unshared workloads.** If every request is a unique short\n  prompt, RadixAttention adds little — SGLang is still a fine server, but the\n  headline benefit needs shared structure.\n- **Ignoring time-to-first-token.** Throughput optimizations can trade TTFT for\n  tokens\u002Fsec. If your UX needs fast first tokens (chat), tune for it explicitly.\n- **Constraint overuse.** Constraining every generation adds overhead; use\n  constraints where structure is required, plain generation elsewhere.\n- **Undersized GPU memory planning.** KV cache sizing interacts with batch size\n  and sequence length. Plan memory for your p99 sequence lengths, not the mean.\n- **No evals on quantized models.** Quantization can subtly degrade quality on\n  your specific task. Evaluate the quantized model on your task before\n  production.\n- **Treating the DSL as required.** You can use SGLang purely as an\n  OpenAI-compatible server. Adopt the frontend language where it helps; don't\n  rewrite working clients unnecessarily.\n- **Skipping soak tests.** Serving looks fine at 10 concurrent requests and falls\n  over at 200. Load-test at realistic peaks plus headroom.\n",{"name":5,"category":24,"description":7},"ai-research","main","af388a77c2fe5e0b4fd879cca4300a0ffa9a6579","skills\u002Fams-sglang-guide",1,"https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis","https:\u002F\u002Fgithub.com\u002FTanNhatCMS","https:\u002F\u002Fmrkiet.name.vn","2026-11-07 14:23:08",{"source":8,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add TanNhatCMS\u002Fjarvis --skill sglang-guide",{"tool":39,"command":40},"bunx","bunx skills add TanNhatCMS\u002Fjarvis --skill sglang-guide",{"tool":42,"command":43},"pnpm","pnpm dlx skills add TanNhatCMS\u002Fjarvis --skill sglang-guide","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fams-sglang-guide\u002FSKILL.md",true,[],[48,55,63,70,77,84],{"skillNo":49,"name":50,"nameZh":6,"description":51,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":52,"categories":53,"keywords":54,"publishedAt":19},"01M7EWYJR0J8DARY7E59Z6Y9MV","web-scrape","Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open\u002Ffetch fails on JS-heavy pages, or when you need clean markdown\u002Fstructured extraction from a site. Requires FIRECRAWL_API_KEY.","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fweb-scrape\u002FSKILL.md",[],[],{"skillNo":56,"name":57,"nameZh":6,"description":58,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":59,"sourceHtmlUrl":60,"categories":61,"keywords":62,"publishedAt":19},"01M7EWYBN5P6F2AJG3FR24T9BP","design-taste-frontend","tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML\u002FCSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, \"AI-made\", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Ftaste-design\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Ftaste-design\u002FSKILL.md",[],[],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":19},"01M7EWY8FQFNQ2GB7NZZWGSSDX","test-driven-development","Use when implementing any feature or bugfix, before writing implementation code","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-tdd\u002FSKILL.md",[],[],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M7EWY5SHBFRH1HDHVWM9K7TV","systematic-debugging","Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-debugging\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":19},"01M7EWY3NM0Z0JAKS7HDZ24K66","requesting-code-review","Use when completing tasks, implementing major features, or before merging to verify work meets requirements","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-code-review\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":19},"01M7EWY0WRMWX4M9ZFY0WKC593","brainstorming","You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-brainstorming\u002FSKILL.md",[],[]]