[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7EWAP6JSC4AP2X65N1H2T89":3,"skill-same:01M7EWAP6JSC4AP2X65N1H2T89":47},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":25,"commitSha":26,"skillDirectory":27,"forksCount":28,"repositoryHtmlUrl":29,"repositoryHomepageUrl":6,"ownerHtmlUrl":30,"ownerBlogUrl":31,"ownerDescription":6,"updatedAt":32,"install":33,"resources":46},"01M7EWAP6JSC4AP2X65N1H2T89","vllm-serving",null,"Serve LLMs at scale with high-throughput inference servers — continuous batching, paged attention, tensor parallelism, and production operations. Use when self-hosting models for many users or high QPS.","TanNhatCMS\u002Fjarvis","TanNhatCMS","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F113796420?v=4","01M753MV17KG5CD3152G3WQEQY","KietNT","01M7EKEDYVYQ6PWVJJMZ911VMX",0,"MIT","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fams-vllm-serving\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: vllm-serving\ndescription: Serve LLMs at scale with high-throughput inference servers — continuous batching, paged attention, tensor parallelism, and production operations. Use when self-hosting models for many users or high QPS.\ncategory: ai-research\n---\n\n# High-Throughput LLM Serving\n\nServing LLMs efficiently is a systems problem: GPU memory is finite, requests arrive unpredictably, \nand naive serving wastes most of the hardware. Modern inference servers use continuous batching and \npaged memory management to multiply throughput — this skill covers operating them.\n\n## Overview\n\nThe key techniques: continuous batching (requests join and leave the batch dynamically instead of \nwaiting for fixed batches), paged KV-cache management (memory allocated in blocks, eliminating \nfragmentation), and tensor parallelism (splitting large models across GPUs). Together they raise \nthroughput several-fold over naive serving. Your job: configure for your workload, monitor the \nright metrics, and operate reliably.\n\n## When to use\n\n- Self-hosting open models for production traffic.\n- Batch inference over large datasets where throughput is cost.\n- Low-latency interactive serving with concurrent users.\n- Choosing between serving frameworks or sizing GPU infrastructure.\n\n## Core concepts\n\n- **Continuous batching**: the scheduler packs requests at the token level — new requests join \nmid-generation, finished ones leave. Utilization stays high under variable load.\n- **Paged KV cache**: the attention key-value cache managed in fixed blocks, like OS virtual \nmemory. Kills fragmentation; enables larger batches and longer contexts.\n- **Tensor parallelism**: splitting model layers across multiple GPUs for models that don't fit on \none. Needed for large models; adds communication overhead.\n- **Quantization for serving**: weight quantization (FP8\u002FINT8\u002FINT4) shrinks memory, allowing bigger \nbatches or smaller GPUs. Pair with the server's supported formats.\n- **Scheduling policies**: prioritizing by arrival, deadline, or prefix-sharing. Prefix caching \n(shared system prompts) saves recomputation across requests.\n- **Metrics that matter**: time-to-first-token (TTFT), inter-token latency, tokens\u002Fsec\u002FGPU, queue \nwait time, and GPU memory utilization. Optimize for your workload's sensitivity.\n\n## Practical workflow\n\n1. Size the deployment: model size + quantization → GPU count and memory headroom for KV cache.\n2. Configure: max batch tokens, max sequences, KV cache allocation — start from recommended \ndefaults for your GPU, then tune.\n3. Load-test with realistic traffic: concurrent users, real prompt-length distribution. Measure \nTTFT, latency, throughput.\n4. Tune the knobs that matter: batch sizes for throughput, parallelism for large models, prefix \ncaching for shared prompts.\n5. Set up monitoring and autoscaling: queue depth, latency percentiles, error rates — alert \nbefore users notice.\n6. Plan operations: rolling updates, model versioning, request logging (redacted), and capacity \nheadroom for spikes.\n\n```text\nServing checklist:\n[ ] GPU sizing: model + KV cache headroom calculated\n[ ] Quantization chosen and quality-validated\n[ ] Load-tested with realistic traffic mix\n[ ] TTFT + inter-token latency within targets\n[ ] Monitoring: queue depth, latency p95\u002Fp99, errors\n[ ] Autoscaling + rolling update plan\n```\n\n## Common pitfalls\n\n- **Under-provisioned KV cache**: long contexts + many concurrent requests exhaust memory. Size for \nyour actual context lengths.\n- **Benchmarking with toy prompts**: uniform short prompts hide real-world variance. Test with your \ndistribution.\n- **Ignoring TTFT**: optimizing tokens\u002Fsec while users wait seconds for the first token. Match \nmetrics to UX.\n- **No quantization**: serving FP16 when INT8\u002FFP8 would double capacity at negligible quality cost. \nValidate and quantize.\n- **Single-GPU thinking**: large models need tensor parallelism; misconfigured parallelism wastes \nGPUs.\n- **No backpressure**: accepting unlimited requests until latency explodes. Queue with limits; shed \nor degrade gracefully.\n","# High-Throughput LLM Serving\n\nServing LLMs efficiently is a systems problem: GPU memory is finite, requests arrive unpredictably, \nand naive serving wastes most of the hardware. Modern inference servers use continuous batching and \npaged memory management to multiply throughput — this skill covers operating them.\n\n## Overview\n\nThe key techniques: continuous batching (requests join and leave the batch dynamically instead of \nwaiting for fixed batches), paged KV-cache management (memory allocated in blocks, eliminating \nfragmentation), and tensor parallelism (splitting large models across GPUs). Together they raise \nthroughput several-fold over naive serving. Your job: configure for your workload, monitor the \nright metrics, and operate reliably.\n\n## When to use\n\n- Self-hosting open models for production traffic.\n- Batch inference over large datasets where throughput is cost.\n- Low-latency interactive serving with concurrent users.\n- Choosing between serving frameworks or sizing GPU infrastructure.\n\n## Core concepts\n\n- **Continuous batching**: the scheduler packs requests at the token level — new requests join \nmid-generation, finished ones leave. Utilization stays high under variable load.\n- **Paged KV cache**: the attention key-value cache managed in fixed blocks, like OS virtual \nmemory. Kills fragmentation; enables larger batches and longer contexts.\n- **Tensor parallelism**: splitting model layers across multiple GPUs for models that don't fit on \none. Needed for large models; adds communication overhead.\n- **Quantization for serving**: weight quantization (FP8\u002FINT8\u002FINT4) shrinks memory, allowing bigger \nbatches or smaller GPUs. Pair with the server's supported formats.\n- **Scheduling policies**: prioritizing by arrival, deadline, or prefix-sharing. Prefix caching \n(shared system prompts) saves recomputation across requests.\n- **Metrics that matter**: time-to-first-token (TTFT), inter-token latency, tokens\u002Fsec\u002FGPU, queue \nwait time, and GPU memory utilization. Optimize for your workload's sensitivity.\n\n## Practical workflow\n\n1. Size the deployment: model size + quantization → GPU count and memory headroom for KV cache.\n2. Configure: max batch tokens, max sequences, KV cache allocation — start from recommended \ndefaults for your GPU, then tune.\n3. Load-test with realistic traffic: concurrent users, real prompt-length distribution. Measure \nTTFT, latency, throughput.\n4. Tune the knobs that matter: batch sizes for throughput, parallelism for large models, prefix \ncaching for shared prompts.\n5. Set up monitoring and autoscaling: queue depth, latency percentiles, error rates — alert \nbefore users notice.\n6. Plan operations: rolling updates, model versioning, request logging (redacted), and capacity \nheadroom for spikes.\n\n```text\nServing checklist:\n[ ] GPU sizing: model + KV cache headroom calculated\n[ ] Quantization chosen and quality-validated\n[ ] Load-tested with realistic traffic mix\n[ ] TTFT + inter-token latency within targets\n[ ] Monitoring: queue depth, latency p95\u002Fp99, errors\n[ ] Autoscaling + rolling update plan\n```\n\n## Common pitfalls\n\n- **Under-provisioned KV cache**: long contexts + many concurrent requests exhaust memory. Size for \nyour actual context lengths.\n- **Benchmarking with toy prompts**: uniform short prompts hide real-world variance. Test with your \ndistribution.\n- **Ignoring TTFT**: optimizing tokens\u002Fsec while users wait seconds for the first token. Match \nmetrics to UX.\n- **No quantization**: serving FP16 when INT8\u002FFP8 would double capacity at negligible quality cost. \nValidate and quantize.\n- **Single-GPU thinking**: large models need tensor parallelism; misconfigured parallelism wastes \nGPUs.\n- **No backpressure**: accepting unlimited requests until latency explodes. Queue with limits; shed \nor degrade gracefully.\n",{"name":5,"category":24,"description":7},"ai-research","main","af388a77c2fe5e0b4fd879cca4300a0ffa9a6579","skills\u002Fams-vllm-serving",1,"https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis","https:\u002F\u002Fgithub.com\u002FTanNhatCMS","https:\u002F\u002Fmrkiet.name.vn","2026-11-07 14:23:08",{"source":8,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add TanNhatCMS\u002Fjarvis --skill vllm-serving",{"tool":39,"command":40},"bunx","bunx skills add TanNhatCMS\u002Fjarvis --skill vllm-serving",{"tool":42,"command":43},"pnpm","pnpm dlx skills add TanNhatCMS\u002Fjarvis --skill vllm-serving","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fams-vllm-serving\u002FSKILL.md",true,[],[48,55,63,70,77,84],{"skillNo":49,"name":50,"nameZh":6,"description":51,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":52,"categories":53,"keywords":54,"publishedAt":19},"01M7EWYJR0J8DARY7E59Z6Y9MV","web-scrape","Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open\u002Ffetch fails on JS-heavy pages, or when you need clean markdown\u002Fstructured extraction from a site. Requires FIRECRAWL_API_KEY.","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fweb-scrape\u002FSKILL.md",[],[],{"skillNo":56,"name":57,"nameZh":6,"description":58,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":59,"sourceHtmlUrl":60,"categories":61,"keywords":62,"publishedAt":19},"01M7EWYBN5P6F2AJG3FR24T9BP","design-taste-frontend","tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML\u002FCSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, \"AI-made\", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Ftaste-design\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Ftaste-design\u002FSKILL.md",[],[],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":19},"01M7EWY8FQFNQ2GB7NZZWGSSDX","test-driven-development","Use when implementing any feature or bugfix, before writing implementation code","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-tdd\u002FSKILL.md",[],[],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M7EWY5SHBFRH1HDHVWM9K7TV","systematic-debugging","Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-debugging\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":19},"01M7EWY3NM0Z0JAKS7HDZ24K66","requesting-code-review","Use when completing tasks, implementing major features, or before merging to verify work meets requirements","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-code-review\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":19},"01M7EWY0WRMWX4M9ZFY0WKC593","brainstorming","You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-brainstorming\u002FSKILL.md",[],[]]