[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MB5296ZNSAQGATPNRC4CVH":3,"skill-same:01M6MB5296ZNSAQGATPNRC4CVH":90},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":41,"linkOnly":42,"markdownContent":43,"markdownBody":44,"frontMatter":45,"refName":52,"commitSha":53,"skillDirectory":54,"forksCount":55,"repositoryHtmlUrl":56,"repositoryHomepageUrl":57,"ownerHtmlUrl":58,"ownerBlogUrl":59,"ownerDescription":60,"updatedAt":61,"install":62,"resources":75},"01M6MB5296ZNSAQGATPNRC4CVH","sglang",null,"Fast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON\u002Fregex outputs, constrained decoding, agentic workflows with tool calls, or when you need 5× faster inference than vLLM with prefix sharing. Powers 300,000+ GPUs at xAI, AMD, NVIDIA, and LinkedIn.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1554,"MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fsglang\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40],"SGLang","use","when","outputs","tool calls","serving","for","at","you","scale","regex","caching","agentic","json","tool","sharing","vLLM","NVIDIA","agentic workflows","workflows","inference","llms","2026-11-10 10:12:30",false,"---\nname: sglang\ndescription: Fast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON\u002Fregex outputs, constrained decoding, agentic workflows with tool calls, or when you need 5× faster inference than vLLM with prefix sharing. Powers 300,000+ GPUs at xAI, AMD, NVIDIA, and LinkedIn.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-inference-and-ops\n  maintainer: Kalaris Labs\n  tags: Inference Serving, SGLang, Structured Generation, RadixAttention, Prefix Caching, Constrained Decoding, Agents, JSON Output, Fast Inference, Production Scale\n  dependencies: sglang, torch, transformers\n---\n\n# SGLang\n\nHigh-performance serving framework for LLMs and VLMs with RadixAttention for automatic prefix caching.\n\n## When to use SGLang\n\n**Use SGLang when:**\n- Need structured outputs (JSON, regex, grammar)\n- Building agents with repeated prefixes (system prompts, tools)\n- Agentic workflows with function calling\n- Multi-turn conversations with shared context\n- Need faster JSON decoding (3× vs standard)\n\n**Use vLLM instead when:**\n- Simple text generation without structure\n- Don't need prefix caching\n- Want mature, widely-tested production system\n\n**Use TensorRT-LLM instead when:**\n- Maximum single-request latency (no batching needed)\n- NVIDIA-only deployment\n- Need FP8\u002FINT4 quantization on H100\n\n## Quick start\n\n### Installation\n\n```bash\n# pip install (recommended)\npip install \"sglang[all]\"\n\n# With FlashInfer (faster, CUDA 11.8\u002F12.1)\npip install sglang[all] flashinfer -i https:\u002F\u002Fflashinfer.ai\u002Fwhl\u002Fcu121\u002Ftorch2.4\u002F\n\n# From source\ngit clone https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang.git\ncd sglang\npip install -e \"python[all]\"\n```\n\n### Launch server\n\n```bash\n# Basic server (Llama 3-8B)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000\n\n# With RadixAttention (automatic prefix caching)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000 \\\n    --enable-radix-cache  # Default: enabled\n\n# Multi-GPU (tensor parallelism)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-70B-Instruct \\\n    --tp 4 \\\n    --port 30000\n```\n\n### Basic inference\n\n```python\nimport sglang as sgl\n\n# Set backend\nsgl.set_default_backend(sgl.OpenAI(\"http:\u002F\u002Flocalhost:30000\u002Fv1\"))\n\n# Simple generation\n@sgl.function\ndef simple_gen(s, question):\n    s += \"Q: \" + question + \"\\n\"\n    s += \"A:\" + sgl.gen(\"answer\", max_tokens=100)\n\n# Run\nstate = simple_gen.run(question=\"What is the capital of France?\")\nprint(state[\"answer\"])\n# Output: \"The capital of France is Paris.\"\n```\n\n### Structured JSON output\n\n```python\nimport sglang as sgl\n\n@sgl.function\ndef extract_person(s, text):\n    s += f\"Extract person information from: {text}\\n\"\n    s += \"Output JSON:\\n\"\n\n    # Constrained JSON generation\n    s += sgl.gen(\n        \"json_output\",\n        max_tokens=200,\n        regex=r'\\{\"name\": \"[^\"]+\", \"age\": \\d+, \"occupation\": \"[^\"]+\"\\}'\n    )\n\n# Run\nstate = extract_person.run(\n    text=\"John Smith is a 35-year-old software engineer.\"\n)\nprint(state[\"json_output\"])\n# Output: {\"name\": \"John Smith\", \"age\": 35, \"occupation\": \"software engineer\"}\n```\n\n## RadixAttention (Key Innovation)\n\n**What it does**: Automatically caches and reuses common prefixes across requests.\n\n**Performance**:\n- **5× faster** for agentic workloads with shared system prompts\n- **10× faster** for few-shot prompting with repeated examples\n- **Zero configuration** - works automatically\n\n**How it works**:\n1. Builds radix tree of all processed tokens\n2. Automatically detects shared prefixes\n3. Reuses KV cache for matching prefixes\n4. Only computes new tokens\n\n**Example** (Agent with system prompt):\n\n```\nRequest 1: [SYSTEM_PROMPT] + \"What's the weather?\"\n→ Computes full prompt (1000 tokens)\n\nRequest 2: [SAME_SYSTEM_PROMPT] + \"Book a flight\"\n→ Reuses system prompt KV cache (998 tokens)\n→ Only computes 2 new tokens\n→ 5× faster!\n```\n\n## Structured generation patterns\n\n### JSON with schema\n\n```python\n@sgl.function\ndef structured_extraction(s, article):\n    s += f\"Article: {article}\\n\\n\"\n    s += \"Extract key information as JSON:\\n\"\n\n    # JSON schema constraint\n    schema = {\n        \"type\": \"object\",\n        \"properties\": {\n            \"title\": {\"type\": \"string\"},\n            \"author\": {\"type\": \"string\"},\n            \"summary\": {\"type\": \"string\"},\n            \"sentiment\": {\"type\": \"string\", \"enum\": [\"positive\", \"negative\", \"neutral\"]}\n        },\n        \"required\": [\"title\", \"author\", \"summary\", \"sentiment\"]\n    }\n\n    s += sgl.gen(\"info\", max_tokens=300, json_schema=schema)\n\nstate = structured_extraction.run(article=\"...\")\nprint(state[\"info\"])\n# Output: Valid JSON matching schema\n```\n\n### Regex-constrained generation\n\n```python\n@sgl.function\ndef extract_email(s, text):\n    s += f\"Extract email from: {text}\\n\"\n    s += \"Email: \"\n\n    # Email regex pattern\n    s += sgl.gen(\n        \"email\",\n        max_tokens=50,\n        regex=r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}'\n    )\n\nstate = extract_email.run(text=\"Contact john.doe@example.com for details\")\nprint(state[\"email\"])\n# Output: \"john.doe@example.com\"\n```\n\n### Grammar-based generation\n\n```python\n@sgl.function\ndef generate_code(s, description):\n    s += f\"Generate Python code for: {description}\\n\"\n    s += \"```python\\n\"\n\n    # EBNF grammar for Python\n    python_grammar = \"\"\"\n    ?start: function_def\n    function_def: \"def\" NAME \"(\" [parameters] \"):\" suite\n    parameters: parameter (\",\" parameter)*\n    parameter: NAME\n    suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT\n    \"\"\"\n\n    s += sgl.gen(\"code\", max_tokens=200, grammar=python_grammar)\n    s += \"\\n```\"\n```\n\n## Agent workflows with function calling\n\n```python\nimport sglang as sgl\n\n# Define tools\ntools = [\n    {\n        \"name\": \"get_weather\",\n        \"description\": \"Get weather for a location\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"location\": {\"type\": \"string\"}\n            }\n        }\n    },\n    {\n        \"name\": \"book_flight\",\n        \"description\": \"Book a flight\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"from\": {\"type\": \"string\"},\n                \"to\": {\"type\": \"string\"},\n                \"date\": {\"type\": \"string\"}\n            }\n        }\n    }\n]\n\n@sgl.function\ndef agent_workflow(s, user_query, tools):\n    # System prompt (cached with RadixAttention)\n    s += \"You are a helpful assistant with access to tools.\\n\"\n    s += f\"Available tools: {tools}\\n\\n\"\n\n    # User query\n    s += f\"User: {user_query}\\n\"\n    s += \"Assistant: \"\n\n    # Generate with function calling\n    s += sgl.gen(\n        \"response\",\n        max_tokens=200,\n        tools=tools,  # SGLang handles tool call format\n        stop=[\"User:\", \"\\n\\n\"]\n    )\n\n# Multiple queries reuse system prompt\nstate1 = agent_workflow.run(\n    user_query=\"What's the weather in NYC?\",\n    tools=tools\n)\n# First call: Computes full system prompt\n\nstate2 = agent_workflow.run(\n    user_query=\"Book a flight to LA\",\n    tools=tools\n)\n# Second call: Reuses system prompt (5× faster)\n```\n\n## Performance benchmarks\n\n### RadixAttention speedup\n\n**Few-shot prompting** (10 examples in prompt):\n- vLLM: 2.5 sec\u002Frequest\n- SGLang: **0.25 sec\u002Frequest** (10× faster)\n- Throughput: 4× higher\n\n**Agent workflows** (1000-token system prompt):\n- vLLM: 1.8 sec\u002Frequest\n- SGLang: **0.35 sec\u002Frequest** (5× faster)\n\n**JSON decoding**:\n- Standard: 45 tok\u002Fs\n- SGLang: **135 tok\u002Fs** (3× faster)\n\n### Throughput (Llama 3-8B, A100)\n\n| Workload | vLLM | SGLang | Speedup |\n|----------|------|--------|---------|\n| Simple generation | 2500 tok\u002Fs | 2800 tok\u002Fs | 1.12× |\n| Few-shot (10 examples) | 500 tok\u002Fs | 5000 tok\u002Fs | 10× |\n| Agent (tool calls) | 800 tok\u002Fs | 4000 tok\u002Fs | 5× |\n| JSON output | 600 tok\u002Fs | 2400 tok\u002Fs | 4× |\n\n## Multi-turn conversations\n\n```python\n@sgl.function\ndef multi_turn_chat(s, history, new_message):\n    # System prompt (always cached)\n    s += \"You are a helpful AI assistant.\\n\\n\"\n\n    # Conversation history (cached as it grows)\n    for msg in history:\n        s += f\"{msg['role']}: {msg['content']}\\n\"\n\n    # New user message (only new part)\n    s += f\"User: {new_message}\\n\"\n    s += \"Assistant: \"\n    s += sgl.gen(\"response\", max_tokens=200)\n\n# Turn 1\nhistory = []\nstate = multi_turn_chat.run(history=history, new_message=\"Hi there!\")\nhistory.append({\"role\": \"User\", \"content\": \"Hi there!\"})\nhistory.append({\"role\": \"Assistant\", \"content\": state[\"response\"]})\n\n# Turn 2 (reuses Turn 1 KV cache)\nstate = multi_turn_chat.run(history=history, new_message=\"What's 2+2?\")\n# Only computes new message (much faster!)\n\n# Turn 3 (reuses Turn 1 + Turn 2 KV cache)\nstate = multi_turn_chat.run(history=history, new_message=\"Tell me a joke\")\n# Progressively faster as history grows\n```\n\n## Advanced features\n\n### Speculative decoding\n\n```bash\n# Launch with draft model (2-3× faster)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-70B-Instruct \\\n    --speculative-model meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --speculative-num-steps 5\n```\n\n### Multi-modal (vision models)\n\n```python\n@sgl.function\ndef describe_image(s, image_path):\n    s += sgl.image(image_path)\n    s += \"Describe this image in detail: \"\n    s += sgl.gen(\"description\", max_tokens=200)\n\nstate = describe_image.run(image_path=\"photo.jpg\")\nprint(state[\"description\"])\n```\n\n### Batching and parallel requests\n\n```python\n# Automatic batching (continuous batching)\nstates = sgl.run_batch(\n    [\n        simple_gen.bind(question=\"What is AI?\"),\n        simple_gen.bind(question=\"What is ML?\"),\n        simple_gen.bind(question=\"What is DL?\"),\n    ]\n)\n\n# All 3 processed in single batch (efficient)\n```\n\n## OpenAI-compatible API\n\n```bash\n# Start server with OpenAI API\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000\n\n# Use with OpenAI client\ncurl http:\u002F\u002Flocalhost:30000\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"model\": \"default\",\n    \"messages\": [\n      {\"role\": \"system\", \"content\": \"You are helpful\"},\n      {\"role\": \"user\", \"content\": \"Hello\"}\n    ],\n    \"temperature\": 0.7,\n    \"max_tokens\": 100\n  }'\n\n# Works with OpenAI Python SDK\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http:\u002F\u002Flocalhost:30000\u002Fv1\", api_key=\"EMPTY\")\n\nresponse = client.chat.completions.create(\n    model=\"default\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello\"}]\n)\n```\n\n## Supported models\n\n**Text models**:\n- Llama 2, Llama 3, Llama 3.1, Llama 3.2\n- Mistral, Mixtral\n- Qwen, Qwen2, QwQ\n- DeepSeek-V2, DeepSeek-V3\n- Gemma, Phi-3\n\n**Vision models**:\n- LLaVA, LLaVA-OneVision\n- Phi-3-Vision\n- Qwen2-VL\n\n**100+ models** from HuggingFace\n\n## Hardware support\n\n**NVIDIA**: A100, H100, L4, T4 (CUDA 11.8+)\n**AMD**: MI300, MI250 (ROCm 6.0+)\n**Intel**: Xeon with GPU (coming soon)\n**Apple**: M1\u002FM2\u002FM3 via MPS (experimental)\n\n## References\n\n- **[Structured Generation Guide](references\u002Fstructured-generation.md)** - JSON schemas, regex, grammars, validation\n- **[RadixAttention Deep Dive](references\u002Fradix-attention.md)** - How it works, optimization, benchmarks\n- **[Production Deployment](references\u002Fdeployment.md)** - Multi-GPU, monitoring, autoscaling\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang\n- **Docs**: https:\u002F\u002Fsgl-project.github.io\u002F\n- **Paper**: RadixAttention (arXiv:2312.07104)\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002Fsglang\n\n## Agent operating procedure\n\n1. **Check the environment.** Confirm hardware, framework and server versions, model format, and expected load.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Serve or log a single request or run end to end before scaling.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Measure latency, throughput and output correctness against a reference; check resource usage and costs.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| The server fails to start or OOMs | Check model size versus memory, quantization and parallelism settings. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Do not expose services or credentials publicly; confirm cloud costs before provisioning.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `tensorrt-llm`: Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency.\n- `outlines`: Generates guaranteed-valid structured output from LLMs with Outlines (dottxt.ai), constraining token sampling via finite state machines for…\n- `llama-cpp`: Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware.\n","# SGLang\n\nHigh-performance serving framework for LLMs and VLMs with RadixAttention for automatic prefix caching.\n\n## When to use SGLang\n\n**Use SGLang when:**\n- Need structured outputs (JSON, regex, grammar)\n- Building agents with repeated prefixes (system prompts, tools)\n- Agentic workflows with function calling\n- Multi-turn conversations with shared context\n- Need faster JSON decoding (3× vs standard)\n\n**Use vLLM instead when:**\n- Simple text generation without structure\n- Don't need prefix caching\n- Want mature, widely-tested production system\n\n**Use TensorRT-LLM instead when:**\n- Maximum single-request latency (no batching needed)\n- NVIDIA-only deployment\n- Need FP8\u002FINT4 quantization on H100\n\n## Quick start\n\n### Installation\n\n```bash\n# pip install (recommended)\npip install \"sglang[all]\"\n\n# With FlashInfer (faster, CUDA 11.8\u002F12.1)\npip install sglang[all] flashinfer -i https:\u002F\u002Fflashinfer.ai\u002Fwhl\u002Fcu121\u002Ftorch2.4\u002F\n\n# From source\ngit clone https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang.git\ncd sglang\npip install -e \"python[all]\"\n```\n\n### Launch server\n\n```bash\n# Basic server (Llama 3-8B)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000\n\n# With RadixAttention (automatic prefix caching)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000 \\\n    --enable-radix-cache  # Default: enabled\n\n# Multi-GPU (tensor parallelism)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-70B-Instruct \\\n    --tp 4 \\\n    --port 30000\n```\n\n### Basic inference\n\n```python\nimport sglang as sgl\n\n# Set backend\nsgl.set_default_backend(sgl.OpenAI(\"http:\u002F\u002Flocalhost:30000\u002Fv1\"))\n\n# Simple generation\n@sgl.function\ndef simple_gen(s, question):\n    s += \"Q: \" + question + \"\\n\"\n    s += \"A:\" + sgl.gen(\"answer\", max_tokens=100)\n\n# Run\nstate = simple_gen.run(question=\"What is the capital of France?\")\nprint(state[\"answer\"])\n# Output: \"The capital of France is Paris.\"\n```\n\n### Structured JSON output\n\n```python\nimport sglang as sgl\n\n@sgl.function\ndef extract_person(s, text):\n    s += f\"Extract person information from: {text}\\n\"\n    s += \"Output JSON:\\n\"\n\n    # Constrained JSON generation\n    s += sgl.gen(\n        \"json_output\",\n        max_tokens=200,\n        regex=r'\\{\"name\": \"[^\"]+\", \"age\": \\d+, \"occupation\": \"[^\"]+\"\\}'\n    )\n\n# Run\nstate = extract_person.run(\n    text=\"John Smith is a 35-year-old software engineer.\"\n)\nprint(state[\"json_output\"])\n# Output: {\"name\": \"John Smith\", \"age\": 35, \"occupation\": \"software engineer\"}\n```\n\n## RadixAttention (Key Innovation)\n\n**What it does**: Automatically caches and reuses common prefixes across requests.\n\n**Performance**:\n- **5× faster** for agentic workloads with shared system prompts\n- **10× faster** for few-shot prompting with repeated examples\n- **Zero configuration** - works automatically\n\n**How it works**:\n1. Builds radix tree of all processed tokens\n2. Automatically detects shared prefixes\n3. Reuses KV cache for matching prefixes\n4. Only computes new tokens\n\n**Example** (Agent with system prompt):\n\n```\nRequest 1: [SYSTEM_PROMPT] + \"What's the weather?\"\n→ Computes full prompt (1000 tokens)\n\nRequest 2: [SAME_SYSTEM_PROMPT] + \"Book a flight\"\n→ Reuses system prompt KV cache (998 tokens)\n→ Only computes 2 new tokens\n→ 5× faster!\n```\n\n## Structured generation patterns\n\n### JSON with schema\n\n```python\n@sgl.function\ndef structured_extraction(s, article):\n    s += f\"Article: {article}\\n\\n\"\n    s += \"Extract key information as JSON:\\n\"\n\n    # JSON schema constraint\n    schema = {\n        \"type\": \"object\",\n        \"properties\": {\n            \"title\": {\"type\": \"string\"},\n            \"author\": {\"type\": \"string\"},\n            \"summary\": {\"type\": \"string\"},\n            \"sentiment\": {\"type\": \"string\", \"enum\": [\"positive\", \"negative\", \"neutral\"]}\n        },\n        \"required\": [\"title\", \"author\", \"summary\", \"sentiment\"]\n    }\n\n    s += sgl.gen(\"info\", max_tokens=300, json_schema=schema)\n\nstate = structured_extraction.run(article=\"...\")\nprint(state[\"info\"])\n# Output: Valid JSON matching schema\n```\n\n### Regex-constrained generation\n\n```python\n@sgl.function\ndef extract_email(s, text):\n    s += f\"Extract email from: {text}\\n\"\n    s += \"Email: \"\n\n    # Email regex pattern\n    s += sgl.gen(\n        \"email\",\n        max_tokens=50,\n        regex=r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}'\n    )\n\nstate = extract_email.run(text=\"Contact john.doe@example.com for details\")\nprint(state[\"email\"])\n# Output: \"john.doe@example.com\"\n```\n\n### Grammar-based generation\n\n```python\n@sgl.function\ndef generate_code(s, description):\n    s += f\"Generate Python code for: {description}\\n\"\n    s += \"```python\\n\"\n\n    # EBNF grammar for Python\n    python_grammar = \"\"\"\n    ?start: function_def\n    function_def: \"def\" NAME \"(\" [parameters] \"):\" suite\n    parameters: parameter (\",\" parameter)*\n    parameter: NAME\n    suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT\n    \"\"\"\n\n    s += sgl.gen(\"code\", max_tokens=200, grammar=python_grammar)\n    s += \"\\n```\"\n```\n\n## Agent workflows with function calling\n\n```python\nimport sglang as sgl\n\n# Define tools\ntools = [\n    {\n        \"name\": \"get_weather\",\n        \"description\": \"Get weather for a location\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"location\": {\"type\": \"string\"}\n            }\n        }\n    },\n    {\n        \"name\": \"book_flight\",\n        \"description\": \"Book a flight\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"from\": {\"type\": \"string\"},\n                \"to\": {\"type\": \"string\"},\n                \"date\": {\"type\": \"string\"}\n            }\n        }\n    }\n]\n\n@sgl.function\ndef agent_workflow(s, user_query, tools):\n    # System prompt (cached with RadixAttention)\n    s += \"You are a helpful assistant with access to tools.\\n\"\n    s += f\"Available tools: {tools}\\n\\n\"\n\n    # User query\n    s += f\"User: {user_query}\\n\"\n    s += \"Assistant: \"\n\n    # Generate with function calling\n    s += sgl.gen(\n        \"response\",\n        max_tokens=200,\n        tools=tools,  # SGLang handles tool call format\n        stop=[\"User:\", \"\\n\\n\"]\n    )\n\n# Multiple queries reuse system prompt\nstate1 = agent_workflow.run(\n    user_query=\"What's the weather in NYC?\",\n    tools=tools\n)\n# First call: Computes full system prompt\n\nstate2 = agent_workflow.run(\n    user_query=\"Book a flight to LA\",\n    tools=tools\n)\n# Second call: Reuses system prompt (5× faster)\n```\n\n## Performance benchmarks\n\n### RadixAttention speedup\n\n**Few-shot prompting** (10 examples in prompt):\n- vLLM: 2.5 sec\u002Frequest\n- SGLang: **0.25 sec\u002Frequest** (10× faster)\n- Throughput: 4× higher\n\n**Agent workflows** (1000-token system prompt):\n- vLLM: 1.8 sec\u002Frequest\n- SGLang: **0.35 sec\u002Frequest** (5× faster)\n\n**JSON decoding**:\n- Standard: 45 tok\u002Fs\n- SGLang: **135 tok\u002Fs** (3× faster)\n\n### Throughput (Llama 3-8B, A100)\n\n| Workload | vLLM | SGLang | Speedup |\n|----------|------|--------|---------|\n| Simple generation | 2500 tok\u002Fs | 2800 tok\u002Fs | 1.12× |\n| Few-shot (10 examples) | 500 tok\u002Fs | 5000 tok\u002Fs | 10× |\n| Agent (tool calls) | 800 tok\u002Fs | 4000 tok\u002Fs | 5× |\n| JSON output | 600 tok\u002Fs | 2400 tok\u002Fs | 4× |\n\n## Multi-turn conversations\n\n```python\n@sgl.function\ndef multi_turn_chat(s, history, new_message):\n    # System prompt (always cached)\n    s += \"You are a helpful AI assistant.\\n\\n\"\n\n    # Conversation history (cached as it grows)\n    for msg in history:\n        s += f\"{msg['role']}: {msg['content']}\\n\"\n\n    # New user message (only new part)\n    s += f\"User: {new_message}\\n\"\n    s += \"Assistant: \"\n    s += sgl.gen(\"response\", max_tokens=200)\n\n# Turn 1\nhistory = []\nstate = multi_turn_chat.run(history=history, new_message=\"Hi there!\")\nhistory.append({\"role\": \"User\", \"content\": \"Hi there!\"})\nhistory.append({\"role\": \"Assistant\", \"content\": state[\"response\"]})\n\n# Turn 2 (reuses Turn 1 KV cache)\nstate = multi_turn_chat.run(history=history, new_message=\"What's 2+2?\")\n# Only computes new message (much faster!)\n\n# Turn 3 (reuses Turn 1 + Turn 2 KV cache)\nstate = multi_turn_chat.run(history=history, new_message=\"Tell me a joke\")\n# Progressively faster as history grows\n```\n\n## Advanced features\n\n### Speculative decoding\n\n```bash\n# Launch with draft model (2-3× faster)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-70B-Instruct \\\n    --speculative-model meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --speculative-num-steps 5\n```\n\n### Multi-modal (vision models)\n\n```python\n@sgl.function\ndef describe_image(s, image_path):\n    s += sgl.image(image_path)\n    s += \"Describe this image in detail: \"\n    s += sgl.gen(\"description\", max_tokens=200)\n\nstate = describe_image.run(image_path=\"photo.jpg\")\nprint(state[\"description\"])\n```\n\n### Batching and parallel requests\n\n```python\n# Automatic batching (continuous batching)\nstates = sgl.run_batch(\n    [\n        simple_gen.bind(question=\"What is AI?\"),\n        simple_gen.bind(question=\"What is ML?\"),\n        simple_gen.bind(question=\"What is DL?\"),\n    ]\n)\n\n# All 3 processed in single batch (efficient)\n```\n\n## OpenAI-compatible API\n\n```bash\n# Start server with OpenAI API\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000\n\n# Use with OpenAI client\ncurl http:\u002F\u002Flocalhost:30000\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"model\": \"default\",\n    \"messages\": [\n      {\"role\": \"system\", \"content\": \"You are helpful\"},\n      {\"role\": \"user\", \"content\": \"Hello\"}\n    ],\n    \"temperature\": 0.7,\n    \"max_tokens\": 100\n  }'\n\n# Works with OpenAI Python SDK\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http:\u002F\u002Flocalhost:30000\u002Fv1\", api_key=\"EMPTY\")\n\nresponse = client.chat.completions.create(\n    model=\"default\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello\"}]\n)\n```\n\n## Supported models\n\n**Text models**:\n- Llama 2, Llama 3, Llama 3.1, Llama 3.2\n- Mistral, Mixtral\n- Qwen, Qwen2, QwQ\n- DeepSeek-V2, DeepSeek-V3\n- Gemma, Phi-3\n\n**Vision models**:\n- LLaVA, LLaVA-OneVision\n- Phi-3-Vision\n- Qwen2-VL\n\n**100+ models** from HuggingFace\n\n## Hardware support\n\n**NVIDIA**: A100, H100, L4, T4 (CUDA 11.8+)\n**AMD**: MI300, MI250 (ROCm 6.0+)\n**Intel**: Xeon with GPU (coming soon)\n**Apple**: M1\u002FM2\u002FM3 via MPS (experimental)\n\n## References\n\n- **[Structured Generation Guide](references\u002Fstructured-generation.md)** - JSON schemas, regex, grammars, validation\n- **[RadixAttention Deep Dive](references\u002Fradix-attention.md)** - How it works, optimization, benchmarks\n- **[Production Deployment](references\u002Fdeployment.md)** - Multi-GPU, monitoring, autoscaling\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang\n- **Docs**: https:\u002F\u002Fsgl-project.github.io\u002F\n- **Paper**: RadixAttention (arXiv:2312.07104)\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002Fsglang\n\n## Agent operating procedure\n\n1. **Check the environment.** Confirm hardware, framework and server versions, model format, and expected load.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Serve or log a single request or run end to end before scaling.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Measure latency, throughput and output correctness against a reference; check resource usage and costs.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| The server fails to start or OOMs | Check model size versus memory, quantization and parallelism settings. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Do not expose services or credentials publicly; confirm cloud costs before provisioning.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `tensorrt-llm`: Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency.\n- `outlines`: Generates guaranteed-valid structured output from LLMs with Outlines (dottxt.ai), constraining token sampling via finite state machines for…\n- `llama-cpp`: Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware.\n",{"name":5,"license":15,"metadata":46,"description":7},{"tags":47,"version":48,"category":49,"maintainer":50,"dependencies":51},"Inference Serving, SGLang, Structured Generation, RadixAttention, Prefix Caching, Constrained Decoding, Agents, JSON Output, Fast Inference, Production Scale","1.0.0","ml-inference-and-ops","Kalaris Labs","sglang, torch, transformers","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fsglang",468,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-11 02:22:54",{"source":8,"skillName":5,"commands":63,"prompt":73,"zipAvailable":74},[64,67,70],{"tool":65,"command":66},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill sglang",{"tool":68,"command":69},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill sglang",{"tool":71,"command":72},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill sglang","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fsglang\u002FSKILL.md",true,[76,82,86],{"path":77,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Fdeployment.md","REFERENCE","参考文档",10020,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fsglang\u002Freferences\u002Fdeployment.md",{"path":83,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":84,"sourceHtmlUrl":85},"references\u002Fradix-attention.md",11005,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fsglang\u002Freferences\u002Fradix-attention.md",{"path":87,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Fstructured-generation.md",13468,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fsglang\u002Freferences\u002Fstructured-generation.md",[91,117,142,176,195,209],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":41},"01M716A6ETNB9B1SQ3GXZC75VR","deja-search","deja-vu memory — search the user's past AI coding sessions with the deja CLI. Holds what they already settled and never wrote into the repo (naming rules, banned or already-tried tools, required checks before merge, preferred values, who owns what). Use before your first edit in a task, before you call a change done or ready to merge, before you recommend a tool, library, design or reviewer, before re-debugging an error that may already be solved, when they say 'didn't we fix this before', 'what did we decide about X' or 'I already have that', and before telling them something on this machine does not exist.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fvshulcz\u002Fdeja-vu\u002Fskills\u002Fdeja-search\u002FSKILL.md",[],[99,100,101,102,103,104,105,106,107,108,109,21,27,110,111,112,113,114,115,116],"search","deja search","error","ready","never","memory","re","decide","fix","design","debugging","done","reviewer","io","change","cli","merge","past",{"skillNo":118,"name":119,"nameZh":6,"description":120,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":121,"categories":122,"keywords":123,"publishedAt":41},"01M716A3J2W5SENSE9VGMXB375","deja-reflect","End-of-session reflection and memory curation with deja-vu. Review what was decided, what commands broke and what repaired them, and record durable project rules or decisions with deja remember so the next session starts with them.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fvshulcz\u002Fdeja-vu\u002Fskills\u002Fdeja-reflect\u002FSKILL.md",[],[124,125,126,127,104,128,129,130,131,132,133,134,135,112,136,137,138,139,140,141],"reflect","deja reflect","brew","remember","project rules","project","session","end","reflection","Rules","decisions","review","with","record","github","next","https","node",{"skillNo":143,"name":144,"nameZh":6,"description":145,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":146,"categories":147,"keywords":159,"publishedAt":41},"01M715T3RR4BAYSR57HPPTJ0PF","revid","Make, edit, export and post AI videos with Revid (revid.ai) through the Revid MCP server. Use whenever the user wants a short-form or long-form video made (from an idea, a script, a URL, an article, a PDF, a tweet, a product page, their own clips, or a song), wants to find what is going viral on TikTok, YouTube, Instagram or X before making something, wants to edit or re-export an existing Revid project, generate a standalone AI image or clip, make a series episode, or publish and schedule a finished video to connected social accounts. Covers setup (hosted MCP with OAuth, or an API key), the 36 tools, credits, polling and the rules for publishing. Routes to the more specific revid-* skills when one fits.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftmakersas\u002Frevid-skills\u002Fskills\u002Frevid\u002FSKILL.md",[148,153,156],{"categoryNo":149,"categoryName":150,"taxonomyType":151,"taxonomyTypeDesc":152},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",{"categoryNo":154,"categoryName":155,"taxonomyType":151,"taxonomyTypeDesc":152},"01JBSEEDCAT000000000000016","图像与多媒体",{"categoryNo":157,"categoryName":158,"taxonomyType":151,"taxonomyTypeDesc":152},"01JBSEEDCAT000000000000023","Skill与智能体开发",[144,160,161,162,163,164,129,165,105,166,167,168,169,21,170,171,172,173,174,175],"Video","short-form","standalone","PDF","viral","tiktok","short","social","MCP","song","URL","image","mcp server","schedule","api","publish",{"skillNo":177,"name":178,"nameZh":6,"description":179,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":180,"categories":181,"keywords":182,"publishedAt":41},"01M71569ZM2X3H5RBFK8PVKMK2","review-study","Review a QuestionPunk survey before launch for wording, answer choices, branching, and missing settings. Use for check my survey or review this draft before publishing.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fquestionpunk\u002Fcodex-plugin\u002Fskills\u002Freview-study\u002FSKILL.md",[],[183,184,135,25,185,186,187,188,20,189,190,191,192,193,194],"study","review study","answer","missing","survey","publishing","check","draft","settings","launch","my","branching",{"skillNo":196,"name":197,"nameZh":6,"description":198,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":199,"categories":200,"keywords":201,"publishedAt":41},"01M71567GV0C35YQBY0DPGR86B","manage-study","Find QuestionPunk studies, check status, publish an explicitly requested draft, or retrieve its participant link. Use for find my survey, publish this study, or how many people responded.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fquestionpunk\u002Fcodex-plugin\u002Fskills\u002Fmanage-study\u002FSKILL.md",[],[202,183,203,25,187,20,204,189,190,175,205,206,207,208,193],"manage study","manage","people","how","find","link","status",{"skillNo":210,"name":211,"nameZh":6,"description":212,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":6,"sourceHtmlUrl":213,"categories":214,"keywords":215,"publishedAt":41},"01M71565A8TCA01AZQZRGRYMFZ","create-study","Create a saved QuestionPunk survey draft from a research brief or edit specified questions. Use for requests such as make a customer feedback survey or shorten my onboarding study.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fquestionpunk\u002Fcodex-plugin\u002Fskills\u002Fcreate-study\u002FSKILL.md",[],[216,183,217,218,219,220,25,221,187,222,20,223,224,190,225,226,227,228,229,193],"create study","create","brief","as","feedback","customer feedback","research brief","shorten","saved","research","requests","customer","questions","edit"]