[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6FC4F2CT2W5REKTXPESPRNQ":3,"skill-same:01M6FC4F2CT2W5REKTXPESPRNQ":90},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":41,"linkOnly":42,"markdownContent":43,"markdownBody":44,"frontMatter":45,"refName":53,"commitSha":54,"skillDirectory":55,"forksCount":56,"repositoryHtmlUrl":57,"repositoryHomepageUrl":6,"ownerHtmlUrl":58,"ownerBlogUrl":59,"ownerDescription":60,"updatedAt":61,"install":62,"resources":75},"01M6FC4F2CT2W5REKTXPESPRNQ","sglang",null,"Fast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON\u002Fregex outputs, constrained decoding, agentic workflows with tool calls, or when you need 5× faster inference than vLLM with prefix sharing. Powers 300,000+ GPUs at xAI, AMD, NVIDIA, and LinkedIn.","Dicklesworthstone\u002Fpi_agent_rust","Dicklesworthstone","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F35050222?v=4","01M6CG26AR942CSVMN1M5E3DSE","Jeff Emanuel","01M6CG26EPP71PSHW1VDS71121",1979,"MIT","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Finference-serving-sglang\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40],"SGLang","JSON Output","when","at","you","regex","caching","xai","tool","Transformers","Production Scale","Fast Inference","agentic","Constrained Decoding","Prefix Caching","RadixAttention","Structured Generation","Inference Serving","NVIDIA","agentic workflows","workflows","llms","2026-11-10 10:12:30",false,"---\nname: sglang\ndescription: Fast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON\u002Fregex outputs, constrained decoding, agentic workflows with tool calls, or when you need 5× faster inference than vLLM with prefix sharing. Powers 300,000+ GPUs at xAI, AMD, NVIDIA, and LinkedIn.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Inference Serving, SGLang, Structured Generation, RadixAttention, Prefix Caching, Constrained Decoding, Agents, JSON Output, Fast Inference, Production Scale]\ndependencies: [sglang, torch, transformers]\n---\n\n# SGLang\n\nHigh-performance serving framework for LLMs and VLMs with RadixAttention for automatic prefix caching.\n\n## When to use SGLang\n\n**Use SGLang when:**\n- Need structured outputs (JSON, regex, grammar)\n- Building agents with repeated prefixes (system prompts, tools)\n- Agentic workflows with function calling\n- Multi-turn conversations with shared context\n- Need faster JSON decoding (3× vs standard)\n\n**Use vLLM instead when:**\n- Simple text generation without structure\n- Don't need prefix caching\n- Want mature, widely-tested production system\n\n**Use TensorRT-LLM instead when:**\n- Maximum single-request latency (no batching needed)\n- NVIDIA-only deployment\n- Need FP8\u002FINT4 quantization on H100\n\n## Quick start\n\n### Installation\n\n```bash\n# pip install (recommended)\npip install \"sglang[all]\"\n\n# With FlashInfer (faster, CUDA 11.8\u002F12.1)\npip install sglang[all] flashinfer -i https:\u002F\u002Fflashinfer.ai\u002Fwhl\u002Fcu121\u002Ftorch2.4\u002F\n\n# From source\ngit clone https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang.git\ncd sglang\npip install -e \"python[all]\"\n```\n\n### Launch server\n\n```bash\n# Basic server (Llama 3-8B)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000\n\n# With RadixAttention (automatic prefix caching)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000 \\\n    --enable-radix-cache  # Default: enabled\n\n# Multi-GPU (tensor parallelism)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-70B-Instruct \\\n    --tp 4 \\\n    --port 30000\n```\n\n### Basic inference\n\n```python\nimport sglang as sgl\n\n# Set backend\nsgl.set_default_backend(sgl.OpenAI(\"http:\u002F\u002Flocalhost:30000\u002Fv1\"))\n\n# Simple generation\n@sgl.function\ndef simple_gen(s, question):\n    s += \"Q: \" + question + \"\\n\"\n    s += \"A:\" + sgl.gen(\"answer\", max_tokens=100)\n\n# Run\nstate = simple_gen.run(question=\"What is the capital of France?\")\nprint(state[\"answer\"])\n# Output: \"The capital of France is Paris.\"\n```\n\n### Structured JSON output\n\n```python\nimport sglang as sgl\n\n@sgl.function\ndef extract_person(s, text):\n    s += f\"Extract person information from: {text}\\n\"\n    s += \"Output JSON:\\n\"\n\n    # Constrained JSON generation\n    s += sgl.gen(\n        \"json_output\",\n        max_tokens=200,\n        regex=r'\\{\"name\": \"[^\"]+\", \"age\": \\d+, \"occupation\": \"[^\"]+\"\\}'\n    )\n\n# Run\nstate = extract_person.run(\n    text=\"John Smith is a 35-year-old software engineer.\"\n)\nprint(state[\"json_output\"])\n# Output: {\"name\": \"John Smith\", \"age\": 35, \"occupation\": \"software engineer\"}\n```\n\n## RadixAttention (Key Innovation)\n\n**What it does**: Automatically caches and reuses common prefixes across requests.\n\n**Performance**:\n- **5× faster** for agentic workloads with shared system prompts\n- **10× faster** for few-shot prompting with repeated examples\n- **Zero configuration** - works automatically\n\n**How it works**:\n1. Builds radix tree of all processed tokens\n2. Automatically detects shared prefixes\n3. Reuses KV cache for matching prefixes\n4. Only computes new tokens\n\n**Example** (Agent with system prompt):\n\n```\nRequest 1: [SYSTEM_PROMPT] + \"What's the weather?\"\n→ Computes full prompt (1000 tokens)\n\nRequest 2: [SAME_SYSTEM_PROMPT] + \"Book a flight\"\n→ Reuses system prompt KV cache (998 tokens)\n→ Only computes 2 new tokens\n→ 5× faster!\n```\n\n## Structured generation patterns\n\n### JSON with schema\n\n```python\n@sgl.function\ndef structured_extraction(s, article):\n    s += f\"Article: {article}\\n\\n\"\n    s += \"Extract key information as JSON:\\n\"\n\n    # JSON schema constraint\n    schema = {\n        \"type\": \"object\",\n        \"properties\": {\n            \"title\": {\"type\": \"string\"},\n            \"author\": {\"type\": \"string\"},\n            \"summary\": {\"type\": \"string\"},\n            \"sentiment\": {\"type\": \"string\", \"enum\": [\"positive\", \"negative\", \"neutral\"]}\n        },\n        \"required\": [\"title\", \"author\", \"summary\", \"sentiment\"]\n    }\n\n    s += sgl.gen(\"info\", max_tokens=300, json_schema=schema)\n\nstate = structured_extraction.run(article=\"...\")\nprint(state[\"info\"])\n# Output: Valid JSON matching schema\n```\n\n### Regex-constrained generation\n\n```python\n@sgl.function\ndef extract_email(s, text):\n    s += f\"Extract email from: {text}\\n\"\n    s += \"Email: \"\n\n    # Email regex pattern\n    s += sgl.gen(\n        \"email\",\n        max_tokens=50,\n        regex=r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}'\n    )\n\nstate = extract_email.run(text=\"Contact john.doe@example.com for details\")\nprint(state[\"email\"])\n# Output: \"john.doe@example.com\"\n```\n\n### Grammar-based generation\n\n```python\n@sgl.function\ndef generate_code(s, description):\n    s += f\"Generate Python code for: {description}\\n\"\n    s += \"```python\\n\"\n\n    # EBNF grammar for Python\n    python_grammar = \"\"\"\n    ?start: function_def\n    function_def: \"def\" NAME \"(\" [parameters] \"):\" suite\n    parameters: parameter (\",\" parameter)*\n    parameter: NAME\n    suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT\n    \"\"\"\n\n    s += sgl.gen(\"code\", max_tokens=200, grammar=python_grammar)\n    s += \"\\n```\"\n```\n\n## Agent workflows with function calling\n\n```python\nimport sglang as sgl\n\n# Define tools\ntools = [\n    {\n        \"name\": \"get_weather\",\n        \"description\": \"Get weather for a location\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"location\": {\"type\": \"string\"}\n            }\n        }\n    },\n    {\n        \"name\": \"book_flight\",\n        \"description\": \"Book a flight\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"from\": {\"type\": \"string\"},\n                \"to\": {\"type\": \"string\"},\n                \"date\": {\"type\": \"string\"}\n            }\n        }\n    }\n]\n\n@sgl.function\ndef agent_workflow(s, user_query, tools):\n    # System prompt (cached with RadixAttention)\n    s += \"You are a helpful assistant with access to tools.\\n\"\n    s += f\"Available tools: {tools}\\n\\n\"\n\n    # User query\n    s += f\"User: {user_query}\\n\"\n    s += \"Assistant: \"\n\n    # Generate with function calling\n    s += sgl.gen(\n        \"response\",\n        max_tokens=200,\n        tools=tools,  # SGLang handles tool call format\n        stop=[\"User:\", \"\\n\\n\"]\n    )\n\n# Multiple queries reuse system prompt\nstate1 = agent_workflow.run(\n    user_query=\"What's the weather in NYC?\",\n    tools=tools\n)\n# First call: Computes full system prompt\n\nstate2 = agent_workflow.run(\n    user_query=\"Book a flight to LA\",\n    tools=tools\n)\n# Second call: Reuses system prompt (5× faster)\n```\n\n## Performance benchmarks\n\n### RadixAttention speedup\n\n**Few-shot prompting** (10 examples in prompt):\n- vLLM: 2.5 sec\u002Frequest\n- SGLang: **0.25 sec\u002Frequest** (10× faster)\n- Throughput: 4× higher\n\n**Agent workflows** (1000-token system prompt):\n- vLLM: 1.8 sec\u002Frequest\n- SGLang: **0.35 sec\u002Frequest** (5× faster)\n\n**JSON decoding**:\n- Standard: 45 tok\u002Fs\n- SGLang: **135 tok\u002Fs** (3× faster)\n\n### Throughput (Llama 3-8B, A100)\n\n| Workload | vLLM | SGLang | Speedup |\n|----------|------|--------|---------|\n| Simple generation | 2500 tok\u002Fs | 2800 tok\u002Fs | 1.12× |\n| Few-shot (10 examples) | 500 tok\u002Fs | 5000 tok\u002Fs | 10× |\n| Agent (tool calls) | 800 tok\u002Fs | 4000 tok\u002Fs | 5× |\n| JSON output | 600 tok\u002Fs | 2400 tok\u002Fs | 4× |\n\n## Multi-turn conversations\n\n```python\n@sgl.function\ndef multi_turn_chat(s, history, new_message):\n    # System prompt (always cached)\n    s += \"You are a helpful AI assistant.\\n\\n\"\n\n    # Conversation history (cached as it grows)\n    for msg in history:\n        s += f\"{msg['role']}: {msg['content']}\\n\"\n\n    # New user message (only new part)\n    s += f\"User: {new_message}\\n\"\n    s += \"Assistant: \"\n    s += sgl.gen(\"response\", max_tokens=200)\n\n# Turn 1\nhistory = []\nstate = multi_turn_chat.run(history=history, new_message=\"Hi there!\")\nhistory.append({\"role\": \"User\", \"content\": \"Hi there!\"})\nhistory.append({\"role\": \"Assistant\", \"content\": state[\"response\"]})\n\n# Turn 2 (reuses Turn 1 KV cache)\nstate = multi_turn_chat.run(history=history, new_message=\"What's 2+2?\")\n# Only computes new message (much faster!)\n\n# Turn 3 (reuses Turn 1 + Turn 2 KV cache)\nstate = multi_turn_chat.run(history=history, new_message=\"Tell me a joke\")\n# Progressively faster as history grows\n```\n\n## Advanced features\n\n### Speculative decoding\n\n```bash\n# Launch with draft model (2-3× faster)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-70B-Instruct \\\n    --speculative-model meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --speculative-num-steps 5\n```\n\n### Multi-modal (vision models)\n\n```python\n@sgl.function\ndef describe_image(s, image_path):\n    s += sgl.image(image_path)\n    s += \"Describe this image in detail: \"\n    s += sgl.gen(\"description\", max_tokens=200)\n\nstate = describe_image.run(image_path=\"photo.jpg\")\nprint(state[\"description\"])\n```\n\n### Batching and parallel requests\n\n```python\n# Automatic batching (continuous batching)\nstates = sgl.run_batch(\n    [\n        simple_gen.bind(question=\"What is AI?\"),\n        simple_gen.bind(question=\"What is ML?\"),\n        simple_gen.bind(question=\"What is DL?\"),\n    ]\n)\n\n# All 3 processed in single batch (efficient)\n```\n\n## OpenAI-compatible API\n\n```bash\n# Start server with OpenAI API\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000\n\n# Use with OpenAI client\ncurl http:\u002F\u002Flocalhost:30000\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"model\": \"default\",\n    \"messages\": [\n      {\"role\": \"system\", \"content\": \"You are helpful\"},\n      {\"role\": \"user\", \"content\": \"Hello\"}\n    ],\n    \"temperature\": 0.7,\n    \"max_tokens\": 100\n  }'\n\n# Works with OpenAI Python SDK\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http:\u002F\u002Flocalhost:30000\u002Fv1\", api_key=\"EMPTY\")\n\nresponse = client.chat.completions.create(\n    model=\"default\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello\"}]\n)\n```\n\n## Supported models\n\n**Text models**:\n- Llama 2, Llama 3, Llama 3.1, Llama 3.2\n- Mistral, Mixtral\n- Qwen, Qwen2, QwQ\n- DeepSeek-V2, DeepSeek-V3\n- Gemma, Phi-3\n\n**Vision models**:\n- LLaVA, LLaVA-OneVision\n- Phi-3-Vision\n- Qwen2-VL\n\n**100+ models** from HuggingFace\n\n## Hardware support\n\n**NVIDIA**: A100, H100, L4, T4 (CUDA 11.8+)\n**AMD**: MI300, MI250 (ROCm 6.0+)\n**Intel**: Xeon with GPU (coming soon)\n**Apple**: M1\u002FM2\u002FM3 via MPS (experimental)\n\n## References\n\n- **[Structured Generation Guide](references\u002Fstructured-generation.md)** - JSON schemas, regex, grammars, validation\n- **[RadixAttention Deep Dive](references\u002Fradix-attention.md)** - How it works, optimization, benchmarks\n- **[Production Deployment](references\u002Fdeployment.md)** - Multi-GPU, monitoring, autoscaling\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang\n- **Docs**: https:\u002F\u002Fsgl-project.github.io\u002F\n- **Paper**: RadixAttention (arXiv:2312.07104)\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002Fsglang\n\n\n","# SGLang\n\nHigh-performance serving framework for LLMs and VLMs with RadixAttention for automatic prefix caching.\n\n## When to use SGLang\n\n**Use SGLang when:**\n- Need structured outputs (JSON, regex, grammar)\n- Building agents with repeated prefixes (system prompts, tools)\n- Agentic workflows with function calling\n- Multi-turn conversations with shared context\n- Need faster JSON decoding (3× vs standard)\n\n**Use vLLM instead when:**\n- Simple text generation without structure\n- Don't need prefix caching\n- Want mature, widely-tested production system\n\n**Use TensorRT-LLM instead when:**\n- Maximum single-request latency (no batching needed)\n- NVIDIA-only deployment\n- Need FP8\u002FINT4 quantization on H100\n\n## Quick start\n\n### Installation\n\n```bash\n# pip install (recommended)\npip install \"sglang[all]\"\n\n# With FlashInfer (faster, CUDA 11.8\u002F12.1)\npip install sglang[all] flashinfer -i https:\u002F\u002Fflashinfer.ai\u002Fwhl\u002Fcu121\u002Ftorch2.4\u002F\n\n# From source\ngit clone https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang.git\ncd sglang\npip install -e \"python[all]\"\n```\n\n### Launch server\n\n```bash\n# Basic server (Llama 3-8B)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000\n\n# With RadixAttention (automatic prefix caching)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000 \\\n    --enable-radix-cache  # Default: enabled\n\n# Multi-GPU (tensor parallelism)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-70B-Instruct \\\n    --tp 4 \\\n    --port 30000\n```\n\n### Basic inference\n\n```python\nimport sglang as sgl\n\n# Set backend\nsgl.set_default_backend(sgl.OpenAI(\"http:\u002F\u002Flocalhost:30000\u002Fv1\"))\n\n# Simple generation\n@sgl.function\ndef simple_gen(s, question):\n    s += \"Q: \" + question + \"\\n\"\n    s += \"A:\" + sgl.gen(\"answer\", max_tokens=100)\n\n# Run\nstate = simple_gen.run(question=\"What is the capital of France?\")\nprint(state[\"answer\"])\n# Output: \"The capital of France is Paris.\"\n```\n\n### Structured JSON output\n\n```python\nimport sglang as sgl\n\n@sgl.function\ndef extract_person(s, text):\n    s += f\"Extract person information from: {text}\\n\"\n    s += \"Output JSON:\\n\"\n\n    # Constrained JSON generation\n    s += sgl.gen(\n        \"json_output\",\n        max_tokens=200,\n        regex=r'\\{\"name\": \"[^\"]+\", \"age\": \\d+, \"occupation\": \"[^\"]+\"\\}'\n    )\n\n# Run\nstate = extract_person.run(\n    text=\"John Smith is a 35-year-old software engineer.\"\n)\nprint(state[\"json_output\"])\n# Output: {\"name\": \"John Smith\", \"age\": 35, \"occupation\": \"software engineer\"}\n```\n\n## RadixAttention (Key Innovation)\n\n**What it does**: Automatically caches and reuses common prefixes across requests.\n\n**Performance**:\n- **5× faster** for agentic workloads with shared system prompts\n- **10× faster** for few-shot prompting with repeated examples\n- **Zero configuration** - works automatically\n\n**How it works**:\n1. Builds radix tree of all processed tokens\n2. Automatically detects shared prefixes\n3. Reuses KV cache for matching prefixes\n4. Only computes new tokens\n\n**Example** (Agent with system prompt):\n\n```\nRequest 1: [SYSTEM_PROMPT] + \"What's the weather?\"\n→ Computes full prompt (1000 tokens)\n\nRequest 2: [SAME_SYSTEM_PROMPT] + \"Book a flight\"\n→ Reuses system prompt KV cache (998 tokens)\n→ Only computes 2 new tokens\n→ 5× faster!\n```\n\n## Structured generation patterns\n\n### JSON with schema\n\n```python\n@sgl.function\ndef structured_extraction(s, article):\n    s += f\"Article: {article}\\n\\n\"\n    s += \"Extract key information as JSON:\\n\"\n\n    # JSON schema constraint\n    schema = {\n        \"type\": \"object\",\n        \"properties\": {\n            \"title\": {\"type\": \"string\"},\n            \"author\": {\"type\": \"string\"},\n            \"summary\": {\"type\": \"string\"},\n            \"sentiment\": {\"type\": \"string\", \"enum\": [\"positive\", \"negative\", \"neutral\"]}\n        },\n        \"required\": [\"title\", \"author\", \"summary\", \"sentiment\"]\n    }\n\n    s += sgl.gen(\"info\", max_tokens=300, json_schema=schema)\n\nstate = structured_extraction.run(article=\"...\")\nprint(state[\"info\"])\n# Output: Valid JSON matching schema\n```\n\n### Regex-constrained generation\n\n```python\n@sgl.function\ndef extract_email(s, text):\n    s += f\"Extract email from: {text}\\n\"\n    s += \"Email: \"\n\n    # Email regex pattern\n    s += sgl.gen(\n        \"email\",\n        max_tokens=50,\n        regex=r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}'\n    )\n\nstate = extract_email.run(text=\"Contact john.doe@example.com for details\")\nprint(state[\"email\"])\n# Output: \"john.doe@example.com\"\n```\n\n### Grammar-based generation\n\n```python\n@sgl.function\ndef generate_code(s, description):\n    s += f\"Generate Python code for: {description}\\n\"\n    s += \"```python\\n\"\n\n    # EBNF grammar for Python\n    python_grammar = \"\"\"\n    ?start: function_def\n    function_def: \"def\" NAME \"(\" [parameters] \"):\" suite\n    parameters: parameter (\",\" parameter)*\n    parameter: NAME\n    suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT\n    \"\"\"\n\n    s += sgl.gen(\"code\", max_tokens=200, grammar=python_grammar)\n    s += \"\\n```\"\n```\n\n## Agent workflows with function calling\n\n```python\nimport sglang as sgl\n\n# Define tools\ntools = [\n    {\n        \"name\": \"get_weather\",\n        \"description\": \"Get weather for a location\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"location\": {\"type\": \"string\"}\n            }\n        }\n    },\n    {\n        \"name\": \"book_flight\",\n        \"description\": \"Book a flight\",\n        \"parameters\": {\n            \"type\": \"object\",\n            \"properties\": {\n                \"from\": {\"type\": \"string\"},\n                \"to\": {\"type\": \"string\"},\n                \"date\": {\"type\": \"string\"}\n            }\n        }\n    }\n]\n\n@sgl.function\ndef agent_workflow(s, user_query, tools):\n    # System prompt (cached with RadixAttention)\n    s += \"You are a helpful assistant with access to tools.\\n\"\n    s += f\"Available tools: {tools}\\n\\n\"\n\n    # User query\n    s += f\"User: {user_query}\\n\"\n    s += \"Assistant: \"\n\n    # Generate with function calling\n    s += sgl.gen(\n        \"response\",\n        max_tokens=200,\n        tools=tools,  # SGLang handles tool call format\n        stop=[\"User:\", \"\\n\\n\"]\n    )\n\n# Multiple queries reuse system prompt\nstate1 = agent_workflow.run(\n    user_query=\"What's the weather in NYC?\",\n    tools=tools\n)\n# First call: Computes full system prompt\n\nstate2 = agent_workflow.run(\n    user_query=\"Book a flight to LA\",\n    tools=tools\n)\n# Second call: Reuses system prompt (5× faster)\n```\n\n## Performance benchmarks\n\n### RadixAttention speedup\n\n**Few-shot prompting** (10 examples in prompt):\n- vLLM: 2.5 sec\u002Frequest\n- SGLang: **0.25 sec\u002Frequest** (10× faster)\n- Throughput: 4× higher\n\n**Agent workflows** (1000-token system prompt):\n- vLLM: 1.8 sec\u002Frequest\n- SGLang: **0.35 sec\u002Frequest** (5× faster)\n\n**JSON decoding**:\n- Standard: 45 tok\u002Fs\n- SGLang: **135 tok\u002Fs** (3× faster)\n\n### Throughput (Llama 3-8B, A100)\n\n| Workload | vLLM | SGLang | Speedup |\n|----------|------|--------|---------|\n| Simple generation | 2500 tok\u002Fs | 2800 tok\u002Fs | 1.12× |\n| Few-shot (10 examples) | 500 tok\u002Fs | 5000 tok\u002Fs | 10× |\n| Agent (tool calls) | 800 tok\u002Fs | 4000 tok\u002Fs | 5× |\n| JSON output | 600 tok\u002Fs | 2400 tok\u002Fs | 4× |\n\n## Multi-turn conversations\n\n```python\n@sgl.function\ndef multi_turn_chat(s, history, new_message):\n    # System prompt (always cached)\n    s += \"You are a helpful AI assistant.\\n\\n\"\n\n    # Conversation history (cached as it grows)\n    for msg in history:\n        s += f\"{msg['role']}: {msg['content']}\\n\"\n\n    # New user message (only new part)\n    s += f\"User: {new_message}\\n\"\n    s += \"Assistant: \"\n    s += sgl.gen(\"response\", max_tokens=200)\n\n# Turn 1\nhistory = []\nstate = multi_turn_chat.run(history=history, new_message=\"Hi there!\")\nhistory.append({\"role\": \"User\", \"content\": \"Hi there!\"})\nhistory.append({\"role\": \"Assistant\", \"content\": state[\"response\"]})\n\n# Turn 2 (reuses Turn 1 KV cache)\nstate = multi_turn_chat.run(history=history, new_message=\"What's 2+2?\")\n# Only computes new message (much faster!)\n\n# Turn 3 (reuses Turn 1 + Turn 2 KV cache)\nstate = multi_turn_chat.run(history=history, new_message=\"Tell me a joke\")\n# Progressively faster as history grows\n```\n\n## Advanced features\n\n### Speculative decoding\n\n```bash\n# Launch with draft model (2-3× faster)\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-70B-Instruct \\\n    --speculative-model meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --speculative-num-steps 5\n```\n\n### Multi-modal (vision models)\n\n```python\n@sgl.function\ndef describe_image(s, image_path):\n    s += sgl.image(image_path)\n    s += \"Describe this image in detail: \"\n    s += sgl.gen(\"description\", max_tokens=200)\n\nstate = describe_image.run(image_path=\"photo.jpg\")\nprint(state[\"description\"])\n```\n\n### Batching and parallel requests\n\n```python\n# Automatic batching (continuous batching)\nstates = sgl.run_batch(\n    [\n        simple_gen.bind(question=\"What is AI?\"),\n        simple_gen.bind(question=\"What is ML?\"),\n        simple_gen.bind(question=\"What is DL?\"),\n    ]\n)\n\n# All 3 processed in single batch (efficient)\n```\n\n## OpenAI-compatible API\n\n```bash\n# Start server with OpenAI API\npython -m sglang.launch_server \\\n    --model-path meta-llama\u002FMeta-Llama-3-8B-Instruct \\\n    --port 30000\n\n# Use with OpenAI client\ncurl http:\u002F\u002Flocalhost:30000\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"model\": \"default\",\n    \"messages\": [\n      {\"role\": \"system\", \"content\": \"You are helpful\"},\n      {\"role\": \"user\", \"content\": \"Hello\"}\n    ],\n    \"temperature\": 0.7,\n    \"max_tokens\": 100\n  }'\n\n# Works with OpenAI Python SDK\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http:\u002F\u002Flocalhost:30000\u002Fv1\", api_key=\"EMPTY\")\n\nresponse = client.chat.completions.create(\n    model=\"default\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello\"}]\n)\n```\n\n## Supported models\n\n**Text models**:\n- Llama 2, Llama 3, Llama 3.1, Llama 3.2\n- Mistral, Mixtral\n- Qwen, Qwen2, QwQ\n- DeepSeek-V2, DeepSeek-V3\n- Gemma, Phi-3\n\n**Vision models**:\n- LLaVA, LLaVA-OneVision\n- Phi-3-Vision\n- Qwen2-VL\n\n**100+ models** from HuggingFace\n\n## Hardware support\n\n**NVIDIA**: A100, H100, L4, T4 (CUDA 11.8+)\n**AMD**: MI300, MI250 (ROCm 6.0+)\n**Intel**: Xeon with GPU (coming soon)\n**Apple**: M1\u002FM2\u002FM3 via MPS (experimental)\n\n## References\n\n- **[Structured Generation Guide](references\u002Fstructured-generation.md)** - JSON schemas, regex, grammars, validation\n- **[RadixAttention Deep Dive](references\u002Fradix-attention.md)** - How it works, optimization, benchmarks\n- **[Production Deployment](references\u002Fdeployment.md)** - Multi-GPU, monitoring, autoscaling\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fsgl-project\u002Fsglang\n- **Docs**: https:\u002F\u002Fsgl-project.github.io\u002F\n- **Paper**: RadixAttention (arXiv:2312.07104)\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002Fsglang\n\n\n",{"name":5,"tags":46,"author":48,"license":15,"version":49,"description":7,"dependencies":50},[36,19,35,34,33,32,47,20,30,29],"Agents","Orchestra Research","1.0.0",[5,51,52],"torch","transformers","main","90b50ed0778745a80fd5bfb13b1843d45b9e82f7","tests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Finference-serving-sglang",231,"https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust","https:\u002F\u002Fgithub.com\u002FDicklesworthstone","https:\u002F\u002Fwww.jeffreyemanuel.com\u002F","Building in NY","2026-11-11 01:23:40",{"source":8,"skillName":5,"commands":63,"prompt":73,"zipAvailable":74},[64,67,70],{"tool":65,"command":66},"npx","npx skills add Dicklesworthstone\u002Fpi_agent_rust --skill sglang",{"tool":68,"command":69},"bunx","bunx skills add Dicklesworthstone\u002Fpi_agent_rust --skill sglang",{"tool":71,"command":72},"pnpm","pnpm dlx skills add Dicklesworthstone\u002Fpi_agent_rust --skill sglang","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Finference-serving-sglang\u002FSKILL.md",true,[76,82,86],{"path":77,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Fdeployment.md","REFERENCE","参考文档",10020,"https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Finference-serving-sglang\u002Freferences\u002Fdeployment.md",{"path":83,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":84,"sourceHtmlUrl":85},"references\u002Fradix-attention.md",11005,"https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Finference-serving-sglang\u002Freferences\u002Fradix-attention.md",{"path":87,"resourceType":78,"resourceTypeDesc":79,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Fstructured-generation.md",13468,"https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Finference-serving-sglang\u002Freferences\u002Fstructured-generation.md",[91,132,157,183,208,230],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":96,"sourceHtmlUrl":97,"categories":98,"keywords":104,"publishedAt":41},"01M6FEGPPK0ZTKCCM0R3E3KS33","web-artifacts-builder","Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn\u002Fui). Use for complex artifacts requiring state management, routing, or shadcn\u002Fui components - not for simple single-file HTML\u002FJSX artifacts.","Apache-2.0","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Futilities\u002Fweb-artifacts-builder\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Futilities\u002Fweb-artifacts-builder\u002FSKILL.md",[99],{"categoryNo":100,"categoryName":101,"taxonomyType":102,"taxonomyTypeDesc":103},"01JBSEEDCAT000000000000009","前端开发","SCENARIO","使用场景",[105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131],"web artifacts builder","web","artifacts","artifacts builder","builder","frontend","css","components","react","component","routing","state management","state","shadcn","file","ui","Tools","Tailwind","use","html","single-file","simple","ui components","multi","for","jsx","suite",{"skillNo":133,"name":134,"nameZh":6,"description":135,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":136,"sourceHtmlUrl":137,"categories":138,"keywords":143,"publishedAt":41},"01M6FEE9VTK1W4GY21AKREJXRG","webapp-testing","Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fsecurity\u002Fwebapp-testing\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fsecurity\u002Fwebapp-testing\u002FSKILL.md",[139,142],{"categoryNo":140,"categoryName":141,"taxonomyType":102,"taxonomyTypeDesc":103},"01JBSEEDCAT000000000000008","测试与质量",{"categoryNo":100,"categoryName":101,"taxonomyType":102,"taxonomyTypeDesc":103},[144,145,146,110,147,148,106,149,150,151,120,152,153,154,155,156,129],"testing","webapp testing","webapp","screenshots","playwright","debugging","toolkit","browser","logs","applications","behavior","local","with",{"skillNo":158,"name":159,"nameZh":6,"description":160,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":161,"sourceHtmlUrl":162,"categories":163,"keywords":164,"publishedAt":41},"01M6FE20F7ZTQBFWKF4MPJ2SBX","neuropixels-analysis","Neuropixels neural recording analysis. Load SpikeGLX\u002FOpenEphys data, preprocess, motion correction, Kilosort4 spike sorting, quality metrics, Allen\u002FIBL curation, AI-assisted visual analysis, for Neuropixels 1.0\u002F2.0 extracellular electrophysiology. Use when working with neural recordings, spike sorting, extracellular electrophysiology, or when the user mentions Neuropixels, SpikeGLX, Open Ephys, Kilosort, quality metrics, or unit curation.","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fscientific\u002Fneuropixels-analysis\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fscientific\u002Fneuropixels-analysis\u002FSKILL.md",[],[165,166,167,168,169,170,171,172,123,173,174,175,176,177,178,179,156,180,181,182,129,21],"analysis","neuropixels analysis","spike","metrics","data","open","quality","user","recordings","Neural","motion","visual","quality metrics","unit","sorting","curation","recording","load",{"skillNo":184,"name":185,"nameZh":6,"description":186,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":187,"sourceHtmlUrl":188,"categories":189,"keywords":193,"publishedAt":41},"01M6FDP0PDK6F1GGX395C7625Z","skill-creator","Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Claude's capabilities with specialized knowledge, workflows, or tool integrations.","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Fskill-creator\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Fskill-creator\u002FSKILL.md",[190],{"categoryNo":191,"categoryName":192,"taxonomyType":102,"taxonomyTypeDesc":103},"01JBSEEDCAT000000000000023","Skill与智能体开发",[194,195,39,196,197,198,199,27,200,201,202,203,204,205,156,129,206,21,207],"skill creator","creator","create","update","capabilities","knowledge","integrations","specialized","guide","new skill","existing skill","users","Existing","new",{"skillNo":209,"name":210,"nameZh":6,"description":211,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":212,"sourceHtmlUrl":213,"categories":214,"keywords":216,"publishedAt":41},"01M6FDM8Y7VK7NYG9SM7QPA3PW","notebooklm","Use this skill to query your Google NotebookLM notebooks directly from Claude Code for source-grounded, citation-backed answers from Gemini. Browser automation, library management, persistent auth. Drastically reduced hallucinations through document-only responses.","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Fnotebooklm\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Fnotebooklm\u002FSKILL.md",[215],{"categoryNo":191,"categoryName":192,"taxonomyType":102,"taxonomyTypeDesc":103},[210,217,218,219,220,221,222,151,223,224,123,225,226,227,228,129,229],"google","browser automation","gemini","automation","auth","code","library","management","only","document","query","Citation","source",{"skillNo":231,"name":232,"nameZh":6,"description":233,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":95,"licenseFileUrl":234,"sourceHtmlUrl":235,"categories":236,"keywords":243,"publishedAt":41},"01M6FDHPXE10J3XGTCH5HBHS1M","slack-gif-creator","Knowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like \"make me a GIF of X doing Y for Slack.\"","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fslack-gif-creator\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fslack-gif-creator\u002FSKILL.md",[237,240],{"categoryNo":238,"categoryName":239,"taxonomyType":102,"taxonomyTypeDesc":103},"01JBSEEDCAT000000000000005","邮件与沟通",{"categoryNo":241,"categoryName":242,"taxonomyType":102,"taxonomyTypeDesc":103},"01JBSEEDCAT000000000000016","图像与多媒体",[244,245,246,195,247,121,248,249,199,123,250,205,251,252,253,129,254,21],"slack gif creator","slack","gif","validation","constraints","Animation","me","make","Utilities","concepts","request"]