[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7A6RDAPWDZD237RABAWDZFQ":3,"skill-same:01M7A6RDAPWDZD237RABAWDZFQ":76},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":34,"commitSha":35,"skillDirectory":36,"forksCount":37,"repositoryHtmlUrl":38,"repositoryHomepageUrl":39,"ownerHtmlUrl":40,"ownerBlogUrl":41,"ownerDescription":42,"updatedAt":43,"install":44,"resources":57},"01M7A6RDAPWDZD237RABAWDZFQ","jetson-llm-serve",null,"Stand up vLLM or SGLang serving on Jetson, using upstream vLLM on Thor and Orin JetPack 7.2+, and NVIDIA-AI-IOT vLLM on older Orin.","themusebook\u002Fskills","themusebook","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F349927247?v=4","01M7A5C1RV5S3V5FF6ZACVDNRN","musebook","01M7A5C1SFC3KRZAXNN60XJ7KM",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fjetson-llm-serve\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: jetson-llm-serve\ndescription: Stand up vLLM or SGLang serving on Jetson, using upstream vLLM on Thor and Orin JetPack 7.2+, and NVIDIA-AI-IOT vLLM on older Orin.\nversion: 0.0.1\nlicense: \"Apache-2.0\"\nmetadata:\n  author: \"Jetson Team\"\n  tags: [jetson, llm, serving]\n  languages: [markdown]\n  data-classification: public\n---\n\n# Jetson LLM Serve\n\nEncodes the [Jetson AI Lab GenAI tutorial](https:\u002F\u002Fwww.jetson-ai-lab.com\u002Ftutorials\u002Fgenai-on-jetson-llms-vlms\u002F): on Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`); on older Orin, pick the NVIDIA-AI-IOT prebuilt vLLM container; on Thor, use upstream vLLM 0.20+ or validated native vLLM 0.20+, and use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) when SGLang is requested. Set MAXN, make Hugging Face credentials\u002Fcache available, and launch an OpenAI-compatible server. Works for both LLMs and VLMs.\n\n## Purpose\n\nProvide a Jetson-appropriate serving recipe for an LLM or VLM using vLLM or SGLang, including runtime path, launch command, endpoint, and verification step.\n\n## When to use\n\n- \"Run \u002F serve \u002F host this model on a Jetson.\"\n- \"Start a vLLM server I can hit from Open WebUI \u002F my app.\"\n- After `jetson-inference-mem-tune` produced launch flags and the user wants to actually start the server.\n\nFor recipe-only questions, answer from this document without starting\ncontainers. Run live pre-flight checks only when the user asks you to check this\ndevice or execute the deployment.\n\n## Prerequisites\n\n- Run on the Jetson host or a shell with Docker access to the Jetson GPU runtime.\n- Know the target Jetson generation (`thor` or `orin`) and the model identifier or local checkpoint path.\n- Use `HF_TOKEN` only when the model is gated\u002Fprivate; public models should omit the token environment variable.\n- Use `jetson-inference-mem-tune` first when memory headroom or launch flags are uncertain.\n\n## Instructions\n\nFor recipe questions, provide a complete launch recipe instead of trying to call\n`jetson-llm-serve` as a tool. A complete answer includes:\n\n- The Jetson-appropriate runtime path: upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) on Thor, NVIDIA-AI-IOT vLLM container on older Orin, or upstream vLLM 0.20+ on Orin JetPack 7.2 \u002F L4T r39+.\n- The model checkpoint \u002F Hugging Face repo the user named.\n- A `docker run` + server command sketch with `--host 0.0.0.0 --port 8000`.\n- The OpenAI-compatible endpoint: `http:\u002F\u002F\u003Cjetson-ip>:8000\u002Fv1`.\n- A verification step such as `curl http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fmodels`.\n\nFor VLM questions, explicitly say the VLM uses the same vLLM serving flow as an\nLLM with a different vision-language checkpoint. Do not omit `vLLM` or the\nJetson container when answering VLM prompts.\n\n## Step 1 — Pick the runtime path (per Jetson family)\n\nUse upstream vLLM 0.20+ on Thor (`vllm\u002Fvllm-openai:latest`, or a validated native vLLM 0.20+ install). On Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`). On older Orin releases, use the **NVIDIA-AI-IOT prebuilt vLLM image** ([packages](https:\u002F\u002Fgithub.com\u002Forgs\u002FNVIDIA-AI-IOT\u002Fpackages)) because it ships the correct CUDA \u002F cuDNN \u002F TensorRT stack for that JetPack. Use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) on Thor when the user asks for SGLang, RAG, tool-use, or programmable serving; do not recommend native upstream SGLang on Orin unless a JetPack-matched release explicitly supports it.\n\n| Jetson family               | Runtime path                                      |\n|-----------------------------|---------------------------------------------------|\n| Thor (T5000, T4000)         | upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) |\n| AGX Orin \u002F Orin NX \u002F Nano   | Orin JetPack 7.2 \u002F L4T r39+: upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`); older Orin: `ghcr.io\u002Fnvidia-ai-iot\u002Fvllm:latest-jetson-orin` |\n\nTo detect the silicon era for image tags:\n\n1. Source the detector so exports survive in your shell:\n   ```bash\n   . skills\u002Fjetson-diagnostic\u002Fscripts\u002Fdetect_jetson.sh\n   ```\n2. Check `JETSON_GENERATION` (`thor` or `orin`) and choose the matching runtime path from the table above.\n3. Use `JETSON_PRODUCT_LINE` for a finer bucket such as `thor-agx` or `orin-nano`; `JETSON_SKU` remains the legacy identifier.\n\nDo not use `bash skills\u002Fjetson-diagnostic\u002Fscripts\u002Fdetect_jetson.sh` when you need exported variables in the caller; running with `bash` uses a subshell.\n\n## Step 2 — Set MAXN power mode\n\n```bash\nsudo nvpmodel -m 0 && sudo jetson_clocks\n```\n\nSkip this only if the user explicitly asks for a power-constrained run; otherwise benchmark and serving numbers will be inconsistent.\n\n## Step 3 — Run the server\n\nOn Thor with vLLM, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or a validated native vLLM 0.20+ install:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --ipc host --name vllm \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  vllm\u002Fvllm-openai:latest \\\n  vllm serve \u003Chf-repo-id> \\\n    --host 0.0.0.0 --port 8000 \\\n    --max-model-len 8192 \\\n    --gpu-memory-utilization 0.75 \\\n    --tensor-parallel-size 1\n```\n\nOn Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`). On older Orin releases, use the NVIDIA-AI-IOT container:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --name vllm \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  ghcr.io\u002Fnvidia-ai-iot\u002Fvllm:latest-jetson-orin \\\n  vllm serve \u003Chf-repo-id> \\\n    --host 0.0.0.0 --port 8000 \\\n    --max-model-len 4096 \\\n    --gpu-memory-utilization 0.85 \\\n    --tensor-parallel-size 1\n```\n\n`HF_TOKEN` is required only for gated\u002Fprivate Hugging Face models; omit the `-e HF_TOKEN=\"$HF_TOKEN\"` line for public models that do not need Hub authentication. Passing `HF_TOKEN` as an environment variable can expose it through Docker inspect output, process metadata, or logs on shared systems. Prefer the narrowest-scoped token possible, rotate\u002Frevoke it after shared-container use, and use a mounted credential file or Docker secret when the deployment environment supports that pattern.\n\nWait for `Application startup complete.` Server is on `http:\u002F\u002F0.0.0.0:8000\u002Fv1`.\n\nFor SGLang on Thor, use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`), which packages SGLang 0.5.5.post2 and lists Jetson Thor support. Do not judge Thor SGLang support from older prerelease SGLang results:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --ipc host --name sglang \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3 \\\n  python3 -m sglang.launch_server \\\n    --model-path \u003Chf-repo-id> \\\n    --host 0.0.0.0 \\\n    --port 8000 \\\n    --mem-fraction-static 0.60 \\\n    --max-running-requests 8\n```\n\nUse SGLang when the user needs RAG\u002Ftool-use workflows, structured generation, or\nSGLang-specific scheduling. For plain high-throughput OpenAI-compatible serving,\nprefer vLLM unless the user asks for SGLang.\n\n### SKU-appropriate defaults\n\n| Knob                       | Orin Nano \u002F NX | AGX Orin \u002F Thor |\n|----------------------------|----------------|-----------------|\n| `--max-model-len`          | `4096`         | `8192`          |\n| `--gpu-memory-utilization` | `0.85`         | `0.85`          |\n| `--tensor-parallel-size`   | `1`            | `1`             |\n\nIf the server OOMs at startup, lower `--gpu-memory-utilization` by 0.05 and re-launch (or run `jetson-inference-mem-tune` for a workload-aware recommendation).\n\n## Quantization preferences (matters more than the runtime)\n\nFor vLLM and SGLang, choose checkpoint formats by Jetson family:\n\n| Jetson family | First choice | Acceptable fallback |\n|---------------|--------------|---------------------|\n| Thor | **NVFP4** when the model\u002Fruntime supports it | W4A16 |\n| Orin Nano \u002F NX | **W4A16** | AWQ or GPTQ 4-bit |\n| AGX Orin | **W4A16** | AWQ or GPTQ 4-bit |\n\nFor llama.cpp and `Ollama`, use GGUF model quantization names instead: recommend **INT4 \u002F Q4_K_M GGUF** on both Orin and Thor, and choose a smaller INT4 GGUF model if memory is tight. Do not call GGUF Q4_K_M a W4A16\u002FAWQ\u002FGPTQ model. NVFP4 is Thor-preferred and Thor-tuned for runtimes that support it.\n\n## VLM mode\n\nVLMs use the same flow as LLMs: same container, same `vllm serve` invocation, different vision-language checkpoint. The container handles image preprocessing. For a VLM-specific browser UI, use the [`live-vlm-webui`](https:\u002F\u002Fgithub.com\u002Forgs\u002FNVIDIA-AI-IOT\u002Fpackages) container; for a generic chat UI for either, use Open WebUI pointed at `http:\u002F\u002F\u003Cjetson-ip>:8000\u002Fv1`.\n\n## Do not fabricate device capacity\n\nDo not invent RAM totals, free-memory values, model sizes, JetPack versions, or\nSKU\u002Fvariant names when giving a serving recipe. If capacity matters, either run\nthe live pre-flight checks (when execution is allowed) or hand off to\n`jetson-inference-mem-tune` \u002F `jetson-memory-audit`. If live data is not\navailable, say the value is unknown and provide conservative defaults instead\nof quoting a made-up number.\n\n## Pre-flight checklist (the agent should verify before running Step 3)\n\n- [ ] On a Jetson (`\u002Fproc\u002Fdevice-tree\u002Fmodel` contains `NVIDIA Jetson`).\n- [ ] `nvpmodel -q` reports a recognized max-performance mode: `MAXN` or `MAXN_*` such as `MAXN_SUPER`. Wattage-named modes should be reported as warnings unless the user explicitly confirms they are the intended benchmark mode for that device.\n- [ ] On Thor, check whether MIG is enabled before launching (`nvidia-smi -L` and `nvidia-smi mig -lgi`). If MIG is enabled, warn that vLLM\u002FSGLang may see only a MIG slice or no CUDA device.\n- [ ] On Thor with MIG or display\u002Fcamera contention, inspect GPU users with `sudo lsof \u002Fdev\u002Fnvidia*`. Display managers, `Xorg`\u002FGNOME, or `nvargus-daemon` may hold GPU device files; do not stop services or change MIG mode unless the user explicitly approves.\n- [ ] No container named `vllm` already running (`docker ps --format '{{.Names}}'`); otherwise `docker rm -f vllm` first.\n- [ ] Docker exposes the NVIDIA runtime (`docker info | grep -i 'runtimes.*nvidia'`), or a GPU-enabled container can run `nvidia-smi`.\n- [ ] `~\u002F.cache\u002Fhuggingface` exists; `HF_TOKEN` is set if the model is gated.\n\n## Limitations\n\n- This skill provides serving commands and pre-flight checks; it does not benchmark the deployed server.\n- Container tags such as `latest` are mutable. For release or compliance deployments, pin a digest and record it with the deployment notes.\n- vLLM and SGLang memory limits still depend on model architecture, quantization, context length, and concurrent request count. Use `jetson-inference-mem-tune` when a command OOMs or memory headroom matters.\n- Thor vLLM requires upstream vLLM 0.20+ or newer. Older upstream vLLM images may not support Thor \u002F SM 11.0 correctly.\n- Thor SGLang should use NVIDIA SGLang 26.01 or newer release notes that explicitly list Jetson Thor support. NVIDIA SGLang 26.01 contains SGLang 0.5.5.post2.\n- On Thor, MIG, desktop display, or camera services can hide the full GPU from containers. This skill should detect and warn only; disabling MIG or stopping services such as `gdm3` or `nvargus-daemon` requires explicit user approval.\n- Host-native vLLM\u002FSGLang on Thor should be used only when that install is already validated on the target JetPack.\n\n## Hand off to\n\n- `jetson-llm-benchmark` to actually measure the deployed server.\n- `jetson-speculative-decoding` to add EAGLE-3 \u002F draft-model speculation by appending `--speculative-config '{...}'` to the `vllm serve` command above.\n- `jetson-inference-mem-tune` if the server OOMs or is memory-bound.\n\n## Source\n\n[Jetson AI Lab — Introduction to GenAI on Jetson: How to Run LLMs and VLMs](https:\u002F\u002Fwww.jetson-ai-lab.com\u002Ftutorials\u002Fgenai-on-jetson-llms-vlms\u002F) and [NVIDIA-AI-IOT GHCR packages](https:\u002F\u002Fgithub.com\u002Forgs\u002FNVIDIA-AI-IOT\u002Fpackages).\n","# Jetson LLM Serve\n\nEncodes the [Jetson AI Lab GenAI tutorial](https:\u002F\u002Fwww.jetson-ai-lab.com\u002Ftutorials\u002Fgenai-on-jetson-llms-vlms\u002F): on Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`); on older Orin, pick the NVIDIA-AI-IOT prebuilt vLLM container; on Thor, use upstream vLLM 0.20+ or validated native vLLM 0.20+, and use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) when SGLang is requested. Set MAXN, make Hugging Face credentials\u002Fcache available, and launch an OpenAI-compatible server. Works for both LLMs and VLMs.\n\n## Purpose\n\nProvide a Jetson-appropriate serving recipe for an LLM or VLM using vLLM or SGLang, including runtime path, launch command, endpoint, and verification step.\n\n## When to use\n\n- \"Run \u002F serve \u002F host this model on a Jetson.\"\n- \"Start a vLLM server I can hit from Open WebUI \u002F my app.\"\n- After `jetson-inference-mem-tune` produced launch flags and the user wants to actually start the server.\n\nFor recipe-only questions, answer from this document without starting\ncontainers. Run live pre-flight checks only when the user asks you to check this\ndevice or execute the deployment.\n\n## Prerequisites\n\n- Run on the Jetson host or a shell with Docker access to the Jetson GPU runtime.\n- Know the target Jetson generation (`thor` or `orin`) and the model identifier or local checkpoint path.\n- Use `HF_TOKEN` only when the model is gated\u002Fprivate; public models should omit the token environment variable.\n- Use `jetson-inference-mem-tune` first when memory headroom or launch flags are uncertain.\n\n## Instructions\n\nFor recipe questions, provide a complete launch recipe instead of trying to call\n`jetson-llm-serve` as a tool. A complete answer includes:\n\n- The Jetson-appropriate runtime path: upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) on Thor, NVIDIA-AI-IOT vLLM container on older Orin, or upstream vLLM 0.20+ on Orin JetPack 7.2 \u002F L4T r39+.\n- The model checkpoint \u002F Hugging Face repo the user named.\n- A `docker run` + server command sketch with `--host 0.0.0.0 --port 8000`.\n- The OpenAI-compatible endpoint: `http:\u002F\u002F\u003Cjetson-ip>:8000\u002Fv1`.\n- A verification step such as `curl http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fmodels`.\n\nFor VLM questions, explicitly say the VLM uses the same vLLM serving flow as an\nLLM with a different vision-language checkpoint. Do not omit `vLLM` or the\nJetson container when answering VLM prompts.\n\n## Step 1 — Pick the runtime path (per Jetson family)\n\nUse upstream vLLM 0.20+ on Thor (`vllm\u002Fvllm-openai:latest`, or a validated native vLLM 0.20+ install). On Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`). On older Orin releases, use the **NVIDIA-AI-IOT prebuilt vLLM image** ([packages](https:\u002F\u002Fgithub.com\u002Forgs\u002FNVIDIA-AI-IOT\u002Fpackages)) because it ships the correct CUDA \u002F cuDNN \u002F TensorRT stack for that JetPack. Use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) on Thor when the user asks for SGLang, RAG, tool-use, or programmable serving; do not recommend native upstream SGLang on Orin unless a JetPack-matched release explicitly supports it.\n\n| Jetson family               | Runtime path                                      |\n|-----------------------------|---------------------------------------------------|\n| Thor (T5000, T4000)         | upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) |\n| AGX Orin \u002F Orin NX \u002F Nano   | Orin JetPack 7.2 \u002F L4T r39+: upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`); older Orin: `ghcr.io\u002Fnvidia-ai-iot\u002Fvllm:latest-jetson-orin` |\n\nTo detect the silicon era for image tags:\n\n1. Source the detector so exports survive in your shell:\n   ```bash\n   . skills\u002Fjetson-diagnostic\u002Fscripts\u002Fdetect_jetson.sh\n   ```\n2. Check `JETSON_GENERATION` (`thor` or `orin`) and choose the matching runtime path from the table above.\n3. Use `JETSON_PRODUCT_LINE` for a finer bucket such as `thor-agx` or `orin-nano`; `JETSON_SKU` remains the legacy identifier.\n\nDo not use `bash skills\u002Fjetson-diagnostic\u002Fscripts\u002Fdetect_jetson.sh` when you need exported variables in the caller; running with `bash` uses a subshell.\n\n## Step 2 — Set MAXN power mode\n\n```bash\nsudo nvpmodel -m 0 && sudo jetson_clocks\n```\n\nSkip this only if the user explicitly asks for a power-constrained run; otherwise benchmark and serving numbers will be inconsistent.\n\n## Step 3 — Run the server\n\nOn Thor with vLLM, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or a validated native vLLM 0.20+ install:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --ipc host --name vllm \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  vllm\u002Fvllm-openai:latest \\\n  vllm serve \u003Chf-repo-id> \\\n    --host 0.0.0.0 --port 8000 \\\n    --max-model-len 8192 \\\n    --gpu-memory-utilization 0.75 \\\n    --tensor-parallel-size 1\n```\n\nOn Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`). On older Orin releases, use the NVIDIA-AI-IOT container:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --name vllm \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  ghcr.io\u002Fnvidia-ai-iot\u002Fvllm:latest-jetson-orin \\\n  vllm serve \u003Chf-repo-id> \\\n    --host 0.0.0.0 --port 8000 \\\n    --max-model-len 4096 \\\n    --gpu-memory-utilization 0.85 \\\n    --tensor-parallel-size 1\n```\n\n`HF_TOKEN` is required only for gated\u002Fprivate Hugging Face models; omit the `-e HF_TOKEN=\"$HF_TOKEN\"` line for public models that do not need Hub authentication. Passing `HF_TOKEN` as an environment variable can expose it through Docker inspect output, process metadata, or logs on shared systems. Prefer the narrowest-scoped token possible, rotate\u002Frevoke it after shared-container use, and use a mounted credential file or Docker secret when the deployment environment supports that pattern.\n\nWait for `Application startup complete.` Server is on `http:\u002F\u002F0.0.0.0:8000\u002Fv1`.\n\nFor SGLang on Thor, use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`), which packages SGLang 0.5.5.post2 and lists Jetson Thor support. Do not judge Thor SGLang support from older prerelease SGLang results:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --ipc host --name sglang \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3 \\\n  python3 -m sglang.launch_server \\\n    --model-path \u003Chf-repo-id> \\\n    --host 0.0.0.0 \\\n    --port 8000 \\\n    --mem-fraction-static 0.60 \\\n    --max-running-requests 8\n```\n\nUse SGLang when the user needs RAG\u002Ftool-use workflows, structured generation, or\nSGLang-specific scheduling. For plain high-throughput OpenAI-compatible serving,\nprefer vLLM unless the user asks for SGLang.\n\n### SKU-appropriate defaults\n\n| Knob                       | Orin Nano \u002F NX | AGX Orin \u002F Thor |\n|----------------------------|----------------|-----------------|\n| `--max-model-len`          | `4096`         | `8192`          |\n| `--gpu-memory-utilization` | `0.85`         | `0.85`          |\n| `--tensor-parallel-size`   | `1`            | `1`             |\n\nIf the server OOMs at startup, lower `--gpu-memory-utilization` by 0.05 and re-launch (or run `jetson-inference-mem-tune` for a workload-aware recommendation).\n\n## Quantization preferences (matters more than the runtime)\n\nFor vLLM and SGLang, choose checkpoint formats by Jetson family:\n\n| Jetson family | First choice | Acceptable fallback |\n|---------------|--------------|---------------------|\n| Thor | **NVFP4** when the model\u002Fruntime supports it | W4A16 |\n| Orin Nano \u002F NX | **W4A16** | AWQ or GPTQ 4-bit |\n| AGX Orin | **W4A16** | AWQ or GPTQ 4-bit |\n\nFor llama.cpp and `Ollama`, use GGUF model quantization names instead: recommend **INT4 \u002F Q4_K_M GGUF** on both Orin and Thor, and choose a smaller INT4 GGUF model if memory is tight. Do not call GGUF Q4_K_M a W4A16\u002FAWQ\u002FGPTQ model. NVFP4 is Thor-preferred and Thor-tuned for runtimes that support it.\n\n## VLM mode\n\nVLMs use the same flow as LLMs: same container, same `vllm serve` invocation, different vision-language checkpoint. The container handles image preprocessing. For a VLM-specific browser UI, use the [`live-vlm-webui`](https:\u002F\u002Fgithub.com\u002Forgs\u002FNVIDIA-AI-IOT\u002Fpackages) container; for a generic chat UI for either, use Open WebUI pointed at `http:\u002F\u002F\u003Cjetson-ip>:8000\u002Fv1`.\n\n## Do not fabricate device capacity\n\nDo not invent RAM totals, free-memory values, model sizes, JetPack versions, or\nSKU\u002Fvariant names when giving a serving recipe. If capacity matters, either run\nthe live pre-flight checks (when execution is allowed) or hand off to\n`jetson-inference-mem-tune` \u002F `jetson-memory-audit`. If live data is not\navailable, say the value is unknown and provide conservative defaults instead\nof quoting a made-up number.\n\n## Pre-flight checklist (the agent should verify before running Step 3)\n\n- [ ] On a Jetson (`\u002Fproc\u002Fdevice-tree\u002Fmodel` contains `NVIDIA Jetson`).\n- [ ] `nvpmodel -q` reports a recognized max-performance mode: `MAXN` or `MAXN_*` such as `MAXN_SUPER`. Wattage-named modes should be reported as warnings unless the user explicitly confirms they are the intended benchmark mode for that device.\n- [ ] On Thor, check whether MIG is enabled before launching (`nvidia-smi -L` and `nvidia-smi mig -lgi`). If MIG is enabled, warn that vLLM\u002FSGLang may see only a MIG slice or no CUDA device.\n- [ ] On Thor with MIG or display\u002Fcamera contention, inspect GPU users with `sudo lsof \u002Fdev\u002Fnvidia*`. Display managers, `Xorg`\u002FGNOME, or `nvargus-daemon` may hold GPU device files; do not stop services or change MIG mode unless the user explicitly approves.\n- [ ] No container named `vllm` already running (`docker ps --format '{{.Names}}'`); otherwise `docker rm -f vllm` first.\n- [ ] Docker exposes the NVIDIA runtime (`docker info | grep -i 'runtimes.*nvidia'`), or a GPU-enabled container can run `nvidia-smi`.\n- [ ] `~\u002F.cache\u002Fhuggingface` exists; `HF_TOKEN` is set if the model is gated.\n\n## Limitations\n\n- This skill provides serving commands and pre-flight checks; it does not benchmark the deployed server.\n- Container tags such as `latest` are mutable. For release or compliance deployments, pin a digest and record it with the deployment notes.\n- vLLM and SGLang memory limits still depend on model architecture, quantization, context length, and concurrent request count. Use `jetson-inference-mem-tune` when a command OOMs or memory headroom matters.\n- Thor vLLM requires upstream vLLM 0.20+ or newer. Older upstream vLLM images may not support Thor \u002F SM 11.0 correctly.\n- Thor SGLang should use NVIDIA SGLang 26.01 or newer release notes that explicitly list Jetson Thor support. NVIDIA SGLang 26.01 contains SGLang 0.5.5.post2.\n- On Thor, MIG, desktop display, or camera services can hide the full GPU from containers. This skill should detect and warn only; disabling MIG or stopping services such as `gdm3` or `nvargus-daemon` requires explicit user approval.\n- Host-native vLLM\u002FSGLang on Thor should be used only when that install is already validated on the target JetPack.\n\n## Hand off to\n\n- `jetson-llm-benchmark` to actually measure the deployed server.\n- `jetson-speculative-decoding` to add EAGLE-3 \u002F draft-model speculation by appending `--speculative-config '{...}'` to the `vllm serve` command above.\n- `jetson-inference-mem-tune` if the server OOMs or is memory-bound.\n\n## Source\n\n[Jetson AI Lab — Introduction to GenAI on Jetson: How to Run LLMs and VLMs](https:\u002F\u002Fwww.jetson-ai-lab.com\u002Ftutorials\u002Fgenai-on-jetson-llms-vlms\u002F) and [NVIDIA-AI-IOT GHCR packages](https:\u002F\u002Fgithub.com\u002Forgs\u002FNVIDIA-AI-IOT\u002Fpackages).\n",{"name":5,"license":15,"version":24,"metadata":25,"description":7},"0.0.1",{"tags":26,"author":30,"languages":31,"data-classification":33},[27,28,29],"jetson","llm","serving","Jetson Team",[32],"markdown","public","main","1c523aba5cdf867a79f078847075492d7778092a","public\u002Fapi\u002Fskills\u002Fnvidia\u002Fjetson-llm-serve",1,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills","https:\u002F\u002Fskills.x402.wtf","https:\u002F\u002Fgithub.com\u002Fthemusebook","https:\u002F\u002Fmusebook.trade","The official Github of musebook","2026-11-02 23:42:26",{"source":8,"skillName":5,"commands":45,"prompt":55,"zipAvailable":56},[46,49,52],{"tool":47,"command":48},"npx","npx skills add themusebook\u002Fskills --skill jetson-llm-serve",{"tool":50,"command":51},"bunx","bunx skills add themusebook\u002Fskills --skill jetson-llm-serve",{"tool":53,"command":54},"pnpm","pnpm dlx skills add themusebook\u002Fskills --skill jetson-llm-serve","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fjetson-llm-serve\u002FSKILL.md",true,[58,64,68,72],{"path":59,"resourceType":60,"resourceTypeDesc":61,"sizeBytes":62,"sourceHtmlUrl":63},"BENCHMARK.md","REFERENCE","参考文档",3310,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fjetson-llm-serve\u002FBENCHMARK.md",{"path":65,"resourceType":60,"resourceTypeDesc":61,"sizeBytes":66,"sourceHtmlUrl":67},"metadata.json",329,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fjetson-llm-serve\u002Fmetadata.json",{"path":69,"resourceType":60,"resourceTypeDesc":61,"sizeBytes":70,"sourceHtmlUrl":71},"skill-card.md",3727,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fjetson-llm-serve\u002Fskill-card.md",{"path":73,"resourceType":60,"resourceTypeDesc":61,"sizeBytes":74,"sourceHtmlUrl":75},"verification.json",1326,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Fjetson-llm-serve\u002Fverification.json",[77,86,93,100,107,114],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":82,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":19},"01M7AB34SPKVSDRRG009PM1CZH","youtube-clipper","YouTube 视频智能剪辑工具。下载视频和字幕，AI 分析生成精细章节（几分钟级别）， 用户选择片段后自动剪辑、翻译字幕为中英双语、烧录字幕到视频，并生成总结文案。 使用场景：当用户需要剪辑 YouTube 视频、生成短视频片段、制作双语字幕版本时。 关键词：视频剪辑、YouTube、字幕翻译、双语字幕、视频下载、clip video","MIT","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fyoutube-clipper\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fyoutube-clipper\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":19},"01M7AB18BH6N7KR7939HRNWVMQ","viem-integration","Integrate EVM blockchains using viem. Use when user says \"read blockchain data\", \"send transaction\", \"interact with smart contract\", \"connect to Ethereum\", \"use viem\", \"use wagmi\", \"wallet integration\", \"viem setup\", or mentions blockchain\u002FEVM development with TypeScript.","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fviem-integration\u002FSKILL.md",[],[],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":19},"01M7AB12GHGP48WCNXDQQ9222H","v4-security-foundations","Security-first Uniswap v4 hook development. Use when user mentions \"v4 hooks\", \"hook security\", \"PoolManager\", \"beforeSwap\", \"afterSwap\", or asks about V4 hook best practices, vulnerabilities, or audit requirements.","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fv4-security-foundations\u002FSKILL.md",[],[],{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":19},"01M7AB0XM3PSR7V3Q2ZRR0Q0FB","v4-sdk-integration","App-layer SDK guide for building swap and liquidity experiences directly with the Uniswap v4 SDK. Use when user asks about \"v4 sdk\", \"uniswap v4\", \"v4 swap\", \"v4 liquidity\", \"PoolManager\", \"V4Planner\", \"StateView\", \"PositionManager\", \"pool state\", \"v4 position\", \"uniswap sdk\", or when building swap\u002Fliquidity UX directly with SDKs rather than via the Trading API.","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fv4-sdk-integration\u002FSKILL.md",[],[],{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":6,"sourceHtmlUrl":111,"categories":112,"keywords":113,"publishedAt":19},"01M7AB0SGA93ASA588DJW4P9WT","v4-hook-generator","Generate Uniswap v4 hook contracts via OpenZeppelin MCP. Use when building custom swap logic, async swaps, hook-owned liquidity, custom curves, dynamic fees, MEV protection, limit orders, or oracle hooks.","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fv4-hook-generator\u002FSKILL.md",[],[],{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":6,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":19},"01M7AAZYH75H4CS03WSW2EZ8DM","testing","For testing with Light Protocol programs and clients on localnet, devnet, and mainnet validation.","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Ftesting\u002FSKILL.md",[],[]]