[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M78A7EW0XTNQCSBWDPE4FM0T":3,"skill-same:01M78A7EW0XTNQCSBWDPE4FM0T":76},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":34,"commitSha":35,"skillDirectory":36,"forksCount":14,"repositoryHtmlUrl":37,"repositoryHomepageUrl":6,"ownerHtmlUrl":38,"ownerBlogUrl":39,"ownerDescription":40,"updatedAt":41,"install":42,"resources":55},"01M78A7EW0XTNQCSBWDPE4FM0T","jetson-llm-serve",null,"Stand up vLLM or SGLang serving on Jetson, using upstream vLLM on Thor and Orin JetPack 7.2+, and NVIDIA-AI-IOT vLLM on older Orin.","Solizardking\u002Ftrain2earn","Solizardking","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F108096810?v=4","01M786KDB10AYV042N0F9BD5B5","Clawd","01M786KDC5C3NPE4W44C5641EZ",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fnvidia\u002Fjetson-llm-serve\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: jetson-llm-serve\ndescription: Stand up vLLM or SGLang serving on Jetson, using upstream vLLM on Thor and Orin JetPack 7.2+, and NVIDIA-AI-IOT vLLM on older Orin.\nversion: 0.0.1\nlicense: \"Apache-2.0\"\nmetadata:\n  author: \"Jetson Team\"\n  tags: [jetson, llm, serving]\n  languages: [markdown]\n  data-classification: public\n---\n\n# Jetson LLM Serve\n\nEncodes the [Jetson AI Lab GenAI tutorial](https:\u002F\u002Fwww.jetson-ai-lab.com\u002Ftutorials\u002Fgenai-on-jetson-llms-vlms\u002F): on Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`); on older Orin, pick the NVIDIA-AI-IOT prebuilt vLLM container; on Thor, use upstream vLLM 0.20+ or validated native vLLM 0.20+, and use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) when SGLang is requested. Set MAXN, make Hugging Face credentials\u002Fcache available, and launch an OpenAI-compatible server. Works for both LLMs and VLMs.\n\n## Purpose\n\nProvide a Jetson-appropriate serving recipe for an LLM or VLM using vLLM or SGLang, including runtime path, launch command, endpoint, and verification step.\n\n## When to use\n\n- \"Run \u002F serve \u002F host this model on a Jetson.\"\n- \"Start a vLLM server I can hit from Open WebUI \u002F my app.\"\n- After `jetson-inference-mem-tune` produced launch flags and the user wants to actually start the server.\n\nFor recipe-only questions, answer from this document without starting\ncontainers. Run live pre-flight checks only when the user asks you to check this\ndevice or execute the deployment.\n\n## Prerequisites\n\n- Run on the Jetson host or a shell with Docker access to the Jetson GPU runtime.\n- Know the target Jetson generation (`thor` or `orin`) and the model identifier or local checkpoint path.\n- Use `HF_TOKEN` only when the model is gated\u002Fprivate; public models should omit the token environment variable.\n- Use `jetson-inference-mem-tune` first when memory headroom or launch flags are uncertain.\n\n## Instructions\n\nFor recipe questions, provide a complete launch recipe instead of trying to call\n`jetson-llm-serve` as a tool. A complete answer includes:\n\n- The Jetson-appropriate runtime path: upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) on Thor, NVIDIA-AI-IOT vLLM container on older Orin, or upstream vLLM 0.20+ on Orin JetPack 7.2 \u002F L4T r39+.\n- The model checkpoint \u002F Hugging Face repo the user named.\n- A `docker run` + server command sketch with `--host 0.0.0.0 --port 8000`.\n- The OpenAI-compatible endpoint: `http:\u002F\u002F\u003Cjetson-ip>:8000\u002Fv1`.\n- A verification step such as `curl http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fmodels`.\n\nFor VLM questions, explicitly say the VLM uses the same vLLM serving flow as an\nLLM with a different vision-language checkpoint. Do not omit `vLLM` or the\nJetson container when answering VLM prompts.\n\n## Step 1 — Pick the runtime path (per Jetson family)\n\nUse upstream vLLM 0.20+ on Thor (`vllm\u002Fvllm-openai:latest`, or a validated native vLLM 0.20+ install). On Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`). On older Orin releases, use the **NVIDIA-AI-IOT prebuilt vLLM image** ([packages](https:\u002F\u002Fgithub.com\u002Forgs\u002FNVIDIA-AI-IOT\u002Fpackages)) because it ships the correct CUDA \u002F cuDNN \u002F TensorRT stack for that JetPack. Use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) on Thor when the user asks for SGLang, RAG, tool-use, or programmable serving; do not recommend native upstream SGLang on Orin unless a JetPack-matched release explicitly supports it.\n\n| Jetson family               | Runtime path                                      |\n|-----------------------------|---------------------------------------------------|\n| Thor (T5000, T4000)         | upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) |\n| AGX Orin \u002F Orin NX \u002F Nano   | Orin JetPack 7.2 \u002F L4T r39+: upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`); older Orin: `ghcr.io\u002Fnvidia-ai-iot\u002Fvllm:latest-jetson-orin` |\n\nTo detect the silicon era for image tags:\n\n1. Source the detector so exports survive in your shell:\n   ```bash\n   . skills\u002Fjetson-diagnostic\u002Fscripts\u002Fdetect_jetson.sh\n   ```\n2. Check `JETSON_GENERATION` (`thor` or `orin`) and choose the matching runtime path from the table above.\n3. Use `JETSON_PRODUCT_LINE` for a finer bucket such as `thor-agx` or `orin-nano`; `JETSON_SKU` remains the legacy identifier.\n\nDo not use `bash skills\u002Fjetson-diagnostic\u002Fscripts\u002Fdetect_jetson.sh` when you need exported variables in the caller; running with `bash` uses a subshell.\n\n## Step 2 — Set MAXN power mode\n\n```bash\nsudo nvpmodel -m 0 && sudo jetson_clocks\n```\n\nSkip this only if the user explicitly asks for a power-constrained run; otherwise benchmark and serving numbers will be inconsistent.\n\n## Step 3 — Run the server\n\nOn Thor with vLLM, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or a validated native vLLM 0.20+ install:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --ipc host --name vllm \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  vllm\u002Fvllm-openai:latest \\\n  vllm serve \u003Chf-repo-id> \\\n    --host 0.0.0.0 --port 8000 \\\n    --max-model-len 8192 \\\n    --gpu-memory-utilization 0.75 \\\n    --tensor-parallel-size 1\n```\n\nOn Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`). On older Orin releases, use the NVIDIA-AI-IOT container:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --name vllm \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  ghcr.io\u002Fnvidia-ai-iot\u002Fvllm:latest-jetson-orin \\\n  vllm serve \u003Chf-repo-id> \\\n    --host 0.0.0.0 --port 8000 \\\n    --max-model-len 4096 \\\n    --gpu-memory-utilization 0.85 \\\n    --tensor-parallel-size 1\n```\n\n`HF_TOKEN` is required only for gated\u002Fprivate Hugging Face models; omit the `-e HF_TOKEN=\"$HF_TOKEN\"` line for public models that do not need Hub authentication. Passing `HF_TOKEN` as an environment variable can expose it through Docker inspect output, process metadata, or logs on shared systems. Prefer the narrowest-scoped token possible, rotate\u002Frevoke it after shared-container use, and use a mounted credential file or Docker secret when the deployment environment supports that pattern.\n\nWait for `Application startup complete.` Server is on `http:\u002F\u002F0.0.0.0:8000\u002Fv1`.\n\nFor SGLang on Thor, use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`), which packages SGLang 0.5.5.post2 and lists Jetson Thor support. Do not judge Thor SGLang support from older prerelease SGLang results:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --ipc host --name sglang \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3 \\\n  python3 -m sglang.launch_server \\\n    --model-path \u003Chf-repo-id> \\\n    --host 0.0.0.0 \\\n    --port 8000 \\\n    --mem-fraction-static 0.60 \\\n    --max-running-requests 8\n```\n\nUse SGLang when the user needs RAG\u002Ftool-use workflows, structured generation, or\nSGLang-specific scheduling. For plain high-throughput OpenAI-compatible serving,\nprefer vLLM unless the user asks for SGLang.\n\n### SKU-appropriate defaults\n\n| Knob                       | Orin Nano \u002F NX | AGX Orin \u002F Thor |\n|----------------------------|----------------|-----------------|\n| `--max-model-len`          | `4096`         | `8192`          |\n| `--gpu-memory-utilization` | `0.85`         | `0.85`          |\n| `--tensor-parallel-size`   | `1`            | `1`             |\n\nIf the server OOMs at startup, lower `--gpu-memory-utilization` by 0.05 and re-launch (or run `jetson-inference-mem-tune` for a workload-aware recommendation).\n\n## Quantization preferences (matters more than the runtime)\n\nFor vLLM and SGLang, choose checkpoint formats by Jetson family:\n\n| Jetson family | First choice | Acceptable fallback |\n|---------------|--------------|---------------------|\n| Thor | **NVFP4** when the model\u002Fruntime supports it | W4A16 |\n| Orin Nano \u002F NX | **W4A16** | AWQ or GPTQ 4-bit |\n| AGX Orin | **W4A16** | AWQ or GPTQ 4-bit |\n\nFor llama.cpp and `Ollama`, use GGUF model quantization names instead: recommend **INT4 \u002F Q4_K_M GGUF** on both Orin and Thor, and choose a smaller INT4 GGUF model if memory is tight. Do not call GGUF Q4_K_M a W4A16\u002FAWQ\u002FGPTQ model. NVFP4 is Thor-preferred and Thor-tuned for runtimes that support it.\n\n## VLM mode\n\nVLMs use the same flow as LLMs: same container, same `vllm serve` invocation, different vision-language checkpoint. The container handles image preprocessing. For a VLM-specific browser UI, use the [`live-vlm-webui`](https:\u002F\u002Fgithub.com\u002Forgs\u002FNVIDIA-AI-IOT\u002Fpackages) container; for a generic chat UI for either, use Open WebUI pointed at `http:\u002F\u002F\u003Cjetson-ip>:8000\u002Fv1`.\n\n## Do not fabricate device capacity\n\nDo not invent RAM totals, free-memory values, model sizes, JetPack versions, or\nSKU\u002Fvariant names when giving a serving recipe. If capacity matters, either run\nthe live pre-flight checks (when execution is allowed) or hand off to\n`jetson-inference-mem-tune` \u002F `jetson-memory-audit`. If live data is not\navailable, say the value is unknown and provide conservative defaults instead\nof quoting a made-up number.\n\n## Pre-flight checklist (the agent should verify before running Step 3)\n\n- [ ] On a Jetson (`\u002Fproc\u002Fdevice-tree\u002Fmodel` contains `NVIDIA Jetson`).\n- [ ] `nvpmodel -q` reports a recognized max-performance mode: `MAXN` or `MAXN_*` such as `MAXN_SUPER`. Wattage-named modes should be reported as warnings unless the user explicitly confirms they are the intended benchmark mode for that device.\n- [ ] On Thor, check whether MIG is enabled before launching (`nvidia-smi -L` and `nvidia-smi mig -lgi`). If MIG is enabled, warn that vLLM\u002FSGLang may see only a MIG slice or no CUDA device.\n- [ ] On Thor with MIG or display\u002Fcamera contention, inspect GPU users with `sudo lsof \u002Fdev\u002Fnvidia*`. Display managers, `Xorg`\u002FGNOME, or `nvargus-daemon` may hold GPU device files; do not stop services or change MIG mode unless the user explicitly approves.\n- [ ] No container named `vllm` already running (`docker ps --format '{{.Names}}'`); otherwise `docker rm -f vllm` first.\n- [ ] Docker exposes the NVIDIA runtime (`docker info | grep -i 'runtimes.*nvidia'`), or a GPU-enabled container can run `nvidia-smi`.\n- [ ] `~\u002F.cache\u002Fhuggingface` exists; `HF_TOKEN` is set if the model is gated.\n\n## Limitations\n\n- This skill provides serving commands and pre-flight checks; it does not benchmark the deployed server.\n- Container tags such as `latest` are mutable. For release or compliance deployments, pin a digest and record it with the deployment notes.\n- vLLM and SGLang memory limits still depend on model architecture, quantization, context length, and concurrent request count. Use `jetson-inference-mem-tune` when a command OOMs or memory headroom matters.\n- Thor vLLM requires upstream vLLM 0.20+ or newer. Older upstream vLLM images may not support Thor \u002F SM 11.0 correctly.\n- Thor SGLang should use NVIDIA SGLang 26.01 or newer release notes that explicitly list Jetson Thor support. NVIDIA SGLang 26.01 contains SGLang 0.5.5.post2.\n- On Thor, MIG, desktop display, or camera services can hide the full GPU from containers. This skill should detect and warn only; disabling MIG or stopping services such as `gdm3` or `nvargus-daemon` requires explicit user approval.\n- Host-native vLLM\u002FSGLang on Thor should be used only when that install is already validated on the target JetPack.\n\n## Hand off to\n\n- `jetson-llm-benchmark` to actually measure the deployed server.\n- `jetson-speculative-decoding` to add EAGLE-3 \u002F draft-model speculation by appending `--speculative-config '{...}'` to the `vllm serve` command above.\n- `jetson-inference-mem-tune` if the server OOMs or is memory-bound.\n\n## Source\n\n[Jetson AI Lab — Introduction to GenAI on Jetson: How to Run LLMs and VLMs](https:\u002F\u002Fwww.jetson-ai-lab.com\u002Ftutorials\u002Fgenai-on-jetson-llms-vlms\u002F) and [NVIDIA-AI-IOT GHCR packages](https:\u002F\u002Fgithub.com\u002Forgs\u002FNVIDIA-AI-IOT\u002Fpackages).\n","# Jetson LLM Serve\n\nEncodes the [Jetson AI Lab GenAI tutorial](https:\u002F\u002Fwww.jetson-ai-lab.com\u002Ftutorials\u002Fgenai-on-jetson-llms-vlms\u002F): on Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`); on older Orin, pick the NVIDIA-AI-IOT prebuilt vLLM container; on Thor, use upstream vLLM 0.20+ or validated native vLLM 0.20+, and use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) when SGLang is requested. Set MAXN, make Hugging Face credentials\u002Fcache available, and launch an OpenAI-compatible server. Works for both LLMs and VLMs.\n\n## Purpose\n\nProvide a Jetson-appropriate serving recipe for an LLM or VLM using vLLM or SGLang, including runtime path, launch command, endpoint, and verification step.\n\n## When to use\n\n- \"Run \u002F serve \u002F host this model on a Jetson.\"\n- \"Start a vLLM server I can hit from Open WebUI \u002F my app.\"\n- After `jetson-inference-mem-tune` produced launch flags and the user wants to actually start the server.\n\nFor recipe-only questions, answer from this document without starting\ncontainers. Run live pre-flight checks only when the user asks you to check this\ndevice or execute the deployment.\n\n## Prerequisites\n\n- Run on the Jetson host or a shell with Docker access to the Jetson GPU runtime.\n- Know the target Jetson generation (`thor` or `orin`) and the model identifier or local checkpoint path.\n- Use `HF_TOKEN` only when the model is gated\u002Fprivate; public models should omit the token environment variable.\n- Use `jetson-inference-mem-tune` first when memory headroom or launch flags are uncertain.\n\n## Instructions\n\nFor recipe questions, provide a complete launch recipe instead of trying to call\n`jetson-llm-serve` as a tool. A complete answer includes:\n\n- The Jetson-appropriate runtime path: upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) on Thor, NVIDIA-AI-IOT vLLM container on older Orin, or upstream vLLM 0.20+ on Orin JetPack 7.2 \u002F L4T r39+.\n- The model checkpoint \u002F Hugging Face repo the user named.\n- A `docker run` + server command sketch with `--host 0.0.0.0 --port 8000`.\n- The OpenAI-compatible endpoint: `http:\u002F\u002F\u003Cjetson-ip>:8000\u002Fv1`.\n- A verification step such as `curl http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fmodels`.\n\nFor VLM questions, explicitly say the VLM uses the same vLLM serving flow as an\nLLM with a different vision-language checkpoint. Do not omit `vLLM` or the\nJetson container when answering VLM prompts.\n\n## Step 1 — Pick the runtime path (per Jetson family)\n\nUse upstream vLLM 0.20+ on Thor (`vllm\u002Fvllm-openai:latest`, or a validated native vLLM 0.20+ install). On Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`). On older Orin releases, use the **NVIDIA-AI-IOT prebuilt vLLM image** ([packages](https:\u002F\u002Fgithub.com\u002Forgs\u002FNVIDIA-AI-IOT\u002Fpackages)) because it ships the correct CUDA \u002F cuDNN \u002F TensorRT stack for that JetPack. Use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) on Thor when the user asks for SGLang, RAG, tool-use, or programmable serving; do not recommend native upstream SGLang on Orin unless a JetPack-matched release explicitly supports it.\n\n| Jetson family               | Runtime path                                      |\n|-----------------------------|---------------------------------------------------|\n| Thor (T5000, T4000)         | upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) |\n| AGX Orin \u002F Orin NX \u002F Nano   | Orin JetPack 7.2 \u002F L4T r39+: upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`); older Orin: `ghcr.io\u002Fnvidia-ai-iot\u002Fvllm:latest-jetson-orin` |\n\nTo detect the silicon era for image tags:\n\n1. Source the detector so exports survive in your shell:\n   ```bash\n   . skills\u002Fjetson-diagnostic\u002Fscripts\u002Fdetect_jetson.sh\n   ```\n2. Check `JETSON_GENERATION` (`thor` or `orin`) and choose the matching runtime path from the table above.\n3. Use `JETSON_PRODUCT_LINE` for a finer bucket such as `thor-agx` or `orin-nano`; `JETSON_SKU` remains the legacy identifier.\n\nDo not use `bash skills\u002Fjetson-diagnostic\u002Fscripts\u002Fdetect_jetson.sh` when you need exported variables in the caller; running with `bash` uses a subshell.\n\n## Step 2 — Set MAXN power mode\n\n```bash\nsudo nvpmodel -m 0 && sudo jetson_clocks\n```\n\nSkip this only if the user explicitly asks for a power-constrained run; otherwise benchmark and serving numbers will be inconsistent.\n\n## Step 3 — Run the server\n\nOn Thor with vLLM, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or a validated native vLLM 0.20+ install:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --ipc host --name vllm \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  vllm\u002Fvllm-openai:latest \\\n  vllm serve \u003Chf-repo-id> \\\n    --host 0.0.0.0 --port 8000 \\\n    --max-model-len 8192 \\\n    --gpu-memory-utilization 0.75 \\\n    --tensor-parallel-size 1\n```\n\nOn Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`). On older Orin releases, use the NVIDIA-AI-IOT container:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --name vllm \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  ghcr.io\u002Fnvidia-ai-iot\u002Fvllm:latest-jetson-orin \\\n  vllm serve \u003Chf-repo-id> \\\n    --host 0.0.0.0 --port 8000 \\\n    --max-model-len 4096 \\\n    --gpu-memory-utilization 0.85 \\\n    --tensor-parallel-size 1\n```\n\n`HF_TOKEN` is required only for gated\u002Fprivate Hugging Face models; omit the `-e HF_TOKEN=\"$HF_TOKEN\"` line for public models that do not need Hub authentication. Passing `HF_TOKEN` as an environment variable can expose it through Docker inspect output, process metadata, or logs on shared systems. Prefer the narrowest-scoped token possible, rotate\u002Frevoke it after shared-container use, and use a mounted credential file or Docker secret when the deployment environment supports that pattern.\n\nWait for `Application startup complete.` Server is on `http:\u002F\u002F0.0.0.0:8000\u002Fv1`.\n\nFor SGLang on Thor, use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`), which packages SGLang 0.5.5.post2 and lists Jetson Thor support. Do not judge Thor SGLang support from older prerelease SGLang results:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --ipc host --name sglang \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3 \\\n  python3 -m sglang.launch_server \\\n    --model-path \u003Chf-repo-id> \\\n    --host 0.0.0.0 \\\n    --port 8000 \\\n    --mem-fraction-static 0.60 \\\n    --max-running-requests 8\n```\n\nUse SGLang when the user needs RAG\u002Ftool-use workflows, structured generation, or\nSGLang-specific scheduling. For plain high-throughput OpenAI-compatible serving,\nprefer vLLM unless the user asks for SGLang.\n\n### SKU-appropriate defaults\n\n| Knob                       | Orin Nano \u002F NX | AGX Orin \u002F Thor |\n|----------------------------|----------------|-----------------|\n| `--max-model-len`          | `4096`         | `8192`          |\n| `--gpu-memory-utilization` | `0.85`         | `0.85`          |\n| `--tensor-parallel-size`   | `1`            | `1`             |\n\nIf the server OOMs at startup, lower `--gpu-memory-utilization` by 0.05 and re-launch (or run `jetson-inference-mem-tune` for a workload-aware recommendation).\n\n## Quantization preferences (matters more than the runtime)\n\nFor vLLM and SGLang, choose checkpoint formats by Jetson family:\n\n| Jetson family | First choice | Acceptable fallback |\n|---------------|--------------|---------------------|\n| Thor | **NVFP4** when the model\u002Fruntime supports it | W4A16 |\n| Orin Nano \u002F NX | **W4A16** | AWQ or GPTQ 4-bit |\n| AGX Orin | **W4A16** | AWQ or GPTQ 4-bit |\n\nFor llama.cpp and `Ollama`, use GGUF model quantization names instead: recommend **INT4 \u002F Q4_K_M GGUF** on both Orin and Thor, and choose a smaller INT4 GGUF model if memory is tight. Do not call GGUF Q4_K_M a W4A16\u002FAWQ\u002FGPTQ model. NVFP4 is Thor-preferred and Thor-tuned for runtimes that support it.\n\n## VLM mode\n\nVLMs use the same flow as LLMs: same container, same `vllm serve` invocation, different vision-language checkpoint. The container handles image preprocessing. For a VLM-specific browser UI, use the [`live-vlm-webui`](https:\u002F\u002Fgithub.com\u002Forgs\u002FNVIDIA-AI-IOT\u002Fpackages) container; for a generic chat UI for either, use Open WebUI pointed at `http:\u002F\u002F\u003Cjetson-ip>:8000\u002Fv1`.\n\n## Do not fabricate device capacity\n\nDo not invent RAM totals, free-memory values, model sizes, JetPack versions, or\nSKU\u002Fvariant names when giving a serving recipe. If capacity matters, either run\nthe live pre-flight checks (when execution is allowed) or hand off to\n`jetson-inference-mem-tune` \u002F `jetson-memory-audit`. If live data is not\navailable, say the value is unknown and provide conservative defaults instead\nof quoting a made-up number.\n\n## Pre-flight checklist (the agent should verify before running Step 3)\n\n- [ ] On a Jetson (`\u002Fproc\u002Fdevice-tree\u002Fmodel` contains `NVIDIA Jetson`).\n- [ ] `nvpmodel -q` reports a recognized max-performance mode: `MAXN` or `MAXN_*` such as `MAXN_SUPER`. Wattage-named modes should be reported as warnings unless the user explicitly confirms they are the intended benchmark mode for that device.\n- [ ] On Thor, check whether MIG is enabled before launching (`nvidia-smi -L` and `nvidia-smi mig -lgi`). If MIG is enabled, warn that vLLM\u002FSGLang may see only a MIG slice or no CUDA device.\n- [ ] On Thor with MIG or display\u002Fcamera contention, inspect GPU users with `sudo lsof \u002Fdev\u002Fnvidia*`. Display managers, `Xorg`\u002FGNOME, or `nvargus-daemon` may hold GPU device files; do not stop services or change MIG mode unless the user explicitly approves.\n- [ ] No container named `vllm` already running (`docker ps --format '{{.Names}}'`); otherwise `docker rm -f vllm` first.\n- [ ] Docker exposes the NVIDIA runtime (`docker info | grep -i 'runtimes.*nvidia'`), or a GPU-enabled container can run `nvidia-smi`.\n- [ ] `~\u002F.cache\u002Fhuggingface` exists; `HF_TOKEN` is set if the model is gated.\n\n## Limitations\n\n- This skill provides serving commands and pre-flight checks; it does not benchmark the deployed server.\n- Container tags such as `latest` are mutable. For release or compliance deployments, pin a digest and record it with the deployment notes.\n- vLLM and SGLang memory limits still depend on model architecture, quantization, context length, and concurrent request count. Use `jetson-inference-mem-tune` when a command OOMs or memory headroom matters.\n- Thor vLLM requires upstream vLLM 0.20+ or newer. Older upstream vLLM images may not support Thor \u002F SM 11.0 correctly.\n- Thor SGLang should use NVIDIA SGLang 26.01 or newer release notes that explicitly list Jetson Thor support. NVIDIA SGLang 26.01 contains SGLang 0.5.5.post2.\n- On Thor, MIG, desktop display, or camera services can hide the full GPU from containers. This skill should detect and warn only; disabling MIG or stopping services such as `gdm3` or `nvargus-daemon` requires explicit user approval.\n- Host-native vLLM\u002FSGLang on Thor should be used only when that install is already validated on the target JetPack.\n\n## Hand off to\n\n- `jetson-llm-benchmark` to actually measure the deployed server.\n- `jetson-speculative-decoding` to add EAGLE-3 \u002F draft-model speculation by appending `--speculative-config '{...}'` to the `vllm serve` command above.\n- `jetson-inference-mem-tune` if the server OOMs or is memory-bound.\n\n## Source\n\n[Jetson AI Lab — Introduction to GenAI on Jetson: How to Run LLMs and VLMs](https:\u002F\u002Fwww.jetson-ai-lab.com\u002Ftutorials\u002Fgenai-on-jetson-llms-vlms\u002F) and [NVIDIA-AI-IOT GHCR packages](https:\u002F\u002Fgithub.com\u002Forgs\u002FNVIDIA-AI-IOT\u002Fpackages).\n",{"name":5,"license":15,"version":24,"metadata":25,"description":7},"0.0.1",{"tags":26,"author":30,"languages":31,"data-classification":33},[27,28,29],"jetson","llm","serving","Jetson Team",[32],"markdown","public","main","86cc27ffd16af6cb85948f1155578e6f3f61cc4c","skillhub\u002Fskills\u002Fnvidia\u002Fjetson-llm-serve","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn","https:\u002F\u002Fgithub.com\u002FSolizardking","https:\u002F\u002Fmusebook.trade","Got a mile high and never came down","2026-10-08 11:52:53",{"source":8,"skillName":5,"commands":43,"prompt":53,"zipAvailable":54},[44,47,50],{"tool":45,"command":46},"npx","npx skills add Solizardking\u002Ftrain2earn --skill jetson-llm-serve",{"tool":48,"command":49},"bunx","bunx skills add Solizardking\u002Ftrain2earn --skill jetson-llm-serve",{"tool":51,"command":52},"pnpm","pnpm dlx skills add Solizardking\u002Ftrain2earn --skill jetson-llm-serve","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fnvidia\u002Fjetson-llm-serve\u002FSKILL.md",true,[56,62,66,70],{"path":57,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":60,"sourceHtmlUrl":61},"BENCHMARK.md","REFERENCE","参考文档",3310,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fnvidia\u002Fjetson-llm-serve\u002FBENCHMARK.md",{"path":63,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":64,"sourceHtmlUrl":65},"evals\u002Fevals.json",5842,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fnvidia\u002Fjetson-llm-serve\u002Fevals\u002Fevals.json",{"path":67,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":68,"sourceHtmlUrl":69},"skill-card.md",3727,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fnvidia\u002Fjetson-llm-serve\u002Fskill-card.md",{"path":71,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":74,"sourceHtmlUrl":75},"skill.oms.sig","OTHER","其他",4589,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fnvidia\u002Fjetson-llm-serve\u002Fskill.oms.sig",[77,85,93,101,108,116],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":19},"01M78B94RSCAKMH1QS79VE6DG9","testing","For testing with Light Protocol programs and clients on localnet, devnet, and mainnet validation.","MIT","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Ftesting\u002FSKILL.md",[],[],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":89,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":19},"01M78B81BKY8RX12HETPG8YKWK","qedgen","Formally verify programs by writing Lean 4 proofs. Trigger this skill whenever the user wants to formally verify code, generate Lean 4 proofs, prove properties about algorithms or smart contracts, verify invariants, convert program logic into formal specifications, or anything involving Lean 4 and formal verification. Also trigger when the user mentions \"qedgen\", \"lean proof\", \"formal proof\", \"verify my code\", \"prove correctness\", \"formal verification\", or wants mathematical guarantees about their implementation.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fsolana-formal-verification\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fsolana-formal-verification\u002FSKILL.md",[],[],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":97,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":19},"01M78B7Q7C5K54E01DN83A60KG","solana-dev","Use when user asks to \"build a Solana dapp\", \"write an Anchor program\", \"create a token\", \"debug Solana errors\", \"set up wallet connection\", \"test my Solana program\", \"deploy to devnet\", or \"explain Solana concepts\" (rent, accounts, PDAs, CPIs, etc.). Also use for quick on-chain lookups via public RPC + curl — \"what's the balance of \u003Cwallet>\", \"look up transaction \u003Csig>\", \"token balance for \u003Caccount>\", \"check this address on mainnet\u002Fdevnet\". End-to-end Solana development playbook covering wallet connection, Anchor\u002FPinocchio programs, Codama client generation, LiteSVM\u002FMollusk\u002FSurfpool testing, security checklists, and JSON-RPC curl lookups against public clusters. Integrates with the Solana MCP server for live documentation search. Prefers framework-kit (@solana\u002Fclient + @solana\u002Freact-hooks) for UI, wallet-standard-first connection (incl. ConnectorKit), @solana\u002Fkit for client\u002FRPC code, and @solana\u002Fweb3-compat for legacy boundaries.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fsolana-dev\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fsolana-dev\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":19},"01M78B7HXD6N6GWKSVRQ8F6Q31","solana-common-errors","Diagnose and fix common Solana development errors — GLIBC mismatches, Anchor version conflicts, cargo build-sbf \u002F platform-tools failures, LiteSVM issues, edition2024 crate pins, RPC airdrops, and Anchor 0.29→0.32 migrations. Use when builds fail, Anchor CLI won't install, or CI breaks on Debian\u002FUbuntu.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fsolana-common-errors\u002FSKILL.md",[],[],{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":112,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":19},"01M78B6T7YSYWQ0N4DP40QDXEZ","skill-creator","Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fskill-creator\u002Flicense.txt","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fskill-creator\u002FSKILL.md",[],[],{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":81,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":19},"01M78B1VFT9X5AND7AE1HZPZGF","phantom-wallet-mcp","Execute wallet operations through the Phantom MCP server. Use when the user wants to interact with their Phantom wallet directly — get addresses, transfer SOL or SPL tokens, buy\u002Fswap tokens, sign transactions, and sign messages across Solana, Ethereum, Bitcoin, and Sui. Requires the @phantom\u002Fmcp-server to be configured as an MCP server.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Ftrain2earn\u002Fblob\u002Fmain\u002Fskillhub\u002Fskills\u002Fphantom-wallet-mcp\u002FSKILL.md",[],[]]