[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7CPNX5Y7YK1RAM8K4TJX37D":3,"skill-same:01M7CPNX5Y7YK1RAM8K4TJX37D":56},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":34,"commitSha":35,"skillDirectory":36,"forksCount":37,"repositoryHtmlUrl":38,"repositoryHomepageUrl":6,"ownerHtmlUrl":39,"ownerBlogUrl":6,"ownerDescription":40,"updatedAt":41,"install":42,"resources":55},"01M7CPNX5Y7YK1RAM8K4TJX37D","jetson-speculative-decoding",null,"Add EAGLE-3 or draft-model speculative decoding to a Jetson vLLM server when TPOT is the bottleneck.","om-scogo\u002Fskillsh-scraper","om-scogo","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160466840?v=4","01M79AWB6DFVCQ7RK70CS34B96","Om Achrekar","01M79AWB6ZM9SKR8KDH1WSSM22",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fnvidia-skills\u002Fjetson-speculative-decoding\u002Fskills\u002Fjetson-speculative-decoding\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: jetson-speculative-decoding\ndescription: Add EAGLE-3 or draft-model speculative decoding to a Jetson vLLM server when TPOT is the bottleneck.\nversion: 0.0.1\nlicense: \"Apache-2.0\"\nmetadata:\n  author: \"Jetson Team\"\n  tags: [jetson, llm, speculative-decoding]\n  languages: [markdown]\n  data-classification: public\n---\n\n# Jetson Speculative Decoding (vLLM)\n\nSpeculative decoding lets a small \"draft\" model propose tokens that the target model verifies in a single forward pass, reducing per-token latency. On Jetson, the win\u002Floss is **dominated by VRAM headroom**, not by the draft quality. This skill encodes the parts an LLM won't already know.\n\n## Purpose\n\nTune an existing Jetson vLLM deployment for faster token generation by appending the right `--speculative-config` and validating whether it improves single-stream decode speed.\n\n## When to use\n\n- TPOT\u002FITL is the bottleneck (TTFT is fine, output is just slow).\n- Workload is single-stream or low-concurrency (≤2). Speculation usually loses at high concurrency.\n- Jetson family is **Thor or AGX Orin**. Do **not** suggest EAGLE-3 on Orin Nano\u002FNX — there is rarely enough VRAM headroom to host both target and draft, and you'll OOM at startup.\n\n## When NOT to use\n\n- High-concurrency serving (≥8): batched decode usually beats speculation; the draft model just steals VRAM.\n- Models without a published EAGLE-3 head — do not train one ad-hoc as a \"fix\".\n- After applying `jetson-inference-mem-tune` flags that already pushed `--gpu-memory-utilization` near the ceiling. Free at least ~2 GB first.\n\n## Prerequisites\n\n- A working vLLM server recipe from `jetson-llm-serve`.\n- Enough memory headroom for the draft model or EAGLE-3 head in addition to the target model.\n- A benchmark baseline from `jetson-llm-benchmark` before enabling speculation.\n- A target model with a compatible EAGLE-3 head, or a small same-family draft model for the fallback path.\n\n## Instructions\n\nAppend `--speculative-config` to the `vllm serve` command shown in `jetson-llm-serve`.\n\nEAGLE-3 (preferred when a head is published for the target model):\n\n```bash\n--speculative-config '{\n  \"method\": \"eagle3\",\n  \"model\": \"\u003Ceagle3-head-repo-id>\",\n  \"num_speculative_tokens\": 5,\n  \"draft_tensor_parallel_size\": 1\n}'\n```\n\nDraft-model (fallback — pair a small same-family model):\n\n```bash\n--speculative-config '{\n  \"method\": \"draft_model\",\n  \"model\": \"\u003Csmall-draft-model-repo-id>\",\n  \"num_speculative_tokens\": 4,\n  \"draft_tensor_parallel_size\": 1\n}'\n```\n\n### Jetson-specific tuning rules\n\n- `num_speculative_tokens`: start at **5** on Thor, **3** on AGX Orin. Higher values pay off only if the draft acceptance rate is >0.6.\n- Always pair with the same vLLM runtime path used by `jetson-llm-serve`: upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or validated native vLLM 0.20+ on Thor, upstream vLLM 0.20+ on Orin JetPack 7.2 \u002F L4T r39+, or the NVIDIA-AI-IOT vLLM image on older Orin. Do not use an Orin NVIDIA-AI-IOT vLLM image on Thor. Older runtimes may lack EAGLE-3 or the current `--speculative-config` shape.\n- Drop `--gpu-memory-utilization` by ~0.05 vs the non-speculative baseline to give the draft model headroom.\n\n## How to verify it actually helped\n\n1. Run `jetson-llm-benchmark` (vLLM path) at `--concurrency 1` **before and after** enabling speculation.\n2. Acceptance: target ≥30% improvement in `throughput_tok_s` and ≥20% drop in `tpot_ms_p50` at concurrency 1.\n3. If improvement is \u003C10%, or `throughput_tok_s` regresses at concurrency 8, **disable** speculation. The draft model is costing more than it returns.\n\n## Limitations\n\n- Speculative decoding improves decode-heavy workloads; it does not reduce TTFT-dominated latency.\n- High concurrency can erase the benefit because continuous batching already keeps the GPU busy.\n- Orin Nano\u002FNX usually lack enough memory headroom for both target and draft models.\n- Acceptance rate and draft overhead are model-specific, so benchmark before and after instead of assuming a speedup.\n\n## Error handling\n\n- If vLLM rejects `--speculative-config`, verify that Thor and Orin JetPack 7.2 \u002F L4T r39+ are using vLLM 0.20+ and that older Orin is using a JetPack-matched NVIDIA-AI-IOT vLLM image; then switch back to the non-speculative serving command if the runtime still rejects it.\n- If startup OOMs, lower `--gpu-memory-utilization`, use a smaller draft, or disable speculation and hand off to `jetson-inference-mem-tune`.\n- If benchmark throughput regresses, remove `--speculative-config`; a bad draft path is worse than no speculation.\n\n## Hand off to\n\n- `jetson-llm-benchmark` to quantify the change.\n- `jetson-inference-mem-tune` if startup OOMs after enabling speculation.\n\n## Source\n\nvLLM speculative decoding docs and the [Jetson AI Lab GenAI tutorial](https:\u002F\u002Fwww.jetson-ai-lab.com\u002Ftutorials\u002Fgenai-on-jetson-llms-vlms\u002F).\n","# Jetson Speculative Decoding (vLLM)\n\nSpeculative decoding lets a small \"draft\" model propose tokens that the target model verifies in a single forward pass, reducing per-token latency. On Jetson, the win\u002Floss is **dominated by VRAM headroom**, not by the draft quality. This skill encodes the parts an LLM won't already know.\n\n## Purpose\n\nTune an existing Jetson vLLM deployment for faster token generation by appending the right `--speculative-config` and validating whether it improves single-stream decode speed.\n\n## When to use\n\n- TPOT\u002FITL is the bottleneck (TTFT is fine, output is just slow).\n- Workload is single-stream or low-concurrency (≤2). Speculation usually loses at high concurrency.\n- Jetson family is **Thor or AGX Orin**. Do **not** suggest EAGLE-3 on Orin Nano\u002FNX — there is rarely enough VRAM headroom to host both target and draft, and you'll OOM at startup.\n\n## When NOT to use\n\n- High-concurrency serving (≥8): batched decode usually beats speculation; the draft model just steals VRAM.\n- Models without a published EAGLE-3 head — do not train one ad-hoc as a \"fix\".\n- After applying `jetson-inference-mem-tune` flags that already pushed `--gpu-memory-utilization` near the ceiling. Free at least ~2 GB first.\n\n## Prerequisites\n\n- A working vLLM server recipe from `jetson-llm-serve`.\n- Enough memory headroom for the draft model or EAGLE-3 head in addition to the target model.\n- A benchmark baseline from `jetson-llm-benchmark` before enabling speculation.\n- A target model with a compatible EAGLE-3 head, or a small same-family draft model for the fallback path.\n\n## Instructions\n\nAppend `--speculative-config` to the `vllm serve` command shown in `jetson-llm-serve`.\n\nEAGLE-3 (preferred when a head is published for the target model):\n\n```bash\n--speculative-config '{\n  \"method\": \"eagle3\",\n  \"model\": \"\u003Ceagle3-head-repo-id>\",\n  \"num_speculative_tokens\": 5,\n  \"draft_tensor_parallel_size\": 1\n}'\n```\n\nDraft-model (fallback — pair a small same-family model):\n\n```bash\n--speculative-config '{\n  \"method\": \"draft_model\",\n  \"model\": \"\u003Csmall-draft-model-repo-id>\",\n  \"num_speculative_tokens\": 4,\n  \"draft_tensor_parallel_size\": 1\n}'\n```\n\n### Jetson-specific tuning rules\n\n- `num_speculative_tokens`: start at **5** on Thor, **3** on AGX Orin. Higher values pay off only if the draft acceptance rate is >0.6.\n- Always pair with the same vLLM runtime path used by `jetson-llm-serve`: upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or validated native vLLM 0.20+ on Thor, upstream vLLM 0.20+ on Orin JetPack 7.2 \u002F L4T r39+, or the NVIDIA-AI-IOT vLLM image on older Orin. Do not use an Orin NVIDIA-AI-IOT vLLM image on Thor. Older runtimes may lack EAGLE-3 or the current `--speculative-config` shape.\n- Drop `--gpu-memory-utilization` by ~0.05 vs the non-speculative baseline to give the draft model headroom.\n\n## How to verify it actually helped\n\n1. Run `jetson-llm-benchmark` (vLLM path) at `--concurrency 1` **before and after** enabling speculation.\n2. Acceptance: target ≥30% improvement in `throughput_tok_s` and ≥20% drop in `tpot_ms_p50` at concurrency 1.\n3. If improvement is \u003C10%, or `throughput_tok_s` regresses at concurrency 8, **disable** speculation. The draft model is costing more than it returns.\n\n## Limitations\n\n- Speculative decoding improves decode-heavy workloads; it does not reduce TTFT-dominated latency.\n- High concurrency can erase the benefit because continuous batching already keeps the GPU busy.\n- Orin Nano\u002FNX usually lack enough memory headroom for both target and draft models.\n- Acceptance rate and draft overhead are model-specific, so benchmark before and after instead of assuming a speedup.\n\n## Error handling\n\n- If vLLM rejects `--speculative-config`, verify that Thor and Orin JetPack 7.2 \u002F L4T r39+ are using vLLM 0.20+ and that older Orin is using a JetPack-matched NVIDIA-AI-IOT vLLM image; then switch back to the non-speculative serving command if the runtime still rejects it.\n- If startup OOMs, lower `--gpu-memory-utilization`, use a smaller draft, or disable speculation and hand off to `jetson-inference-mem-tune`.\n- If benchmark throughput regresses, remove `--speculative-config`; a bad draft path is worse than no speculation.\n\n## Hand off to\n\n- `jetson-llm-benchmark` to quantify the change.\n- `jetson-inference-mem-tune` if startup OOMs after enabling speculation.\n\n## Source\n\nvLLM speculative decoding docs and the [Jetson AI Lab GenAI tutorial](https:\u002F\u002Fwww.jetson-ai-lab.com\u002Ftutorials\u002Fgenai-on-jetson-llms-vlms\u002F).\n",{"name":5,"license":15,"version":24,"metadata":25,"description":7},"0.0.1",{"tags":26,"author":30,"languages":31,"data-classification":33},[27,28,29],"jetson","llm","speculative-decoding","Jetson Team",[32],"markdown","public","main","8859218955d9504cd18b59e7892100718039d3b4","data\u002Fskills\u002Fnvidia-skills\u002Fjetson-speculative-decoding\u002Fskills\u002Fjetson-speculative-decoding",1,"https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper","https:\u002F\u002Fgithub.com\u002Fom-scogo","SDE-1 @ Scogo ","2026-07-18 12:22:46",{"source":8,"skillName":5,"commands":43,"prompt":53,"zipAvailable":54},[44,47,50],{"tool":45,"command":46},"npx","npx skills add om-scogo\u002Fskillsh-scraper --skill jetson-speculative-decoding",{"tool":48,"command":49},"bunx","bunx skills add om-scogo\u002Fskillsh-scraper --skill jetson-speculative-decoding",{"tool":51,"command":52},"pnpm","pnpm dlx skills add om-scogo\u002Fskillsh-scraper --skill jetson-speculative-decoding","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fnvidia-skills\u002Fjetson-speculative-decoding\u002Fskills\u002Fjetson-speculative-decoding\u002FSKILL.md",true,[],[57,65,70,77,82,89],{"skillNo":58,"name":59,"nameZh":6,"description":60,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":61,"licenseFileUrl":6,"sourceHtmlUrl":62,"categories":63,"keywords":64,"publishedAt":19},"01M7D6MC9FAAEV4CMNYX8W7JTG","arxiv-search","Search arXiv physics, math, and computer science preprints using natural language queries. Powered by Valyu semantic search.","MIT","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fyorkeccak-scientific-skills\u002Farxiv-search\u002Fskills\u002Farxiv-search\u002FSKILL.md",[],[],{"skillNo":66,"name":59,"nameZh":6,"description":60,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":61,"licenseFileUrl":6,"sourceHtmlUrl":67,"categories":68,"keywords":69,"publishedAt":19},"01M7D6M85CKKN87TA98VCQV4H7","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fyorkeccak-scientific-skills\u002Farxiv-search\u002FSkill.md",[],[],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":61,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M7D4ZTCS4DR3CE4JY06X84T2","working-with-legacy-code","Safely change and test untested codebases using Feathers' \"Working Effectively with Legacy Code\". Use when the user mentions \"legacy code\", \"no tests\", \"untested codebase\", \"how do I test this\", \"seams\", \"characterization tests\", \"golden master\", \"sprout method\", \"afraid to change this code\", \"monster method\", \"dependency breaking\", or \"inherited a messy codebase\". Also trigger when changing code without tests safely, getting a class under test when constructors, statics, or singletons block it, adding features to tangled modules, or planning incremental test coverage for an old codebase. Covers the legacy-code change algorithm, seams, characterization tests, sprout\u002Fwrap, and dependency-breaking techniques. For refactoring code that already has tests, see refactoring-patterns. For day-to-day code quality, see clean-code.","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fwondelai-skills\u002Fworking-with-legacy-code\u002Fworking-with-legacy-code\u002FSKILL.md",[],[],{"skillNo":78,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":61,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":81,"publishedAt":19},"01M7D4ZHZKJD2RSNGAF63V14P4","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fwondelai-skills\u002Fworking-with-legacy-code\u002FSkill.md",[],[],{"skillNo":83,"name":84,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":61,"licenseFileUrl":6,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":19},"01M7D4ZB4CJBFW5JE67CPH2QRV","web-typography","Select, pair, and implement typefaces for web projects. Use when the user mentions \"font pairing\", \"which typeface\", \"line height\", \"responsive typography\", \"web font loading\", \"type hierarchy\", \"variable fonts\", \"FOUT\u002FFOIT\", \"typographic scale\", or \"the text is hard to read\". Also trigger when choosing between system fonts and web fonts, optimizing font-loading performance, or designing readable long-form content. Covers readability evaluation, CSS implementation, and performance optimization. For overall UI design systems, see refactoring-ui. For dramatic typographic experiences, see top-design.","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fwondelai-skills\u002Fweb-typography\u002Fweb-typography\u002FSKILL.md",[],[],{"skillNo":90,"name":84,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":61,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":19},"01M7D4Z4MVYVY3NMJ18BNV6WNA","https:\u002F\u002Fgithub.com\u002Fom-scogo\u002Fskillsh-scraper\u002Fblob\u002Fmain\u002Fdata\u002Fskills\u002Fwondelai-skills\u002Fweb-typography\u002FSkill.md",[],[]]