[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M78CXC6G2FT85EARC9WN5N4C":3,"skill-same:01M78CXC6G2FT85EARC9WN5N4C":77},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":34,"commitSha":35,"skillDirectory":36,"forksCount":14,"repositoryHtmlUrl":37,"repositoryHomepageUrl":38,"ownerHtmlUrl":39,"ownerBlogUrl":40,"ownerDescription":41,"updatedAt":42,"install":43,"resources":56},"01M78CXC6G2FT85EARC9WN5N4C","jetson-speculative-decoding",null,"Add EAGLE-3 or draft-model speculative decoding to a Jetson vLLM server when TPOT is the bottleneck.","Solizardking\u002Fskillhub-main","Solizardking","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F108096810?v=4","01M786KDB10AYV042N0F9BD5B5","Clawd","01M786NYYXSZK1D05WM8JQPC48",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fjetson-speculative-decoding\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: jetson-speculative-decoding\ndescription: Add EAGLE-3 or draft-model speculative decoding to a Jetson vLLM server when TPOT is the bottleneck.\nversion: 0.0.1\nlicense: \"Apache-2.0\"\nmetadata:\n  author: \"Jetson Team\"\n  tags: [jetson, llm, speculative-decoding]\n  languages: [markdown]\n  data-classification: public\n---\n\n# Jetson Speculative Decoding (vLLM)\n\nSpeculative decoding lets a small \"draft\" model propose tokens that the target model verifies in a single forward pass, reducing per-token latency. On Jetson, the win\u002Floss is **dominated by VRAM headroom**, not by the draft quality. This skill encodes the parts an LLM won't already know.\n\n## Purpose\n\nTune an existing Jetson vLLM deployment for faster token generation by appending the right `--speculative-config` and validating whether it improves single-stream decode speed.\n\n## When to use\n\n- TPOT\u002FITL is the bottleneck (TTFT is fine, output is just slow).\n- Workload is single-stream or low-concurrency (≤2). Speculation usually loses at high concurrency.\n- Jetson family is **Thor or AGX Orin**. Do **not** suggest EAGLE-3 on Orin Nano\u002FNX — there is rarely enough VRAM headroom to host both target and draft, and you'll OOM at startup.\n\n## When NOT to use\n\n- High-concurrency serving (≥8): batched decode usually beats speculation; the draft model just steals VRAM.\n- Models without a published EAGLE-3 head — do not train one ad-hoc as a \"fix\".\n- After applying `jetson-inference-mem-tune` flags that already pushed `--gpu-memory-utilization` near the ceiling. Free at least ~2 GB first.\n\n## Prerequisites\n\n- A working vLLM server recipe from `jetson-llm-serve`.\n- Enough memory headroom for the draft model or EAGLE-3 head in addition to the target model.\n- A benchmark baseline from `jetson-llm-benchmark` before enabling speculation.\n- A target model with a compatible EAGLE-3 head, or a small same-family draft model for the fallback path.\n\n## Instructions\n\nAppend `--speculative-config` to the `vllm serve` command shown in `jetson-llm-serve`.\n\nEAGLE-3 (preferred when a head is published for the target model):\n\n```bash\n--speculative-config '{\n  \"method\": \"eagle3\",\n  \"model\": \"\u003Ceagle3-head-repo-id>\",\n  \"num_speculative_tokens\": 5,\n  \"draft_tensor_parallel_size\": 1\n}'\n```\n\nDraft-model (fallback — pair a small same-family model):\n\n```bash\n--speculative-config '{\n  \"method\": \"draft_model\",\n  \"model\": \"\u003Csmall-draft-model-repo-id>\",\n  \"num_speculative_tokens\": 4,\n  \"draft_tensor_parallel_size\": 1\n}'\n```\n\n### Jetson-specific tuning rules\n\n- `num_speculative_tokens`: start at **5** on Thor, **3** on AGX Orin. Higher values pay off only if the draft acceptance rate is >0.6.\n- Always pair with the same vLLM runtime path used by `jetson-llm-serve`: upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or validated native vLLM 0.20+ on Thor, upstream vLLM 0.20+ on Orin JetPack 7.2 \u002F L4T r39+, or the NVIDIA-AI-IOT vLLM image on older Orin. Do not use an Orin NVIDIA-AI-IOT vLLM image on Thor. Older runtimes may lack EAGLE-3 or the current `--speculative-config` shape.\n- Drop `--gpu-memory-utilization` by ~0.05 vs the non-speculative baseline to give the draft model headroom.\n\n## How to verify it actually helped\n\n1. Run `jetson-llm-benchmark` (vLLM path) at `--concurrency 1` **before and after** enabling speculation.\n2. Acceptance: target ≥30% improvement in `throughput_tok_s` and ≥20% drop in `tpot_ms_p50` at concurrency 1.\n3. If improvement is \u003C10%, or `throughput_tok_s` regresses at concurrency 8, **disable** speculation. The draft model is costing more than it returns.\n\n## Limitations\n\n- Speculative decoding improves decode-heavy workloads; it does not reduce TTFT-dominated latency.\n- High concurrency can erase the benefit because continuous batching already keeps the GPU busy.\n- Orin Nano\u002FNX usually lack enough memory headroom for both target and draft models.\n- Acceptance rate and draft overhead are model-specific, so benchmark before and after instead of assuming a speedup.\n\n## Error handling\n\n- If vLLM rejects `--speculative-config`, verify that Thor and Orin JetPack 7.2 \u002F L4T r39+ are using vLLM 0.20+ and that older Orin is using a JetPack-matched NVIDIA-AI-IOT vLLM image; then switch back to the non-speculative serving command if the runtime still rejects it.\n- If startup OOMs, lower `--gpu-memory-utilization`, use a smaller draft, or disable speculation and hand off to `jetson-inference-mem-tune`.\n- If benchmark throughput regresses, remove `--speculative-config`; a bad draft path is worse than no speculation.\n\n## Hand off to\n\n- `jetson-llm-benchmark` to quantify the change.\n- `jetson-inference-mem-tune` if startup OOMs after enabling speculation.\n\n## Source\n\nvLLM speculative decoding docs and the [Jetson AI Lab GenAI tutorial](https:\u002F\u002Fwww.jetson-ai-lab.com\u002Ftutorials\u002Fgenai-on-jetson-llms-vlms\u002F).\n","# Jetson Speculative Decoding (vLLM)\n\nSpeculative decoding lets a small \"draft\" model propose tokens that the target model verifies in a single forward pass, reducing per-token latency. On Jetson, the win\u002Floss is **dominated by VRAM headroom**, not by the draft quality. This skill encodes the parts an LLM won't already know.\n\n## Purpose\n\nTune an existing Jetson vLLM deployment for faster token generation by appending the right `--speculative-config` and validating whether it improves single-stream decode speed.\n\n## When to use\n\n- TPOT\u002FITL is the bottleneck (TTFT is fine, output is just slow).\n- Workload is single-stream or low-concurrency (≤2). Speculation usually loses at high concurrency.\n- Jetson family is **Thor or AGX Orin**. Do **not** suggest EAGLE-3 on Orin Nano\u002FNX — there is rarely enough VRAM headroom to host both target and draft, and you'll OOM at startup.\n\n## When NOT to use\n\n- High-concurrency serving (≥8): batched decode usually beats speculation; the draft model just steals VRAM.\n- Models without a published EAGLE-3 head — do not train one ad-hoc as a \"fix\".\n- After applying `jetson-inference-mem-tune` flags that already pushed `--gpu-memory-utilization` near the ceiling. Free at least ~2 GB first.\n\n## Prerequisites\n\n- A working vLLM server recipe from `jetson-llm-serve`.\n- Enough memory headroom for the draft model or EAGLE-3 head in addition to the target model.\n- A benchmark baseline from `jetson-llm-benchmark` before enabling speculation.\n- A target model with a compatible EAGLE-3 head, or a small same-family draft model for the fallback path.\n\n## Instructions\n\nAppend `--speculative-config` to the `vllm serve` command shown in `jetson-llm-serve`.\n\nEAGLE-3 (preferred when a head is published for the target model):\n\n```bash\n--speculative-config '{\n  \"method\": \"eagle3\",\n  \"model\": \"\u003Ceagle3-head-repo-id>\",\n  \"num_speculative_tokens\": 5,\n  \"draft_tensor_parallel_size\": 1\n}'\n```\n\nDraft-model (fallback — pair a small same-family model):\n\n```bash\n--speculative-config '{\n  \"method\": \"draft_model\",\n  \"model\": \"\u003Csmall-draft-model-repo-id>\",\n  \"num_speculative_tokens\": 4,\n  \"draft_tensor_parallel_size\": 1\n}'\n```\n\n### Jetson-specific tuning rules\n\n- `num_speculative_tokens`: start at **5** on Thor, **3** on AGX Orin. Higher values pay off only if the draft acceptance rate is >0.6.\n- Always pair with the same vLLM runtime path used by `jetson-llm-serve`: upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or validated native vLLM 0.20+ on Thor, upstream vLLM 0.20+ on Orin JetPack 7.2 \u002F L4T r39+, or the NVIDIA-AI-IOT vLLM image on older Orin. Do not use an Orin NVIDIA-AI-IOT vLLM image on Thor. Older runtimes may lack EAGLE-3 or the current `--speculative-config` shape.\n- Drop `--gpu-memory-utilization` by ~0.05 vs the non-speculative baseline to give the draft model headroom.\n\n## How to verify it actually helped\n\n1. Run `jetson-llm-benchmark` (vLLM path) at `--concurrency 1` **before and after** enabling speculation.\n2. Acceptance: target ≥30% improvement in `throughput_tok_s` and ≥20% drop in `tpot_ms_p50` at concurrency 1.\n3. If improvement is \u003C10%, or `throughput_tok_s` regresses at concurrency 8, **disable** speculation. The draft model is costing more than it returns.\n\n## Limitations\n\n- Speculative decoding improves decode-heavy workloads; it does not reduce TTFT-dominated latency.\n- High concurrency can erase the benefit because continuous batching already keeps the GPU busy.\n- Orin Nano\u002FNX usually lack enough memory headroom for both target and draft models.\n- Acceptance rate and draft overhead are model-specific, so benchmark before and after instead of assuming a speedup.\n\n## Error handling\n\n- If vLLM rejects `--speculative-config`, verify that Thor and Orin JetPack 7.2 \u002F L4T r39+ are using vLLM 0.20+ and that older Orin is using a JetPack-matched NVIDIA-AI-IOT vLLM image; then switch back to the non-speculative serving command if the runtime still rejects it.\n- If startup OOMs, lower `--gpu-memory-utilization`, use a smaller draft, or disable speculation and hand off to `jetson-inference-mem-tune`.\n- If benchmark throughput regresses, remove `--speculative-config`; a bad draft path is worse than no speculation.\n\n## Hand off to\n\n- `jetson-llm-benchmark` to quantify the change.\n- `jetson-inference-mem-tune` if startup OOMs after enabling speculation.\n\n## Source\n\nvLLM speculative decoding docs and the [Jetson AI Lab GenAI tutorial](https:\u002F\u002Fwww.jetson-ai-lab.com\u002Ftutorials\u002Fgenai-on-jetson-llms-vlms\u002F).\n",{"name":5,"license":15,"version":24,"metadata":25,"description":7},"0.0.1",{"tags":26,"author":30,"languages":31,"data-classification":33},[27,28,29],"jetson","llm","speculative-decoding","Jetson Team",[32],"markdown","public","main","be3de0244dc1a57e323d8160787ebb40fb64930e","skills\u002Fnvidia\u002Fjetson-speculative-decoding","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main","https:\u002F\u002Fskills.x402.wtf","https:\u002F\u002Fgithub.com\u002FSolizardking","https:\u002F\u002Fmusebook.trade","Got a mile high and never came down","2026-10-08 23:20:13",{"source":8,"skillName":5,"commands":44,"prompt":54,"zipAvailable":55},[45,48,51],{"tool":46,"command":47},"npx","npx skills add Solizardking\u002Fskillhub-main --skill jetson-speculative-decoding",{"tool":49,"command":50},"bunx","bunx skills add Solizardking\u002Fskillhub-main --skill jetson-speculative-decoding",{"tool":52,"command":53},"pnpm","pnpm dlx skills add Solizardking\u002Fskillhub-main --skill jetson-speculative-decoding","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fjetson-speculative-decoding\u002FSKILL.md",true,[57,63,67,71],{"path":58,"resourceType":59,"resourceTypeDesc":60,"sizeBytes":61,"sourceHtmlUrl":62},"BENCHMARK.md","REFERENCE","参考文档",3357,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fjetson-speculative-decoding\u002FBENCHMARK.md",{"path":64,"resourceType":59,"resourceTypeDesc":60,"sizeBytes":65,"sourceHtmlUrl":66},"evals\u002Fevals.json",6522,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fjetson-speculative-decoding\u002Fevals\u002Fevals.json",{"path":68,"resourceType":59,"resourceTypeDesc":60,"sizeBytes":69,"sourceHtmlUrl":70},"skill-card.md",3675,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fjetson-speculative-decoding\u002Fskill-card.md",{"path":72,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":75,"sourceHtmlUrl":76},"skill.oms.sig","OTHER","其他",4605,"https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fjetson-speculative-decoding\u002Fskill.oms.sig",[78,86,93,100,107,115],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":82,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":19},"01M78E0TGCAMNPS207BA3G07S7","zkrouter","Self-hosted OpenAI-compatible LLM router — smart tier-based routing, ZK-stamped routing decisions, mode overrides, free OpenRouter at install via the Birth bot. Save 60-80% on inference costs by routing to the cheapest capable model across 12+ OpenRouter models.","MIT","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fzkrouter\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":82,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":19},"01M78E0R1Z8BDT6EDTHY17MTNB","zk","For custom ZK Solana programs and privacy-preserving applications to prevent double spending. Guide to integrate rent-free nullifier PDAs for double-spend prevention.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fzk\u002FSKILL.md",[],[],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":82,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":19},"01M78E0P3ZTZ1YWF8MD5MM23QE","zk-omni-messaging","Zero-knowledge omnichain messaging between Robinhood Chain and Solana using CheshireZkOmniMessenger (msgType 4) with nullifier anti-replay and the zk-omni-relayer service.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fzk-omni-messaging\u002FSKILL.md",[],[],{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":82,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":19},"01M78E0JXZ2PNB1896VXAGDBWD","zerion-wallet-data","Use when the user asks to check a wallet's portfolio, token balances, DeFi positions, PnL, transaction history, or NFTs. Provides interpreted on-chain data across 40+ EVM chains and Solana via x402 pay-per-request or API key — no raw RPC decoding needed.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fzerion-wallet-data\u002FSKILL.md",[],[],{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":82,"licenseFileUrl":111,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":19},"01M78E0G14FH9GRRTQYMKD5E59","youtube-clipper","YouTube 视频智能剪辑工具。下载视频和字幕，AI 分析生成精细章节（几分钟级别）， 用户选择片段后自动剪辑、翻译字幕为中英双语、烧录字幕到视频，并生成总结文案。 使用场景：当用户需要剪辑 YouTube 视频、生成短视频片段、制作双语字幕版本时。 关键词：视频剪辑、YouTube、字幕翻译、双语字幕、视频下载、clip video","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fyoutube-clipper\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fyoutube-clipper\u002FSKILL.md",[],[],{"skillNo":116,"name":117,"nameZh":6,"description":118,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":82,"licenseFileUrl":6,"sourceHtmlUrl":119,"categories":120,"keywords":121,"publishedAt":19},"01M78E0C2ZYA99CFFDNB7ZMK5S","yield-optimization","Multi-chain yield optimization via yield.xyz (StakeKit). Discover 2,988+ yield opportunities across 75+ blockchains, build deposit\u002Fwithdrawal transactions with shell scripts, and sign them with a MoonPay wallet. Use when the user wants to earn yield, find the best lending or staking rate, enter\u002Fexit a position, or check portfolio balances.","https:\u002F\u002Fgithub.com\u002FSolizardking\u002Fskillhub-main\u002Fblob\u002Fmain\u002Fskills\u002Fyield-optimization\u002FSKILL.md",[],[]]