AGENT SKILL

jetson-speculative-decoding

Add EAGLE-3 or draft-model speculative decoding to a Jetson vLLM server when TPOT is the bottleneck.

License 许可
Apache-2.0
最近更新
2026-10-08
Stars
★ 0
Forks
0

01 前言 Front Matter

name
jetson-speculative-decoding
license
Apache-2.0
version
0.0.1
metadata
{ "tags": [ "jetson", "llm", "speculative-decoding" ], "author": "Jetson Team", "languages": [ "markdown" ], "data-classification": "public" }
description
Add EAGLE-3 or draft-model speculative decoding to a Jetson vLLM server when TPOT is the bottleneck.

02 资源文件 4

03 SKILL.md 正文

Jetson Speculative Decoding (vLLM)

Speculative decoding lets a small "draft" model propose tokens that the target model verifies in a single forward pass, reducing per-token latency. On Jetson, the win/loss is dominated by VRAM headroom, not by the draft quality. This skill encodes the parts an LLM won't already know.

Purpose

Tune an existing Jetson vLLM deployment for faster token generation by appending the right --speculative-config and validating whether it improves single-stream decode speed.

When to use

  • TPOT/ITL is the bottleneck (TTFT is fine, output is just slow).
  • Workload is single-stream or low-concurrency (≤2). Speculation usually loses at high concurrency.
  • Jetson family is Thor or AGX Orin. Do not suggest EAGLE-3 on Orin Nano/NX — there is rarely enough VRAM headroom to host both target and draft, and you'll OOM at startup.

When NOT to use

  • High-concurrency serving (≥8): batched decode usually beats speculation; the draft model just steals VRAM.
  • Models without a published EAGLE-3 head — do not train one ad-hoc as a "fix".
  • After applying jetson-inference-mem-tune flags that already pushed --gpu-memory-utilization near the ceiling. Free at least ~2 GB first.

Prerequisites

  • A working vLLM server recipe from jetson-llm-serve.
  • Enough memory headroom for the draft model or EAGLE-3 head in addition to the target model.
  • A benchmark baseline from jetson-llm-benchmark before enabling speculation.
  • A target model with a compatible EAGLE-3 head, or a small same-family draft model for the fallback path.

Instructions

Append --speculative-config to the vllm serve command shown in jetson-llm-serve.

EAGLE-3 (preferred when a head is published for the target model):

--speculative-config '{
  "method": "eagle3",
  "model": "<eagle3-head-repo-id>",
  "num_speculative_tokens": 5,
  "draft_tensor_parallel_size": 1
}'

Draft-model (fallback — pair a small same-family model):

--speculative-config '{
  "method": "draft_model",
  "model": "<small-draft-model-repo-id>",
  "num_speculative_tokens": 4,
  "draft_tensor_parallel_size": 1
}'

Jetson-specific tuning rules

  • num_speculative_tokens: start at 5 on Thor, 3 on AGX Orin. Higher values pay off only if the draft acceptance rate is >0.6.
  • Always pair with the same vLLM runtime path used by jetson-llm-serve: upstream vLLM 0.20+ (vllm/vllm-openai:latest) or validated native vLLM 0.20+ on Thor, upstream vLLM 0.20+ on Orin JetPack 7.2 / L4T r39+, or the NVIDIA-AI-IOT vLLM image on older Orin. Do not use an Orin NVIDIA-AI-IOT vLLM image on Thor. Older runtimes may lack EAGLE-3 or the current --speculative-config shape.
  • Drop --gpu-memory-utilization by ~0.05 vs the non-speculative baseline to give the draft model headroom.

How to verify it actually helped

  1. Run jetson-llm-benchmark (vLLM path) at --concurrency 1 before and after enabling speculation.
  2. Acceptance: target ≥30% improvement in throughput_tok_s and ≥20% drop in tpot_ms_p50 at concurrency 1.
  3. If improvement is <10%, or throughput_tok_s regresses at concurrency 8, disable speculation. The draft model is costing more than it returns.

Limitations

  • Speculative decoding improves decode-heavy workloads; it does not reduce TTFT-dominated latency.
  • High concurrency can erase the benefit because continuous batching already keeps the GPU busy.
  • Orin Nano/NX usually lack enough memory headroom for both target and draft models.
  • Acceptance rate and draft overhead are model-specific, so benchmark before and after instead of assuming a speedup.

Error handling

  • If vLLM rejects --speculative-config, verify that Thor and Orin JetPack 7.2 / L4T r39+ are using vLLM 0.20+ and that older Orin is using a JetPack-matched NVIDIA-AI-IOT vLLM image; then switch back to the non-speculative serving command if the runtime still rejects it.
  • If startup OOMs, lower --gpu-memory-utilization, use a smaller draft, or disable speculation and hand off to jetson-inference-mem-tune.
  • If benchmark throughput regresses, remove --speculative-config; a bad draft path is worse than no speculation.

Hand off to

  • jetson-llm-benchmark to quantify the change.
  • jetson-inference-mem-tune if startup OOMs after enabling speculation.

Source

vLLM speculative decoding docs and the Jetson AI Lab GenAI tutorial.

04 同仓库更多 Skills

Solizardking
Solizardking/skillhub-main

zkrouter

Self-hosted OpenAI-compatible LLM router — smart tier-based routing, ZK-stamped routing decisions, mode overrides, free OpenRouter at install via the Birth bot. Save 60-80% on inference costs by routing to the cheapest capable model across 12+ OpenRouter models.

★ 0
Solizardking
Solizardking/skillhub-main

zk

For custom ZK Solana programs and privacy-preserving applications to prevent double spending. Guide to integrate rent-free nullifier PDAs for double-spend prevention.

★ 0
Solizardking
Solizardking/skillhub-main

zk-omni-messaging

Zero-knowledge omnichain messaging between Robinhood Chain and Solana using CheshireZkOmniMessenger (msgType 4) with nullifier anti-replay and the zk-omni-relayer service.

★ 0
Solizardking
Solizardking/skillhub-main

zerion-wallet-data

Use when the user asks to check a wallet's portfolio, token balances, DeFi positions, PnL, transaction history, or NFTs. Provides interpreted on-chain data across 40+ EVM chains and Solana via x402 pay-per-request or API key — no raw RPC decoding needed.

★ 0
Solizardking
Solizardking/skillhub-main

youtube-clipper

YouTube 视频智能剪辑工具。下载视频和字幕,AI 分析生成精细章节(几分钟级别), 用户选择片段后自动剪辑、翻译字幕为中英双语、烧录字幕到视频,并生成总结文案。 使用场景:当用户需要剪辑 YouTube 视频、生成短视频片段、制作双语字幕版本时。 关键词:视频剪辑、YouTube、字幕翻译、双语字幕、视频下载、clip video

★ 0
Solizardking
Solizardking/skillhub-main

yield-optimization

Multi-chain yield optimization via yield.xyz (StakeKit). Discover 2,988+ yield opportunities across 75+ blockchains, build deposit/withdrawal transactions with shell scripts, and sign them with a MoonPay wallet. Use when the user wants to earn yield, find the best lending or staking rate, enter/exit a position, or check portfolio balances.

★ 0