[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EDZTXG8YCCWDTCMNQ55GYF":3,"skill-same:01M6EDZTXG8YCCWDTCMNQ55GYF":85},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":53,"commitSha":54,"skillDirectory":55,"forksCount":56,"repositoryHtmlUrl":57,"repositoryHomepageUrl":6,"ownerHtmlUrl":58,"ownerBlogUrl":6,"ownerDescription":59,"updatedAt":60,"install":61,"resources":74},"01M6EDZTXG8YCCWDTCMNQ55GYF","speculative-decoding",null,"Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques. Use when optimizing inference speed (1.5-3.6× speedup), reducing latency for real-time applications, or deploying models with limited compute. Covers draft models, tree-based attention, Jacobi iteration, parallel token generation, and production deployment strategies.","luokai0\u002Fai-agent-skills-by-luo-kai","luokai0","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F266954973?v=4","01M6E16VNT42JB8B3X3254QWJ7","luo kai","01M6E16VTFVP6S5XJF1V1MPV90",14,"MIT","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fspeculative-decoding\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"speculative decoding","iteration","Latency Reduction","Parallel Generation","Tree Attention","Draft Models","Lookahead Decoding","medusa","Emerging Techniques","Accelerate","latency","Fast Inference","token","attention","strategies","time","deployment strategies","parallel","real-time","Transformers","2026-11-10 10:12:30",false,"---\nname: speculative-decoding\ndescription: Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques. Use when optimizing inference speed (1.5-3.6× speedup), reducing latency for real-time applications, or deploying models with limited compute. Covers draft models, tree-based attention, Jacobi iteration, parallel token generation, and production deployment strategies.\nversion: 1.0.0\nauthor: luo-kai\nlicense: MIT\ntags: [Emerging Techniques, Speculative Decoding, Medusa, Lookahead Decoding, Fast Inference, Draft Models, Tree Attention, Parallel Generation, Latency Reduction, Inference Optimization]\ndependencies: [transformers, torch]\n---\n\n# Speculative Decoding: Accelerating LLM Inference\n\n## When to Use This Skill\n\nUse Speculative Decoding when you need to:\n- **Speed up inference** by 1.5-3.6× without quality loss\n- **Reduce latency** for real-time applications (chatbots, code generation)\n- **Optimize throughput** for high-volume serving\n- **Deploy efficiently** on limited hardware\n- **Generate faster** without changing model architecture\n\n**Key Techniques**: Draft model speculative decoding, Medusa (multiple heads), Lookahead Decoding (Jacobi iteration)\n\n**Papers**: Medusa (arXiv 2401.10774), Lookahead Decoding (ICML 2024), Speculative Decoding Survey (ACL 2024)\n\n## Installation\n\n```bash\n# Standard speculative decoding (transformers)\npip install transformers accelerate\n\n# Medusa (multiple decoding heads)\ngit clone https:\u002F\u002Fgithub.com\u002FFasterDecoding\u002FMedusa\ncd Medusa\npip install -e .\n\n# Lookahead Decoding\ngit clone https:\u002F\u002Fgithub.com\u002Fhao-ai-lab\u002FLookaheadDecoding\ncd LookaheadDecoding\npip install -e .\n\n# Optional: vLLM with speculative decoding\npip install vllm\n```\n\n## Quick Start\n\n### Basic Speculative Decoding (Draft Model)\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load target model (large, slow)\ntarget_model = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-70b-hf\",\n    device_map=\"auto\",\n    torch_dtype=torch.float16\n)\n\n# Load draft model (small, fast)\ndraft_model = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    device_map=\"auto\",\n    torch_dtype=torch.float16\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-70b-hf\")\n\n# Generate with speculative decoding\nprompt = \"Explain quantum computing in simple terms:\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\n# Transformers 4.36+ supports assisted generation\noutputs = target_model.generate(\n    **inputs,\n    assistant_model=draft_model,  # Enable speculative decoding\n    max_new_tokens=256,\n    do_sample=True,\n    temperature=0.7,\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\nprint(response)\n```\n\n### Medusa (Multiple Decoding Heads)\n\n```python\nfrom medusa.model.medusa_model import MedusaModel\n\n# Load Medusa-enhanced model\nmodel = MedusaModel.from_pretrained(\n    \"FasterDecoding\u002Fmedusa-vicuna-7b-v1.3\",  # Pre-trained with Medusa heads\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"FasterDecoding\u002Fmedusa-vicuna-7b-v1.3\")\n\n# Generate with Medusa (2-3× speedup)\nprompt = \"Write a Python function to calculate fibonacci numbers:\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.medusa_generate(\n    **inputs,\n    max_new_tokens=256,\n    temperature=0.7,\n    posterior_threshold=0.09,  # Acceptance threshold\n    posterior_alpha=0.3,       # Tree construction parameter\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\n```\n\n### Lookahead Decoding (Jacobi Iteration)\n\n```python\nfrom lookahead.lookahead_decoding import LookaheadDecoding\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n\n# Initialize lookahead decoding\nlookahead = LookaheadDecoding(\n    model=model,\n    tokenizer=tokenizer,\n    window_size=15,    # Lookahead window (W)\n    ngram_size=5,      # N-gram size (N)\n    guess_size=5       # Number of parallel guesses\n)\n\n# Generate (1.5-2.3× speedup)\nprompt = \"Implement quicksort in Python:\"\noutput = lookahead.generate(prompt, max_new_tokens=256)\nprint(output)\n```\n\n## Core Concepts\n\n### 1. Speculative Decoding (Draft Model)\n\n**Idea**: Use small draft model to generate candidates, large target model to verify in parallel.\n\n**Algorithm**:\n1. Draft model generates K tokens speculatively\n2. Target model evaluates all K tokens in parallel (single forward pass)\n3. Accept tokens where draft and target agree\n4. Reject first disagreement, continue from there\n\n```python\ndef speculative_decode(target_model, draft_model, prompt, K=4):\n    \"\"\"Speculative decoding algorithm.\"\"\"\n    # 1. Generate K draft tokens\n    draft_tokens = draft_model.generate(prompt, max_new_tokens=K)\n\n    # 2. Target model evaluates all K tokens in one forward pass\n    target_logits = target_model(draft_tokens)  # Parallel!\n\n    # 3. Accept\u002Freject based on probability match\n    accepted = []\n    for i in range(K):\n        p_draft = softmax(draft_model.logits[i])\n        p_target = softmax(target_logits[i])\n\n        # Acceptance probability\n        if random.random() \u003C min(1, p_target[draft_tokens[i]] \u002F p_draft[draft_tokens[i]]):\n            accepted.append(draft_tokens[i])\n        else:\n            break  # Reject, resample from target\n\n    return accepted\n```\n\n**Performance**:\n- Speedup: 1.5-2× with good draft model\n- Zero quality loss (mathematically equivalent to target model)\n- Best when draft model is 5-10× smaller than target\n\n### 2. Medusa (Multiple Decoding Heads)\n\n**Source**: arXiv 2401.10774 (2024)\n\n**Innovation**: Add multiple prediction heads to existing model, predict future tokens without separate draft model.\n\n**Architecture**:\n```\nInput → Base LLM (frozen) → Hidden State\n                                ├→ Head 1 (predicts token t+1)\n                                ├→ Head 2 (predicts token t+2)\n                                ├→ Head 3 (predicts token t+3)\n                                └→ Head 4 (predicts token t+4)\n```\n\n**Training**:\n- **Medusa-1**: Freeze base LLM, train only heads\n  - 2.2× speedup, lossless\n- **Medusa-2**: Fine-tune base LLM + heads together\n  - 2.3-3.6× speedup, better quality\n\n**Tree-based Attention**:\n```python\n# Medusa constructs tree of candidates\n# Example: Predict 2 steps ahead with top-2 per step\n\n#         Root\n#        \u002F    \\\n#      T1a    T1b  (Step 1: 2 candidates)\n#     \u002F  \\    \u002F \\\n#  T2a  T2b T2c T2d  (Step 2: 4 candidates total)\n\n# Single forward pass evaluates entire tree!\n```\n\n**Advantages**:\n- No separate draft model needed\n- Minimal training (only heads)\n- Compatible with any LLM\n\n### 3. Lookahead Decoding (Jacobi Iteration)\n\n**Source**: ICML 2024\n\n**Core idea**: Reformulate autoregressive decoding as solving system of equations, solve in parallel using Jacobi iteration.\n\n**Mathematical formulation**:\n```\nTraditional:  y_t = f(x, y_1, ..., y_{t-1})  (sequential)\nJacobi:       y_t^{(k+1)} = f(x, y_1^{(k)}, ..., y_{t-1}^{(k)})  (parallel)\n```\n\n**Two branches**:\n\n1. **Lookahead Branch**: Generate n-grams in parallel\n   - Window size W: How many steps to look ahead\n   - N-gram size N: How many past tokens to use\n\n2. **Verification Branch**: Verify promising n-grams\n   - Match n-grams with generated tokens\n   - Accept if first token matches\n\n```python\nclass LookaheadDecoding:\n    def __init__(self, model, window_size=15, ngram_size=5):\n        self.model = model\n        self.W = window_size  # Lookahead window\n        self.N = ngram_size   # N-gram size\n\n    def generate_step(self, tokens):\n        # Lookahead branch: Generate W × N candidates\n        candidates = {}\n        for w in range(1, self.W + 1):\n            for n in range(1, self.N + 1):\n                # Generate n-gram starting at position w\n                ngram = self.generate_ngram(tokens, start=w, length=n)\n                candidates[(w, n)] = ngram\n\n        # Verification branch: Find matching n-grams\n        verified = []\n        for ngram in candidates.values():\n            if ngram[0] == tokens[-1]:  # First token matches last input\n                if self.verify(tokens, ngram):\n                    verified.append(ngram)\n\n        # Accept longest verified n-gram\n        return max(verified, key=len) if verified else [self.model.generate_next(tokens)]\n```\n\n**Performance**:\n- Speedup: 1.5-2.3× (up to 3.6× for code generation)\n- No draft model or training needed\n- Works out-of-the-box with any model\n\n## Method Comparison\n\n| Method | Speedup | Training Needed | Draft Model | Quality Loss |\n|--------|---------|-----------------|-------------|--------------|\n| **Draft Model Speculative** | 1.5-2× | No | Yes (external) | None |\n| **Medusa** | 2-3.6× | Minimal (heads only) | No (built-in heads) | None |\n| **Lookahead** | 1.5-2.3× | None | No | None |\n| **Naive Batching** | 1.2-1.5× | No | No | None |\n\n## Advanced Patterns\n\n### Training Medusa Heads\n\n```python\nfrom medusa.model.medusa_model import MedusaModel\nfrom medusa.model.kv_cache import initialize_past_key_values\nimport torch.nn as nn\n\n# 1. Load base model\nbase_model = AutoModelForCausalLM.from_pretrained(\n    \"lmsys\u002Fvicuna-7b-v1.3\",\n    torch_dtype=torch.float16\n)\n\n# 2. Add Medusa heads\nnum_heads = 4\nmedusa_heads = nn.ModuleList([\n    nn.Linear(base_model.config.hidden_size, base_model.config.vocab_size, bias=False)\n    for _ in range(num_heads)\n])\n\n# 3. Training loop (freeze base model for Medusa-1)\nfor param in base_model.parameters():\n    param.requires_grad = False  # Freeze base\n\noptimizer = torch.optim.Adam(medusa_heads.parameters(), lr=1e-3)\n\nfor batch in dataloader:\n    # Forward pass\n    hidden_states = base_model(**batch, output_hidden_states=True).hidden_states[-1]\n\n    # Predict future tokens with each head\n    loss = 0\n    for i, head in enumerate(medusa_heads):\n        logits = head(hidden_states)\n        # Target: tokens shifted by (i+1) positions\n        target = batch['input_ids'][:, i+1:]\n        loss += F.cross_entropy(logits[:, :-i-1], target)\n\n    # Backward\n    optimizer.zero_grad()\n    loss.backward()\n    optimizer.step()\n```\n\n### Hybrid: Speculative + Medusa\n\n```python\n# Use Medusa as draft model for speculative decoding\ndraft_medusa = MedusaModel.from_pretrained(\"medusa-vicuna-7b\")\ntarget_model = AutoModelForCausalLM.from_pretrained(\"vicuna-33b\")\n\n# Draft generates multiple candidates with Medusa\ndraft_tokens = draft_medusa.medusa_generate(prompt, max_new_tokens=5)\n\n# Target verifies in single forward pass\noutputs = target_model.generate(\n    prompt,\n    assistant_model=draft_medusa,  # Use Medusa as draft\n    max_new_tokens=256\n)\n\n# Combines benefits: Medusa speed + large model quality\n```\n\n### Optimal Draft Model Selection\n\n```python\ndef select_draft_model(target_model_size, target):\n    \"\"\"Select optimal draft model for speculative decoding.\"\"\"\n    # Rule: Draft should be 5-10× smaller\n    if target_model_size == \"70B\":\n        return \"7B\"  # 10× smaller\n    elif target_model_size == \"33B\":\n        return \"7B\"  # 5× smaller\n    elif target_model_size == \"13B\":\n        return \"1B\"  # 13× smaller\n    else:\n        return None  # Target too small, use Medusa\u002FLookahead instead\n\n# Example\ndraft = select_draft_model(\"70B\", target_model)\n# Returns \"7B\" → Use Llama-2-7b as draft for Llama-2-70b\n```\n\n## Best Practices\n\n### 1. Choose the Right Method\n\n```python\n# New deployment → Medusa (best overall speedup, no draft model)\nif deploying_new_model:\n    use_method = \"Medusa\"\n\n# Existing deployment with small model available → Draft speculative\nelif have_small_version_of_model:\n    use_method = \"Draft Model Speculative\"\n\n# Want zero training\u002Fsetup → Lookahead\nelif want_plug_and_play:\n    use_method = \"Lookahead Decoding\"\n```\n\n### 2. Hyperparameter Tuning\n\n**Draft Model Speculative**:\n```python\n# K = number of speculative tokens\nK = 4  # Good default\nK = 2  # Conservative (higher acceptance)\nK = 8  # Aggressive (lower acceptance, but more when accepted)\n\n# Rule: Larger K → more speedup IF draft model is good\n```\n\n**Medusa**:\n```python\n# Posterior threshold (acceptance confidence)\nposterior_threshold = 0.09  # Standard (from paper)\nposterior_threshold = 0.05  # More conservative (slower, higher quality)\nposterior_threshold = 0.15  # More aggressive (faster, may degrade quality)\n\n# Tree depth (how many steps ahead)\nmedusa_choices = [[0], [0, 0], [0, 1], [0, 0, 0]]  # Depth 3 (standard)\n```\n\n**Lookahead**:\n```python\n# Window size W (lookahead distance)\n# N-gram size N (context for generation)\n\n# 7B model (more resources)\nW, N = 15, 5\n\n# 13B model (moderate)\nW, N = 10, 5\n\n# 33B+ model (limited resources)\nW, N = 7, 5\n```\n\n### 3. Production Deployment\n\n```python\n# vLLM with speculative decoding\nfrom vllm import LLM, SamplingParams\n\n# Initialize with draft model\nllm = LLM(\n    model=\"meta-llama\u002FLlama-2-70b-hf\",\n    speculative_model=\"meta-llama\u002FLlama-2-7b-hf\",  # Draft model\n    num_speculative_tokens=5,\n    use_v2_block_manager=True,\n)\n\n# Generate\nprompts = [\"Tell me about AI:\", \"Explain quantum physics:\"]\nsampling_params = SamplingParams(temperature=0.7, max_tokens=256)\n\noutputs = llm.generate(prompts, sampling_params)\nfor output in outputs:\n    print(output.outputs[0].text)\n```\n\n## Resources\n\n- **Medusa Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.10774\n- **Medusa GitHub**: https:\u002F\u002Fgithub.com\u002FFasterDecoding\u002FMedusa\n- **Lookahead Decoding (ICML 2024)**: https:\u002F\u002Flmsys.org\u002Fblog\u002F2023-11-21-lookahead-decoding\u002F\n- **Lookahead GitHub**: https:\u002F\u002Fgithub.com\u002Fhao-ai-lab\u002FLookaheadDecoding\n- **Speculative Decoding Survey (ACL 2024)**: https:\u002F\u002Faclanthology.org\u002F2024.findings-acl.456.pdf\n- **Comprehensive Survey**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.07851\n\n## See Also\n\n- `references\u002Fdraft_model.md` - Draft model selection and training\n- `references\u002Fmedusa.md` - Medusa architecture and training\n- `references\u002Flookahead.md` - Lookahead decoding implementation details\n\n\n","# Speculative Decoding: Accelerating LLM Inference\n\n## When to Use This Skill\n\nUse Speculative Decoding when you need to:\n- **Speed up inference** by 1.5-3.6× without quality loss\n- **Reduce latency** for real-time applications (chatbots, code generation)\n- **Optimize throughput** for high-volume serving\n- **Deploy efficiently** on limited hardware\n- **Generate faster** without changing model architecture\n\n**Key Techniques**: Draft model speculative decoding, Medusa (multiple heads), Lookahead Decoding (Jacobi iteration)\n\n**Papers**: Medusa (arXiv 2401.10774), Lookahead Decoding (ICML 2024), Speculative Decoding Survey (ACL 2024)\n\n## Installation\n\n```bash\n# Standard speculative decoding (transformers)\npip install transformers accelerate\n\n# Medusa (multiple decoding heads)\ngit clone https:\u002F\u002Fgithub.com\u002FFasterDecoding\u002FMedusa\ncd Medusa\npip install -e .\n\n# Lookahead Decoding\ngit clone https:\u002F\u002Fgithub.com\u002Fhao-ai-lab\u002FLookaheadDecoding\ncd LookaheadDecoding\npip install -e .\n\n# Optional: vLLM with speculative decoding\npip install vllm\n```\n\n## Quick Start\n\n### Basic Speculative Decoding (Draft Model)\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load target model (large, slow)\ntarget_model = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-70b-hf\",\n    device_map=\"auto\",\n    torch_dtype=torch.float16\n)\n\n# Load draft model (small, fast)\ndraft_model = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    device_map=\"auto\",\n    torch_dtype=torch.float16\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-70b-hf\")\n\n# Generate with speculative decoding\nprompt = \"Explain quantum computing in simple terms:\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\n# Transformers 4.36+ supports assisted generation\noutputs = target_model.generate(\n    **inputs,\n    assistant_model=draft_model,  # Enable speculative decoding\n    max_new_tokens=256,\n    do_sample=True,\n    temperature=0.7,\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\nprint(response)\n```\n\n### Medusa (Multiple Decoding Heads)\n\n```python\nfrom medusa.model.medusa_model import MedusaModel\n\n# Load Medusa-enhanced model\nmodel = MedusaModel.from_pretrained(\n    \"FasterDecoding\u002Fmedusa-vicuna-7b-v1.3\",  # Pre-trained with Medusa heads\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"FasterDecoding\u002Fmedusa-vicuna-7b-v1.3\")\n\n# Generate with Medusa (2-3× speedup)\nprompt = \"Write a Python function to calculate fibonacci numbers:\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.medusa_generate(\n    **inputs,\n    max_new_tokens=256,\n    temperature=0.7,\n    posterior_threshold=0.09,  # Acceptance threshold\n    posterior_alpha=0.3,       # Tree construction parameter\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\n```\n\n### Lookahead Decoding (Jacobi Iteration)\n\n```python\nfrom lookahead.lookahead_decoding import LookaheadDecoding\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n\n# Initialize lookahead decoding\nlookahead = LookaheadDecoding(\n    model=model,\n    tokenizer=tokenizer,\n    window_size=15,    # Lookahead window (W)\n    ngram_size=5,      # N-gram size (N)\n    guess_size=5       # Number of parallel guesses\n)\n\n# Generate (1.5-2.3× speedup)\nprompt = \"Implement quicksort in Python:\"\noutput = lookahead.generate(prompt, max_new_tokens=256)\nprint(output)\n```\n\n## Core Concepts\n\n### 1. Speculative Decoding (Draft Model)\n\n**Idea**: Use small draft model to generate candidates, large target model to verify in parallel.\n\n**Algorithm**:\n1. Draft model generates K tokens speculatively\n2. Target model evaluates all K tokens in parallel (single forward pass)\n3. Accept tokens where draft and target agree\n4. Reject first disagreement, continue from there\n\n```python\ndef speculative_decode(target_model, draft_model, prompt, K=4):\n    \"\"\"Speculative decoding algorithm.\"\"\"\n    # 1. Generate K draft tokens\n    draft_tokens = draft_model.generate(prompt, max_new_tokens=K)\n\n    # 2. Target model evaluates all K tokens in one forward pass\n    target_logits = target_model(draft_tokens)  # Parallel!\n\n    # 3. Accept\u002Freject based on probability match\n    accepted = []\n    for i in range(K):\n        p_draft = softmax(draft_model.logits[i])\n        p_target = softmax(target_logits[i])\n\n        # Acceptance probability\n        if random.random() \u003C min(1, p_target[draft_tokens[i]] \u002F p_draft[draft_tokens[i]]):\n            accepted.append(draft_tokens[i])\n        else:\n            break  # Reject, resample from target\n\n    return accepted\n```\n\n**Performance**:\n- Speedup: 1.5-2× with good draft model\n- Zero quality loss (mathematically equivalent to target model)\n- Best when draft model is 5-10× smaller than target\n\n### 2. Medusa (Multiple Decoding Heads)\n\n**Source**: arXiv 2401.10774 (2024)\n\n**Innovation**: Add multiple prediction heads to existing model, predict future tokens without separate draft model.\n\n**Architecture**:\n```\nInput → Base LLM (frozen) → Hidden State\n                                ├→ Head 1 (predicts token t+1)\n                                ├→ Head 2 (predicts token t+2)\n                                ├→ Head 3 (predicts token t+3)\n                                └→ Head 4 (predicts token t+4)\n```\n\n**Training**:\n- **Medusa-1**: Freeze base LLM, train only heads\n  - 2.2× speedup, lossless\n- **Medusa-2**: Fine-tune base LLM + heads together\n  - 2.3-3.6× speedup, better quality\n\n**Tree-based Attention**:\n```python\n# Medusa constructs tree of candidates\n# Example: Predict 2 steps ahead with top-2 per step\n\n#         Root\n#        \u002F    \\\n#      T1a    T1b  (Step 1: 2 candidates)\n#     \u002F  \\    \u002F \\\n#  T2a  T2b T2c T2d  (Step 2: 4 candidates total)\n\n# Single forward pass evaluates entire tree!\n```\n\n**Advantages**:\n- No separate draft model needed\n- Minimal training (only heads)\n- Compatible with any LLM\n\n### 3. Lookahead Decoding (Jacobi Iteration)\n\n**Source**: ICML 2024\n\n**Core idea**: Reformulate autoregressive decoding as solving system of equations, solve in parallel using Jacobi iteration.\n\n**Mathematical formulation**:\n```\nTraditional:  y_t = f(x, y_1, ..., y_{t-1})  (sequential)\nJacobi:       y_t^{(k+1)} = f(x, y_1^{(k)}, ..., y_{t-1}^{(k)})  (parallel)\n```\n\n**Two branches**:\n\n1. **Lookahead Branch**: Generate n-grams in parallel\n   - Window size W: How many steps to look ahead\n   - N-gram size N: How many past tokens to use\n\n2. **Verification Branch**: Verify promising n-grams\n   - Match n-grams with generated tokens\n   - Accept if first token matches\n\n```python\nclass LookaheadDecoding:\n    def __init__(self, model, window_size=15, ngram_size=5):\n        self.model = model\n        self.W = window_size  # Lookahead window\n        self.N = ngram_size   # N-gram size\n\n    def generate_step(self, tokens):\n        # Lookahead branch: Generate W × N candidates\n        candidates = {}\n        for w in range(1, self.W + 1):\n            for n in range(1, self.N + 1):\n                # Generate n-gram starting at position w\n                ngram = self.generate_ngram(tokens, start=w, length=n)\n                candidates[(w, n)] = ngram\n\n        # Verification branch: Find matching n-grams\n        verified = []\n        for ngram in candidates.values():\n            if ngram[0] == tokens[-1]:  # First token matches last input\n                if self.verify(tokens, ngram):\n                    verified.append(ngram)\n\n        # Accept longest verified n-gram\n        return max(verified, key=len) if verified else [self.model.generate_next(tokens)]\n```\n\n**Performance**:\n- Speedup: 1.5-2.3× (up to 3.6× for code generation)\n- No draft model or training needed\n- Works out-of-the-box with any model\n\n## Method Comparison\n\n| Method | Speedup | Training Needed | Draft Model | Quality Loss |\n|--------|---------|-----------------|-------------|--------------|\n| **Draft Model Speculative** | 1.5-2× | No | Yes (external) | None |\n| **Medusa** | 2-3.6× | Minimal (heads only) | No (built-in heads) | None |\n| **Lookahead** | 1.5-2.3× | None | No | None |\n| **Naive Batching** | 1.2-1.5× | No | No | None |\n\n## Advanced Patterns\n\n### Training Medusa Heads\n\n```python\nfrom medusa.model.medusa_model import MedusaModel\nfrom medusa.model.kv_cache import initialize_past_key_values\nimport torch.nn as nn\n\n# 1. Load base model\nbase_model = AutoModelForCausalLM.from_pretrained(\n    \"lmsys\u002Fvicuna-7b-v1.3\",\n    torch_dtype=torch.float16\n)\n\n# 2. Add Medusa heads\nnum_heads = 4\nmedusa_heads = nn.ModuleList([\n    nn.Linear(base_model.config.hidden_size, base_model.config.vocab_size, bias=False)\n    for _ in range(num_heads)\n])\n\n# 3. Training loop (freeze base model for Medusa-1)\nfor param in base_model.parameters():\n    param.requires_grad = False  # Freeze base\n\noptimizer = torch.optim.Adam(medusa_heads.parameters(), lr=1e-3)\n\nfor batch in dataloader:\n    # Forward pass\n    hidden_states = base_model(**batch, output_hidden_states=True).hidden_states[-1]\n\n    # Predict future tokens with each head\n    loss = 0\n    for i, head in enumerate(medusa_heads):\n        logits = head(hidden_states)\n        # Target: tokens shifted by (i+1) positions\n        target = batch['input_ids'][:, i+1:]\n        loss += F.cross_entropy(logits[:, :-i-1], target)\n\n    # Backward\n    optimizer.zero_grad()\n    loss.backward()\n    optimizer.step()\n```\n\n### Hybrid: Speculative + Medusa\n\n```python\n# Use Medusa as draft model for speculative decoding\ndraft_medusa = MedusaModel.from_pretrained(\"medusa-vicuna-7b\")\ntarget_model = AutoModelForCausalLM.from_pretrained(\"vicuna-33b\")\n\n# Draft generates multiple candidates with Medusa\ndraft_tokens = draft_medusa.medusa_generate(prompt, max_new_tokens=5)\n\n# Target verifies in single forward pass\noutputs = target_model.generate(\n    prompt,\n    assistant_model=draft_medusa,  # Use Medusa as draft\n    max_new_tokens=256\n)\n\n# Combines benefits: Medusa speed + large model quality\n```\n\n### Optimal Draft Model Selection\n\n```python\ndef select_draft_model(target_model_size, target):\n    \"\"\"Select optimal draft model for speculative decoding.\"\"\"\n    # Rule: Draft should be 5-10× smaller\n    if target_model_size == \"70B\":\n        return \"7B\"  # 10× smaller\n    elif target_model_size == \"33B\":\n        return \"7B\"  # 5× smaller\n    elif target_model_size == \"13B\":\n        return \"1B\"  # 13× smaller\n    else:\n        return None  # Target too small, use Medusa\u002FLookahead instead\n\n# Example\ndraft = select_draft_model(\"70B\", target_model)\n# Returns \"7B\" → Use Llama-2-7b as draft for Llama-2-70b\n```\n\n## Best Practices\n\n### 1. Choose the Right Method\n\n```python\n# New deployment → Medusa (best overall speedup, no draft model)\nif deploying_new_model:\n    use_method = \"Medusa\"\n\n# Existing deployment with small model available → Draft speculative\nelif have_small_version_of_model:\n    use_method = \"Draft Model Speculative\"\n\n# Want zero training\u002Fsetup → Lookahead\nelif want_plug_and_play:\n    use_method = \"Lookahead Decoding\"\n```\n\n### 2. Hyperparameter Tuning\n\n**Draft Model Speculative**:\n```python\n# K = number of speculative tokens\nK = 4  # Good default\nK = 2  # Conservative (higher acceptance)\nK = 8  # Aggressive (lower acceptance, but more when accepted)\n\n# Rule: Larger K → more speedup IF draft model is good\n```\n\n**Medusa**:\n```python\n# Posterior threshold (acceptance confidence)\nposterior_threshold = 0.09  # Standard (from paper)\nposterior_threshold = 0.05  # More conservative (slower, higher quality)\nposterior_threshold = 0.15  # More aggressive (faster, may degrade quality)\n\n# Tree depth (how many steps ahead)\nmedusa_choices = [[0], [0, 0], [0, 1], [0, 0, 0]]  # Depth 3 (standard)\n```\n\n**Lookahead**:\n```python\n# Window size W (lookahead distance)\n# N-gram size N (context for generation)\n\n# 7B model (more resources)\nW, N = 15, 5\n\n# 13B model (moderate)\nW, N = 10, 5\n\n# 33B+ model (limited resources)\nW, N = 7, 5\n```\n\n### 3. Production Deployment\n\n```python\n# vLLM with speculative decoding\nfrom vllm import LLM, SamplingParams\n\n# Initialize with draft model\nllm = LLM(\n    model=\"meta-llama\u002FLlama-2-70b-hf\",\n    speculative_model=\"meta-llama\u002FLlama-2-7b-hf\",  # Draft model\n    num_speculative_tokens=5,\n    use_v2_block_manager=True,\n)\n\n# Generate\nprompts = [\"Tell me about AI:\", \"Explain quantum physics:\"]\nsampling_params = SamplingParams(temperature=0.7, max_tokens=256)\n\noutputs = llm.generate(prompts, sampling_params)\nfor output in outputs:\n    print(output.outputs[0].text)\n```\n\n## Resources\n\n- **Medusa Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.10774\n- **Medusa GitHub**: https:\u002F\u002Fgithub.com\u002FFasterDecoding\u002FMedusa\n- **Lookahead Decoding (ICML 2024)**: https:\u002F\u002Flmsys.org\u002Fblog\u002F2023-11-21-lookahead-decoding\u002F\n- **Lookahead GitHub**: https:\u002F\u002Fgithub.com\u002Fhao-ai-lab\u002FLookaheadDecoding\n- **Speculative Decoding Survey (ACL 2024)**: https:\u002F\u002Faclanthology.org\u002F2024.findings-acl.456.pdf\n- **Comprehensive Survey**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.07851\n\n## See Also\n\n- `references\u002Fdraft_model.md` - Draft model selection and training\n- `references\u002Fmedusa.md` - Medusa architecture and training\n- `references\u002Flookahead.md` - Lookahead decoding implementation details\n\n\n",{"name":5,"tags":44,"author":48,"license":15,"version":49,"description":7,"dependencies":50},[27,45,46,25,30,24,23,22,21,47],"Speculative Decoding","Medusa","Inference Optimization","luo-kai","1.0.0",[51,52],"transformers","torch","main","4d2fb89cdfb75f3f6fd3c6c5ec9ad90ea129eb3c","ai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fspeculative-decoding",2,"https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai","https:\u002F\u002Fgithub.com\u002Fluokai0","a solo builder.","2026-05-06 04:29:04",{"source":8,"skillName":5,"commands":62,"prompt":72,"zipAvailable":73},[63,66,69],{"tool":64,"command":65},"npx","npx skills add luokai0\u002Fai-agent-skills-by-luo-kai --skill speculative-decoding",{"tool":67,"command":68},"bunx","bunx skills add luokai0\u002Fai-agent-skills-by-luo-kai --skill speculative-decoding",{"tool":70,"command":71},"pnpm","pnpm dlx skills add luokai0\u002Fai-agent-skills-by-luo-kai --skill speculative-decoding","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fspeculative-decoding\u002FSKILL.md",true,[75,81],{"path":76,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":79,"sourceHtmlUrl":80},"references\u002Flookahead.md","REFERENCE","参考文档",8890,"https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fspeculative-decoding\u002Freferences\u002Flookahead.md",{"path":82,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Fmedusa.md",9851,"https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F18-ai-agents-and-automation (by Luo Kai)\u002F16-other-agents\u002Fspeculative-decoding\u002Freferences\u002Fmedusa.md",[86,112,129,159,192,229],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":39},"01M6F2SKTE208174C7S7JJCZC7","write-contracts","Generates secure Aptos Move V2 smart contracts with Object model, Digital Asset integration, security patterns, and storage type guidance. Includes comprehensive storage decision framework for optimal data structure selection. Triggers on: 'write contract', 'create NFT collection', 'build marketplace', 'implement minting', 'generate Move module', 'create token contract', 'build DAO', 'implement staking'. Ask storage questions when: 'store', 'track', 'registry', 'mapping', 'list', 'collection'.","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fwrite-contracts\u002FSKILL.md",[],[93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,31,109,110,111],"write","contracts","write contracts","storage","secure","create","generate","collection","model","decision","ask","smart-contracts","selection","objects","structure","nft","aptos","critical","move",{"skillNo":113,"name":114,"nameZh":6,"description":115,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":116,"categories":117,"keywords":118,"publishedAt":39},"01M6F2RFXJ6T9MKW0A7N7GM45B","web-search","通用网络搜索技能，支持多引擎搜索（百度、必应、DuckDuckGo），无需API密钥即可获取实时信息","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fweb-search-ex-skill\u002FSKILL.md",[],[119,120,121,122,123,124,125,126,127,128],"web","search","web search","搜索","baidu","bing","duckduckgo","百度","多引擎","internet",{"skillNo":130,"name":131,"nameZh":6,"description":132,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":133,"categories":134,"keywords":140,"publishedAt":39},"01M6F2K968G1FZEZFPW2E6B222","solo-build","Execute implementation plan tasks with TDD workflow, auto-commit, and phase gates. Use when user says \"build it\", \"start building\", \"execute plan\", \"implement tasks\", \"ship it\", or references a track ID. Do NOT use for planning (use \u002Fplan) or scaffolding (use \u002Fscaffold).","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fsolo-build\u002FSKILL.md",[135],{"categoryNo":136,"categoryName":137,"taxonomyType":138,"taxonomyTypeDesc":139},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[141,142,143,144,145,146,147,148,120,149,150,151,93,152,153,154,155,156,157,158],"build","solo","solo build","MCP","workflow","commit","tasks","code","Bash","plan","read","auto","references","context7","query","scaffolding","ship it","project",{"skillNo":160,"name":161,"nameZh":6,"description":162,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":163,"categories":164,"keywords":171,"publishedAt":39},"01M6F2GKN39SGHSPWYB3662YC8","react-email","Use when building HTML email templates with React components, adding a visual email editor to an application using the React Email visual editor, rendering emails to HTML, or sending emails with Resend. Covers welcome emails, password resets, notifications, order confirmations, newsletters, transactional emails, and the embeddable email editor component.","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Freact-email-skills\u002FSKILL.md",[165,168],{"categoryNo":166,"categoryName":167,"taxonomyType":138,"taxonomyTypeDesc":139},"01JBSEEDCAT000000000000005","邮件与沟通",{"categoryNo":169,"categoryName":170,"taxonomyType":138,"taxonomyTypeDesc":139},"01JBSEEDCAT000000000000009","前端开发",[172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191],"react","email","react email","github","editor","components","react components","component","welcome","https","notifications","password","node","use","html","resend","visual","emails","rendering","with",{"skillNo":193,"name":194,"nameZh":6,"description":195,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":196,"categories":197,"keywords":210,"publishedAt":39},"01M6F2FBVGVDFSG8SSFJGXA40C","APITester Agent-Driven API Testing","Test API endpoints and document responses. Define tests in plain English, run them, get formatted results. Agent-driven Postman alternative.","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fopenclaw-api-tester\u002FSKILL.md",[198,201,204,205],{"categoryNo":199,"categoryName":200,"taxonomyType":138,"taxonomyTypeDesc":139},"01JBSEEDCAT000000000000008","测试与质量",{"categoryNo":202,"categoryName":203,"taxonomyType":138,"taxonomyTypeDesc":139},"01JBSEEDCAT000000000000010","后端与接口",{"categoryNo":136,"categoryName":137,"taxonomyType":138,"taxonomyTypeDesc":139},{"categoryNo":206,"categoryName":207,"taxonomyType":208,"taxonomyTypeDesc":209},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228],"testing","api","api testing","driven","APITester Agent Driven API Testing","test","run","postman","results","plain english","tester","endpoints","document","get","define","tests","english","api-tester",{"skillNo":230,"name":231,"nameZh":6,"description":232,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":233,"categories":234,"keywords":235,"publishedAt":39},"01M6F273Y3NGAPVJZ608BC0G9G","hubspot-implementation-plan","Generate a phased implementation plan from a HubSpot audit report. Creates prioritized, sequenced cleanup processes with effort estimates, dependencies, and automation feasibility. Use after running \u002Fhubspot-audit.","https:\u002F\u002Fgithub.com\u002Fluokai0\u002Fai-agent-skills-by-luo-kai\u002Fblob\u002Fmain\u002Fai-agent-skills\u002F22-clawhub-skills\u002Fhubspot-implementation-plan\u002FSKILL.md",[],[150,236,237,238,239,240,241,99,242,243,244,245,246,185,247,248,191,249],"implementation","hubspot","hubspot implementation plan","automation","feasibility","cleanup","planning","report","audit","audit report","phased implementation","dependencies","processes","hubspot audit"]