[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M781XT3ZTSSNPXE19B83T0VR":3,"skill-same:01M781XT3ZTSSNPXE19B83T0VR":71},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":40,"commitSha":41,"skillDirectory":42,"forksCount":43,"repositoryHtmlUrl":44,"repositoryHomepageUrl":6,"ownerHtmlUrl":45,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":46,"install":47,"resources":60},"01M781XT3ZTSSNPXE19B83T0VR","speculative-decoding",null,"Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques. Use when optimizing inference speed (1.5-3.6× speedup), reducing latency for real-time applications, or deploying models with limited compute. Covers draft models, tree-based attention, Jacobi iteration, parallel token generation, and production deployment strategies.","Heer-Patel69\u002FAetheris","Heer-Patel69","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F225642641?v=4","01M77YA5MD24QFSR03S12N4VK5","Heer Dipakbhai  Patel ","01M77YA5N2Z9P54X25VFPFMZJP",1,"MIT","https:\u002F\u002Fgithub.com\u002FHeer-Patel69\u002FAetheris\u002Fblob\u002Fmain\u002Fthird party\u002Fclaude-code-templates-main\u002Fclaude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-speculative-decoding\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: speculative-decoding\ndescription: Accelerate LLM inference using speculative decoding, Medusa multiple heads, and lookahead decoding techniques. Use when optimizing inference speed (1.5-3.6× speedup), reducing latency for real-time applications, or deploying models with limited compute. Covers draft models, tree-based attention, Jacobi iteration, parallel token generation, and production deployment strategies.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Emerging Techniques, Speculative Decoding, Medusa, Lookahead Decoding, Fast Inference, Draft Models, Tree Attention, Parallel Generation, Latency Reduction, Inference Optimization]\ndependencies: [transformers, torch]\n---\n\n# Speculative Decoding: Accelerating LLM Inference\n\n## When to Use This Skill\n\nUse Speculative Decoding when you need to:\n- **Speed up inference** by 1.5-3.6× without quality loss\n- **Reduce latency** for real-time applications (chatbots, code generation)\n- **Optimize throughput** for high-volume serving\n- **Deploy efficiently** on limited hardware\n- **Generate faster** without changing model architecture\n\n**Key Techniques**: Draft model speculative decoding, Medusa (multiple heads), Lookahead Decoding (Jacobi iteration)\n\n**Papers**: Medusa (arXiv 2401.10774), Lookahead Decoding (ICML 2024), Speculative Decoding Survey (ACL 2024)\n\n## Installation\n\n```bash\n# Standard speculative decoding (transformers)\npip install transformers accelerate\n\n# Medusa (multiple decoding heads)\ngit clone https:\u002F\u002Fgithub.com\u002FFasterDecoding\u002FMedusa\ncd Medusa\npip install -e .\n\n# Lookahead Decoding\ngit clone https:\u002F\u002Fgithub.com\u002Fhao-ai-lab\u002FLookaheadDecoding\ncd LookaheadDecoding\npip install -e .\n\n# Optional: vLLM with speculative decoding\npip install vllm\n```\n\n## Quick Start\n\n### Basic Speculative Decoding (Draft Model)\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load target model (large, slow)\ntarget_model = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-70b-hf\",\n    device_map=\"auto\",\n    torch_dtype=torch.float16\n)\n\n# Load draft model (small, fast)\ndraft_model = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    device_map=\"auto\",\n    torch_dtype=torch.float16\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-70b-hf\")\n\n# Generate with speculative decoding\nprompt = \"Explain quantum computing in simple terms:\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\n# Transformers 4.36+ supports assisted generation\noutputs = target_model.generate(\n    **inputs,\n    assistant_model=draft_model,  # Enable speculative decoding\n    max_new_tokens=256,\n    do_sample=True,\n    temperature=0.7,\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\nprint(response)\n```\n\n### Medusa (Multiple Decoding Heads)\n\n```python\nfrom medusa.model.medusa_model import MedusaModel\n\n# Load Medusa-enhanced model\nmodel = MedusaModel.from_pretrained(\n    \"FasterDecoding\u002Fmedusa-vicuna-7b-v1.3\",  # Pre-trained with Medusa heads\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"FasterDecoding\u002Fmedusa-vicuna-7b-v1.3\")\n\n# Generate with Medusa (2-3× speedup)\nprompt = \"Write a Python function to calculate fibonacci numbers:\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.medusa_generate(\n    **inputs,\n    max_new_tokens=256,\n    temperature=0.7,\n    posterior_threshold=0.09,  # Acceptance threshold\n    posterior_alpha=0.3,       # Tree construction parameter\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\n```\n\n### Lookahead Decoding (Jacobi Iteration)\n\n```python\nfrom lookahead.lookahead_decoding import LookaheadDecoding\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n\n# Initialize lookahead decoding\nlookahead = LookaheadDecoding(\n    model=model,\n    tokenizer=tokenizer,\n    window_size=15,    # Lookahead window (W)\n    ngram_size=5,      # N-gram size (N)\n    guess_size=5       # Number of parallel guesses\n)\n\n# Generate (1.5-2.3× speedup)\nprompt = \"Implement quicksort in Python:\"\noutput = lookahead.generate(prompt, max_new_tokens=256)\nprint(output)\n```\n\n## Core Concepts\n\n### 1. Speculative Decoding (Draft Model)\n\n**Idea**: Use small draft model to generate candidates, large target model to verify in parallel.\n\n**Algorithm**:\n1. Draft model generates K tokens speculatively\n2. Target model evaluates all K tokens in parallel (single forward pass)\n3. Accept tokens where draft and target agree\n4. Reject first disagreement, continue from there\n\n```python\ndef speculative_decode(target_model, draft_model, prompt, K=4):\n    \"\"\"Speculative decoding algorithm.\"\"\"\n    # 1. Generate K draft tokens\n    draft_tokens = draft_model.generate(prompt, max_new_tokens=K)\n\n    # 2. Target model evaluates all K tokens in one forward pass\n    target_logits = target_model(draft_tokens)  # Parallel!\n\n    # 3. Accept\u002Freject based on probability match\n    accepted = []\n    for i in range(K):\n        p_draft = softmax(draft_model.logits[i])\n        p_target = softmax(target_logits[i])\n\n        # Acceptance probability\n        if random.random() \u003C min(1, p_target[draft_tokens[i]] \u002F p_draft[draft_tokens[i]]):\n            accepted.append(draft_tokens[i])\n        else:\n            break  # Reject, resample from target\n\n    return accepted\n```\n\n**Performance**:\n- Speedup: 1.5-2× with good draft model\n- Zero quality loss (mathematically equivalent to target model)\n- Best when draft model is 5-10× smaller than target\n\n### 2. Medusa (Multiple Decoding Heads)\n\n**Source**: arXiv 2401.10774 (2024)\n\n**Innovation**: Add multiple prediction heads to existing model, predict future tokens without separate draft model.\n\n**Architecture**:\n```\nInput → Base LLM (frozen) → Hidden State\n                                ├→ Head 1 (predicts token t+1)\n                                ├→ Head 2 (predicts token t+2)\n                                ├→ Head 3 (predicts token t+3)\n                                └→ Head 4 (predicts token t+4)\n```\n\n**Training**:\n- **Medusa-1**: Freeze base LLM, train only heads\n  - 2.2× speedup, lossless\n- **Medusa-2**: Fine-tune base LLM + heads together\n  - 2.3-3.6× speedup, better quality\n\n**Tree-based Attention**:\n```python\n# Medusa constructs tree of candidates\n# Example: Predict 2 steps ahead with top-2 per step\n\n#         Root\n#        \u002F    \\\n#      T1a    T1b  (Step 1: 2 candidates)\n#     \u002F  \\    \u002F \\\n#  T2a  T2b T2c T2d  (Step 2: 4 candidates total)\n\n# Single forward pass evaluates entire tree!\n```\n\n**Advantages**:\n- No separate draft model needed\n- Minimal training (only heads)\n- Compatible with any LLM\n\n### 3. Lookahead Decoding (Jacobi Iteration)\n\n**Source**: ICML 2024\n\n**Core idea**: Reformulate autoregressive decoding as solving system of equations, solve in parallel using Jacobi iteration.\n\n**Mathematical formulation**:\n```\nTraditional:  y_t = f(x, y_1, ..., y_{t-1})  (sequential)\nJacobi:       y_t^{(k+1)} = f(x, y_1^{(k)}, ..., y_{t-1}^{(k)})  (parallel)\n```\n\n**Two branches**:\n\n1. **Lookahead Branch**: Generate n-grams in parallel\n   - Window size W: How many steps to look ahead\n   - N-gram size N: How many past tokens to use\n\n2. **Verification Branch**: Verify promising n-grams\n   - Match n-grams with generated tokens\n   - Accept if first token matches\n\n```python\nclass LookaheadDecoding:\n    def __init__(self, model, window_size=15, ngram_size=5):\n        self.model = model\n        self.W = window_size  # Lookahead window\n        self.N = ngram_size   # N-gram size\n\n    def generate_step(self, tokens):\n        # Lookahead branch: Generate W × N candidates\n        candidates = {}\n        for w in range(1, self.W + 1):\n            for n in range(1, self.N + 1):\n                # Generate n-gram starting at position w\n                ngram = self.generate_ngram(tokens, start=w, length=n)\n                candidates[(w, n)] = ngram\n\n        # Verification branch: Find matching n-grams\n        verified = []\n        for ngram in candidates.values():\n            if ngram[0] == tokens[-1]:  # First token matches last input\n                if self.verify(tokens, ngram):\n                    verified.append(ngram)\n\n        # Accept longest verified n-gram\n        return max(verified, key=len) if verified else [self.model.generate_next(tokens)]\n```\n\n**Performance**:\n- Speedup: 1.5-2.3× (up to 3.6× for code generation)\n- No draft model or training needed\n- Works out-of-the-box with any model\n\n## Method Comparison\n\n| Method | Speedup | Training Needed | Draft Model | Quality Loss |\n|--------|---------|-----------------|-------------|--------------|\n| **Draft Model Speculative** | 1.5-2× | No | Yes (external) | None |\n| **Medusa** | 2-3.6× | Minimal (heads only) | No (built-in heads) | None |\n| **Lookahead** | 1.5-2.3× | None | No | None |\n| **Naive Batching** | 1.2-1.5× | No | No | None |\n\n## Advanced Patterns\n\n### Training Medusa Heads\n\n```python\nfrom medusa.model.medusa_model import MedusaModel\nfrom medusa.model.kv_cache import initialize_past_key_values\nimport torch.nn as nn\n\n# 1. Load base model\nbase_model = AutoModelForCausalLM.from_pretrained(\n    \"lmsys\u002Fvicuna-7b-v1.3\",\n    torch_dtype=torch.float16\n)\n\n# 2. Add Medusa heads\nnum_heads = 4\nmedusa_heads = nn.ModuleList([\n    nn.Linear(base_model.config.hidden_size, base_model.config.vocab_size, bias=False)\n    for _ in range(num_heads)\n])\n\n# 3. Training loop (freeze base model for Medusa-1)\nfor param in base_model.parameters():\n    param.requires_grad = False  # Freeze base\n\noptimizer = torch.optim.Adam(medusa_heads.parameters(), lr=1e-3)\n\nfor batch in dataloader:\n    # Forward pass\n    hidden_states = base_model(**batch, output_hidden_states=True).hidden_states[-1]\n\n    # Predict future tokens with each head\n    loss = 0\n    for i, head in enumerate(medusa_heads):\n        logits = head(hidden_states)\n        # Target: tokens shifted by (i+1) positions\n        target = batch['input_ids'][:, i+1:]\n        loss += F.cross_entropy(logits[:, :-i-1], target)\n\n    # Backward\n    optimizer.zero_grad()\n    loss.backward()\n    optimizer.step()\n```\n\n### Hybrid: Speculative + Medusa\n\n```python\n# Use Medusa as draft model for speculative decoding\ndraft_medusa = MedusaModel.from_pretrained(\"medusa-vicuna-7b\")\ntarget_model = AutoModelForCausalLM.from_pretrained(\"vicuna-33b\")\n\n# Draft generates multiple candidates with Medusa\ndraft_tokens = draft_medusa.medusa_generate(prompt, max_new_tokens=5)\n\n# Target verifies in single forward pass\noutputs = target_model.generate(\n    prompt,\n    assistant_model=draft_medusa,  # Use Medusa as draft\n    max_new_tokens=256\n)\n\n# Combines benefits: Medusa speed + large model quality\n```\n\n### Optimal Draft Model Selection\n\n```python\ndef select_draft_model(target_model_size, target):\n    \"\"\"Select optimal draft model for speculative decoding.\"\"\"\n    # Rule: Draft should be 5-10× smaller\n    if target_model_size == \"70B\":\n        return \"7B\"  # 10× smaller\n    elif target_model_size == \"33B\":\n        return \"7B\"  # 5× smaller\n    elif target_model_size == \"13B\":\n        return \"1B\"  # 13× smaller\n    else:\n        return None  # Target too small, use Medusa\u002FLookahead instead\n\n# Example\ndraft = select_draft_model(\"70B\", target_model)\n# Returns \"7B\" → Use Llama-2-7b as draft for Llama-2-70b\n```\n\n## Best Practices\n\n### 1. Choose the Right Method\n\n```python\n# New deployment → Medusa (best overall speedup, no draft model)\nif deploying_new_model:\n    use_method = \"Medusa\"\n\n# Existing deployment with small model available → Draft speculative\nelif have_small_version_of_model:\n    use_method = \"Draft Model Speculative\"\n\n# Want zero training\u002Fsetup → Lookahead\nelif want_plug_and_play:\n    use_method = \"Lookahead Decoding\"\n```\n\n### 2. Hyperparameter Tuning\n\n**Draft Model Speculative**:\n```python\n# K = number of speculative tokens\nK = 4  # Good default\nK = 2  # Conservative (higher acceptance)\nK = 8  # Aggressive (lower acceptance, but more when accepted)\n\n# Rule: Larger K → more speedup IF draft model is good\n```\n\n**Medusa**:\n```python\n# Posterior threshold (acceptance confidence)\nposterior_threshold = 0.09  # Standard (from paper)\nposterior_threshold = 0.05  # More conservative (slower, higher quality)\nposterior_threshold = 0.15  # More aggressive (faster, may degrade quality)\n\n# Tree depth (how many steps ahead)\nmedusa_choices = [[0], [0, 0], [0, 1], [0, 0, 0]]  # Depth 3 (standard)\n```\n\n**Lookahead**:\n```python\n# Window size W (lookahead distance)\n# N-gram size N (context for generation)\n\n# 7B model (more resources)\nW, N = 15, 5\n\n# 13B model (moderate)\nW, N = 10, 5\n\n# 33B+ model (limited resources)\nW, N = 7, 5\n```\n\n### 3. Production Deployment\n\n```python\n# vLLM with speculative decoding\nfrom vllm import LLM, SamplingParams\n\n# Initialize with draft model\nllm = LLM(\n    model=\"meta-llama\u002FLlama-2-70b-hf\",\n    speculative_model=\"meta-llama\u002FLlama-2-7b-hf\",  # Draft model\n    num_speculative_tokens=5,\n    use_v2_block_manager=True,\n)\n\n# Generate\nprompts = [\"Tell me about AI:\", \"Explain quantum physics:\"]\nsampling_params = SamplingParams(temperature=0.7, max_tokens=256)\n\noutputs = llm.generate(prompts, sampling_params)\nfor output in outputs:\n    print(output.outputs[0].text)\n```\n\n## Resources\n\n- **Medusa Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.10774\n- **Medusa GitHub**: https:\u002F\u002Fgithub.com\u002FFasterDecoding\u002FMedusa\n- **Lookahead Decoding (ICML 2024)**: https:\u002F\u002Flmsys.org\u002Fblog\u002F2023-11-21-lookahead-decoding\u002F\n- **Lookahead GitHub**: https:\u002F\u002Fgithub.com\u002Fhao-ai-lab\u002FLookaheadDecoding\n- **Speculative Decoding Survey (ACL 2024)**: https:\u002F\u002Faclanthology.org\u002F2024.findings-acl.456.pdf\n- **Comprehensive Survey**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.07851\n\n## See Also\n\n- `references\u002Fdraft_model.md` - Draft model selection and training\n- `references\u002Fmedusa.md` - Medusa architecture and training\n- `references\u002Flookahead.md` - Lookahead decoding implementation details\n\n\n","# Speculative Decoding: Accelerating LLM Inference\n\n## When to Use This Skill\n\nUse Speculative Decoding when you need to:\n- **Speed up inference** by 1.5-3.6× without quality loss\n- **Reduce latency** for real-time applications (chatbots, code generation)\n- **Optimize throughput** for high-volume serving\n- **Deploy efficiently** on limited hardware\n- **Generate faster** without changing model architecture\n\n**Key Techniques**: Draft model speculative decoding, Medusa (multiple heads), Lookahead Decoding (Jacobi iteration)\n\n**Papers**: Medusa (arXiv 2401.10774), Lookahead Decoding (ICML 2024), Speculative Decoding Survey (ACL 2024)\n\n## Installation\n\n```bash\n# Standard speculative decoding (transformers)\npip install transformers accelerate\n\n# Medusa (multiple decoding heads)\ngit clone https:\u002F\u002Fgithub.com\u002FFasterDecoding\u002FMedusa\ncd Medusa\npip install -e .\n\n# Lookahead Decoding\ngit clone https:\u002F\u002Fgithub.com\u002Fhao-ai-lab\u002FLookaheadDecoding\ncd LookaheadDecoding\npip install -e .\n\n# Optional: vLLM with speculative decoding\npip install vllm\n```\n\n## Quick Start\n\n### Basic Speculative Decoding (Draft Model)\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load target model (large, slow)\ntarget_model = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-70b-hf\",\n    device_map=\"auto\",\n    torch_dtype=torch.float16\n)\n\n# Load draft model (small, fast)\ndraft_model = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    device_map=\"auto\",\n    torch_dtype=torch.float16\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-70b-hf\")\n\n# Generate with speculative decoding\nprompt = \"Explain quantum computing in simple terms:\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\n# Transformers 4.36+ supports assisted generation\noutputs = target_model.generate(\n    **inputs,\n    assistant_model=draft_model,  # Enable speculative decoding\n    max_new_tokens=256,\n    do_sample=True,\n    temperature=0.7,\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\nprint(response)\n```\n\n### Medusa (Multiple Decoding Heads)\n\n```python\nfrom medusa.model.medusa_model import MedusaModel\n\n# Load Medusa-enhanced model\nmodel = MedusaModel.from_pretrained(\n    \"FasterDecoding\u002Fmedusa-vicuna-7b-v1.3\",  # Pre-trained with Medusa heads\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"FasterDecoding\u002Fmedusa-vicuna-7b-v1.3\")\n\n# Generate with Medusa (2-3× speedup)\nprompt = \"Write a Python function to calculate fibonacci numbers:\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.medusa_generate(\n    **inputs,\n    max_new_tokens=256,\n    temperature=0.7,\n    posterior_threshold=0.09,  # Acceptance threshold\n    posterior_alpha=0.3,       # Tree construction parameter\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\n```\n\n### Lookahead Decoding (Jacobi Iteration)\n\n```python\nfrom lookahead.lookahead_decoding import LookaheadDecoding\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n\n# Initialize lookahead decoding\nlookahead = LookaheadDecoding(\n    model=model,\n    tokenizer=tokenizer,\n    window_size=15,    # Lookahead window (W)\n    ngram_size=5,      # N-gram size (N)\n    guess_size=5       # Number of parallel guesses\n)\n\n# Generate (1.5-2.3× speedup)\nprompt = \"Implement quicksort in Python:\"\noutput = lookahead.generate(prompt, max_new_tokens=256)\nprint(output)\n```\n\n## Core Concepts\n\n### 1. Speculative Decoding (Draft Model)\n\n**Idea**: Use small draft model to generate candidates, large target model to verify in parallel.\n\n**Algorithm**:\n1. Draft model generates K tokens speculatively\n2. Target model evaluates all K tokens in parallel (single forward pass)\n3. Accept tokens where draft and target agree\n4. Reject first disagreement, continue from there\n\n```python\ndef speculative_decode(target_model, draft_model, prompt, K=4):\n    \"\"\"Speculative decoding algorithm.\"\"\"\n    # 1. Generate K draft tokens\n    draft_tokens = draft_model.generate(prompt, max_new_tokens=K)\n\n    # 2. Target model evaluates all K tokens in one forward pass\n    target_logits = target_model(draft_tokens)  # Parallel!\n\n    # 3. Accept\u002Freject based on probability match\n    accepted = []\n    for i in range(K):\n        p_draft = softmax(draft_model.logits[i])\n        p_target = softmax(target_logits[i])\n\n        # Acceptance probability\n        if random.random() \u003C min(1, p_target[draft_tokens[i]] \u002F p_draft[draft_tokens[i]]):\n            accepted.append(draft_tokens[i])\n        else:\n            break  # Reject, resample from target\n\n    return accepted\n```\n\n**Performance**:\n- Speedup: 1.5-2× with good draft model\n- Zero quality loss (mathematically equivalent to target model)\n- Best when draft model is 5-10× smaller than target\n\n### 2. Medusa (Multiple Decoding Heads)\n\n**Source**: arXiv 2401.10774 (2024)\n\n**Innovation**: Add multiple prediction heads to existing model, predict future tokens without separate draft model.\n\n**Architecture**:\n```\nInput → Base LLM (frozen) → Hidden State\n                                ├→ Head 1 (predicts token t+1)\n                                ├→ Head 2 (predicts token t+2)\n                                ├→ Head 3 (predicts token t+3)\n                                └→ Head 4 (predicts token t+4)\n```\n\n**Training**:\n- **Medusa-1**: Freeze base LLM, train only heads\n  - 2.2× speedup, lossless\n- **Medusa-2**: Fine-tune base LLM + heads together\n  - 2.3-3.6× speedup, better quality\n\n**Tree-based Attention**:\n```python\n# Medusa constructs tree of candidates\n# Example: Predict 2 steps ahead with top-2 per step\n\n#         Root\n#        \u002F    \\\n#      T1a    T1b  (Step 1: 2 candidates)\n#     \u002F  \\    \u002F \\\n#  T2a  T2b T2c T2d  (Step 2: 4 candidates total)\n\n# Single forward pass evaluates entire tree!\n```\n\n**Advantages**:\n- No separate draft model needed\n- Minimal training (only heads)\n- Compatible with any LLM\n\n### 3. Lookahead Decoding (Jacobi Iteration)\n\n**Source**: ICML 2024\n\n**Core idea**: Reformulate autoregressive decoding as solving system of equations, solve in parallel using Jacobi iteration.\n\n**Mathematical formulation**:\n```\nTraditional:  y_t = f(x, y_1, ..., y_{t-1})  (sequential)\nJacobi:       y_t^{(k+1)} = f(x, y_1^{(k)}, ..., y_{t-1}^{(k)})  (parallel)\n```\n\n**Two branches**:\n\n1. **Lookahead Branch**: Generate n-grams in parallel\n   - Window size W: How many steps to look ahead\n   - N-gram size N: How many past tokens to use\n\n2. **Verification Branch**: Verify promising n-grams\n   - Match n-grams with generated tokens\n   - Accept if first token matches\n\n```python\nclass LookaheadDecoding:\n    def __init__(self, model, window_size=15, ngram_size=5):\n        self.model = model\n        self.W = window_size  # Lookahead window\n        self.N = ngram_size   # N-gram size\n\n    def generate_step(self, tokens):\n        # Lookahead branch: Generate W × N candidates\n        candidates = {}\n        for w in range(1, self.W + 1):\n            for n in range(1, self.N + 1):\n                # Generate n-gram starting at position w\n                ngram = self.generate_ngram(tokens, start=w, length=n)\n                candidates[(w, n)] = ngram\n\n        # Verification branch: Find matching n-grams\n        verified = []\n        for ngram in candidates.values():\n            if ngram[0] == tokens[-1]:  # First token matches last input\n                if self.verify(tokens, ngram):\n                    verified.append(ngram)\n\n        # Accept longest verified n-gram\n        return max(verified, key=len) if verified else [self.model.generate_next(tokens)]\n```\n\n**Performance**:\n- Speedup: 1.5-2.3× (up to 3.6× for code generation)\n- No draft model or training needed\n- Works out-of-the-box with any model\n\n## Method Comparison\n\n| Method | Speedup | Training Needed | Draft Model | Quality Loss |\n|--------|---------|-----------------|-------------|--------------|\n| **Draft Model Speculative** | 1.5-2× | No | Yes (external) | None |\n| **Medusa** | 2-3.6× | Minimal (heads only) | No (built-in heads) | None |\n| **Lookahead** | 1.5-2.3× | None | No | None |\n| **Naive Batching** | 1.2-1.5× | No | No | None |\n\n## Advanced Patterns\n\n### Training Medusa Heads\n\n```python\nfrom medusa.model.medusa_model import MedusaModel\nfrom medusa.model.kv_cache import initialize_past_key_values\nimport torch.nn as nn\n\n# 1. Load base model\nbase_model = AutoModelForCausalLM.from_pretrained(\n    \"lmsys\u002Fvicuna-7b-v1.3\",\n    torch_dtype=torch.float16\n)\n\n# 2. Add Medusa heads\nnum_heads = 4\nmedusa_heads = nn.ModuleList([\n    nn.Linear(base_model.config.hidden_size, base_model.config.vocab_size, bias=False)\n    for _ in range(num_heads)\n])\n\n# 3. Training loop (freeze base model for Medusa-1)\nfor param in base_model.parameters():\n    param.requires_grad = False  # Freeze base\n\noptimizer = torch.optim.Adam(medusa_heads.parameters(), lr=1e-3)\n\nfor batch in dataloader:\n    # Forward pass\n    hidden_states = base_model(**batch, output_hidden_states=True).hidden_states[-1]\n\n    # Predict future tokens with each head\n    loss = 0\n    for i, head in enumerate(medusa_heads):\n        logits = head(hidden_states)\n        # Target: tokens shifted by (i+1) positions\n        target = batch['input_ids'][:, i+1:]\n        loss += F.cross_entropy(logits[:, :-i-1], target)\n\n    # Backward\n    optimizer.zero_grad()\n    loss.backward()\n    optimizer.step()\n```\n\n### Hybrid: Speculative + Medusa\n\n```python\n# Use Medusa as draft model for speculative decoding\ndraft_medusa = MedusaModel.from_pretrained(\"medusa-vicuna-7b\")\ntarget_model = AutoModelForCausalLM.from_pretrained(\"vicuna-33b\")\n\n# Draft generates multiple candidates with Medusa\ndraft_tokens = draft_medusa.medusa_generate(prompt, max_new_tokens=5)\n\n# Target verifies in single forward pass\noutputs = target_model.generate(\n    prompt,\n    assistant_model=draft_medusa,  # Use Medusa as draft\n    max_new_tokens=256\n)\n\n# Combines benefits: Medusa speed + large model quality\n```\n\n### Optimal Draft Model Selection\n\n```python\ndef select_draft_model(target_model_size, target):\n    \"\"\"Select optimal draft model for speculative decoding.\"\"\"\n    # Rule: Draft should be 5-10× smaller\n    if target_model_size == \"70B\":\n        return \"7B\"  # 10× smaller\n    elif target_model_size == \"33B\":\n        return \"7B\"  # 5× smaller\n    elif target_model_size == \"13B\":\n        return \"1B\"  # 13× smaller\n    else:\n        return None  # Target too small, use Medusa\u002FLookahead instead\n\n# Example\ndraft = select_draft_model(\"70B\", target_model)\n# Returns \"7B\" → Use Llama-2-7b as draft for Llama-2-70b\n```\n\n## Best Practices\n\n### 1. Choose the Right Method\n\n```python\n# New deployment → Medusa (best overall speedup, no draft model)\nif deploying_new_model:\n    use_method = \"Medusa\"\n\n# Existing deployment with small model available → Draft speculative\nelif have_small_version_of_model:\n    use_method = \"Draft Model Speculative\"\n\n# Want zero training\u002Fsetup → Lookahead\nelif want_plug_and_play:\n    use_method = \"Lookahead Decoding\"\n```\n\n### 2. Hyperparameter Tuning\n\n**Draft Model Speculative**:\n```python\n# K = number of speculative tokens\nK = 4  # Good default\nK = 2  # Conservative (higher acceptance)\nK = 8  # Aggressive (lower acceptance, but more when accepted)\n\n# Rule: Larger K → more speedup IF draft model is good\n```\n\n**Medusa**:\n```python\n# Posterior threshold (acceptance confidence)\nposterior_threshold = 0.09  # Standard (from paper)\nposterior_threshold = 0.05  # More conservative (slower, higher quality)\nposterior_threshold = 0.15  # More aggressive (faster, may degrade quality)\n\n# Tree depth (how many steps ahead)\nmedusa_choices = [[0], [0, 0], [0, 1], [0, 0, 0]]  # Depth 3 (standard)\n```\n\n**Lookahead**:\n```python\n# Window size W (lookahead distance)\n# N-gram size N (context for generation)\n\n# 7B model (more resources)\nW, N = 15, 5\n\n# 13B model (moderate)\nW, N = 10, 5\n\n# 33B+ model (limited resources)\nW, N = 7, 5\n```\n\n### 3. Production Deployment\n\n```python\n# vLLM with speculative decoding\nfrom vllm import LLM, SamplingParams\n\n# Initialize with draft model\nllm = LLM(\n    model=\"meta-llama\u002FLlama-2-70b-hf\",\n    speculative_model=\"meta-llama\u002FLlama-2-7b-hf\",  # Draft model\n    num_speculative_tokens=5,\n    use_v2_block_manager=True,\n)\n\n# Generate\nprompts = [\"Tell me about AI:\", \"Explain quantum physics:\"]\nsampling_params = SamplingParams(temperature=0.7, max_tokens=256)\n\noutputs = llm.generate(prompts, sampling_params)\nfor output in outputs:\n    print(output.outputs[0].text)\n```\n\n## Resources\n\n- **Medusa Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.10774\n- **Medusa GitHub**: https:\u002F\u002Fgithub.com\u002FFasterDecoding\u002FMedusa\n- **Lookahead Decoding (ICML 2024)**: https:\u002F\u002Flmsys.org\u002Fblog\u002F2023-11-21-lookahead-decoding\u002F\n- **Lookahead GitHub**: https:\u002F\u002Fgithub.com\u002Fhao-ai-lab\u002FLookaheadDecoding\n- **Speculative Decoding Survey (ACL 2024)**: https:\u002F\u002Faclanthology.org\u002F2024.findings-acl.456.pdf\n- **Comprehensive Survey**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.07851\n\n## See Also\n\n- `references\u002Fdraft_model.md` - Draft model selection and training\n- `references\u002Fmedusa.md` - Medusa architecture and training\n- `references\u002Flookahead.md` - Lookahead decoding implementation details\n\n\n",{"name":5,"tags":24,"author":35,"license":15,"version":36,"description":7,"dependencies":37},[25,26,27,28,29,30,31,32,33,34],"Emerging Techniques","Speculative Decoding","Medusa","Lookahead Decoding","Fast Inference","Draft Models","Tree Attention","Parallel Generation","Latency Reduction","Inference Optimization","Orchestra Research","1.0.0",[38,39],"transformers","torch","main","0edefbe9101b21ed9b961f914c13ae42b268e2a4","third party\u002Fclaude-code-templates-main\u002Fclaude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-speculative-decoding",0,"https:\u002F\u002Fgithub.com\u002FHeer-Patel69\u002FAetheris","https:\u002F\u002Fgithub.com\u002FHeer-Patel69","2026-07-19 01:50:57",{"source":8,"skillName":5,"commands":48,"prompt":58,"zipAvailable":59},[49,52,55],{"tool":50,"command":51},"npx","npx skills add Heer-Patel69\u002FAetheris --skill speculative-decoding",{"tool":53,"command":54},"bunx","bunx skills add Heer-Patel69\u002FAetheris --skill speculative-decoding",{"tool":56,"command":57},"pnpm","pnpm dlx skills add Heer-Patel69\u002FAetheris --skill speculative-decoding","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FHeer-Patel69\u002FAetheris\u002Fblob\u002Fmain\u002Fthird party\u002Fclaude-code-templates-main\u002Fclaude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-speculative-decoding\u002FSKILL.md",true,[61,67],{"path":62,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":65,"sourceHtmlUrl":66},"references\u002Flookahead.md","REFERENCE","参考文档",8890,"https:\u002F\u002Fgithub.com\u002FHeer-Patel69\u002FAetheris\u002Fblob\u002Fmain\u002Fthird party\u002Fclaude-code-templates-main\u002Fclaude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-speculative-decoding\u002Freferences\u002Flookahead.md",{"path":68,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":69,"sourceHtmlUrl":70},"references\u002Fmedusa.md",9851,"https:\u002F\u002Fgithub.com\u002FHeer-Patel69\u002FAetheris\u002Fblob\u002Fmain\u002Fthird party\u002Fclaude-code-templates-main\u002Fclaude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-speculative-decoding\u002Freferences\u002Fmedusa.md",[72,81,88,95,102,109],{"skillNo":73,"name":74,"nameZh":6,"description":75,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":76,"licenseFileUrl":77,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":19},"01M786K8RTQF44DN3R4DQRNHH0","yeet","Use only when the user explicitly asks to stage, commit, push, and open a GitHub pull request in one flow using the GitHub CLI (`gh`).","Apache-2.0","https:\u002F\u002Fgithub.com\u002FHeer-Patel69\u002FAetheris\u002Fblob\u002Fmain\u002Fthird party\u002Fclaude-code-templates-main\u002Fclaude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fyeet\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FHeer-Patel69\u002FAetheris\u002Fblob\u002Fmain\u002Fthird party\u002Fclaude-code-templates-main\u002Fclaude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fyeet\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M786FRBBC04CFM75BAVZK2DG","dependabot-review","Review and manage Dependabot PRs. Categorizes by risk, checks CI status, auto-merges safe updates, and reports issues. Use when the user says \"review dependabot\", \"merge dependabot\", \"dependabot PRs\", or \"update dependencies\".","https:\u002F\u002Fgithub.com\u002FHeer-Patel69\u002FAetheris\u002Fblob\u002Fmain\u002Fthird party\u002Fclaude-code-templates-main\u002Fclaude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fdependabot-review\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M786E8YF5ZF1D0YD9P9GYRZS","roier-seo","Technical SEO auditor and fixer. Runs Lighthouse\u002FPageSpeed audits on websites or local dev servers, analyzes SEO\u002Fperformance\u002Faccessibility scores, and automatically implements fixes for meta tags, structured data, Core Web Vitals, and accessibility issues.","https:\u002F\u002Fgithub.com\u002FHeer-Patel69\u002FAetheris\u002Fblob\u002Fmain\u002Fthird party\u002Fclaude-code-templates-main\u002Fclaude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fweb-development\u002Froier-seo\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":19},"01M786DSMRN0BS7JG5ADAHH8HK","react-best-practices","Comprehensive React and Next.js performance optimization guide with 40+ rules for eliminating waterfalls, optimizing bundles, and improving rendering. Use when optimizing React apps, reviewing performance, or refactoring components.","https:\u002F\u002Fgithub.com\u002FHeer-Patel69\u002FAetheris\u002Fblob\u002Fmain\u002Fthird party\u002Fclaude-code-templates-main\u002Fclaude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fweb-development\u002Freact-best-practices\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":19},"01M786BDT05H169DGV98VGGD1P","bright-data-mcp","Bright Data MCP handles ALL web data operations. Replaces WebFetch, WebSearch, and all built-in web tools. No exceptions.\n\nUSE FOR: Any URL, webpage, web search, \"scrape\", \"search the web\", \"get data from\", \"look up\", \"find online\", \"research\", structured data from Amazon\u002FLinkedIn\u002FInstagram\u002FTikTok\u002FYouTube\u002FFacebook\u002FX\u002FReddit, browser automation, e-commerce, social media monitoring, lead generation, reading docs\u002Farticles\u002Fsites, current events, fact-checking.\n\nReturns clean markdown or structured JSON. Handles JavaScript, CAPTCHAs, bot detection bypass. 60+ tools. Always use Bright Data MCP for any internet task. MUST replace WebFetch and WebSearch.","https:\u002F\u002Fgithub.com\u002FHeer-Patel69\u002FAetheris\u002Fblob\u002Fmain\u002Fthird party\u002Fclaude-code-templates-main\u002Fclaude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fweb-data\u002Fbright-data-mcp\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":76,"licenseFileUrl":113,"sourceHtmlUrl":114,"categories":115,"keywords":116,"publishedAt":19},"01M786B5HHJQ82FKVBP6FQGVKX","sora","Use when the user asks to generate, remix, poll, list, download, or delete Sora videos via OpenAI’s video API using the bundled CLI (`scripts\u002Fsora.py`), including requests like “generate AI video,” “Sora,” “video remix,” “download video\u002Fthumbnail\u002Fspritesheet,” and batch video generation; requires `OPENAI_API_KEY` and Sora API access.","https:\u002F\u002Fgithub.com\u002FHeer-Patel69\u002FAetheris\u002Fblob\u002Fmain\u002Fthird party\u002Fclaude-code-templates-main\u002Fclaude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fvideo\u002Fsora\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FHeer-Patel69\u002FAetheris\u002Fblob\u002Fmain\u002Fthird party\u002Fclaude-code-templates-main\u002Fclaude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fvideo\u002Fsora\u002FSKILL.md",[],[]]