[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EDTY5PS4F1HWTXYDACDBN6":3,"skill-same:01M6EDTY5PS4F1HWTXYDACDBN6":46},{"skillNo":4,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":21,"frontMatter":22,"refName":23,"commitSha":24,"skillDirectory":25,"forksCount":26,"repositoryHtmlUrl":27,"repositoryHomepageUrl":28,"ownerHtmlUrl":29,"ownerBlogUrl":6,"ownerDescription":30,"updatedAt":31,"install":32,"resources":45},"01M6EDTY5PS4F1HWTXYDACDBN6","speculative-decoding",null,"ComeOnOliver\u002Fskillshub","ComeOnOliver","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F43215937?v=4","01M6E5JNFECJAAVFE0W7SMWWD2","Haoran Shu","01M6E5JNNDQS7AJZDYFQ8TPZCK",67,"MIT","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002FOrchestra-Research\u002FAI-Research-SKILLs\u002Fspeculative-decoding\u002FSKILL.md",[],[18],"speculative decoding","2026-11-10 10:12:30",false,"# Speculative Decoding: Accelerating LLM Inference\n\n## When to Use This Skill\n\nUse Speculative Decoding when you need to:\n- **Speed up inference** by 1.5-3.6× without quality loss\n- **Reduce latency** for real-time applications (chatbots, code generation)\n- **Optimize throughput** for high-volume serving\n- **Deploy efficiently** on limited hardware\n- **Generate faster** without changing model architecture\n\n**Key Techniques**: Draft model speculative decoding, Medusa (multiple heads), Lookahead Decoding (Jacobi iteration)\n\n**Papers**: Medusa (arXiv 2401.10774), Lookahead Decoding (ICML 2024), Speculative Decoding Survey (ACL 2024)\n\n## Installation\n\n```bash\n# Standard speculative decoding (transformers)\npip install transformers accelerate\n\n# Medusa (multiple decoding heads)\ngit clone https:\u002F\u002Fgithub.com\u002FFasterDecoding\u002FMedusa\ncd Medusa\npip install -e .\n\n# Lookahead Decoding\ngit clone https:\u002F\u002Fgithub.com\u002Fhao-ai-lab\u002FLookaheadDecoding\ncd LookaheadDecoding\npip install -e .\n\n# Optional: vLLM with speculative decoding\npip install vllm\n```\n\n## Quick Start\n\n### Basic Speculative Decoding (Draft Model)\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load target model (large, slow)\ntarget_model = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-70b-hf\",\n    device_map=\"auto\",\n    torch_dtype=torch.float16\n)\n\n# Load draft model (small, fast)\ndraft_model = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    device_map=\"auto\",\n    torch_dtype=torch.float16\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-70b-hf\")\n\n# Generate with speculative decoding\nprompt = \"Explain quantum computing in simple terms:\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\n# Transformers 4.36+ supports assisted generation\noutputs = target_model.generate(\n    **inputs,\n    assistant_model=draft_model,  # Enable speculative decoding\n    max_new_tokens=256,\n    do_sample=True,\n    temperature=0.7,\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\nprint(response)\n```\n\n### Medusa (Multiple Decoding Heads)\n\n```python\nfrom medusa.model.medusa_model import MedusaModel\n\n# Load Medusa-enhanced model\nmodel = MedusaModel.from_pretrained(\n    \"FasterDecoding\u002Fmedusa-vicuna-7b-v1.3\",  # Pre-trained with Medusa heads\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"FasterDecoding\u002Fmedusa-vicuna-7b-v1.3\")\n\n# Generate with Medusa (2-3× speedup)\nprompt = \"Write a Python function to calculate fibonacci numbers:\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda\")\n\noutputs = model.medusa_generate(\n    **inputs,\n    max_new_tokens=256,\n    temperature=0.7,\n    posterior_threshold=0.09,  # Acceptance threshold\n    posterior_alpha=0.3,       # Tree construction parameter\n)\n\nresponse = tokenizer.decode(outputs[0], skip_special_tokens=True)\n```\n\n### Lookahead Decoding (Jacobi Iteration)\n\n```python\nfrom lookahead.lookahead_decoding import LookaheadDecoding\n\n# Load model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-7b-hf\")\n\n# Initialize lookahead decoding\nlookahead = LookaheadDecoding(\n    model=model,\n    tokenizer=tokenizer,\n    window_size=15,    # Lookahead window (W)\n    ngram_size=5,      # N-gram size (N)\n    guess_size=5       # Number of parallel guesses\n)\n\n# Generate (1.5-2.3× speedup)\nprompt = \"Implement quicksort in Python:\"\noutput = lookahead.generate(prompt, max_new_tokens=256)\nprint(output)\n```\n\n## Core Concepts\n\n### 1. Speculative Decoding (Draft Model)\n\n**Idea**: Use small draft model to generate candidates, large target model to verify in parallel.\n\n**Algorithm**:\n1. Draft model generates K tokens speculatively\n2. Target model evaluates all K tokens in parallel (single forward pass)\n3. Accept tokens where draft and target agree\n4. Reject first disagreement, continue from there\n\n```python\ndef speculative_decode(target_model, draft_model, prompt, K=4):\n    \"\"\"Speculative decoding algorithm.\"\"\"\n    # 1. Generate K draft tokens\n    draft_tokens = draft_model.generate(prompt, max_new_tokens=K)\n\n    # 2. Target model evaluates all K tokens in one forward pass\n    target_logits = target_model(draft_tokens)  # Parallel!\n\n    # 3. Accept\u002Freject based on probability match\n    accepted = []\n    for i in range(K):\n        p_draft = softmax(draft_model.logits[i])\n        p_target = softmax(target_logits[i])\n\n        # Acceptance probability\n        if random.random() \u003C min(1, p_target[draft_tokens[i]] \u002F p_draft[draft_tokens[i]]):\n            accepted.append(draft_tokens[i])\n        else:\n            break  # Reject, resample from target\n\n    return accepted\n```\n\n**Performance**:\n- Speedup: 1.5-2× with good draft model\n- Zero quality loss (mathematically equivalent to target model)\n- Best when draft model is 5-10× smaller than target\n\n### 2. Medusa (Multiple Decoding Heads)\n\n**Source**: arXiv 2401.10774 (2024)\n\n**Innovation**: Add multiple prediction heads to existing model, predict future tokens without separate draft model.\n\n**Architecture**:\n```\nInput → Base LLM (frozen) → Hidden State\n                                ├→ Head 1 (predicts token t+1)\n                                ├→ Head 2 (predicts token t+2)\n                                ├→ Head 3 (predicts token t+3)\n                                └→ Head 4 (predicts token t+4)\n```\n\n**Training**:\n- **Medusa-1**: Freeze base LLM, train only heads\n  - 2.2× speedup, lossless\n- **Medusa-2**: Fine-tune base LLM + heads together\n  - 2.3-3.6× speedup, better quality\n\n**Tree-based Attention**:\n```python\n# Medusa constructs tree of candidates\n# Example: Predict 2 steps ahead with top-2 per step\n\n#         Root\n#        \u002F    \\\n#      T1a    T1b  (Step 1: 2 candidates)\n#     \u002F  \\    \u002F \\\n#  T2a  T2b T2c T2d  (Step 2: 4 candidates total)\n\n# Single forward pass evaluates entire tree!\n```\n\n**Advantages**:\n- No separate draft model needed\n- Minimal training (only heads)\n- Compatible with any LLM\n\n### 3. Lookahead Decoding (Jacobi Iteration)\n\n**Source**: ICML 2024\n\n**Core idea**: Reformulate autoregressive decoding as solving system of equations, solve in parallel using Jacobi iteration.\n\n**Mathematical formulation**:\n```\nTraditional:  y_t = f(x, y_1, ..., y_{t-1})  (sequential)\nJacobi:       y_t^{(k+1)} = f(x, y_1^{(k)}, ..., y_{t-1}^{(k)})  (parallel)\n```\n\n**Two branches**:\n\n1. **Lookahead Branch**: Generate n-grams in parallel\n   - Window size W: How many steps to look ahead\n   - N-gram size N: How many past tokens to use\n\n2. **Verification Branch**: Verify promising n-grams\n   - Match n-grams with generated tokens\n   - Accept if first token matches\n\n```python\nclass LookaheadDecoding:\n    def __init__(self, model, window_size=15, ngram_size=5):\n        self.model = model\n        self.W = window_size  # Lookahead window\n        self.N = ngram_size   # N-gram size\n\n    def generate_step(self, tokens):\n        # Lookahead branch: Generate W × N candidates\n        candidates = {}\n        for w in range(1, self.W + 1):\n            for n in range(1, self.N + 1):\n                # Generate n-gram starting at position w\n                ngram = self.generate_ngram(tokens, start=w, length=n)\n                candidates[(w, n)] = ngram\n\n        # Verification branch: Find matching n-grams\n        verified = []\n        for ngram in candidates.values():\n            if ngram[0] == tokens[-1]:  # First token matches last input\n                if self.verify(tokens, ngram):\n                    verified.append(ngram)\n\n        # Accept longest verified n-gram\n        return max(verified, key=len) if verified else [self.model.generate_next(tokens)]\n```\n\n**Performance**:\n- Speedup: 1.5-2.3× (up to 3.6× for code generation)\n- No draft model or training needed\n- Works out-of-the-box with any model\n\n## Method Comparison\n\n| Method | Speedup | Training Needed | Draft Model | Quality Loss |\n|--------|---------|-----------------|-------------|--------------|\n| **Draft Model Speculative** | 1.5-2× | No | Yes (external) | None |\n| **Medusa** | 2-3.6× | Minimal (heads only) | No (built-in heads) | None |\n| **Lookahead** | 1.5-2.3× | None | No | None |\n| **Naive Batching** | 1.2-1.5× | No | No | None |\n\n## Advanced Patterns\n\n### Training Medusa Heads\n\n```python\nfrom medusa.model.medusa_model import MedusaModel\nfrom medusa.model.kv_cache import initialize_past_key_values\nimport torch.nn as nn\n\n# 1. Load base model\nbase_model = AutoModelForCausalLM.from_pretrained(\n    \"lmsys\u002Fvicuna-7b-v1.3\",\n    torch_dtype=torch.float16\n)\n\n# 2. Add Medusa heads\nnum_heads = 4\nmedusa_heads = nn.ModuleList([\n    nn.Linear(base_model.config.hidden_size, base_model.config.vocab_size, bias=False)\n    for _ in range(num_heads)\n])\n\n# 3. Training loop (freeze base model for Medusa-1)\nfor param in base_model.parameters():\n    param.requires_grad = False  # Freeze base\n\noptimizer = torch.optim.Adam(medusa_heads.parameters(), lr=1e-3)\n\nfor batch in dataloader:\n    # Forward pass\n    hidden_states = base_model(**batch, output_hidden_states=True).hidden_states[-1]\n\n    # Predict future tokens with each head\n    loss = 0\n    for i, head in enumerate(medusa_heads):\n        logits = head(hidden_states)\n        # Target: tokens shifted by (i+1) positions\n        target = batch['input_ids'][:, i+1:]\n        loss += F.cross_entropy(logits[:, :-i-1], target)\n\n    # Backward\n    optimizer.zero_grad()\n    loss.backward()\n    optimizer.step()\n```\n\n### Hybrid: Speculative + Medusa\n\n```python\n# Use Medusa as draft model for speculative decoding\ndraft_medusa = MedusaModel.from_pretrained(\"medusa-vicuna-7b\")\ntarget_model = AutoModelForCausalLM.from_pretrained(\"vicuna-33b\")\n\n# Draft generates multiple candidates with Medusa\ndraft_tokens = draft_medusa.medusa_generate(prompt, max_new_tokens=5)\n\n# Target verifies in single forward pass\noutputs = target_model.generate(\n    prompt,\n    assistant_model=draft_medusa,  # Use Medusa as draft\n    max_new_tokens=256\n)\n\n# Combines benefits: Medusa speed + large model quality\n```\n\n### Optimal Draft Model Selection\n\n```python\ndef select_draft_model(target_model_size, target):\n    \"\"\"Select optimal draft model for speculative decoding.\"\"\"\n    # Rule: Draft should be 5-10× smaller\n    if target_model_size == \"70B\":\n        return \"7B\"  # 10× smaller\n    elif target_model_size == \"33B\":\n        return \"7B\"  # 5× smaller\n    elif target_model_size == \"13B\":\n        return \"1B\"  # 13× smaller\n    else:\n        return None  # Target too small, use Medusa\u002FLookahead instead\n\n# Example\ndraft = select_draft_model(\"70B\", target_model)\n# Returns \"7B\" → Use Llama-2-7b as draft for Llama-2-70b\n```\n\n## Best Practices\n\n### 1. Choose the Right Method\n\n```python\n# New deployment → Medusa (best overall speedup, no draft model)\nif deploying_new_model:\n    use_method = \"Medusa\"\n\n# Existing deployment with small model available → Draft speculative\nelif have_small_version_of_model:\n    use_method = \"Draft Model Speculative\"\n\n# Want zero training\u002Fsetup → Lookahead\nelif want_plug_and_play:\n    use_method = \"Lookahead Decoding\"\n```\n\n### 2. Hyperparameter Tuning\n\n**Draft Model Speculative**:\n```python\n# K = number of speculative tokens\nK = 4  # Good default\nK = 2  # Conservative (higher acceptance)\nK = 8  # Aggressive (lower acceptance, but more when accepted)\n\n# Rule: Larger K → more speedup IF draft model is good\n```\n\n**Medusa**:\n```python\n# Posterior threshold (acceptance confidence)\nposterior_threshold = 0.09  # Standard (from paper)\nposterior_threshold = 0.05  # More conservative (slower, higher quality)\nposterior_threshold = 0.15  # More aggressive (faster, may degrade quality)\n\n# Tree depth (how many steps ahead)\nmedusa_choices = [[0], [0, 0], [0, 1], [0, 0, 0]]  # Depth 3 (standard)\n```\n\n**Lookahead**:\n```python\n# Window size W (lookahead distance)\n# N-gram size N (context for generation)\n\n# 7B model (more resources)\nW, N = 15, 5\n\n# 13B model (moderate)\nW, N = 10, 5\n\n# 33B+ model (limited resources)\nW, N = 7, 5\n```\n\n### 3. Production Deployment\n\n```python\n# vLLM with speculative decoding\nfrom vllm import LLM, SamplingParams\n\n# Initialize with draft model\nllm = LLM(\n    model=\"meta-llama\u002FLlama-2-70b-hf\",\n    speculative_model=\"meta-llama\u002FLlama-2-7b-hf\",  # Draft model\n    num_speculative_tokens=5,\n    use_v2_block_manager=True,\n)\n\n# Generate\nprompts = [\"Tell me about AI:\", \"Explain quantum physics:\"]\nsampling_params = SamplingParams(temperature=0.7, max_tokens=256)\n\noutputs = llm.generate(prompts, sampling_params)\nfor output in outputs:\n    print(output.outputs[0].text)\n```\n\n## Resources\n\n- **Medusa Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.10774\n- **Medusa GitHub**: https:\u002F\u002Fgithub.com\u002FFasterDecoding\u002FMedusa\n- **Lookahead Decoding (ICML 2024)**: https:\u002F\u002Flmsys.org\u002Fblog\u002F2023-11-21-lookahead-decoding\u002F\n- **Lookahead GitHub**: https:\u002F\u002Fgithub.com\u002Fhao-ai-lab\u002FLookaheadDecoding\n- **Speculative Decoding Survey (ACL 2024)**: https:\u002F\u002Faclanthology.org\u002F2024.findings-acl.456.pdf\n- **Comprehensive Survey**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2401.07851\n\n## See Also\n\n- `references\u002Fdraft_model.md` - Draft model selection and training\n- `references\u002Fmedusa.md` - Medusa architecture and training\n- `references\u002Flookahead.md` - Lookahead decoding implementation details\n",{},"main","def8531e65114c0fca8fb8551c1871ee0eed705c","skills\u002FOrchestra-Research\u002FAI-Research-SKILLs\u002Fspeculative-decoding",24,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub","https:\u002F\u002Fskillshub.wtf","https:\u002F\u002Fgithub.com\u002FComeOnOliver","AI & Blockchain Engineer\r\n","2026-06-24 22:07:19",{"source":7,"skillName":5,"commands":33,"prompt":43,"zipAvailable":44},[34,37,40],{"tool":35,"command":36},"npx","npx skills add ComeOnOliver\u002Fskillshub --skill speculative-decoding",{"tool":38,"command":39},"bunx","bunx skills add ComeOnOliver\u002Fskillshub --skill speculative-decoding",{"tool":41,"command":42},"pnpm","pnpm dlx skills add ComeOnOliver\u002Fskillshub --skill speculative-decoding","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002FOrchestra-Research\u002FAI-Research-SKILLs\u002Fspeculative-decoding\u002FSKILL.md",true,[],[47,61,71,80,86,93],{"skillNo":48,"name":49,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":50,"categories":51,"keywords":57,"publishedAt":19},"01M6FKJN13GMG4KY57J3QD8GY5","model-hierarchy-skill","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzscole\u002Fmodel-hierarchy-skill\u002Fmodel-hierarchy-skill\u002FSKILL.md",[52],{"categoryNo":53,"categoryName":54,"taxonomyType":55,"taxonomyTypeDesc":56},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[58,59,60],"model","model hierarchy","model hierarchy skill",{"skillNo":62,"name":63,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":64,"categories":65,"keywords":67,"publishedAt":19},"01M6FKJJJ6GCAMR4PPPRD12M5X","zoom-mcp","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-mcp\u002FSKILL.md",[66],{"categoryNo":53,"categoryName":54,"taxonomyType":55,"taxonomyTypeDesc":56},[68,69,70],"MCP","zoom","zoom mcp",{"skillNo":72,"name":73,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":19},"01M6FKJE3KYT3KPEZMEEHYKA6K","zoom-apps-sdk","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-apps-sdk\u002FSKILL.md",[],[77,78,79,69],"sdk","zoom apps sdk","apps",{"skillNo":81,"name":82,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":19},"01M6FKJ9EDH9Q5VWDF8HAJRSP0","windows","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwindows\u002FSKILL.md",[],[82],{"skillNo":87,"name":88,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":92,"publishedAt":19},"01M6FKJ666BEEFC3AE9GJ72Q93","whiteboard","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwhiteboard\u002FSKILL.md",[91],{"categoryNo":53,"categoryName":54,"taxonomyType":55,"taxonomyTypeDesc":56},[88],{"skillNo":94,"name":95,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":19},"01M6FKJ21WFCDGTJQ2Q91T9DVW","websockets","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwebsockets\u002FSKILL.md",[],[95]]