[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M72FEKBJ1863VEKAYX27REYV":3,"skill-same:01M72FEKBJ1863VEKAYX27REYV":68},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":41,"commitSha":42,"skillDirectory":43,"forksCount":14,"repositoryHtmlUrl":44,"repositoryHomepageUrl":6,"ownerHtmlUrl":45,"ownerBlogUrl":6,"ownerDescription":46,"updatedAt":47,"install":48,"resources":61},"01M72FEKBJ1863VEKAYX27REYV","knowledge-distillation",null,"Compress large language models using knowledge distillation from teacher to student models. Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs. Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.","cocyuhao\u002Fai-dev-tooling-research-archive","cocyuhao","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F269083740?v=4","01M72F908ET5R76KP9NXS6Y2GJ","Hao Yu","01M72F909376KCNJVKG8MEPY2Q",0,"MIT","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-knowledge-distillation\u002FSKILL.md",[],[],"2026-11-10 18:39:20",false,"---\nname: knowledge-distillation\ndescription: Compress large language models using knowledge distillation from teacher to student models. Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs. Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Emerging Techniques, Knowledge Distillation, Model Compression, Teacher-Student, MiniLLM, Reverse KLD, Soft Targets, Temperature Scaling, Logit Distillation, Model Transfer]\ndependencies: [transformers, torch, datasets]\n---\n\n# Knowledge Distillation: Compressing LLMs\n\n## When to Use This Skill\n\nUse Knowledge Distillation when you need to:\n- **Compress models** from 70B → 7B while retaining 90%+ performance\n- **Transfer capabilities** from proprietary models (GPT-4) to open-source (LLaMA, Mistral)\n- **Reduce inference costs** by deploying smaller student models\n- **Create specialized models** by distilling domain-specific knowledge\n- **Improve small models** using synthetic data from large teachers\n\n**Key Techniques**: Temperature scaling, soft targets, reverse KLD (MiniLLM), logit distillation, response distillation\n\n**Papers**: Hinton et al. 2015 (arXiv 1503.02531), MiniLLM (arXiv 2306.08543), KD Survey (arXiv 2402.13116)\n\n## Installation\n\n```bash\n# Standard transformers\npip install transformers datasets accelerate\n\n# For training\npip install torch deepspeed wandb\n\n# Optional: MiniLLM implementation\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLMOps\ncd LMOps\u002Fminillm\npip install -e .\n```\n\n## Quick Start\n\n### Basic Knowledge Distillation\n\n```python\nimport torch\nimport torch.nn.functional as F\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments\n\n# 1. Load teacher (large) and student (small) models\nteacher = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-70b-hf\",  # Large teacher\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\n\nstudent = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",  # Small student\n    torch_dtype=torch.float16,\n    device_map=\"cuda:0\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-70b-hf\")\n\n# 2. Define distillation loss\ndef distillation_loss(student_logits, teacher_logits, labels, temperature=2.0, alpha=0.5):\n    \"\"\"\n    Combine hard loss (cross-entropy) with soft loss (KL divergence).\n\n    Args:\n        temperature: Softens probability distributions (higher = softer)\n        alpha: Weight for distillation loss (1-alpha for hard loss)\n    \"\"\"\n    # Hard loss: Standard cross-entropy with true labels\n    hard_loss = F.cross_entropy(student_logits.view(-1, student_logits.size(-1)), labels.view(-1))\n\n    # Soft loss: KL divergence between student and teacher\n    soft_targets = F.softmax(teacher_logits \u002F temperature, dim=-1)\n    soft_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n    soft_loss = F.kl_div(soft_student, soft_targets, reduction='batchmean') * (temperature ** 2)\n\n    # Combined loss\n    return alpha * soft_loss + (1 - alpha) * hard_loss\n\n# 3. Training loop\nfor batch in dataloader:\n    # Teacher forward (no grad)\n    with torch.no_grad():\n        teacher_outputs = teacher(**batch)\n        teacher_logits = teacher_outputs.logits\n\n    # Student forward\n    student_outputs = student(**batch)\n    student_logits = student_outputs.logits\n\n    # Compute distillation loss\n    loss = distillation_loss(\n        student_logits,\n        teacher_logits,\n        batch['labels'],\n        temperature=2.0,\n        alpha=0.7  # 70% soft, 30% hard\n    )\n\n    # Backward and optimize\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n```\n\n### MiniLLM (Reverse KLD)\n\n**Source**: arXiv 2306.08543 (2024)\n\n**Innovation**: Use reverse KLD instead of forward KLD for better generative model distillation.\n\n```python\ndef reverse_kl_loss(student_logits, teacher_logits, temperature=1.0):\n    \"\"\"\n    Reverse KL divergence: KL(Teacher || Student)\n    Better for generative models than forward KL.\n    \"\"\"\n    # Teacher distribution (target)\n    p_teacher = F.softmax(teacher_logits \u002F temperature, dim=-1)\n\n    # Student distribution (model)\n    log_p_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n\n    # Reverse KL: Sum over teacher, student learns to cover teacher's modes\n    reverse_kl = -(p_teacher * log_p_student).sum(dim=-1).mean()\n\n    return reverse_kl * (temperature ** 2)\n\n# Training with MiniLLM\nfor batch in dataloader:\n    with torch.no_grad():\n        teacher_logits = teacher(**batch).logits\n\n    student_logits = student(**batch).logits\n\n    # Reverse KLD (better for generation)\n    loss = reverse_kl_loss(student_logits, teacher_logits, temperature=1.0)\n\n    loss.backward()\n    optimizer.step()\n```\n\n**Why reverse KL?**\n- **Forward KL** (standard): Student learns to match teacher's *mean*\n- **Reverse KL** (MiniLLM): Student learns to *cover* all teacher's modes\n- Better for diverse text generation\n\n### Response Distillation\n\n```python\n# Generate synthetic data from teacher, train student to imitate\n\n# 1. Generate synthetic responses from teacher\nprompts = [\"Explain AI:\", \"What is ML?\", \"Define NLP:\"]\n\nteacher_responses = []\nfor prompt in prompts:\n    inputs = tokenizer(prompt, return_tensors='pt').to(teacher.device)\n    outputs = teacher.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)\n    response = tokenizer.decode(outputs[0], skip_special_tokens=True)\n    teacher_responses.append(response)\n\n# 2. Train student on teacher's responses (standard fine-tuning)\ntrain_dataset = [\n    {\"text\": f\"{prompt}\\n{response}\"}\n    for prompt, response in zip(prompts, teacher_responses)\n]\n\n# 3. Fine-tune student\ntrainer = Trainer(\n    model=student,\n    args=TrainingArguments(output_dir=\".\u002Fstudent\", num_train_epochs=3, learning_rate=2e-5),\n    train_dataset=train_dataset,\n)\ntrainer.train()\n```\n\n## Core Concepts\n\n### 1. Temperature Scaling\n\n**Purpose**: Soften probability distributions to expose teacher's uncertainty.\n\n```python\n# Low temperature (T=1): Sharp distribution\nlogits = [3.0, 2.0, 1.0]\nprobs_T1 = softmax(logits \u002F 1.0)  # [0.67, 0.24, 0.09]\n\n# High temperature (T=4): Soft distribution\nprobs_T4 = softmax(logits \u002F 4.0)  # [0.42, 0.34, 0.24]\n\n# Higher T reveals more information about relative rankings\n```\n\n**Rule**: Use T=2-5 for distillation (2 is common default).\n\n### 2. Loss Function Components\n\n```python\n# Total loss = alpha * soft_loss + (1 - alpha) * hard_loss\n\n# Soft loss: Learn from teacher's knowledge\nsoft_loss = KL(student || teacher)\n\n# Hard loss: Learn from ground truth labels\nhard_loss = CrossEntropy(student_output, true_labels)\n\n# Typical values:\nalpha = 0.5  # Balanced\nalpha = 0.7  # More emphasis on teacher\nalpha = 0.3  # More emphasis on labels\n```\n\n### 3. Forward vs Reverse KLD\n\n```python\n# Forward KL: KL(Student || Teacher)\n# - Student matches teacher's average behavior\n# - Mode-seeking: Student focuses on teacher's highest probability modes\n# - Good for classification\n\n# Reverse KL: KL(Teacher || Student)\n# - Student covers all of teacher's behaviors\n# - Mode-covering: Student learns diverse behaviors\n# - Good for generation (MiniLLM)\n```\n\n## Training Strategies\n\n### Strategy 1: Logit Distillation\n\n```python\n# Train student to match teacher's logits directly\n\ndef logit_distillation_trainer(student, teacher, dataloader, temperature=2.0):\n    optimizer = torch.optim.AdamW(student.parameters(), lr=2e-5)\n\n    for epoch in range(3):\n        for batch in dataloader:\n            # Get logits\n            with torch.no_grad():\n                teacher_logits = teacher(**batch).logits\n\n            student_logits = student(**batch).logits\n\n            # MSE on logits (alternative to KLD)\n            loss = F.mse_loss(student_logits, teacher_logits)\n\n            # Or use KLD\n            # loss = F.kl_div(\n            #     F.log_softmax(student_logits\u002Ftemperature, dim=-1),\n            #     F.softmax(teacher_logits\u002Ftemperature, dim=-1),\n            #     reduction='batchmean'\n            # ) * (temperature ** 2)\n\n            loss.backward()\n            optimizer.step()\n            optimizer.zero_grad()\n\n    return student\n```\n\n### Strategy 2: Two-Stage Distillation\n\n```python\n# Stage 1: Distill from teacher\nstudent = distill(teacher, student, epochs=5)\n\n# Stage 2: Fine-tune on task-specific data\nstudent = fine_tune(student, task_data, epochs=3)\n\n# Results in better task performance than single-stage\n```\n\n### Strategy 3: Multi-Teacher Distillation\n\n```python\n# Learn from multiple expert teachers\n\ndef multi_teacher_distillation(student, teachers, batch):\n    \"\"\"Distill from ensemble of teachers.\"\"\"\n    teacher_logits_list = []\n\n    # Get logits from all teachers\n    with torch.no_grad():\n        for teacher in teachers:\n            logits = teacher(**batch).logits\n            teacher_logits_list.append(logits)\n\n    # Average teacher predictions\n    avg_teacher_logits = torch.stack(teacher_logits_list).mean(dim=0)\n\n    # Student learns from ensemble\n    student_logits = student(**batch).logits\n    loss = F.kl_div(\n        F.log_softmax(student_logits, dim=-1),\n        F.softmax(avg_teacher_logits, dim=-1),\n        reduction='batchmean'\n    )\n\n    return loss\n```\n\n## Production Deployment\n\n### Complete Training Script\n\n```python\nfrom transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling\n\ndef train_distilled_model(\n    teacher_name=\"meta-llama\u002FLlama-2-70b-hf\",\n    student_name=\"meta-llama\u002FLlama-2-7b-hf\",\n    output_dir=\".\u002Fdistilled-llama-7b\",\n    temperature=2.0,\n    alpha=0.7,\n):\n    # Load models\n    teacher = AutoModelForCausalLM.from_pretrained(teacher_name, torch_dtype=torch.float16, device_map=\"auto\")\n    student = AutoModelForCausalLM.from_pretrained(student_name, torch_dtype=torch.float16)\n    tokenizer = AutoTokenizer.from_pretrained(teacher_name)\n\n    # Custom trainer with distillation\n    class DistillationTrainer(Trainer):\n        def compute_loss(self, model, inputs, return_outputs=False):\n            # Student forward\n            outputs_student = model(**inputs)\n            student_logits = outputs_student.logits\n\n            # Teacher forward (no grad)\n            with torch.no_grad():\n                outputs_teacher = teacher(**inputs)\n                teacher_logits = outputs_teacher.logits\n\n            # Distillation loss\n            soft_targets = F.softmax(teacher_logits \u002F temperature, dim=-1)\n            soft_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n            soft_loss = F.kl_div(soft_student, soft_targets, reduction='batchmean') * (temperature ** 2)\n\n            # Hard loss\n            hard_loss = outputs_student.loss\n\n            # Combined\n            loss = alpha * soft_loss + (1 - alpha) * hard_loss\n\n            return (loss, outputs_student) if return_outputs else loss\n\n    # Training arguments\n    training_args = TrainingArguments(\n        output_dir=output_dir,\n        num_train_epochs=3,\n        per_device_train_batch_size=4,\n        gradient_accumulation_steps=8,\n        learning_rate=2e-5,\n        warmup_steps=500,\n        logging_steps=100,\n        save_steps=1000,\n        bf16=True,\n        gradient_checkpointing=True,\n    )\n\n    # Train\n    trainer = DistillationTrainer(\n        model=student,\n        args=training_args,\n        train_dataset=train_dataset,\n        data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),\n    )\n\n    trainer.train()\n    student.save_pretrained(output_dir)\n    tokenizer.save_pretrained(output_dir)\n\n# Usage\ntrain_distilled_model(\n    teacher_name=\"meta-llama\u002FLlama-2-70b-hf\",\n    student_name=\"meta-llama\u002FLlama-2-7b-hf\",\n    temperature=2.0,\n    alpha=0.7\n)\n```\n\n## Best Practices\n\n### 1. Hyperparameter Selection\n\n```python\n# Temperature\nT = 1.0  # Sharp (less knowledge transfer)\nT = 2.0  # Standard (good balance)\nT = 5.0  # Soft (more knowledge transfer)\n\n# Alpha (weight)\nalpha = 0.5  # Balanced\nalpha = 0.7  # Emphasize teacher knowledge\nalpha = 0.9  # Strong distillation\n\n# Rule: Higher T + higher alpha = stronger distillation\n```\n\n### 2. Model Size Ratio\n\n```python\n# Good ratios (teacher\u002Fstudent)\n70B \u002F 7B = 10×    # Excellent\n13B \u002F 1B = 13×    # Good\n7B \u002F 1B = 7×      # Acceptable\n\n# Avoid too large gap\n70B \u002F 1B = 70×    # Too large, ineffective\n```\n\n### 3. Data Quality\n\n```python\n# Best: Use teacher-generated data + real data\ntrain_data = {\n    \"teacher_generated\": 70%,  # Diverse, high-quality\n    \"real_data\": 30%            # Ground truth\n}\n\n# Avoid: Only real data (doesn't utilize teacher fully)\n```\n\n## Evaluation\n\n```python\nfrom transformers import pipeline\n\n# Compare student vs teacher\nteacher_pipe = pipeline(\"text-generation\", model=teacher)\nstudent_pipe = pipeline(\"text-generation\", model=student)\n\nprompts = [\"Explain quantum computing:\", \"What is AI?\"]\n\nfor prompt in prompts:\n    teacher_out = teacher_pipe(prompt, max_new_tokens=100)\n    student_out = student_pipe(prompt, max_new_tokens=100)\n\n    print(f\"Prompt: {prompt}\")\n    print(f\"Teacher: {teacher_out[0]['generated_text']}\")\n    print(f\"Student: {student_out[0]['generated_text']}\")\n    print(f\"Match quality: {calculate_similarity(teacher_out, student_out):.2f}\")\n```\n\n## Resources\n\n- **Hinton et al. 2015 (Foundational)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F1503.02531\n- **MiniLLM (Reverse KLD)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.08543\n- **KD Survey for LLMs (2024)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2402.13116\n- **MiniLLM GitHub**: https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLMOps\u002Ftree\u002Fmain\u002Fminillm\n\n\n","# Knowledge Distillation: Compressing LLMs\n\n## When to Use This Skill\n\nUse Knowledge Distillation when you need to:\n- **Compress models** from 70B → 7B while retaining 90%+ performance\n- **Transfer capabilities** from proprietary models (GPT-4) to open-source (LLaMA, Mistral)\n- **Reduce inference costs** by deploying smaller student models\n- **Create specialized models** by distilling domain-specific knowledge\n- **Improve small models** using synthetic data from large teachers\n\n**Key Techniques**: Temperature scaling, soft targets, reverse KLD (MiniLLM), logit distillation, response distillation\n\n**Papers**: Hinton et al. 2015 (arXiv 1503.02531), MiniLLM (arXiv 2306.08543), KD Survey (arXiv 2402.13116)\n\n## Installation\n\n```bash\n# Standard transformers\npip install transformers datasets accelerate\n\n# For training\npip install torch deepspeed wandb\n\n# Optional: MiniLLM implementation\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLMOps\ncd LMOps\u002Fminillm\npip install -e .\n```\n\n## Quick Start\n\n### Basic Knowledge Distillation\n\n```python\nimport torch\nimport torch.nn.functional as F\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments\n\n# 1. Load teacher (large) and student (small) models\nteacher = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-70b-hf\",  # Large teacher\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\n\nstudent = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",  # Small student\n    torch_dtype=torch.float16,\n    device_map=\"cuda:0\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-70b-hf\")\n\n# 2. Define distillation loss\ndef distillation_loss(student_logits, teacher_logits, labels, temperature=2.0, alpha=0.5):\n    \"\"\"\n    Combine hard loss (cross-entropy) with soft loss (KL divergence).\n\n    Args:\n        temperature: Softens probability distributions (higher = softer)\n        alpha: Weight for distillation loss (1-alpha for hard loss)\n    \"\"\"\n    # Hard loss: Standard cross-entropy with true labels\n    hard_loss = F.cross_entropy(student_logits.view(-1, student_logits.size(-1)), labels.view(-1))\n\n    # Soft loss: KL divergence between student and teacher\n    soft_targets = F.softmax(teacher_logits \u002F temperature, dim=-1)\n    soft_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n    soft_loss = F.kl_div(soft_student, soft_targets, reduction='batchmean') * (temperature ** 2)\n\n    # Combined loss\n    return alpha * soft_loss + (1 - alpha) * hard_loss\n\n# 3. Training loop\nfor batch in dataloader:\n    # Teacher forward (no grad)\n    with torch.no_grad():\n        teacher_outputs = teacher(**batch)\n        teacher_logits = teacher_outputs.logits\n\n    # Student forward\n    student_outputs = student(**batch)\n    student_logits = student_outputs.logits\n\n    # Compute distillation loss\n    loss = distillation_loss(\n        student_logits,\n        teacher_logits,\n        batch['labels'],\n        temperature=2.0,\n        alpha=0.7  # 70% soft, 30% hard\n    )\n\n    # Backward and optimize\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n```\n\n### MiniLLM (Reverse KLD)\n\n**Source**: arXiv 2306.08543 (2024)\n\n**Innovation**: Use reverse KLD instead of forward KLD for better generative model distillation.\n\n```python\ndef reverse_kl_loss(student_logits, teacher_logits, temperature=1.0):\n    \"\"\"\n    Reverse KL divergence: KL(Teacher || Student)\n    Better for generative models than forward KL.\n    \"\"\"\n    # Teacher distribution (target)\n    p_teacher = F.softmax(teacher_logits \u002F temperature, dim=-1)\n\n    # Student distribution (model)\n    log_p_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n\n    # Reverse KL: Sum over teacher, student learns to cover teacher's modes\n    reverse_kl = -(p_teacher * log_p_student).sum(dim=-1).mean()\n\n    return reverse_kl * (temperature ** 2)\n\n# Training with MiniLLM\nfor batch in dataloader:\n    with torch.no_grad():\n        teacher_logits = teacher(**batch).logits\n\n    student_logits = student(**batch).logits\n\n    # Reverse KLD (better for generation)\n    loss = reverse_kl_loss(student_logits, teacher_logits, temperature=1.0)\n\n    loss.backward()\n    optimizer.step()\n```\n\n**Why reverse KL?**\n- **Forward KL** (standard): Student learns to match teacher's *mean*\n- **Reverse KL** (MiniLLM): Student learns to *cover* all teacher's modes\n- Better for diverse text generation\n\n### Response Distillation\n\n```python\n# Generate synthetic data from teacher, train student to imitate\n\n# 1. Generate synthetic responses from teacher\nprompts = [\"Explain AI:\", \"What is ML?\", \"Define NLP:\"]\n\nteacher_responses = []\nfor prompt in prompts:\n    inputs = tokenizer(prompt, return_tensors='pt').to(teacher.device)\n    outputs = teacher.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)\n    response = tokenizer.decode(outputs[0], skip_special_tokens=True)\n    teacher_responses.append(response)\n\n# 2. Train student on teacher's responses (standard fine-tuning)\ntrain_dataset = [\n    {\"text\": f\"{prompt}\\n{response}\"}\n    for prompt, response in zip(prompts, teacher_responses)\n]\n\n# 3. Fine-tune student\ntrainer = Trainer(\n    model=student,\n    args=TrainingArguments(output_dir=\".\u002Fstudent\", num_train_epochs=3, learning_rate=2e-5),\n    train_dataset=train_dataset,\n)\ntrainer.train()\n```\n\n## Core Concepts\n\n### 1. Temperature Scaling\n\n**Purpose**: Soften probability distributions to expose teacher's uncertainty.\n\n```python\n# Low temperature (T=1): Sharp distribution\nlogits = [3.0, 2.0, 1.0]\nprobs_T1 = softmax(logits \u002F 1.0)  # [0.67, 0.24, 0.09]\n\n# High temperature (T=4): Soft distribution\nprobs_T4 = softmax(logits \u002F 4.0)  # [0.42, 0.34, 0.24]\n\n# Higher T reveals more information about relative rankings\n```\n\n**Rule**: Use T=2-5 for distillation (2 is common default).\n\n### 2. Loss Function Components\n\n```python\n# Total loss = alpha * soft_loss + (1 - alpha) * hard_loss\n\n# Soft loss: Learn from teacher's knowledge\nsoft_loss = KL(student || teacher)\n\n# Hard loss: Learn from ground truth labels\nhard_loss = CrossEntropy(student_output, true_labels)\n\n# Typical values:\nalpha = 0.5  # Balanced\nalpha = 0.7  # More emphasis on teacher\nalpha = 0.3  # More emphasis on labels\n```\n\n### 3. Forward vs Reverse KLD\n\n```python\n# Forward KL: KL(Student || Teacher)\n# - Student matches teacher's average behavior\n# - Mode-seeking: Student focuses on teacher's highest probability modes\n# - Good for classification\n\n# Reverse KL: KL(Teacher || Student)\n# - Student covers all of teacher's behaviors\n# - Mode-covering: Student learns diverse behaviors\n# - Good for generation (MiniLLM)\n```\n\n## Training Strategies\n\n### Strategy 1: Logit Distillation\n\n```python\n# Train student to match teacher's logits directly\n\ndef logit_distillation_trainer(student, teacher, dataloader, temperature=2.0):\n    optimizer = torch.optim.AdamW(student.parameters(), lr=2e-5)\n\n    for epoch in range(3):\n        for batch in dataloader:\n            # Get logits\n            with torch.no_grad():\n                teacher_logits = teacher(**batch).logits\n\n            student_logits = student(**batch).logits\n\n            # MSE on logits (alternative to KLD)\n            loss = F.mse_loss(student_logits, teacher_logits)\n\n            # Or use KLD\n            # loss = F.kl_div(\n            #     F.log_softmax(student_logits\u002Ftemperature, dim=-1),\n            #     F.softmax(teacher_logits\u002Ftemperature, dim=-1),\n            #     reduction='batchmean'\n            # ) * (temperature ** 2)\n\n            loss.backward()\n            optimizer.step()\n            optimizer.zero_grad()\n\n    return student\n```\n\n### Strategy 2: Two-Stage Distillation\n\n```python\n# Stage 1: Distill from teacher\nstudent = distill(teacher, student, epochs=5)\n\n# Stage 2: Fine-tune on task-specific data\nstudent = fine_tune(student, task_data, epochs=3)\n\n# Results in better task performance than single-stage\n```\n\n### Strategy 3: Multi-Teacher Distillation\n\n```python\n# Learn from multiple expert teachers\n\ndef multi_teacher_distillation(student, teachers, batch):\n    \"\"\"Distill from ensemble of teachers.\"\"\"\n    teacher_logits_list = []\n\n    # Get logits from all teachers\n    with torch.no_grad():\n        for teacher in teachers:\n            logits = teacher(**batch).logits\n            teacher_logits_list.append(logits)\n\n    # Average teacher predictions\n    avg_teacher_logits = torch.stack(teacher_logits_list).mean(dim=0)\n\n    # Student learns from ensemble\n    student_logits = student(**batch).logits\n    loss = F.kl_div(\n        F.log_softmax(student_logits, dim=-1),\n        F.softmax(avg_teacher_logits, dim=-1),\n        reduction='batchmean'\n    )\n\n    return loss\n```\n\n## Production Deployment\n\n### Complete Training Script\n\n```python\nfrom transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling\n\ndef train_distilled_model(\n    teacher_name=\"meta-llama\u002FLlama-2-70b-hf\",\n    student_name=\"meta-llama\u002FLlama-2-7b-hf\",\n    output_dir=\".\u002Fdistilled-llama-7b\",\n    temperature=2.0,\n    alpha=0.7,\n):\n    # Load models\n    teacher = AutoModelForCausalLM.from_pretrained(teacher_name, torch_dtype=torch.float16, device_map=\"auto\")\n    student = AutoModelForCausalLM.from_pretrained(student_name, torch_dtype=torch.float16)\n    tokenizer = AutoTokenizer.from_pretrained(teacher_name)\n\n    # Custom trainer with distillation\n    class DistillationTrainer(Trainer):\n        def compute_loss(self, model, inputs, return_outputs=False):\n            # Student forward\n            outputs_student = model(**inputs)\n            student_logits = outputs_student.logits\n\n            # Teacher forward (no grad)\n            with torch.no_grad():\n                outputs_teacher = teacher(**inputs)\n                teacher_logits = outputs_teacher.logits\n\n            # Distillation loss\n            soft_targets = F.softmax(teacher_logits \u002F temperature, dim=-1)\n            soft_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n            soft_loss = F.kl_div(soft_student, soft_targets, reduction='batchmean') * (temperature ** 2)\n\n            # Hard loss\n            hard_loss = outputs_student.loss\n\n            # Combined\n            loss = alpha * soft_loss + (1 - alpha) * hard_loss\n\n            return (loss, outputs_student) if return_outputs else loss\n\n    # Training arguments\n    training_args = TrainingArguments(\n        output_dir=output_dir,\n        num_train_epochs=3,\n        per_device_train_batch_size=4,\n        gradient_accumulation_steps=8,\n        learning_rate=2e-5,\n        warmup_steps=500,\n        logging_steps=100,\n        save_steps=1000,\n        bf16=True,\n        gradient_checkpointing=True,\n    )\n\n    # Train\n    trainer = DistillationTrainer(\n        model=student,\n        args=training_args,\n        train_dataset=train_dataset,\n        data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),\n    )\n\n    trainer.train()\n    student.save_pretrained(output_dir)\n    tokenizer.save_pretrained(output_dir)\n\n# Usage\ntrain_distilled_model(\n    teacher_name=\"meta-llama\u002FLlama-2-70b-hf\",\n    student_name=\"meta-llama\u002FLlama-2-7b-hf\",\n    temperature=2.0,\n    alpha=0.7\n)\n```\n\n## Best Practices\n\n### 1. Hyperparameter Selection\n\n```python\n# Temperature\nT = 1.0  # Sharp (less knowledge transfer)\nT = 2.0  # Standard (good balance)\nT = 5.0  # Soft (more knowledge transfer)\n\n# Alpha (weight)\nalpha = 0.5  # Balanced\nalpha = 0.7  # Emphasize teacher knowledge\nalpha = 0.9  # Strong distillation\n\n# Rule: Higher T + higher alpha = stronger distillation\n```\n\n### 2. Model Size Ratio\n\n```python\n# Good ratios (teacher\u002Fstudent)\n70B \u002F 7B = 10×    # Excellent\n13B \u002F 1B = 13×    # Good\n7B \u002F 1B = 7×      # Acceptable\n\n# Avoid too large gap\n70B \u002F 1B = 70×    # Too large, ineffective\n```\n\n### 3. Data Quality\n\n```python\n# Best: Use teacher-generated data + real data\ntrain_data = {\n    \"teacher_generated\": 70%,  # Diverse, high-quality\n    \"real_data\": 30%            # Ground truth\n}\n\n# Avoid: Only real data (doesn't utilize teacher fully)\n```\n\n## Evaluation\n\n```python\nfrom transformers import pipeline\n\n# Compare student vs teacher\nteacher_pipe = pipeline(\"text-generation\", model=teacher)\nstudent_pipe = pipeline(\"text-generation\", model=student)\n\nprompts = [\"Explain quantum computing:\", \"What is AI?\"]\n\nfor prompt in prompts:\n    teacher_out = teacher_pipe(prompt, max_new_tokens=100)\n    student_out = student_pipe(prompt, max_new_tokens=100)\n\n    print(f\"Prompt: {prompt}\")\n    print(f\"Teacher: {teacher_out[0]['generated_text']}\")\n    print(f\"Student: {student_out[0]['generated_text']}\")\n    print(f\"Match quality: {calculate_similarity(teacher_out, student_out):.2f}\")\n```\n\n## Resources\n\n- **Hinton et al. 2015 (Foundational)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F1503.02531\n- **MiniLLM (Reverse KLD)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.08543\n- **KD Survey for LLMs (2024)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2402.13116\n- **MiniLLM GitHub**: https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLMOps\u002Ftree\u002Fmain\u002Fminillm\n\n\n",{"name":5,"tags":24,"author":35,"license":15,"version":36,"description":7,"dependencies":37},[25,26,27,28,29,30,31,32,33,34],"Emerging Techniques","Knowledge Distillation","Model Compression","Teacher-Student","MiniLLM","Reverse KLD","Soft Targets","Temperature Scaling","Logit Distillation","Model Transfer","Orchestra Research","1.0.0",[38,39,40],"transformers","torch","datasets","main","f4a24bd1ab047411d4ae57ee9a96701a7706e779","claude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-knowledge-distillation","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive","https:\u002F\u002Fgithub.com\u002Fcocyuhao","Beijing Wuzi University","2026-05-12 10:57:02",{"source":8,"skillName":5,"commands":49,"prompt":59,"zipAvailable":60},[50,53,56],{"tool":51,"command":52},"npx","npx skills add cocyuhao\u002Fai-dev-tooling-research-archive --skill knowledge-distillation",{"tool":54,"command":55},"bunx","bunx skills add cocyuhao\u002Fai-dev-tooling-research-archive --skill knowledge-distillation",{"tool":57,"command":58},"pnpm","pnpm dlx skills add cocyuhao\u002Fai-dev-tooling-research-archive --skill knowledge-distillation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-knowledge-distillation\u002FSKILL.md",true,[62],{"path":63,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":66,"sourceHtmlUrl":67},"references\u002Fminillm.md","REFERENCE","参考文档",9117,"https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-knowledge-distillation\u002Freferences\u002Fminillm.md",[69,77,85,92,99,106],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":76},"01M72JQ593HE3SRJD3446CCKA8","web-access","所有联网操作必须通过此 skill 处理，包括：搜索、网页抓取、登录后操作、网络交互等。 触发场景：用户要求搜索信息、查看网页内容、访问需要登录的网站、操作网页界面、抓取社交媒体内容（小红书、微博、推特等）、读取动态渲染页面、以及任何需要真实浏览器环境的网络任务。","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fweb-access\u002FSKILL.md",[],[],"2026-11-10 22:39:42",{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":81,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":76},"01M72JKXJYFT2RBJ2HP5ZRA853","last30days","Multi-query social research across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web. Intelligent query planning with Gemini\u002FOpenAI fallback.","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Flast30days-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Flast30days-skill\u002FSKILL.md",[],[],{"skillNo":86,"name":87,"nameZh":6,"description":88,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":89,"categories":90,"keywords":91,"publishedAt":76},"01M72JKMRQSD0ZY66KQ863HVY5","writing-plans","Use when you have a spec or requirements for a multi-step task. Creates comprehensive implementation plans with bite-sized tasks, exact file paths, and complete code examples.","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fsoftware-development\u002Fwriting-plans\u002FSKILL.md",[],[],{"skillNo":93,"name":94,"nameZh":6,"description":95,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":76},"01M72JKHCV8DT6VWSD4KS5YWT9","test-driven-development","Use when implementing any feature or bugfix, before writing implementation code. Enforces RED-GREEN-REFACTOR cycle with test-first approach.","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fsoftware-development\u002Ftest-driven-development\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":76},"01M72JKD7VX7EQM38R704F3KT1","systematic-debugging","Use when encountering any bug, test failure, or unexpected behavior. 4-phase root cause investigation — NO fixes without understanding the problem first.","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fsoftware-development\u002Fsystematic-debugging\u002FSKILL.md",[],[],{"skillNo":107,"name":108,"nameZh":6,"description":109,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":76},"01M72JKA9XXB92FS8JHPCYW8CM","subagent-driven-development","Use when executing implementation plans with independent tasks. Dispatches fresh delegate_task per task with two-stage review (spec compliance then code quality).","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fsoftware-development\u002Fsubagent-driven-development\u002FSKILL.md",[],[]]