[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6FC2GC6SPS0PKE0M6CBA2SJ":3,"skill-same:01M6FC2GC6SPS0PKE0M6CBA2SJ":88},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":47,"linkOnly":48,"markdownContent":49,"markdownBody":50,"frontMatter":51,"refName":59,"commitSha":60,"skillDirectory":61,"forksCount":62,"repositoryHtmlUrl":63,"repositoryHomepageUrl":6,"ownerHtmlUrl":64,"ownerBlogUrl":65,"ownerDescription":66,"updatedAt":67,"install":68,"resources":81},"01M6FC2GC6SPS0PKE0M6CBA2SJ","knowledge-distillation",null,"Compress large language models using knowledge distillation from teacher to student models. Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs. Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.","Dicklesworthstone\u002Fpi_agent_rust","Dicklesworthstone","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F35050222?v=4","01M6CG26AR942CSVMN1M5E3DSE","Jeff Emanuel","01M6CG26EPP71PSHW1VDS71121",1986,"MIT","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-knowledge-distillation\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000021","资料检索与研究","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46],"knowledge","distillation","Knowledge Distillation","Emerging Techniques","when","teacher","student","Model Transfer","Logit Distillation","Temperature Scaling","Soft Targets","Reverse KLD","MiniLLM","Teacher-Student","performance","Model Compression","scaling","strategies","model","open","Transformers","capabilities","training","2026-11-10 10:12:30",false,"---\nname: knowledge-distillation\ndescription: Compress large language models using knowledge distillation from teacher to student models. Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs. Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Emerging Techniques, Knowledge Distillation, Model Compression, Teacher-Student, MiniLLM, Reverse KLD, Soft Targets, Temperature Scaling, Logit Distillation, Model Transfer]\ndependencies: [transformers, torch, datasets]\n---\n\n# Knowledge Distillation: Compressing LLMs\n\n## When to Use This Skill\n\nUse Knowledge Distillation when you need to:\n- **Compress models** from 70B → 7B while retaining 90%+ performance\n- **Transfer capabilities** from proprietary models (GPT-4) to open-source (LLaMA, Mistral)\n- **Reduce inference costs** by deploying smaller student models\n- **Create specialized models** by distilling domain-specific knowledge\n- **Improve small models** using synthetic data from large teachers\n\n**Key Techniques**: Temperature scaling, soft targets, reverse KLD (MiniLLM), logit distillation, response distillation\n\n**Papers**: Hinton et al. 2015 (arXiv 1503.02531), MiniLLM (arXiv 2306.08543), KD Survey (arXiv 2402.13116)\n\n## Installation\n\n```bash\n# Standard transformers\npip install transformers datasets accelerate\n\n# For training\npip install torch deepspeed wandb\n\n# Optional: MiniLLM implementation\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLMOps\ncd LMOps\u002Fminillm\npip install -e .\n```\n\n## Quick Start\n\n### Basic Knowledge Distillation\n\n```python\nimport torch\nimport torch.nn.functional as F\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments\n\n# 1. Load teacher (large) and student (small) models\nteacher = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-70b-hf\",  # Large teacher\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\n\nstudent = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",  # Small student\n    torch_dtype=torch.float16,\n    device_map=\"cuda:0\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-70b-hf\")\n\n# 2. Define distillation loss\ndef distillation_loss(student_logits, teacher_logits, labels, temperature=2.0, alpha=0.5):\n    \"\"\"\n    Combine hard loss (cross-entropy) with soft loss (KL divergence).\n\n    Args:\n        temperature: Softens probability distributions (higher = softer)\n        alpha: Weight for distillation loss (1-alpha for hard loss)\n    \"\"\"\n    # Hard loss: Standard cross-entropy with true labels\n    hard_loss = F.cross_entropy(student_logits.view(-1, student_logits.size(-1)), labels.view(-1))\n\n    # Soft loss: KL divergence between student and teacher\n    soft_targets = F.softmax(teacher_logits \u002F temperature, dim=-1)\n    soft_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n    soft_loss = F.kl_div(soft_student, soft_targets, reduction='batchmean') * (temperature ** 2)\n\n    # Combined loss\n    return alpha * soft_loss + (1 - alpha) * hard_loss\n\n# 3. Training loop\nfor batch in dataloader:\n    # Teacher forward (no grad)\n    with torch.no_grad():\n        teacher_outputs = teacher(**batch)\n        teacher_logits = teacher_outputs.logits\n\n    # Student forward\n    student_outputs = student(**batch)\n    student_logits = student_outputs.logits\n\n    # Compute distillation loss\n    loss = distillation_loss(\n        student_logits,\n        teacher_logits,\n        batch['labels'],\n        temperature=2.0,\n        alpha=0.7  # 70% soft, 30% hard\n    )\n\n    # Backward and optimize\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n```\n\n### MiniLLM (Reverse KLD)\n\n**Source**: arXiv 2306.08543 (2024)\n\n**Innovation**: Use reverse KLD instead of forward KLD for better generative model distillation.\n\n```python\ndef reverse_kl_loss(student_logits, teacher_logits, temperature=1.0):\n    \"\"\"\n    Reverse KL divergence: KL(Teacher || Student)\n    Better for generative models than forward KL.\n    \"\"\"\n    # Teacher distribution (target)\n    p_teacher = F.softmax(teacher_logits \u002F temperature, dim=-1)\n\n    # Student distribution (model)\n    log_p_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n\n    # Reverse KL: Sum over teacher, student learns to cover teacher's modes\n    reverse_kl = -(p_teacher * log_p_student).sum(dim=-1).mean()\n\n    return reverse_kl * (temperature ** 2)\n\n# Training with MiniLLM\nfor batch in dataloader:\n    with torch.no_grad():\n        teacher_logits = teacher(**batch).logits\n\n    student_logits = student(**batch).logits\n\n    # Reverse KLD (better for generation)\n    loss = reverse_kl_loss(student_logits, teacher_logits, temperature=1.0)\n\n    loss.backward()\n    optimizer.step()\n```\n\n**Why reverse KL?**\n- **Forward KL** (standard): Student learns to match teacher's *mean*\n- **Reverse KL** (MiniLLM): Student learns to *cover* all teacher's modes\n- Better for diverse text generation\n\n### Response Distillation\n\n```python\n# Generate synthetic data from teacher, train student to imitate\n\n# 1. Generate synthetic responses from teacher\nprompts = [\"Explain AI:\", \"What is ML?\", \"Define NLP:\"]\n\nteacher_responses = []\nfor prompt in prompts:\n    inputs = tokenizer(prompt, return_tensors='pt').to(teacher.device)\n    outputs = teacher.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)\n    response = tokenizer.decode(outputs[0], skip_special_tokens=True)\n    teacher_responses.append(response)\n\n# 2. Train student on teacher's responses (standard fine-tuning)\ntrain_dataset = [\n    {\"text\": f\"{prompt}\\n{response}\"}\n    for prompt, response in zip(prompts, teacher_responses)\n]\n\n# 3. Fine-tune student\ntrainer = Trainer(\n    model=student,\n    args=TrainingArguments(output_dir=\".\u002Fstudent\", num_train_epochs=3, learning_rate=2e-5),\n    train_dataset=train_dataset,\n)\ntrainer.train()\n```\n\n## Core Concepts\n\n### 1. Temperature Scaling\n\n**Purpose**: Soften probability distributions to expose teacher's uncertainty.\n\n```python\n# Low temperature (T=1): Sharp distribution\nlogits = [3.0, 2.0, 1.0]\nprobs_T1 = softmax(logits \u002F 1.0)  # [0.67, 0.24, 0.09]\n\n# High temperature (T=4): Soft distribution\nprobs_T4 = softmax(logits \u002F 4.0)  # [0.42, 0.34, 0.24]\n\n# Higher T reveals more information about relative rankings\n```\n\n**Rule**: Use T=2-5 for distillation (2 is common default).\n\n### 2. Loss Function Components\n\n```python\n# Total loss = alpha * soft_loss + (1 - alpha) * hard_loss\n\n# Soft loss: Learn from teacher's knowledge\nsoft_loss = KL(student || teacher)\n\n# Hard loss: Learn from ground truth labels\nhard_loss = CrossEntropy(student_output, true_labels)\n\n# Typical values:\nalpha = 0.5  # Balanced\nalpha = 0.7  # More emphasis on teacher\nalpha = 0.3  # More emphasis on labels\n```\n\n### 3. Forward vs Reverse KLD\n\n```python\n# Forward KL: KL(Student || Teacher)\n# - Student matches teacher's average behavior\n# - Mode-seeking: Student focuses on teacher's highest probability modes\n# - Good for classification\n\n# Reverse KL: KL(Teacher || Student)\n# - Student covers all of teacher's behaviors\n# - Mode-covering: Student learns diverse behaviors\n# - Good for generation (MiniLLM)\n```\n\n## Training Strategies\n\n### Strategy 1: Logit Distillation\n\n```python\n# Train student to match teacher's logits directly\n\ndef logit_distillation_trainer(student, teacher, dataloader, temperature=2.0):\n    optimizer = torch.optim.AdamW(student.parameters(), lr=2e-5)\n\n    for epoch in range(3):\n        for batch in dataloader:\n            # Get logits\n            with torch.no_grad():\n                teacher_logits = teacher(**batch).logits\n\n            student_logits = student(**batch).logits\n\n            # MSE on logits (alternative to KLD)\n            loss = F.mse_loss(student_logits, teacher_logits)\n\n            # Or use KLD\n            # loss = F.kl_div(\n            #     F.log_softmax(student_logits\u002Ftemperature, dim=-1),\n            #     F.softmax(teacher_logits\u002Ftemperature, dim=-1),\n            #     reduction='batchmean'\n            # ) * (temperature ** 2)\n\n            loss.backward()\n            optimizer.step()\n            optimizer.zero_grad()\n\n    return student\n```\n\n### Strategy 2: Two-Stage Distillation\n\n```python\n# Stage 1: Distill from teacher\nstudent = distill(teacher, student, epochs=5)\n\n# Stage 2: Fine-tune on task-specific data\nstudent = fine_tune(student, task_data, epochs=3)\n\n# Results in better task performance than single-stage\n```\n\n### Strategy 3: Multi-Teacher Distillation\n\n```python\n# Learn from multiple expert teachers\n\ndef multi_teacher_distillation(student, teachers, batch):\n    \"\"\"Distill from ensemble of teachers.\"\"\"\n    teacher_logits_list = []\n\n    # Get logits from all teachers\n    with torch.no_grad():\n        for teacher in teachers:\n            logits = teacher(**batch).logits\n            teacher_logits_list.append(logits)\n\n    # Average teacher predictions\n    avg_teacher_logits = torch.stack(teacher_logits_list).mean(dim=0)\n\n    # Student learns from ensemble\n    student_logits = student(**batch).logits\n    loss = F.kl_div(\n        F.log_softmax(student_logits, dim=-1),\n        F.softmax(avg_teacher_logits, dim=-1),\n        reduction='batchmean'\n    )\n\n    return loss\n```\n\n## Production Deployment\n\n### Complete Training Script\n\n```python\nfrom transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling\n\ndef train_distilled_model(\n    teacher_name=\"meta-llama\u002FLlama-2-70b-hf\",\n    student_name=\"meta-llama\u002FLlama-2-7b-hf\",\n    output_dir=\".\u002Fdistilled-llama-7b\",\n    temperature=2.0,\n    alpha=0.7,\n):\n    # Load models\n    teacher = AutoModelForCausalLM.from_pretrained(teacher_name, torch_dtype=torch.float16, device_map=\"auto\")\n    student = AutoModelForCausalLM.from_pretrained(student_name, torch_dtype=torch.float16)\n    tokenizer = AutoTokenizer.from_pretrained(teacher_name)\n\n    # Custom trainer with distillation\n    class DistillationTrainer(Trainer):\n        def compute_loss(self, model, inputs, return_outputs=False):\n            # Student forward\n            outputs_student = model(**inputs)\n            student_logits = outputs_student.logits\n\n            # Teacher forward (no grad)\n            with torch.no_grad():\n                outputs_teacher = teacher(**inputs)\n                teacher_logits = outputs_teacher.logits\n\n            # Distillation loss\n            soft_targets = F.softmax(teacher_logits \u002F temperature, dim=-1)\n            soft_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n            soft_loss = F.kl_div(soft_student, soft_targets, reduction='batchmean') * (temperature ** 2)\n\n            # Hard loss\n            hard_loss = outputs_student.loss\n\n            # Combined\n            loss = alpha * soft_loss + (1 - alpha) * hard_loss\n\n            return (loss, outputs_student) if return_outputs else loss\n\n    # Training arguments\n    training_args = TrainingArguments(\n        output_dir=output_dir,\n        num_train_epochs=3,\n        per_device_train_batch_size=4,\n        gradient_accumulation_steps=8,\n        learning_rate=2e-5,\n        warmup_steps=500,\n        logging_steps=100,\n        save_steps=1000,\n        bf16=True,\n        gradient_checkpointing=True,\n    )\n\n    # Train\n    trainer = DistillationTrainer(\n        model=student,\n        args=training_args,\n        train_dataset=train_dataset,\n        data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),\n    )\n\n    trainer.train()\n    student.save_pretrained(output_dir)\n    tokenizer.save_pretrained(output_dir)\n\n# Usage\ntrain_distilled_model(\n    teacher_name=\"meta-llama\u002FLlama-2-70b-hf\",\n    student_name=\"meta-llama\u002FLlama-2-7b-hf\",\n    temperature=2.0,\n    alpha=0.7\n)\n```\n\n## Best Practices\n\n### 1. Hyperparameter Selection\n\n```python\n# Temperature\nT = 1.0  # Sharp (less knowledge transfer)\nT = 2.0  # Standard (good balance)\nT = 5.0  # Soft (more knowledge transfer)\n\n# Alpha (weight)\nalpha = 0.5  # Balanced\nalpha = 0.7  # Emphasize teacher knowledge\nalpha = 0.9  # Strong distillation\n\n# Rule: Higher T + higher alpha = stronger distillation\n```\n\n### 2. Model Size Ratio\n\n```python\n# Good ratios (teacher\u002Fstudent)\n70B \u002F 7B = 10×    # Excellent\n13B \u002F 1B = 13×    # Good\n7B \u002F 1B = 7×      # Acceptable\n\n# Avoid too large gap\n70B \u002F 1B = 70×    # Too large, ineffective\n```\n\n### 3. Data Quality\n\n```python\n# Best: Use teacher-generated data + real data\ntrain_data = {\n    \"teacher_generated\": 70%,  # Diverse, high-quality\n    \"real_data\": 30%            # Ground truth\n}\n\n# Avoid: Only real data (doesn't utilize teacher fully)\n```\n\n## Evaluation\n\n```python\nfrom transformers import pipeline\n\n# Compare student vs teacher\nteacher_pipe = pipeline(\"text-generation\", model=teacher)\nstudent_pipe = pipeline(\"text-generation\", model=student)\n\nprompts = [\"Explain quantum computing:\", \"What is AI?\"]\n\nfor prompt in prompts:\n    teacher_out = teacher_pipe(prompt, max_new_tokens=100)\n    student_out = student_pipe(prompt, max_new_tokens=100)\n\n    print(f\"Prompt: {prompt}\")\n    print(f\"Teacher: {teacher_out[0]['generated_text']}\")\n    print(f\"Student: {student_out[0]['generated_text']}\")\n    print(f\"Match quality: {calculate_similarity(teacher_out, student_out):.2f}\")\n```\n\n## Resources\n\n- **Hinton et al. 2015 (Foundational)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F1503.02531\n- **MiniLLM (Reverse KLD)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.08543\n- **KD Survey for LLMs (2024)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2402.13116\n- **MiniLLM GitHub**: https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLMOps\u002Ftree\u002Fmain\u002Fminillm\n\n\n","# Knowledge Distillation: Compressing LLMs\n\n## When to Use This Skill\n\nUse Knowledge Distillation when you need to:\n- **Compress models** from 70B → 7B while retaining 90%+ performance\n- **Transfer capabilities** from proprietary models (GPT-4) to open-source (LLaMA, Mistral)\n- **Reduce inference costs** by deploying smaller student models\n- **Create specialized models** by distilling domain-specific knowledge\n- **Improve small models** using synthetic data from large teachers\n\n**Key Techniques**: Temperature scaling, soft targets, reverse KLD (MiniLLM), logit distillation, response distillation\n\n**Papers**: Hinton et al. 2015 (arXiv 1503.02531), MiniLLM (arXiv 2306.08543), KD Survey (arXiv 2402.13116)\n\n## Installation\n\n```bash\n# Standard transformers\npip install transformers datasets accelerate\n\n# For training\npip install torch deepspeed wandb\n\n# Optional: MiniLLM implementation\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLMOps\ncd LMOps\u002Fminillm\npip install -e .\n```\n\n## Quick Start\n\n### Basic Knowledge Distillation\n\n```python\nimport torch\nimport torch.nn.functional as F\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments\n\n# 1. Load teacher (large) and student (small) models\nteacher = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-70b-hf\",  # Large teacher\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\n\nstudent = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",  # Small student\n    torch_dtype=torch.float16,\n    device_map=\"cuda:0\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-70b-hf\")\n\n# 2. Define distillation loss\ndef distillation_loss(student_logits, teacher_logits, labels, temperature=2.0, alpha=0.5):\n    \"\"\"\n    Combine hard loss (cross-entropy) with soft loss (KL divergence).\n\n    Args:\n        temperature: Softens probability distributions (higher = softer)\n        alpha: Weight for distillation loss (1-alpha for hard loss)\n    \"\"\"\n    # Hard loss: Standard cross-entropy with true labels\n    hard_loss = F.cross_entropy(student_logits.view(-1, student_logits.size(-1)), labels.view(-1))\n\n    # Soft loss: KL divergence between student and teacher\n    soft_targets = F.softmax(teacher_logits \u002F temperature, dim=-1)\n    soft_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n    soft_loss = F.kl_div(soft_student, soft_targets, reduction='batchmean') * (temperature ** 2)\n\n    # Combined loss\n    return alpha * soft_loss + (1 - alpha) * hard_loss\n\n# 3. Training loop\nfor batch in dataloader:\n    # Teacher forward (no grad)\n    with torch.no_grad():\n        teacher_outputs = teacher(**batch)\n        teacher_logits = teacher_outputs.logits\n\n    # Student forward\n    student_outputs = student(**batch)\n    student_logits = student_outputs.logits\n\n    # Compute distillation loss\n    loss = distillation_loss(\n        student_logits,\n        teacher_logits,\n        batch['labels'],\n        temperature=2.0,\n        alpha=0.7  # 70% soft, 30% hard\n    )\n\n    # Backward and optimize\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n```\n\n### MiniLLM (Reverse KLD)\n\n**Source**: arXiv 2306.08543 (2024)\n\n**Innovation**: Use reverse KLD instead of forward KLD for better generative model distillation.\n\n```python\ndef reverse_kl_loss(student_logits, teacher_logits, temperature=1.0):\n    \"\"\"\n    Reverse KL divergence: KL(Teacher || Student)\n    Better for generative models than forward KL.\n    \"\"\"\n    # Teacher distribution (target)\n    p_teacher = F.softmax(teacher_logits \u002F temperature, dim=-1)\n\n    # Student distribution (model)\n    log_p_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n\n    # Reverse KL: Sum over teacher, student learns to cover teacher's modes\n    reverse_kl = -(p_teacher * log_p_student).sum(dim=-1).mean()\n\n    return reverse_kl * (temperature ** 2)\n\n# Training with MiniLLM\nfor batch in dataloader:\n    with torch.no_grad():\n        teacher_logits = teacher(**batch).logits\n\n    student_logits = student(**batch).logits\n\n    # Reverse KLD (better for generation)\n    loss = reverse_kl_loss(student_logits, teacher_logits, temperature=1.0)\n\n    loss.backward()\n    optimizer.step()\n```\n\n**Why reverse KL?**\n- **Forward KL** (standard): Student learns to match teacher's *mean*\n- **Reverse KL** (MiniLLM): Student learns to *cover* all teacher's modes\n- Better for diverse text generation\n\n### Response Distillation\n\n```python\n# Generate synthetic data from teacher, train student to imitate\n\n# 1. Generate synthetic responses from teacher\nprompts = [\"Explain AI:\", \"What is ML?\", \"Define NLP:\"]\n\nteacher_responses = []\nfor prompt in prompts:\n    inputs = tokenizer(prompt, return_tensors='pt').to(teacher.device)\n    outputs = teacher.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)\n    response = tokenizer.decode(outputs[0], skip_special_tokens=True)\n    teacher_responses.append(response)\n\n# 2. Train student on teacher's responses (standard fine-tuning)\ntrain_dataset = [\n    {\"text\": f\"{prompt}\\n{response}\"}\n    for prompt, response in zip(prompts, teacher_responses)\n]\n\n# 3. Fine-tune student\ntrainer = Trainer(\n    model=student,\n    args=TrainingArguments(output_dir=\".\u002Fstudent\", num_train_epochs=3, learning_rate=2e-5),\n    train_dataset=train_dataset,\n)\ntrainer.train()\n```\n\n## Core Concepts\n\n### 1. Temperature Scaling\n\n**Purpose**: Soften probability distributions to expose teacher's uncertainty.\n\n```python\n# Low temperature (T=1): Sharp distribution\nlogits = [3.0, 2.0, 1.0]\nprobs_T1 = softmax(logits \u002F 1.0)  # [0.67, 0.24, 0.09]\n\n# High temperature (T=4): Soft distribution\nprobs_T4 = softmax(logits \u002F 4.0)  # [0.42, 0.34, 0.24]\n\n# Higher T reveals more information about relative rankings\n```\n\n**Rule**: Use T=2-5 for distillation (2 is common default).\n\n### 2. Loss Function Components\n\n```python\n# Total loss = alpha * soft_loss + (1 - alpha) * hard_loss\n\n# Soft loss: Learn from teacher's knowledge\nsoft_loss = KL(student || teacher)\n\n# Hard loss: Learn from ground truth labels\nhard_loss = CrossEntropy(student_output, true_labels)\n\n# Typical values:\nalpha = 0.5  # Balanced\nalpha = 0.7  # More emphasis on teacher\nalpha = 0.3  # More emphasis on labels\n```\n\n### 3. Forward vs Reverse KLD\n\n```python\n# Forward KL: KL(Student || Teacher)\n# - Student matches teacher's average behavior\n# - Mode-seeking: Student focuses on teacher's highest probability modes\n# - Good for classification\n\n# Reverse KL: KL(Teacher || Student)\n# - Student covers all of teacher's behaviors\n# - Mode-covering: Student learns diverse behaviors\n# - Good for generation (MiniLLM)\n```\n\n## Training Strategies\n\n### Strategy 1: Logit Distillation\n\n```python\n# Train student to match teacher's logits directly\n\ndef logit_distillation_trainer(student, teacher, dataloader, temperature=2.0):\n    optimizer = torch.optim.AdamW(student.parameters(), lr=2e-5)\n\n    for epoch in range(3):\n        for batch in dataloader:\n            # Get logits\n            with torch.no_grad():\n                teacher_logits = teacher(**batch).logits\n\n            student_logits = student(**batch).logits\n\n            # MSE on logits (alternative to KLD)\n            loss = F.mse_loss(student_logits, teacher_logits)\n\n            # Or use KLD\n            # loss = F.kl_div(\n            #     F.log_softmax(student_logits\u002Ftemperature, dim=-1),\n            #     F.softmax(teacher_logits\u002Ftemperature, dim=-1),\n            #     reduction='batchmean'\n            # ) * (temperature ** 2)\n\n            loss.backward()\n            optimizer.step()\n            optimizer.zero_grad()\n\n    return student\n```\n\n### Strategy 2: Two-Stage Distillation\n\n```python\n# Stage 1: Distill from teacher\nstudent = distill(teacher, student, epochs=5)\n\n# Stage 2: Fine-tune on task-specific data\nstudent = fine_tune(student, task_data, epochs=3)\n\n# Results in better task performance than single-stage\n```\n\n### Strategy 3: Multi-Teacher Distillation\n\n```python\n# Learn from multiple expert teachers\n\ndef multi_teacher_distillation(student, teachers, batch):\n    \"\"\"Distill from ensemble of teachers.\"\"\"\n    teacher_logits_list = []\n\n    # Get logits from all teachers\n    with torch.no_grad():\n        for teacher in teachers:\n            logits = teacher(**batch).logits\n            teacher_logits_list.append(logits)\n\n    # Average teacher predictions\n    avg_teacher_logits = torch.stack(teacher_logits_list).mean(dim=0)\n\n    # Student learns from ensemble\n    student_logits = student(**batch).logits\n    loss = F.kl_div(\n        F.log_softmax(student_logits, dim=-1),\n        F.softmax(avg_teacher_logits, dim=-1),\n        reduction='batchmean'\n    )\n\n    return loss\n```\n\n## Production Deployment\n\n### Complete Training Script\n\n```python\nfrom transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling\n\ndef train_distilled_model(\n    teacher_name=\"meta-llama\u002FLlama-2-70b-hf\",\n    student_name=\"meta-llama\u002FLlama-2-7b-hf\",\n    output_dir=\".\u002Fdistilled-llama-7b\",\n    temperature=2.0,\n    alpha=0.7,\n):\n    # Load models\n    teacher = AutoModelForCausalLM.from_pretrained(teacher_name, torch_dtype=torch.float16, device_map=\"auto\")\n    student = AutoModelForCausalLM.from_pretrained(student_name, torch_dtype=torch.float16)\n    tokenizer = AutoTokenizer.from_pretrained(teacher_name)\n\n    # Custom trainer with distillation\n    class DistillationTrainer(Trainer):\n        def compute_loss(self, model, inputs, return_outputs=False):\n            # Student forward\n            outputs_student = model(**inputs)\n            student_logits = outputs_student.logits\n\n            # Teacher forward (no grad)\n            with torch.no_grad():\n                outputs_teacher = teacher(**inputs)\n                teacher_logits = outputs_teacher.logits\n\n            # Distillation loss\n            soft_targets = F.softmax(teacher_logits \u002F temperature, dim=-1)\n            soft_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n            soft_loss = F.kl_div(soft_student, soft_targets, reduction='batchmean') * (temperature ** 2)\n\n            # Hard loss\n            hard_loss = outputs_student.loss\n\n            # Combined\n            loss = alpha * soft_loss + (1 - alpha) * hard_loss\n\n            return (loss, outputs_student) if return_outputs else loss\n\n    # Training arguments\n    training_args = TrainingArguments(\n        output_dir=output_dir,\n        num_train_epochs=3,\n        per_device_train_batch_size=4,\n        gradient_accumulation_steps=8,\n        learning_rate=2e-5,\n        warmup_steps=500,\n        logging_steps=100,\n        save_steps=1000,\n        bf16=True,\n        gradient_checkpointing=True,\n    )\n\n    # Train\n    trainer = DistillationTrainer(\n        model=student,\n        args=training_args,\n        train_dataset=train_dataset,\n        data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),\n    )\n\n    trainer.train()\n    student.save_pretrained(output_dir)\n    tokenizer.save_pretrained(output_dir)\n\n# Usage\ntrain_distilled_model(\n    teacher_name=\"meta-llama\u002FLlama-2-70b-hf\",\n    student_name=\"meta-llama\u002FLlama-2-7b-hf\",\n    temperature=2.0,\n    alpha=0.7\n)\n```\n\n## Best Practices\n\n### 1. Hyperparameter Selection\n\n```python\n# Temperature\nT = 1.0  # Sharp (less knowledge transfer)\nT = 2.0  # Standard (good balance)\nT = 5.0  # Soft (more knowledge transfer)\n\n# Alpha (weight)\nalpha = 0.5  # Balanced\nalpha = 0.7  # Emphasize teacher knowledge\nalpha = 0.9  # Strong distillation\n\n# Rule: Higher T + higher alpha = stronger distillation\n```\n\n### 2. Model Size Ratio\n\n```python\n# Good ratios (teacher\u002Fstudent)\n70B \u002F 7B = 10×    # Excellent\n13B \u002F 1B = 13×    # Good\n7B \u002F 1B = 7×      # Acceptable\n\n# Avoid too large gap\n70B \u002F 1B = 70×    # Too large, ineffective\n```\n\n### 3. Data Quality\n\n```python\n# Best: Use teacher-generated data + real data\ntrain_data = {\n    \"teacher_generated\": 70%,  # Diverse, high-quality\n    \"real_data\": 30%            # Ground truth\n}\n\n# Avoid: Only real data (doesn't utilize teacher fully)\n```\n\n## Evaluation\n\n```python\nfrom transformers import pipeline\n\n# Compare student vs teacher\nteacher_pipe = pipeline(\"text-generation\", model=teacher)\nstudent_pipe = pipeline(\"text-generation\", model=student)\n\nprompts = [\"Explain quantum computing:\", \"What is AI?\"]\n\nfor prompt in prompts:\n    teacher_out = teacher_pipe(prompt, max_new_tokens=100)\n    student_out = student_pipe(prompt, max_new_tokens=100)\n\n    print(f\"Prompt: {prompt}\")\n    print(f\"Teacher: {teacher_out[0]['generated_text']}\")\n    print(f\"Student: {student_out[0]['generated_text']}\")\n    print(f\"Match quality: {calculate_similarity(teacher_out, student_out):.2f}\")\n```\n\n## Resources\n\n- **Hinton et al. 2015 (Foundational)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F1503.02531\n- **MiniLLM (Reverse KLD)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.08543\n- **KD Survey for LLMs (2024)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2402.13116\n- **MiniLLM GitHub**: https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLMOps\u002Ftree\u002Fmain\u002Fminillm\n\n\n",{"name":5,"tags":52,"author":53,"license":15,"version":54,"description":7,"dependencies":55},[27,26,39,37,36,35,34,33,32,31],"Orchestra Research","1.0.0",[56,57,58],"transformers","torch","datasets","main","90b50ed0778745a80fd5bfb13b1843d45b9e82f7","tests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-knowledge-distillation",233,"https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust","https:\u002F\u002Fgithub.com\u002FDicklesworthstone","https:\u002F\u002Fwww.jeffreyemanuel.com\u002F","Building in NY","2026-11-13 03:33:43",{"source":8,"skillName":5,"commands":69,"prompt":79,"zipAvailable":80},[70,73,76],{"tool":71,"command":72},"npx","npx skills add Dicklesworthstone\u002Fpi_agent_rust --skill knowledge-distillation",{"tool":74,"command":75},"bunx","bunx skills add Dicklesworthstone\u002Fpi_agent_rust --skill knowledge-distillation",{"tool":77,"command":78},"pnpm","pnpm dlx skills add Dicklesworthstone\u002Fpi_agent_rust --skill knowledge-distillation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-knowledge-distillation\u002FSKILL.md",true,[82],{"path":83,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":86,"sourceHtmlUrl":87},"references\u002Fminillm.md","REFERENCE","参考文档",9117,"https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-knowledge-distillation\u002Freferences\u002Fminillm.md",[89,128,153,178,203,225],{"skillNo":90,"name":91,"nameZh":6,"description":92,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":93,"licenseFileUrl":94,"sourceHtmlUrl":95,"categories":96,"keywords":100,"publishedAt":47},"01M6FEGPPK0ZTKCCM0R3E3KS33","web-artifacts-builder","Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn\u002Fui). Use for complex artifacts requiring state management, routing, or shadcn\u002Fui components - not for simple single-file HTML\u002FJSX artifacts.","Apache-2.0","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Futilities\u002Fweb-artifacts-builder\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Futilities\u002Fweb-artifacts-builder\u002FSKILL.md",[97],{"categoryNo":98,"categoryName":99,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000009","前端开发",[101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127],"web artifacts builder","web","artifacts","artifacts builder","builder","frontend","css","components","react","component","routing","state management","state","shadcn","file","ui","Tools","Tailwind","use","html","single-file","simple","ui components","multi","for","jsx","suite",{"skillNo":129,"name":130,"nameZh":6,"description":131,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":93,"licenseFileUrl":132,"sourceHtmlUrl":133,"categories":134,"keywords":139,"publishedAt":47},"01M6FEE9VTK1W4GY21AKREJXRG","webapp-testing","Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fsecurity\u002Fwebapp-testing\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fsecurity\u002Fwebapp-testing\u002FSKILL.md",[135,138],{"categoryNo":136,"categoryName":137,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000008","测试与质量",{"categoryNo":98,"categoryName":99,"taxonomyType":21,"taxonomyTypeDesc":22},[140,141,142,106,143,144,102,145,146,147,116,148,149,150,151,152,125],"testing","webapp testing","webapp","screenshots","playwright","debugging","toolkit","browser","logs","applications","behavior","local","with",{"skillNo":154,"name":155,"nameZh":6,"description":156,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":157,"sourceHtmlUrl":158,"categories":159,"keywords":160,"publishedAt":47},"01M6FE20F7ZTQBFWKF4MPJ2SBX","neuropixels-analysis","Neuropixels neural recording analysis. Load SpikeGLX\u002FOpenEphys data, preprocess, motion correction, Kilosort4 spike sorting, quality metrics, Allen\u002FIBL curation, AI-assisted visual analysis, for Neuropixels 1.0\u002F2.0 extracellular electrophysiology. Use when working with neural recordings, spike sorting, extracellular electrophysiology, or when the user mentions Neuropixels, SpikeGLX, Open Ephys, Kilosort, quality metrics, or unit curation.","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fscientific\u002Fneuropixels-analysis\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fscientific\u002Fneuropixels-analysis\u002FSKILL.md",[],[161,162,163,164,165,43,166,167,119,168,169,170,171,172,173,174,152,175,176,177,125,28],"analysis","neuropixels analysis","spike","metrics","data","quality","user","recordings","Neural","motion","visual","quality metrics","unit","sorting","curation","recording","load",{"skillNo":179,"name":180,"nameZh":6,"description":181,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":93,"licenseFileUrl":182,"sourceHtmlUrl":183,"categories":184,"keywords":188,"publishedAt":47},"01M6FDP0PDK6F1GGX395C7625Z","skill-creator","Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Claude's capabilities with specialized knowledge, workflows, or tool integrations.","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Fskill-creator\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Fskill-creator\u002FSKILL.md",[185],{"categoryNo":186,"categoryName":187,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发",[189,190,191,192,193,45,24,194,195,196,197,198,199,200,152,125,201,28,202],"skill creator","creator","workflows","create","update","tool","integrations","specialized","guide","new skill","existing skill","users","Existing","new",{"skillNo":204,"name":205,"nameZh":6,"description":206,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":207,"sourceHtmlUrl":208,"categories":209,"keywords":211,"publishedAt":47},"01M6FDM8Y7VK7NYG9SM7QPA3PW","notebooklm","Use this skill to query your Google NotebookLM notebooks directly from Claude Code for source-grounded, citation-backed answers from Gemini. Browser automation, library management, persistent auth. Drastically reduced hallucinations through document-only responses.","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Fnotebooklm\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Fnotebooklm\u002FSKILL.md",[210],{"categoryNo":186,"categoryName":187,"taxonomyType":21,"taxonomyTypeDesc":22},[205,212,213,214,215,216,217,147,218,219,119,220,221,222,223,125,224],"google","browser automation","gemini","automation","auth","code","library","management","only","document","query","Citation","source",{"skillNo":226,"name":227,"nameZh":6,"description":228,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":93,"licenseFileUrl":229,"sourceHtmlUrl":230,"categories":231,"keywords":238,"publishedAt":47},"01M6FDHPXE10J3XGTCH5HBHS1M","slack-gif-creator","Knowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like \"make me a GIF of X doing Y for Slack.\"","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fslack-gif-creator\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FDicklesworthstone\u002Fpi_agent_rust\u002Fblob\u002Fmain\u002Ftests\u002Fext_conformance\u002Fartifacts\u002Ftemplates-davila7\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fenterprise-communication\u002Fslack-gif-creator\u002FSKILL.md",[232,235],{"categoryNo":233,"categoryName":234,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000005","邮件与沟通",{"categoryNo":236,"categoryName":237,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000016","图像与多媒体",[239,240,241,190,242,117,243,244,24,119,245,200,246,247,248,125,249,28],"slack gif creator","slack","gif","validation","constraints","Animation","me","make","Utilities","concepts","request"]