[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M79TNC2ZN4DAH53CSDVN4NWE":3,"skill-same:01M79TNC2ZN4DAH53CSDVN4NWE":66},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":40,"commitSha":41,"skillDirectory":42,"forksCount":13,"repositoryHtmlUrl":43,"repositoryHomepageUrl":6,"ownerHtmlUrl":44,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":45,"install":46,"resources":59},"01M79TNC2ZN4DAH53CSDVN4NWE","knowledge-distillation",null,"Compress large language models using knowledge distillation from teacher to student models. Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs. Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.","KrismithReddy12\u002Fclaude-code-templates","KrismithReddy12","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F182916557?v=4","01M79T8DNGJRHG48G3H3Q02MK8","01M79TA8ZWV2PP7R4ARWVYWZFE",0,"MIT","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-knowledge-distillation\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: knowledge-distillation\ndescription: Compress large language models using knowledge distillation from teacher to student models. Use when deploying smaller models with retained performance, transferring GPT-4 capabilities to open-source models, or reducing inference costs. Covers temperature scaling, soft targets, reverse KLD, logit distillation, and MiniLLM training strategies.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Emerging Techniques, Knowledge Distillation, Model Compression, Teacher-Student, MiniLLM, Reverse KLD, Soft Targets, Temperature Scaling, Logit Distillation, Model Transfer]\ndependencies: [transformers, torch, datasets]\n---\n\n# Knowledge Distillation: Compressing LLMs\n\n## When to Use This Skill\n\nUse Knowledge Distillation when you need to:\n- **Compress models** from 70B → 7B while retaining 90%+ performance\n- **Transfer capabilities** from proprietary models (GPT-4) to open-source (LLaMA, Mistral)\n- **Reduce inference costs** by deploying smaller student models\n- **Create specialized models** by distilling domain-specific knowledge\n- **Improve small models** using synthetic data from large teachers\n\n**Key Techniques**: Temperature scaling, soft targets, reverse KLD (MiniLLM), logit distillation, response distillation\n\n**Papers**: Hinton et al. 2015 (arXiv 1503.02531), MiniLLM (arXiv 2306.08543), KD Survey (arXiv 2402.13116)\n\n## Installation\n\n```bash\n# Standard transformers\npip install transformers datasets accelerate\n\n# For training\npip install torch deepspeed wandb\n\n# Optional: MiniLLM implementation\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLMOps\ncd LMOps\u002Fminillm\npip install -e .\n```\n\n## Quick Start\n\n### Basic Knowledge Distillation\n\n```python\nimport torch\nimport torch.nn.functional as F\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments\n\n# 1. Load teacher (large) and student (small) models\nteacher = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-70b-hf\",  # Large teacher\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\n\nstudent = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",  # Small student\n    torch_dtype=torch.float16,\n    device_map=\"cuda:0\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-70b-hf\")\n\n# 2. Define distillation loss\ndef distillation_loss(student_logits, teacher_logits, labels, temperature=2.0, alpha=0.5):\n    \"\"\"\n    Combine hard loss (cross-entropy) with soft loss (KL divergence).\n\n    Args:\n        temperature: Softens probability distributions (higher = softer)\n        alpha: Weight for distillation loss (1-alpha for hard loss)\n    \"\"\"\n    # Hard loss: Standard cross-entropy with true labels\n    hard_loss = F.cross_entropy(student_logits.view(-1, student_logits.size(-1)), labels.view(-1))\n\n    # Soft loss: KL divergence between student and teacher\n    soft_targets = F.softmax(teacher_logits \u002F temperature, dim=-1)\n    soft_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n    soft_loss = F.kl_div(soft_student, soft_targets, reduction='batchmean') * (temperature ** 2)\n\n    # Combined loss\n    return alpha * soft_loss + (1 - alpha) * hard_loss\n\n# 3. Training loop\nfor batch in dataloader:\n    # Teacher forward (no grad)\n    with torch.no_grad():\n        teacher_outputs = teacher(**batch)\n        teacher_logits = teacher_outputs.logits\n\n    # Student forward\n    student_outputs = student(**batch)\n    student_logits = student_outputs.logits\n\n    # Compute distillation loss\n    loss = distillation_loss(\n        student_logits,\n        teacher_logits,\n        batch['labels'],\n        temperature=2.0,\n        alpha=0.7  # 70% soft, 30% hard\n    )\n\n    # Backward and optimize\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n```\n\n### MiniLLM (Reverse KLD)\n\n**Source**: arXiv 2306.08543 (2024)\n\n**Innovation**: Use reverse KLD instead of forward KLD for better generative model distillation.\n\n```python\ndef reverse_kl_loss(student_logits, teacher_logits, temperature=1.0):\n    \"\"\"\n    Reverse KL divergence: KL(Teacher || Student)\n    Better for generative models than forward KL.\n    \"\"\"\n    # Teacher distribution (target)\n    p_teacher = F.softmax(teacher_logits \u002F temperature, dim=-1)\n\n    # Student distribution (model)\n    log_p_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n\n    # Reverse KL: Sum over teacher, student learns to cover teacher's modes\n    reverse_kl = -(p_teacher * log_p_student).sum(dim=-1).mean()\n\n    return reverse_kl * (temperature ** 2)\n\n# Training with MiniLLM\nfor batch in dataloader:\n    with torch.no_grad():\n        teacher_logits = teacher(**batch).logits\n\n    student_logits = student(**batch).logits\n\n    # Reverse KLD (better for generation)\n    loss = reverse_kl_loss(student_logits, teacher_logits, temperature=1.0)\n\n    loss.backward()\n    optimizer.step()\n```\n\n**Why reverse KL?**\n- **Forward KL** (standard): Student learns to match teacher's *mean*\n- **Reverse KL** (MiniLLM): Student learns to *cover* all teacher's modes\n- Better for diverse text generation\n\n### Response Distillation\n\n```python\n# Generate synthetic data from teacher, train student to imitate\n\n# 1. Generate synthetic responses from teacher\nprompts = [\"Explain AI:\", \"What is ML?\", \"Define NLP:\"]\n\nteacher_responses = []\nfor prompt in prompts:\n    inputs = tokenizer(prompt, return_tensors='pt').to(teacher.device)\n    outputs = teacher.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)\n    response = tokenizer.decode(outputs[0], skip_special_tokens=True)\n    teacher_responses.append(response)\n\n# 2. Train student on teacher's responses (standard fine-tuning)\ntrain_dataset = [\n    {\"text\": f\"{prompt}\\n{response}\"}\n    for prompt, response in zip(prompts, teacher_responses)\n]\n\n# 3. Fine-tune student\ntrainer = Trainer(\n    model=student,\n    args=TrainingArguments(output_dir=\".\u002Fstudent\", num_train_epochs=3, learning_rate=2e-5),\n    train_dataset=train_dataset,\n)\ntrainer.train()\n```\n\n## Core Concepts\n\n### 1. Temperature Scaling\n\n**Purpose**: Soften probability distributions to expose teacher's uncertainty.\n\n```python\n# Low temperature (T=1): Sharp distribution\nlogits = [3.0, 2.0, 1.0]\nprobs_T1 = softmax(logits \u002F 1.0)  # [0.67, 0.24, 0.09]\n\n# High temperature (T=4): Soft distribution\nprobs_T4 = softmax(logits \u002F 4.0)  # [0.42, 0.34, 0.24]\n\n# Higher T reveals more information about relative rankings\n```\n\n**Rule**: Use T=2-5 for distillation (2 is common default).\n\n### 2. Loss Function Components\n\n```python\n# Total loss = alpha * soft_loss + (1 - alpha) * hard_loss\n\n# Soft loss: Learn from teacher's knowledge\nsoft_loss = KL(student || teacher)\n\n# Hard loss: Learn from ground truth labels\nhard_loss = CrossEntropy(student_output, true_labels)\n\n# Typical values:\nalpha = 0.5  # Balanced\nalpha = 0.7  # More emphasis on teacher\nalpha = 0.3  # More emphasis on labels\n```\n\n### 3. Forward vs Reverse KLD\n\n```python\n# Forward KL: KL(Student || Teacher)\n# - Student matches teacher's average behavior\n# - Mode-seeking: Student focuses on teacher's highest probability modes\n# - Good for classification\n\n# Reverse KL: KL(Teacher || Student)\n# - Student covers all of teacher's behaviors\n# - Mode-covering: Student learns diverse behaviors\n# - Good for generation (MiniLLM)\n```\n\n## Training Strategies\n\n### Strategy 1: Logit Distillation\n\n```python\n# Train student to match teacher's logits directly\n\ndef logit_distillation_trainer(student, teacher, dataloader, temperature=2.0):\n    optimizer = torch.optim.AdamW(student.parameters(), lr=2e-5)\n\n    for epoch in range(3):\n        for batch in dataloader:\n            # Get logits\n            with torch.no_grad():\n                teacher_logits = teacher(**batch).logits\n\n            student_logits = student(**batch).logits\n\n            # MSE on logits (alternative to KLD)\n            loss = F.mse_loss(student_logits, teacher_logits)\n\n            # Or use KLD\n            # loss = F.kl_div(\n            #     F.log_softmax(student_logits\u002Ftemperature, dim=-1),\n            #     F.softmax(teacher_logits\u002Ftemperature, dim=-1),\n            #     reduction='batchmean'\n            # ) * (temperature ** 2)\n\n            loss.backward()\n            optimizer.step()\n            optimizer.zero_grad()\n\n    return student\n```\n\n### Strategy 2: Two-Stage Distillation\n\n```python\n# Stage 1: Distill from teacher\nstudent = distill(teacher, student, epochs=5)\n\n# Stage 2: Fine-tune on task-specific data\nstudent = fine_tune(student, task_data, epochs=3)\n\n# Results in better task performance than single-stage\n```\n\n### Strategy 3: Multi-Teacher Distillation\n\n```python\n# Learn from multiple expert teachers\n\ndef multi_teacher_distillation(student, teachers, batch):\n    \"\"\"Distill from ensemble of teachers.\"\"\"\n    teacher_logits_list = []\n\n    # Get logits from all teachers\n    with torch.no_grad():\n        for teacher in teachers:\n            logits = teacher(**batch).logits\n            teacher_logits_list.append(logits)\n\n    # Average teacher predictions\n    avg_teacher_logits = torch.stack(teacher_logits_list).mean(dim=0)\n\n    # Student learns from ensemble\n    student_logits = student(**batch).logits\n    loss = F.kl_div(\n        F.log_softmax(student_logits, dim=-1),\n        F.softmax(avg_teacher_logits, dim=-1),\n        reduction='batchmean'\n    )\n\n    return loss\n```\n\n## Production Deployment\n\n### Complete Training Script\n\n```python\nfrom transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling\n\ndef train_distilled_model(\n    teacher_name=\"meta-llama\u002FLlama-2-70b-hf\",\n    student_name=\"meta-llama\u002FLlama-2-7b-hf\",\n    output_dir=\".\u002Fdistilled-llama-7b\",\n    temperature=2.0,\n    alpha=0.7,\n):\n    # Load models\n    teacher = AutoModelForCausalLM.from_pretrained(teacher_name, torch_dtype=torch.float16, device_map=\"auto\")\n    student = AutoModelForCausalLM.from_pretrained(student_name, torch_dtype=torch.float16)\n    tokenizer = AutoTokenizer.from_pretrained(teacher_name)\n\n    # Custom trainer with distillation\n    class DistillationTrainer(Trainer):\n        def compute_loss(self, model, inputs, return_outputs=False):\n            # Student forward\n            outputs_student = model(**inputs)\n            student_logits = outputs_student.logits\n\n            # Teacher forward (no grad)\n            with torch.no_grad():\n                outputs_teacher = teacher(**inputs)\n                teacher_logits = outputs_teacher.logits\n\n            # Distillation loss\n            soft_targets = F.softmax(teacher_logits \u002F temperature, dim=-1)\n            soft_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n            soft_loss = F.kl_div(soft_student, soft_targets, reduction='batchmean') * (temperature ** 2)\n\n            # Hard loss\n            hard_loss = outputs_student.loss\n\n            # Combined\n            loss = alpha * soft_loss + (1 - alpha) * hard_loss\n\n            return (loss, outputs_student) if return_outputs else loss\n\n    # Training arguments\n    training_args = TrainingArguments(\n        output_dir=output_dir,\n        num_train_epochs=3,\n        per_device_train_batch_size=4,\n        gradient_accumulation_steps=8,\n        learning_rate=2e-5,\n        warmup_steps=500,\n        logging_steps=100,\n        save_steps=1000,\n        bf16=True,\n        gradient_checkpointing=True,\n    )\n\n    # Train\n    trainer = DistillationTrainer(\n        model=student,\n        args=training_args,\n        train_dataset=train_dataset,\n        data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),\n    )\n\n    trainer.train()\n    student.save_pretrained(output_dir)\n    tokenizer.save_pretrained(output_dir)\n\n# Usage\ntrain_distilled_model(\n    teacher_name=\"meta-llama\u002FLlama-2-70b-hf\",\n    student_name=\"meta-llama\u002FLlama-2-7b-hf\",\n    temperature=2.0,\n    alpha=0.7\n)\n```\n\n## Best Practices\n\n### 1. Hyperparameter Selection\n\n```python\n# Temperature\nT = 1.0  # Sharp (less knowledge transfer)\nT = 2.0  # Standard (good balance)\nT = 5.0  # Soft (more knowledge transfer)\n\n# Alpha (weight)\nalpha = 0.5  # Balanced\nalpha = 0.7  # Emphasize teacher knowledge\nalpha = 0.9  # Strong distillation\n\n# Rule: Higher T + higher alpha = stronger distillation\n```\n\n### 2. Model Size Ratio\n\n```python\n# Good ratios (teacher\u002Fstudent)\n70B \u002F 7B = 10×    # Excellent\n13B \u002F 1B = 13×    # Good\n7B \u002F 1B = 7×      # Acceptable\n\n# Avoid too large gap\n70B \u002F 1B = 70×    # Too large, ineffective\n```\n\n### 3. Data Quality\n\n```python\n# Best: Use teacher-generated data + real data\ntrain_data = {\n    \"teacher_generated\": 70%,  # Diverse, high-quality\n    \"real_data\": 30%            # Ground truth\n}\n\n# Avoid: Only real data (doesn't utilize teacher fully)\n```\n\n## Evaluation\n\n```python\nfrom transformers import pipeline\n\n# Compare student vs teacher\nteacher_pipe = pipeline(\"text-generation\", model=teacher)\nstudent_pipe = pipeline(\"text-generation\", model=student)\n\nprompts = [\"Explain quantum computing:\", \"What is AI?\"]\n\nfor prompt in prompts:\n    teacher_out = teacher_pipe(prompt, max_new_tokens=100)\n    student_out = student_pipe(prompt, max_new_tokens=100)\n\n    print(f\"Prompt: {prompt}\")\n    print(f\"Teacher: {teacher_out[0]['generated_text']}\")\n    print(f\"Student: {student_out[0]['generated_text']}\")\n    print(f\"Match quality: {calculate_similarity(teacher_out, student_out):.2f}\")\n```\n\n## Resources\n\n- **Hinton et al. 2015 (Foundational)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F1503.02531\n- **MiniLLM (Reverse KLD)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.08543\n- **KD Survey for LLMs (2024)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2402.13116\n- **MiniLLM GitHub**: https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLMOps\u002Ftree\u002Fmain\u002Fminillm\n\n\n","# Knowledge Distillation: Compressing LLMs\n\n## When to Use This Skill\n\nUse Knowledge Distillation when you need to:\n- **Compress models** from 70B → 7B while retaining 90%+ performance\n- **Transfer capabilities** from proprietary models (GPT-4) to open-source (LLaMA, Mistral)\n- **Reduce inference costs** by deploying smaller student models\n- **Create specialized models** by distilling domain-specific knowledge\n- **Improve small models** using synthetic data from large teachers\n\n**Key Techniques**: Temperature scaling, soft targets, reverse KLD (MiniLLM), logit distillation, response distillation\n\n**Papers**: Hinton et al. 2015 (arXiv 1503.02531), MiniLLM (arXiv 2306.08543), KD Survey (arXiv 2402.13116)\n\n## Installation\n\n```bash\n# Standard transformers\npip install transformers datasets accelerate\n\n# For training\npip install torch deepspeed wandb\n\n# Optional: MiniLLM implementation\ngit clone https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLMOps\ncd LMOps\u002Fminillm\npip install -e .\n```\n\n## Quick Start\n\n### Basic Knowledge Distillation\n\n```python\nimport torch\nimport torch.nn.functional as F\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments\n\n# 1. Load teacher (large) and student (small) models\nteacher = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-70b-hf\",  # Large teacher\n    torch_dtype=torch.float16,\n    device_map=\"auto\"\n)\n\nstudent = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",  # Small student\n    torch_dtype=torch.float16,\n    device_map=\"cuda:0\"\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-70b-hf\")\n\n# 2. Define distillation loss\ndef distillation_loss(student_logits, teacher_logits, labels, temperature=2.0, alpha=0.5):\n    \"\"\"\n    Combine hard loss (cross-entropy) with soft loss (KL divergence).\n\n    Args:\n        temperature: Softens probability distributions (higher = softer)\n        alpha: Weight for distillation loss (1-alpha for hard loss)\n    \"\"\"\n    # Hard loss: Standard cross-entropy with true labels\n    hard_loss = F.cross_entropy(student_logits.view(-1, student_logits.size(-1)), labels.view(-1))\n\n    # Soft loss: KL divergence between student and teacher\n    soft_targets = F.softmax(teacher_logits \u002F temperature, dim=-1)\n    soft_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n    soft_loss = F.kl_div(soft_student, soft_targets, reduction='batchmean') * (temperature ** 2)\n\n    # Combined loss\n    return alpha * soft_loss + (1 - alpha) * hard_loss\n\n# 3. Training loop\nfor batch in dataloader:\n    # Teacher forward (no grad)\n    with torch.no_grad():\n        teacher_outputs = teacher(**batch)\n        teacher_logits = teacher_outputs.logits\n\n    # Student forward\n    student_outputs = student(**batch)\n    student_logits = student_outputs.logits\n\n    # Compute distillation loss\n    loss = distillation_loss(\n        student_logits,\n        teacher_logits,\n        batch['labels'],\n        temperature=2.0,\n        alpha=0.7  # 70% soft, 30% hard\n    )\n\n    # Backward and optimize\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n```\n\n### MiniLLM (Reverse KLD)\n\n**Source**: arXiv 2306.08543 (2024)\n\n**Innovation**: Use reverse KLD instead of forward KLD for better generative model distillation.\n\n```python\ndef reverse_kl_loss(student_logits, teacher_logits, temperature=1.0):\n    \"\"\"\n    Reverse KL divergence: KL(Teacher || Student)\n    Better for generative models than forward KL.\n    \"\"\"\n    # Teacher distribution (target)\n    p_teacher = F.softmax(teacher_logits \u002F temperature, dim=-1)\n\n    # Student distribution (model)\n    log_p_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n\n    # Reverse KL: Sum over teacher, student learns to cover teacher's modes\n    reverse_kl = -(p_teacher * log_p_student).sum(dim=-1).mean()\n\n    return reverse_kl * (temperature ** 2)\n\n# Training with MiniLLM\nfor batch in dataloader:\n    with torch.no_grad():\n        teacher_logits = teacher(**batch).logits\n\n    student_logits = student(**batch).logits\n\n    # Reverse KLD (better for generation)\n    loss = reverse_kl_loss(student_logits, teacher_logits, temperature=1.0)\n\n    loss.backward()\n    optimizer.step()\n```\n\n**Why reverse KL?**\n- **Forward KL** (standard): Student learns to match teacher's *mean*\n- **Reverse KL** (MiniLLM): Student learns to *cover* all teacher's modes\n- Better for diverse text generation\n\n### Response Distillation\n\n```python\n# Generate synthetic data from teacher, train student to imitate\n\n# 1. Generate synthetic responses from teacher\nprompts = [\"Explain AI:\", \"What is ML?\", \"Define NLP:\"]\n\nteacher_responses = []\nfor prompt in prompts:\n    inputs = tokenizer(prompt, return_tensors='pt').to(teacher.device)\n    outputs = teacher.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)\n    response = tokenizer.decode(outputs[0], skip_special_tokens=True)\n    teacher_responses.append(response)\n\n# 2. Train student on teacher's responses (standard fine-tuning)\ntrain_dataset = [\n    {\"text\": f\"{prompt}\\n{response}\"}\n    for prompt, response in zip(prompts, teacher_responses)\n]\n\n# 3. Fine-tune student\ntrainer = Trainer(\n    model=student,\n    args=TrainingArguments(output_dir=\".\u002Fstudent\", num_train_epochs=3, learning_rate=2e-5),\n    train_dataset=train_dataset,\n)\ntrainer.train()\n```\n\n## Core Concepts\n\n### 1. Temperature Scaling\n\n**Purpose**: Soften probability distributions to expose teacher's uncertainty.\n\n```python\n# Low temperature (T=1): Sharp distribution\nlogits = [3.0, 2.0, 1.0]\nprobs_T1 = softmax(logits \u002F 1.0)  # [0.67, 0.24, 0.09]\n\n# High temperature (T=4): Soft distribution\nprobs_T4 = softmax(logits \u002F 4.0)  # [0.42, 0.34, 0.24]\n\n# Higher T reveals more information about relative rankings\n```\n\n**Rule**: Use T=2-5 for distillation (2 is common default).\n\n### 2. Loss Function Components\n\n```python\n# Total loss = alpha * soft_loss + (1 - alpha) * hard_loss\n\n# Soft loss: Learn from teacher's knowledge\nsoft_loss = KL(student || teacher)\n\n# Hard loss: Learn from ground truth labels\nhard_loss = CrossEntropy(student_output, true_labels)\n\n# Typical values:\nalpha = 0.5  # Balanced\nalpha = 0.7  # More emphasis on teacher\nalpha = 0.3  # More emphasis on labels\n```\n\n### 3. Forward vs Reverse KLD\n\n```python\n# Forward KL: KL(Student || Teacher)\n# - Student matches teacher's average behavior\n# - Mode-seeking: Student focuses on teacher's highest probability modes\n# - Good for classification\n\n# Reverse KL: KL(Teacher || Student)\n# - Student covers all of teacher's behaviors\n# - Mode-covering: Student learns diverse behaviors\n# - Good for generation (MiniLLM)\n```\n\n## Training Strategies\n\n### Strategy 1: Logit Distillation\n\n```python\n# Train student to match teacher's logits directly\n\ndef logit_distillation_trainer(student, teacher, dataloader, temperature=2.0):\n    optimizer = torch.optim.AdamW(student.parameters(), lr=2e-5)\n\n    for epoch in range(3):\n        for batch in dataloader:\n            # Get logits\n            with torch.no_grad():\n                teacher_logits = teacher(**batch).logits\n\n            student_logits = student(**batch).logits\n\n            # MSE on logits (alternative to KLD)\n            loss = F.mse_loss(student_logits, teacher_logits)\n\n            # Or use KLD\n            # loss = F.kl_div(\n            #     F.log_softmax(student_logits\u002Ftemperature, dim=-1),\n            #     F.softmax(teacher_logits\u002Ftemperature, dim=-1),\n            #     reduction='batchmean'\n            # ) * (temperature ** 2)\n\n            loss.backward()\n            optimizer.step()\n            optimizer.zero_grad()\n\n    return student\n```\n\n### Strategy 2: Two-Stage Distillation\n\n```python\n# Stage 1: Distill from teacher\nstudent = distill(teacher, student, epochs=5)\n\n# Stage 2: Fine-tune on task-specific data\nstudent = fine_tune(student, task_data, epochs=3)\n\n# Results in better task performance than single-stage\n```\n\n### Strategy 3: Multi-Teacher Distillation\n\n```python\n# Learn from multiple expert teachers\n\ndef multi_teacher_distillation(student, teachers, batch):\n    \"\"\"Distill from ensemble of teachers.\"\"\"\n    teacher_logits_list = []\n\n    # Get logits from all teachers\n    with torch.no_grad():\n        for teacher in teachers:\n            logits = teacher(**batch).logits\n            teacher_logits_list.append(logits)\n\n    # Average teacher predictions\n    avg_teacher_logits = torch.stack(teacher_logits_list).mean(dim=0)\n\n    # Student learns from ensemble\n    student_logits = student(**batch).logits\n    loss = F.kl_div(\n        F.log_softmax(student_logits, dim=-1),\n        F.softmax(avg_teacher_logits, dim=-1),\n        reduction='batchmean'\n    )\n\n    return loss\n```\n\n## Production Deployment\n\n### Complete Training Script\n\n```python\nfrom transformers import Trainer, TrainingArguments, DataCollatorForLanguageModeling\n\ndef train_distilled_model(\n    teacher_name=\"meta-llama\u002FLlama-2-70b-hf\",\n    student_name=\"meta-llama\u002FLlama-2-7b-hf\",\n    output_dir=\".\u002Fdistilled-llama-7b\",\n    temperature=2.0,\n    alpha=0.7,\n):\n    # Load models\n    teacher = AutoModelForCausalLM.from_pretrained(teacher_name, torch_dtype=torch.float16, device_map=\"auto\")\n    student = AutoModelForCausalLM.from_pretrained(student_name, torch_dtype=torch.float16)\n    tokenizer = AutoTokenizer.from_pretrained(teacher_name)\n\n    # Custom trainer with distillation\n    class DistillationTrainer(Trainer):\n        def compute_loss(self, model, inputs, return_outputs=False):\n            # Student forward\n            outputs_student = model(**inputs)\n            student_logits = outputs_student.logits\n\n            # Teacher forward (no grad)\n            with torch.no_grad():\n                outputs_teacher = teacher(**inputs)\n                teacher_logits = outputs_teacher.logits\n\n            # Distillation loss\n            soft_targets = F.softmax(teacher_logits \u002F temperature, dim=-1)\n            soft_student = F.log_softmax(student_logits \u002F temperature, dim=-1)\n            soft_loss = F.kl_div(soft_student, soft_targets, reduction='batchmean') * (temperature ** 2)\n\n            # Hard loss\n            hard_loss = outputs_student.loss\n\n            # Combined\n            loss = alpha * soft_loss + (1 - alpha) * hard_loss\n\n            return (loss, outputs_student) if return_outputs else loss\n\n    # Training arguments\n    training_args = TrainingArguments(\n        output_dir=output_dir,\n        num_train_epochs=3,\n        per_device_train_batch_size=4,\n        gradient_accumulation_steps=8,\n        learning_rate=2e-5,\n        warmup_steps=500,\n        logging_steps=100,\n        save_steps=1000,\n        bf16=True,\n        gradient_checkpointing=True,\n    )\n\n    # Train\n    trainer = DistillationTrainer(\n        model=student,\n        args=training_args,\n        train_dataset=train_dataset,\n        data_collator=DataCollatorForLanguageModeling(tokenizer, mlm=False),\n    )\n\n    trainer.train()\n    student.save_pretrained(output_dir)\n    tokenizer.save_pretrained(output_dir)\n\n# Usage\ntrain_distilled_model(\n    teacher_name=\"meta-llama\u002FLlama-2-70b-hf\",\n    student_name=\"meta-llama\u002FLlama-2-7b-hf\",\n    temperature=2.0,\n    alpha=0.7\n)\n```\n\n## Best Practices\n\n### 1. Hyperparameter Selection\n\n```python\n# Temperature\nT = 1.0  # Sharp (less knowledge transfer)\nT = 2.0  # Standard (good balance)\nT = 5.0  # Soft (more knowledge transfer)\n\n# Alpha (weight)\nalpha = 0.5  # Balanced\nalpha = 0.7  # Emphasize teacher knowledge\nalpha = 0.9  # Strong distillation\n\n# Rule: Higher T + higher alpha = stronger distillation\n```\n\n### 2. Model Size Ratio\n\n```python\n# Good ratios (teacher\u002Fstudent)\n70B \u002F 7B = 10×    # Excellent\n13B \u002F 1B = 13×    # Good\n7B \u002F 1B = 7×      # Acceptable\n\n# Avoid too large gap\n70B \u002F 1B = 70×    # Too large, ineffective\n```\n\n### 3. Data Quality\n\n```python\n# Best: Use teacher-generated data + real data\ntrain_data = {\n    \"teacher_generated\": 70%,  # Diverse, high-quality\n    \"real_data\": 30%            # Ground truth\n}\n\n# Avoid: Only real data (doesn't utilize teacher fully)\n```\n\n## Evaluation\n\n```python\nfrom transformers import pipeline\n\n# Compare student vs teacher\nteacher_pipe = pipeline(\"text-generation\", model=teacher)\nstudent_pipe = pipeline(\"text-generation\", model=student)\n\nprompts = [\"Explain quantum computing:\", \"What is AI?\"]\n\nfor prompt in prompts:\n    teacher_out = teacher_pipe(prompt, max_new_tokens=100)\n    student_out = student_pipe(prompt, max_new_tokens=100)\n\n    print(f\"Prompt: {prompt}\")\n    print(f\"Teacher: {teacher_out[0]['generated_text']}\")\n    print(f\"Student: {student_out[0]['generated_text']}\")\n    print(f\"Match quality: {calculate_similarity(teacher_out, student_out):.2f}\")\n```\n\n## Resources\n\n- **Hinton et al. 2015 (Foundational)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F1503.02531\n- **MiniLLM (Reverse KLD)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.08543\n- **KD Survey for LLMs (2024)**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2402.13116\n- **MiniLLM GitHub**: https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FLMOps\u002Ftree\u002Fmain\u002Fminillm\n\n\n",{"name":5,"tags":23,"author":34,"license":14,"version":35,"description":7,"dependencies":36},[24,25,26,27,28,29,30,31,32,33],"Emerging Techniques","Knowledge Distillation","Model Compression","Teacher-Student","MiniLLM","Reverse KLD","Soft Targets","Temperature Scaling","Logit Distillation","Model Transfer","Orchestra Research","1.0.0",[37,38,39],"transformers","torch","datasets","main","e3744adb7654df443f12927f3e001252b3a26b6d","cli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-knowledge-distillation","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates","https:\u002F\u002Fgithub.com\u002FKrismithReddy12","2026-07-29 08:15:10",{"source":8,"skillName":5,"commands":47,"prompt":57,"zipAvailable":58},[48,51,54],{"tool":49,"command":50},"npx","npx skills add KrismithReddy12\u002Fclaude-code-templates --skill knowledge-distillation",{"tool":52,"command":53},"bunx","bunx skills add KrismithReddy12\u002Fclaude-code-templates --skill knowledge-distillation",{"tool":55,"command":56},"pnpm","pnpm dlx skills add KrismithReddy12\u002Fclaude-code-templates --skill knowledge-distillation","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-knowledge-distillation\u002FSKILL.md",true,[60],{"path":61,"resourceType":62,"resourceTypeDesc":63,"sizeBytes":64,"sourceHtmlUrl":65},"references\u002Fminillm.md","REFERENCE","参考文档",9117,"https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-knowledge-distillation\u002Freferences\u002Fminillm.md",[67,74,83,90,97,104],{"skillNo":68,"name":69,"nameZh":6,"description":70,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":71,"categories":72,"keywords":73,"publishedAt":18},"01M79YMXJ07S2BQVB7JAM3657Q","zapier-make-patterns","No-code automation democratizes workflow building. Zapier and Make (formerly Integromat) let non-developers automate business processes without writing code. But no-code doesn't mean no-complexity - these platforms have their own patterns, pitfalls, and breaking points.  This skill covers when to use which platform, how to build reliable automations, and when to graduate to code-based solutions. Key insight: Zapier optimizes for simplicity and integrations (7000+ apps), Make optimizes for power","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fzapier-make-patterns\u002FSKILL.md",[],[],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":78,"licenseFileUrl":79,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":18},"01M79YMVPMK3Y4JDP2KDJKQ2ZD","yeet","Use only when the user explicitly asks to stage, commit, push, and open a GitHub pull request in one flow using the GitHub CLI (`gh`).","Apache-2.0","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fyeet\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fyeet\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":18},"01M79YMMQ4XXMPJHDQHY8FR7ME","workflow-automation","Workflow automation is the infrastructure that makes AI agents reliable. Without durable execution, a network hiccup during a 10-step payment flow means lost money and angry customers. With it, workflows resume exactly where they left off.  This skill covers the platforms (n8n, Temporal, Inngest) and patterns (sequential, parallel, orchestrator-worker) that turn brittle scripts into production-grade automation.  Key insight: The platforms make different tradeoffs. n8n optimizes for accessibility","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fworkflow-automation\u002FSKILL.md",[],[],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":94,"categories":95,"keywords":96,"publishedAt":18},"01M79YMHQY3CP1XXMQ00TGQFAW","trigger-dev","Trigger.dev expert for background jobs, AI workflows, and reliable async execution with excellent developer experience and TypeScript-first design. Use when: trigger.dev, trigger dev, background task, ai background job, long running task.","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Ftrigger-dev\u002FSKILL.md",[],[],{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":18},"01M79YME4Z3DRW0MT6BNYW1D7P","slack-automation","Automate Slack workspace operations including messaging, search, channel management, and reaction workflows through Composio's Slack toolkit.","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fslack-automation\u002FSKILL.md",[],[],{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":18},"01M79YMAX7NFYDFXFZH2TJS7SS","planning-with-files","Transforms workflow to use Manus-style persistent markdown files for planning, progress tracking, and knowledge storage. Use when starting complex tasks, multi-step projects, research tasks, or when the user mentions planning, organizing work, tracking progress, or wants structured output.","https:\u002F\u002Fgithub.com\u002FKrismithReddy12\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fplanning-with-files\u002FSKILL.md",[],[]]