[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M72CPF4DDDF6Q30H5541XDN9":3,"skill-same:01M72CPF4DDDF6Q30H5541XDN9":79},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":41,"commitSha":42,"skillDirectory":43,"forksCount":13,"repositoryHtmlUrl":44,"repositoryHomepageUrl":6,"ownerHtmlUrl":45,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":46,"install":47,"resources":60},"01M72CPF4DDDF6Q30H5541XDN9","model-merging",null,"Merge multiple fine-tuned models using mergekit to combine capabilities without retraining. Use when creating specialized models by blending domain-specific expertise (math + coding + chat), improving performance beyond single models, or experimenting rapidly with model variants. Covers SLERP, TIES-Merging, DARE, Task Arithmetic, linear merging, and production deployment strategies.","Sansar35\u002Fclaude-agent","Sansar35","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F145020281?v=4","01M724CQ2SCC86MGNSD87D0D4K","01M724CQ3965G9WJ3QXE3AJ5YY",1,"MIT","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fmodel-merging\u002FSKILL.md",[],[],"2026-11-10 18:39:20",false,"---\nname: model-merging\ndescription: Merge multiple fine-tuned models using mergekit to combine capabilities without retraining. Use when creating specialized models by blending domain-specific expertise (math + coding + chat), improving performance beyond single models, or experimenting rapidly with model variants. Covers SLERP, TIES-Merging, DARE, Task Arithmetic, linear merging, and production deployment strategies.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Emerging Techniques, Model Merging, Mergekit, SLERP, TIES, DARE, Task Arithmetic, Model Fusion, No Retraining, Multi-Capability, Arcee AI]\ndependencies: [mergekit, transformers, torch]\n---\n\n# Model Merging: Combining Pre-trained Models\n\n## When to Use This Skill\n\nUse Model Merging when you need to:\n- **Combine capabilities** from multiple fine-tuned models without retraining\n- **Create specialized models** by blending domain-specific expertise (math + coding + chat)\n- **Improve performance** beyond single models (often +5-10% on benchmarks)\n- **Reduce training costs** - no GPUs needed, merges run on CPU\n- **Experiment rapidly** - create new model variants in minutes, not days\n- **Preserve multiple skills** - merge without catastrophic forgetting\n\n**Success Stories**: Marcoro14-7B-slerp (best on Open LLM Leaderboard 02\u002F2024), many top HuggingFace models use merging\n\n**Tools**: mergekit (Arcee AI), LazyMergekit, Model Soup\n\n## Installation\n\n```bash\n# Install mergekit\ngit clone https:\u002F\u002Fgithub.com\u002Farcee-ai\u002Fmergekit.git\ncd mergekit\npip install -e .\n\n# Or via pip\npip install mergekit\n\n# Optional: Transformer library\npip install transformers torch\n```\n\n## Quick Start\n\n### Simple Linear Merge\n\n```yaml\n# config.yml - Merge two models with equal weights\nmerge_method: linear\nmodels:\n  - model: mistralai\u002FMistral-7B-v0.1\n    parameters:\n      weight: 0.5\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    parameters:\n      weight: 0.5\ndtype: bfloat16\n```\n\n```bash\n# Run merge\nmergekit-yaml config.yml .\u002Fmerged-model --cuda\n\n# Use merged model\npython -m transformers.models.auto --model_name_or_path .\u002Fmerged-model\n```\n\n### SLERP Merge (Best for 2 Models)\n\n```yaml\n# config.yml - Spherical interpolation\nmerge_method: slerp\nslices:\n  - sources:\n      - model: mistralai\u002FMistral-7B-v0.1\n        layer_range: [0, 32]\n      - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n        layer_range: [0, 32]\nparameters:\n  t: 0.5  # Interpolation factor (0=model1, 1=model2)\ndtype: bfloat16\n```\n\n## Core Concepts\n\n### 1. Merge Methods\n\n**Linear (Model Soup)**\n- Simple weighted average of parameters\n- Fast, works well for similar models\n- Can merge 2+ models\n\n```python\nmerged_weights = w1 * model1_weights + w2 * model2_weights + w3 * model3_weights\n# where w1 + w2 + w3 = 1\n```\n\n**SLERP (Spherical Linear Interpolation)**\n- Interpolates along sphere in weight space\n- Preserves magnitude of weight vectors\n- Best for merging 2 models\n- Smoother than linear\n\n```python\n# SLERP formula\nmerged = (sin((1-t)*θ) \u002F sin(θ)) * model1 + (sin(t*θ) \u002F sin(θ)) * model2\n# where θ = arccos(dot(model1, model2))\n# t ∈ [0, 1]\n```\n\n**Task Arithmetic**\n- Extract \"task vectors\" (fine-tuned - base)\n- Combine task vectors, add to base\n- Good for merging multiple specialized models\n\n```python\n# Task vector\ntask_vector = finetuned_model - base_model\n\n# Merge multiple task vectors\nmerged = base_model + α₁*task_vector₁ + α₂*task_vector₂\n```\n\n**TIES-Merging**\n- Task arithmetic + sparsification\n- Resolves sign conflicts in parameters\n- Best for merging many task-specific models\n\n**DARE (Drop And REscale)**\n- Randomly drops fine-tuned parameters\n- Rescales remaining parameters\n- Reduces redundancy, maintains performance\n\n### 2. Configuration Structure\n\n```yaml\n# Basic structure\nmerge_method: \u003Cmethod>  # linear, slerp, ties, dare_ties, task_arithmetic\nbase_model: \u003Cpath>      # Optional: base model for task arithmetic\n\nmodels:\n  - model: \u003Cpath\u002Fto\u002Fmodel1>\n    parameters:\n      weight: \u003Cfloat>   # Merge weight\n      density: \u003Cfloat>  # For TIES\u002FDARE\n\n  - model: \u003Cpath\u002Fto\u002Fmodel2>\n    parameters:\n      weight: \u003Cfloat>\n\nparameters:\n  # Method-specific parameters\n\ndtype: \u003Cdtype>  # bfloat16, float16, float32\n\n# Optional\nslices:  # Layer-wise merging\ntokenizer:  # Tokenizer configuration\n```\n\n## Merge Methods Guide\n\n### Linear Merge\n\n**Best for**: Simple model combinations, equal weighting\n\n```yaml\nmerge_method: linear\nmodels:\n  - model: WizardLM\u002FWizardMath-7B-V1.1\n    parameters:\n      weight: 0.4\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    parameters:\n      weight: 0.3\n  - model: NousResearch\u002FNous-Hermes-2-Mistral-7B-DPO\n    parameters:\n      weight: 0.3\ndtype: bfloat16\n```\n\n### SLERP Merge\n\n**Best for**: Two models, smooth interpolation\n\n```yaml\nmerge_method: slerp\nslices:\n  - sources:\n      - model: mistralai\u002FMistral-7B-v0.1\n        layer_range: [0, 32]\n      - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n        layer_range: [0, 32]\nparameters:\n  t: 0.5  # 0.0 = first model, 1.0 = second model\ndtype: bfloat16\n```\n\n**Layer-specific SLERP:**\n\n```yaml\nmerge_method: slerp\nslices:\n  - sources:\n      - model: model_a\n        layer_range: [0, 32]\n      - model: model_b\n        layer_range: [0, 32]\nparameters:\n  t:\n    - filter: self_attn    # Attention layers\n      value: 0.3\n    - filter: mlp          # MLP layers\n      value: 0.7\n    - value: 0.5           # Default for other layers\ndtype: bfloat16\n```\n\n### Task Arithmetic\n\n**Best for**: Combining specialized skills\n\n```yaml\nmerge_method: task_arithmetic\nbase_model: mistralai\u002FMistral-7B-v0.1\nmodels:\n  - model: WizardLM\u002FWizardMath-7B-V1.1  # Math\n    parameters:\n      weight: 0.5\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B  # Chat\n    parameters:\n      weight: 0.3\n  - model: ajibawa-2023\u002FCode-Mistral-7B  # Code\n    parameters:\n      weight: 0.2\ndtype: bfloat16\n```\n\n### TIES-Merging\n\n**Best for**: Many models, resolving conflicts\n\n```yaml\nmerge_method: ties\nbase_model: mistralai\u002FMistral-7B-v0.1\nmodels:\n  - model: WizardLM\u002FWizardMath-7B-V1.1\n    parameters:\n      density: 0.5  # Keep top 50% of parameters\n      weight: 1.0\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    parameters:\n      density: 0.5\n      weight: 1.0\n  - model: NousResearch\u002FNous-Hermes-2-Mistral-7B-DPO\n    parameters:\n      density: 0.5\n      weight: 1.0\nparameters:\n  normalize: true\ndtype: bfloat16\n```\n\n### DARE Merge\n\n**Best for**: Reducing redundancy\n\n```yaml\nmerge_method: dare_ties\nbase_model: mistralai\u002FMistral-7B-v0.1\nmodels:\n  - model: WizardLM\u002FWizardMath-7B-V1.1\n    parameters:\n      density: 0.5    # Drop 50% of deltas\n      weight: 0.6\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    parameters:\n      density: 0.5\n      weight: 0.4\nparameters:\n  int8_mask: true  # Use int8 for masks (saves memory)\ndtype: bfloat16\n```\n\n## Advanced Patterns\n\n### Layer-wise Merging\n\n```yaml\n# Different models for different layers\nmerge_method: passthrough\nslices:\n  - sources:\n      - model: mistralai\u002FMistral-7B-v0.1\n        layer_range: [0, 16]   # First half\n  - sources:\n      - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n        layer_range: [16, 32]  # Second half\ndtype: bfloat16\n```\n\n### MoE from Merged Models\n\n```yaml\n# Create Mixture of Experts\nmerge_method: moe\nbase_model: mistralai\u002FMistral-7B-v0.1\nexperts:\n  - source_model: WizardLM\u002FWizardMath-7B-V1.1\n    positive_prompts:\n      - \"math\"\n      - \"calculate\"\n  - source_model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    positive_prompts:\n      - \"chat\"\n      - \"conversation\"\n  - source_model: ajibawa-2023\u002FCode-Mistral-7B\n    positive_prompts:\n      - \"code\"\n      - \"python\"\ndtype: bfloat16\n```\n\n### Tokenizer Merging\n\n```yaml\nmerge_method: linear\nmodels:\n  - model: mistralai\u002FMistral-7B-v0.1\n  - model: custom\u002Fspecialized-model\n\ntokenizer:\n  source: \"union\"  # Combine vocabularies from both models\n  tokens:\n    \u003C|special_token|>:\n      source: \"custom\u002Fspecialized-model\"\n```\n\n## Best Practices\n\n### 1. Model Compatibility\n\n```python\n# ✅ Good: Same architecture\nmodels = [\n    \"mistralai\u002FMistral-7B-v0.1\",\n    \"teknium\u002FOpenHermes-2.5-Mistral-7B\",  # Both Mistral 7B\n]\n\n# ❌ Bad: Different architectures\nmodels = [\n    \"meta-llama\u002FLlama-2-7b-hf\",  # Llama\n    \"mistralai\u002FMistral-7B-v0.1\",  # Mistral (incompatible!)\n]\n```\n\n### 2. Weight Selection\n\n```yaml\n# ✅ Good: Weights sum to 1.0\nmodels:\n  - model: model_a\n    parameters:\n      weight: 0.6\n  - model: model_b\n    parameters:\n      weight: 0.4  # 0.6 + 0.4 = 1.0\n\n# ⚠️  Acceptable: Weights don't sum to 1 (for task arithmetic)\nmodels:\n  - model: model_a\n    parameters:\n      weight: 0.8\n  - model: model_b\n    parameters:\n      weight: 0.8  # May boost performance\n```\n\n### 3. Method Selection\n\n```python\n# Choose merge method based on use case:\n\n# 2 models, smooth blend → SLERP\nmerge_method = \"slerp\"\n\n# 3+ models, simple average → Linear\nmerge_method = \"linear\"\n\n# Multiple task-specific models → Task Arithmetic or TIES\nmerge_method = \"ties\"\n\n# Want to reduce redundancy → DARE\nmerge_method = \"dare_ties\"\n```\n\n### 4. Density Tuning (TIES\u002FDARE)\n\n```yaml\n# Start conservative (keep more parameters)\nparameters:\n  density: 0.8  # Keep 80%\n\n# If performance good, increase sparsity\nparameters:\n  density: 0.5  # Keep 50%\n\n# If performance degrades, reduce sparsity\nparameters:\n  density: 0.9  # Keep 90%\n```\n\n### 5. Layer-specific Merging\n\n```yaml\n# Preserve base model's beginning and end\nmerge_method: passthrough\nslices:\n  - sources:\n      - model: base_model\n        layer_range: [0, 2]     # Keep first layers\n  - sources:\n      - model: merged_middle    # Merge middle layers\n        layer_range: [2, 30]\n  - sources:\n      - model: base_model\n        layer_range: [30, 32]   # Keep last layers\n```\n\n## Evaluation & Testing\n\n### Benchmark Merged Models\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load merged model\nmodel = AutoModelForCausalLM.from_pretrained(\".\u002Fmerged-model\")\ntokenizer = AutoTokenizer.from_pretrained(\".\u002Fmerged-model\")\n\n# Test on various tasks\ntest_prompts = {\n    \"math\": \"Calculate: 25 * 17 =\",\n    \"code\": \"Write a Python function to reverse a string:\",\n    \"chat\": \"What is the capital of France?\",\n}\n\nfor task, prompt in test_prompts.items():\n    inputs = tokenizer(prompt, return_tensors=\"pt\")\n    outputs = model.generate(**inputs, max_length=100)\n    print(f\"{task}: {tokenizer.decode(outputs[0])}\")\n```\n\n### Common Benchmarks\n\n- **Open LLM Leaderboard**: General capabilities\n- **MT-Bench**: Multi-turn conversation\n- **MMLU**: Multitask accuracy\n- **HumanEval**: Code generation\n- **GSM8K**: Math reasoning\n\n## Production Deployment\n\n### Save and Upload\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load merged model\nmodel = AutoModelForCausalLM.from_pretrained(\".\u002Fmerged-model\")\ntokenizer = AutoTokenizer.from_pretrained(\".\u002Fmerged-model\")\n\n# Upload to HuggingFace Hub\nmodel.push_to_hub(\"username\u002Fmy-merged-model\")\ntokenizer.push_to_hub(\"username\u002Fmy-merged-model\")\n```\n\n### Quantize Merged Model\n\n```bash\n# Quantize with GGUF\npython convert.py .\u002Fmerged-model --outtype f16 --outfile merged-model.gguf\n\n# Quantize with GPTQ\npython quantize_gptq.py .\u002Fmerged-model --bits 4 --group_size 128\n```\n\n## Common Pitfalls\n\n### ❌ Pitfall 1: Merging Incompatible Models\n\n```yaml\n# Wrong: Different architectures\nmodels:\n  - model: meta-llama\u002FLlama-2-7b  # Llama architecture\n  - model: mistralai\u002FMistral-7B   # Mistral architecture\n```\n\n**Fix**: Only merge models with same architecture\n\n### ❌ Pitfall 2: Over-weighting One Model\n\n```yaml\n# Suboptimal: One model dominates\nmodels:\n  - model: model_a\n    parameters:\n      weight: 0.95  # Too high\n  - model: model_b\n    parameters:\n      weight: 0.05  # Too low\n```\n\n**Fix**: Use more balanced weights (0.3-0.7 range)\n\n### ❌ Pitfall 3: Not Evaluating\n\n```bash\n# Wrong: Merge and deploy without testing\nmergekit-yaml config.yml .\u002Fmerged-model\n# Deploy immediately (risky!)\n```\n\n**Fix**: Always benchmark before deploying\n\n## Resources\n\n- **mergekit GitHub**: https:\u002F\u002Fgithub.com\u002Farcee-ai\u002Fmergekit\n- **HuggingFace Tutorial**: https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fmlabonne\u002Fmerge-models\n- **LazyMergekit**: Automated merging notebook\n- **TIES Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.01708\n- **DARE Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2311.03099\n\n## See Also\n\n- `references\u002Fmethods.md` - Deep dive into merge algorithms\n- `references\u002Fexamples.md` - Real-world merge configurations\n- `references\u002Fevaluation.md` - Benchmarking and testing strategies\n\n\n","# Model Merging: Combining Pre-trained Models\n\n## When to Use This Skill\n\nUse Model Merging when you need to:\n- **Combine capabilities** from multiple fine-tuned models without retraining\n- **Create specialized models** by blending domain-specific expertise (math + coding + chat)\n- **Improve performance** beyond single models (often +5-10% on benchmarks)\n- **Reduce training costs** - no GPUs needed, merges run on CPU\n- **Experiment rapidly** - create new model variants in minutes, not days\n- **Preserve multiple skills** - merge without catastrophic forgetting\n\n**Success Stories**: Marcoro14-7B-slerp (best on Open LLM Leaderboard 02\u002F2024), many top HuggingFace models use merging\n\n**Tools**: mergekit (Arcee AI), LazyMergekit, Model Soup\n\n## Installation\n\n```bash\n# Install mergekit\ngit clone https:\u002F\u002Fgithub.com\u002Farcee-ai\u002Fmergekit.git\ncd mergekit\npip install -e .\n\n# Or via pip\npip install mergekit\n\n# Optional: Transformer library\npip install transformers torch\n```\n\n## Quick Start\n\n### Simple Linear Merge\n\n```yaml\n# config.yml - Merge two models with equal weights\nmerge_method: linear\nmodels:\n  - model: mistralai\u002FMistral-7B-v0.1\n    parameters:\n      weight: 0.5\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    parameters:\n      weight: 0.5\ndtype: bfloat16\n```\n\n```bash\n# Run merge\nmergekit-yaml config.yml .\u002Fmerged-model --cuda\n\n# Use merged model\npython -m transformers.models.auto --model_name_or_path .\u002Fmerged-model\n```\n\n### SLERP Merge (Best for 2 Models)\n\n```yaml\n# config.yml - Spherical interpolation\nmerge_method: slerp\nslices:\n  - sources:\n      - model: mistralai\u002FMistral-7B-v0.1\n        layer_range: [0, 32]\n      - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n        layer_range: [0, 32]\nparameters:\n  t: 0.5  # Interpolation factor (0=model1, 1=model2)\ndtype: bfloat16\n```\n\n## Core Concepts\n\n### 1. Merge Methods\n\n**Linear (Model Soup)**\n- Simple weighted average of parameters\n- Fast, works well for similar models\n- Can merge 2+ models\n\n```python\nmerged_weights = w1 * model1_weights + w2 * model2_weights + w3 * model3_weights\n# where w1 + w2 + w3 = 1\n```\n\n**SLERP (Spherical Linear Interpolation)**\n- Interpolates along sphere in weight space\n- Preserves magnitude of weight vectors\n- Best for merging 2 models\n- Smoother than linear\n\n```python\n# SLERP formula\nmerged = (sin((1-t)*θ) \u002F sin(θ)) * model1 + (sin(t*θ) \u002F sin(θ)) * model2\n# where θ = arccos(dot(model1, model2))\n# t ∈ [0, 1]\n```\n\n**Task Arithmetic**\n- Extract \"task vectors\" (fine-tuned - base)\n- Combine task vectors, add to base\n- Good for merging multiple specialized models\n\n```python\n# Task vector\ntask_vector = finetuned_model - base_model\n\n# Merge multiple task vectors\nmerged = base_model + α₁*task_vector₁ + α₂*task_vector₂\n```\n\n**TIES-Merging**\n- Task arithmetic + sparsification\n- Resolves sign conflicts in parameters\n- Best for merging many task-specific models\n\n**DARE (Drop And REscale)**\n- Randomly drops fine-tuned parameters\n- Rescales remaining parameters\n- Reduces redundancy, maintains performance\n\n### 2. Configuration Structure\n\n```yaml\n# Basic structure\nmerge_method: \u003Cmethod>  # linear, slerp, ties, dare_ties, task_arithmetic\nbase_model: \u003Cpath>      # Optional: base model for task arithmetic\n\nmodels:\n  - model: \u003Cpath\u002Fto\u002Fmodel1>\n    parameters:\n      weight: \u003Cfloat>   # Merge weight\n      density: \u003Cfloat>  # For TIES\u002FDARE\n\n  - model: \u003Cpath\u002Fto\u002Fmodel2>\n    parameters:\n      weight: \u003Cfloat>\n\nparameters:\n  # Method-specific parameters\n\ndtype: \u003Cdtype>  # bfloat16, float16, float32\n\n# Optional\nslices:  # Layer-wise merging\ntokenizer:  # Tokenizer configuration\n```\n\n## Merge Methods Guide\n\n### Linear Merge\n\n**Best for**: Simple model combinations, equal weighting\n\n```yaml\nmerge_method: linear\nmodels:\n  - model: WizardLM\u002FWizardMath-7B-V1.1\n    parameters:\n      weight: 0.4\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    parameters:\n      weight: 0.3\n  - model: NousResearch\u002FNous-Hermes-2-Mistral-7B-DPO\n    parameters:\n      weight: 0.3\ndtype: bfloat16\n```\n\n### SLERP Merge\n\n**Best for**: Two models, smooth interpolation\n\n```yaml\nmerge_method: slerp\nslices:\n  - sources:\n      - model: mistralai\u002FMistral-7B-v0.1\n        layer_range: [0, 32]\n      - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n        layer_range: [0, 32]\nparameters:\n  t: 0.5  # 0.0 = first model, 1.0 = second model\ndtype: bfloat16\n```\n\n**Layer-specific SLERP:**\n\n```yaml\nmerge_method: slerp\nslices:\n  - sources:\n      - model: model_a\n        layer_range: [0, 32]\n      - model: model_b\n        layer_range: [0, 32]\nparameters:\n  t:\n    - filter: self_attn    # Attention layers\n      value: 0.3\n    - filter: mlp          # MLP layers\n      value: 0.7\n    - value: 0.5           # Default for other layers\ndtype: bfloat16\n```\n\n### Task Arithmetic\n\n**Best for**: Combining specialized skills\n\n```yaml\nmerge_method: task_arithmetic\nbase_model: mistralai\u002FMistral-7B-v0.1\nmodels:\n  - model: WizardLM\u002FWizardMath-7B-V1.1  # Math\n    parameters:\n      weight: 0.5\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B  # Chat\n    parameters:\n      weight: 0.3\n  - model: ajibawa-2023\u002FCode-Mistral-7B  # Code\n    parameters:\n      weight: 0.2\ndtype: bfloat16\n```\n\n### TIES-Merging\n\n**Best for**: Many models, resolving conflicts\n\n```yaml\nmerge_method: ties\nbase_model: mistralai\u002FMistral-7B-v0.1\nmodels:\n  - model: WizardLM\u002FWizardMath-7B-V1.1\n    parameters:\n      density: 0.5  # Keep top 50% of parameters\n      weight: 1.0\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    parameters:\n      density: 0.5\n      weight: 1.0\n  - model: NousResearch\u002FNous-Hermes-2-Mistral-7B-DPO\n    parameters:\n      density: 0.5\n      weight: 1.0\nparameters:\n  normalize: true\ndtype: bfloat16\n```\n\n### DARE Merge\n\n**Best for**: Reducing redundancy\n\n```yaml\nmerge_method: dare_ties\nbase_model: mistralai\u002FMistral-7B-v0.1\nmodels:\n  - model: WizardLM\u002FWizardMath-7B-V1.1\n    parameters:\n      density: 0.5    # Drop 50% of deltas\n      weight: 0.6\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    parameters:\n      density: 0.5\n      weight: 0.4\nparameters:\n  int8_mask: true  # Use int8 for masks (saves memory)\ndtype: bfloat16\n```\n\n## Advanced Patterns\n\n### Layer-wise Merging\n\n```yaml\n# Different models for different layers\nmerge_method: passthrough\nslices:\n  - sources:\n      - model: mistralai\u002FMistral-7B-v0.1\n        layer_range: [0, 16]   # First half\n  - sources:\n      - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n        layer_range: [16, 32]  # Second half\ndtype: bfloat16\n```\n\n### MoE from Merged Models\n\n```yaml\n# Create Mixture of Experts\nmerge_method: moe\nbase_model: mistralai\u002FMistral-7B-v0.1\nexperts:\n  - source_model: WizardLM\u002FWizardMath-7B-V1.1\n    positive_prompts:\n      - \"math\"\n      - \"calculate\"\n  - source_model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    positive_prompts:\n      - \"chat\"\n      - \"conversation\"\n  - source_model: ajibawa-2023\u002FCode-Mistral-7B\n    positive_prompts:\n      - \"code\"\n      - \"python\"\ndtype: bfloat16\n```\n\n### Tokenizer Merging\n\n```yaml\nmerge_method: linear\nmodels:\n  - model: mistralai\u002FMistral-7B-v0.1\n  - model: custom\u002Fspecialized-model\n\ntokenizer:\n  source: \"union\"  # Combine vocabularies from both models\n  tokens:\n    \u003C|special_token|>:\n      source: \"custom\u002Fspecialized-model\"\n```\n\n## Best Practices\n\n### 1. Model Compatibility\n\n```python\n# ✅ Good: Same architecture\nmodels = [\n    \"mistralai\u002FMistral-7B-v0.1\",\n    \"teknium\u002FOpenHermes-2.5-Mistral-7B\",  # Both Mistral 7B\n]\n\n# ❌ Bad: Different architectures\nmodels = [\n    \"meta-llama\u002FLlama-2-7b-hf\",  # Llama\n    \"mistralai\u002FMistral-7B-v0.1\",  # Mistral (incompatible!)\n]\n```\n\n### 2. Weight Selection\n\n```yaml\n# ✅ Good: Weights sum to 1.0\nmodels:\n  - model: model_a\n    parameters:\n      weight: 0.6\n  - model: model_b\n    parameters:\n      weight: 0.4  # 0.6 + 0.4 = 1.0\n\n# ⚠️  Acceptable: Weights don't sum to 1 (for task arithmetic)\nmodels:\n  - model: model_a\n    parameters:\n      weight: 0.8\n  - model: model_b\n    parameters:\n      weight: 0.8  # May boost performance\n```\n\n### 3. Method Selection\n\n```python\n# Choose merge method based on use case:\n\n# 2 models, smooth blend → SLERP\nmerge_method = \"slerp\"\n\n# 3+ models, simple average → Linear\nmerge_method = \"linear\"\n\n# Multiple task-specific models → Task Arithmetic or TIES\nmerge_method = \"ties\"\n\n# Want to reduce redundancy → DARE\nmerge_method = \"dare_ties\"\n```\n\n### 4. Density Tuning (TIES\u002FDARE)\n\n```yaml\n# Start conservative (keep more parameters)\nparameters:\n  density: 0.8  # Keep 80%\n\n# If performance good, increase sparsity\nparameters:\n  density: 0.5  # Keep 50%\n\n# If performance degrades, reduce sparsity\nparameters:\n  density: 0.9  # Keep 90%\n```\n\n### 5. Layer-specific Merging\n\n```yaml\n# Preserve base model's beginning and end\nmerge_method: passthrough\nslices:\n  - sources:\n      - model: base_model\n        layer_range: [0, 2]     # Keep first layers\n  - sources:\n      - model: merged_middle    # Merge middle layers\n        layer_range: [2, 30]\n  - sources:\n      - model: base_model\n        layer_range: [30, 32]   # Keep last layers\n```\n\n## Evaluation & Testing\n\n### Benchmark Merged Models\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load merged model\nmodel = AutoModelForCausalLM.from_pretrained(\".\u002Fmerged-model\")\ntokenizer = AutoTokenizer.from_pretrained(\".\u002Fmerged-model\")\n\n# Test on various tasks\ntest_prompts = {\n    \"math\": \"Calculate: 25 * 17 =\",\n    \"code\": \"Write a Python function to reverse a string:\",\n    \"chat\": \"What is the capital of France?\",\n}\n\nfor task, prompt in test_prompts.items():\n    inputs = tokenizer(prompt, return_tensors=\"pt\")\n    outputs = model.generate(**inputs, max_length=100)\n    print(f\"{task}: {tokenizer.decode(outputs[0])}\")\n```\n\n### Common Benchmarks\n\n- **Open LLM Leaderboard**: General capabilities\n- **MT-Bench**: Multi-turn conversation\n- **MMLU**: Multitask accuracy\n- **HumanEval**: Code generation\n- **GSM8K**: Math reasoning\n\n## Production Deployment\n\n### Save and Upload\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load merged model\nmodel = AutoModelForCausalLM.from_pretrained(\".\u002Fmerged-model\")\ntokenizer = AutoTokenizer.from_pretrained(\".\u002Fmerged-model\")\n\n# Upload to HuggingFace Hub\nmodel.push_to_hub(\"username\u002Fmy-merged-model\")\ntokenizer.push_to_hub(\"username\u002Fmy-merged-model\")\n```\n\n### Quantize Merged Model\n\n```bash\n# Quantize with GGUF\npython convert.py .\u002Fmerged-model --outtype f16 --outfile merged-model.gguf\n\n# Quantize with GPTQ\npython quantize_gptq.py .\u002Fmerged-model --bits 4 --group_size 128\n```\n\n## Common Pitfalls\n\n### ❌ Pitfall 1: Merging Incompatible Models\n\n```yaml\n# Wrong: Different architectures\nmodels:\n  - model: meta-llama\u002FLlama-2-7b  # Llama architecture\n  - model: mistralai\u002FMistral-7B   # Mistral architecture\n```\n\n**Fix**: Only merge models with same architecture\n\n### ❌ Pitfall 2: Over-weighting One Model\n\n```yaml\n# Suboptimal: One model dominates\nmodels:\n  - model: model_a\n    parameters:\n      weight: 0.95  # Too high\n  - model: model_b\n    parameters:\n      weight: 0.05  # Too low\n```\n\n**Fix**: Use more balanced weights (0.3-0.7 range)\n\n### ❌ Pitfall 3: Not Evaluating\n\n```bash\n# Wrong: Merge and deploy without testing\nmergekit-yaml config.yml .\u002Fmerged-model\n# Deploy immediately (risky!)\n```\n\n**Fix**: Always benchmark before deploying\n\n## Resources\n\n- **mergekit GitHub**: https:\u002F\u002Fgithub.com\u002Farcee-ai\u002Fmergekit\n- **HuggingFace Tutorial**: https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fmlabonne\u002Fmerge-models\n- **LazyMergekit**: Automated merging notebook\n- **TIES Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.01708\n- **DARE Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2311.03099\n\n## See Also\n\n- `references\u002Fmethods.md` - Deep dive into merge algorithms\n- `references\u002Fexamples.md` - Real-world merge configurations\n- `references\u002Fevaluation.md` - Benchmarking and testing strategies\n\n\n",{"name":5,"tags":23,"author":35,"license":14,"version":36,"description":7,"dependencies":37},[24,25,26,27,28,29,30,31,32,33,34],"Emerging Techniques","Model Merging","Mergekit","SLERP","TIES","DARE","Task Arithmetic","Model Fusion","No Retraining","Multi-Capability","Arcee AI","Orchestra Research","1.0.0",[38,39,40],"mergekit","transformers","torch","main","d046d088ce5a79a083f0abc0fe1b7ded73bc5b1f","claude-config\u002Fskills\u002Fmodel-merging","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent","https:\u002F\u002Fgithub.com\u002FSansar35","2026-05-10 16:04:53",{"source":8,"skillName":5,"commands":48,"prompt":58,"zipAvailable":59},[49,52,55],{"tool":50,"command":51},"npx","npx skills add Sansar35\u002Fclaude-agent --skill model-merging",{"tool":53,"command":54},"bunx","bunx skills add Sansar35\u002Fclaude-agent --skill model-merging",{"tool":56,"command":57},"pnpm","pnpm dlx skills add Sansar35\u002Fclaude-agent --skill model-merging","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fmodel-merging\u002FSKILL.md",true,[61,67,71,75],{"path":62,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":65,"sourceHtmlUrl":66},".skill-source.json","REFERENCE","参考文档",632,"https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fmodel-merging\u002F.skill-source.json",{"path":68,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":69,"sourceHtmlUrl":70},"references\u002Fevaluation.md",11705,"https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fmodel-merging\u002Freferences\u002Fevaluation.md",{"path":72,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":73,"sourceHtmlUrl":74},"references\u002Fexamples.md",8813,"https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fmodel-merging\u002Freferences\u002Fexamples.md",{"path":76,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":77,"sourceHtmlUrl":78},"references\u002Fmethods.md",9562,"https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fmodel-merging\u002Freferences\u002Fmethods.md",[80,87,94,101,108,116],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":18},"01M72DVAN1JYEA302MT0C5MGD6","zustand-store-ts","Create Zustand stores following established patterns with proper TypeScript types and middleware.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzustand-store-ts\u002FSKILL.md",[],[],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":18},"01M72DV8J5NBRRM4RPDFRV95N7","zod-validation-expert","Expert in Zod — TypeScript-first schema validation. Covers parsing, custom errors, refinements, type inference, and integration with React Hook Form, Next.js, and tRPC.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzod-validation-expert\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":18},"01M72DV5EMHW0BHP9BZ167MDCB","zinc-database","Access ZINC (230M+ purchasable compounds). Search by ZINC ID\u002FSMILES, similarity searches, 3D-ready structures for docking, analog discovery, for virtual screening and drug discovery.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzinc-database\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":18},"01M72DV216QY0E693ZPEVQ0DBS","zarr-python","Chunked N-D arrays for cloud storage. Compressed arrays, parallel I\u002FO, S3\u002FGCS integration, NumPy\u002FDask\u002FXarray compatible, for large-scale scientific computing pipelines.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzarr-python\u002FSKILL.md",[],[],{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":112,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":18},"01M72DTXQREMCRKA8NHCVKHN4W","zapier-workflows","Manage and trigger pre-built Zapier workflows and MCP tool orchestration. Use when user mentions workflows, Zaps, automations, daily digest, research, search, lead tracking, expenses, or asks to \"run\" any process. Also handles Perplexity-based research and Google Sheets data tracking.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzapier-workflows\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzapier-workflows\u002FSKILL.md",[],[],{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":18},"01M72DTT17TBSJ0Y8J2XZ4C9TZ","zapier-make-patterns","No-code automation democratizes workflow building. Zapier and Make (formerly Integromat) let non-developers automate business processes without writing code. But no-code doesn't mean no-complexity - these platforms have their own patterns, pitfalls, and breaking points.  This skill covers when to use which platform, how to build reliable automations, and when to graduate to code-based solutions. Key insight: Zapier optimizes for simplicity and integrations (7000+ apps), Make optimizes for power","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzapier-make-patterns\u002FSKILL.md",[],[]]