[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7CKCQPDZFYSNB40EXWRCM2X":3,"skill-same:01M7CKCQPDZFYSNB40EXWRCM2X":78},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":42,"commitSha":43,"skillDirectory":44,"forksCount":45,"repositoryHtmlUrl":46,"repositoryHomepageUrl":6,"ownerHtmlUrl":47,"ownerBlogUrl":6,"ownerDescription":48,"updatedAt":49,"install":50,"resources":63},"01M7CKCQPDZFYSNB40EXWRCM2X","model-merging",null,"Merge multiple fine-tuned models using mergekit to combine capabilities without retraining. Use when creating specialized models by blending domain-specific expertise (math + coding + chat), improving performance beyond single models, or experimenting rapidly with model variants. Covers SLERP, TIES-Merging, DARE, Task Arithmetic, linear merging, and production deployment strategies.","Carchaloneo22\u002FClaude-Code-template","Carchaloneo22","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F159488757?v=4","01M7CK5MQXAFZE3W7SAP69KNYR","Carlos Chaparro","01M7CK5MRFE26CDPEQDN5RSQPK",2,"MIT","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-model-merging\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: model-merging\ndescription: Merge multiple fine-tuned models using mergekit to combine capabilities without retraining. Use when creating specialized models by blending domain-specific expertise (math + coding + chat), improving performance beyond single models, or experimenting rapidly with model variants. Covers SLERP, TIES-Merging, DARE, Task Arithmetic, linear merging, and production deployment strategies.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Emerging Techniques, Model Merging, Mergekit, SLERP, TIES, DARE, Task Arithmetic, Model Fusion, No Retraining, Multi-Capability, Arcee AI]\ndependencies: [mergekit, transformers, torch]\n---\n\n# Model Merging: Combining Pre-trained Models\n\n## When to Use This Skill\n\nUse Model Merging when you need to:\n- **Combine capabilities** from multiple fine-tuned models without retraining\n- **Create specialized models** by blending domain-specific expertise (math + coding + chat)\n- **Improve performance** beyond single models (often +5-10% on benchmarks)\n- **Reduce training costs** - no GPUs needed, merges run on CPU\n- **Experiment rapidly** - create new model variants in minutes, not days\n- **Preserve multiple skills** - merge without catastrophic forgetting\n\n**Success Stories**: Marcoro14-7B-slerp (best on Open LLM Leaderboard 02\u002F2024), many top HuggingFace models use merging\n\n**Tools**: mergekit (Arcee AI), LazyMergekit, Model Soup\n\n## Installation\n\n```bash\n# Install mergekit\ngit clone https:\u002F\u002Fgithub.com\u002Farcee-ai\u002Fmergekit.git\ncd mergekit\npip install -e .\n\n# Or via pip\npip install mergekit\n\n# Optional: Transformer library\npip install transformers torch\n```\n\n## Quick Start\n\n### Simple Linear Merge\n\n```yaml\n# config.yml - Merge two models with equal weights\nmerge_method: linear\nmodels:\n  - model: mistralai\u002FMistral-7B-v0.1\n    parameters:\n      weight: 0.5\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    parameters:\n      weight: 0.5\ndtype: bfloat16\n```\n\n```bash\n# Run merge\nmergekit-yaml config.yml .\u002Fmerged-model --cuda\n\n# Use merged model\npython -m transformers.models.auto --model_name_or_path .\u002Fmerged-model\n```\n\n### SLERP Merge (Best for 2 Models)\n\n```yaml\n# config.yml - Spherical interpolation\nmerge_method: slerp\nslices:\n  - sources:\n      - model: mistralai\u002FMistral-7B-v0.1\n        layer_range: [0, 32]\n      - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n        layer_range: [0, 32]\nparameters:\n  t: 0.5  # Interpolation factor (0=model1, 1=model2)\ndtype: bfloat16\n```\n\n## Core Concepts\n\n### 1. Merge Methods\n\n**Linear (Model Soup)**\n- Simple weighted average of parameters\n- Fast, works well for similar models\n- Can merge 2+ models\n\n```python\nmerged_weights = w1 * model1_weights + w2 * model2_weights + w3 * model3_weights\n# where w1 + w2 + w3 = 1\n```\n\n**SLERP (Spherical Linear Interpolation)**\n- Interpolates along sphere in weight space\n- Preserves magnitude of weight vectors\n- Best for merging 2 models\n- Smoother than linear\n\n```python\n# SLERP formula\nmerged = (sin((1-t)*θ) \u002F sin(θ)) * model1 + (sin(t*θ) \u002F sin(θ)) * model2\n# where θ = arccos(dot(model1, model2))\n# t ∈ [0, 1]\n```\n\n**Task Arithmetic**\n- Extract \"task vectors\" (fine-tuned - base)\n- Combine task vectors, add to base\n- Good for merging multiple specialized models\n\n```python\n# Task vector\ntask_vector = finetuned_model - base_model\n\n# Merge multiple task vectors\nmerged = base_model + α₁*task_vector₁ + α₂*task_vector₂\n```\n\n**TIES-Merging**\n- Task arithmetic + sparsification\n- Resolves sign conflicts in parameters\n- Best for merging many task-specific models\n\n**DARE (Drop And REscale)**\n- Randomly drops fine-tuned parameters\n- Rescales remaining parameters\n- Reduces redundancy, maintains performance\n\n### 2. Configuration Structure\n\n```yaml\n# Basic structure\nmerge_method: \u003Cmethod>  # linear, slerp, ties, dare_ties, task_arithmetic\nbase_model: \u003Cpath>      # Optional: base model for task arithmetic\n\nmodels:\n  - model: \u003Cpath\u002Fto\u002Fmodel1>\n    parameters:\n      weight: \u003Cfloat>   # Merge weight\n      density: \u003Cfloat>  # For TIES\u002FDARE\n\n  - model: \u003Cpath\u002Fto\u002Fmodel2>\n    parameters:\n      weight: \u003Cfloat>\n\nparameters:\n  # Method-specific parameters\n\ndtype: \u003Cdtype>  # bfloat16, float16, float32\n\n# Optional\nslices:  # Layer-wise merging\ntokenizer:  # Tokenizer configuration\n```\n\n## Merge Methods Guide\n\n### Linear Merge\n\n**Best for**: Simple model combinations, equal weighting\n\n```yaml\nmerge_method: linear\nmodels:\n  - model: WizardLM\u002FWizardMath-7B-V1.1\n    parameters:\n      weight: 0.4\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    parameters:\n      weight: 0.3\n  - model: NousResearch\u002FNous-Hermes-2-Mistral-7B-DPO\n    parameters:\n      weight: 0.3\ndtype: bfloat16\n```\n\n### SLERP Merge\n\n**Best for**: Two models, smooth interpolation\n\n```yaml\nmerge_method: slerp\nslices:\n  - sources:\n      - model: mistralai\u002FMistral-7B-v0.1\n        layer_range: [0, 32]\n      - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n        layer_range: [0, 32]\nparameters:\n  t: 0.5  # 0.0 = first model, 1.0 = second model\ndtype: bfloat16\n```\n\n**Layer-specific SLERP:**\n\n```yaml\nmerge_method: slerp\nslices:\n  - sources:\n      - model: model_a\n        layer_range: [0, 32]\n      - model: model_b\n        layer_range: [0, 32]\nparameters:\n  t:\n    - filter: self_attn    # Attention layers\n      value: 0.3\n    - filter: mlp          # MLP layers\n      value: 0.7\n    - value: 0.5           # Default for other layers\ndtype: bfloat16\n```\n\n### Task Arithmetic\n\n**Best for**: Combining specialized skills\n\n```yaml\nmerge_method: task_arithmetic\nbase_model: mistralai\u002FMistral-7B-v0.1\nmodels:\n  - model: WizardLM\u002FWizardMath-7B-V1.1  # Math\n    parameters:\n      weight: 0.5\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B  # Chat\n    parameters:\n      weight: 0.3\n  - model: ajibawa-2023\u002FCode-Mistral-7B  # Code\n    parameters:\n      weight: 0.2\ndtype: bfloat16\n```\n\n### TIES-Merging\n\n**Best for**: Many models, resolving conflicts\n\n```yaml\nmerge_method: ties\nbase_model: mistralai\u002FMistral-7B-v0.1\nmodels:\n  - model: WizardLM\u002FWizardMath-7B-V1.1\n    parameters:\n      density: 0.5  # Keep top 50% of parameters\n      weight: 1.0\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    parameters:\n      density: 0.5\n      weight: 1.0\n  - model: NousResearch\u002FNous-Hermes-2-Mistral-7B-DPO\n    parameters:\n      density: 0.5\n      weight: 1.0\nparameters:\n  normalize: true\ndtype: bfloat16\n```\n\n### DARE Merge\n\n**Best for**: Reducing redundancy\n\n```yaml\nmerge_method: dare_ties\nbase_model: mistralai\u002FMistral-7B-v0.1\nmodels:\n  - model: WizardLM\u002FWizardMath-7B-V1.1\n    parameters:\n      density: 0.5    # Drop 50% of deltas\n      weight: 0.6\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    parameters:\n      density: 0.5\n      weight: 0.4\nparameters:\n  int8_mask: true  # Use int8 for masks (saves memory)\ndtype: bfloat16\n```\n\n## Advanced Patterns\n\n### Layer-wise Merging\n\n```yaml\n# Different models for different layers\nmerge_method: passthrough\nslices:\n  - sources:\n      - model: mistralai\u002FMistral-7B-v0.1\n        layer_range: [0, 16]   # First half\n  - sources:\n      - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n        layer_range: [16, 32]  # Second half\ndtype: bfloat16\n```\n\n### MoE from Merged Models\n\n```yaml\n# Create Mixture of Experts\nmerge_method: moe\nbase_model: mistralai\u002FMistral-7B-v0.1\nexperts:\n  - source_model: WizardLM\u002FWizardMath-7B-V1.1\n    positive_prompts:\n      - \"math\"\n      - \"calculate\"\n  - source_model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    positive_prompts:\n      - \"chat\"\n      - \"conversation\"\n  - source_model: ajibawa-2023\u002FCode-Mistral-7B\n    positive_prompts:\n      - \"code\"\n      - \"python\"\ndtype: bfloat16\n```\n\n### Tokenizer Merging\n\n```yaml\nmerge_method: linear\nmodels:\n  - model: mistralai\u002FMistral-7B-v0.1\n  - model: custom\u002Fspecialized-model\n\ntokenizer:\n  source: \"union\"  # Combine vocabularies from both models\n  tokens:\n    \u003C|special_token|>:\n      source: \"custom\u002Fspecialized-model\"\n```\n\n## Best Practices\n\n### 1. Model Compatibility\n\n```python\n# ✅ Good: Same architecture\nmodels = [\n    \"mistralai\u002FMistral-7B-v0.1\",\n    \"teknium\u002FOpenHermes-2.5-Mistral-7B\",  # Both Mistral 7B\n]\n\n# ❌ Bad: Different architectures\nmodels = [\n    \"meta-llama\u002FLlama-2-7b-hf\",  # Llama\n    \"mistralai\u002FMistral-7B-v0.1\",  # Mistral (incompatible!)\n]\n```\n\n### 2. Weight Selection\n\n```yaml\n# ✅ Good: Weights sum to 1.0\nmodels:\n  - model: model_a\n    parameters:\n      weight: 0.6\n  - model: model_b\n    parameters:\n      weight: 0.4  # 0.6 + 0.4 = 1.0\n\n# ⚠️  Acceptable: Weights don't sum to 1 (for task arithmetic)\nmodels:\n  - model: model_a\n    parameters:\n      weight: 0.8\n  - model: model_b\n    parameters:\n      weight: 0.8  # May boost performance\n```\n\n### 3. Method Selection\n\n```python\n# Choose merge method based on use case:\n\n# 2 models, smooth blend → SLERP\nmerge_method = \"slerp\"\n\n# 3+ models, simple average → Linear\nmerge_method = \"linear\"\n\n# Multiple task-specific models → Task Arithmetic or TIES\nmerge_method = \"ties\"\n\n# Want to reduce redundancy → DARE\nmerge_method = \"dare_ties\"\n```\n\n### 4. Density Tuning (TIES\u002FDARE)\n\n```yaml\n# Start conservative (keep more parameters)\nparameters:\n  density: 0.8  # Keep 80%\n\n# If performance good, increase sparsity\nparameters:\n  density: 0.5  # Keep 50%\n\n# If performance degrades, reduce sparsity\nparameters:\n  density: 0.9  # Keep 90%\n```\n\n### 5. Layer-specific Merging\n\n```yaml\n# Preserve base model's beginning and end\nmerge_method: passthrough\nslices:\n  - sources:\n      - model: base_model\n        layer_range: [0, 2]     # Keep first layers\n  - sources:\n      - model: merged_middle    # Merge middle layers\n        layer_range: [2, 30]\n  - sources:\n      - model: base_model\n        layer_range: [30, 32]   # Keep last layers\n```\n\n## Evaluation & Testing\n\n### Benchmark Merged Models\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load merged model\nmodel = AutoModelForCausalLM.from_pretrained(\".\u002Fmerged-model\")\ntokenizer = AutoTokenizer.from_pretrained(\".\u002Fmerged-model\")\n\n# Test on various tasks\ntest_prompts = {\n    \"math\": \"Calculate: 25 * 17 =\",\n    \"code\": \"Write a Python function to reverse a string:\",\n    \"chat\": \"What is the capital of France?\",\n}\n\nfor task, prompt in test_prompts.items():\n    inputs = tokenizer(prompt, return_tensors=\"pt\")\n    outputs = model.generate(**inputs, max_length=100)\n    print(f\"{task}: {tokenizer.decode(outputs[0])}\")\n```\n\n### Common Benchmarks\n\n- **Open LLM Leaderboard**: General capabilities\n- **MT-Bench**: Multi-turn conversation\n- **MMLU**: Multitask accuracy\n- **HumanEval**: Code generation\n- **GSM8K**: Math reasoning\n\n## Production Deployment\n\n### Save and Upload\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load merged model\nmodel = AutoModelForCausalLM.from_pretrained(\".\u002Fmerged-model\")\ntokenizer = AutoTokenizer.from_pretrained(\".\u002Fmerged-model\")\n\n# Upload to HuggingFace Hub\nmodel.push_to_hub(\"username\u002Fmy-merged-model\")\ntokenizer.push_to_hub(\"username\u002Fmy-merged-model\")\n```\n\n### Quantize Merged Model\n\n```bash\n# Quantize with GGUF\npython convert.py .\u002Fmerged-model --outtype f16 --outfile merged-model.gguf\n\n# Quantize with GPTQ\npython quantize_gptq.py .\u002Fmerged-model --bits 4 --group_size 128\n```\n\n## Common Pitfalls\n\n### ❌ Pitfall 1: Merging Incompatible Models\n\n```yaml\n# Wrong: Different architectures\nmodels:\n  - model: meta-llama\u002FLlama-2-7b  # Llama architecture\n  - model: mistralai\u002FMistral-7B   # Mistral architecture\n```\n\n**Fix**: Only merge models with same architecture\n\n### ❌ Pitfall 2: Over-weighting One Model\n\n```yaml\n# Suboptimal: One model dominates\nmodels:\n  - model: model_a\n    parameters:\n      weight: 0.95  # Too high\n  - model: model_b\n    parameters:\n      weight: 0.05  # Too low\n```\n\n**Fix**: Use more balanced weights (0.3-0.7 range)\n\n### ❌ Pitfall 3: Not Evaluating\n\n```bash\n# Wrong: Merge and deploy without testing\nmergekit-yaml config.yml .\u002Fmerged-model\n# Deploy immediately (risky!)\n```\n\n**Fix**: Always benchmark before deploying\n\n## Resources\n\n- **mergekit GitHub**: https:\u002F\u002Fgithub.com\u002Farcee-ai\u002Fmergekit\n- **HuggingFace Tutorial**: https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fmlabonne\u002Fmerge-models\n- **LazyMergekit**: Automated merging notebook\n- **TIES Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.01708\n- **DARE Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2311.03099\n\n## See Also\n\n- `references\u002Fmethods.md` - Deep dive into merge algorithms\n- `references\u002Fexamples.md` - Real-world merge configurations\n- `references\u002Fevaluation.md` - Benchmarking and testing strategies\n\n\n","# Model Merging: Combining Pre-trained Models\n\n## When to Use This Skill\n\nUse Model Merging when you need to:\n- **Combine capabilities** from multiple fine-tuned models without retraining\n- **Create specialized models** by blending domain-specific expertise (math + coding + chat)\n- **Improve performance** beyond single models (often +5-10% on benchmarks)\n- **Reduce training costs** - no GPUs needed, merges run on CPU\n- **Experiment rapidly** - create new model variants in minutes, not days\n- **Preserve multiple skills** - merge without catastrophic forgetting\n\n**Success Stories**: Marcoro14-7B-slerp (best on Open LLM Leaderboard 02\u002F2024), many top HuggingFace models use merging\n\n**Tools**: mergekit (Arcee AI), LazyMergekit, Model Soup\n\n## Installation\n\n```bash\n# Install mergekit\ngit clone https:\u002F\u002Fgithub.com\u002Farcee-ai\u002Fmergekit.git\ncd mergekit\npip install -e .\n\n# Or via pip\npip install mergekit\n\n# Optional: Transformer library\npip install transformers torch\n```\n\n## Quick Start\n\n### Simple Linear Merge\n\n```yaml\n# config.yml - Merge two models with equal weights\nmerge_method: linear\nmodels:\n  - model: mistralai\u002FMistral-7B-v0.1\n    parameters:\n      weight: 0.5\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    parameters:\n      weight: 0.5\ndtype: bfloat16\n```\n\n```bash\n# Run merge\nmergekit-yaml config.yml .\u002Fmerged-model --cuda\n\n# Use merged model\npython -m transformers.models.auto --model_name_or_path .\u002Fmerged-model\n```\n\n### SLERP Merge (Best for 2 Models)\n\n```yaml\n# config.yml - Spherical interpolation\nmerge_method: slerp\nslices:\n  - sources:\n      - model: mistralai\u002FMistral-7B-v0.1\n        layer_range: [0, 32]\n      - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n        layer_range: [0, 32]\nparameters:\n  t: 0.5  # Interpolation factor (0=model1, 1=model2)\ndtype: bfloat16\n```\n\n## Core Concepts\n\n### 1. Merge Methods\n\n**Linear (Model Soup)**\n- Simple weighted average of parameters\n- Fast, works well for similar models\n- Can merge 2+ models\n\n```python\nmerged_weights = w1 * model1_weights + w2 * model2_weights + w3 * model3_weights\n# where w1 + w2 + w3 = 1\n```\n\n**SLERP (Spherical Linear Interpolation)**\n- Interpolates along sphere in weight space\n- Preserves magnitude of weight vectors\n- Best for merging 2 models\n- Smoother than linear\n\n```python\n# SLERP formula\nmerged = (sin((1-t)*θ) \u002F sin(θ)) * model1 + (sin(t*θ) \u002F sin(θ)) * model2\n# where θ = arccos(dot(model1, model2))\n# t ∈ [0, 1]\n```\n\n**Task Arithmetic**\n- Extract \"task vectors\" (fine-tuned - base)\n- Combine task vectors, add to base\n- Good for merging multiple specialized models\n\n```python\n# Task vector\ntask_vector = finetuned_model - base_model\n\n# Merge multiple task vectors\nmerged = base_model + α₁*task_vector₁ + α₂*task_vector₂\n```\n\n**TIES-Merging**\n- Task arithmetic + sparsification\n- Resolves sign conflicts in parameters\n- Best for merging many task-specific models\n\n**DARE (Drop And REscale)**\n- Randomly drops fine-tuned parameters\n- Rescales remaining parameters\n- Reduces redundancy, maintains performance\n\n### 2. Configuration Structure\n\n```yaml\n# Basic structure\nmerge_method: \u003Cmethod>  # linear, slerp, ties, dare_ties, task_arithmetic\nbase_model: \u003Cpath>      # Optional: base model for task arithmetic\n\nmodels:\n  - model: \u003Cpath\u002Fto\u002Fmodel1>\n    parameters:\n      weight: \u003Cfloat>   # Merge weight\n      density: \u003Cfloat>  # For TIES\u002FDARE\n\n  - model: \u003Cpath\u002Fto\u002Fmodel2>\n    parameters:\n      weight: \u003Cfloat>\n\nparameters:\n  # Method-specific parameters\n\ndtype: \u003Cdtype>  # bfloat16, float16, float32\n\n# Optional\nslices:  # Layer-wise merging\ntokenizer:  # Tokenizer configuration\n```\n\n## Merge Methods Guide\n\n### Linear Merge\n\n**Best for**: Simple model combinations, equal weighting\n\n```yaml\nmerge_method: linear\nmodels:\n  - model: WizardLM\u002FWizardMath-7B-V1.1\n    parameters:\n      weight: 0.4\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    parameters:\n      weight: 0.3\n  - model: NousResearch\u002FNous-Hermes-2-Mistral-7B-DPO\n    parameters:\n      weight: 0.3\ndtype: bfloat16\n```\n\n### SLERP Merge\n\n**Best for**: Two models, smooth interpolation\n\n```yaml\nmerge_method: slerp\nslices:\n  - sources:\n      - model: mistralai\u002FMistral-7B-v0.1\n        layer_range: [0, 32]\n      - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n        layer_range: [0, 32]\nparameters:\n  t: 0.5  # 0.0 = first model, 1.0 = second model\ndtype: bfloat16\n```\n\n**Layer-specific SLERP:**\n\n```yaml\nmerge_method: slerp\nslices:\n  - sources:\n      - model: model_a\n        layer_range: [0, 32]\n      - model: model_b\n        layer_range: [0, 32]\nparameters:\n  t:\n    - filter: self_attn    # Attention layers\n      value: 0.3\n    - filter: mlp          # MLP layers\n      value: 0.7\n    - value: 0.5           # Default for other layers\ndtype: bfloat16\n```\n\n### Task Arithmetic\n\n**Best for**: Combining specialized skills\n\n```yaml\nmerge_method: task_arithmetic\nbase_model: mistralai\u002FMistral-7B-v0.1\nmodels:\n  - model: WizardLM\u002FWizardMath-7B-V1.1  # Math\n    parameters:\n      weight: 0.5\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B  # Chat\n    parameters:\n      weight: 0.3\n  - model: ajibawa-2023\u002FCode-Mistral-7B  # Code\n    parameters:\n      weight: 0.2\ndtype: bfloat16\n```\n\n### TIES-Merging\n\n**Best for**: Many models, resolving conflicts\n\n```yaml\nmerge_method: ties\nbase_model: mistralai\u002FMistral-7B-v0.1\nmodels:\n  - model: WizardLM\u002FWizardMath-7B-V1.1\n    parameters:\n      density: 0.5  # Keep top 50% of parameters\n      weight: 1.0\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    parameters:\n      density: 0.5\n      weight: 1.0\n  - model: NousResearch\u002FNous-Hermes-2-Mistral-7B-DPO\n    parameters:\n      density: 0.5\n      weight: 1.0\nparameters:\n  normalize: true\ndtype: bfloat16\n```\n\n### DARE Merge\n\n**Best for**: Reducing redundancy\n\n```yaml\nmerge_method: dare_ties\nbase_model: mistralai\u002FMistral-7B-v0.1\nmodels:\n  - model: WizardLM\u002FWizardMath-7B-V1.1\n    parameters:\n      density: 0.5    # Drop 50% of deltas\n      weight: 0.6\n  - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    parameters:\n      density: 0.5\n      weight: 0.4\nparameters:\n  int8_mask: true  # Use int8 for masks (saves memory)\ndtype: bfloat16\n```\n\n## Advanced Patterns\n\n### Layer-wise Merging\n\n```yaml\n# Different models for different layers\nmerge_method: passthrough\nslices:\n  - sources:\n      - model: mistralai\u002FMistral-7B-v0.1\n        layer_range: [0, 16]   # First half\n  - sources:\n      - model: teknium\u002FOpenHermes-2.5-Mistral-7B\n        layer_range: [16, 32]  # Second half\ndtype: bfloat16\n```\n\n### MoE from Merged Models\n\n```yaml\n# Create Mixture of Experts\nmerge_method: moe\nbase_model: mistralai\u002FMistral-7B-v0.1\nexperts:\n  - source_model: WizardLM\u002FWizardMath-7B-V1.1\n    positive_prompts:\n      - \"math\"\n      - \"calculate\"\n  - source_model: teknium\u002FOpenHermes-2.5-Mistral-7B\n    positive_prompts:\n      - \"chat\"\n      - \"conversation\"\n  - source_model: ajibawa-2023\u002FCode-Mistral-7B\n    positive_prompts:\n      - \"code\"\n      - \"python\"\ndtype: bfloat16\n```\n\n### Tokenizer Merging\n\n```yaml\nmerge_method: linear\nmodels:\n  - model: mistralai\u002FMistral-7B-v0.1\n  - model: custom\u002Fspecialized-model\n\ntokenizer:\n  source: \"union\"  # Combine vocabularies from both models\n  tokens:\n    \u003C|special_token|>:\n      source: \"custom\u002Fspecialized-model\"\n```\n\n## Best Practices\n\n### 1. Model Compatibility\n\n```python\n# ✅ Good: Same architecture\nmodels = [\n    \"mistralai\u002FMistral-7B-v0.1\",\n    \"teknium\u002FOpenHermes-2.5-Mistral-7B\",  # Both Mistral 7B\n]\n\n# ❌ Bad: Different architectures\nmodels = [\n    \"meta-llama\u002FLlama-2-7b-hf\",  # Llama\n    \"mistralai\u002FMistral-7B-v0.1\",  # Mistral (incompatible!)\n]\n```\n\n### 2. Weight Selection\n\n```yaml\n# ✅ Good: Weights sum to 1.0\nmodels:\n  - model: model_a\n    parameters:\n      weight: 0.6\n  - model: model_b\n    parameters:\n      weight: 0.4  # 0.6 + 0.4 = 1.0\n\n# ⚠️  Acceptable: Weights don't sum to 1 (for task arithmetic)\nmodels:\n  - model: model_a\n    parameters:\n      weight: 0.8\n  - model: model_b\n    parameters:\n      weight: 0.8  # May boost performance\n```\n\n### 3. Method Selection\n\n```python\n# Choose merge method based on use case:\n\n# 2 models, smooth blend → SLERP\nmerge_method = \"slerp\"\n\n# 3+ models, simple average → Linear\nmerge_method = \"linear\"\n\n# Multiple task-specific models → Task Arithmetic or TIES\nmerge_method = \"ties\"\n\n# Want to reduce redundancy → DARE\nmerge_method = \"dare_ties\"\n```\n\n### 4. Density Tuning (TIES\u002FDARE)\n\n```yaml\n# Start conservative (keep more parameters)\nparameters:\n  density: 0.8  # Keep 80%\n\n# If performance good, increase sparsity\nparameters:\n  density: 0.5  # Keep 50%\n\n# If performance degrades, reduce sparsity\nparameters:\n  density: 0.9  # Keep 90%\n```\n\n### 5. Layer-specific Merging\n\n```yaml\n# Preserve base model's beginning and end\nmerge_method: passthrough\nslices:\n  - sources:\n      - model: base_model\n        layer_range: [0, 2]     # Keep first layers\n  - sources:\n      - model: merged_middle    # Merge middle layers\n        layer_range: [2, 30]\n  - sources:\n      - model: base_model\n        layer_range: [30, 32]   # Keep last layers\n```\n\n## Evaluation & Testing\n\n### Benchmark Merged Models\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load merged model\nmodel = AutoModelForCausalLM.from_pretrained(\".\u002Fmerged-model\")\ntokenizer = AutoTokenizer.from_pretrained(\".\u002Fmerged-model\")\n\n# Test on various tasks\ntest_prompts = {\n    \"math\": \"Calculate: 25 * 17 =\",\n    \"code\": \"Write a Python function to reverse a string:\",\n    \"chat\": \"What is the capital of France?\",\n}\n\nfor task, prompt in test_prompts.items():\n    inputs = tokenizer(prompt, return_tensors=\"pt\")\n    outputs = model.generate(**inputs, max_length=100)\n    print(f\"{task}: {tokenizer.decode(outputs[0])}\")\n```\n\n### Common Benchmarks\n\n- **Open LLM Leaderboard**: General capabilities\n- **MT-Bench**: Multi-turn conversation\n- **MMLU**: Multitask accuracy\n- **HumanEval**: Code generation\n- **GSM8K**: Math reasoning\n\n## Production Deployment\n\n### Save and Upload\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load merged model\nmodel = AutoModelForCausalLM.from_pretrained(\".\u002Fmerged-model\")\ntokenizer = AutoTokenizer.from_pretrained(\".\u002Fmerged-model\")\n\n# Upload to HuggingFace Hub\nmodel.push_to_hub(\"username\u002Fmy-merged-model\")\ntokenizer.push_to_hub(\"username\u002Fmy-merged-model\")\n```\n\n### Quantize Merged Model\n\n```bash\n# Quantize with GGUF\npython convert.py .\u002Fmerged-model --outtype f16 --outfile merged-model.gguf\n\n# Quantize with GPTQ\npython quantize_gptq.py .\u002Fmerged-model --bits 4 --group_size 128\n```\n\n## Common Pitfalls\n\n### ❌ Pitfall 1: Merging Incompatible Models\n\n```yaml\n# Wrong: Different architectures\nmodels:\n  - model: meta-llama\u002FLlama-2-7b  # Llama architecture\n  - model: mistralai\u002FMistral-7B   # Mistral architecture\n```\n\n**Fix**: Only merge models with same architecture\n\n### ❌ Pitfall 2: Over-weighting One Model\n\n```yaml\n# Suboptimal: One model dominates\nmodels:\n  - model: model_a\n    parameters:\n      weight: 0.95  # Too high\n  - model: model_b\n    parameters:\n      weight: 0.05  # Too low\n```\n\n**Fix**: Use more balanced weights (0.3-0.7 range)\n\n### ❌ Pitfall 3: Not Evaluating\n\n```bash\n# Wrong: Merge and deploy without testing\nmergekit-yaml config.yml .\u002Fmerged-model\n# Deploy immediately (risky!)\n```\n\n**Fix**: Always benchmark before deploying\n\n## Resources\n\n- **mergekit GitHub**: https:\u002F\u002Fgithub.com\u002Farcee-ai\u002Fmergekit\n- **HuggingFace Tutorial**: https:\u002F\u002Fhuggingface.co\u002Fblog\u002Fmlabonne\u002Fmerge-models\n- **LazyMergekit**: Automated merging notebook\n- **TIES Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2306.01708\n- **DARE Paper**: https:\u002F\u002Farxiv.org\u002Fabs\u002F2311.03099\n\n## See Also\n\n- `references\u002Fmethods.md` - Deep dive into merge algorithms\n- `references\u002Fexamples.md` - Real-world merge configurations\n- `references\u002Fevaluation.md` - Benchmarking and testing strategies\n\n\n",{"name":5,"tags":24,"author":36,"license":15,"version":37,"description":7,"dependencies":38},[25,26,27,28,29,30,31,32,33,34,35],"Emerging Techniques","Model Merging","Mergekit","SLERP","TIES","DARE","Task Arithmetic","Model Fusion","No Retraining","Multi-Capability","Arcee AI","Orchestra Research","1.0.0",[39,40,41],"mergekit","transformers","torch","add-x-twitter-scraper-skill","7bc347b2f7dc2dde2761a965585925e0d0171748","cli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-model-merging",0,"https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template","https:\u002F\u002Fgithub.com\u002FCarchaloneo22","Ingeniero Informático | Ingeniero en Desarrollo de Software | Consultor TIC en Gestión Publica | Docente Análisis y Desarrollo de Software. ","2026-09-12 11:10:56",{"source":8,"skillName":5,"commands":51,"prompt":61,"zipAvailable":62},[52,55,58],{"tool":53,"command":54},"npx","npx skills add Carchaloneo22\u002FClaude-Code-template --skill model-merging",{"tool":56,"command":57},"bunx","bunx skills add Carchaloneo22\u002FClaude-Code-template --skill model-merging",{"tool":59,"command":60},"pnpm","pnpm dlx skills add Carchaloneo22\u002FClaude-Code-template --skill model-merging","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-model-merging\u002FSKILL.md",true,[64,70,74],{"path":65,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":68,"sourceHtmlUrl":69},"references\u002Fevaluation.md","REFERENCE","参考文档",11705,"https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-model-merging\u002Freferences\u002Fevaluation.md",{"path":71,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":72,"sourceHtmlUrl":73},"references\u002Fexamples.md",8813,"https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-model-merging\u002Freferences\u002Fexamples.md",{"path":75,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":76,"sourceHtmlUrl":77},"references\u002Fmethods.md",9562,"https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Femerging-techniques-model-merging\u002Freferences\u002Fmethods.md",[79,86,95,102,109,116],{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":19},"01M7CPQFVGKX7H3Q5Y9CN0XAZN","zapier-make-patterns","No-code automation democratizes workflow building. Zapier and Make (formerly Integromat) let non-developers automate business processes without writing code. But no-code doesn't mean no-complexity - these platforms have their own patterns, pitfalls, and breaking points.  This skill covers when to use which platform, how to build reliable automations, and when to graduate to code-based solutions. Key insight: Zapier optimizes for simplicity and integrations (7000+ apps), Make optimizes for power","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fzapier-make-patterns\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":90,"licenseFileUrl":91,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M7CPQDR2HMKAYHAAY99EA4B2","yeet","Use only when the user explicitly asks to stage, commit, push, and open a GitHub pull request in one flow using the GitHub CLI (`gh`).","Apache-2.0","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fyeet\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fyeet\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":19},"01M7CPQ7K5MAQSXEB027HZFM2G","workflow-automation","Workflow automation is the infrastructure that makes AI agents reliable. Without durable execution, a network hiccup during a 10-step payment flow means lost money and angry customers. With it, workflows resume exactly where they left off.  This skill covers the platforms (n8n, Temporal, Inngest) and patterns (sequential, parallel, orchestrator-worker) that turn brittle scripts into production-grade automation.  Key insight: The platforms make different tradeoffs. n8n optimizes for accessibility","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fworkflow-automation\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":19},"01M7CPQ34F18JHMEM8NHQS1N7K","trigger-dev","Trigger.dev expert for background jobs, AI workflows, and reliable async execution with excellent developer experience and TypeScript-first design. Use when: trigger.dev, trigger dev, background task, ai background job, long running task.","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Ftrigger-dev\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":19},"01M7CPPZSXXDB1SAWT5N63ACZT","planning-with-files","Transforms workflow to use Manus-style persistent markdown files for planning, progress tracking, and knowledge storage. Use when starting complex tasks, multi-step projects, research tasks, or when the user mentions planning, organizing work, tracking progress, or wants structured output.","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fplanning-with-files\u002FSKILL.md",[],[],{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":19},"01M7CPPTQSBVWSK0R10FJNMP0F","n8n-workflow-patterns","Proven workflow architectural patterns from real n8n workflows. Use when building new workflows, designing workflow structure, choosing workflow patterns, planning workflow architecture, or asking about webhook processing, HTTP API integration, database operations, AI agent workflows, or scheduled tasks.","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fn8n\u002Fn8n-workflow-patterns\u002FSKILL.md",[],[]]