[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M74YQX9DBMHTMT3ZP0F3R19Y":3,"skill-same:01M74YQX9DBMHTMT3ZP0F3R19Y":64},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":30,"commitSha":31,"skillDirectory":32,"forksCount":33,"repositoryHtmlUrl":34,"repositoryHomepageUrl":35,"ownerHtmlUrl":36,"ownerBlogUrl":37,"ownerDescription":38,"updatedAt":39,"install":40,"resources":53},"01M74YQX9DBMHTMT3ZP0F3R19Y","gguf-quantization",null,"GGUF format and llama.cpp quantization for efficient CPU\u002FGPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.","KalarisLabs\u002Fresearch-agent-skills","KalarisLabs","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F300878568?v=4","01M74Y7MT23GXXC56VM6TVSXZY","Kalarislabs","01M74Y7MTQM9REXZX4SNXWX95C",132,"MIT","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fgguf-quantization\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: gguf-quantization\ndescription: GGUF format and llama.cpp quantization for efficient CPU\u002FGPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-training\n  maintainer: Kalaris Labs\n  tags: GGUF, Quantization, llama.cpp, CPU Inference, Apple Silicon, Model Compression, Optimization\n  dependencies: llama-cpp-python>=0.2.0\n---\n\n# GGUF - Quantization Format for llama.cpp\n\nThe GGUF (GPT-Generated Unified Format) is the standard file format for llama.cpp, enabling efficient inference on CPUs, Apple Silicon, and GPUs with flexible quantization options.\n\n## When to use GGUF\n\n**Use GGUF when:**\n- Deploying on consumer hardware (laptops, desktops)\n- Running on Apple Silicon (M1\u002FM2\u002FM3) with Metal acceleration\n- Need CPU inference without GPU requirements\n- Want flexible quantization (Q2_K to Q8_0)\n- Using local AI tools (LM Studio, Ollama, text-generation-webui)\n\n**Key advantages:**\n- **Universal hardware**: CPU, Apple Silicon, NVIDIA, AMD support\n- **No Python runtime**: Pure C\u002FC++ inference\n- **Flexible quantization**: 2-8 bit with various methods (K-quants)\n- **Ecosystem support**: LM Studio, Ollama, koboldcpp, and more\n- **imatrix**: Importance matrix for better low-bit quality\n\n**Use alternatives instead:**\n- **AWQ\u002FGPTQ**: Maximum accuracy with calibration on NVIDIA GPUs\n- **HQQ**: Fast calibration-free quantization for HuggingFace\n- **bitsandbytes**: Simple integration with transformers library\n- **TensorRT-LLM**: Production NVIDIA deployment with maximum speed\n\n## Quick start\n\n### Installation\n\n```bash\n# Clone llama.cpp\ngit clone https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\ncd llama.cpp\n\n# Build (CPU)\nmake\n\n# Build with CUDA (NVIDIA)\nmake GGML_CUDA=1\n\n# Build with Metal (Apple Silicon)\nmake GGML_METAL=1\n\n# Install Python bindings (optional)\npip install llama-cpp-python\n```\n\n### Convert model to GGUF\n\n```bash\n# Install requirements\npip install -r requirements.txt\n\n# Convert HuggingFace model to GGUF (FP16)\npython convert_hf_to_gguf.py .\u002Fpath\u002Fto\u002Fmodel --outfile model-f16.gguf\n\n# Or specify output type\npython convert_hf_to_gguf.py .\u002Fpath\u002Fto\u002Fmodel \\\n    --outfile model-f16.gguf \\\n    --outtype f16\n```\n\n### Quantize model\n\n```bash\n# Basic quantization to Q4_K_M\n.\u002Fllama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M\n\n# Quantize with importance matrix (better quality)\n.\u002Fllama-imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix\n.\u002Fllama-quantize --imatrix model.imatrix model-f16.gguf model-q4_k_m.gguf Q4_K_M\n```\n\n### Run inference\n\n```bash\n# CLI inference\n.\u002Fllama-cli -m model-q4_k_m.gguf -p \"Hello, how are you?\"\n\n# Interactive mode\n.\u002Fllama-cli -m model-q4_k_m.gguf --interactive\n\n# With GPU offload\n.\u002Fllama-cli -m model-q4_k_m.gguf -ngl 35 -p \"Hello!\"\n```\n\n## Quantization types\n\n### K-quant methods (recommended)\n\n| Type | Bits | Size (7B) | Quality | Use Case |\n|------|------|-----------|---------|----------|\n| Q2_K | 2.5 | ~2.8 GB | Low | Extreme compression |\n| Q3_K_S | 3.0 | ~3.0 GB | Low-Med | Memory constrained |\n| Q3_K_M | 3.3 | ~3.3 GB | Medium | Balance |\n| Q4_K_S | 4.0 | ~3.8 GB | Med-High | Good balance |\n| Q4_K_M | 4.5 | ~4.1 GB | High | **Recommended default** |\n| Q5_K_S | 5.0 | ~4.6 GB | High | Quality focused |\n| Q5_K_M | 5.5 | ~4.8 GB | Very High | High quality |\n| Q6_K | 6.0 | ~5.5 GB | Excellent | Near-original |\n| Q8_0 | 8.0 | ~7.2 GB | Best | Maximum quality |\n\n### Legacy methods\n\n| Type | Description |\n|------|-------------|\n| Q4_0 | 4-bit, basic |\n| Q4_1 | 4-bit with delta |\n| Q5_0 | 5-bit, basic |\n| Q5_1 | 5-bit with delta |\n\n**Recommendation**: Use K-quant methods (Q4_K_M, Q5_K_M) for best quality\u002Fsize ratio.\n\n## Conversion workflows\n\n### Workflow 1: HuggingFace to GGUF\n\n```bash\n# 1. Download model\nhuggingface-cli download meta-llama\u002FLlama-3.1-8B --local-dir .\u002Fllama-3.1-8b\n\n# 2. Convert to GGUF (FP16)\npython convert_hf_to_gguf.py .\u002Fllama-3.1-8b \\\n    --outfile llama-3.1-8b-f16.gguf \\\n    --outtype f16\n\n# 3. Quantize\n.\u002Fllama-quantize llama-3.1-8b-f16.gguf llama-3.1-8b-q4_k_m.gguf Q4_K_M\n\n# 4. Test\n.\u002Fllama-cli -m llama-3.1-8b-q4_k_m.gguf -p \"Hello!\" -n 50\n```\n\n### Workflow 2: With importance matrix (better quality)\n\n```bash\n# 1. Convert to GGUF\npython convert_hf_to_gguf.py .\u002Fmodel --outfile model-f16.gguf\n\n# 2. Create calibration text (diverse samples)\ncat > calibration.txt \u003C\u003C 'EOF'\nThe quick brown fox jumps over the lazy dog.\nMachine learning is a subset of artificial intelligence.\nPython is a popular programming language.\n# Add more diverse text samples...\nEOF\n\n# 3. Generate importance matrix\n.\u002Fllama-imatrix -m model-f16.gguf \\\n    -f calibration.txt \\\n    --chunk 512 \\\n    -o model.imatrix \\\n    -ngl 35  # GPU layers if available\n\n# 4. Quantize with imatrix\n.\u002Fllama-quantize --imatrix model.imatrix \\\n    model-f16.gguf \\\n    model-q4_k_m.gguf \\\n    Q4_K_M\n```\n\n### Workflow 3: Multiple quantizations\n\n```bash\n#!\u002Fbin\u002Fbash\nMODEL=\"llama-3.1-8b-f16.gguf\"\nIMATRIX=\"llama-3.1-8b.imatrix\"\n\n# Generate imatrix once\n.\u002Fllama-imatrix -m $MODEL -f wiki.txt -o $IMATRIX -ngl 35\n\n# Create multiple quantizations\nfor QUANT in Q4_K_M Q5_K_M Q6_K Q8_0; do\n    OUTPUT=\"llama-3.1-8b-${QUANT,,}.gguf\"\n    .\u002Fllama-quantize --imatrix $IMATRIX $MODEL $OUTPUT $QUANT\n    echo \"Created: $OUTPUT ($(du -h $OUTPUT | cut -f1))\"\ndone\n```\n\n## Python usage\n\n### llama-cpp-python\n\n```python\nfrom llama_cpp import Llama\n\n# Load model\nllm = Llama(\n    model_path=\".\u002Fmodel-q4_k_m.gguf\",\n    n_ctx=4096,          # Context window\n    n_gpu_layers=35,     # GPU offload (0 for CPU only)\n    n_threads=8          # CPU threads\n)\n\n# Generate\noutput = llm(\n    \"What is machine learning?\",\n    max_tokens=256,\n    temperature=0.7,\n    stop=[\"\u003C\u002Fs>\", \"\\n\\n\"]\n)\nprint(output[\"choices\"][0][\"text\"])\n```\n\n### Chat completion\n\n```python\nfrom llama_cpp import Llama\n\nllm = Llama(\n    model_path=\".\u002Fmodel-q4_k_m.gguf\",\n    n_ctx=4096,\n    n_gpu_layers=35,\n    chat_format=\"llama-3\"  # Or \"chatml\", \"mistral\", etc.\n)\n\nmessages = [\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n    {\"role\": \"user\", \"content\": \"What is Python?\"}\n]\n\nresponse = llm.create_chat_completion(\n    messages=messages,\n    max_tokens=256,\n    temperature=0.7\n)\nprint(response[\"choices\"][0][\"message\"][\"content\"])\n```\n\n### Streaming\n\n```python\nfrom llama_cpp import Llama\n\nllm = Llama(model_path=\".\u002Fmodel-q4_k_m.gguf\", n_gpu_layers=35)\n\n# Stream tokens\nfor chunk in llm(\n    \"Explain quantum computing:\",\n    max_tokens=256,\n    stream=True\n):\n    print(chunk[\"choices\"][0][\"text\"], end=\"\", flush=True)\n```\n\n## Server mode\n\n### Start OpenAI-compatible server\n\n```bash\n# Start server\n.\u002Fllama-server -m model-q4_k_m.gguf \\\n    --host 0.0.0.0 \\\n    --port 8080 \\\n    -ngl 35 \\\n    -c 4096\n\n# Or with Python bindings\npython -m llama_cpp.server \\\n    --model model-q4_k_m.gguf \\\n    --n_gpu_layers 35 \\\n    --host 0.0.0.0 \\\n    --port 8080\n```\n\n### Use with OpenAI client\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http:\u002F\u002Flocalhost:8080\u002Fv1\",\n    api_key=\"not-needed\"\n)\n\nresponse = client.chat.completions.create(\n    model=\"local-model\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n    max_tokens=256\n)\nprint(response.choices[0].message.content)\n```\n\n## Hardware optimization\n\n### Apple Silicon (Metal)\n\n```bash\n# Build with Metal\nmake clean && make GGML_METAL=1\n\n# Run with Metal acceleration\n.\u002Fllama-cli -m model.gguf -ngl 99 -p \"Hello\"\n\n# Python with Metal\nllm = Llama(\n    model_path=\"model.gguf\",\n    n_gpu_layers=99,     # Offload all layers\n    n_threads=1          # Metal handles parallelism\n)\n```\n\n### NVIDIA CUDA\n\n```bash\n# Build with CUDA\nmake clean && make GGML_CUDA=1\n\n# Run with CUDA\n.\u002Fllama-cli -m model.gguf -ngl 35 -p \"Hello\"\n\n# Specify GPU\nCUDA_VISIBLE_DEVICES=0 .\u002Fllama-cli -m model.gguf -ngl 35\n```\n\n### CPU optimization\n\n```bash\n# Build with AVX2\u002FAVX512\nmake clean && make\n\n# Run with optimal threads\n.\u002Fllama-cli -m model.gguf -t 8 -p \"Hello\"\n\n# Python CPU config\nllm = Llama(\n    model_path=\"model.gguf\",\n    n_gpu_layers=0,      # CPU only\n    n_threads=8,         # Match physical cores\n    n_batch=512          # Batch size for prompt processing\n)\n```\n\n## Integration with tools\n\n### Ollama\n\n```bash\n# Create Modelfile\ncat > Modelfile \u003C\u003C 'EOF'\nFROM .\u002Fmodel-q4_k_m.gguf\nTEMPLATE \"\"\"{{ .System }}\n{{ .Prompt }}\"\"\"\nPARAMETER temperature 0.7\nPARAMETER num_ctx 4096\nEOF\n\n# Create Ollama model\nollama create mymodel -f Modelfile\n\n# Run\nollama run mymodel \"Hello!\"\n```\n\n### LM Studio\n\n1. Place GGUF file in `~\u002F.cache\u002Flm-studio\u002Fmodels\u002F`\n2. Open LM Studio and select the model\n3. Configure context length and GPU offload\n4. Start inference\n\n### text-generation-webui\n\n```bash\n# Place in models folder\ncp model-q4_k_m.gguf text-generation-webui\u002Fmodels\u002F\n\n# Start with llama.cpp loader\npython server.py --model model-q4_k_m.gguf --loader llama.cpp --n-gpu-layers 35\n```\n\n## Best practices\n\n1. **Use K-quants**: Q4_K_M offers best quality\u002Fsize balance\n2. **Use imatrix**: Always use importance matrix for Q4 and below\n3. **GPU offload**: Offload as many layers as VRAM allows\n4. **Context length**: Start with 4096, increase if needed\n5. **Thread count**: Match physical CPU cores, not logical\n6. **Batch size**: Increase n_batch for faster prompt processing\n\n## Common issues\n\n**Model loads slowly:**\n```bash\n# Use mmap for faster loading\n.\u002Fllama-cli -m model.gguf --mmap\n```\n\n**Out of memory:**\n```bash\n# Reduce GPU layers\n.\u002Fllama-cli -m model.gguf -ngl 20  # Reduce from 35\n\n# Or use smaller quantization\n.\u002Fllama-quantize model-f16.gguf model-q3_k_m.gguf Q3_K_M\n```\n\n**Poor quality at low bits:**\n```bash\n# Always use imatrix for Q4 and below\n.\u002Fllama-imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix\n.\u002Fllama-quantize --imatrix model.imatrix model-f16.gguf model-q4_k_m.gguf Q4_K_M\n```\n\n## References\n\n- **[Advanced Usage](references\u002Fadvanced-usage.md)** - Batching, speculative decoding, custom builds\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, debugging, benchmarks\n\n## Resources\n\n- **Repository**: https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\n- **Python Bindings**: https:\u002F\u002Fgithub.com\u002Fabetlen\u002Fllama-cpp-python\n- **Pre-quantized Models**: https:\u002F\u002Fhuggingface.co\u002FTheBloke\n- **GGUF Converter**: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fggml-org\u002Fgguf-my-repo\n- **License**: MIT\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `llama-cpp`: Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware.\n- `hqq-quantization`: Half-Quadratic Quantization for LLMs without calibration data.\n- `awq-quantization`: Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss.\n","# GGUF - Quantization Format for llama.cpp\n\nThe GGUF (GPT-Generated Unified Format) is the standard file format for llama.cpp, enabling efficient inference on CPUs, Apple Silicon, and GPUs with flexible quantization options.\n\n## When to use GGUF\n\n**Use GGUF when:**\n- Deploying on consumer hardware (laptops, desktops)\n- Running on Apple Silicon (M1\u002FM2\u002FM3) with Metal acceleration\n- Need CPU inference without GPU requirements\n- Want flexible quantization (Q2_K to Q8_0)\n- Using local AI tools (LM Studio, Ollama, text-generation-webui)\n\n**Key advantages:**\n- **Universal hardware**: CPU, Apple Silicon, NVIDIA, AMD support\n- **No Python runtime**: Pure C\u002FC++ inference\n- **Flexible quantization**: 2-8 bit with various methods (K-quants)\n- **Ecosystem support**: LM Studio, Ollama, koboldcpp, and more\n- **imatrix**: Importance matrix for better low-bit quality\n\n**Use alternatives instead:**\n- **AWQ\u002FGPTQ**: Maximum accuracy with calibration on NVIDIA GPUs\n- **HQQ**: Fast calibration-free quantization for HuggingFace\n- **bitsandbytes**: Simple integration with transformers library\n- **TensorRT-LLM**: Production NVIDIA deployment with maximum speed\n\n## Quick start\n\n### Installation\n\n```bash\n# Clone llama.cpp\ngit clone https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\ncd llama.cpp\n\n# Build (CPU)\nmake\n\n# Build with CUDA (NVIDIA)\nmake GGML_CUDA=1\n\n# Build with Metal (Apple Silicon)\nmake GGML_METAL=1\n\n# Install Python bindings (optional)\npip install llama-cpp-python\n```\n\n### Convert model to GGUF\n\n```bash\n# Install requirements\npip install -r requirements.txt\n\n# Convert HuggingFace model to GGUF (FP16)\npython convert_hf_to_gguf.py .\u002Fpath\u002Fto\u002Fmodel --outfile model-f16.gguf\n\n# Or specify output type\npython convert_hf_to_gguf.py .\u002Fpath\u002Fto\u002Fmodel \\\n    --outfile model-f16.gguf \\\n    --outtype f16\n```\n\n### Quantize model\n\n```bash\n# Basic quantization to Q4_K_M\n.\u002Fllama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M\n\n# Quantize with importance matrix (better quality)\n.\u002Fllama-imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix\n.\u002Fllama-quantize --imatrix model.imatrix model-f16.gguf model-q4_k_m.gguf Q4_K_M\n```\n\n### Run inference\n\n```bash\n# CLI inference\n.\u002Fllama-cli -m model-q4_k_m.gguf -p \"Hello, how are you?\"\n\n# Interactive mode\n.\u002Fllama-cli -m model-q4_k_m.gguf --interactive\n\n# With GPU offload\n.\u002Fllama-cli -m model-q4_k_m.gguf -ngl 35 -p \"Hello!\"\n```\n\n## Quantization types\n\n### K-quant methods (recommended)\n\n| Type | Bits | Size (7B) | Quality | Use Case |\n|------|------|-----------|---------|----------|\n| Q2_K | 2.5 | ~2.8 GB | Low | Extreme compression |\n| Q3_K_S | 3.0 | ~3.0 GB | Low-Med | Memory constrained |\n| Q3_K_M | 3.3 | ~3.3 GB | Medium | Balance |\n| Q4_K_S | 4.0 | ~3.8 GB | Med-High | Good balance |\n| Q4_K_M | 4.5 | ~4.1 GB | High | **Recommended default** |\n| Q5_K_S | 5.0 | ~4.6 GB | High | Quality focused |\n| Q5_K_M | 5.5 | ~4.8 GB | Very High | High quality |\n| Q6_K | 6.0 | ~5.5 GB | Excellent | Near-original |\n| Q8_0 | 8.0 | ~7.2 GB | Best | Maximum quality |\n\n### Legacy methods\n\n| Type | Description |\n|------|-------------|\n| Q4_0 | 4-bit, basic |\n| Q4_1 | 4-bit with delta |\n| Q5_0 | 5-bit, basic |\n| Q5_1 | 5-bit with delta |\n\n**Recommendation**: Use K-quant methods (Q4_K_M, Q5_K_M) for best quality\u002Fsize ratio.\n\n## Conversion workflows\n\n### Workflow 1: HuggingFace to GGUF\n\n```bash\n# 1. Download model\nhuggingface-cli download meta-llama\u002FLlama-3.1-8B --local-dir .\u002Fllama-3.1-8b\n\n# 2. Convert to GGUF (FP16)\npython convert_hf_to_gguf.py .\u002Fllama-3.1-8b \\\n    --outfile llama-3.1-8b-f16.gguf \\\n    --outtype f16\n\n# 3. Quantize\n.\u002Fllama-quantize llama-3.1-8b-f16.gguf llama-3.1-8b-q4_k_m.gguf Q4_K_M\n\n# 4. Test\n.\u002Fllama-cli -m llama-3.1-8b-q4_k_m.gguf -p \"Hello!\" -n 50\n```\n\n### Workflow 2: With importance matrix (better quality)\n\n```bash\n# 1. Convert to GGUF\npython convert_hf_to_gguf.py .\u002Fmodel --outfile model-f16.gguf\n\n# 2. Create calibration text (diverse samples)\ncat > calibration.txt \u003C\u003C 'EOF'\nThe quick brown fox jumps over the lazy dog.\nMachine learning is a subset of artificial intelligence.\nPython is a popular programming language.\n# Add more diverse text samples...\nEOF\n\n# 3. Generate importance matrix\n.\u002Fllama-imatrix -m model-f16.gguf \\\n    -f calibration.txt \\\n    --chunk 512 \\\n    -o model.imatrix \\\n    -ngl 35  # GPU layers if available\n\n# 4. Quantize with imatrix\n.\u002Fllama-quantize --imatrix model.imatrix \\\n    model-f16.gguf \\\n    model-q4_k_m.gguf \\\n    Q4_K_M\n```\n\n### Workflow 3: Multiple quantizations\n\n```bash\n#!\u002Fbin\u002Fbash\nMODEL=\"llama-3.1-8b-f16.gguf\"\nIMATRIX=\"llama-3.1-8b.imatrix\"\n\n# Generate imatrix once\n.\u002Fllama-imatrix -m $MODEL -f wiki.txt -o $IMATRIX -ngl 35\n\n# Create multiple quantizations\nfor QUANT in Q4_K_M Q5_K_M Q6_K Q8_0; do\n    OUTPUT=\"llama-3.1-8b-${QUANT,,}.gguf\"\n    .\u002Fllama-quantize --imatrix $IMATRIX $MODEL $OUTPUT $QUANT\n    echo \"Created: $OUTPUT ($(du -h $OUTPUT | cut -f1))\"\ndone\n```\n\n## Python usage\n\n### llama-cpp-python\n\n```python\nfrom llama_cpp import Llama\n\n# Load model\nllm = Llama(\n    model_path=\".\u002Fmodel-q4_k_m.gguf\",\n    n_ctx=4096,          # Context window\n    n_gpu_layers=35,     # GPU offload (0 for CPU only)\n    n_threads=8          # CPU threads\n)\n\n# Generate\noutput = llm(\n    \"What is machine learning?\",\n    max_tokens=256,\n    temperature=0.7,\n    stop=[\"\u003C\u002Fs>\", \"\\n\\n\"]\n)\nprint(output[\"choices\"][0][\"text\"])\n```\n\n### Chat completion\n\n```python\nfrom llama_cpp import Llama\n\nllm = Llama(\n    model_path=\".\u002Fmodel-q4_k_m.gguf\",\n    n_ctx=4096,\n    n_gpu_layers=35,\n    chat_format=\"llama-3\"  # Or \"chatml\", \"mistral\", etc.\n)\n\nmessages = [\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n    {\"role\": \"user\", \"content\": \"What is Python?\"}\n]\n\nresponse = llm.create_chat_completion(\n    messages=messages,\n    max_tokens=256,\n    temperature=0.7\n)\nprint(response[\"choices\"][0][\"message\"][\"content\"])\n```\n\n### Streaming\n\n```python\nfrom llama_cpp import Llama\n\nllm = Llama(model_path=\".\u002Fmodel-q4_k_m.gguf\", n_gpu_layers=35)\n\n# Stream tokens\nfor chunk in llm(\n    \"Explain quantum computing:\",\n    max_tokens=256,\n    stream=True\n):\n    print(chunk[\"choices\"][0][\"text\"], end=\"\", flush=True)\n```\n\n## Server mode\n\n### Start OpenAI-compatible server\n\n```bash\n# Start server\n.\u002Fllama-server -m model-q4_k_m.gguf \\\n    --host 0.0.0.0 \\\n    --port 8080 \\\n    -ngl 35 \\\n    -c 4096\n\n# Or with Python bindings\npython -m llama_cpp.server \\\n    --model model-q4_k_m.gguf \\\n    --n_gpu_layers 35 \\\n    --host 0.0.0.0 \\\n    --port 8080\n```\n\n### Use with OpenAI client\n\n```python\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http:\u002F\u002Flocalhost:8080\u002Fv1\",\n    api_key=\"not-needed\"\n)\n\nresponse = client.chat.completions.create(\n    model=\"local-model\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n    max_tokens=256\n)\nprint(response.choices[0].message.content)\n```\n\n## Hardware optimization\n\n### Apple Silicon (Metal)\n\n```bash\n# Build with Metal\nmake clean && make GGML_METAL=1\n\n# Run with Metal acceleration\n.\u002Fllama-cli -m model.gguf -ngl 99 -p \"Hello\"\n\n# Python with Metal\nllm = Llama(\n    model_path=\"model.gguf\",\n    n_gpu_layers=99,     # Offload all layers\n    n_threads=1          # Metal handles parallelism\n)\n```\n\n### NVIDIA CUDA\n\n```bash\n# Build with CUDA\nmake clean && make GGML_CUDA=1\n\n# Run with CUDA\n.\u002Fllama-cli -m model.gguf -ngl 35 -p \"Hello\"\n\n# Specify GPU\nCUDA_VISIBLE_DEVICES=0 .\u002Fllama-cli -m model.gguf -ngl 35\n```\n\n### CPU optimization\n\n```bash\n# Build with AVX2\u002FAVX512\nmake clean && make\n\n# Run with optimal threads\n.\u002Fllama-cli -m model.gguf -t 8 -p \"Hello\"\n\n# Python CPU config\nllm = Llama(\n    model_path=\"model.gguf\",\n    n_gpu_layers=0,      # CPU only\n    n_threads=8,         # Match physical cores\n    n_batch=512          # Batch size for prompt processing\n)\n```\n\n## Integration with tools\n\n### Ollama\n\n```bash\n# Create Modelfile\ncat > Modelfile \u003C\u003C 'EOF'\nFROM .\u002Fmodel-q4_k_m.gguf\nTEMPLATE \"\"\"{{ .System }}\n{{ .Prompt }}\"\"\"\nPARAMETER temperature 0.7\nPARAMETER num_ctx 4096\nEOF\n\n# Create Ollama model\nollama create mymodel -f Modelfile\n\n# Run\nollama run mymodel \"Hello!\"\n```\n\n### LM Studio\n\n1. Place GGUF file in `~\u002F.cache\u002Flm-studio\u002Fmodels\u002F`\n2. Open LM Studio and select the model\n3. Configure context length and GPU offload\n4. Start inference\n\n### text-generation-webui\n\n```bash\n# Place in models folder\ncp model-q4_k_m.gguf text-generation-webui\u002Fmodels\u002F\n\n# Start with llama.cpp loader\npython server.py --model model-q4_k_m.gguf --loader llama.cpp --n-gpu-layers 35\n```\n\n## Best practices\n\n1. **Use K-quants**: Q4_K_M offers best quality\u002Fsize balance\n2. **Use imatrix**: Always use importance matrix for Q4 and below\n3. **GPU offload**: Offload as many layers as VRAM allows\n4. **Context length**: Start with 4096, increase if needed\n5. **Thread count**: Match physical CPU cores, not logical\n6. **Batch size**: Increase n_batch for faster prompt processing\n\n## Common issues\n\n**Model loads slowly:**\n```bash\n# Use mmap for faster loading\n.\u002Fllama-cli -m model.gguf --mmap\n```\n\n**Out of memory:**\n```bash\n# Reduce GPU layers\n.\u002Fllama-cli -m model.gguf -ngl 20  # Reduce from 35\n\n# Or use smaller quantization\n.\u002Fllama-quantize model-f16.gguf model-q3_k_m.gguf Q3_K_M\n```\n\n**Poor quality at low bits:**\n```bash\n# Always use imatrix for Q4 and below\n.\u002Fllama-imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix\n.\u002Fllama-quantize --imatrix model.imatrix model-f16.gguf model-q4_k_m.gguf Q4_K_M\n```\n\n## References\n\n- **[Advanced Usage](references\u002Fadvanced-usage.md)** - Batching, speculative decoding, custom builds\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, debugging, benchmarks\n\n## Resources\n\n- **Repository**: https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\n- **Python Bindings**: https:\u002F\u002Fgithub.com\u002Fabetlen\u002Fllama-cpp-python\n- **Pre-quantized Models**: https:\u002F\u002Fhuggingface.co\u002FTheBloke\n- **GGUF Converter**: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fggml-org\u002Fgguf-my-repo\n- **License**: MIT\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `llama-cpp`: Runs LLM inference on CPU, Apple Silicon, and consumer GPUs without NVIDIA hardware.\n- `hqq-quantization`: Half-Quadratic Quantization for LLMs without calibration data.\n- `awq-quantization`: Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss.\n",{"name":5,"license":15,"metadata":24,"description":7},{"tags":25,"version":26,"category":27,"maintainer":28,"dependencies":29},"GGUF, Quantization, llama.cpp, CPU Inference, Apple Silicon, Model Compression, Optimization","1.0.0","ml-training","Kalaris Labs","llama-cpp-python>=0.2.0","main","29fe36d26127205057d5e0ed8425f6b0e9a8c13a","skills\u002Fgguf-quantization",16,"https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills","https:\u002F\u002Fdocs.kalarislabs.com\u002Fresearch-agent-skills","https:\u002F\u002Fgithub.com\u002FKalarisLabs","kalarislabs.com","Applied AI lab building the infrastructure for the agentic era","2026-11-14 17:49:08",{"source":8,"skillName":5,"commands":41,"prompt":51,"zipAvailable":52},[42,45,48],{"tool":43,"command":44},"npx","npx skills add KalarisLabs\u002Fresearch-agent-skills --skill gguf-quantization",{"tool":46,"command":47},"bunx","bunx skills add KalarisLabs\u002Fresearch-agent-skills --skill gguf-quantization",{"tool":49,"command":50},"pnpm","pnpm dlx skills add KalarisLabs\u002Fresearch-agent-skills --skill gguf-quantization","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fgguf-quantization\u002FSKILL.md",true,[54,60],{"path":55,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":58,"sourceHtmlUrl":59},"references\u002Fadvanced-usage.md","REFERENCE","参考文档",10887,"https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fgguf-quantization\u002Freferences\u002Fadvanced-usage.md",{"path":61,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":62,"sourceHtmlUrl":63},"references\u002Ftroubleshooting.md",8904,"https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fgguf-quantization\u002Freferences\u002Ftroubleshooting.md",[65,72,79,86,93,100],{"skillNo":66,"name":67,"nameZh":6,"description":68,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":69,"categories":70,"keywords":71,"publishedAt":19},"01M74Z8MGT305R11SNY5G28QH5","plos","Prepare manuscripts for PLOS journals (PLOS ONE, PLOS Biology, PLOS Computational Biology, PLOS Genetics, PLOS Medicine, PLOS Pathogens, PLOS Neglected Tropical Diseases, PLOS Global Public Health and others), covering structure, abstract and author summary, Vancouver-style references, the mandatory data availability policy, ethics and financial disclosure statements, figure requirements (PACE), the PLOS LaTeX template, reporting guidelines and PLOS ONE's publication criteria. Use when targeting any PLOS journal or checking compliance with PLOS policies.","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fplos\u002FSKILL.md",[],[],{"skillNo":73,"name":74,"nameZh":6,"description":75,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":76,"categories":77,"keywords":78,"publishedAt":19},"01M74Z8CCN9APXYHNQ95V1EBGT","pkpd-modeling","Pharmacokinetic and pharmacodynamic modelling and simulation - non-compartmental analysis, compartmental and population PK, PK\u002FPD and exposure-response, TMDD, PBPK orientation, bioequivalence, allometric scaling and first-in-human dose, drug interaction prediction, and Bayesian therapeutic drug monitoring. Use when analysing concentration-time data, deriving exposure metrics, fitting PK or PD models, or evaluating dosing regimens. Triggers include \"pharmacokinetics\", \"pharmacodynamics\", \"PK\u002FPD\", \"NCA\", \"non-compartmental\", \"AUC\", \"Cmax\", \"lambda z\", \"half-life\", \"clearance\", \"volume of distribution\", \"compartmental model\", \"population PK\", \"popPK\", \"NONMEM\", \"nlmixr2\", \"Pharmpy\", \"Monolix\", \"exposure-response\", \"Emax\", \"EC50\", \"indirect response\", \"effect compartment\", \"TMDD\", \"PBPK\", \"bioequivalence\", \"RSABE\", \"ABEL\", \"allometric scaling\", \"first-in-human\", \"MABEL\", \"drug-drug interaction\", \"DDI\", \"ICH M12\", \"concentration-QTc\", \"therapeutic drug monitoring\", \"MIPD\", and \"dosing regimen\".","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fpkpd-modeling\u002FSKILL.md",[],[],{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":19},"01M74Z86YHJJ1FM3X87BHN76KZ","pinecone","Guides use of Pinecone, a managed serverless vector database, through its Python client and the LangChain and LlamaIndex integrations. Covers creating indexes, upserting and querying vectors, metadata filtering, namespaces, hybrid dense and sparse search, index management, and deleting vectors. Use when building a production RAG system on a hosted vector store, adding semantic search or recommendations without running infrastructure, isolating per-user or per-tenant data with namespaces, combining dense and sparse vectors in one query, or filtering results by metadata. Do not use for self-hosted or local stores (Chroma, Weaviate) or offline similarity search (FAISS).","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fpinecone\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":19},"01M74Z80T6E89SAJ8MYDD0767Q","pi-agent","Build with and use Pi, the minimal terminal coding harness. Use for installing Pi, configuring providers\u002Fmodels\u002Fsettings\u002Fenvironment variables, creating Pi skills\u002Fextensions\u002Fpackages\u002Fthemes\u002Fprompt templates, embedding Pi through the SDK, integrating over RPC or JSON event streams, parsing sessions, running local models through the llama.cpp router, developing custom Pi providers and TUI components, or using ecosystem packages such as pi-subagents (delegation\u002Forchestration), pi-mcp-adapter (MCP servers), pi-interview (interactive forms), and pi-web-access (web search, fetching, video understanding).","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fpi-agent\u002FSKILL.md",[],[],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":19},"01M74Z7WF1Y8Q3VFAQPPPRAFMV","phoenix-observability","Open-source AI observability platform for LLM tracing, evaluation, and monitoring. Use when debugging LLM applications with detailed traces, running evaluations on datasets, or monitoring production AI systems with real-time insights.","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fphoenix-observability\u002FSKILL.md",[],[],{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":19},"01M74Z7HNEH1A2AVYR3G1X2QV6","peft-fine-tuning","Fine-tunes LLMs with Hugging Face PEFT, using LoRA, QLoRA, IA3, AdaLoRA, prefix tuning, and prompt tuning so that under 1% of parameters are trained. Covers rank, alpha, and target module selection, loading and merging adapters, multi-adapter serving, and integration with TRL SFTTrainer, Axolotl, and vLLM. Use when fine-tuning 7B-70B models on limited GPU memory, when running QLoRA on a single 24GB GPU, when choosing LoRA rank and alpha, when merging or swapping adapters on one base model, or when debugging CUDA OOM or an adapter that does not apply. Not for full fine-tuning of small models under 1B parameters or cases needing all weights updated.","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fpeft-fine-tuning\u002FSKILL.md",[],[]]