[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6GACVEAF474HBJCYFZ4MD9V":3,"skill-same:01M6GACVEAF474HBJCYFZ4MD9V":74},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":52,"commitSha":53,"skillDirectory":54,"forksCount":55,"repositoryHtmlUrl":56,"repositoryHomepageUrl":6,"ownerHtmlUrl":57,"ownerBlogUrl":6,"ownerDescription":58,"updatedAt":59,"install":60,"resources":73},"01M6GACVEAF474HBJCYFZ4MD9V","gguf-quantization",null,"GGUF format and llama.cpp quantization for efficient CPU\u002FGPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.","zapabob\u002FSkills","zapabob","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F130830376?v=4","01M6DBMYZAGRTFQDSQYW3W764P","峯岸 亮","01M6DBMZ3GKDSJ01NZ9XTZD7JD",2,"MIT","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Farchive\u002Fskill-md-reviewed\u002Fsha256\u002F89\u002F8944bce3f696818423bd3433340b730b121a561c757ed749814e90d34680b81f\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"GGUF","Quantization","gguf quantization","requirements","Model Compression","CPU Inference","llama.cpp","CPU","silicon","bit","apple","python","models","use","model","apple silicon","Optimization","gpu","cpp","research","2026-11-10 10:12:30",false,"---\r\nname: gguf-quantization\r\ndescription: GGUF format and llama.cpp quantization for efficient CPU\u002FGPU inference. Use when deploying models on consumer hardware, Apple Silicon, or when needing flexible quantization from 2-8 bit without GPU requirements.\r\nversion: 1.0.0\r\nauthor: Orchestra Research\r\nlicense: MIT\r\ndependencies: [llama-cpp-python>=0.2.0]\r\nmetadata:\r\n  hermes:\r\n    tags: [GGUF, Quantization, llama.cpp, CPU Inference, Apple Silicon, Model Compression, Optimization]\r\n\r\n---\r\n\r\n# GGUF - Quantization Format for llama.cpp\r\n\r\nThe GGUF (GPT-Generated Unified Format) is the standard file format for llama.cpp, enabling efficient inference on CPUs, Apple Silicon, and GPUs with flexible quantization options.\r\n\r\n## When to use GGUF\r\n\r\n**Use GGUF when:**\r\n- Deploying on consumer hardware (laptops, desktops)\r\n- Running on Apple Silicon (M1\u002FM2\u002FM3) with Metal acceleration\r\n- Need CPU inference without GPU requirements\r\n- Want flexible quantization (Q2_K to Q8_0)\r\n- Using local AI tools (LM Studio, Ollama, text-generation-webui)\r\n\r\n**Key advantages:**\r\n- **Universal hardware**: CPU, Apple Silicon, NVIDIA, AMD support\r\n- **No Python runtime**: Pure C\u002FC++ inference\r\n- **Flexible quantization**: 2-8 bit with various methods (K-quants)\r\n- **Ecosystem support**: LM Studio, Ollama, koboldcpp, and more\r\n- **imatrix**: Importance matrix for better low-bit quality\r\n\r\n**Use alternatives instead:**\r\n- **AWQ\u002FGPTQ**: Maximum accuracy with calibration on NVIDIA GPUs\r\n- **HQQ**: Fast calibration-free quantization for HuggingFace\r\n- **bitsandbytes**: Simple integration with transformers library\r\n- **TensorRT-LLM**: Production NVIDIA deployment with maximum speed\r\n\r\n## Quick start\r\n\r\n### Installation\r\n\r\n```bash\r\n# Clone llama.cpp\r\ngit clone https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\r\ncd llama.cpp\r\n\r\n# Build (CPU)\r\nmake\r\n\r\n# Build with CUDA (NVIDIA)\r\nmake GGML_CUDA=1\r\n\r\n# Build with Metal (Apple Silicon)\r\nmake GGML_METAL=1\r\n\r\n# Install Python bindings (optional)\r\npip install llama-cpp-python\r\n```\r\n\r\n### Convert model to GGUF\r\n\r\n```bash\r\n# Install requirements\r\npip install -r requirements.txt\r\n\r\n# Convert HuggingFace model to GGUF (FP16)\r\npython convert_hf_to_gguf.py .\u002Fpath\u002Fto\u002Fmodel --outfile model-f16.gguf\r\n\r\n# Or specify output type\r\npython convert_hf_to_gguf.py .\u002Fpath\u002Fto\u002Fmodel \\\r\n    --outfile model-f16.gguf \\\r\n    --outtype f16\r\n```\r\n\r\n### Quantize model\r\n\r\n```bash\r\n# Basic quantization to Q4_K_M\r\n.\u002Fllama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M\r\n\r\n# Quantize with importance matrix (better quality)\r\n.\u002Fllama-imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix\r\n.\u002Fllama-quantize --imatrix model.imatrix model-f16.gguf model-q4_k_m.gguf Q4_K_M\r\n```\r\n\r\n### Run inference\r\n\r\n```bash\r\n# CLI inference\r\n.\u002Fllama-cli -m model-q4_k_m.gguf -p \"Hello, how are you?\"\r\n\r\n# Interactive mode\r\n.\u002Fllama-cli -m model-q4_k_m.gguf --interactive\r\n\r\n# With GPU offload\r\n.\u002Fllama-cli -m model-q4_k_m.gguf -ngl 35 -p \"Hello!\"\r\n```\r\n\r\n## Quantization types\r\n\r\n### K-quant methods (recommended)\r\n\r\n| Type | Bits | Size (7B) | Quality | Use Case |\r\n|------|------|-----------|---------|----------|\r\n| Q2_K | 2.5 | ~2.8 GB | Low | Extreme compression |\r\n| Q3_K_S | 3.0 | ~3.0 GB | Low-Med | Memory constrained |\r\n| Q3_K_M | 3.3 | ~3.3 GB | Medium | Balance |\r\n| Q4_K_S | 4.0 | ~3.8 GB | Med-High | Good balance |\r\n| Q4_K_M | 4.5 | ~4.1 GB | High | **Recommended default** |\r\n| Q5_K_S | 5.0 | ~4.6 GB | High | Quality focused |\r\n| Q5_K_M | 5.5 | ~4.8 GB | Very High | High quality |\r\n| Q6_K | 6.0 | ~5.5 GB | Excellent | Near-original |\r\n| Q8_0 | 8.0 | ~7.2 GB | Best | Maximum quality |\r\n\r\n### Legacy methods\r\n\r\n| Type | Description |\r\n|------|-------------|\r\n| Q4_0 | 4-bit, basic |\r\n| Q4_1 | 4-bit with delta |\r\n| Q5_0 | 5-bit, basic |\r\n| Q5_1 | 5-bit with delta |\r\n\r\n**Recommendation**: Use K-quant methods (Q4_K_M, Q5_K_M) for best quality\u002Fsize ratio.\r\n\r\n## Conversion workflows\r\n\r\n### Workflow 1: HuggingFace to GGUF\r\n\r\n```bash\r\n# 1. Download model\r\nhuggingface-cli download meta-llama\u002FLlama-3.1-8B --local-dir .\u002Fllama-3.1-8b\r\n\r\n# 2. Convert to GGUF (FP16)\r\npython convert_hf_to_gguf.py .\u002Fllama-3.1-8b \\\r\n    --outfile llama-3.1-8b-f16.gguf \\\r\n    --outtype f16\r\n\r\n# 3. Quantize\r\n.\u002Fllama-quantize llama-3.1-8b-f16.gguf llama-3.1-8b-q4_k_m.gguf Q4_K_M\r\n\r\n# 4. Test\r\n.\u002Fllama-cli -m llama-3.1-8b-q4_k_m.gguf -p \"Hello!\" -n 50\r\n```\r\n\r\n### Workflow 2: With importance matrix (better quality)\r\n\r\n```bash\r\n# 1. Convert to GGUF\r\npython convert_hf_to_gguf.py .\u002Fmodel --outfile model-f16.gguf\r\n\r\n# 2. Create calibration text (diverse samples)\r\ncat > calibration.txt \u003C\u003C 'EOF'\r\nThe quick brown fox jumps over the lazy dog.\r\nMachine learning is a subset of artificial intelligence.\r\nPython is a popular programming language.\r\n# Add more diverse text samples...\r\nEOF\r\n\r\n# 3. Generate importance matrix\r\n.\u002Fllama-imatrix -m model-f16.gguf \\\r\n    -f calibration.txt \\\r\n    --chunk 512 \\\r\n    -o model.imatrix \\\r\n    -ngl 35  # GPU layers if available\r\n\r\n# 4. Quantize with imatrix\r\n.\u002Fllama-quantize --imatrix model.imatrix \\\r\n    model-f16.gguf \\\r\n    model-q4_k_m.gguf \\\r\n    Q4_K_M\r\n```\r\n\r\n### Workflow 3: Multiple quantizations\r\n\r\n```bash\r\n#!\u002Fbin\u002Fbash\r\nMODEL=\"llama-3.1-8b-f16.gguf\"\r\nIMATRIX=\"llama-3.1-8b.imatrix\"\r\n\r\n# Generate imatrix once\r\n.\u002Fllama-imatrix -m $MODEL -f wiki.txt -o $IMATRIX -ngl 35\r\n\r\n# Create multiple quantizations\r\nfor QUANT in Q4_K_M Q5_K_M Q6_K Q8_0; do\r\n    OUTPUT=\"llama-3.1-8b-${QUANT,,}.gguf\"\r\n    .\u002Fllama-quantize --imatrix $IMATRIX $MODEL $OUTPUT $QUANT\r\n    echo \"Created: $OUTPUT ($(du -h $OUTPUT | cut -f1))\"\r\ndone\r\n```\r\n\r\n## Python usage\r\n\r\n### llama-cpp-python\r\n\r\n```python\r\nfrom llama_cpp import Llama\r\n\r\n# Load model\r\nllm = Llama(\r\n    model_path=\".\u002Fmodel-q4_k_m.gguf\",\r\n    n_ctx=4096,          # Context window\r\n    n_gpu_layers=35,     # GPU offload (0 for CPU only)\r\n    n_threads=8          # CPU threads\r\n)\r\n\r\n# Generate\r\noutput = llm(\r\n    \"What is machine learning?\",\r\n    max_tokens=256,\r\n    temperature=0.7,\r\n    stop=[\"\u003C\u002Fs>\", \"\\n\\n\"]\r\n)\r\nprint(output[\"choices\"][0][\"text\"])\r\n```\r\n\r\n### Chat completion\r\n\r\n```python\r\nfrom llama_cpp import Llama\r\n\r\nllm = Llama(\r\n    model_path=\".\u002Fmodel-q4_k_m.gguf\",\r\n    n_ctx=4096,\r\n    n_gpu_layers=35,\r\n    chat_format=\"llama-3\"  # Or \"chatml\", \"mistral\", etc.\r\n)\r\n\r\nmessages = [\r\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\r\n    {\"role\": \"user\", \"content\": \"What is Python?\"}\r\n]\r\n\r\nresponse = llm.create_chat_completion(\r\n    messages=messages,\r\n    max_tokens=256,\r\n    temperature=0.7\r\n)\r\nprint(response[\"choices\"][0][\"message\"][\"content\"])\r\n```\r\n\r\n### Streaming\r\n\r\n```python\r\nfrom llama_cpp import Llama\r\n\r\nllm = Llama(model_path=\".\u002Fmodel-q4_k_m.gguf\", n_gpu_layers=35)\r\n\r\n# Stream tokens\r\nfor chunk in llm(\r\n    \"Explain quantum computing:\",\r\n    max_tokens=256,\r\n    stream=True\r\n):\r\n    print(chunk[\"choices\"][0][\"text\"], end=\"\", flush=True)\r\n```\r\n\r\n## Server mode\r\n\r\n### Start OpenAI-compatible server\r\n\r\n```bash\r\n# Start server\r\n.\u002Fllama-server -m model-q4_k_m.gguf \\\r\n    --host 0.0.0.0 \\\r\n    --port 8080 \\\r\n    -ngl 35 \\\r\n    -c 4096\r\n\r\n# Or with Python bindings\r\npython -m llama_cpp.server \\\r\n    --model model-q4_k_m.gguf \\\r\n    --n_gpu_layers 35 \\\r\n    --host 0.0.0.0 \\\r\n    --port 8080\r\n```\r\n\r\n### Use with OpenAI client\r\n\r\n```python\r\nfrom openai import OpenAI\r\n\r\nclient = OpenAI(\r\n    base_url=\"http:\u002F\u002Flocalhost:8080\u002Fv1\",\r\n    api_key=\"not-needed\"\r\n)\r\n\r\nresponse = client.chat.completions.create(\r\n    model=\"local-model\",\r\n    messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\r\n    max_tokens=256\r\n)\r\nprint(response.choices[0].message.content)\r\n```\r\n\r\n## Hardware optimization\r\n\r\n### Apple Silicon (Metal)\r\n\r\n```bash\r\n# Build with Metal\r\nmake clean && make GGML_METAL=1\r\n\r\n# Run with Metal acceleration\r\n.\u002Fllama-cli -m model.gguf -ngl 99 -p \"Hello\"\r\n\r\n# Python with Metal\r\nllm = Llama(\r\n    model_path=\"model.gguf\",\r\n    n_gpu_layers=99,     # Offload all layers\r\n    n_threads=1          # Metal handles parallelism\r\n)\r\n```\r\n\r\n### NVIDIA CUDA\r\n\r\n```bash\r\n# Build with CUDA\r\nmake clean && make GGML_CUDA=1\r\n\r\n# Run with CUDA\r\n.\u002Fllama-cli -m model.gguf -ngl 35 -p \"Hello\"\r\n\r\n# Specify GPU\r\nCUDA_VISIBLE_DEVICES=0 .\u002Fllama-cli -m model.gguf -ngl 35\r\n```\r\n\r\n### CPU optimization\r\n\r\n```bash\r\n# Build with AVX2\u002FAVX512\r\nmake clean && make\r\n\r\n# Run with optimal threads\r\n.\u002Fllama-cli -m model.gguf -t 8 -p \"Hello\"\r\n\r\n# Python CPU config\r\nllm = Llama(\r\n    model_path=\"model.gguf\",\r\n    n_gpu_layers=0,      # CPU only\r\n    n_threads=8,         # Match physical cores\r\n    n_batch=512          # Batch size for prompt processing\r\n)\r\n```\r\n\r\n## Integration with tools\r\n\r\n### Ollama\r\n\r\n```bash\r\n# Create Modelfile\r\ncat > Modelfile \u003C\u003C 'EOF'\r\nFROM .\u002Fmodel-q4_k_m.gguf\r\nTEMPLATE \"\"\"{{ .System }}\r\n{{ .Prompt }}\"\"\"\r\nPARAMETER temperature 0.7\r\nPARAMETER num_ctx 4096\r\nEOF\r\n\r\n# Create Ollama model\r\nollama create mymodel -f Modelfile\r\n\r\n# Run\r\nollama run mymodel \"Hello!\"\r\n```\r\n\r\n### LM Studio\r\n\r\n1. Place GGUF file in `~\u002F.cache\u002Flm-studio\u002Fmodels\u002F`\r\n2. Open LM Studio and select the model\r\n3. Configure context length and GPU offload\r\n4. Start inference\r\n\r\n### text-generation-webui\r\n\r\n```bash\r\n# Place in models folder\r\ncp model-q4_k_m.gguf text-generation-webui\u002Fmodels\u002F\r\n\r\n# Start with llama.cpp loader\r\npython server.py --model model-q4_k_m.gguf --loader llama.cpp --n-gpu-layers 35\r\n```\r\n\r\n## Best practices\r\n\r\n1. **Use K-quants**: Q4_K_M offers best quality\u002Fsize balance\r\n2. **Use imatrix**: Always use importance matrix for Q4 and below\r\n3. **GPU offload**: Offload as many layers as VRAM allows\r\n4. **Context length**: Start with 4096, increase if needed\r\n5. **Thread count**: Match physical CPU cores, not logical\r\n6. **Batch size**: Increase n_batch for faster prompt processing\r\n\r\n## Common issues\r\n\r\n**Model loads slowly:**\r\n```bash\r\n# Use mmap for faster loading\r\n.\u002Fllama-cli -m model.gguf --mmap\r\n```\r\n\r\n**Out of memory:**\r\n```bash\r\n# Reduce GPU layers\r\n.\u002Fllama-cli -m model.gguf -ngl 20  # Reduce from 35\r\n\r\n# Or use smaller quantization\r\n.\u002Fllama-quantize model-f16.gguf model-q3_k_m.gguf Q3_K_M\r\n```\r\n\r\n**Poor quality at low bits:**\r\n```bash\r\n# Always use imatrix for Q4 and below\r\n.\u002Fllama-imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix\r\n.\u002Fllama-quantize --imatrix model.imatrix model-f16.gguf model-q4_k_m.gguf Q4_K_M\r\n```\r\n\r\n## References\r\n\r\n- **[Advanced Usage](references\u002Fadvanced-usage.md)** - Batching, speculative decoding, custom builds\r\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, debugging, benchmarks\r\n\r\n## Resources\r\n\r\n- **Repository**: https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\r\n- **Python Bindings**: https:\u002F\u002Fgithub.com\u002Fabetlen\u002Fllama-cpp-python\r\n- **Pre-quantized Models**: https:\u002F\u002Fhuggingface.co\u002FTheBloke\r\n- **GGUF Converter**: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fggml-org\u002Fgguf-my-repo\r\n- **License**: MIT\r\n","# GGUF - Quantization Format for llama.cpp\r\n\r\nThe GGUF (GPT-Generated Unified Format) is the standard file format for llama.cpp, enabling efficient inference on CPUs, Apple Silicon, and GPUs with flexible quantization options.\r\n\r\n## When to use GGUF\r\n\r\n**Use GGUF when:**\r\n- Deploying on consumer hardware (laptops, desktops)\r\n- Running on Apple Silicon (M1\u002FM2\u002FM3) with Metal acceleration\r\n- Need CPU inference without GPU requirements\r\n- Want flexible quantization (Q2_K to Q8_0)\r\n- Using local AI tools (LM Studio, Ollama, text-generation-webui)\r\n\r\n**Key advantages:**\r\n- **Universal hardware**: CPU, Apple Silicon, NVIDIA, AMD support\r\n- **No Python runtime**: Pure C\u002FC++ inference\r\n- **Flexible quantization**: 2-8 bit with various methods (K-quants)\r\n- **Ecosystem support**: LM Studio, Ollama, koboldcpp, and more\r\n- **imatrix**: Importance matrix for better low-bit quality\r\n\r\n**Use alternatives instead:**\r\n- **AWQ\u002FGPTQ**: Maximum accuracy with calibration on NVIDIA GPUs\r\n- **HQQ**: Fast calibration-free quantization for HuggingFace\r\n- **bitsandbytes**: Simple integration with transformers library\r\n- **TensorRT-LLM**: Production NVIDIA deployment with maximum speed\r\n\r\n## Quick start\r\n\r\n### Installation\r\n\r\n```bash\r\n# Clone llama.cpp\r\ngit clone https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\r\ncd llama.cpp\r\n\r\n# Build (CPU)\r\nmake\r\n\r\n# Build with CUDA (NVIDIA)\r\nmake GGML_CUDA=1\r\n\r\n# Build with Metal (Apple Silicon)\r\nmake GGML_METAL=1\r\n\r\n# Install Python bindings (optional)\r\npip install llama-cpp-python\r\n```\r\n\r\n### Convert model to GGUF\r\n\r\n```bash\r\n# Install requirements\r\npip install -r requirements.txt\r\n\r\n# Convert HuggingFace model to GGUF (FP16)\r\npython convert_hf_to_gguf.py .\u002Fpath\u002Fto\u002Fmodel --outfile model-f16.gguf\r\n\r\n# Or specify output type\r\npython convert_hf_to_gguf.py .\u002Fpath\u002Fto\u002Fmodel \\\r\n    --outfile model-f16.gguf \\\r\n    --outtype f16\r\n```\r\n\r\n### Quantize model\r\n\r\n```bash\r\n# Basic quantization to Q4_K_M\r\n.\u002Fllama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M\r\n\r\n# Quantize with importance matrix (better quality)\r\n.\u002Fllama-imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix\r\n.\u002Fllama-quantize --imatrix model.imatrix model-f16.gguf model-q4_k_m.gguf Q4_K_M\r\n```\r\n\r\n### Run inference\r\n\r\n```bash\r\n# CLI inference\r\n.\u002Fllama-cli -m model-q4_k_m.gguf -p \"Hello, how are you?\"\r\n\r\n# Interactive mode\r\n.\u002Fllama-cli -m model-q4_k_m.gguf --interactive\r\n\r\n# With GPU offload\r\n.\u002Fllama-cli -m model-q4_k_m.gguf -ngl 35 -p \"Hello!\"\r\n```\r\n\r\n## Quantization types\r\n\r\n### K-quant methods (recommended)\r\n\r\n| Type | Bits | Size (7B) | Quality | Use Case |\r\n|------|------|-----------|---------|----------|\r\n| Q2_K | 2.5 | ~2.8 GB | Low | Extreme compression |\r\n| Q3_K_S | 3.0 | ~3.0 GB | Low-Med | Memory constrained |\r\n| Q3_K_M | 3.3 | ~3.3 GB | Medium | Balance |\r\n| Q4_K_S | 4.0 | ~3.8 GB | Med-High | Good balance |\r\n| Q4_K_M | 4.5 | ~4.1 GB | High | **Recommended default** |\r\n| Q5_K_S | 5.0 | ~4.6 GB | High | Quality focused |\r\n| Q5_K_M | 5.5 | ~4.8 GB | Very High | High quality |\r\n| Q6_K | 6.0 | ~5.5 GB | Excellent | Near-original |\r\n| Q8_0 | 8.0 | ~7.2 GB | Best | Maximum quality |\r\n\r\n### Legacy methods\r\n\r\n| Type | Description |\r\n|------|-------------|\r\n| Q4_0 | 4-bit, basic |\r\n| Q4_1 | 4-bit with delta |\r\n| Q5_0 | 5-bit, basic |\r\n| Q5_1 | 5-bit with delta |\r\n\r\n**Recommendation**: Use K-quant methods (Q4_K_M, Q5_K_M) for best quality\u002Fsize ratio.\r\n\r\n## Conversion workflows\r\n\r\n### Workflow 1: HuggingFace to GGUF\r\n\r\n```bash\r\n# 1. Download model\r\nhuggingface-cli download meta-llama\u002FLlama-3.1-8B --local-dir .\u002Fllama-3.1-8b\r\n\r\n# 2. Convert to GGUF (FP16)\r\npython convert_hf_to_gguf.py .\u002Fllama-3.1-8b \\\r\n    --outfile llama-3.1-8b-f16.gguf \\\r\n    --outtype f16\r\n\r\n# 3. Quantize\r\n.\u002Fllama-quantize llama-3.1-8b-f16.gguf llama-3.1-8b-q4_k_m.gguf Q4_K_M\r\n\r\n# 4. Test\r\n.\u002Fllama-cli -m llama-3.1-8b-q4_k_m.gguf -p \"Hello!\" -n 50\r\n```\r\n\r\n### Workflow 2: With importance matrix (better quality)\r\n\r\n```bash\r\n# 1. Convert to GGUF\r\npython convert_hf_to_gguf.py .\u002Fmodel --outfile model-f16.gguf\r\n\r\n# 2. Create calibration text (diverse samples)\r\ncat > calibration.txt \u003C\u003C 'EOF'\r\nThe quick brown fox jumps over the lazy dog.\r\nMachine learning is a subset of artificial intelligence.\r\nPython is a popular programming language.\r\n# Add more diverse text samples...\r\nEOF\r\n\r\n# 3. Generate importance matrix\r\n.\u002Fllama-imatrix -m model-f16.gguf \\\r\n    -f calibration.txt \\\r\n    --chunk 512 \\\r\n    -o model.imatrix \\\r\n    -ngl 35  # GPU layers if available\r\n\r\n# 4. Quantize with imatrix\r\n.\u002Fllama-quantize --imatrix model.imatrix \\\r\n    model-f16.gguf \\\r\n    model-q4_k_m.gguf \\\r\n    Q4_K_M\r\n```\r\n\r\n### Workflow 3: Multiple quantizations\r\n\r\n```bash\r\n#!\u002Fbin\u002Fbash\r\nMODEL=\"llama-3.1-8b-f16.gguf\"\r\nIMATRIX=\"llama-3.1-8b.imatrix\"\r\n\r\n# Generate imatrix once\r\n.\u002Fllama-imatrix -m $MODEL -f wiki.txt -o $IMATRIX -ngl 35\r\n\r\n# Create multiple quantizations\r\nfor QUANT in Q4_K_M Q5_K_M Q6_K Q8_0; do\r\n    OUTPUT=\"llama-3.1-8b-${QUANT,,}.gguf\"\r\n    .\u002Fllama-quantize --imatrix $IMATRIX $MODEL $OUTPUT $QUANT\r\n    echo \"Created: $OUTPUT ($(du -h $OUTPUT | cut -f1))\"\r\ndone\r\n```\r\n\r\n## Python usage\r\n\r\n### llama-cpp-python\r\n\r\n```python\r\nfrom llama_cpp import Llama\r\n\r\n# Load model\r\nllm = Llama(\r\n    model_path=\".\u002Fmodel-q4_k_m.gguf\",\r\n    n_ctx=4096,          # Context window\r\n    n_gpu_layers=35,     # GPU offload (0 for CPU only)\r\n    n_threads=8          # CPU threads\r\n)\r\n\r\n# Generate\r\noutput = llm(\r\n    \"What is machine learning?\",\r\n    max_tokens=256,\r\n    temperature=0.7,\r\n    stop=[\"\u003C\u002Fs>\", \"\\n\\n\"]\r\n)\r\nprint(output[\"choices\"][0][\"text\"])\r\n```\r\n\r\n### Chat completion\r\n\r\n```python\r\nfrom llama_cpp import Llama\r\n\r\nllm = Llama(\r\n    model_path=\".\u002Fmodel-q4_k_m.gguf\",\r\n    n_ctx=4096,\r\n    n_gpu_layers=35,\r\n    chat_format=\"llama-3\"  # Or \"chatml\", \"mistral\", etc.\r\n)\r\n\r\nmessages = [\r\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\r\n    {\"role\": \"user\", \"content\": \"What is Python?\"}\r\n]\r\n\r\nresponse = llm.create_chat_completion(\r\n    messages=messages,\r\n    max_tokens=256,\r\n    temperature=0.7\r\n)\r\nprint(response[\"choices\"][0][\"message\"][\"content\"])\r\n```\r\n\r\n### Streaming\r\n\r\n```python\r\nfrom llama_cpp import Llama\r\n\r\nllm = Llama(model_path=\".\u002Fmodel-q4_k_m.gguf\", n_gpu_layers=35)\r\n\r\n# Stream tokens\r\nfor chunk in llm(\r\n    \"Explain quantum computing:\",\r\n    max_tokens=256,\r\n    stream=True\r\n):\r\n    print(chunk[\"choices\"][0][\"text\"], end=\"\", flush=True)\r\n```\r\n\r\n## Server mode\r\n\r\n### Start OpenAI-compatible server\r\n\r\n```bash\r\n# Start server\r\n.\u002Fllama-server -m model-q4_k_m.gguf \\\r\n    --host 0.0.0.0 \\\r\n    --port 8080 \\\r\n    -ngl 35 \\\r\n    -c 4096\r\n\r\n# Or with Python bindings\r\npython -m llama_cpp.server \\\r\n    --model model-q4_k_m.gguf \\\r\n    --n_gpu_layers 35 \\\r\n    --host 0.0.0.0 \\\r\n    --port 8080\r\n```\r\n\r\n### Use with OpenAI client\r\n\r\n```python\r\nfrom openai import OpenAI\r\n\r\nclient = OpenAI(\r\n    base_url=\"http:\u002F\u002Flocalhost:8080\u002Fv1\",\r\n    api_key=\"not-needed\"\r\n)\r\n\r\nresponse = client.chat.completions.create(\r\n    model=\"local-model\",\r\n    messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\r\n    max_tokens=256\r\n)\r\nprint(response.choices[0].message.content)\r\n```\r\n\r\n## Hardware optimization\r\n\r\n### Apple Silicon (Metal)\r\n\r\n```bash\r\n# Build with Metal\r\nmake clean && make GGML_METAL=1\r\n\r\n# Run with Metal acceleration\r\n.\u002Fllama-cli -m model.gguf -ngl 99 -p \"Hello\"\r\n\r\n# Python with Metal\r\nllm = Llama(\r\n    model_path=\"model.gguf\",\r\n    n_gpu_layers=99,     # Offload all layers\r\n    n_threads=1          # Metal handles parallelism\r\n)\r\n```\r\n\r\n### NVIDIA CUDA\r\n\r\n```bash\r\n# Build with CUDA\r\nmake clean && make GGML_CUDA=1\r\n\r\n# Run with CUDA\r\n.\u002Fllama-cli -m model.gguf -ngl 35 -p \"Hello\"\r\n\r\n# Specify GPU\r\nCUDA_VISIBLE_DEVICES=0 .\u002Fllama-cli -m model.gguf -ngl 35\r\n```\r\n\r\n### CPU optimization\r\n\r\n```bash\r\n# Build with AVX2\u002FAVX512\r\nmake clean && make\r\n\r\n# Run with optimal threads\r\n.\u002Fllama-cli -m model.gguf -t 8 -p \"Hello\"\r\n\r\n# Python CPU config\r\nllm = Llama(\r\n    model_path=\"model.gguf\",\r\n    n_gpu_layers=0,      # CPU only\r\n    n_threads=8,         # Match physical cores\r\n    n_batch=512          # Batch size for prompt processing\r\n)\r\n```\r\n\r\n## Integration with tools\r\n\r\n### Ollama\r\n\r\n```bash\r\n# Create Modelfile\r\ncat > Modelfile \u003C\u003C 'EOF'\r\nFROM .\u002Fmodel-q4_k_m.gguf\r\nTEMPLATE \"\"\"{{ .System }}\r\n{{ .Prompt }}\"\"\"\r\nPARAMETER temperature 0.7\r\nPARAMETER num_ctx 4096\r\nEOF\r\n\r\n# Create Ollama model\r\nollama create mymodel -f Modelfile\r\n\r\n# Run\r\nollama run mymodel \"Hello!\"\r\n```\r\n\r\n### LM Studio\r\n\r\n1. Place GGUF file in `~\u002F.cache\u002Flm-studio\u002Fmodels\u002F`\r\n2. Open LM Studio and select the model\r\n3. Configure context length and GPU offload\r\n4. Start inference\r\n\r\n### text-generation-webui\r\n\r\n```bash\r\n# Place in models folder\r\ncp model-q4_k_m.gguf text-generation-webui\u002Fmodels\u002F\r\n\r\n# Start with llama.cpp loader\r\npython server.py --model model-q4_k_m.gguf --loader llama.cpp --n-gpu-layers 35\r\n```\r\n\r\n## Best practices\r\n\r\n1. **Use K-quants**: Q4_K_M offers best quality\u002Fsize balance\r\n2. **Use imatrix**: Always use importance matrix for Q4 and below\r\n3. **GPU offload**: Offload as many layers as VRAM allows\r\n4. **Context length**: Start with 4096, increase if needed\r\n5. **Thread count**: Match physical CPU cores, not logical\r\n6. **Batch size**: Increase n_batch for faster prompt processing\r\n\r\n## Common issues\r\n\r\n**Model loads slowly:**\r\n```bash\r\n# Use mmap for faster loading\r\n.\u002Fllama-cli -m model.gguf --mmap\r\n```\r\n\r\n**Out of memory:**\r\n```bash\r\n# Reduce GPU layers\r\n.\u002Fllama-cli -m model.gguf -ngl 20  # Reduce from 35\r\n\r\n# Or use smaller quantization\r\n.\u002Fllama-quantize model-f16.gguf model-q3_k_m.gguf Q3_K_M\r\n```\r\n\r\n**Poor quality at low bits:**\r\n```bash\r\n# Always use imatrix for Q4 and below\r\n.\u002Fllama-imatrix -m model-f16.gguf -f calibration.txt -o model.imatrix\r\n.\u002Fllama-quantize --imatrix model.imatrix model-f16.gguf model-q4_k_m.gguf Q4_K_M\r\n```\r\n\r\n## References\r\n\r\n- **[Advanced Usage](references\u002Fadvanced-usage.md)** - Batching, speculative decoding, custom builds\r\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, debugging, benchmarks\r\n\r\n## Resources\r\n\r\n- **Repository**: https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\r\n- **Python Bindings**: https:\u002F\u002Fgithub.com\u002Fabetlen\u002Fllama-cpp-python\r\n- **Pre-quantized Models**: https:\u002F\u002Fhuggingface.co\u002FTheBloke\r\n- **GGUF Converter**: https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fggml-org\u002Fgguf-my-repo\r\n- **License**: MIT\r\n",{"name":5,"author":44,"license":15,"version":45,"metadata":46,"description":7,"dependencies":50},"Orchestra Research","1.0.0",{"hermes":47},{"tags":48},[19,20,25,24,49,23,35],"Apple Silicon",[51],"llama-cpp-python>=0.2.0","main","0d1434f89f4bc6b07e9d272accd8c1013768820f","archive\u002Fskill-md-reviewed\u002Fsha256\u002F89\u002F8944bce3f696818423bd3433340b730b121a561c757ed749814e90d34680b81f",0,"https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills","https:\u002F\u002Fgithub.com\u002Fzapabob","The Open University of JAPAN　2019-now\r\n\r\nTokyo University of Science　2007-2010","2026-10-26 16:29:01",{"source":8,"skillName":5,"commands":61,"prompt":71,"zipAvailable":72},[62,65,68],{"tool":63,"command":64},"npx","npx skills add zapabob\u002FSkills --skill gguf-quantization",{"tool":66,"command":67},"bunx","bunx skills add zapabob\u002FSkills --skill gguf-quantization",{"tool":69,"command":70},"pnpm","pnpm dlx skills add zapabob\u002FSkills --skill gguf-quantization","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Farchive\u002Fskill-md-reviewed\u002Fsha256\u002F89\u002F8944bce3f696818423bd3433340b730b121a561c757ed749814e90d34680b81f\u002FSKILL.md",true,[],[75,113,148,173,200,217],{"skillNo":76,"name":77,"nameZh":6,"description":78,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":94,"publishedAt":39},"01M6GJTE1GESGGF9K028Y7753E","xurl","X\u002FTwitter via xurl CLI: post, search, DM, media, v2 API.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fxurl\u002Fvariants\u002Fhermes-agent\u002FSKILL.md",[81,86,89],{"categoryNo":82,"categoryName":83,"taxonomyType":84,"taxonomyTypeDesc":85},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",{"categoryNo":87,"categoryName":88,"taxonomyType":84,"taxonomyTypeDesc":85},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":90,"categoryName":91,"taxonomyType":92,"taxonomyTypeDesc":93},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[77,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,52,110,111,112],"openclaw","macos","cli","github","linux","windows","official","search","https","social-media","api","Media","hermes","hermes agent","twitter","social","via","post",{"skillNo":114,"name":115,"nameZh":6,"description":116,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":117,"categories":118,"keywords":130,"publishedAt":39},"01M6GJT8S1PXQ8WPXT1HSNBWH2","writing-plans","Write implementation plans: bite-sized tasks, paths, code.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fwriting-plans\u002Fvariants\u002Fhermes-agent\u002FSKILL.md",[119,122,125,126,127],{"categoryNo":120,"categoryName":121,"taxonomyType":84,"taxonomyTypeDesc":85},"01JBSEEDCAT000000000000015","视觉设计",{"categoryNo":123,"categoryName":124,"taxonomyType":84,"taxonomyTypeDesc":85},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":87,"categoryName":88,"taxonomyType":84,"taxonomyTypeDesc":85},{"categoryNo":90,"categoryName":91,"taxonomyType":92,"taxonomyTypeDesc":93},{"categoryNo":128,"categoryName":129,"taxonomyType":92,"taxonomyTypeDesc":93},"01JBSEEDCAT000000000000036","内容创作者",[131,132,133,134,96,135,99,136,100,137,138,139,140,141,142,143,144,145,146,147],"writing","writing plans","plans","design","review","workflow","documentation","tasks","development","code","planning","code-review","subagent","write","superpowers","paths","test-driven-development",{"skillNo":149,"name":150,"nameZh":6,"description":151,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":152,"categories":153,"keywords":157,"publishedAt":39},"01M6GJST0H5X7WHBJ3BZK0DVKH","whisper","OpenAI's general-purpose speech recognition model. Supports 99 languages, transcription, translation to English, and language identification. Six model sizes from tiny (39M params) to large (1550M params). Use for speech-to-text, podcast transcription, or multilingual audio processing. Best for robust, multilingual ASR.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fwhisper\u002Fvariants\u002Fhermes-agent-optional\u002FSKILL.md",[154],{"categoryNo":155,"categoryName":156,"taxonomyType":84,"taxonomyTypeDesc":85},"01JBSEEDCAT000000000000016","图像与多媒体",[150,96,99,158,159,160,161,162,163,164,33,165,32,166,167,168,169,170,171,172],"openai","speech","multimodal","Transformers","ASR","Audio","text","speech recognition","general","multilingual","translation","english","identification","processing","Audio Processing",{"skillNo":174,"name":175,"nameZh":6,"description":176,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":177,"categories":178,"keywords":182,"publishedAt":39},"01M6GJSMRQ8EETZTYMG72P70EA","weights-and-biases","W&B: log ML experiments, sweeps, model registry, dashboards.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fweights-and-biases\u002Fvariants\u002Fhermes-agent\u002FSKILL.md",[179],{"categoryNo":180,"categoryName":181,"taxonomyType":84,"taxonomyTypeDesc":85},"01JBSEEDCAT000000000000013","分析与可视化",[183,96,184,185,100,186,187,188,189,190,191,33,192,193,194,195,196,197,198,199],"weights and biases","tensorflow","experiments","pytorch","visualization","registry","real-time","tuning","dashboards","experiment","log","time","Experiment Tracking","Model Registry","Hyperparameter Tuning","WandB","Real-Time Visualization",{"skillNo":201,"name":202,"nameZh":6,"description":203,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":204,"categories":205,"keywords":207,"publishedAt":39},"01M6GJS19KDPRY112JTR326S06","watchers","Poll RSS, JSON APIs, and GitHub with watermark dedup.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fwatchers\u002Fvariants\u002Fhermes-agent-optional\u002FSKILL.md",[206],{"categoryNo":87,"categoryName":88,"taxonomyType":84,"taxonomyTypeDesc":85},[202,96,208,98,99,209,210,211,212,213,214,107,108,215,216],"terminal","automation","http","monitoring","cron","json","rss","devops","with",{"skillNo":218,"name":219,"nameZh":6,"description":220,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":221,"categories":222,"keywords":225,"publishedAt":39},"01M6GJRHJD2A6Q830SWF9W48BC","serving-llms-vllm","vLLM: high-throughput LLM serving, OpenAI API, quantization.","https:\u002F\u002Fgithub.com\u002Fzapabob\u002FSkills\u002Fblob\u002Fmain\u002Fregistry\u002Fskills\u002Fvllm\u002Fvariants\u002Fhermes-agent\u002FSKILL.md",[223,224],{"categoryNo":82,"categoryName":83,"taxonomyType":84,"taxonomyTypeDesc":85},{"categoryNo":90,"categoryName":91,"taxonomyType":92,"taxonomyTypeDesc":93},[226,227,228,99,38,158,229,230,231,161,232,105,233,234,235,236,20,237,238,239],"llms","vLLM","serving llms vllm","openai api","tensor","Inference Serving","Production","llm serving","High Throughput","PagedAttention","Continuous Batching","Tensor Parallelism","high-throughput","throughput"]