[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M74YR74WS32T87R8YTZ3WMXC":3,"skill-same:01M74YR74WS32T87R8YTZ3WMXC":68},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":30,"commitSha":31,"skillDirectory":32,"forksCount":33,"repositoryHtmlUrl":34,"repositoryHomepageUrl":35,"ownerHtmlUrl":36,"ownerBlogUrl":37,"ownerDescription":38,"updatedAt":39,"install":40,"resources":53},"01M74YR74WS32T87R8YTZ3WMXC","gptq",null,"Quantizes LLMs to 4-bit (also 3-bit) with GPTQ using group-wise quantization (group size 128 by default), via AutoGPTQ and transformers. Covers loading pre-quantized GPTQ models, quantizing your own model, choosing group size, selecting ExLlamaV2, Marlin, or Triton kernels, and QLoRA fine-tuning with PEFT. Use when fitting 70B-class models onto limited or consumer GPUs, cutting memory about 4x versus FP16, speeding up inference, finding pre-quantized checkpoints on HuggingFace, or fine-tuning a quantized model with LoRA. For slightly better accuracy on newer GPUs use AWQ instead, and for simple 8-bit or on-the-fly quantization use bitsandbytes.","KalarisLabs\u002Fresearch-agent-skills","KalarisLabs","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F300878568?v=4","01M74Y7MT23GXXC56VM6TVSXZY","Kalarislabs","01M74Y7MTQM9REXZX4SNXWX95C",132,"MIT","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fgptq\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: gptq\ndescription: Quantizes LLMs to 4-bit (also 3-bit) with GPTQ using group-wise quantization (group size 128 by default), via AutoGPTQ and transformers. Covers loading pre-quantized GPTQ models, quantizing your own model, choosing group size, selecting ExLlamaV2, Marlin, or Triton kernels, and QLoRA fine-tuning with PEFT. Use when fitting 70B-class models onto limited or consumer GPUs, cutting memory about 4x versus FP16, speeding up inference, finding pre-quantized checkpoints on HuggingFace, or fine-tuning a quantized model with LoRA. For slightly better accuracy on newer GPUs use AWQ instead, and for simple 8-bit or on-the-fly quantization use bitsandbytes.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-training\n  maintainer: Kalaris Labs\n  tags: Optimization, GPTQ, Quantization, 4-Bit, Post-Training, Memory Optimization, Consumer GPUs, Fast Inference, QLoRA, Group-Wise Quantization\n  dependencies: auto-gptq, transformers, optimum, peft\n---\n\n# GPTQ (Generative Pre-trained Transformer Quantization)\n\nPost-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization.\n\n## When to use GPTQ\n\n**Use GPTQ when:**\n- Need to fit large models (70B+) on limited GPU memory\n- Want 4× memory reduction with \u003C2% accuracy loss\n- Deploying on consumer GPUs (RTX 4090, 3090)\n- Need faster inference (3-4× speedup vs FP16)\n\n**Use AWQ instead when:**\n- Need slightly better accuracy (\u003C1% loss)\n- Have newer GPUs (Ampere, Ada)\n- Want Marlin kernel support (2× faster on some GPUs)\n\n**Use bitsandbytes instead when:**\n- Need simple integration with transformers\n- Want 8-bit quantization (less compression, better quality)\n- Don't need pre-quantized model files\n\n## Quick start\n\n### Installation\n\n```bash\n# Install AutoGPTQ\npip install auto-gptq\n\n# With Triton (Linux only, faster)\npip install auto-gptq[triton]\n\n# With CUDA extensions (faster)\npip install auto-gptq --no-build-isolation\n\n# Full installation\npip install auto-gptq transformers accelerate\n```\n\n### Load pre-quantized model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Load quantized model from HuggingFace\nmodel_name = \"TheBloke\u002FLlama-2-7B-Chat-GPTQ\"\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=False  # Set True on Linux for speed\n)\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Generate\nprompt = \"Explain quantum computing\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda:0\")\noutputs = model.generate(**inputs, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\n### Quantize your own model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel_name = \"meta-llama\u002FLlama-2-7b-chat-hf\"\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Quantization config\nquantize_config = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Group size (recommended: 128)\n    desc_act=False,      # Activation order (False for CUDA kernel)\n    damp_percent=0.01    # Dampening factor\n)\n\n# Load model for quantization\nmodel = AutoGPTQForCausalLM.from_pretrained(\n    model_name,\n    quantize_config=quantize_config\n)\n\n# Prepare calibration data\ndataset = load_dataset(\"c4\", split=\"train\", streaming=True)\ncalibration_data = [\n    tokenizer(example[\"text\"])[\"input_ids\"][:512]\n    for example in dataset.take(128)\n]\n\n# Quantize\nmodel.quantize(calibration_data)\n\n# Save quantized model\nmodel.save_quantized(\"llama-2-7b-gptq\")\ntokenizer.save_pretrained(\"llama-2-7b-gptq\")\n\n# Push to HuggingFace\nmodel.push_to_hub(\"username\u002Fllama-2-7b-gptq\")\n```\n\n## Group-wise quantization\n\n**How GPTQ works**:\n1. **Group weights**: Divide each weight matrix into groups (typically 128 elements)\n2. **Quantize per-group**: Each group has its own scale\u002Fzero-point\n3. **Minimize error**: Uses Hessian information to minimize quantization error\n4. **Result**: 4-bit weights with near-FP16 accuracy\n\n**Group size trade-off**:\n\n| Group Size | Model Size | Accuracy | Speed | Recommendation |\n|------------|------------|----------|-------|----------------|\n| -1 (per-column) | Smallest | Best | Slowest | Research only |\n| 32 | Smaller | Better | Slower | High accuracy needed |\n| **128** | Medium | Good | **Fast** | **Recommended default** |\n| 256 | Larger | Lower | Faster | Speed critical |\n| 1024 | Largest | Lowest | Fastest | Not recommended |\n\n**Example**:\n```\nWeight matrix: [1024, 4096] = 4.2M elements\n\nGroup size = 128:\n- Groups: 4.2M \u002F 128 = 32,768 groups\n- Each group: own 4-bit scale + zero-point\n- Result: Better granularity → better accuracy\n```\n\n## Quantization configurations\n\n### Standard 4-bit (recommended)\n\n```python\nfrom auto_gptq import BaseQuantizeConfig\n\nconfig = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Standard group size\n    desc_act=False,      # Faster CUDA kernel\n    damp_percent=0.01    # Dampening factor\n)\n```\n\n**Performance**:\n- Memory: 4× reduction (70B model: 140GB → 35GB)\n- Accuracy: ~1.5% perplexity increase\n- Speed: 3-4× faster than FP16\n\n### High accuracy (3-bit with larger groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=3,              # 3-bit (more compression)\n    group_size=128,      # Keep standard group size\n    desc_act=True,       # Better accuracy (slower)\n    damp_percent=0.01\n)\n```\n\n**Trade-off**:\n- Memory: 5× reduction\n- Accuracy: ~3% perplexity increase\n- Speed: 5× faster (but less accurate)\n\n### Maximum accuracy (4-bit with small groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=32,       # Smaller groups (better accuracy)\n    desc_act=True,       # Activation reordering\n    damp_percent=0.005   # Lower dampening\n)\n```\n\n**Trade-off**:\n- Memory: 3.5× reduction (slightly larger)\n- Accuracy: ~0.8% perplexity increase (best)\n- Speed: 2-3× faster (kernel overhead)\n\n## Kernel backends\n\n### ExLlamaV2 (default, fastest)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_exllama=True,      # Use ExLlamaV2\n    exllama_config={\"version\": 2}\n)\n```\n\n**Performance**: 1.5-2× faster than Triton\n\n### Marlin (Ampere+ GPUs)\n\n```python\n# Quantize with Marlin format\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=128,\n    desc_act=False  # Required for Marlin\n)\n\nmodel.quantize(calibration_data, use_marlin=True)\n\n# Load with Marlin\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_marlin=True  # 2× faster on A100\u002FH100\n)\n```\n\n**Requirements**:\n- NVIDIA Ampere or newer (A100, H100, RTX 40xx)\n- Compute capability ≥ 8.0\n\n### Triton (Linux only)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=True  # Linux only\n)\n```\n\n**Performance**: 1.2-1.5× faster than CUDA backend\n\n## Integration with transformers\n\n### Direct transformers usage\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load quantized model (transformers auto-detects GPTQ)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-13B-Chat-GPTQ\",\n    device_map=\"auto\",\n    trust_remote_code=False\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"TheBloke\u002FLlama-2-13B-Chat-GPTQ\")\n\n# Use like any transformers model\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_new_tokens=100)\n```\n\n### QLoRA fine-tuning (GPTQ + LoRA)\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model\n\n# Load GPTQ model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-7B-GPTQ\",\n    device_map=\"auto\"\n)\n\n# Prepare for LoRA training\nmodel = prepare_model_for_kbit_training(model)\n\n# LoRA config\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"v_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\n# Add LoRA adapters\nmodel = get_peft_model(model, lora_config)\n\n# Fine-tune (memory efficient!)\n# 70B model trainable on single A100 80GB\n```\n\n## Performance benchmarks\n\n### Memory reduction\n\n| Model | FP16 | GPTQ 4-bit | Reduction |\n|-------|------|------------|-----------|\n| Llama 2-7B | 14 GB | 3.5 GB | 4× |\n| Llama 2-13B | 26 GB | 6.5 GB | 4× |\n| Llama 2-70B | 140 GB | 35 GB | 4× |\n| Llama 3-405B | 810 GB | 203 GB | 4× |\n\n**Enables**:\n- 70B on single A100 80GB (vs 2× A100 needed for FP16)\n- 405B on 3× A100 80GB (vs 11× A100 needed for FP16)\n- 13B on RTX 4090 24GB (vs OOM with FP16)\n\n### Inference speed (Llama 2-7B, A100)\n\n| Precision | Tokens\u002Fsec | vs FP16 |\n|-----------|------------|---------|\n| FP16 | 25 tok\u002Fs | 1× |\n| GPTQ 4-bit (CUDA) | 85 tok\u002Fs | 3.4× |\n| GPTQ 4-bit (ExLlama) | 105 tok\u002Fs | 4.2× |\n| GPTQ 4-bit (Marlin) | 120 tok\u002Fs | 4.8× |\n\n### Accuracy (perplexity on WikiText-2)\n\n| Model | FP16 | GPTQ 4-bit (g=128) | Degradation |\n|-------|------|---------------------|-------------|\n| Llama 2-7B | 5.47 | 5.55 | +1.5% |\n| Llama 2-13B | 4.88 | 4.95 | +1.4% |\n| Llama 2-70B | 3.32 | 3.38 | +1.8% |\n\n**Excellent quality preservation** - less than 2% degradation!\n\n## Common patterns\n\n### Multi-GPU deployment\n\n```python\n# Automatic device mapping\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-GPTQ\",\n    device_map=\"auto\",  # Automatically split across GPUs\n    max_memory={0: \"40GB\", 1: \"40GB\"}  # Limit per GPU\n)\n\n# Manual device mapping\ndevice_map = {\n    \"model.embed_tokens\": 0,\n    \"model.layers.0-39\": 0,  # First 40 layers on GPU 0\n    \"model.layers.40-79\": 1,  # Last 40 layers on GPU 1\n    \"model.norm\": 1,\n    \"lm_head\": 1\n}\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device_map=device_map\n)\n```\n\n### CPU offloading\n\n```python\n# Offload some layers to CPU (for very large models)\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-405B-GPTQ\",\n    device_map=\"auto\",\n    max_memory={\n        0: \"80GB\",  # GPU 0\n        1: \"80GB\",  # GPU 1\n        2: \"80GB\",  # GPU 2\n        \"cpu\": \"200GB\"  # Offload overflow to CPU\n    }\n)\n```\n\n### Batch inference\n\n```python\n# Process multiple prompts efficiently\nprompts = [\n    \"Explain AI\",\n    \"Explain ML\",\n    \"Explain DL\"\n]\n\ninputs = tokenizer(prompts, return_tensors=\"pt\", padding=True).to(\"cuda\")\n\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=100,\n    pad_token_id=tokenizer.eos_token_id\n)\n\nfor i, output in enumerate(outputs):\n    print(f\"Prompt {i}: {tokenizer.decode(output)}\")\n```\n\n## Finding pre-quantized models\n\n**TheBloke on HuggingFace**:\n- https:\u002F\u002Fhuggingface.co\u002FTheBloke\n- 1000+ models in GPTQ format\n- Multiple group sizes (32, 128)\n- Both CUDA and Marlin formats\n\n**Search**:\n```bash\n# Find GPTQ models on HuggingFace\nhttps:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n```\n\n**Download**:\n```python\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Automatically downloads from HuggingFace\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-Chat-GPTQ\",\n    device=\"cuda:0\"\n)\n```\n\n## Supported models\n\n- **LLaMA family**: Llama 2, Llama 3, Code Llama\n- **Mistral**: Mistral 7B, Mixtral 8x7B, 8x22B\n- **Qwen**: Qwen, Qwen2, QwQ\n- **DeepSeek**: V2, V3\n- **Phi**: Phi-2, Phi-3\n- **Yi, Falcon, BLOOM, OPT**\n- **100+ models** on HuggingFace\n\n## References\n\n- **[Calibration Guide](references\u002Fcalibration.md)** - Dataset selection, quantization process, quality optimization\n- **[Integration Guide](references\u002Fintegration.md)** - Transformers, PEFT, vLLM, TensorRT-LLM\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, performance optimization\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FAutoGPTQ\u002FAutoGPTQ\n- **Paper**: GPTQ: Accurate Post-Training Quantization (arXiv:2210.17323)\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002Fautogptq\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `quantizing-models-bitsandbytes`: Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss.\n- `hqq-quantization`: Half-Quadratic Quantization for LLMs without calibration data.\n- `awq-quantization`: Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss.\n","# GPTQ (Generative Pre-trained Transformer Quantization)\n\nPost-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization.\n\n## When to use GPTQ\n\n**Use GPTQ when:**\n- Need to fit large models (70B+) on limited GPU memory\n- Want 4× memory reduction with \u003C2% accuracy loss\n- Deploying on consumer GPUs (RTX 4090, 3090)\n- Need faster inference (3-4× speedup vs FP16)\n\n**Use AWQ instead when:**\n- Need slightly better accuracy (\u003C1% loss)\n- Have newer GPUs (Ampere, Ada)\n- Want Marlin kernel support (2× faster on some GPUs)\n\n**Use bitsandbytes instead when:**\n- Need simple integration with transformers\n- Want 8-bit quantization (less compression, better quality)\n- Don't need pre-quantized model files\n\n## Quick start\n\n### Installation\n\n```bash\n# Install AutoGPTQ\npip install auto-gptq\n\n# With Triton (Linux only, faster)\npip install auto-gptq[triton]\n\n# With CUDA extensions (faster)\npip install auto-gptq --no-build-isolation\n\n# Full installation\npip install auto-gptq transformers accelerate\n```\n\n### Load pre-quantized model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Load quantized model from HuggingFace\nmodel_name = \"TheBloke\u002FLlama-2-7B-Chat-GPTQ\"\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=False  # Set True on Linux for speed\n)\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Generate\nprompt = \"Explain quantum computing\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda:0\")\noutputs = model.generate(**inputs, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\n### Quantize your own model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel_name = \"meta-llama\u002FLlama-2-7b-chat-hf\"\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Quantization config\nquantize_config = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Group size (recommended: 128)\n    desc_act=False,      # Activation order (False for CUDA kernel)\n    damp_percent=0.01    # Dampening factor\n)\n\n# Load model for quantization\nmodel = AutoGPTQForCausalLM.from_pretrained(\n    model_name,\n    quantize_config=quantize_config\n)\n\n# Prepare calibration data\ndataset = load_dataset(\"c4\", split=\"train\", streaming=True)\ncalibration_data = [\n    tokenizer(example[\"text\"])[\"input_ids\"][:512]\n    for example in dataset.take(128)\n]\n\n# Quantize\nmodel.quantize(calibration_data)\n\n# Save quantized model\nmodel.save_quantized(\"llama-2-7b-gptq\")\ntokenizer.save_pretrained(\"llama-2-7b-gptq\")\n\n# Push to HuggingFace\nmodel.push_to_hub(\"username\u002Fllama-2-7b-gptq\")\n```\n\n## Group-wise quantization\n\n**How GPTQ works**:\n1. **Group weights**: Divide each weight matrix into groups (typically 128 elements)\n2. **Quantize per-group**: Each group has its own scale\u002Fzero-point\n3. **Minimize error**: Uses Hessian information to minimize quantization error\n4. **Result**: 4-bit weights with near-FP16 accuracy\n\n**Group size trade-off**:\n\n| Group Size | Model Size | Accuracy | Speed | Recommendation |\n|------------|------------|----------|-------|----------------|\n| -1 (per-column) | Smallest | Best | Slowest | Research only |\n| 32 | Smaller | Better | Slower | High accuracy needed |\n| **128** | Medium | Good | **Fast** | **Recommended default** |\n| 256 | Larger | Lower | Faster | Speed critical |\n| 1024 | Largest | Lowest | Fastest | Not recommended |\n\n**Example**:\n```\nWeight matrix: [1024, 4096] = 4.2M elements\n\nGroup size = 128:\n- Groups: 4.2M \u002F 128 = 32,768 groups\n- Each group: own 4-bit scale + zero-point\n- Result: Better granularity → better accuracy\n```\n\n## Quantization configurations\n\n### Standard 4-bit (recommended)\n\n```python\nfrom auto_gptq import BaseQuantizeConfig\n\nconfig = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Standard group size\n    desc_act=False,      # Faster CUDA kernel\n    damp_percent=0.01    # Dampening factor\n)\n```\n\n**Performance**:\n- Memory: 4× reduction (70B model: 140GB → 35GB)\n- Accuracy: ~1.5% perplexity increase\n- Speed: 3-4× faster than FP16\n\n### High accuracy (3-bit with larger groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=3,              # 3-bit (more compression)\n    group_size=128,      # Keep standard group size\n    desc_act=True,       # Better accuracy (slower)\n    damp_percent=0.01\n)\n```\n\n**Trade-off**:\n- Memory: 5× reduction\n- Accuracy: ~3% perplexity increase\n- Speed: 5× faster (but less accurate)\n\n### Maximum accuracy (4-bit with small groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=32,       # Smaller groups (better accuracy)\n    desc_act=True,       # Activation reordering\n    damp_percent=0.005   # Lower dampening\n)\n```\n\n**Trade-off**:\n- Memory: 3.5× reduction (slightly larger)\n- Accuracy: ~0.8% perplexity increase (best)\n- Speed: 2-3× faster (kernel overhead)\n\n## Kernel backends\n\n### ExLlamaV2 (default, fastest)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_exllama=True,      # Use ExLlamaV2\n    exllama_config={\"version\": 2}\n)\n```\n\n**Performance**: 1.5-2× faster than Triton\n\n### Marlin (Ampere+ GPUs)\n\n```python\n# Quantize with Marlin format\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=128,\n    desc_act=False  # Required for Marlin\n)\n\nmodel.quantize(calibration_data, use_marlin=True)\n\n# Load with Marlin\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_marlin=True  # 2× faster on A100\u002FH100\n)\n```\n\n**Requirements**:\n- NVIDIA Ampere or newer (A100, H100, RTX 40xx)\n- Compute capability ≥ 8.0\n\n### Triton (Linux only)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=True  # Linux only\n)\n```\n\n**Performance**: 1.2-1.5× faster than CUDA backend\n\n## Integration with transformers\n\n### Direct transformers usage\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load quantized model (transformers auto-detects GPTQ)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-13B-Chat-GPTQ\",\n    device_map=\"auto\",\n    trust_remote_code=False\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"TheBloke\u002FLlama-2-13B-Chat-GPTQ\")\n\n# Use like any transformers model\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_new_tokens=100)\n```\n\n### QLoRA fine-tuning (GPTQ + LoRA)\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model\n\n# Load GPTQ model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-7B-GPTQ\",\n    device_map=\"auto\"\n)\n\n# Prepare for LoRA training\nmodel = prepare_model_for_kbit_training(model)\n\n# LoRA config\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"v_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\n# Add LoRA adapters\nmodel = get_peft_model(model, lora_config)\n\n# Fine-tune (memory efficient!)\n# 70B model trainable on single A100 80GB\n```\n\n## Performance benchmarks\n\n### Memory reduction\n\n| Model | FP16 | GPTQ 4-bit | Reduction |\n|-------|------|------------|-----------|\n| Llama 2-7B | 14 GB | 3.5 GB | 4× |\n| Llama 2-13B | 26 GB | 6.5 GB | 4× |\n| Llama 2-70B | 140 GB | 35 GB | 4× |\n| Llama 3-405B | 810 GB | 203 GB | 4× |\n\n**Enables**:\n- 70B on single A100 80GB (vs 2× A100 needed for FP16)\n- 405B on 3× A100 80GB (vs 11× A100 needed for FP16)\n- 13B on RTX 4090 24GB (vs OOM with FP16)\n\n### Inference speed (Llama 2-7B, A100)\n\n| Precision | Tokens\u002Fsec | vs FP16 |\n|-----------|------------|---------|\n| FP16 | 25 tok\u002Fs | 1× |\n| GPTQ 4-bit (CUDA) | 85 tok\u002Fs | 3.4× |\n| GPTQ 4-bit (ExLlama) | 105 tok\u002Fs | 4.2× |\n| GPTQ 4-bit (Marlin) | 120 tok\u002Fs | 4.8× |\n\n### Accuracy (perplexity on WikiText-2)\n\n| Model | FP16 | GPTQ 4-bit (g=128) | Degradation |\n|-------|------|---------------------|-------------|\n| Llama 2-7B | 5.47 | 5.55 | +1.5% |\n| Llama 2-13B | 4.88 | 4.95 | +1.4% |\n| Llama 2-70B | 3.32 | 3.38 | +1.8% |\n\n**Excellent quality preservation** - less than 2% degradation!\n\n## Common patterns\n\n### Multi-GPU deployment\n\n```python\n# Automatic device mapping\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-GPTQ\",\n    device_map=\"auto\",  # Automatically split across GPUs\n    max_memory={0: \"40GB\", 1: \"40GB\"}  # Limit per GPU\n)\n\n# Manual device mapping\ndevice_map = {\n    \"model.embed_tokens\": 0,\n    \"model.layers.0-39\": 0,  # First 40 layers on GPU 0\n    \"model.layers.40-79\": 1,  # Last 40 layers on GPU 1\n    \"model.norm\": 1,\n    \"lm_head\": 1\n}\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device_map=device_map\n)\n```\n\n### CPU offloading\n\n```python\n# Offload some layers to CPU (for very large models)\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-405B-GPTQ\",\n    device_map=\"auto\",\n    max_memory={\n        0: \"80GB\",  # GPU 0\n        1: \"80GB\",  # GPU 1\n        2: \"80GB\",  # GPU 2\n        \"cpu\": \"200GB\"  # Offload overflow to CPU\n    }\n)\n```\n\n### Batch inference\n\n```python\n# Process multiple prompts efficiently\nprompts = [\n    \"Explain AI\",\n    \"Explain ML\",\n    \"Explain DL\"\n]\n\ninputs = tokenizer(prompts, return_tensors=\"pt\", padding=True).to(\"cuda\")\n\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=100,\n    pad_token_id=tokenizer.eos_token_id\n)\n\nfor i, output in enumerate(outputs):\n    print(f\"Prompt {i}: {tokenizer.decode(output)}\")\n```\n\n## Finding pre-quantized models\n\n**TheBloke on HuggingFace**:\n- https:\u002F\u002Fhuggingface.co\u002FTheBloke\n- 1000+ models in GPTQ format\n- Multiple group sizes (32, 128)\n- Both CUDA and Marlin formats\n\n**Search**:\n```bash\n# Find GPTQ models on HuggingFace\nhttps:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n```\n\n**Download**:\n```python\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Automatically downloads from HuggingFace\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-Chat-GPTQ\",\n    device=\"cuda:0\"\n)\n```\n\n## Supported models\n\n- **LLaMA family**: Llama 2, Llama 3, Code Llama\n- **Mistral**: Mistral 7B, Mixtral 8x7B, 8x22B\n- **Qwen**: Qwen, Qwen2, QwQ\n- **DeepSeek**: V2, V3\n- **Phi**: Phi-2, Phi-3\n- **Yi, Falcon, BLOOM, OPT**\n- **100+ models** on HuggingFace\n\n## References\n\n- **[Calibration Guide](references\u002Fcalibration.md)** - Dataset selection, quantization process, quality optimization\n- **[Integration Guide](references\u002Fintegration.md)** - Transformers, PEFT, vLLM, TensorRT-LLM\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, performance optimization\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FAutoGPTQ\u002FAutoGPTQ\n- **Paper**: GPTQ: Accurate Post-Training Quantization (arXiv:2210.17323)\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002Fautogptq\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `quantizing-models-bitsandbytes`: Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss.\n- `hqq-quantization`: Half-Quadratic Quantization for LLMs without calibration data.\n- `awq-quantization`: Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss.\n",{"name":5,"license":15,"metadata":24,"description":7},{"tags":25,"version":26,"category":27,"maintainer":28,"dependencies":29},"Optimization, GPTQ, Quantization, 4-Bit, Post-Training, Memory Optimization, Consumer GPUs, Fast Inference, QLoRA, Group-Wise Quantization","1.0.0","ml-training","Kalaris Labs","auto-gptq, transformers, optimum, peft","main","29fe36d26127205057d5e0ed8425f6b0e9a8c13a","skills\u002Fgptq",16,"https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills","https:\u002F\u002Fdocs.kalarislabs.com\u002Fresearch-agent-skills","https:\u002F\u002Fgithub.com\u002FKalarisLabs","kalarislabs.com","Applied AI lab building the infrastructure for the agentic era","2026-11-14 17:49:08",{"source":8,"skillName":5,"commands":41,"prompt":51,"zipAvailable":52},[42,45,48],{"tool":43,"command":44},"npx","npx skills add KalarisLabs\u002Fresearch-agent-skills --skill gptq",{"tool":46,"command":47},"bunx","bunx skills add KalarisLabs\u002Fresearch-agent-skills --skill gptq",{"tool":49,"command":50},"pnpm","pnpm dlx skills add KalarisLabs\u002Fresearch-agent-skills --skill gptq","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fgptq\u002FSKILL.md",true,[54,60,64],{"path":55,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":58,"sourceHtmlUrl":59},"references\u002Fcalibration.md","REFERENCE","参考文档",8164,"https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fgptq\u002Freferences\u002Fcalibration.md",{"path":61,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":62,"sourceHtmlUrl":63},"references\u002Fintegration.md",2797,"https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fgptq\u002Freferences\u002Fintegration.md",{"path":65,"resourceType":56,"resourceTypeDesc":57,"sizeBytes":66,"sourceHtmlUrl":67},"references\u002Ftroubleshooting.md",1899,"https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fgptq\u002Freferences\u002Ftroubleshooting.md",[69,76,83,90,97,104],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":19},"01M74Z8MGT305R11SNY5G28QH5","plos","Prepare manuscripts for PLOS journals (PLOS ONE, PLOS Biology, PLOS Computational Biology, PLOS Genetics, PLOS Medicine, PLOS Pathogens, PLOS Neglected Tropical Diseases, PLOS Global Public Health and others), covering structure, abstract and author summary, Vancouver-style references, the mandatory data availability policy, ethics and financial disclosure statements, figure requirements (PACE), the PLOS LaTeX template, reporting guidelines and PLOS ONE's publication criteria. Use when targeting any PLOS journal or checking compliance with PLOS policies.","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fplos\u002FSKILL.md",[],[],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":19},"01M74Z8CCN9APXYHNQ95V1EBGT","pkpd-modeling","Pharmacokinetic and pharmacodynamic modelling and simulation - non-compartmental analysis, compartmental and population PK, PK\u002FPD and exposure-response, TMDD, PBPK orientation, bioequivalence, allometric scaling and first-in-human dose, drug interaction prediction, and Bayesian therapeutic drug monitoring. Use when analysing concentration-time data, deriving exposure metrics, fitting PK or PD models, or evaluating dosing regimens. Triggers include \"pharmacokinetics\", \"pharmacodynamics\", \"PK\u002FPD\", \"NCA\", \"non-compartmental\", \"AUC\", \"Cmax\", \"lambda z\", \"half-life\", \"clearance\", \"volume of distribution\", \"compartmental model\", \"population PK\", \"popPK\", \"NONMEM\", \"nlmixr2\", \"Pharmpy\", \"Monolix\", \"exposure-response\", \"Emax\", \"EC50\", \"indirect response\", \"effect compartment\", \"TMDD\", \"PBPK\", \"bioequivalence\", \"RSABE\", \"ABEL\", \"allometric scaling\", \"first-in-human\", \"MABEL\", \"drug-drug interaction\", \"DDI\", \"ICH M12\", \"concentration-QTc\", \"therapeutic drug monitoring\", \"MIPD\", and \"dosing regimen\".","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fpkpd-modeling\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":19},"01M74Z86YHJJ1FM3X87BHN76KZ","pinecone","Guides use of Pinecone, a managed serverless vector database, through its Python client and the LangChain and LlamaIndex integrations. Covers creating indexes, upserting and querying vectors, metadata filtering, namespaces, hybrid dense and sparse search, index management, and deleting vectors. Use when building a production RAG system on a hosted vector store, adding semantic search or recommendations without running infrastructure, isolating per-user or per-tenant data with namespaces, combining dense and sparse vectors in one query, or filtering results by metadata. Do not use for self-hosted or local stores (Chroma, Weaviate) or offline similarity search (FAISS).","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fpinecone\u002FSKILL.md",[],[],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":94,"categories":95,"keywords":96,"publishedAt":19},"01M74Z80T6E89SAJ8MYDD0767Q","pi-agent","Build with and use Pi, the minimal terminal coding harness. Use for installing Pi, configuring providers\u002Fmodels\u002Fsettings\u002Fenvironment variables, creating Pi skills\u002Fextensions\u002Fpackages\u002Fthemes\u002Fprompt templates, embedding Pi through the SDK, integrating over RPC or JSON event streams, parsing sessions, running local models through the llama.cpp router, developing custom Pi providers and TUI components, or using ecosystem packages such as pi-subagents (delegation\u002Forchestration), pi-mcp-adapter (MCP servers), pi-interview (interactive forms), and pi-web-access (web search, fetching, video understanding).","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fpi-agent\u002FSKILL.md",[],[],{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":19},"01M74Z7WF1Y8Q3VFAQPPPRAFMV","phoenix-observability","Open-source AI observability platform for LLM tracing, evaluation, and monitoring. Use when debugging LLM applications with detailed traces, running evaluations on datasets, or monitoring production AI systems with real-time insights.","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fphoenix-observability\u002FSKILL.md",[],[],{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":19},"01M74Z7HNEH1A2AVYR3G1X2QV6","peft-fine-tuning","Fine-tunes LLMs with Hugging Face PEFT, using LoRA, QLoRA, IA3, AdaLoRA, prefix tuning, and prompt tuning so that under 1% of parameters are trained. Covers rank, alpha, and target module selection, loading and merging adapters, multi-adapter serving, and integration with TRL SFTTrainer, Axolotl, and vLLM. Use when fine-tuning 7B-70B models on limited GPU memory, when running QLoRA on a single 24GB GPU, when choosing LoRA rank and alpha, when merging or swapping adapters on one base model, or when debugging CUDA OOM or an adapter that does not apply. Not for full fine-tuning of small models under 1B parameters or cases needing all weights updated.","https:\u002F\u002Fgithub.com\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fpeft-fine-tuning\u002FSKILL.md",[],[]]