[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6QERVSJYJSGGDQVPKNSXATK":3,"skill-same:01M6QERVSJYJSGGDQVPKNSXATK":87},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":50,"commitSha":51,"skillDirectory":52,"forksCount":14,"repositoryHtmlUrl":53,"repositoryHomepageUrl":54,"ownerHtmlUrl":55,"ownerBlogUrl":56,"ownerDescription":57,"updatedAt":58,"install":59,"resources":72},"01M6QERVSJYJSGGDQVPKNSXATK","gptq",null,"Quantizes LLMs to 4-bit (also 3-bit) with GPTQ using group-wise quantization (group size 128 by default), via AutoGPTQ and transformers. Covers loading pre-quantized GPTQ models, quantizing your own model, choosing group size, selecting ExLlamaV2, Marlin, or Triton kernels, and QLoRA fine-tuning with PEFT. Use when fitting 70B-class models onto limited or consumer GPUs, cutting memory about 4x versus FP16, speeding up inference, finding pre-quantized checkpoints on HuggingFace, or fine-tuning a quantized model with LoRA. For slightly better accuracy on newer GPUs use AWQ instead, and for simple 8-bit or on-the-fly quantization use bitsandbytes.","Tashima-Tarsh\u002FDisha6.6","Tashima-Tarsh","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F269906498?v=4","01M6PBH98X03XERH0HFT4PFB1P","NITISH KUMAR","01M6PBH9DA0M6KM3Z12P848F0K",0,"MIT","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-gptq\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"GPTQ","AWQ","accuracy","for","kernels","triton","8-Bit","Bitsandbytes","Memory Optimization","4-Bit","training","bit","auto","use","fly","Post-Training","HuggingFace","QLoRA","LoRA","Fine-Tuning","2026-11-10 10:12:30",false,"---\nname: gptq\ndescription: Quantizes LLMs to 4-bit (also 3-bit) with GPTQ using group-wise quantization (group size 128 by default), via AutoGPTQ and transformers. Covers loading pre-quantized GPTQ models, quantizing your own model, choosing group size, selecting ExLlamaV2, Marlin, or Triton kernels, and QLoRA fine-tuning with PEFT. Use when fitting 70B-class models onto limited or consumer GPUs, cutting memory about 4x versus FP16, speeding up inference, finding pre-quantized checkpoints on HuggingFace, or fine-tuning a quantized model with LoRA. For slightly better accuracy on newer GPUs use AWQ instead, and for simple 8-bit or on-the-fly quantization use bitsandbytes.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-training\n  maintainer: Kalaris Labs\n  tags: Optimization, GPTQ, Quantization, 4-Bit, Post-Training, Memory Optimization, Consumer GPUs, Fast Inference, QLoRA, Group-Wise Quantization\n  dependencies: auto-gptq, transformers, optimum, peft\n---\n\n# GPTQ (Generative Pre-trained Transformer Quantization)\n\nPost-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization.\n\n## When to use GPTQ\n\n**Use GPTQ when:**\n- Need to fit large models (70B+) on limited GPU memory\n- Want 4× memory reduction with \u003C2% accuracy loss\n- Deploying on consumer GPUs (RTX 4090, 3090)\n- Need faster inference (3-4× speedup vs FP16)\n\n**Use AWQ instead when:**\n- Need slightly better accuracy (\u003C1% loss)\n- Have newer GPUs (Ampere, Ada)\n- Want Marlin kernel support (2× faster on some GPUs)\n\n**Use bitsandbytes instead when:**\n- Need simple integration with transformers\n- Want 8-bit quantization (less compression, better quality)\n- Don't need pre-quantized model files\n\n## Quick start\n\n### Installation\n\n```bash\n# Install AutoGPTQ\npip install auto-gptq\n\n# With Triton (Linux only, faster)\npip install auto-gptq[triton]\n\n# With CUDA extensions (faster)\npip install auto-gptq --no-build-isolation\n\n# Full installation\npip install auto-gptq transformers accelerate\n```\n\n### Load pre-quantized model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Load quantized model from HuggingFace\nmodel_name = \"TheBloke\u002FLlama-2-7B-Chat-GPTQ\"\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=False  # Set True on Linux for speed\n)\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Generate\nprompt = \"Explain quantum computing\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda:0\")\noutputs = model.generate(**inputs, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\n### Quantize your own model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel_name = \"meta-llama\u002FLlama-2-7b-chat-hf\"\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Quantization config\nquantize_config = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Group size (recommended: 128)\n    desc_act=False,      # Activation order (False for CUDA kernel)\n    damp_percent=0.01    # Dampening factor\n)\n\n# Load model for quantization\nmodel = AutoGPTQForCausalLM.from_pretrained(\n    model_name,\n    quantize_config=quantize_config\n)\n\n# Prepare calibration data\ndataset = load_dataset(\"c4\", split=\"train\", streaming=True)\ncalibration_data = [\n    tokenizer(example[\"text\"])[\"input_ids\"][:512]\n    for example in dataset.take(128)\n]\n\n# Quantize\nmodel.quantize(calibration_data)\n\n# Save quantized model\nmodel.save_quantized(\"llama-2-7b-gptq\")\ntokenizer.save_pretrained(\"llama-2-7b-gptq\")\n\n# Push to HuggingFace\nmodel.push_to_hub(\"username\u002Fllama-2-7b-gptq\")\n```\n\n## Group-wise quantization\n\n**How GPTQ works**:\n1. **Group weights**: Divide each weight matrix into groups (typically 128 elements)\n2. **Quantize per-group**: Each group has its own scale\u002Fzero-point\n3. **Minimize error**: Uses Hessian information to minimize quantization error\n4. **Result**: 4-bit weights with near-FP16 accuracy\n\n**Group size trade-off**:\n\n| Group Size | Model Size | Accuracy | Speed | Recommendation |\n|------------|------------|----------|-------|----------------|\n| -1 (per-column) | Smallest | Best | Slowest | Research only |\n| 32 | Smaller | Better | Slower | High accuracy needed |\n| **128** | Medium | Good | **Fast** | **Recommended default** |\n| 256 | Larger | Lower | Faster | Speed critical |\n| 1024 | Largest | Lowest | Fastest | Not recommended |\n\n**Example**:\n```\nWeight matrix: [1024, 4096] = 4.2M elements\n\nGroup size = 128:\n- Groups: 4.2M \u002F 128 = 32,768 groups\n- Each group: own 4-bit scale + zero-point\n- Result: Better granularity → better accuracy\n```\n\n## Quantization configurations\n\n### Standard 4-bit (recommended)\n\n```python\nfrom auto_gptq import BaseQuantizeConfig\n\nconfig = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Standard group size\n    desc_act=False,      # Faster CUDA kernel\n    damp_percent=0.01    # Dampening factor\n)\n```\n\n**Performance**:\n- Memory: 4× reduction (70B model: 140GB → 35GB)\n- Accuracy: ~1.5% perplexity increase\n- Speed: 3-4× faster than FP16\n\n### High accuracy (3-bit with larger groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=3,              # 3-bit (more compression)\n    group_size=128,      # Keep standard group size\n    desc_act=True,       # Better accuracy (slower)\n    damp_percent=0.01\n)\n```\n\n**Trade-off**:\n- Memory: 5× reduction\n- Accuracy: ~3% perplexity increase\n- Speed: 5× faster (but less accurate)\n\n### Maximum accuracy (4-bit with small groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=32,       # Smaller groups (better accuracy)\n    desc_act=True,       # Activation reordering\n    damp_percent=0.005   # Lower dampening\n)\n```\n\n**Trade-off**:\n- Memory: 3.5× reduction (slightly larger)\n- Accuracy: ~0.8% perplexity increase (best)\n- Speed: 2-3× faster (kernel overhead)\n\n## Kernel backends\n\n### ExLlamaV2 (default, fastest)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_exllama=True,      # Use ExLlamaV2\n    exllama_config={\"version\": 2}\n)\n```\n\n**Performance**: 1.5-2× faster than Triton\n\n### Marlin (Ampere+ GPUs)\n\n```python\n# Quantize with Marlin format\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=128,\n    desc_act=False  # Required for Marlin\n)\n\nmodel.quantize(calibration_data, use_marlin=True)\n\n# Load with Marlin\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_marlin=True  # 2× faster on A100\u002FH100\n)\n```\n\n**Requirements**:\n- NVIDIA Ampere or newer (A100, H100, RTX 40xx)\n- Compute capability ≥ 8.0\n\n### Triton (Linux only)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=True  # Linux only\n)\n```\n\n**Performance**: 1.2-1.5× faster than CUDA backend\n\n## Integration with transformers\n\n### Direct transformers usage\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load quantized model (transformers auto-detects GPTQ)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-13B-Chat-GPTQ\",\n    device_map=\"auto\",\n    trust_remote_code=False\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"TheBloke\u002FLlama-2-13B-Chat-GPTQ\")\n\n# Use like any transformers model\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_new_tokens=100)\n```\n\n### QLoRA fine-tuning (GPTQ + LoRA)\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model\n\n# Load GPTQ model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-7B-GPTQ\",\n    device_map=\"auto\"\n)\n\n# Prepare for LoRA training\nmodel = prepare_model_for_kbit_training(model)\n\n# LoRA config\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"v_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\n# Add LoRA adapters\nmodel = get_peft_model(model, lora_config)\n\n# Fine-tune (memory efficient!)\n# 70B model trainable on single A100 80GB\n```\n\n## Performance benchmarks\n\n### Memory reduction\n\n| Model | FP16 | GPTQ 4-bit | Reduction |\n|-------|------|------------|-----------|\n| Llama 2-7B | 14 GB | 3.5 GB | 4× |\n| Llama 2-13B | 26 GB | 6.5 GB | 4× |\n| Llama 2-70B | 140 GB | 35 GB | 4× |\n| Llama 3-405B | 810 GB | 203 GB | 4× |\n\n**Enables**:\n- 70B on single A100 80GB (vs 2× A100 needed for FP16)\n- 405B on 3× A100 80GB (vs 11× A100 needed for FP16)\n- 13B on RTX 4090 24GB (vs OOM with FP16)\n\n### Inference speed (Llama 2-7B, A100)\n\n| Precision | Tokens\u002Fsec | vs FP16 |\n|-----------|------------|---------|\n| FP16 | 25 tok\u002Fs | 1× |\n| GPTQ 4-bit (CUDA) | 85 tok\u002Fs | 3.4× |\n| GPTQ 4-bit (ExLlama) | 105 tok\u002Fs | 4.2× |\n| GPTQ 4-bit (Marlin) | 120 tok\u002Fs | 4.8× |\n\n### Accuracy (perplexity on WikiText-2)\n\n| Model | FP16 | GPTQ 4-bit (g=128) | Degradation |\n|-------|------|---------------------|-------------|\n| Llama 2-7B | 5.47 | 5.55 | +1.5% |\n| Llama 2-13B | 4.88 | 4.95 | +1.4% |\n| Llama 2-70B | 3.32 | 3.38 | +1.8% |\n\n**Excellent quality preservation** - less than 2% degradation!\n\n## Common patterns\n\n### Multi-GPU deployment\n\n```python\n# Automatic device mapping\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-GPTQ\",\n    device_map=\"auto\",  # Automatically split across GPUs\n    max_memory={0: \"40GB\", 1: \"40GB\"}  # Limit per GPU\n)\n\n# Manual device mapping\ndevice_map = {\n    \"model.embed_tokens\": 0,\n    \"model.layers.0-39\": 0,  # First 40 layers on GPU 0\n    \"model.layers.40-79\": 1,  # Last 40 layers on GPU 1\n    \"model.norm\": 1,\n    \"lm_head\": 1\n}\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device_map=device_map\n)\n```\n\n### CPU offloading\n\n```python\n# Offload some layers to CPU (for very large models)\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-405B-GPTQ\",\n    device_map=\"auto\",\n    max_memory={\n        0: \"80GB\",  # GPU 0\n        1: \"80GB\",  # GPU 1\n        2: \"80GB\",  # GPU 2\n        \"cpu\": \"200GB\"  # Offload overflow to CPU\n    }\n)\n```\n\n### Batch inference\n\n```python\n# Process multiple prompts efficiently\nprompts = [\n    \"Explain AI\",\n    \"Explain ML\",\n    \"Explain DL\"\n]\n\ninputs = tokenizer(prompts, return_tensors=\"pt\", padding=True).to(\"cuda\")\n\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=100,\n    pad_token_id=tokenizer.eos_token_id\n)\n\nfor i, output in enumerate(outputs):\n    print(f\"Prompt {i}: {tokenizer.decode(output)}\")\n```\n\n## Finding pre-quantized models\n\n**TheBloke on HuggingFace**:\n- https:\u002F\u002Fhuggingface.co\u002FTheBloke\n- 1000+ models in GPTQ format\n- Multiple group sizes (32, 128)\n- Both CUDA and Marlin formats\n\n**Search**:\n```bash\n# Find GPTQ models on HuggingFace\nhttps:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n```\n\n**Download**:\n```python\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Automatically downloads from HuggingFace\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-Chat-GPTQ\",\n    device=\"cuda:0\"\n)\n```\n\n## Supported models\n\n- **LLaMA family**: Llama 2, Llama 3, Code Llama\n- **Mistral**: Mistral 7B, Mixtral 8x7B, 8x22B\n- **Qwen**: Qwen, Qwen2, QwQ\n- **DeepSeek**: V2, V3\n- **Phi**: Phi-2, Phi-3\n- **Yi, Falcon, BLOOM, OPT**\n- **100+ models** on HuggingFace\n\n## References\n\n- **[Calibration Guide](references\u002Fcalibration.md)** - Dataset selection, quantization process, quality optimization\n- **[Integration Guide](references\u002Fintegration.md)** - Transformers, PEFT, vLLM, TensorRT-LLM\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, performance optimization\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FAutoGPTQ\u002FAutoGPTQ\n- **Paper**: GPTQ: Accurate Post-Training Quantization (arXiv:2210.17323)\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002Fautogptq\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `quantizing-models-bitsandbytes`: Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss.\n- `hqq-quantization`: Half-Quadratic Quantization for LLMs without calibration data.\n- `awq-quantization`: Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss.\n","# GPTQ (Generative Pre-trained Transformer Quantization)\n\nPost-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization.\n\n## When to use GPTQ\n\n**Use GPTQ when:**\n- Need to fit large models (70B+) on limited GPU memory\n- Want 4× memory reduction with \u003C2% accuracy loss\n- Deploying on consumer GPUs (RTX 4090, 3090)\n- Need faster inference (3-4× speedup vs FP16)\n\n**Use AWQ instead when:**\n- Need slightly better accuracy (\u003C1% loss)\n- Have newer GPUs (Ampere, Ada)\n- Want Marlin kernel support (2× faster on some GPUs)\n\n**Use bitsandbytes instead when:**\n- Need simple integration with transformers\n- Want 8-bit quantization (less compression, better quality)\n- Don't need pre-quantized model files\n\n## Quick start\n\n### Installation\n\n```bash\n# Install AutoGPTQ\npip install auto-gptq\n\n# With Triton (Linux only, faster)\npip install auto-gptq[triton]\n\n# With CUDA extensions (faster)\npip install auto-gptq --no-build-isolation\n\n# Full installation\npip install auto-gptq transformers accelerate\n```\n\n### Load pre-quantized model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Load quantized model from HuggingFace\nmodel_name = \"TheBloke\u002FLlama-2-7B-Chat-GPTQ\"\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=False  # Set True on Linux for speed\n)\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Generate\nprompt = \"Explain quantum computing\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda:0\")\noutputs = model.generate(**inputs, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\n### Quantize your own model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel_name = \"meta-llama\u002FLlama-2-7b-chat-hf\"\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Quantization config\nquantize_config = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Group size (recommended: 128)\n    desc_act=False,      # Activation order (False for CUDA kernel)\n    damp_percent=0.01    # Dampening factor\n)\n\n# Load model for quantization\nmodel = AutoGPTQForCausalLM.from_pretrained(\n    model_name,\n    quantize_config=quantize_config\n)\n\n# Prepare calibration data\ndataset = load_dataset(\"c4\", split=\"train\", streaming=True)\ncalibration_data = [\n    tokenizer(example[\"text\"])[\"input_ids\"][:512]\n    for example in dataset.take(128)\n]\n\n# Quantize\nmodel.quantize(calibration_data)\n\n# Save quantized model\nmodel.save_quantized(\"llama-2-7b-gptq\")\ntokenizer.save_pretrained(\"llama-2-7b-gptq\")\n\n# Push to HuggingFace\nmodel.push_to_hub(\"username\u002Fllama-2-7b-gptq\")\n```\n\n## Group-wise quantization\n\n**How GPTQ works**:\n1. **Group weights**: Divide each weight matrix into groups (typically 128 elements)\n2. **Quantize per-group**: Each group has its own scale\u002Fzero-point\n3. **Minimize error**: Uses Hessian information to minimize quantization error\n4. **Result**: 4-bit weights with near-FP16 accuracy\n\n**Group size trade-off**:\n\n| Group Size | Model Size | Accuracy | Speed | Recommendation |\n|------------|------------|----------|-------|----------------|\n| -1 (per-column) | Smallest | Best | Slowest | Research only |\n| 32 | Smaller | Better | Slower | High accuracy needed |\n| **128** | Medium | Good | **Fast** | **Recommended default** |\n| 256 | Larger | Lower | Faster | Speed critical |\n| 1024 | Largest | Lowest | Fastest | Not recommended |\n\n**Example**:\n```\nWeight matrix: [1024, 4096] = 4.2M elements\n\nGroup size = 128:\n- Groups: 4.2M \u002F 128 = 32,768 groups\n- Each group: own 4-bit scale + zero-point\n- Result: Better granularity → better accuracy\n```\n\n## Quantization configurations\n\n### Standard 4-bit (recommended)\n\n```python\nfrom auto_gptq import BaseQuantizeConfig\n\nconfig = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Standard group size\n    desc_act=False,      # Faster CUDA kernel\n    damp_percent=0.01    # Dampening factor\n)\n```\n\n**Performance**:\n- Memory: 4× reduction (70B model: 140GB → 35GB)\n- Accuracy: ~1.5% perplexity increase\n- Speed: 3-4× faster than FP16\n\n### High accuracy (3-bit with larger groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=3,              # 3-bit (more compression)\n    group_size=128,      # Keep standard group size\n    desc_act=True,       # Better accuracy (slower)\n    damp_percent=0.01\n)\n```\n\n**Trade-off**:\n- Memory: 5× reduction\n- Accuracy: ~3% perplexity increase\n- Speed: 5× faster (but less accurate)\n\n### Maximum accuracy (4-bit with small groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=32,       # Smaller groups (better accuracy)\n    desc_act=True,       # Activation reordering\n    damp_percent=0.005   # Lower dampening\n)\n```\n\n**Trade-off**:\n- Memory: 3.5× reduction (slightly larger)\n- Accuracy: ~0.8% perplexity increase (best)\n- Speed: 2-3× faster (kernel overhead)\n\n## Kernel backends\n\n### ExLlamaV2 (default, fastest)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_exllama=True,      # Use ExLlamaV2\n    exllama_config={\"version\": 2}\n)\n```\n\n**Performance**: 1.5-2× faster than Triton\n\n### Marlin (Ampere+ GPUs)\n\n```python\n# Quantize with Marlin format\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=128,\n    desc_act=False  # Required for Marlin\n)\n\nmodel.quantize(calibration_data, use_marlin=True)\n\n# Load with Marlin\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_marlin=True  # 2× faster on A100\u002FH100\n)\n```\n\n**Requirements**:\n- NVIDIA Ampere or newer (A100, H100, RTX 40xx)\n- Compute capability ≥ 8.0\n\n### Triton (Linux only)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=True  # Linux only\n)\n```\n\n**Performance**: 1.2-1.5× faster than CUDA backend\n\n## Integration with transformers\n\n### Direct transformers usage\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load quantized model (transformers auto-detects GPTQ)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-13B-Chat-GPTQ\",\n    device_map=\"auto\",\n    trust_remote_code=False\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"TheBloke\u002FLlama-2-13B-Chat-GPTQ\")\n\n# Use like any transformers model\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_new_tokens=100)\n```\n\n### QLoRA fine-tuning (GPTQ + LoRA)\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model\n\n# Load GPTQ model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-7B-GPTQ\",\n    device_map=\"auto\"\n)\n\n# Prepare for LoRA training\nmodel = prepare_model_for_kbit_training(model)\n\n# LoRA config\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"v_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\n# Add LoRA adapters\nmodel = get_peft_model(model, lora_config)\n\n# Fine-tune (memory efficient!)\n# 70B model trainable on single A100 80GB\n```\n\n## Performance benchmarks\n\n### Memory reduction\n\n| Model | FP16 | GPTQ 4-bit | Reduction |\n|-------|------|------------|-----------|\n| Llama 2-7B | 14 GB | 3.5 GB | 4× |\n| Llama 2-13B | 26 GB | 6.5 GB | 4× |\n| Llama 2-70B | 140 GB | 35 GB | 4× |\n| Llama 3-405B | 810 GB | 203 GB | 4× |\n\n**Enables**:\n- 70B on single A100 80GB (vs 2× A100 needed for FP16)\n- 405B on 3× A100 80GB (vs 11× A100 needed for FP16)\n- 13B on RTX 4090 24GB (vs OOM with FP16)\n\n### Inference speed (Llama 2-7B, A100)\n\n| Precision | Tokens\u002Fsec | vs FP16 |\n|-----------|------------|---------|\n| FP16 | 25 tok\u002Fs | 1× |\n| GPTQ 4-bit (CUDA) | 85 tok\u002Fs | 3.4× |\n| GPTQ 4-bit (ExLlama) | 105 tok\u002Fs | 4.2× |\n| GPTQ 4-bit (Marlin) | 120 tok\u002Fs | 4.8× |\n\n### Accuracy (perplexity on WikiText-2)\n\n| Model | FP16 | GPTQ 4-bit (g=128) | Degradation |\n|-------|------|---------------------|-------------|\n| Llama 2-7B | 5.47 | 5.55 | +1.5% |\n| Llama 2-13B | 4.88 | 4.95 | +1.4% |\n| Llama 2-70B | 3.32 | 3.38 | +1.8% |\n\n**Excellent quality preservation** - less than 2% degradation!\n\n## Common patterns\n\n### Multi-GPU deployment\n\n```python\n# Automatic device mapping\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-GPTQ\",\n    device_map=\"auto\",  # Automatically split across GPUs\n    max_memory={0: \"40GB\", 1: \"40GB\"}  # Limit per GPU\n)\n\n# Manual device mapping\ndevice_map = {\n    \"model.embed_tokens\": 0,\n    \"model.layers.0-39\": 0,  # First 40 layers on GPU 0\n    \"model.layers.40-79\": 1,  # Last 40 layers on GPU 1\n    \"model.norm\": 1,\n    \"lm_head\": 1\n}\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device_map=device_map\n)\n```\n\n### CPU offloading\n\n```python\n# Offload some layers to CPU (for very large models)\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-405B-GPTQ\",\n    device_map=\"auto\",\n    max_memory={\n        0: \"80GB\",  # GPU 0\n        1: \"80GB\",  # GPU 1\n        2: \"80GB\",  # GPU 2\n        \"cpu\": \"200GB\"  # Offload overflow to CPU\n    }\n)\n```\n\n### Batch inference\n\n```python\n# Process multiple prompts efficiently\nprompts = [\n    \"Explain AI\",\n    \"Explain ML\",\n    \"Explain DL\"\n]\n\ninputs = tokenizer(prompts, return_tensors=\"pt\", padding=True).to(\"cuda\")\n\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=100,\n    pad_token_id=tokenizer.eos_token_id\n)\n\nfor i, output in enumerate(outputs):\n    print(f\"Prompt {i}: {tokenizer.decode(output)}\")\n```\n\n## Finding pre-quantized models\n\n**TheBloke on HuggingFace**:\n- https:\u002F\u002Fhuggingface.co\u002FTheBloke\n- 1000+ models in GPTQ format\n- Multiple group sizes (32, 128)\n- Both CUDA and Marlin formats\n\n**Search**:\n```bash\n# Find GPTQ models on HuggingFace\nhttps:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n```\n\n**Download**:\n```python\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Automatically downloads from HuggingFace\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-Chat-GPTQ\",\n    device=\"cuda:0\"\n)\n```\n\n## Supported models\n\n- **LLaMA family**: Llama 2, Llama 3, Code Llama\n- **Mistral**: Mistral 7B, Mixtral 8x7B, 8x22B\n- **Qwen**: Qwen, Qwen2, QwQ\n- **DeepSeek**: V2, V3\n- **Phi**: Phi-2, Phi-3\n- **Yi, Falcon, BLOOM, OPT**\n- **100+ models** on HuggingFace\n\n## References\n\n- **[Calibration Guide](references\u002Fcalibration.md)** - Dataset selection, quantization process, quality optimization\n- **[Integration Guide](references\u002Fintegration.md)** - Transformers, PEFT, vLLM, TensorRT-LLM\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, performance optimization\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FAutoGPTQ\u002FAutoGPTQ\n- **Paper**: GPTQ: Accurate Post-Training Quantization (arXiv:2210.17323)\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002Fautogptq\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `quantizing-models-bitsandbytes`: Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss.\n- `hqq-quantization`: Half-Quadratic Quantization for LLMs without calibration data.\n- `awq-quantization`: Activation-aware weight quantization for 4-bit LLM compression with 3x speedup and minimal accuracy loss.\n",{"name":5,"license":15,"metadata":44,"description":7},{"tags":45,"version":46,"category":47,"maintainer":48,"dependencies":49},"Optimization, GPTQ, Quantization, 4-Bit, Post-Training, Memory Optimization, Consumer GPUs, Fast Inference, QLoRA, Group-Wise Quantization","1.0.0","ml-training","Kalaris Labs","auto-gptq, transformers, optimum, peft","main","0da4485785ee599388900be50db1635489d53056",".agents\u002Fskills\u002Fkal-gptq","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6","https:\u002F\u002Fdisha6.6.thenitishkr.in\u002F","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh","www.thenitishkr","thenitishkr | AI • Cyber-security • Research\r\n","2026-11-08 03:33:25",{"source":8,"skillName":5,"commands":60,"prompt":70,"zipAvailable":71},[61,64,67],{"tool":62,"command":63},"npx","npx skills add Tashima-Tarsh\u002FDisha6.6 --skill gptq",{"tool":65,"command":66},"bunx","bunx skills add Tashima-Tarsh\u002FDisha6.6 --skill gptq",{"tool":68,"command":69},"pnpm","pnpm dlx skills add Tashima-Tarsh\u002FDisha6.6 --skill gptq","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-gptq\u002FSKILL.md",true,[73,79,83],{"path":74,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":77,"sourceHtmlUrl":78},"references\u002Fcalibration.md","REFERENCE","参考文档",8164,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-gptq\u002Freferences\u002Fcalibration.md",{"path":80,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":81,"sourceHtmlUrl":82},"references\u002Fintegration.md",2797,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-gptq\u002Freferences\u002Fintegration.md",{"path":84,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":85,"sourceHtmlUrl":86},"references\u002Ftroubleshooting.md",1899,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fkal-gptq\u002Freferences\u002Ftroubleshooting.md",[88,119,143,170,195,219],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":99,"publishedAt":39},"01M6QRD07D4TGNDK6VJ4EHF8BV","youtube-full","Use when the user needs YouTube transcripts, video search, channel browsing, playlist extraction, or content monitoring. Trigger phrases: 'get the transcript for', 'search YouTube for', 'what are the latest videos on', 'list this playlist', 'monitor this channel', or any request involving a YouTube URL, video ID, or @handle. Do NOT use for downloading video or audio files, YouTube engagement data (likes, comments), or private\u002Fage-restricted videos.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fyoutube-full\u002FSKILL.md",[94],{"categoryNo":95,"categoryName":96,"taxonomyType":97,"taxonomyTypeDesc":98},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",[100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,22,116,117,118],"youtube full","youtube","full","transcript","search","monitoring","data","Video","Audio","list","files","do","get","age","trigger","URL","browsing","request","youtube transcripts",{"skillNo":120,"name":121,"nameZh":6,"description":122,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":123,"categories":124,"keywords":125,"publishedAt":39},"01M6QRCJ3112VYPA97YG7KGMC2","x-twitter-growth","X\u002FTwitter growth engine for building audience, crafting viral content, and analyzing engagement. Use when the user wants to grow on X\u002FTwitter, write tweets or threads, analyze their X profile, research competitors on X, plan a posting strategy, or optimize engagement. Complements social-content (generic multi-platform) with X-specific depth: algorithm mechanics, thread engineering, reply strategy, profile optimization, and competitive intelligence via web search.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fx-twitter-growth\u002FSKILL.md",[],[126,127,128,129,104,130,131,132,133,134,135,136,137,138,32,139,140,141,142,22],"twitter","x twitter growth","growth","research","content","engineering","plan","web search","competitors","analyze","engine","write","strategy","platform","grow","posting","competitive",{"skillNo":144,"name":145,"nameZh":6,"description":146,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":147,"categories":148,"keywords":152,"publishedAt":39},"01M6QRBN3EHQNJY42RERP7A149","write-a-skill","Create new agent skills with proper structure, progressive disclosure, and bundled resources. Use when user wants to create, write, build, or author a new skill.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrite-a-skill\u002FSKILL.md",[149],{"categoryNo":150,"categoryName":151,"taxonomyType":97,"taxonomyTypeDesc":98},"01JBSEEDCAT000000000000023","Skill与智能体开发",[153,137,154,155,156,157,158,159,32,160,161,50,162,163,164,165,166,167,168,169],"write a skill","github","create","https","productivity","matt pocock","direct","example","build","new skill","structure","new agent","resources","with","tree","progressive disclosure","author",{"skillNo":171,"name":172,"nameZh":6,"description":173,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":174,"categories":175,"keywords":176,"publishedAt":39},"01M6QRB6G8P33PNGAF9Y69EJ9T","wrap-up","Close out a launched Claude Managed Agent — recap every primitive the founder now owns, regenerate the single-file overview page, and suggest the next 1-2 upgrades. Use when the user says \"wrap up\", \"close this out\", \"what do I own now\", \"give me the summary\", \"recap the agent\", or when the orchestrator routes phase=wrap-up. primitives_inventory.py tables everything owned (agent, environment, session, memory, outcome, deployment); overview_page.py regenerates a self-contained .\u002Fmy-agent\u002Fagent-overview.html; upgrade_suggester.py ranks the next moves from recorded deferrals plus standing hardening steps. Companion to run-without-you; the last stop before phase=done.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrap-up\u002FSKILL.md",[],[177,178,179,180,181,182,183,184,185,32,186,111,187,188,189,190,191,192,193,194],"wrap up","up","wrap","gemini","deployment","run","file","hardening","closeout","stop","upgrade","founder","close","companion","you","tables","directions","phase",{"skillNo":196,"name":197,"nameZh":6,"description":198,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":199,"categories":200,"keywords":202,"publishedAt":39},"01M6QRA5AYC1RBFZ39GTWGS3N8","workflow-builder","Design and write deterministic multi-agent workflow scripts (.js files in .claude\u002Fworkflows\u002F) for Claude Code's Workflow tool. Use when a user wants to build, create, author, scaffold, or run a custom Claude Code workflow, orchestrate sub-agents (fan-out, pipeline, loop, judge-panel), or automate a repeatable multi-step task across fresh-context agents.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fworkflow-builder\u002FSKILL.md",[201],{"categoryNo":150,"categoryName":151,"taxonomyType":97,"taxonomyTypeDesc":98},[203,204,205,206,207,155,182,208,209,210,211,212,137,213,110,214,215,216,217,218],"workflow","workflow builder","builder","design","workflows","code","orchestrate","pipeline","loop","multi-agent","context","across","fresh","scripts","judge","claude code workflow",{"skillNo":220,"name":221,"nameZh":6,"description":222,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":223,"categories":224,"keywords":225,"publishedAt":39},"01M6QR7QMCZESKQA0ERW1E657M","weekly-review","Use when someone wants to run a weekly review, close open loops, audit stalled projects and commitments, get their system back to trusted, restart a lapsed review habit, or says \"\u002Fcs:weekly-review\". Walks David Allen's three-phase loop — GET CLEAR, GET CURRENT, GET CREATIVE — with deterministic scripts that inventory open loops, gate the checklist with named gaps, and score commitment health 0-100.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fweekly-review\u002FSKILL.md",[],[226,227,228,229,230,231,232,233,234,211,235,236,237,238,239,240,112,241,242,194],"review","weekly review","weekly","deterministic scripts","reflect","feeds","inventory","open","team","projects","project-management","scanner","conversation","back","project","sprint","cadence"]