[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M72C35DVVMBAXCPMTSHXECX8":3,"skill-same:01M72C35DVVMBAXCPMTSHXECX8":79},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":41,"commitSha":42,"skillDirectory":43,"forksCount":13,"repositoryHtmlUrl":44,"repositoryHomepageUrl":6,"ownerHtmlUrl":45,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":46,"install":47,"resources":60},"01M72C35DVVMBAXCPMTSHXECX8","gptq",null,"Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with \u003C2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.","Sansar35\u002Fclaude-agent","Sansar35","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F145020281?v=4","01M724CQ2SCC86MGNSD87D0D4K","01M724CQ3965G9WJ3QXE3AJ5YY",1,"MIT","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fgptq\u002FSKILL.md",[],[],"2026-11-10 18:39:20",false,"---\nname: gptq\ndescription: Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with \u003C2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Optimization, GPTQ, Quantization, 4-Bit, Post-Training, Memory Optimization, Consumer GPUs, Fast Inference, QLoRA, Group-Wise Quantization]\ndependencies: [auto-gptq, transformers, optimum, peft]\n---\n\n# GPTQ (Generative Pre-trained Transformer Quantization)\n\nPost-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization.\n\n## When to use GPTQ\n\n**Use GPTQ when:**\n- Need to fit large models (70B+) on limited GPU memory\n- Want 4× memory reduction with \u003C2% accuracy loss\n- Deploying on consumer GPUs (RTX 4090, 3090)\n- Need faster inference (3-4× speedup vs FP16)\n\n**Use AWQ instead when:**\n- Need slightly better accuracy (\u003C1% loss)\n- Have newer GPUs (Ampere, Ada)\n- Want Marlin kernel support (2× faster on some GPUs)\n\n**Use bitsandbytes instead when:**\n- Need simple integration with transformers\n- Want 8-bit quantization (less compression, better quality)\n- Don't need pre-quantized model files\n\n## Quick start\n\n### Installation\n\n```bash\n# Install AutoGPTQ\npip install auto-gptq\n\n# With Triton (Linux only, faster)\npip install auto-gptq[triton]\n\n# With CUDA extensions (faster)\npip install auto-gptq --no-build-isolation\n\n# Full installation\npip install auto-gptq transformers accelerate\n```\n\n### Load pre-quantized model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Load quantized model from HuggingFace\nmodel_name = \"TheBloke\u002FLlama-2-7B-Chat-GPTQ\"\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=False  # Set True on Linux for speed\n)\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Generate\nprompt = \"Explain quantum computing\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda:0\")\noutputs = model.generate(**inputs, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\n### Quantize your own model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel_name = \"meta-llama\u002FLlama-2-7b-chat-hf\"\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Quantization config\nquantize_config = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Group size (recommended: 128)\n    desc_act=False,      # Activation order (False for CUDA kernel)\n    damp_percent=0.01    # Dampening factor\n)\n\n# Load model for quantization\nmodel = AutoGPTQForCausalLM.from_pretrained(\n    model_name,\n    quantize_config=quantize_config\n)\n\n# Prepare calibration data\ndataset = load_dataset(\"c4\", split=\"train\", streaming=True)\ncalibration_data = [\n    tokenizer(example[\"text\"])[\"input_ids\"][:512]\n    for example in dataset.take(128)\n]\n\n# Quantize\nmodel.quantize(calibration_data)\n\n# Save quantized model\nmodel.save_quantized(\"llama-2-7b-gptq\")\ntokenizer.save_pretrained(\"llama-2-7b-gptq\")\n\n# Push to HuggingFace\nmodel.push_to_hub(\"username\u002Fllama-2-7b-gptq\")\n```\n\n## Group-wise quantization\n\n**How GPTQ works**:\n1. **Group weights**: Divide each weight matrix into groups (typically 128 elements)\n2. **Quantize per-group**: Each group has its own scale\u002Fzero-point\n3. **Minimize error**: Uses Hessian information to minimize quantization error\n4. **Result**: 4-bit weights with near-FP16 accuracy\n\n**Group size trade-off**:\n\n| Group Size | Model Size | Accuracy | Speed | Recommendation |\n|------------|------------|----------|-------|----------------|\n| -1 (per-column) | Smallest | Best | Slowest | Research only |\n| 32 | Smaller | Better | Slower | High accuracy needed |\n| **128** | Medium | Good | **Fast** | **Recommended default** |\n| 256 | Larger | Lower | Faster | Speed critical |\n| 1024 | Largest | Lowest | Fastest | Not recommended |\n\n**Example**:\n```\nWeight matrix: [1024, 4096] = 4.2M elements\n\nGroup size = 128:\n- Groups: 4.2M \u002F 128 = 32,768 groups\n- Each group: own 4-bit scale + zero-point\n- Result: Better granularity → better accuracy\n```\n\n## Quantization configurations\n\n### Standard 4-bit (recommended)\n\n```python\nfrom auto_gptq import BaseQuantizeConfig\n\nconfig = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Standard group size\n    desc_act=False,      # Faster CUDA kernel\n    damp_percent=0.01    # Dampening factor\n)\n```\n\n**Performance**:\n- Memory: 4× reduction (70B model: 140GB → 35GB)\n- Accuracy: ~1.5% perplexity increase\n- Speed: 3-4× faster than FP16\n\n### High accuracy (3-bit with larger groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=3,              # 3-bit (more compression)\n    group_size=128,      # Keep standard group size\n    desc_act=True,       # Better accuracy (slower)\n    damp_percent=0.01\n)\n```\n\n**Trade-off**:\n- Memory: 5× reduction\n- Accuracy: ~3% perplexity increase\n- Speed: 5× faster (but less accurate)\n\n### Maximum accuracy (4-bit with small groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=32,       # Smaller groups (better accuracy)\n    desc_act=True,       # Activation reordering\n    damp_percent=0.005   # Lower dampening\n)\n```\n\n**Trade-off**:\n- Memory: 3.5× reduction (slightly larger)\n- Accuracy: ~0.8% perplexity increase (best)\n- Speed: 2-3× faster (kernel overhead)\n\n## Kernel backends\n\n### ExLlamaV2 (default, fastest)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_exllama=True,      # Use ExLlamaV2\n    exllama_config={\"version\": 2}\n)\n```\n\n**Performance**: 1.5-2× faster than Triton\n\n### Marlin (Ampere+ GPUs)\n\n```python\n# Quantize with Marlin format\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=128,\n    desc_act=False  # Required for Marlin\n)\n\nmodel.quantize(calibration_data, use_marlin=True)\n\n# Load with Marlin\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_marlin=True  # 2× faster on A100\u002FH100\n)\n```\n\n**Requirements**:\n- NVIDIA Ampere or newer (A100, H100, RTX 40xx)\n- Compute capability ≥ 8.0\n\n### Triton (Linux only)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=True  # Linux only\n)\n```\n\n**Performance**: 1.2-1.5× faster than CUDA backend\n\n## Integration with transformers\n\n### Direct transformers usage\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load quantized model (transformers auto-detects GPTQ)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-13B-Chat-GPTQ\",\n    device_map=\"auto\",\n    trust_remote_code=False\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"TheBloke\u002FLlama-2-13B-Chat-GPTQ\")\n\n# Use like any transformers model\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_new_tokens=100)\n```\n\n### QLoRA fine-tuning (GPTQ + LoRA)\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model\n\n# Load GPTQ model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-7B-GPTQ\",\n    device_map=\"auto\"\n)\n\n# Prepare for LoRA training\nmodel = prepare_model_for_kbit_training(model)\n\n# LoRA config\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"v_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\n# Add LoRA adapters\nmodel = get_peft_model(model, lora_config)\n\n# Fine-tune (memory efficient!)\n# 70B model trainable on single A100 80GB\n```\n\n## Performance benchmarks\n\n### Memory reduction\n\n| Model | FP16 | GPTQ 4-bit | Reduction |\n|-------|------|------------|-----------|\n| Llama 2-7B | 14 GB | 3.5 GB | 4× |\n| Llama 2-13B | 26 GB | 6.5 GB | 4× |\n| Llama 2-70B | 140 GB | 35 GB | 4× |\n| Llama 3-405B | 810 GB | 203 GB | 4× |\n\n**Enables**:\n- 70B on single A100 80GB (vs 2× A100 needed for FP16)\n- 405B on 3× A100 80GB (vs 11× A100 needed for FP16)\n- 13B on RTX 4090 24GB (vs OOM with FP16)\n\n### Inference speed (Llama 2-7B, A100)\n\n| Precision | Tokens\u002Fsec | vs FP16 |\n|-----------|------------|---------|\n| FP16 | 25 tok\u002Fs | 1× |\n| GPTQ 4-bit (CUDA) | 85 tok\u002Fs | 3.4× |\n| GPTQ 4-bit (ExLlama) | 105 tok\u002Fs | 4.2× |\n| GPTQ 4-bit (Marlin) | 120 tok\u002Fs | 4.8× |\n\n### Accuracy (perplexity on WikiText-2)\n\n| Model | FP16 | GPTQ 4-bit (g=128) | Degradation |\n|-------|------|---------------------|-------------|\n| Llama 2-7B | 5.47 | 5.55 | +1.5% |\n| Llama 2-13B | 4.88 | 4.95 | +1.4% |\n| Llama 2-70B | 3.32 | 3.38 | +1.8% |\n\n**Excellent quality preservation** - less than 2% degradation!\n\n## Common patterns\n\n### Multi-GPU deployment\n\n```python\n# Automatic device mapping\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-GPTQ\",\n    device_map=\"auto\",  # Automatically split across GPUs\n    max_memory={0: \"40GB\", 1: \"40GB\"}  # Limit per GPU\n)\n\n# Manual device mapping\ndevice_map = {\n    \"model.embed_tokens\": 0,\n    \"model.layers.0-39\": 0,  # First 40 layers on GPU 0\n    \"model.layers.40-79\": 1,  # Last 40 layers on GPU 1\n    \"model.norm\": 1,\n    \"lm_head\": 1\n}\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device_map=device_map\n)\n```\n\n### CPU offloading\n\n```python\n# Offload some layers to CPU (for very large models)\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-405B-GPTQ\",\n    device_map=\"auto\",\n    max_memory={\n        0: \"80GB\",  # GPU 0\n        1: \"80GB\",  # GPU 1\n        2: \"80GB\",  # GPU 2\n        \"cpu\": \"200GB\"  # Offload overflow to CPU\n    }\n)\n```\n\n### Batch inference\n\n```python\n# Process multiple prompts efficiently\nprompts = [\n    \"Explain AI\",\n    \"Explain ML\",\n    \"Explain DL\"\n]\n\ninputs = tokenizer(prompts, return_tensors=\"pt\", padding=True).to(\"cuda\")\n\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=100,\n    pad_token_id=tokenizer.eos_token_id\n)\n\nfor i, output in enumerate(outputs):\n    print(f\"Prompt {i}: {tokenizer.decode(output)}\")\n```\n\n## Finding pre-quantized models\n\n**TheBloke on HuggingFace**:\n- https:\u002F\u002Fhuggingface.co\u002FTheBloke\n- 1000+ models in GPTQ format\n- Multiple group sizes (32, 128)\n- Both CUDA and Marlin formats\n\n**Search**:\n```bash\n# Find GPTQ models on HuggingFace\nhttps:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n```\n\n**Download**:\n```python\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Automatically downloads from HuggingFace\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-Chat-GPTQ\",\n    device=\"cuda:0\"\n)\n```\n\n## Supported models\n\n- **LLaMA family**: Llama 2, Llama 3, Code Llama\n- **Mistral**: Mistral 7B, Mixtral 8x7B, 8x22B\n- **Qwen**: Qwen, Qwen2, QwQ\n- **DeepSeek**: V2, V3\n- **Phi**: Phi-2, Phi-3\n- **Yi, Falcon, BLOOM, OPT**\n- **100+ models** on HuggingFace\n\n## References\n\n- **[Calibration Guide](references\u002Fcalibration.md)** - Dataset selection, quantization process, quality optimization\n- **[Integration Guide](references\u002Fintegration.md)** - Transformers, PEFT, vLLM, TensorRT-LLM\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, performance optimization\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FAutoGPTQ\u002FAutoGPTQ\n- **Paper**: GPTQ: Accurate Post-Training Quantization (arXiv:2210.17323)\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002Fautogptq\n\n\n","# GPTQ (Generative Pre-trained Transformer Quantization)\n\nPost-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization.\n\n## When to use GPTQ\n\n**Use GPTQ when:**\n- Need to fit large models (70B+) on limited GPU memory\n- Want 4× memory reduction with \u003C2% accuracy loss\n- Deploying on consumer GPUs (RTX 4090, 3090)\n- Need faster inference (3-4× speedup vs FP16)\n\n**Use AWQ instead when:**\n- Need slightly better accuracy (\u003C1% loss)\n- Have newer GPUs (Ampere, Ada)\n- Want Marlin kernel support (2× faster on some GPUs)\n\n**Use bitsandbytes instead when:**\n- Need simple integration with transformers\n- Want 8-bit quantization (less compression, better quality)\n- Don't need pre-quantized model files\n\n## Quick start\n\n### Installation\n\n```bash\n# Install AutoGPTQ\npip install auto-gptq\n\n# With Triton (Linux only, faster)\npip install auto-gptq[triton]\n\n# With CUDA extensions (faster)\npip install auto-gptq --no-build-isolation\n\n# Full installation\npip install auto-gptq transformers accelerate\n```\n\n### Load pre-quantized model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Load quantized model from HuggingFace\nmodel_name = \"TheBloke\u002FLlama-2-7B-Chat-GPTQ\"\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=False  # Set True on Linux for speed\n)\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Generate\nprompt = \"Explain quantum computing\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda:0\")\noutputs = model.generate(**inputs, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\n### Quantize your own model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel_name = \"meta-llama\u002FLlama-2-7b-chat-hf\"\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Quantization config\nquantize_config = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Group size (recommended: 128)\n    desc_act=False,      # Activation order (False for CUDA kernel)\n    damp_percent=0.01    # Dampening factor\n)\n\n# Load model for quantization\nmodel = AutoGPTQForCausalLM.from_pretrained(\n    model_name,\n    quantize_config=quantize_config\n)\n\n# Prepare calibration data\ndataset = load_dataset(\"c4\", split=\"train\", streaming=True)\ncalibration_data = [\n    tokenizer(example[\"text\"])[\"input_ids\"][:512]\n    for example in dataset.take(128)\n]\n\n# Quantize\nmodel.quantize(calibration_data)\n\n# Save quantized model\nmodel.save_quantized(\"llama-2-7b-gptq\")\ntokenizer.save_pretrained(\"llama-2-7b-gptq\")\n\n# Push to HuggingFace\nmodel.push_to_hub(\"username\u002Fllama-2-7b-gptq\")\n```\n\n## Group-wise quantization\n\n**How GPTQ works**:\n1. **Group weights**: Divide each weight matrix into groups (typically 128 elements)\n2. **Quantize per-group**: Each group has its own scale\u002Fzero-point\n3. **Minimize error**: Uses Hessian information to minimize quantization error\n4. **Result**: 4-bit weights with near-FP16 accuracy\n\n**Group size trade-off**:\n\n| Group Size | Model Size | Accuracy | Speed | Recommendation |\n|------------|------------|----------|-------|----------------|\n| -1 (per-column) | Smallest | Best | Slowest | Research only |\n| 32 | Smaller | Better | Slower | High accuracy needed |\n| **128** | Medium | Good | **Fast** | **Recommended default** |\n| 256 | Larger | Lower | Faster | Speed critical |\n| 1024 | Largest | Lowest | Fastest | Not recommended |\n\n**Example**:\n```\nWeight matrix: [1024, 4096] = 4.2M elements\n\nGroup size = 128:\n- Groups: 4.2M \u002F 128 = 32,768 groups\n- Each group: own 4-bit scale + zero-point\n- Result: Better granularity → better accuracy\n```\n\n## Quantization configurations\n\n### Standard 4-bit (recommended)\n\n```python\nfrom auto_gptq import BaseQuantizeConfig\n\nconfig = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Standard group size\n    desc_act=False,      # Faster CUDA kernel\n    damp_percent=0.01    # Dampening factor\n)\n```\n\n**Performance**:\n- Memory: 4× reduction (70B model: 140GB → 35GB)\n- Accuracy: ~1.5% perplexity increase\n- Speed: 3-4× faster than FP16\n\n### High accuracy (3-bit with larger groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=3,              # 3-bit (more compression)\n    group_size=128,      # Keep standard group size\n    desc_act=True,       # Better accuracy (slower)\n    damp_percent=0.01\n)\n```\n\n**Trade-off**:\n- Memory: 5× reduction\n- Accuracy: ~3% perplexity increase\n- Speed: 5× faster (but less accurate)\n\n### Maximum accuracy (4-bit with small groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=32,       # Smaller groups (better accuracy)\n    desc_act=True,       # Activation reordering\n    damp_percent=0.005   # Lower dampening\n)\n```\n\n**Trade-off**:\n- Memory: 3.5× reduction (slightly larger)\n- Accuracy: ~0.8% perplexity increase (best)\n- Speed: 2-3× faster (kernel overhead)\n\n## Kernel backends\n\n### ExLlamaV2 (default, fastest)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_exllama=True,      # Use ExLlamaV2\n    exllama_config={\"version\": 2}\n)\n```\n\n**Performance**: 1.5-2× faster than Triton\n\n### Marlin (Ampere+ GPUs)\n\n```python\n# Quantize with Marlin format\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=128,\n    desc_act=False  # Required for Marlin\n)\n\nmodel.quantize(calibration_data, use_marlin=True)\n\n# Load with Marlin\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_marlin=True  # 2× faster on A100\u002FH100\n)\n```\n\n**Requirements**:\n- NVIDIA Ampere or newer (A100, H100, RTX 40xx)\n- Compute capability ≥ 8.0\n\n### Triton (Linux only)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=True  # Linux only\n)\n```\n\n**Performance**: 1.2-1.5× faster than CUDA backend\n\n## Integration with transformers\n\n### Direct transformers usage\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load quantized model (transformers auto-detects GPTQ)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-13B-Chat-GPTQ\",\n    device_map=\"auto\",\n    trust_remote_code=False\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"TheBloke\u002FLlama-2-13B-Chat-GPTQ\")\n\n# Use like any transformers model\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_new_tokens=100)\n```\n\n### QLoRA fine-tuning (GPTQ + LoRA)\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model\n\n# Load GPTQ model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-7B-GPTQ\",\n    device_map=\"auto\"\n)\n\n# Prepare for LoRA training\nmodel = prepare_model_for_kbit_training(model)\n\n# LoRA config\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"v_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\n# Add LoRA adapters\nmodel = get_peft_model(model, lora_config)\n\n# Fine-tune (memory efficient!)\n# 70B model trainable on single A100 80GB\n```\n\n## Performance benchmarks\n\n### Memory reduction\n\n| Model | FP16 | GPTQ 4-bit | Reduction |\n|-------|------|------------|-----------|\n| Llama 2-7B | 14 GB | 3.5 GB | 4× |\n| Llama 2-13B | 26 GB | 6.5 GB | 4× |\n| Llama 2-70B | 140 GB | 35 GB | 4× |\n| Llama 3-405B | 810 GB | 203 GB | 4× |\n\n**Enables**:\n- 70B on single A100 80GB (vs 2× A100 needed for FP16)\n- 405B on 3× A100 80GB (vs 11× A100 needed for FP16)\n- 13B on RTX 4090 24GB (vs OOM with FP16)\n\n### Inference speed (Llama 2-7B, A100)\n\n| Precision | Tokens\u002Fsec | vs FP16 |\n|-----------|------------|---------|\n| FP16 | 25 tok\u002Fs | 1× |\n| GPTQ 4-bit (CUDA) | 85 tok\u002Fs | 3.4× |\n| GPTQ 4-bit (ExLlama) | 105 tok\u002Fs | 4.2× |\n| GPTQ 4-bit (Marlin) | 120 tok\u002Fs | 4.8× |\n\n### Accuracy (perplexity on WikiText-2)\n\n| Model | FP16 | GPTQ 4-bit (g=128) | Degradation |\n|-------|------|---------------------|-------------|\n| Llama 2-7B | 5.47 | 5.55 | +1.5% |\n| Llama 2-13B | 4.88 | 4.95 | +1.4% |\n| Llama 2-70B | 3.32 | 3.38 | +1.8% |\n\n**Excellent quality preservation** - less than 2% degradation!\n\n## Common patterns\n\n### Multi-GPU deployment\n\n```python\n# Automatic device mapping\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-GPTQ\",\n    device_map=\"auto\",  # Automatically split across GPUs\n    max_memory={0: \"40GB\", 1: \"40GB\"}  # Limit per GPU\n)\n\n# Manual device mapping\ndevice_map = {\n    \"model.embed_tokens\": 0,\n    \"model.layers.0-39\": 0,  # First 40 layers on GPU 0\n    \"model.layers.40-79\": 1,  # Last 40 layers on GPU 1\n    \"model.norm\": 1,\n    \"lm_head\": 1\n}\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device_map=device_map\n)\n```\n\n### CPU offloading\n\n```python\n# Offload some layers to CPU (for very large models)\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-405B-GPTQ\",\n    device_map=\"auto\",\n    max_memory={\n        0: \"80GB\",  # GPU 0\n        1: \"80GB\",  # GPU 1\n        2: \"80GB\",  # GPU 2\n        \"cpu\": \"200GB\"  # Offload overflow to CPU\n    }\n)\n```\n\n### Batch inference\n\n```python\n# Process multiple prompts efficiently\nprompts = [\n    \"Explain AI\",\n    \"Explain ML\",\n    \"Explain DL\"\n]\n\ninputs = tokenizer(prompts, return_tensors=\"pt\", padding=True).to(\"cuda\")\n\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=100,\n    pad_token_id=tokenizer.eos_token_id\n)\n\nfor i, output in enumerate(outputs):\n    print(f\"Prompt {i}: {tokenizer.decode(output)}\")\n```\n\n## Finding pre-quantized models\n\n**TheBloke on HuggingFace**:\n- https:\u002F\u002Fhuggingface.co\u002FTheBloke\n- 1000+ models in GPTQ format\n- Multiple group sizes (32, 128)\n- Both CUDA and Marlin formats\n\n**Search**:\n```bash\n# Find GPTQ models on HuggingFace\nhttps:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n```\n\n**Download**:\n```python\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Automatically downloads from HuggingFace\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-Chat-GPTQ\",\n    device=\"cuda:0\"\n)\n```\n\n## Supported models\n\n- **LLaMA family**: Llama 2, Llama 3, Code Llama\n- **Mistral**: Mistral 7B, Mixtral 8x7B, 8x22B\n- **Qwen**: Qwen, Qwen2, QwQ\n- **DeepSeek**: V2, V3\n- **Phi**: Phi-2, Phi-3\n- **Yi, Falcon, BLOOM, OPT**\n- **100+ models** on HuggingFace\n\n## References\n\n- **[Calibration Guide](references\u002Fcalibration.md)** - Dataset selection, quantization process, quality optimization\n- **[Integration Guide](references\u002Fintegration.md)** - Transformers, PEFT, vLLM, TensorRT-LLM\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, performance optimization\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FAutoGPTQ\u002FAutoGPTQ\n- **Paper**: GPTQ: Accurate Post-Training Quantization (arXiv:2210.17323)\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002Fautogptq\n\n\n",{"name":5,"tags":23,"author":34,"license":14,"version":35,"description":7,"dependencies":36},[24,25,26,27,28,29,30,31,32,33],"Optimization","GPTQ","Quantization","4-Bit","Post-Training","Memory Optimization","Consumer GPUs","Fast Inference","QLoRA","Group-Wise Quantization","Orchestra Research","1.0.0",[37,38,39,40],"auto-gptq","transformers","optimum","peft","main","d046d088ce5a79a083f0abc0fe1b7ded73bc5b1f","claude-config\u002Fskills\u002Fgptq","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent","https:\u002F\u002Fgithub.com\u002FSansar35","2026-05-10 16:04:53",{"source":8,"skillName":5,"commands":48,"prompt":58,"zipAvailable":59},[49,52,55],{"tool":50,"command":51},"npx","npx skills add Sansar35\u002Fclaude-agent --skill gptq",{"tool":53,"command":54},"bunx","bunx skills add Sansar35\u002Fclaude-agent --skill gptq",{"tool":56,"command":57},"pnpm","pnpm dlx skills add Sansar35\u002Fclaude-agent --skill gptq","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fgptq\u002FSKILL.md",true,[61,67,71,75],{"path":62,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":65,"sourceHtmlUrl":66},".skill-source.json","REFERENCE","参考文档",541,"https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fgptq\u002F.skill-source.json",{"path":68,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":69,"sourceHtmlUrl":70},"references\u002Fcalibration.md",8164,"https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fgptq\u002Freferences\u002Fcalibration.md",{"path":72,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":73,"sourceHtmlUrl":74},"references\u002Fintegration.md",2797,"https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fgptq\u002Freferences\u002Fintegration.md",{"path":76,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":77,"sourceHtmlUrl":78},"references\u002Ftroubleshooting.md",1899,"https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fgptq\u002Freferences\u002Ftroubleshooting.md",[80,87,94,101,108,116],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":18},"01M72DVAN1JYEA302MT0C5MGD6","zustand-store-ts","Create Zustand stores following established patterns with proper TypeScript types and middleware.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzustand-store-ts\u002FSKILL.md",[],[],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":18},"01M72DV8J5NBRRM4RPDFRV95N7","zod-validation-expert","Expert in Zod — TypeScript-first schema validation. Covers parsing, custom errors, refinements, type inference, and integration with React Hook Form, Next.js, and tRPC.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzod-validation-expert\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":18},"01M72DV5EMHW0BHP9BZ167MDCB","zinc-database","Access ZINC (230M+ purchasable compounds). Search by ZINC ID\u002FSMILES, similarity searches, 3D-ready structures for docking, analog discovery, for virtual screening and drug discovery.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzinc-database\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":18},"01M72DV216QY0E693ZPEVQ0DBS","zarr-python","Chunked N-D arrays for cloud storage. Compressed arrays, parallel I\u002FO, S3\u002FGCS integration, NumPy\u002FDask\u002FXarray compatible, for large-scale scientific computing pipelines.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzarr-python\u002FSKILL.md",[],[],{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":112,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":18},"01M72DTXQREMCRKA8NHCVKHN4W","zapier-workflows","Manage and trigger pre-built Zapier workflows and MCP tool orchestration. Use when user mentions workflows, Zaps, automations, daily digest, research, search, lead tracking, expenses, or asks to \"run\" any process. Also handles Perplexity-based research and Google Sheets data tracking.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzapier-workflows\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzapier-workflows\u002FSKILL.md",[],[],{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":18},"01M72DTT17TBSJ0Y8J2XZ4C9TZ","zapier-make-patterns","No-code automation democratizes workflow building. Zapier and Make (formerly Integromat) let non-developers automate business processes without writing code. But no-code doesn't mean no-complexity - these platforms have their own patterns, pitfalls, and breaking points.  This skill covers when to use which platform, how to build reliable automations, and when to graduate to code-based solutions. Key insight: Zapier optimizes for simplicity and integrations (7000+ apps), Make optimizes for power","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzapier-make-patterns\u002FSKILL.md",[],[]]