[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M73ACXQMSRAXNSAJMRNX3F4H":3,"skill-same:01M73ACXQMSRAXNSAJMRNX3F4H":78},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":42,"commitSha":43,"skillDirectory":44,"forksCount":45,"repositoryHtmlUrl":46,"repositoryHomepageUrl":6,"ownerHtmlUrl":47,"ownerBlogUrl":48,"ownerDescription":6,"updatedAt":49,"install":50,"resources":63},"01M73ACXQMSRAXNSAJMRNX3F4H","gptq",null,"Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with \u003C2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.","qhjqhj00\u002Fresearch-skills-pool","qhjqhj00","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F29684092?v=4","01M72AE0ZGXVM5C57N512DGQG8","Tommy Chien","01M72AE107ESJMHT7QC5SXX144",3,"MIT","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--10-optimization__gptq\u002FSKILL.md",[],[],"2026-11-11 15:11:14",false,"---\nname: gptq\ndescription: Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with \u003C2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Optimization, GPTQ, Quantization, 4-Bit, Post-Training, Memory Optimization, Consumer GPUs, Fast Inference, QLoRA, Group-Wise Quantization]\ndependencies: [auto-gptq, transformers, optimum, peft]\n---\n\n# GPTQ (Generative Pre-trained Transformer Quantization)\n\nPost-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization.\n\n## When to use GPTQ\n\n**Use GPTQ when:**\n- Need to fit large models (70B+) on limited GPU memory\n- Want 4× memory reduction with \u003C2% accuracy loss\n- Deploying on consumer GPUs (RTX 4090, 3090)\n- Need faster inference (3-4× speedup vs FP16)\n\n**Use AWQ instead when:**\n- Need slightly better accuracy (\u003C1% loss)\n- Have newer GPUs (Ampere, Ada)\n- Want Marlin kernel support (2× faster on some GPUs)\n\n**Use bitsandbytes instead when:**\n- Need simple integration with transformers\n- Want 8-bit quantization (less compression, better quality)\n- Don't need pre-quantized model files\n\n## Quick start\n\n### Installation\n\n```bash\n# Install AutoGPTQ\npip install auto-gptq\n\n# With Triton (Linux only, faster)\npip install auto-gptq[triton]\n\n# With CUDA extensions (faster)\npip install auto-gptq --no-build-isolation\n\n# Full installation\npip install auto-gptq transformers accelerate\n```\n\n### Load pre-quantized model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Load quantized model from HuggingFace\nmodel_name = \"TheBloke\u002FLlama-2-7B-Chat-GPTQ\"\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=False  # Set True on Linux for speed\n)\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Generate\nprompt = \"Explain quantum computing\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda:0\")\noutputs = model.generate(**inputs, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\n### Quantize your own model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel_name = \"meta-llama\u002FLlama-2-7b-chat-hf\"\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Quantization config\nquantize_config = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Group size (recommended: 128)\n    desc_act=False,      # Activation order (False for CUDA kernel)\n    damp_percent=0.01    # Dampening factor\n)\n\n# Load model for quantization\nmodel = AutoGPTQForCausalLM.from_pretrained(\n    model_name,\n    quantize_config=quantize_config\n)\n\n# Prepare calibration data\ndataset = load_dataset(\"c4\", split=\"train\", streaming=True)\ncalibration_data = [\n    tokenizer(example[\"text\"])[\"input_ids\"][:512]\n    for example in dataset.take(128)\n]\n\n# Quantize\nmodel.quantize(calibration_data)\n\n# Save quantized model\nmodel.save_quantized(\"llama-2-7b-gptq\")\ntokenizer.save_pretrained(\"llama-2-7b-gptq\")\n\n# Push to HuggingFace\nmodel.push_to_hub(\"username\u002Fllama-2-7b-gptq\")\n```\n\n## Group-wise quantization\n\n**How GPTQ works**:\n1. **Group weights**: Divide each weight matrix into groups (typically 128 elements)\n2. **Quantize per-group**: Each group has its own scale\u002Fzero-point\n3. **Minimize error**: Uses Hessian information to minimize quantization error\n4. **Result**: 4-bit weights with near-FP16 accuracy\n\n**Group size trade-off**:\n\n| Group Size | Model Size | Accuracy | Speed | Recommendation |\n|------------|------------|----------|-------|----------------|\n| -1 (per-column) | Smallest | Best | Slowest | Research only |\n| 32 | Smaller | Better | Slower | High accuracy needed |\n| **128** | Medium | Good | **Fast** | **Recommended default** |\n| 256 | Larger | Lower | Faster | Speed critical |\n| 1024 | Largest | Lowest | Fastest | Not recommended |\n\n**Example**:\n```\nWeight matrix: [1024, 4096] = 4.2M elements\n\nGroup size = 128:\n- Groups: 4.2M \u002F 128 = 32,768 groups\n- Each group: own 4-bit scale + zero-point\n- Result: Better granularity → better accuracy\n```\n\n## Quantization configurations\n\n### Standard 4-bit (recommended)\n\n```python\nfrom auto_gptq import BaseQuantizeConfig\n\nconfig = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Standard group size\n    desc_act=False,      # Faster CUDA kernel\n    damp_percent=0.01    # Dampening factor\n)\n```\n\n**Performance**:\n- Memory: 4× reduction (70B model: 140GB → 35GB)\n- Accuracy: ~1.5% perplexity increase\n- Speed: 3-4× faster than FP16\n\n### High accuracy (3-bit with larger groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=3,              # 3-bit (more compression)\n    group_size=128,      # Keep standard group size\n    desc_act=True,       # Better accuracy (slower)\n    damp_percent=0.01\n)\n```\n\n**Trade-off**:\n- Memory: 5× reduction\n- Accuracy: ~3% perplexity increase\n- Speed: 5× faster (but less accurate)\n\n### Maximum accuracy (4-bit with small groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=32,       # Smaller groups (better accuracy)\n    desc_act=True,       # Activation reordering\n    damp_percent=0.005   # Lower dampening\n)\n```\n\n**Trade-off**:\n- Memory: 3.5× reduction (slightly larger)\n- Accuracy: ~0.8% perplexity increase (best)\n- Speed: 2-3× faster (kernel overhead)\n\n## Kernel backends\n\n### ExLlamaV2 (default, fastest)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_exllama=True,      # Use ExLlamaV2\n    exllama_config={\"version\": 2}\n)\n```\n\n**Performance**: 1.5-2× faster than Triton\n\n### Marlin (Ampere+ GPUs)\n\n```python\n# Quantize with Marlin format\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=128,\n    desc_act=False  # Required for Marlin\n)\n\nmodel.quantize(calibration_data, use_marlin=True)\n\n# Load with Marlin\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_marlin=True  # 2× faster on A100\u002FH100\n)\n```\n\n**Requirements**:\n- NVIDIA Ampere or newer (A100, H100, RTX 40xx)\n- Compute capability ≥ 8.0\n\n### Triton (Linux only)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=True  # Linux only\n)\n```\n\n**Performance**: 1.2-1.5× faster than CUDA backend\n\n## Integration with transformers\n\n### Direct transformers usage\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load quantized model (transformers auto-detects GPTQ)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-13B-Chat-GPTQ\",\n    device_map=\"auto\",\n    trust_remote_code=False\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"TheBloke\u002FLlama-2-13B-Chat-GPTQ\")\n\n# Use like any transformers model\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_new_tokens=100)\n```\n\n### QLoRA fine-tuning (GPTQ + LoRA)\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model\n\n# Load GPTQ model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-7B-GPTQ\",\n    device_map=\"auto\"\n)\n\n# Prepare for LoRA training\nmodel = prepare_model_for_kbit_training(model)\n\n# LoRA config\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"v_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\n# Add LoRA adapters\nmodel = get_peft_model(model, lora_config)\n\n# Fine-tune (memory efficient!)\n# 70B model trainable on single A100 80GB\n```\n\n## Performance benchmarks\n\n### Memory reduction\n\n| Model | FP16 | GPTQ 4-bit | Reduction |\n|-------|------|------------|-----------|\n| Llama 2-7B | 14 GB | 3.5 GB | 4× |\n| Llama 2-13B | 26 GB | 6.5 GB | 4× |\n| Llama 2-70B | 140 GB | 35 GB | 4× |\n| Llama 3-405B | 810 GB | 203 GB | 4× |\n\n**Enables**:\n- 70B on single A100 80GB (vs 2× A100 needed for FP16)\n- 405B on 3× A100 80GB (vs 11× A100 needed for FP16)\n- 13B on RTX 4090 24GB (vs OOM with FP16)\n\n### Inference speed (Llama 2-7B, A100)\n\n| Precision | Tokens\u002Fsec | vs FP16 |\n|-----------|------------|---------|\n| FP16 | 25 tok\u002Fs | 1× |\n| GPTQ 4-bit (CUDA) | 85 tok\u002Fs | 3.4× |\n| GPTQ 4-bit (ExLlama) | 105 tok\u002Fs | 4.2× |\n| GPTQ 4-bit (Marlin) | 120 tok\u002Fs | 4.8× |\n\n### Accuracy (perplexity on WikiText-2)\n\n| Model | FP16 | GPTQ 4-bit (g=128) | Degradation |\n|-------|------|---------------------|-------------|\n| Llama 2-7B | 5.47 | 5.55 | +1.5% |\n| Llama 2-13B | 4.88 | 4.95 | +1.4% |\n| Llama 2-70B | 3.32 | 3.38 | +1.8% |\n\n**Excellent quality preservation** - less than 2% degradation!\n\n## Common patterns\n\n### Multi-GPU deployment\n\n```python\n# Automatic device mapping\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-GPTQ\",\n    device_map=\"auto\",  # Automatically split across GPUs\n    max_memory={0: \"40GB\", 1: \"40GB\"}  # Limit per GPU\n)\n\n# Manual device mapping\ndevice_map = {\n    \"model.embed_tokens\": 0,\n    \"model.layers.0-39\": 0,  # First 40 layers on GPU 0\n    \"model.layers.40-79\": 1,  # Last 40 layers on GPU 1\n    \"model.norm\": 1,\n    \"lm_head\": 1\n}\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device_map=device_map\n)\n```\n\n### CPU offloading\n\n```python\n# Offload some layers to CPU (for very large models)\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-405B-GPTQ\",\n    device_map=\"auto\",\n    max_memory={\n        0: \"80GB\",  # GPU 0\n        1: \"80GB\",  # GPU 1\n        2: \"80GB\",  # GPU 2\n        \"cpu\": \"200GB\"  # Offload overflow to CPU\n    }\n)\n```\n\n### Batch inference\n\n```python\n# Process multiple prompts efficiently\nprompts = [\n    \"Explain AI\",\n    \"Explain ML\",\n    \"Explain DL\"\n]\n\ninputs = tokenizer(prompts, return_tensors=\"pt\", padding=True).to(\"cuda\")\n\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=100,\n    pad_token_id=tokenizer.eos_token_id\n)\n\nfor i, output in enumerate(outputs):\n    print(f\"Prompt {i}: {tokenizer.decode(output)}\")\n```\n\n## Finding pre-quantized models\n\n**TheBloke on HuggingFace**:\n- https:\u002F\u002Fhuggingface.co\u002FTheBloke\n- 1000+ models in GPTQ format\n- Multiple group sizes (32, 128)\n- Both CUDA and Marlin formats\n\n**Search**:\n```bash\n# Find GPTQ models on HuggingFace\nhttps:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n```\n\n**Download**:\n```python\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Automatically downloads from HuggingFace\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-Chat-GPTQ\",\n    device=\"cuda:0\"\n)\n```\n\n## Supported models\n\n- **LLaMA family**: Llama 2, Llama 3, Code Llama\n- **Mistral**: Mistral 7B, Mixtral 8x7B, 8x22B\n- **Qwen**: Qwen, Qwen2, QwQ\n- **DeepSeek**: V2, V3\n- **Phi**: Phi-2, Phi-3\n- **Yi, Falcon, BLOOM, OPT**\n- **100+ models** on HuggingFace\n\n## References\n\n- **[Calibration Guide](references\u002Fcalibration.md)** - Dataset selection, quantization process, quality optimization\n- **[Integration Guide](references\u002Fintegration.md)** - Transformers, PEFT, vLLM, TensorRT-LLM\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, performance optimization\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FAutoGPTQ\u002FAutoGPTQ\n- **Paper**: GPTQ: Accurate Post-Training Quantization (arXiv:2210.17323)\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002Fautogptq\n\n\n","# GPTQ (Generative Pre-trained Transformer Quantization)\n\nPost-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization.\n\n## When to use GPTQ\n\n**Use GPTQ when:**\n- Need to fit large models (70B+) on limited GPU memory\n- Want 4× memory reduction with \u003C2% accuracy loss\n- Deploying on consumer GPUs (RTX 4090, 3090)\n- Need faster inference (3-4× speedup vs FP16)\n\n**Use AWQ instead when:**\n- Need slightly better accuracy (\u003C1% loss)\n- Have newer GPUs (Ampere, Ada)\n- Want Marlin kernel support (2× faster on some GPUs)\n\n**Use bitsandbytes instead when:**\n- Need simple integration with transformers\n- Want 8-bit quantization (less compression, better quality)\n- Don't need pre-quantized model files\n\n## Quick start\n\n### Installation\n\n```bash\n# Install AutoGPTQ\npip install auto-gptq\n\n# With Triton (Linux only, faster)\npip install auto-gptq[triton]\n\n# With CUDA extensions (faster)\npip install auto-gptq --no-build-isolation\n\n# Full installation\npip install auto-gptq transformers accelerate\n```\n\n### Load pre-quantized model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Load quantized model from HuggingFace\nmodel_name = \"TheBloke\u002FLlama-2-7B-Chat-GPTQ\"\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=False  # Set True on Linux for speed\n)\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Generate\nprompt = \"Explain quantum computing\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda:0\")\noutputs = model.generate(**inputs, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\n### Quantize your own model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel_name = \"meta-llama\u002FLlama-2-7b-chat-hf\"\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Quantization config\nquantize_config = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Group size (recommended: 128)\n    desc_act=False,      # Activation order (False for CUDA kernel)\n    damp_percent=0.01    # Dampening factor\n)\n\n# Load model for quantization\nmodel = AutoGPTQForCausalLM.from_pretrained(\n    model_name,\n    quantize_config=quantize_config\n)\n\n# Prepare calibration data\ndataset = load_dataset(\"c4\", split=\"train\", streaming=True)\ncalibration_data = [\n    tokenizer(example[\"text\"])[\"input_ids\"][:512]\n    for example in dataset.take(128)\n]\n\n# Quantize\nmodel.quantize(calibration_data)\n\n# Save quantized model\nmodel.save_quantized(\"llama-2-7b-gptq\")\ntokenizer.save_pretrained(\"llama-2-7b-gptq\")\n\n# Push to HuggingFace\nmodel.push_to_hub(\"username\u002Fllama-2-7b-gptq\")\n```\n\n## Group-wise quantization\n\n**How GPTQ works**:\n1. **Group weights**: Divide each weight matrix into groups (typically 128 elements)\n2. **Quantize per-group**: Each group has its own scale\u002Fzero-point\n3. **Minimize error**: Uses Hessian information to minimize quantization error\n4. **Result**: 4-bit weights with near-FP16 accuracy\n\n**Group size trade-off**:\n\n| Group Size | Model Size | Accuracy | Speed | Recommendation |\n|------------|------------|----------|-------|----------------|\n| -1 (per-column) | Smallest | Best | Slowest | Research only |\n| 32 | Smaller | Better | Slower | High accuracy needed |\n| **128** | Medium | Good | **Fast** | **Recommended default** |\n| 256 | Larger | Lower | Faster | Speed critical |\n| 1024 | Largest | Lowest | Fastest | Not recommended |\n\n**Example**:\n```\nWeight matrix: [1024, 4096] = 4.2M elements\n\nGroup size = 128:\n- Groups: 4.2M \u002F 128 = 32,768 groups\n- Each group: own 4-bit scale + zero-point\n- Result: Better granularity → better accuracy\n```\n\n## Quantization configurations\n\n### Standard 4-bit (recommended)\n\n```python\nfrom auto_gptq import BaseQuantizeConfig\n\nconfig = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Standard group size\n    desc_act=False,      # Faster CUDA kernel\n    damp_percent=0.01    # Dampening factor\n)\n```\n\n**Performance**:\n- Memory: 4× reduction (70B model: 140GB → 35GB)\n- Accuracy: ~1.5% perplexity increase\n- Speed: 3-4× faster than FP16\n\n### High accuracy (3-bit with larger groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=3,              # 3-bit (more compression)\n    group_size=128,      # Keep standard group size\n    desc_act=True,       # Better accuracy (slower)\n    damp_percent=0.01\n)\n```\n\n**Trade-off**:\n- Memory: 5× reduction\n- Accuracy: ~3% perplexity increase\n- Speed: 5× faster (but less accurate)\n\n### Maximum accuracy (4-bit with small groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=32,       # Smaller groups (better accuracy)\n    desc_act=True,       # Activation reordering\n    damp_percent=0.005   # Lower dampening\n)\n```\n\n**Trade-off**:\n- Memory: 3.5× reduction (slightly larger)\n- Accuracy: ~0.8% perplexity increase (best)\n- Speed: 2-3× faster (kernel overhead)\n\n## Kernel backends\n\n### ExLlamaV2 (default, fastest)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_exllama=True,      # Use ExLlamaV2\n    exllama_config={\"version\": 2}\n)\n```\n\n**Performance**: 1.5-2× faster than Triton\n\n### Marlin (Ampere+ GPUs)\n\n```python\n# Quantize with Marlin format\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=128,\n    desc_act=False  # Required for Marlin\n)\n\nmodel.quantize(calibration_data, use_marlin=True)\n\n# Load with Marlin\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_marlin=True  # 2× faster on A100\u002FH100\n)\n```\n\n**Requirements**:\n- NVIDIA Ampere or newer (A100, H100, RTX 40xx)\n- Compute capability ≥ 8.0\n\n### Triton (Linux only)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=True  # Linux only\n)\n```\n\n**Performance**: 1.2-1.5× faster than CUDA backend\n\n## Integration with transformers\n\n### Direct transformers usage\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load quantized model (transformers auto-detects GPTQ)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-13B-Chat-GPTQ\",\n    device_map=\"auto\",\n    trust_remote_code=False\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"TheBloke\u002FLlama-2-13B-Chat-GPTQ\")\n\n# Use like any transformers model\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_new_tokens=100)\n```\n\n### QLoRA fine-tuning (GPTQ + LoRA)\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model\n\n# Load GPTQ model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-7B-GPTQ\",\n    device_map=\"auto\"\n)\n\n# Prepare for LoRA training\nmodel = prepare_model_for_kbit_training(model)\n\n# LoRA config\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"v_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\n# Add LoRA adapters\nmodel = get_peft_model(model, lora_config)\n\n# Fine-tune (memory efficient!)\n# 70B model trainable on single A100 80GB\n```\n\n## Performance benchmarks\n\n### Memory reduction\n\n| Model | FP16 | GPTQ 4-bit | Reduction |\n|-------|------|------------|-----------|\n| Llama 2-7B | 14 GB | 3.5 GB | 4× |\n| Llama 2-13B | 26 GB | 6.5 GB | 4× |\n| Llama 2-70B | 140 GB | 35 GB | 4× |\n| Llama 3-405B | 810 GB | 203 GB | 4× |\n\n**Enables**:\n- 70B on single A100 80GB (vs 2× A100 needed for FP16)\n- 405B on 3× A100 80GB (vs 11× A100 needed for FP16)\n- 13B on RTX 4090 24GB (vs OOM with FP16)\n\n### Inference speed (Llama 2-7B, A100)\n\n| Precision | Tokens\u002Fsec | vs FP16 |\n|-----------|------------|---------|\n| FP16 | 25 tok\u002Fs | 1× |\n| GPTQ 4-bit (CUDA) | 85 tok\u002Fs | 3.4× |\n| GPTQ 4-bit (ExLlama) | 105 tok\u002Fs | 4.2× |\n| GPTQ 4-bit (Marlin) | 120 tok\u002Fs | 4.8× |\n\n### Accuracy (perplexity on WikiText-2)\n\n| Model | FP16 | GPTQ 4-bit (g=128) | Degradation |\n|-------|------|---------------------|-------------|\n| Llama 2-7B | 5.47 | 5.55 | +1.5% |\n| Llama 2-13B | 4.88 | 4.95 | +1.4% |\n| Llama 2-70B | 3.32 | 3.38 | +1.8% |\n\n**Excellent quality preservation** - less than 2% degradation!\n\n## Common patterns\n\n### Multi-GPU deployment\n\n```python\n# Automatic device mapping\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-GPTQ\",\n    device_map=\"auto\",  # Automatically split across GPUs\n    max_memory={0: \"40GB\", 1: \"40GB\"}  # Limit per GPU\n)\n\n# Manual device mapping\ndevice_map = {\n    \"model.embed_tokens\": 0,\n    \"model.layers.0-39\": 0,  # First 40 layers on GPU 0\n    \"model.layers.40-79\": 1,  # Last 40 layers on GPU 1\n    \"model.norm\": 1,\n    \"lm_head\": 1\n}\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device_map=device_map\n)\n```\n\n### CPU offloading\n\n```python\n# Offload some layers to CPU (for very large models)\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-405B-GPTQ\",\n    device_map=\"auto\",\n    max_memory={\n        0: \"80GB\",  # GPU 0\n        1: \"80GB\",  # GPU 1\n        2: \"80GB\",  # GPU 2\n        \"cpu\": \"200GB\"  # Offload overflow to CPU\n    }\n)\n```\n\n### Batch inference\n\n```python\n# Process multiple prompts efficiently\nprompts = [\n    \"Explain AI\",\n    \"Explain ML\",\n    \"Explain DL\"\n]\n\ninputs = tokenizer(prompts, return_tensors=\"pt\", padding=True).to(\"cuda\")\n\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=100,\n    pad_token_id=tokenizer.eos_token_id\n)\n\nfor i, output in enumerate(outputs):\n    print(f\"Prompt {i}: {tokenizer.decode(output)}\")\n```\n\n## Finding pre-quantized models\n\n**TheBloke on HuggingFace**:\n- https:\u002F\u002Fhuggingface.co\u002FTheBloke\n- 1000+ models in GPTQ format\n- Multiple group sizes (32, 128)\n- Both CUDA and Marlin formats\n\n**Search**:\n```bash\n# Find GPTQ models on HuggingFace\nhttps:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n```\n\n**Download**:\n```python\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Automatically downloads from HuggingFace\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-Chat-GPTQ\",\n    device=\"cuda:0\"\n)\n```\n\n## Supported models\n\n- **LLaMA family**: Llama 2, Llama 3, Code Llama\n- **Mistral**: Mistral 7B, Mixtral 8x7B, 8x22B\n- **Qwen**: Qwen, Qwen2, QwQ\n- **DeepSeek**: V2, V3\n- **Phi**: Phi-2, Phi-3\n- **Yi, Falcon, BLOOM, OPT**\n- **100+ models** on HuggingFace\n\n## References\n\n- **[Calibration Guide](references\u002Fcalibration.md)** - Dataset selection, quantization process, quality optimization\n- **[Integration Guide](references\u002Fintegration.md)** - Transformers, PEFT, vLLM, TensorRT-LLM\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, performance optimization\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FAutoGPTQ\u002FAutoGPTQ\n- **Paper**: GPTQ: Accurate Post-Training Quantization (arXiv:2210.17323)\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002Fautogptq\n\n\n",{"name":5,"tags":24,"author":35,"license":15,"version":36,"description":7,"dependencies":37},[25,26,27,28,29,30,31,32,33,34],"Optimization","GPTQ","Quantization","4-Bit","Post-Training","Memory Optimization","Consumer GPUs","Fast Inference","QLoRA","Group-Wise Quantization","Orchestra Research","1.0.0",[38,39,40,41],"auto-gptq","transformers","optimum","peft","master","c247302c836de6438d746b45670990eb71691166","skills_collected\u002FOrchestra-Research--AI-research-SKILLs--10-optimization__gptq",1,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool","https:\u002F\u002Fgithub.com\u002Fqhjqhj00","chien.io","2026-05-11 15:07:59",{"source":8,"skillName":5,"commands":51,"prompt":61,"zipAvailable":62},[52,55,58],{"tool":53,"command":54},"npx","npx skills add qhjqhj00\u002Fresearch-skills-pool --skill gptq",{"tool":56,"command":57},"bunx","bunx skills add qhjqhj00\u002Fresearch-skills-pool --skill gptq",{"tool":59,"command":60},"pnpm","pnpm dlx skills add qhjqhj00\u002Fresearch-skills-pool --skill gptq","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--10-optimization__gptq\u002FSKILL.md",true,[64,70,74],{"path":65,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":68,"sourceHtmlUrl":69},"references\u002Fcalibration.md","REFERENCE","参考文档",8164,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--10-optimization__gptq\u002Freferences\u002Fcalibration.md",{"path":71,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":72,"sourceHtmlUrl":73},"references\u002Fintegration.md",2797,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--10-optimization__gptq\u002Freferences\u002Fintegration.md",{"path":75,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":76,"sourceHtmlUrl":77},"references\u002Ftroubleshooting.md",1899,"https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Fskills_collected\u002FOrchestra-Research--AI-research-SKILLs--10-optimization__gptq\u002Freferences\u002Ftroubleshooting.md",[79,86,93,100,107,114],{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":19},"01M73JJB6EQPFPPVQK2Y9N67Q0","transformer-lens-interpretability","Provides guidance for mechanistic interpretability research using TransformerLens to inspect and manipulate transformer internals via HookPoints and activation caching. Use when reverse-engineering model algorithms, studying attention patterns, or performing activation patching experiments.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F130-transformer-lens\u002FSKILL.md",[],[],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":19},"01M73JHTANRRWEE9HP98TPEHAY","wiki-vitepress","Packages generated wiki Markdown into a VitePress static site with dark theme, dark-mode Mermaid diagrams with click-to-zoom, and production build output. Use when the user wants to create a browsable website from generated wiki pages.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F126-wiki-vitepress\u002FSKILL.md",[],[],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":19},"01M73JHJBMVFW46F1Y8K3SBF43","phoenix-observability","Open-source AI observability platform for LLM tracing, evaluation, and monitoring. Use when debugging LLM applications with detailed traces, running evaluations on datasets, or monitoring production AI systems with real-time insights.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F124-phoenix\u002FSKILL.md",[],[],{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":19},"01M73JHFPA3A2806PMP42ZN86Q","frontend-ui-dark-ts","Build dark-themed React applications using Tailwind CSS with custom theming, glassmorphism effects, and Framer Motion animations. Use when creating dashboards, admin panels, or data-rich interfaces with a refined dark aesthetic.","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F123-frontend-ui-dark-ts\u002FSKILL.md",[],[],{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":111,"categories":112,"keywords":113,"publishedAt":19},"01M73JFVBVK57NRJS49TT2VDCT","tensorboard","Visualize training metrics, debug models with histograms, compare experiments, visualize model graphs, and profile performance with TensorBoard - Google's ML visualization toolkit","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F113-tensorboard\u002FSKILL.md",[],[],{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":118,"licenseFileUrl":6,"sourceHtmlUrl":119,"categories":120,"keywords":121,"publishedAt":19},"01M73JFF6PK6ZCN41C89J7HS2B","markdown-mermaid-writing","Comprehensive markdown and Mermaid diagram writing skill. Use when creating any scientific document, report, analysis, or visualization. Establishes text-based diagrams as the default documentation standard with full style guides (markdown + mermaid), 24 diagram type references, and 9 document templates.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fqhjqhj00\u002Fresearch-skills-pool\u002Fblob\u002Fmaster\u002Ftop100-runs\u002F110-markdown-mermaid-writing\u002FSKILL.md",[],[]]