[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7AY0JDCKC027WBNPRG5VSWV":3,"skill-same:01M7AY0JDCKC027WBNPRG5VSWV":58},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":14,"repositoryHtmlUrl":27,"repositoryHomepageUrl":6,"ownerHtmlUrl":28,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":29,"install":30,"resources":43},"01M7AY0JDCKC027WBNPRG5VSWV","gptq",null,"Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with \u003C2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.","SiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot","SiddhamShah972","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F98964854?v=4","01M7AWVFA8QBCFE7PRDT5XBB2E","Siddham Shah","01M7AWVFB82AN6H6BNQSWEF402",0,"MIT","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fgptq\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\ndescription: \"Post-training 4-bit quantization for LLMs with minimal accuracy loss. Use for deploying large models (70B, 405B) on consumer GPUs, when you need 4× memory reduction with \u003C2% perplexity degradation, or for faster inference (3-4× speedup) vs FP16. Integrates with transformers and PEFT for QLoRA fine-tuning.\"\nlicense: \"MIT\"\n---\n# GPTQ (Generative Pre-trained Transformer Quantization)\n\nPost-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization.\n\n## When to use GPTQ\n\n**Use GPTQ when:**\n- Need to fit large models (70B+) on limited GPU memory\n- Want 4× memory reduction with \u003C2% accuracy loss\n- Deploying on consumer GPUs (RTX 4090, 3090)\n- Need faster inference (3-4× speedup vs FP16)\n\n**Use AWQ instead when:**\n- Need slightly better accuracy (\u003C1% loss)\n- Have newer GPUs (Ampere, Ada)\n- Want Marlin kernel support (2× faster on some GPUs)\n\n**Use bitsandbytes instead when:**\n- Need simple integration with transformers\n- Want 8-bit quantization (less compression, better quality)\n- Don't need pre-quantized model files\n\n## Quick start\n\n### Installation\n\n```bash\n# Install AutoGPTQ\npip install auto-gptq\n\n# With Triton (Linux only, faster)\npip install auto-gptq[triton]\n\n# With CUDA extensions (faster)\npip install auto-gptq --no-build-isolation\n\n# Full installation\npip install auto-gptq transformers accelerate\n```\n\n### Load pre-quantized model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Load quantized model from HuggingFace\nmodel_name = \"TheBloke\u002FLlama-2-7B-Chat-GPTQ\"\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=False  # Set True on Linux for speed\n)\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Generate\nprompt = \"Explain quantum computing\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda:0\")\noutputs = model.generate(**inputs, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\n### Quantize your own model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel_name = \"meta-llama\u002FLlama-2-7b-chat-hf\"\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Quantization config\nquantize_config = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Group size (recommended: 128)\n    desc_act=False,      # Activation order (False for CUDA kernel)\n    damp_percent=0.01    # Dampening factor\n)\n\n# Load model for quantization\nmodel = AutoGPTQForCausalLM.from_pretrained(\n    model_name,\n    quantize_config=quantize_config\n)\n\n# Prepare calibration data\ndataset = load_dataset(\"c4\", split=\"train\", streaming=True)\ncalibration_data = [\n    tokenizer(example[\"text\"])[\"input_ids\"][:512]\n    for example in dataset.take(128)\n]\n\n# Quantize\nmodel.quantize(calibration_data)\n\n# Save quantized model\nmodel.save_quantized(\"llama-2-7b-gptq\")\ntokenizer.save_pretrained(\"llama-2-7b-gptq\")\n\n# Push to HuggingFace\nmodel.push_to_hub(\"username\u002Fllama-2-7b-gptq\")\n```\n\n## Group-wise quantization\n\n**How GPTQ works**:\n1. **Group weights**: Divide each weight matrix into groups (typically 128 elements)\n2. **Quantize per-group**: Each group has its own scale\u002Fzero-point\n3. **Minimize error**: Uses Hessian information to minimize quantization error\n4. **Result**: 4-bit weights with near-FP16 accuracy\n\n**Group size trade-off**:\n\n| Group Size | Model Size | Accuracy | Speed | Recommendation |\n|------------|------------|----------|-------|----------------|\n| -1 (per-column) | Smallest | Best | Slowest | Research only |\n| 32 | Smaller | Better | Slower | High accuracy needed |\n| **128** | Medium | Good | **Fast** | **Recommended default** |\n| 256 | Larger | Lower | Faster | Speed critical |\n| 1024 | Largest | Lowest | Fastest | Not recommended |\n\n**Example**:\n```\nWeight matrix: [1024, 4096] = 4.2M elements\n\nGroup size = 128:\n- Groups: 4.2M \u002F 128 = 32,768 groups\n- Each group: own 4-bit scale + zero-point\n- Result: Better granularity → better accuracy\n```\n\n## Quantization configurations\n\n### Standard 4-bit (recommended)\n\n```python\nfrom auto_gptq import BaseQuantizeConfig\n\nconfig = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Standard group size\n    desc_act=False,      # Faster CUDA kernel\n    damp_percent=0.01    # Dampening factor\n)\n```\n\n**Performance**:\n- Memory: 4× reduction (70B model: 140GB → 35GB)\n- Accuracy: ~1.5% perplexity increase\n- Speed: 3-4× faster than FP16\n\n### High accuracy (3-bit with larger groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=3,              # 3-bit (more compression)\n    group_size=128,      # Keep standard group size\n    desc_act=True,       # Better accuracy (slower)\n    damp_percent=0.01\n)\n```\n\n**Trade-off**:\n- Memory: 5× reduction\n- Accuracy: ~3% perplexity increase\n- Speed: 5× faster (but less accurate)\n\n### Maximum accuracy (4-bit with small groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=32,       # Smaller groups (better accuracy)\n    desc_act=True,       # Activation reordering\n    damp_percent=0.005   # Lower dampening\n)\n```\n\n**Trade-off**:\n- Memory: 3.5× reduction (slightly larger)\n- Accuracy: ~0.8% perplexity increase (best)\n- Speed: 2-3× faster (kernel overhead)\n\n## Kernel backends\n\n### ExLlamaV2 (default, fastest)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_exllama=True,      # Use ExLlamaV2\n    exllama_config={\"version\": 2}\n)\n```\n\n**Performance**: 1.5-2× faster than Triton\n\n### Marlin (Ampere+ GPUs)\n\n```python\n# Quantize with Marlin format\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=128,\n    desc_act=False  # Required for Marlin\n)\n\nmodel.quantize(calibration_data, use_marlin=True)\n\n# Load with Marlin\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_marlin=True  # 2× faster on A100\u002FH100\n)\n```\n\n**Requirements**:\n- NVIDIA Ampere or newer (A100, H100, RTX 40xx)\n- Compute capability ≥ 8.0\n\n### Triton (Linux only)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=True  # Linux only\n)\n```\n\n**Performance**: 1.2-1.5× faster than CUDA backend\n\n## Integration with transformers\n\n### Direct transformers usage\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load quantized model (transformers auto-detects GPTQ)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-13B-Chat-GPTQ\",\n    device_map=\"auto\",\n    trust_remote_code=False\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"TheBloke\u002FLlama-2-13B-Chat-GPTQ\")\n\n# Use like any transformers model\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_new_tokens=100)\n```\n\n### QLoRA fine-tuning (GPTQ + LoRA)\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model\n\n# Load GPTQ model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-7B-GPTQ\",\n    device_map=\"auto\"\n)\n\n# Prepare for LoRA training\nmodel = prepare_model_for_kbit_training(model)\n\n# LoRA config\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"v_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\n# Add LoRA adapters\nmodel = get_peft_model(model, lora_config)\n\n# Fine-tune (memory efficient!)\n# 70B model trainable on single A100 80GB\n```\n\n## Performance benchmarks\n\n### Memory reduction\n\n| Model | FP16 | GPTQ 4-bit | Reduction |\n|-------|------|------------|-----------|\n| Llama 2-7B | 14 GB | 3.5 GB | 4× |\n| Llama 2-13B | 26 GB | 6.5 GB | 4× |\n| Llama 2-70B | 140 GB | 35 GB | 4× |\n| Llama 3-405B | 810 GB | 203 GB | 4× |\n\n**Enables**:\n- 70B on single A100 80GB (vs 2× A100 needed for FP16)\n- 405B on 3× A100 80GB (vs 11× A100 needed for FP16)\n- 13B on RTX 4090 24GB (vs OOM with FP16)\n\n### Inference speed (Llama 2-7B, A100)\n\n| Precision | Tokens\u002Fsec | vs FP16 |\n|-----------|------------|---------|\n| FP16 | 25 tok\u002Fs | 1× |\n| GPTQ 4-bit (CUDA) | 85 tok\u002Fs | 3.4× |\n| GPTQ 4-bit (ExLlama) | 105 tok\u002Fs | 4.2× |\n| GPTQ 4-bit (Marlin) | 120 tok\u002Fs | 4.8× |\n\n### Accuracy (perplexity on WikiText-2)\n\n| Model | FP16 | GPTQ 4-bit (g=128) | Degradation |\n|-------|------|---------------------|-------------|\n| Llama 2-7B | 5.47 | 5.55 | +1.5% |\n| Llama 2-13B | 4.88 | 4.95 | +1.4% |\n| Llama 2-70B | 3.32 | 3.38 | +1.8% |\n\n**Excellent quality preservation** - less than 2% degradation!\n\n## Common patterns\n\n### Multi-GPU deployment\n\n```python\n# Automatic device mapping\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-GPTQ\",\n    device_map=\"auto\",  # Automatically split across GPUs\n    max_memory={0: \"40GB\", 1: \"40GB\"}  # Limit per GPU\n)\n\n# Manual device mapping\ndevice_map = {\n    \"model.embed_tokens\": 0,\n    \"model.layers.0-39\": 0,  # First 40 layers on GPU 0\n    \"model.layers.40-79\": 1,  # Last 40 layers on GPU 1\n    \"model.norm\": 1,\n    \"lm_head\": 1\n}\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device_map=device_map\n)\n```\n\n### CPU offloading\n\n```python\n# Offload some layers to CPU (for very large models)\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-405B-GPTQ\",\n    device_map=\"auto\",\n    max_memory={\n        0: \"80GB\",  # GPU 0\n        1: \"80GB\",  # GPU 1\n        2: \"80GB\",  # GPU 2\n        \"cpu\": \"200GB\"  # Offload overflow to CPU\n    }\n)\n```\n\n### Batch inference\n\n```python\n# Process multiple prompts efficiently\nprompts = [\n    \"Explain AI\",\n    \"Explain ML\",\n    \"Explain DL\"\n]\n\ninputs = tokenizer(prompts, return_tensors=\"pt\", padding=True).to(\"cuda\")\n\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=100,\n    pad_token_id=tokenizer.eos_token_id\n)\n\nfor i, output in enumerate(outputs):\n    print(f\"Prompt {i}: {tokenizer.decode(output)}\")\n```\n\n## Finding pre-quantized models\n\n**TheBloke on HuggingFace**:\n- https:\u002F\u002Fhuggingface.co\u002FTheBloke\n- 1000+ models in GPTQ format\n- Multiple group sizes (32, 128)\n- Both CUDA and Marlin formats\n\n**Search**:\n```bash\n# Find GPTQ models on HuggingFace\nhttps:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n```\n\n**Download**:\n```python\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Automatically downloads from HuggingFace\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-Chat-GPTQ\",\n    device=\"cuda:0\"\n)\n```\n\n## Supported models\n\n- **LLaMA family**: Llama 2, Llama 3, Code Llama\n- **Mistral**: Mistral 7B, Mixtral 8x7B, 8x22B\n- **Qwen**: Qwen, Qwen2, QwQ\n- **DeepSeek**: V2, V3\n- **Phi**: Phi-2, Phi-3\n- **Yi, Falcon, BLOOM, OPT**\n- **100+ models** on HuggingFace\n\n## References\n\n- **[Calibration Guide](references\u002Fcalibration.md)** - Dataset selection, quantization process, quality optimization\n- **[Integration Guide](references\u002Fintegration.md)** - Transformers, PEFT, vLLM, TensorRT-LLM\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, performance optimization\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FAutoGPTQ\u002FAutoGPTQ\n- **Paper**: GPTQ: Accurate Post-Training Quantization (arXiv:2210.17323)\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002Fautogptq\n\n\n","# GPTQ (Generative Pre-trained Transformer Quantization)\n\nPost-training quantization method that compresses LLMs to 4-bit with minimal accuracy loss using group-wise quantization.\n\n## When to use GPTQ\n\n**Use GPTQ when:**\n- Need to fit large models (70B+) on limited GPU memory\n- Want 4× memory reduction with \u003C2% accuracy loss\n- Deploying on consumer GPUs (RTX 4090, 3090)\n- Need faster inference (3-4× speedup vs FP16)\n\n**Use AWQ instead when:**\n- Need slightly better accuracy (\u003C1% loss)\n- Have newer GPUs (Ampere, Ada)\n- Want Marlin kernel support (2× faster on some GPUs)\n\n**Use bitsandbytes instead when:**\n- Need simple integration with transformers\n- Want 8-bit quantization (less compression, better quality)\n- Don't need pre-quantized model files\n\n## Quick start\n\n### Installation\n\n```bash\n# Install AutoGPTQ\npip install auto-gptq\n\n# With Triton (Linux only, faster)\npip install auto-gptq[triton]\n\n# With CUDA extensions (faster)\npip install auto-gptq --no-build-isolation\n\n# Full installation\npip install auto-gptq transformers accelerate\n```\n\n### Load pre-quantized model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Load quantized model from HuggingFace\nmodel_name = \"TheBloke\u002FLlama-2-7B-Chat-GPTQ\"\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=False  # Set True on Linux for speed\n)\n\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Generate\nprompt = \"Explain quantum computing\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(\"cuda:0\")\noutputs = model.generate(**inputs, max_new_tokens=200)\nprint(tokenizer.decode(outputs[0]))\n```\n\n### Quantize your own model\n\n```python\nfrom transformers import AutoTokenizer\nfrom auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\nfrom datasets import load_dataset\n\n# Load model\nmodel_name = \"meta-llama\u002FLlama-2-7b-chat-hf\"\ntokenizer = AutoTokenizer.from_pretrained(model_name)\n\n# Quantization config\nquantize_config = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Group size (recommended: 128)\n    desc_act=False,      # Activation order (False for CUDA kernel)\n    damp_percent=0.01    # Dampening factor\n)\n\n# Load model for quantization\nmodel = AutoGPTQForCausalLM.from_pretrained(\n    model_name,\n    quantize_config=quantize_config\n)\n\n# Prepare calibration data\ndataset = load_dataset(\"c4\", split=\"train\", streaming=True)\ncalibration_data = [\n    tokenizer(example[\"text\"])[\"input_ids\"][:512]\n    for example in dataset.take(128)\n]\n\n# Quantize\nmodel.quantize(calibration_data)\n\n# Save quantized model\nmodel.save_quantized(\"llama-2-7b-gptq\")\ntokenizer.save_pretrained(\"llama-2-7b-gptq\")\n\n# Push to HuggingFace\nmodel.push_to_hub(\"username\u002Fllama-2-7b-gptq\")\n```\n\n## Group-wise quantization\n\n**How GPTQ works**:\n1. **Group weights**: Divide each weight matrix into groups (typically 128 elements)\n2. **Quantize per-group**: Each group has its own scale\u002Fzero-point\n3. **Minimize error**: Uses Hessian information to minimize quantization error\n4. **Result**: 4-bit weights with near-FP16 accuracy\n\n**Group size trade-off**:\n\n| Group Size | Model Size | Accuracy | Speed | Recommendation |\n|------------|------------|----------|-------|----------------|\n| -1 (per-column) | Smallest | Best | Slowest | Research only |\n| 32 | Smaller | Better | Slower | High accuracy needed |\n| **128** | Medium | Good | **Fast** | **Recommended default** |\n| 256 | Larger | Lower | Faster | Speed critical |\n| 1024 | Largest | Lowest | Fastest | Not recommended |\n\n**Example**:\n```\nWeight matrix: [1024, 4096] = 4.2M elements\n\nGroup size = 128:\n- Groups: 4.2M \u002F 128 = 32,768 groups\n- Each group: own 4-bit scale + zero-point\n- Result: Better granularity → better accuracy\n```\n\n## Quantization configurations\n\n### Standard 4-bit (recommended)\n\n```python\nfrom auto_gptq import BaseQuantizeConfig\n\nconfig = BaseQuantizeConfig(\n    bits=4,              # 4-bit quantization\n    group_size=128,      # Standard group size\n    desc_act=False,      # Faster CUDA kernel\n    damp_percent=0.01    # Dampening factor\n)\n```\n\n**Performance**:\n- Memory: 4× reduction (70B model: 140GB → 35GB)\n- Accuracy: ~1.5% perplexity increase\n- Speed: 3-4× faster than FP16\n\n### High accuracy (3-bit with larger groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=3,              # 3-bit (more compression)\n    group_size=128,      # Keep standard group size\n    desc_act=True,       # Better accuracy (slower)\n    damp_percent=0.01\n)\n```\n\n**Trade-off**:\n- Memory: 5× reduction\n- Accuracy: ~3% perplexity increase\n- Speed: 5× faster (but less accurate)\n\n### Maximum accuracy (4-bit with small groups)\n\n```python\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=32,       # Smaller groups (better accuracy)\n    desc_act=True,       # Activation reordering\n    damp_percent=0.005   # Lower dampening\n)\n```\n\n**Trade-off**:\n- Memory: 3.5× reduction (slightly larger)\n- Accuracy: ~0.8% perplexity increase (best)\n- Speed: 2-3× faster (kernel overhead)\n\n## Kernel backends\n\n### ExLlamaV2 (default, fastest)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_exllama=True,      # Use ExLlamaV2\n    exllama_config={\"version\": 2}\n)\n```\n\n**Performance**: 1.5-2× faster than Triton\n\n### Marlin (Ampere+ GPUs)\n\n```python\n# Quantize with Marlin format\nconfig = BaseQuantizeConfig(\n    bits=4,\n    group_size=128,\n    desc_act=False  # Required for Marlin\n)\n\nmodel.quantize(calibration_data, use_marlin=True)\n\n# Load with Marlin\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_marlin=True  # 2× faster on A100\u002FH100\n)\n```\n\n**Requirements**:\n- NVIDIA Ampere or newer (A100, H100, RTX 40xx)\n- Compute capability ≥ 8.0\n\n### Triton (Linux only)\n\n```python\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device=\"cuda:0\",\n    use_triton=True  # Linux only\n)\n```\n\n**Performance**: 1.2-1.5× faster than CUDA backend\n\n## Integration with transformers\n\n### Direct transformers usage\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load quantized model (transformers auto-detects GPTQ)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-13B-Chat-GPTQ\",\n    device_map=\"auto\",\n    trust_remote_code=False\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"TheBloke\u002FLlama-2-13B-Chat-GPTQ\")\n\n# Use like any transformers model\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_new_tokens=100)\n```\n\n### QLoRA fine-tuning (GPTQ + LoRA)\n\n```python\nfrom transformers import AutoModelForCausalLM\nfrom peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model\n\n# Load GPTQ model\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"TheBloke\u002FLlama-2-7B-GPTQ\",\n    device_map=\"auto\"\n)\n\n# Prepare for LoRA training\nmodel = prepare_model_for_kbit_training(model)\n\n# LoRA config\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"v_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\n# Add LoRA adapters\nmodel = get_peft_model(model, lora_config)\n\n# Fine-tune (memory efficient!)\n# 70B model trainable on single A100 80GB\n```\n\n## Performance benchmarks\n\n### Memory reduction\n\n| Model | FP16 | GPTQ 4-bit | Reduction |\n|-------|------|------------|-----------|\n| Llama 2-7B | 14 GB | 3.5 GB | 4× |\n| Llama 2-13B | 26 GB | 6.5 GB | 4× |\n| Llama 2-70B | 140 GB | 35 GB | 4× |\n| Llama 3-405B | 810 GB | 203 GB | 4× |\n\n**Enables**:\n- 70B on single A100 80GB (vs 2× A100 needed for FP16)\n- 405B on 3× A100 80GB (vs 11× A100 needed for FP16)\n- 13B on RTX 4090 24GB (vs OOM with FP16)\n\n### Inference speed (Llama 2-7B, A100)\n\n| Precision | Tokens\u002Fsec | vs FP16 |\n|-----------|------------|---------|\n| FP16 | 25 tok\u002Fs | 1× |\n| GPTQ 4-bit (CUDA) | 85 tok\u002Fs | 3.4× |\n| GPTQ 4-bit (ExLlama) | 105 tok\u002Fs | 4.2× |\n| GPTQ 4-bit (Marlin) | 120 tok\u002Fs | 4.8× |\n\n### Accuracy (perplexity on WikiText-2)\n\n| Model | FP16 | GPTQ 4-bit (g=128) | Degradation |\n|-------|------|---------------------|-------------|\n| Llama 2-7B | 5.47 | 5.55 | +1.5% |\n| Llama 2-13B | 4.88 | 4.95 | +1.4% |\n| Llama 2-70B | 3.32 | 3.38 | +1.8% |\n\n**Excellent quality preservation** - less than 2% degradation!\n\n## Common patterns\n\n### Multi-GPU deployment\n\n```python\n# Automatic device mapping\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-GPTQ\",\n    device_map=\"auto\",  # Automatically split across GPUs\n    max_memory={0: \"40GB\", 1: \"40GB\"}  # Limit per GPU\n)\n\n# Manual device mapping\ndevice_map = {\n    \"model.embed_tokens\": 0,\n    \"model.layers.0-39\": 0,  # First 40 layers on GPU 0\n    \"model.layers.40-79\": 1,  # Last 40 layers on GPU 1\n    \"model.norm\": 1,\n    \"lm_head\": 1\n}\n\nmodel = AutoGPTQForCausalLM.from_quantized(\n    model_name,\n    device_map=device_map\n)\n```\n\n### CPU offloading\n\n```python\n# Offload some layers to CPU (for very large models)\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-405B-GPTQ\",\n    device_map=\"auto\",\n    max_memory={\n        0: \"80GB\",  # GPU 0\n        1: \"80GB\",  # GPU 1\n        2: \"80GB\",  # GPU 2\n        \"cpu\": \"200GB\"  # Offload overflow to CPU\n    }\n)\n```\n\n### Batch inference\n\n```python\n# Process multiple prompts efficiently\nprompts = [\n    \"Explain AI\",\n    \"Explain ML\",\n    \"Explain DL\"\n]\n\ninputs = tokenizer(prompts, return_tensors=\"pt\", padding=True).to(\"cuda\")\n\noutputs = model.generate(\n    **inputs,\n    max_new_tokens=100,\n    pad_token_id=tokenizer.eos_token_id\n)\n\nfor i, output in enumerate(outputs):\n    print(f\"Prompt {i}: {tokenizer.decode(output)}\")\n```\n\n## Finding pre-quantized models\n\n**TheBloke on HuggingFace**:\n- https:\u002F\u002Fhuggingface.co\u002FTheBloke\n- 1000+ models in GPTQ format\n- Multiple group sizes (32, 128)\n- Both CUDA and Marlin formats\n\n**Search**:\n```bash\n# Find GPTQ models on HuggingFace\nhttps:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n```\n\n**Download**:\n```python\nfrom auto_gptq import AutoGPTQForCausalLM\n\n# Automatically downloads from HuggingFace\nmodel = AutoGPTQForCausalLM.from_quantized(\n    \"TheBloke\u002FLlama-2-70B-Chat-GPTQ\",\n    device=\"cuda:0\"\n)\n```\n\n## Supported models\n\n- **LLaMA family**: Llama 2, Llama 3, Code Llama\n- **Mistral**: Mistral 7B, Mixtral 8x7B, 8x22B\n- **Qwen**: Qwen, Qwen2, QwQ\n- **DeepSeek**: V2, V3\n- **Phi**: Phi-2, Phi-3\n- **Yi, Falcon, BLOOM, OPT**\n- **100+ models** on HuggingFace\n\n## References\n\n- **[Calibration Guide](references\u002Fcalibration.md)** - Dataset selection, quantization process, quality optimization\n- **[Integration Guide](references\u002Fintegration.md)** - Transformers, PEFT, vLLM, TensorRT-LLM\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, performance optimization\n\n## Resources\n\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FAutoGPTQ\u002FAutoGPTQ\n- **Paper**: GPTQ: Accurate Post-Training Quantization (arXiv:2210.17323)\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=gptq\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002Fautogptq\n\n\n",{"license":15,"description":7},"main","35cc2c5cb1f57477268c9f68a3a64363e68de836","agent\u002Fskills\u002Fgptq","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot","https:\u002F\u002Fgithub.com\u002FSiddhamShah972","2026-08-20 23:29:35",{"source":8,"skillName":5,"commands":31,"prompt":41,"zipAvailable":42},[32,35,38],{"tool":33,"command":34},"npx","npx skills add SiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot --skill gptq",{"tool":36,"command":37},"bunx","bunx skills add SiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot --skill gptq",{"tool":39,"command":40},"pnpm","pnpm dlx skills add SiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot --skill gptq","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fgptq\u002FSKILL.md",true,[44,50,54],{"path":45,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":48,"sourceHtmlUrl":49},"references\u002Fcalibration.md","REFERENCE","参考文档",8164,"https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fgptq\u002Freferences\u002Fcalibration.md",{"path":51,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":52,"sourceHtmlUrl":53},"references\u002Fintegration.md",2797,"https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fgptq\u002Freferences\u002Fintegration.md",{"path":55,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":56,"sourceHtmlUrl":57},"references\u002Ftroubleshooting.md",1899,"https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fgptq\u002Freferences\u002Ftroubleshooting.md",[59,67,74,81,90,98],{"skillNo":60,"name":61,"nameZh":6,"description":62,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":63,"sourceHtmlUrl":64,"categories":65,"keywords":66,"publishedAt":19},"01M7AZQ1V61JHXWJ370V37AC6D","workorai","WorkorAI talent marketplace skill: candidate job search and employer hiring with white-box match explanations via the WorkorAI MCP server (https:\u002F\u002Fworkorai.com\u002Fmcp). Use when the user asks to find a job, apply to jobs, respond to employer invitations, or when an employer wants to post jobs, search and evaluate candidates, invite them, and review applicants.","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fworkorai\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fworkorai\u002FSKILL.md",[],[],{"skillNo":68,"name":69,"nameZh":6,"description":70,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":71,"categories":72,"keywords":73,"publishedAt":19},"01M7AZP93C6FJ9EXT7QXFQ8HA6","whisper","OpenAI's general-purpose speech recognition model. Supports 99 languages, transcription, translation to English, and language identification. Six model sizes from tiny (39M params) to large (1550M params). Use for speech-to-text, podcast transcription, or multilingual audio processing. Best for robust, multilingual ASR.","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fwhisper\u002FSKILL.md",[],[],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":19},"01M7AZP4RVG6DEBAS7TPJTRP55","weights-and-biases","Track ML experiments with automatic logging, visualize training in real-time, optimize hyperparameters with sweeps, and manage model registry with W&B - collaborative MLOps platform","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fweights-and-biases\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":85,"licenseFileUrl":86,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":19},"01M7AZNYG21QDFHN1CSGRJ9XJM","webapp-testing","Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.","Apache-2.0","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fwebapp-testing\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fwebapp-testing\u002FSKILL.md",[],[],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":85,"licenseFileUrl":94,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":19},"01M7AZNCEWCKPENGX0ZT274M4Q","web-artifacts-builder","Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn\u002Fui). Use for complex artifacts requiring state management, routing, or shadcn\u002Fui components - not for simple single-file HTML\u002FJSX artifacts.","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fweb-artifacts-builder\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fweb-artifacts-builder\u002FSKILL.md",[],[],{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":104,"publishedAt":19},"01M7AZMQVK156KR60STTBGQGAK","vitest","Set up Vitest testing in any project — detects type (Cloudflare Workers, React, Node, library), generates vitest.config.ts, test setup, utilities, and a sample test. Covers mocking patterns, coverage config, workspace setup, Jest migration. Use whenever the user mentions adding tests, setting up Vitest, configuring tests, migrating from Jest, fixing testing infrastructure, or asks 'how do I test this'.","https:\u002F\u002Fgithub.com\u002FSiddhamShah972\u002FAgentic-Forecasting-Insight-Copilot\u002Fblob\u002Fmain\u002Fagent\u002Fskills\u002Fvitest\u002FSKILL.md",[],[]]