[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M72C898ECT7G0BGDX8AXF859":3,"skill-same:01M72C898ECT7G0BGDX8AXF859":69},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":35,"commitSha":36,"skillDirectory":37,"forksCount":13,"repositoryHtmlUrl":38,"repositoryHomepageUrl":6,"ownerHtmlUrl":39,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":40,"install":41,"resources":54},"01M72C898ECT7G0BGDX8AXF859","hqq-quantization",null,"Half-Quadratic Quantization for LLMs without calibration data. Use when quantizing models to 4\u002F3\u002F2-bit precision without needing calibration datasets, for fast quantization workflows, or when deploying with vLLM or HuggingFace Transformers.","Sansar35\u002Fclaude-agent","Sansar35","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F145020281?v=4","01M724CQ2SCC86MGNSD87D0D4K","01M724CQ3965G9WJ3QXE3AJ5YY",1,"MIT","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fhqq-quantization\u002FSKILL.md",[],[],"2026-11-10 18:39:20",false,"---\nname: hqq-quantization\ndescription: Half-Quadratic Quantization for LLMs without calibration data. Use when quantizing models to 4\u002F3\u002F2-bit precision without needing calibration datasets, for fast quantization workflows, or when deploying with vLLM or HuggingFace Transformers.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Quantization, HQQ, Optimization, Memory Efficiency, Inference, Model Compression]\ndependencies: [hqq>=0.2.0, torch>=2.0.0]\n---\n\n# HQQ - Half-Quadratic Quantization\n\nFast, calibration-free weight quantization supporting 8\u002F4\u002F3\u002F2\u002F1-bit precision with multiple optimized backends.\n\n## When to use HQQ\n\n**Use HQQ when:**\n- Quantizing models without calibration data (no dataset needed)\n- Need fast quantization (minutes vs hours for GPTQ\u002FAWQ)\n- Deploying with vLLM or HuggingFace Transformers\n- Fine-tuning quantized models with LoRA\u002FPEFT\n- Experimenting with extreme quantization (2-bit, 1-bit)\n\n**Key advantages:**\n- **No calibration**: Quantize any model instantly without sample data\n- **Multiple backends**: PyTorch, ATEN, TorchAO, Marlin, BitBlas for optimized inference\n- **Flexible precision**: 8\u002F4\u002F3\u002F2\u002F1-bit with configurable group sizes\n- **Framework integration**: Native HuggingFace and vLLM support\n- **PEFT compatible**: Fine-tune quantized models with LoRA\n\n**Use alternatives instead:**\n- **AWQ**: Need calibration-based accuracy, production serving\n- **GPTQ**: Maximum accuracy with calibration data available\n- **bitsandbytes**: Simple 8-bit\u002F4-bit without custom backends\n- **llama.cpp\u002FGGUF**: CPU inference, Apple Silicon deployment\n\n## Quick start\n\n### Installation\n\n```bash\npip install hqq\n\n# With specific backend\npip install hqq[torch]      # PyTorch backend\npip install hqq[torchao]    # TorchAO int4 backend\npip install hqq[bitblas]    # BitBlas backend\npip install hqq[marlin]     # Marlin backend\n```\n\n### Basic quantization\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig, HQQLinear\nimport torch.nn as nn\n\n# Configure quantization\nconfig = BaseQuantizeConfig(\n    nbits=4,           # 4-bit quantization\n    group_size=64,     # Group size for quantization\n    axis=1             # Quantize along output dimension\n)\n\n# Quantize a linear layer\nlinear = nn.Linear(4096, 4096)\nhqq_linear = HQQLinear(linear, config)\n\n# Use normally\noutput = hqq_linear(input_tensor)\n```\n\n### Quantize full model with HuggingFace\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Configure HQQ\nquantization_config = HqqConfig(\n    nbits=4,\n    group_size=64,\n    axis=1\n)\n\n# Load and quantize\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quantization_config,\n    device_map=\"auto\"\n)\n\n# Model is quantized and ready to use\n```\n\n## Core concepts\n\n### Quantization configuration\n\nHQQ uses `BaseQuantizeConfig` to define quantization parameters:\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig\n\n# Standard 4-bit config\nconfig_4bit = BaseQuantizeConfig(\n    nbits=4,           # Bits per weight (1-8)\n    group_size=64,     # Weights per quantization group\n    axis=1             # 0=input dim, 1=output dim\n)\n\n# Aggressive 2-bit config\nconfig_2bit = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,     # Smaller groups for low-bit\n    axis=1\n)\n\n# Mixed precision per layer type\nlayer_configs = {\n    \"self_attn.q_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.k_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.v_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"mlp.gate_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.up_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.down_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n}\n```\n\n### HQQLinear layer\n\nThe core quantized layer that replaces `nn.Linear`:\n\n```python\nfrom hqq.core.quantize import HQQLinear\nimport torch\n\n# Create quantized layer\nlinear = torch.nn.Linear(4096, 4096)\nhqq_layer = HQQLinear(linear, config)\n\n# Access quantized weights\nW_q = hqq_layer.W_q           # Quantized weights\nscale = hqq_layer.scale       # Scale factors\nzero = hqq_layer.zero         # Zero points\n\n# Dequantize for inspection\nW_dequant = hqq_layer.dequantize()\n```\n\n### Backends\n\nHQQ supports multiple inference backends for different hardware:\n\n```python\nfrom hqq.core.quantize import HQQLinear\n\n# Available backends\nbackends = [\n    \"pytorch\",          # Pure PyTorch (default)\n    \"pytorch_compile\",  # torch.compile optimized\n    \"aten\",            # Custom CUDA kernels\n    \"torchao_int4\",    # TorchAO int4 matmul\n    \"gemlite\",         # GemLite CUDA kernels\n    \"bitblas\",         # BitBlas optimized\n    \"marlin\",          # Marlin 4-bit kernels\n]\n\n# Set backend globally\nHQQLinear.set_backend(\"torchao_int4\")\n\n# Or per layer\nhqq_layer.set_backend(\"marlin\")\n```\n\n**Backend selection guide:**\n| Backend | Best For | Requirements |\n|---------|----------|--------------|\n| pytorch | Compatibility | Any GPU |\n| pytorch_compile | Moderate speedup | torch>=2.0 |\n| aten | Good balance | CUDA GPU |\n| torchao_int4 | 4-bit inference | torchao installed |\n| marlin | Maximum 4-bit speed | Ampere+ GPU |\n| bitblas | Flexible bit-widths | bitblas installed |\n\n## HuggingFace integration\n\n### Load pre-quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load HQQ-quantized model from Hub\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# Use normally\ninputs = tokenizer(\"Hello, world!\", return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=50)\n```\n\n### Quantize and save\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Quantize\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Save quantized model\nmodel.save_pretrained(\".\u002Fllama-8b-hqq-4bit\")\n\n# Push to Hub\nmodel.push_to_hub(\"my-org\u002FLlama-3.1-8B-HQQ-4bit\")\n```\n\n### Mixed precision quantization\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Different precision per layer type\nconfig = HqqConfig(\n    nbits=4,\n    group_size=64,\n    # Attention layers: higher precision\n    # MLP layers: lower precision for memory savings\n    dynamic_config={\n        \"attn\": {\"nbits\": 4, \"group_size\": 64},\n        \"mlp\": {\"nbits\": 2, \"group_size\": 32}\n    }\n)\n```\n\n## vLLM integration\n\n### Serve HQQ models with vLLM\n\n```python\nfrom vllm import LLM, SamplingParams\n\n# Load HQQ-quantized model\nllm = LLM(\n    model=\"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    quantization=\"hqq\",\n    dtype=\"float16\"\n)\n\n# Generate\nsampling_params = SamplingParams(temperature=0.7, max_tokens=100)\noutputs = llm.generate([\"What is machine learning?\"], sampling_params)\n```\n\n### vLLM with custom HQQ config\n\n```python\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"meta-llama\u002FLlama-3.1-8B\",\n    quantization=\"hqq\",\n    quantization_config={\n        \"nbits\": 4,\n        \"group_size\": 64\n    }\n)\n```\n\n## PEFT\u002FLoRA fine-tuning\n\n### Fine-tune quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\nfrom peft import LoraConfig, get_peft_model\n\n# Load quantized model\nquant_config = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quant_config,\n    device_map=\"auto\"\n)\n\n# Apply LoRA\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\nmodel = get_peft_model(model, lora_config)\n\n# Train normally with Trainer or custom loop\n```\n\n### QLoRA-style training\n\n```python\nfrom transformers import TrainingArguments, Trainer\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fhqq-lora-output\",\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    learning_rate=2e-4,\n    num_train_epochs=3,\n    fp16=True,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    data_collator=data_collator\n)\n\ntrainer.train()\n```\n\n## Quantization workflows\n\n### Workflow 1: Quick model compression\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig\n\n# 1. Configure quantization\nconfig = HqqConfig(nbits=4, group_size=64)\n\n# 2. Load and quantize (no calibration needed!)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# 3. Verify quality\nprompt = \"The capital of France is\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=20)\nprint(tokenizer.decode(outputs[0]))\n\n# 4. Save\nmodel.save_pretrained(\".\u002Fllama-8b-hqq\")\ntokenizer.save_pretrained(\".\u002Fllama-8b-hqq\")\n```\n\n### Workflow 2: Optimize for inference speed\n\n```python\nfrom hqq.core.quantize import HQQLinear\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# 1. Quantize with optimal backend\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# 2. Set fast backend\nHQQLinear.set_backend(\"marlin\")  # or \"torchao_int4\"\n\n# 3. Compile for additional speedup\nimport torch\nmodel = torch.compile(model)\n\n# 4. Benchmark\nimport time\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(model.device)\nstart = time.time()\nfor _ in range(10):\n    model.generate(**inputs, max_new_tokens=100)\nprint(f\"Avg time: {(time.time() - start) \u002F 10:.2f}s\")\n```\n\n## Best practices\n\n1. **Start with 4-bit**: Best quality\u002Fsize tradeoff for most models\n2. **Use group_size=64**: Good balance; smaller for extreme quantization\n3. **Choose backend wisely**: Marlin for 4-bit Ampere+, TorchAO for flexibility\n4. **Verify quality**: Always test generation quality after quantization\n5. **Mixed precision**: Keep attention at higher precision, compress MLP more\n6. **PEFT training**: Use LoRA r=16-32 for good fine-tuning results\n\n## Common issues\n\n**Out of memory during quantization:**\n```python\n# Quantize layer-by-layer\nfrom hqq.models.hf.base import AutoHQQHFModel\n\nmodel = AutoHQQHFModel.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"sequential\"  # Load layers sequentially\n)\n```\n\n**Slow inference:**\n```python\n# Switch to optimized backend\nfrom hqq.core.quantize import HQQLinear\nHQQLinear.set_backend(\"marlin\")  # Requires Ampere+ GPU\n\n# Or compile\nmodel = torch.compile(model, mode=\"reduce-overhead\")\n```\n\n**Poor quality at 2-bit:**\n```python\n# Use smaller group size\nconfig = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,  # Smaller groups help at low bits\n    axis=1\n)\n```\n\n## References\n\n- **[Advanced Usage](references\u002Fadvanced-usage.md)** - Custom backends, mixed precision, optimization\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, debugging, benchmarks\n\n## Resources\n\n- **Repository**: https:\u002F\u002Fgithub.com\u002Fmobiusml\u002Fhqq\n- **Paper**: Half-Quadratic Quantization\n- **HuggingFace Models**: https:\u002F\u002Fhuggingface.co\u002Fmobiuslabsgmbh\n- **Version**: 0.2.0+\n- **License**: Apache 2.0\n","# HQQ - Half-Quadratic Quantization\n\nFast, calibration-free weight quantization supporting 8\u002F4\u002F3\u002F2\u002F1-bit precision with multiple optimized backends.\n\n## When to use HQQ\n\n**Use HQQ when:**\n- Quantizing models without calibration data (no dataset needed)\n- Need fast quantization (minutes vs hours for GPTQ\u002FAWQ)\n- Deploying with vLLM or HuggingFace Transformers\n- Fine-tuning quantized models with LoRA\u002FPEFT\n- Experimenting with extreme quantization (2-bit, 1-bit)\n\n**Key advantages:**\n- **No calibration**: Quantize any model instantly without sample data\n- **Multiple backends**: PyTorch, ATEN, TorchAO, Marlin, BitBlas for optimized inference\n- **Flexible precision**: 8\u002F4\u002F3\u002F2\u002F1-bit with configurable group sizes\n- **Framework integration**: Native HuggingFace and vLLM support\n- **PEFT compatible**: Fine-tune quantized models with LoRA\n\n**Use alternatives instead:**\n- **AWQ**: Need calibration-based accuracy, production serving\n- **GPTQ**: Maximum accuracy with calibration data available\n- **bitsandbytes**: Simple 8-bit\u002F4-bit without custom backends\n- **llama.cpp\u002FGGUF**: CPU inference, Apple Silicon deployment\n\n## Quick start\n\n### Installation\n\n```bash\npip install hqq\n\n# With specific backend\npip install hqq[torch]      # PyTorch backend\npip install hqq[torchao]    # TorchAO int4 backend\npip install hqq[bitblas]    # BitBlas backend\npip install hqq[marlin]     # Marlin backend\n```\n\n### Basic quantization\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig, HQQLinear\nimport torch.nn as nn\n\n# Configure quantization\nconfig = BaseQuantizeConfig(\n    nbits=4,           # 4-bit quantization\n    group_size=64,     # Group size for quantization\n    axis=1             # Quantize along output dimension\n)\n\n# Quantize a linear layer\nlinear = nn.Linear(4096, 4096)\nhqq_linear = HQQLinear(linear, config)\n\n# Use normally\noutput = hqq_linear(input_tensor)\n```\n\n### Quantize full model with HuggingFace\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Configure HQQ\nquantization_config = HqqConfig(\n    nbits=4,\n    group_size=64,\n    axis=1\n)\n\n# Load and quantize\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quantization_config,\n    device_map=\"auto\"\n)\n\n# Model is quantized and ready to use\n```\n\n## Core concepts\n\n### Quantization configuration\n\nHQQ uses `BaseQuantizeConfig` to define quantization parameters:\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig\n\n# Standard 4-bit config\nconfig_4bit = BaseQuantizeConfig(\n    nbits=4,           # Bits per weight (1-8)\n    group_size=64,     # Weights per quantization group\n    axis=1             # 0=input dim, 1=output dim\n)\n\n# Aggressive 2-bit config\nconfig_2bit = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,     # Smaller groups for low-bit\n    axis=1\n)\n\n# Mixed precision per layer type\nlayer_configs = {\n    \"self_attn.q_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.k_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.v_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"mlp.gate_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.up_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.down_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n}\n```\n\n### HQQLinear layer\n\nThe core quantized layer that replaces `nn.Linear`:\n\n```python\nfrom hqq.core.quantize import HQQLinear\nimport torch\n\n# Create quantized layer\nlinear = torch.nn.Linear(4096, 4096)\nhqq_layer = HQQLinear(linear, config)\n\n# Access quantized weights\nW_q = hqq_layer.W_q           # Quantized weights\nscale = hqq_layer.scale       # Scale factors\nzero = hqq_layer.zero         # Zero points\n\n# Dequantize for inspection\nW_dequant = hqq_layer.dequantize()\n```\n\n### Backends\n\nHQQ supports multiple inference backends for different hardware:\n\n```python\nfrom hqq.core.quantize import HQQLinear\n\n# Available backends\nbackends = [\n    \"pytorch\",          # Pure PyTorch (default)\n    \"pytorch_compile\",  # torch.compile optimized\n    \"aten\",            # Custom CUDA kernels\n    \"torchao_int4\",    # TorchAO int4 matmul\n    \"gemlite\",         # GemLite CUDA kernels\n    \"bitblas\",         # BitBlas optimized\n    \"marlin\",          # Marlin 4-bit kernels\n]\n\n# Set backend globally\nHQQLinear.set_backend(\"torchao_int4\")\n\n# Or per layer\nhqq_layer.set_backend(\"marlin\")\n```\n\n**Backend selection guide:**\n| Backend | Best For | Requirements |\n|---------|----------|--------------|\n| pytorch | Compatibility | Any GPU |\n| pytorch_compile | Moderate speedup | torch>=2.0 |\n| aten | Good balance | CUDA GPU |\n| torchao_int4 | 4-bit inference | torchao installed |\n| marlin | Maximum 4-bit speed | Ampere+ GPU |\n| bitblas | Flexible bit-widths | bitblas installed |\n\n## HuggingFace integration\n\n### Load pre-quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load HQQ-quantized model from Hub\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# Use normally\ninputs = tokenizer(\"Hello, world!\", return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=50)\n```\n\n### Quantize and save\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Quantize\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Save quantized model\nmodel.save_pretrained(\".\u002Fllama-8b-hqq-4bit\")\n\n# Push to Hub\nmodel.push_to_hub(\"my-org\u002FLlama-3.1-8B-HQQ-4bit\")\n```\n\n### Mixed precision quantization\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Different precision per layer type\nconfig = HqqConfig(\n    nbits=4,\n    group_size=64,\n    # Attention layers: higher precision\n    # MLP layers: lower precision for memory savings\n    dynamic_config={\n        \"attn\": {\"nbits\": 4, \"group_size\": 64},\n        \"mlp\": {\"nbits\": 2, \"group_size\": 32}\n    }\n)\n```\n\n## vLLM integration\n\n### Serve HQQ models with vLLM\n\n```python\nfrom vllm import LLM, SamplingParams\n\n# Load HQQ-quantized model\nllm = LLM(\n    model=\"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    quantization=\"hqq\",\n    dtype=\"float16\"\n)\n\n# Generate\nsampling_params = SamplingParams(temperature=0.7, max_tokens=100)\noutputs = llm.generate([\"What is machine learning?\"], sampling_params)\n```\n\n### vLLM with custom HQQ config\n\n```python\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"meta-llama\u002FLlama-3.1-8B\",\n    quantization=\"hqq\",\n    quantization_config={\n        \"nbits\": 4,\n        \"group_size\": 64\n    }\n)\n```\n\n## PEFT\u002FLoRA fine-tuning\n\n### Fine-tune quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\nfrom peft import LoraConfig, get_peft_model\n\n# Load quantized model\nquant_config = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quant_config,\n    device_map=\"auto\"\n)\n\n# Apply LoRA\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\nmodel = get_peft_model(model, lora_config)\n\n# Train normally with Trainer or custom loop\n```\n\n### QLoRA-style training\n\n```python\nfrom transformers import TrainingArguments, Trainer\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fhqq-lora-output\",\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    learning_rate=2e-4,\n    num_train_epochs=3,\n    fp16=True,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    data_collator=data_collator\n)\n\ntrainer.train()\n```\n\n## Quantization workflows\n\n### Workflow 1: Quick model compression\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig\n\n# 1. Configure quantization\nconfig = HqqConfig(nbits=4, group_size=64)\n\n# 2. Load and quantize (no calibration needed!)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# 3. Verify quality\nprompt = \"The capital of France is\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=20)\nprint(tokenizer.decode(outputs[0]))\n\n# 4. Save\nmodel.save_pretrained(\".\u002Fllama-8b-hqq\")\ntokenizer.save_pretrained(\".\u002Fllama-8b-hqq\")\n```\n\n### Workflow 2: Optimize for inference speed\n\n```python\nfrom hqq.core.quantize import HQQLinear\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# 1. Quantize with optimal backend\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# 2. Set fast backend\nHQQLinear.set_backend(\"marlin\")  # or \"torchao_int4\"\n\n# 3. Compile for additional speedup\nimport torch\nmodel = torch.compile(model)\n\n# 4. Benchmark\nimport time\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(model.device)\nstart = time.time()\nfor _ in range(10):\n    model.generate(**inputs, max_new_tokens=100)\nprint(f\"Avg time: {(time.time() - start) \u002F 10:.2f}s\")\n```\n\n## Best practices\n\n1. **Start with 4-bit**: Best quality\u002Fsize tradeoff for most models\n2. **Use group_size=64**: Good balance; smaller for extreme quantization\n3. **Choose backend wisely**: Marlin for 4-bit Ampere+, TorchAO for flexibility\n4. **Verify quality**: Always test generation quality after quantization\n5. **Mixed precision**: Keep attention at higher precision, compress MLP more\n6. **PEFT training**: Use LoRA r=16-32 for good fine-tuning results\n\n## Common issues\n\n**Out of memory during quantization:**\n```python\n# Quantize layer-by-layer\nfrom hqq.models.hf.base import AutoHQQHFModel\n\nmodel = AutoHQQHFModel.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"sequential\"  # Load layers sequentially\n)\n```\n\n**Slow inference:**\n```python\n# Switch to optimized backend\nfrom hqq.core.quantize import HQQLinear\nHQQLinear.set_backend(\"marlin\")  # Requires Ampere+ GPU\n\n# Or compile\nmodel = torch.compile(model, mode=\"reduce-overhead\")\n```\n\n**Poor quality at 2-bit:**\n```python\n# Use smaller group size\nconfig = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,  # Smaller groups help at low bits\n    axis=1\n)\n```\n\n## References\n\n- **[Advanced Usage](references\u002Fadvanced-usage.md)** - Custom backends, mixed precision, optimization\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, debugging, benchmarks\n\n## Resources\n\n- **Repository**: https:\u002F\u002Fgithub.com\u002Fmobiusml\u002Fhqq\n- **Paper**: Half-Quadratic Quantization\n- **HuggingFace Models**: https:\u002F\u002Fhuggingface.co\u002Fmobiuslabsgmbh\n- **Version**: 0.2.0+\n- **License**: Apache 2.0\n",{"name":5,"tags":23,"author":30,"license":14,"version":31,"description":7,"dependencies":32},[24,25,26,27,28,29],"Quantization","HQQ","Optimization","Memory Efficiency","Inference","Model Compression","Orchestra Research","1.0.0",[33,34],"hqq>=0.2.0","torch>=2.0.0","main","d046d088ce5a79a083f0abc0fe1b7ded73bc5b1f","claude-config\u002Fskills\u002Fhqq-quantization","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent","https:\u002F\u002Fgithub.com\u002FSansar35","2026-05-10 16:04:53",{"source":8,"skillName":5,"commands":42,"prompt":52,"zipAvailable":53},[43,46,49],{"tool":44,"command":45},"npx","npx skills add Sansar35\u002Fclaude-agent --skill hqq-quantization",{"tool":47,"command":48},"bunx","bunx skills add Sansar35\u002Fclaude-agent --skill hqq-quantization",{"tool":50,"command":51},"pnpm","pnpm dlx skills add Sansar35\u002Fclaude-agent --skill hqq-quantization","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fhqq-quantization\u002FSKILL.md",true,[55,61,65],{"path":56,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":59,"sourceHtmlUrl":60},".skill-source.json","REFERENCE","参考文档",473,"https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fhqq-quantization\u002F.skill-source.json",{"path":62,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":63,"sourceHtmlUrl":64},"references\u002Fadvanced-usage.md",14077,"https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fhqq-quantization\u002Freferences\u002Fadvanced-usage.md",{"path":66,"resourceType":57,"resourceTypeDesc":58,"sizeBytes":67,"sourceHtmlUrl":68},"references\u002Ftroubleshooting.md",11086,"https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fhqq-quantization\u002Freferences\u002Ftroubleshooting.md",[70,77,84,91,98,106],{"skillNo":71,"name":72,"nameZh":6,"description":73,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":74,"categories":75,"keywords":76,"publishedAt":18},"01M72DVAN1JYEA302MT0C5MGD6","zustand-store-ts","Create Zustand stores following established patterns with proper TypeScript types and middleware.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzustand-store-ts\u002FSKILL.md",[],[],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":18},"01M72DV8J5NBRRM4RPDFRV95N7","zod-validation-expert","Expert in Zod — TypeScript-first schema validation. Covers parsing, custom errors, refinements, type inference, and integration with React Hook Form, Next.js, and tRPC.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzod-validation-expert\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":18},"01M72DV5EMHW0BHP9BZ167MDCB","zinc-database","Access ZINC (230M+ purchasable compounds). Search by ZINC ID\u002FSMILES, similarity searches, 3D-ready structures for docking, analog discovery, for virtual screening and drug discovery.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzinc-database\u002FSKILL.md",[],[],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":18},"01M72DV216QY0E693ZPEVQ0DBS","zarr-python","Chunked N-D arrays for cloud storage. Compressed arrays, parallel I\u002FO, S3\u002FGCS integration, NumPy\u002FDask\u002FXarray compatible, for large-scale scientific computing pipelines.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzarr-python\u002FSKILL.md",[],[],{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":102,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":18},"01M72DTXQREMCRKA8NHCVKHN4W","zapier-workflows","Manage and trigger pre-built Zapier workflows and MCP tool orchestration. Use when user mentions workflows, Zaps, automations, daily digest, research, search, lead tracking, expenses, or asks to \"run\" any process. Also handles Perplexity-based research and Google Sheets data tracking.","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzapier-workflows\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzapier-workflows\u002FSKILL.md",[],[],{"skillNo":107,"name":108,"nameZh":6,"description":109,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":18},"01M72DTT17TBSJ0Y8J2XZ4C9TZ","zapier-make-patterns","No-code automation democratizes workflow building. Zapier and Make (formerly Integromat) let non-developers automate business processes without writing code. But no-code doesn't mean no-complexity - these platforms have their own patterns, pitfalls, and breaking points.  This skill covers when to use which platform, how to build reliable automations, and when to graduate to code-based solutions. Key insight: Zapier optimizes for simplicity and integrations (7000+ apps), Make optimizes for power","https:\u002F\u002Fgithub.com\u002FSansar35\u002Fclaude-agent\u002Fblob\u002Fmain\u002Fclaude-config\u002Fskills\u002Fzapier-make-patterns\u002FSKILL.md",[],[]]