[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7EPMT10YRGF17YVBSVABA71":3,"skill-same:01M7EPMT10YRGF17YVBSVABA71":68},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":36,"commitSha":37,"skillDirectory":38,"forksCount":39,"repositoryHtmlUrl":40,"repositoryHomepageUrl":6,"ownerHtmlUrl":41,"ownerBlogUrl":42,"ownerDescription":6,"updatedAt":43,"install":44,"resources":57},"01M7EPMT10YRGF17YVBSVABA71","optimization-hqq",null,"Half-Quadratic Quantization for LLMs without calibration data. Use when quantizing models to 4\u002F3\u002F2-bit precision without needing calibration datasets, for fast quantization workflows, or when deploying with vLLM or HuggingFace Transformers.","TanNhatCMS\u002Fjarvis","TanNhatCMS","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F113796420?v=4","01M753MV17KG5CD3152G3WQEQY","KietNT","01M7EKEDYVYQ6PWVJJMZ911VMX",0,"MIT","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fams-imp-optimization-hqq\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: optimization-hqq\ndescription: Half-Quadratic Quantization for LLMs without calibration data. Use when quantizing models to 4\u002F3\u002F2-bit precision without needing calibration datasets, for fast quantization workflows, or when deploying with vLLM or HuggingFace Transformers.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Quantization, HQQ, Optimization, Memory Efficiency, Inference, Model Compression]\ndependencies: [hqq>=0.2.0, torch>=2.0.0]\n---\n\n# HQQ - Half-Quadratic Quantization\n\nFast, calibration-free weight quantization supporting 8\u002F4\u002F3\u002F2\u002F1-bit precision with multiple optimized backends.\n\n## When to use HQQ\n\n**Use HQQ when:**\n- Quantizing models without calibration data (no dataset needed)\n- Need fast quantization (minutes vs hours for GPTQ\u002FAWQ)\n- Deploying with vLLM or HuggingFace Transformers\n- Fine-tuning quantized models with LoRA\u002FPEFT\n- Experimenting with extreme quantization (2-bit, 1-bit)\n\n**Key advantages:**\n- **No calibration**: Quantize any model instantly without sample data\n- **Multiple backends**: PyTorch, ATEN, TorchAO, Marlin, BitBlas for optimized inference\n- **Flexible precision**: 8\u002F4\u002F3\u002F2\u002F1-bit with configurable group sizes\n- **Framework integration**: Native HuggingFace and vLLM support\n- **PEFT compatible**: Fine-tune quantized models with LoRA\n\n**Use alternatives instead:**\n- **AWQ**: Need calibration-based accuracy, production serving\n- **GPTQ**: Maximum accuracy with calibration data available\n- **bitsandbytes**: Simple 8-bit\u002F4-bit without custom backends\n- **llama.cpp\u002FGGUF**: CPU inference, Apple Silicon deployment\n\n## Quick start\n\n### Installation\n\n```bash\npip install hqq\n\n# With specific backend\npip install hqq[torch]      # PyTorch backend\npip install hqq[torchao]    # TorchAO int4 backend\npip install hqq[bitblas]    # BitBlas backend\npip install hqq[marlin]     # Marlin backend\n```\n\n### Basic quantization\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig, HQQLinear\nimport torch.nn as nn\n\n# Configure quantization\nconfig = BaseQuantizeConfig(\n    nbits=4,           # 4-bit quantization\n    group_size=64,     # Group size for quantization\n    axis=1             # Quantize along output dimension\n)\n\n# Quantize a linear layer\nlinear = nn.Linear(4096, 4096)\nhqq_linear = HQQLinear(linear, config)\n\n# Use normally\noutput = hqq_linear(input_tensor)\n```\n\n### Quantize full model with HuggingFace\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Configure HQQ\nquantization_config = HqqConfig(\n    nbits=4,\n    group_size=64,\n    axis=1\n)\n\n# Load and quantize\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quantization_config,\n    device_map=\"auto\"\n)\n\n# Model is quantized and ready to use\n```\n\n## Core concepts\n\n### Quantization configuration\n\nHQQ uses `BaseQuantizeConfig` to define quantization parameters:\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig\n\n# Standard 4-bit config\nconfig_4bit = BaseQuantizeConfig(\n    nbits=4,           # Bits per weight (1-8)\n    group_size=64,     # Weights per quantization group\n    axis=1             # 0=input dim, 1=output dim\n)\n\n# Aggressive 2-bit config\nconfig_2bit = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,     # Smaller groups for low-bit\n    axis=1\n)\n\n# Mixed precision per layer type\nlayer_configs = {\n    \"self_attn.q_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.k_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.v_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"mlp.gate_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.up_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.down_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n}\n```\n\n### HQQLinear layer\n\nThe core quantized layer that replaces `nn.Linear`:\n\n```python\nfrom hqq.core.quantize import HQQLinear\nimport torch\n\n# Create quantized layer\nlinear = torch.nn.Linear(4096, 4096)\nhqq_layer = HQQLinear(linear, config)\n\n# Access quantized weights\nW_q = hqq_layer.W_q           # Quantized weights\nscale = hqq_layer.scale       # Scale factors\nzero = hqq_layer.zero         # Zero points\n\n# Dequantize for inspection\nW_dequant = hqq_layer.dequantize()\n```\n\n### Backends\n\nHQQ supports multiple inference backends for different hardware:\n\n```python\nfrom hqq.core.quantize import HQQLinear\n\n# Available backends\nbackends = [\n    \"pytorch\",          # Pure PyTorch (default)\n    \"pytorch_compile\",  # torch.compile optimized\n    \"aten\",            # Custom CUDA kernels\n    \"torchao_int4\",    # TorchAO int4 matmul\n    \"gemlite\",         # GemLite CUDA kernels\n    \"bitblas\",         # BitBlas optimized\n    \"marlin\",          # Marlin 4-bit kernels\n]\n\n# Set backend globally\nHQQLinear.set_backend(\"torchao_int4\")\n\n# Or per layer\nhqq_layer.set_backend(\"marlin\")\n```\n\n**Backend selection guide:**\n| Backend | Best For | Requirements |\n|---------|----------|--------------|\n| pytorch | Compatibility | Any GPU |\n| pytorch_compile | Moderate speedup | torch>=2.0 |\n| aten | Good balance | CUDA GPU |\n| torchao_int4 | 4-bit inference | torchao installed |\n| marlin | Maximum 4-bit speed | Ampere+ GPU |\n| bitblas | Flexible bit-widths | bitblas installed |\n\n## HuggingFace integration\n\n### Load pre-quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load HQQ-quantized model from Hub\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# Use normally\ninputs = tokenizer(\"Hello, world!\", return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=50)\n```\n\n### Quantize and save\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Quantize\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Save quantized model\nmodel.save_pretrained(\".\u002Fllama-8b-hqq-4bit\")\n\n# Push to Hub\nmodel.push_to_hub(\"my-org\u002FLlama-3.1-8B-HQQ-4bit\")\n```\n\n### Mixed precision quantization\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Different precision per layer type\nconfig = HqqConfig(\n    nbits=4,\n    group_size=64,\n    # Attention layers: higher precision\n    # MLP layers: lower precision for memory savings\n    dynamic_config={\n        \"attn\": {\"nbits\": 4, \"group_size\": 64},\n        \"mlp\": {\"nbits\": 2, \"group_size\": 32}\n    }\n)\n```\n\n## vLLM integration\n\n### Serve HQQ models with vLLM\n\n```python\nfrom vllm import LLM, SamplingParams\n\n# Load HQQ-quantized model\nllm = LLM(\n    model=\"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    quantization=\"hqq\",\n    dtype=\"float16\"\n)\n\n# Generate\nsampling_params = SamplingParams(temperature=0.7, max_tokens=100)\noutputs = llm.generate([\"What is machine learning?\"], sampling_params)\n```\n\n### vLLM with custom HQQ config\n\n```python\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"meta-llama\u002FLlama-3.1-8B\",\n    quantization=\"hqq\",\n    quantization_config={\n        \"nbits\": 4,\n        \"group_size\": 64\n    }\n)\n```\n\n## PEFT\u002FLoRA fine-tuning\n\n### Fine-tune quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\nfrom peft import LoraConfig, get_peft_model\n\n# Load quantized model\nquant_config = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quant_config,\n    device_map=\"auto\"\n)\n\n# Apply LoRA\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\nmodel = get_peft_model(model, lora_config)\n\n# Train normally with Trainer or custom loop\n```\n\n### QLoRA-style training\n\n```python\nfrom transformers import TrainingArguments, Trainer\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fhqq-lora-output\",\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    learning_rate=2e-4,\n    num_train_epochs=3,\n    fp16=True,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    data_collator=data_collator\n)\n\ntrainer.train()\n```\n\n## Quantization workflows\n\n### Workflow 1: Quick model compression\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig\n\n# 1. Configure quantization\nconfig = HqqConfig(nbits=4, group_size=64)\n\n# 2. Load and quantize (no calibration needed!)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# 3. Verify quality\nprompt = \"The capital of France is\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=20)\nprint(tokenizer.decode(outputs[0]))\n\n# 4. Save\nmodel.save_pretrained(\".\u002Fllama-8b-hqq\")\ntokenizer.save_pretrained(\".\u002Fllama-8b-hqq\")\n```\n\n### Workflow 2: Optimize for inference speed\n\n```python\nfrom hqq.core.quantize import HQQLinear\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# 1. Quantize with optimal backend\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# 2. Set fast backend\nHQQLinear.set_backend(\"marlin\")  # or \"torchao_int4\"\n\n# 3. Compile for additional speedup\nimport torch\nmodel = torch.compile(model)\n\n# 4. Benchmark\nimport time\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(model.device)\nstart = time.time()\nfor _ in range(10):\n    model.generate(**inputs, max_new_tokens=100)\nprint(f\"Avg time: {(time.time() - start) \u002F 10:.2f}s\")\n```\n\n## Best practices\n\n1. **Start with 4-bit**: Best quality\u002Fsize tradeoff for most models\n2. **Use group_size=64**: Good balance; smaller for extreme quantization\n3. **Choose backend wisely**: Marlin for 4-bit Ampere+, TorchAO for flexibility\n4. **Verify quality**: Always test generation quality after quantization\n5. **Mixed precision**: Keep attention at higher precision, compress MLP more\n6. **PEFT training**: Use LoRA r=16-32 for good fine-tuning results\n\n## Common issues\n\n**Out of memory during quantization:**\n```python\n# Quantize layer-by-layer\nfrom hqq.models.hf.base import AutoHQQHFModel\n\nmodel = AutoHQQHFModel.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"sequential\"  # Load layers sequentially\n)\n```\n\n**Slow inference:**\n```python\n# Switch to optimized backend\nfrom hqq.core.quantize import HQQLinear\nHQQLinear.set_backend(\"marlin\")  # Requires Ampere+ GPU\n\n# Or compile\nmodel = torch.compile(model, mode=\"reduce-overhead\")\n```\n\n**Poor quality at 2-bit:**\n```python\n# Use smaller group size\nconfig = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,  # Smaller groups help at low bits\n    axis=1\n)\n```\n\n## References\n\n- **[Advanced Usage](references\u002Fadvanced-usage.md)** - Custom backends, mixed precision, optimization\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, debugging, benchmarks\n\n## Resources\n\n- **Repository**: https:\u002F\u002Fgithub.com\u002Fmobiusml\u002Fhqq\n- **Paper**: Half-Quadratic Quantization\n- **HuggingFace Models**: https:\u002F\u002Fhuggingface.co\u002Fmobiuslabsgmbh\n- **Version**: 0.2.0+\n- **License**: Apache 2.0\n","# HQQ - Half-Quadratic Quantization\n\nFast, calibration-free weight quantization supporting 8\u002F4\u002F3\u002F2\u002F1-bit precision with multiple optimized backends.\n\n## When to use HQQ\n\n**Use HQQ when:**\n- Quantizing models without calibration data (no dataset needed)\n- Need fast quantization (minutes vs hours for GPTQ\u002FAWQ)\n- Deploying with vLLM or HuggingFace Transformers\n- Fine-tuning quantized models with LoRA\u002FPEFT\n- Experimenting with extreme quantization (2-bit, 1-bit)\n\n**Key advantages:**\n- **No calibration**: Quantize any model instantly without sample data\n- **Multiple backends**: PyTorch, ATEN, TorchAO, Marlin, BitBlas for optimized inference\n- **Flexible precision**: 8\u002F4\u002F3\u002F2\u002F1-bit with configurable group sizes\n- **Framework integration**: Native HuggingFace and vLLM support\n- **PEFT compatible**: Fine-tune quantized models with LoRA\n\n**Use alternatives instead:**\n- **AWQ**: Need calibration-based accuracy, production serving\n- **GPTQ**: Maximum accuracy with calibration data available\n- **bitsandbytes**: Simple 8-bit\u002F4-bit without custom backends\n- **llama.cpp\u002FGGUF**: CPU inference, Apple Silicon deployment\n\n## Quick start\n\n### Installation\n\n```bash\npip install hqq\n\n# With specific backend\npip install hqq[torch]      # PyTorch backend\npip install hqq[torchao]    # TorchAO int4 backend\npip install hqq[bitblas]    # BitBlas backend\npip install hqq[marlin]     # Marlin backend\n```\n\n### Basic quantization\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig, HQQLinear\nimport torch.nn as nn\n\n# Configure quantization\nconfig = BaseQuantizeConfig(\n    nbits=4,           # 4-bit quantization\n    group_size=64,     # Group size for quantization\n    axis=1             # Quantize along output dimension\n)\n\n# Quantize a linear layer\nlinear = nn.Linear(4096, 4096)\nhqq_linear = HQQLinear(linear, config)\n\n# Use normally\noutput = hqq_linear(input_tensor)\n```\n\n### Quantize full model with HuggingFace\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Configure HQQ\nquantization_config = HqqConfig(\n    nbits=4,\n    group_size=64,\n    axis=1\n)\n\n# Load and quantize\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quantization_config,\n    device_map=\"auto\"\n)\n\n# Model is quantized and ready to use\n```\n\n## Core concepts\n\n### Quantization configuration\n\nHQQ uses `BaseQuantizeConfig` to define quantization parameters:\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig\n\n# Standard 4-bit config\nconfig_4bit = BaseQuantizeConfig(\n    nbits=4,           # Bits per weight (1-8)\n    group_size=64,     # Weights per quantization group\n    axis=1             # 0=input dim, 1=output dim\n)\n\n# Aggressive 2-bit config\nconfig_2bit = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,     # Smaller groups for low-bit\n    axis=1\n)\n\n# Mixed precision per layer type\nlayer_configs = {\n    \"self_attn.q_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.k_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.v_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"mlp.gate_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.up_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.down_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n}\n```\n\n### HQQLinear layer\n\nThe core quantized layer that replaces `nn.Linear`:\n\n```python\nfrom hqq.core.quantize import HQQLinear\nimport torch\n\n# Create quantized layer\nlinear = torch.nn.Linear(4096, 4096)\nhqq_layer = HQQLinear(linear, config)\n\n# Access quantized weights\nW_q = hqq_layer.W_q           # Quantized weights\nscale = hqq_layer.scale       # Scale factors\nzero = hqq_layer.zero         # Zero points\n\n# Dequantize for inspection\nW_dequant = hqq_layer.dequantize()\n```\n\n### Backends\n\nHQQ supports multiple inference backends for different hardware:\n\n```python\nfrom hqq.core.quantize import HQQLinear\n\n# Available backends\nbackends = [\n    \"pytorch\",          # Pure PyTorch (default)\n    \"pytorch_compile\",  # torch.compile optimized\n    \"aten\",            # Custom CUDA kernels\n    \"torchao_int4\",    # TorchAO int4 matmul\n    \"gemlite\",         # GemLite CUDA kernels\n    \"bitblas\",         # BitBlas optimized\n    \"marlin\",          # Marlin 4-bit kernels\n]\n\n# Set backend globally\nHQQLinear.set_backend(\"torchao_int4\")\n\n# Or per layer\nhqq_layer.set_backend(\"marlin\")\n```\n\n**Backend selection guide:**\n| Backend | Best For | Requirements |\n|---------|----------|--------------|\n| pytorch | Compatibility | Any GPU |\n| pytorch_compile | Moderate speedup | torch>=2.0 |\n| aten | Good balance | CUDA GPU |\n| torchao_int4 | 4-bit inference | torchao installed |\n| marlin | Maximum 4-bit speed | Ampere+ GPU |\n| bitblas | Flexible bit-widths | bitblas installed |\n\n## HuggingFace integration\n\n### Load pre-quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load HQQ-quantized model from Hub\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# Use normally\ninputs = tokenizer(\"Hello, world!\", return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=50)\n```\n\n### Quantize and save\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Quantize\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Save quantized model\nmodel.save_pretrained(\".\u002Fllama-8b-hqq-4bit\")\n\n# Push to Hub\nmodel.push_to_hub(\"my-org\u002FLlama-3.1-8B-HQQ-4bit\")\n```\n\n### Mixed precision quantization\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Different precision per layer type\nconfig = HqqConfig(\n    nbits=4,\n    group_size=64,\n    # Attention layers: higher precision\n    # MLP layers: lower precision for memory savings\n    dynamic_config={\n        \"attn\": {\"nbits\": 4, \"group_size\": 64},\n        \"mlp\": {\"nbits\": 2, \"group_size\": 32}\n    }\n)\n```\n\n## vLLM integration\n\n### Serve HQQ models with vLLM\n\n```python\nfrom vllm import LLM, SamplingParams\n\n# Load HQQ-quantized model\nllm = LLM(\n    model=\"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    quantization=\"hqq\",\n    dtype=\"float16\"\n)\n\n# Generate\nsampling_params = SamplingParams(temperature=0.7, max_tokens=100)\noutputs = llm.generate([\"What is machine learning?\"], sampling_params)\n```\n\n### vLLM with custom HQQ config\n\n```python\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"meta-llama\u002FLlama-3.1-8B\",\n    quantization=\"hqq\",\n    quantization_config={\n        \"nbits\": 4,\n        \"group_size\": 64\n    }\n)\n```\n\n## PEFT\u002FLoRA fine-tuning\n\n### Fine-tune quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\nfrom peft import LoraConfig, get_peft_model\n\n# Load quantized model\nquant_config = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quant_config,\n    device_map=\"auto\"\n)\n\n# Apply LoRA\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\nmodel = get_peft_model(model, lora_config)\n\n# Train normally with Trainer or custom loop\n```\n\n### QLoRA-style training\n\n```python\nfrom transformers import TrainingArguments, Trainer\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fhqq-lora-output\",\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    learning_rate=2e-4,\n    num_train_epochs=3,\n    fp16=True,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    data_collator=data_collator\n)\n\ntrainer.train()\n```\n\n## Quantization workflows\n\n### Workflow 1: Quick model compression\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig\n\n# 1. Configure quantization\nconfig = HqqConfig(nbits=4, group_size=64)\n\n# 2. Load and quantize (no calibration needed!)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# 3. Verify quality\nprompt = \"The capital of France is\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=20)\nprint(tokenizer.decode(outputs[0]))\n\n# 4. Save\nmodel.save_pretrained(\".\u002Fllama-8b-hqq\")\ntokenizer.save_pretrained(\".\u002Fllama-8b-hqq\")\n```\n\n### Workflow 2: Optimize for inference speed\n\n```python\nfrom hqq.core.quantize import HQQLinear\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# 1. Quantize with optimal backend\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# 2. Set fast backend\nHQQLinear.set_backend(\"marlin\")  # or \"torchao_int4\"\n\n# 3. Compile for additional speedup\nimport torch\nmodel = torch.compile(model)\n\n# 4. Benchmark\nimport time\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(model.device)\nstart = time.time()\nfor _ in range(10):\n    model.generate(**inputs, max_new_tokens=100)\nprint(f\"Avg time: {(time.time() - start) \u002F 10:.2f}s\")\n```\n\n## Best practices\n\n1. **Start with 4-bit**: Best quality\u002Fsize tradeoff for most models\n2. **Use group_size=64**: Good balance; smaller for extreme quantization\n3. **Choose backend wisely**: Marlin for 4-bit Ampere+, TorchAO for flexibility\n4. **Verify quality**: Always test generation quality after quantization\n5. **Mixed precision**: Keep attention at higher precision, compress MLP more\n6. **PEFT training**: Use LoRA r=16-32 for good fine-tuning results\n\n## Common issues\n\n**Out of memory during quantization:**\n```python\n# Quantize layer-by-layer\nfrom hqq.models.hf.base import AutoHQQHFModel\n\nmodel = AutoHQQHFModel.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"sequential\"  # Load layers sequentially\n)\n```\n\n**Slow inference:**\n```python\n# Switch to optimized backend\nfrom hqq.core.quantize import HQQLinear\nHQQLinear.set_backend(\"marlin\")  # Requires Ampere+ GPU\n\n# Or compile\nmodel = torch.compile(model, mode=\"reduce-overhead\")\n```\n\n**Poor quality at 2-bit:**\n```python\n# Use smaller group size\nconfig = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,  # Smaller groups help at low bits\n    axis=1\n)\n```\n\n## References\n\n- **[Advanced Usage](references\u002Fadvanced-usage.md)** - Custom backends, mixed precision, optimization\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, debugging, benchmarks\n\n## Resources\n\n- **Repository**: https:\u002F\u002Fgithub.com\u002Fmobiusml\u002Fhqq\n- **Paper**: Half-Quadratic Quantization\n- **HuggingFace Models**: https:\u002F\u002Fhuggingface.co\u002Fmobiuslabsgmbh\n- **Version**: 0.2.0+\n- **License**: Apache 2.0\n",{"name":5,"tags":24,"author":31,"license":15,"version":32,"description":7,"dependencies":33},[25,26,27,28,29,30],"Quantization","HQQ","Optimization","Memory Efficiency","Inference","Model Compression","Orchestra Research","1.0.0",[34,35],"hqq>=0.2.0","torch>=2.0.0","main","af388a77c2fe5e0b4fd879cca4300a0ffa9a6579","skills\u002Fams-imp-optimization-hqq",1,"https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis","https:\u002F\u002Fgithub.com\u002FTanNhatCMS","https:\u002F\u002Fmrkiet.name.vn","2026-11-07 14:23:08",{"source":8,"skillName":5,"commands":45,"prompt":55,"zipAvailable":56},[46,49,52],{"tool":47,"command":48},"npx","npx skills add TanNhatCMS\u002Fjarvis --skill optimization-hqq",{"tool":50,"command":51},"bunx","bunx skills add TanNhatCMS\u002Fjarvis --skill optimization-hqq",{"tool":53,"command":54},"pnpm","pnpm dlx skills add TanNhatCMS\u002Fjarvis --skill optimization-hqq","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fams-imp-optimization-hqq\u002FSKILL.md",true,[58,64],{"path":59,"resourceType":60,"resourceTypeDesc":61,"sizeBytes":62,"sourceHtmlUrl":63},"references\u002Fadvanced-usage.md","REFERENCE","参考文档",14077,"https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fams-imp-optimization-hqq\u002Freferences\u002Fadvanced-usage.md",{"path":65,"resourceType":60,"resourceTypeDesc":61,"sizeBytes":66,"sourceHtmlUrl":67},"references\u002Ftroubleshooting.md",11086,"https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fams-imp-optimization-hqq\u002Freferences\u002Ftroubleshooting.md",[69,76,84,91,98,105],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":19},"01M7EWYJR0J8DARY7E59Z6Y9MV","web-scrape","Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open\u002Ffetch fails on JS-heavy pages, or when you need clean markdown\u002Fstructured extraction from a site. Requires FIRECRAWL_API_KEY.","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fweb-scrape\u002FSKILL.md",[],[],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":80,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":19},"01M7EWYBN5P6F2AJG3FR24T9BP","design-taste-frontend","tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML\u002FCSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, \"AI-made\", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Ftaste-design\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Ftaste-design\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":19},"01M7EWY8FQFNQ2GB7NZZWGSSDX","test-driven-development","Use when implementing any feature or bugfix, before writing implementation code","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-tdd\u002FSKILL.md",[],[],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":19},"01M7EWY5SHBFRH1HDHVWM9K7TV","systematic-debugging","Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-debugging\u002FSKILL.md",[],[],{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":104,"publishedAt":19},"01M7EWY3NM0Z0JAKS7HDZ24K66","requesting-code-review","Use when completing tasks, implementing major features, or before merging to verify work meets requirements","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-code-review\u002FSKILL.md",[],[],{"skillNo":106,"name":107,"nameZh":6,"description":108,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":109,"categories":110,"keywords":111,"publishedAt":19},"01M7EWY0WRMWX4M9ZFY0WKC593","brainstorming","You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-brainstorming\u002FSKILL.md",[],[]]