[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M72FN39MCQD8FC9T533T40KC":3,"skill-same:01M72FN39MCQD8FC9T533T40KC":67},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":36,"commitSha":37,"skillDirectory":38,"forksCount":14,"repositoryHtmlUrl":39,"repositoryHomepageUrl":6,"ownerHtmlUrl":40,"ownerBlogUrl":6,"ownerDescription":41,"updatedAt":42,"install":43,"resources":56},"01M72FN39MCQD8FC9T533T40KC","hqq-quantization",null,"Half-Quadratic Quantization for LLMs without calibration data. Use when quantizing models to 4\u002F3\u002F2-bit precision without needing calibration datasets, for fast quantization workflows, or when deploying with vLLM or HuggingFace Transformers.","cocyuhao\u002Fai-dev-tooling-research-archive","cocyuhao","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F269083740?v=4","01M72F908ET5R76KP9NXS6Y2GJ","Hao Yu","01M72F909376KCNJVKG8MEPY2Q",0,"MIT","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-hqq\u002FSKILL.md",[],[],"2026-11-10 18:39:20",false,"---\nname: hqq-quantization\ndescription: Half-Quadratic Quantization for LLMs without calibration data. Use when quantizing models to 4\u002F3\u002F2-bit precision without needing calibration datasets, for fast quantization workflows, or when deploying with vLLM or HuggingFace Transformers.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Quantization, HQQ, Optimization, Memory Efficiency, Inference, Model Compression]\ndependencies: [hqq>=0.2.0, torch>=2.0.0]\n---\n\n# HQQ - Half-Quadratic Quantization\n\nFast, calibration-free weight quantization supporting 8\u002F4\u002F3\u002F2\u002F1-bit precision with multiple optimized backends.\n\n## When to use HQQ\n\n**Use HQQ when:**\n- Quantizing models without calibration data (no dataset needed)\n- Need fast quantization (minutes vs hours for GPTQ\u002FAWQ)\n- Deploying with vLLM or HuggingFace Transformers\n- Fine-tuning quantized models with LoRA\u002FPEFT\n- Experimenting with extreme quantization (2-bit, 1-bit)\n\n**Key advantages:**\n- **No calibration**: Quantize any model instantly without sample data\n- **Multiple backends**: PyTorch, ATEN, TorchAO, Marlin, BitBlas for optimized inference\n- **Flexible precision**: 8\u002F4\u002F3\u002F2\u002F1-bit with configurable group sizes\n- **Framework integration**: Native HuggingFace and vLLM support\n- **PEFT compatible**: Fine-tune quantized models with LoRA\n\n**Use alternatives instead:**\n- **AWQ**: Need calibration-based accuracy, production serving\n- **GPTQ**: Maximum accuracy with calibration data available\n- **bitsandbytes**: Simple 8-bit\u002F4-bit without custom backends\n- **llama.cpp\u002FGGUF**: CPU inference, Apple Silicon deployment\n\n## Quick start\n\n### Installation\n\n```bash\npip install hqq\n\n# With specific backend\npip install hqq[torch]      # PyTorch backend\npip install hqq[torchao]    # TorchAO int4 backend\npip install hqq[bitblas]    # BitBlas backend\npip install hqq[marlin]     # Marlin backend\n```\n\n### Basic quantization\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig, HQQLinear\nimport torch.nn as nn\n\n# Configure quantization\nconfig = BaseQuantizeConfig(\n    nbits=4,           # 4-bit quantization\n    group_size=64,     # Group size for quantization\n    axis=1             # Quantize along output dimension\n)\n\n# Quantize a linear layer\nlinear = nn.Linear(4096, 4096)\nhqq_linear = HQQLinear(linear, config)\n\n# Use normally\noutput = hqq_linear(input_tensor)\n```\n\n### Quantize full model with HuggingFace\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Configure HQQ\nquantization_config = HqqConfig(\n    nbits=4,\n    group_size=64,\n    axis=1\n)\n\n# Load and quantize\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quantization_config,\n    device_map=\"auto\"\n)\n\n# Model is quantized and ready to use\n```\n\n## Core concepts\n\n### Quantization configuration\n\nHQQ uses `BaseQuantizeConfig` to define quantization parameters:\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig\n\n# Standard 4-bit config\nconfig_4bit = BaseQuantizeConfig(\n    nbits=4,           # Bits per weight (1-8)\n    group_size=64,     # Weights per quantization group\n    axis=1             # 0=input dim, 1=output dim\n)\n\n# Aggressive 2-bit config\nconfig_2bit = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,     # Smaller groups for low-bit\n    axis=1\n)\n\n# Mixed precision per layer type\nlayer_configs = {\n    \"self_attn.q_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.k_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.v_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"mlp.gate_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.up_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.down_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n}\n```\n\n### HQQLinear layer\n\nThe core quantized layer that replaces `nn.Linear`:\n\n```python\nfrom hqq.core.quantize import HQQLinear\nimport torch\n\n# Create quantized layer\nlinear = torch.nn.Linear(4096, 4096)\nhqq_layer = HQQLinear(linear, config)\n\n# Access quantized weights\nW_q = hqq_layer.W_q           # Quantized weights\nscale = hqq_layer.scale       # Scale factors\nzero = hqq_layer.zero         # Zero points\n\n# Dequantize for inspection\nW_dequant = hqq_layer.dequantize()\n```\n\n### Backends\n\nHQQ supports multiple inference backends for different hardware:\n\n```python\nfrom hqq.core.quantize import HQQLinear\n\n# Available backends\nbackends = [\n    \"pytorch\",          # Pure PyTorch (default)\n    \"pytorch_compile\",  # torch.compile optimized\n    \"aten\",            # Custom CUDA kernels\n    \"torchao_int4\",    # TorchAO int4 matmul\n    \"gemlite\",         # GemLite CUDA kernels\n    \"bitblas\",         # BitBlas optimized\n    \"marlin\",          # Marlin 4-bit kernels\n]\n\n# Set backend globally\nHQQLinear.set_backend(\"torchao_int4\")\n\n# Or per layer\nhqq_layer.set_backend(\"marlin\")\n```\n\n**Backend selection guide:**\n| Backend | Best For | Requirements |\n|---------|----------|--------------|\n| pytorch | Compatibility | Any GPU |\n| pytorch_compile | Moderate speedup | torch>=2.0 |\n| aten | Good balance | CUDA GPU |\n| torchao_int4 | 4-bit inference | torchao installed |\n| marlin | Maximum 4-bit speed | Ampere+ GPU |\n| bitblas | Flexible bit-widths | bitblas installed |\n\n## HuggingFace integration\n\n### Load pre-quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load HQQ-quantized model from Hub\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# Use normally\ninputs = tokenizer(\"Hello, world!\", return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=50)\n```\n\n### Quantize and save\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Quantize\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Save quantized model\nmodel.save_pretrained(\".\u002Fllama-8b-hqq-4bit\")\n\n# Push to Hub\nmodel.push_to_hub(\"my-org\u002FLlama-3.1-8B-HQQ-4bit\")\n```\n\n### Mixed precision quantization\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Different precision per layer type\nconfig = HqqConfig(\n    nbits=4,\n    group_size=64,\n    # Attention layers: higher precision\n    # MLP layers: lower precision for memory savings\n    dynamic_config={\n        \"attn\": {\"nbits\": 4, \"group_size\": 64},\n        \"mlp\": {\"nbits\": 2, \"group_size\": 32}\n    }\n)\n```\n\n## vLLM integration\n\n### Serve HQQ models with vLLM\n\n```python\nfrom vllm import LLM, SamplingParams\n\n# Load HQQ-quantized model\nllm = LLM(\n    model=\"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    quantization=\"hqq\",\n    dtype=\"float16\"\n)\n\n# Generate\nsampling_params = SamplingParams(temperature=0.7, max_tokens=100)\noutputs = llm.generate([\"What is machine learning?\"], sampling_params)\n```\n\n### vLLM with custom HQQ config\n\n```python\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"meta-llama\u002FLlama-3.1-8B\",\n    quantization=\"hqq\",\n    quantization_config={\n        \"nbits\": 4,\n        \"group_size\": 64\n    }\n)\n```\n\n## PEFT\u002FLoRA fine-tuning\n\n### Fine-tune quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\nfrom peft import LoraConfig, get_peft_model\n\n# Load quantized model\nquant_config = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quant_config,\n    device_map=\"auto\"\n)\n\n# Apply LoRA\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\nmodel = get_peft_model(model, lora_config)\n\n# Train normally with Trainer or custom loop\n```\n\n### QLoRA-style training\n\n```python\nfrom transformers import TrainingArguments, Trainer\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fhqq-lora-output\",\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    learning_rate=2e-4,\n    num_train_epochs=3,\n    fp16=True,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    data_collator=data_collator\n)\n\ntrainer.train()\n```\n\n## Quantization workflows\n\n### Workflow 1: Quick model compression\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig\n\n# 1. Configure quantization\nconfig = HqqConfig(nbits=4, group_size=64)\n\n# 2. Load and quantize (no calibration needed!)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# 3. Verify quality\nprompt = \"The capital of France is\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=20)\nprint(tokenizer.decode(outputs[0]))\n\n# 4. Save\nmodel.save_pretrained(\".\u002Fllama-8b-hqq\")\ntokenizer.save_pretrained(\".\u002Fllama-8b-hqq\")\n```\n\n### Workflow 2: Optimize for inference speed\n\n```python\nfrom hqq.core.quantize import HQQLinear\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# 1. Quantize with optimal backend\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# 2. Set fast backend\nHQQLinear.set_backend(\"marlin\")  # or \"torchao_int4\"\n\n# 3. Compile for additional speedup\nimport torch\nmodel = torch.compile(model)\n\n# 4. Benchmark\nimport time\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(model.device)\nstart = time.time()\nfor _ in range(10):\n    model.generate(**inputs, max_new_tokens=100)\nprint(f\"Avg time: {(time.time() - start) \u002F 10:.2f}s\")\n```\n\n## Best practices\n\n1. **Start with 4-bit**: Best quality\u002Fsize tradeoff for most models\n2. **Use group_size=64**: Good balance; smaller for extreme quantization\n3. **Choose backend wisely**: Marlin for 4-bit Ampere+, TorchAO for flexibility\n4. **Verify quality**: Always test generation quality after quantization\n5. **Mixed precision**: Keep attention at higher precision, compress MLP more\n6. **PEFT training**: Use LoRA r=16-32 for good fine-tuning results\n\n## Common issues\n\n**Out of memory during quantization:**\n```python\n# Quantize layer-by-layer\nfrom hqq.models.hf.base import AutoHQQHFModel\n\nmodel = AutoHQQHFModel.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"sequential\"  # Load layers sequentially\n)\n```\n\n**Slow inference:**\n```python\n# Switch to optimized backend\nfrom hqq.core.quantize import HQQLinear\nHQQLinear.set_backend(\"marlin\")  # Requires Ampere+ GPU\n\n# Or compile\nmodel = torch.compile(model, mode=\"reduce-overhead\")\n```\n\n**Poor quality at 2-bit:**\n```python\n# Use smaller group size\nconfig = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,  # Smaller groups help at low bits\n    axis=1\n)\n```\n\n## References\n\n- **[Advanced Usage](references\u002Fadvanced-usage.md)** - Custom backends, mixed precision, optimization\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, debugging, benchmarks\n\n## Resources\n\n- **Repository**: https:\u002F\u002Fgithub.com\u002Fmobiusml\u002Fhqq\n- **Paper**: Half-Quadratic Quantization\n- **HuggingFace Models**: https:\u002F\u002Fhuggingface.co\u002Fmobiuslabsgmbh\n- **Version**: 0.2.0+\n- **License**: Apache 2.0\n","# HQQ - Half-Quadratic Quantization\n\nFast, calibration-free weight quantization supporting 8\u002F4\u002F3\u002F2\u002F1-bit precision with multiple optimized backends.\n\n## When to use HQQ\n\n**Use HQQ when:**\n- Quantizing models without calibration data (no dataset needed)\n- Need fast quantization (minutes vs hours for GPTQ\u002FAWQ)\n- Deploying with vLLM or HuggingFace Transformers\n- Fine-tuning quantized models with LoRA\u002FPEFT\n- Experimenting with extreme quantization (2-bit, 1-bit)\n\n**Key advantages:**\n- **No calibration**: Quantize any model instantly without sample data\n- **Multiple backends**: PyTorch, ATEN, TorchAO, Marlin, BitBlas for optimized inference\n- **Flexible precision**: 8\u002F4\u002F3\u002F2\u002F1-bit with configurable group sizes\n- **Framework integration**: Native HuggingFace and vLLM support\n- **PEFT compatible**: Fine-tune quantized models with LoRA\n\n**Use alternatives instead:**\n- **AWQ**: Need calibration-based accuracy, production serving\n- **GPTQ**: Maximum accuracy with calibration data available\n- **bitsandbytes**: Simple 8-bit\u002F4-bit without custom backends\n- **llama.cpp\u002FGGUF**: CPU inference, Apple Silicon deployment\n\n## Quick start\n\n### Installation\n\n```bash\npip install hqq\n\n# With specific backend\npip install hqq[torch]      # PyTorch backend\npip install hqq[torchao]    # TorchAO int4 backend\npip install hqq[bitblas]    # BitBlas backend\npip install hqq[marlin]     # Marlin backend\n```\n\n### Basic quantization\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig, HQQLinear\nimport torch.nn as nn\n\n# Configure quantization\nconfig = BaseQuantizeConfig(\n    nbits=4,           # 4-bit quantization\n    group_size=64,     # Group size for quantization\n    axis=1             # Quantize along output dimension\n)\n\n# Quantize a linear layer\nlinear = nn.Linear(4096, 4096)\nhqq_linear = HQQLinear(linear, config)\n\n# Use normally\noutput = hqq_linear(input_tensor)\n```\n\n### Quantize full model with HuggingFace\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Configure HQQ\nquantization_config = HqqConfig(\n    nbits=4,\n    group_size=64,\n    axis=1\n)\n\n# Load and quantize\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quantization_config,\n    device_map=\"auto\"\n)\n\n# Model is quantized and ready to use\n```\n\n## Core concepts\n\n### Quantization configuration\n\nHQQ uses `BaseQuantizeConfig` to define quantization parameters:\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig\n\n# Standard 4-bit config\nconfig_4bit = BaseQuantizeConfig(\n    nbits=4,           # Bits per weight (1-8)\n    group_size=64,     # Weights per quantization group\n    axis=1             # 0=input dim, 1=output dim\n)\n\n# Aggressive 2-bit config\nconfig_2bit = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,     # Smaller groups for low-bit\n    axis=1\n)\n\n# Mixed precision per layer type\nlayer_configs = {\n    \"self_attn.q_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.k_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.v_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"mlp.gate_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.up_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.down_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n}\n```\n\n### HQQLinear layer\n\nThe core quantized layer that replaces `nn.Linear`:\n\n```python\nfrom hqq.core.quantize import HQQLinear\nimport torch\n\n# Create quantized layer\nlinear = torch.nn.Linear(4096, 4096)\nhqq_layer = HQQLinear(linear, config)\n\n# Access quantized weights\nW_q = hqq_layer.W_q           # Quantized weights\nscale = hqq_layer.scale       # Scale factors\nzero = hqq_layer.zero         # Zero points\n\n# Dequantize for inspection\nW_dequant = hqq_layer.dequantize()\n```\n\n### Backends\n\nHQQ supports multiple inference backends for different hardware:\n\n```python\nfrom hqq.core.quantize import HQQLinear\n\n# Available backends\nbackends = [\n    \"pytorch\",          # Pure PyTorch (default)\n    \"pytorch_compile\",  # torch.compile optimized\n    \"aten\",            # Custom CUDA kernels\n    \"torchao_int4\",    # TorchAO int4 matmul\n    \"gemlite\",         # GemLite CUDA kernels\n    \"bitblas\",         # BitBlas optimized\n    \"marlin\",          # Marlin 4-bit kernels\n]\n\n# Set backend globally\nHQQLinear.set_backend(\"torchao_int4\")\n\n# Or per layer\nhqq_layer.set_backend(\"marlin\")\n```\n\n**Backend selection guide:**\n| Backend | Best For | Requirements |\n|---------|----------|--------------|\n| pytorch | Compatibility | Any GPU |\n| pytorch_compile | Moderate speedup | torch>=2.0 |\n| aten | Good balance | CUDA GPU |\n| torchao_int4 | 4-bit inference | torchao installed |\n| marlin | Maximum 4-bit speed | Ampere+ GPU |\n| bitblas | Flexible bit-widths | bitblas installed |\n\n## HuggingFace integration\n\n### Load pre-quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load HQQ-quantized model from Hub\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# Use normally\ninputs = tokenizer(\"Hello, world!\", return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=50)\n```\n\n### Quantize and save\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Quantize\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Save quantized model\nmodel.save_pretrained(\".\u002Fllama-8b-hqq-4bit\")\n\n# Push to Hub\nmodel.push_to_hub(\"my-org\u002FLlama-3.1-8B-HQQ-4bit\")\n```\n\n### Mixed precision quantization\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Different precision per layer type\nconfig = HqqConfig(\n    nbits=4,\n    group_size=64,\n    # Attention layers: higher precision\n    # MLP layers: lower precision for memory savings\n    dynamic_config={\n        \"attn\": {\"nbits\": 4, \"group_size\": 64},\n        \"mlp\": {\"nbits\": 2, \"group_size\": 32}\n    }\n)\n```\n\n## vLLM integration\n\n### Serve HQQ models with vLLM\n\n```python\nfrom vllm import LLM, SamplingParams\n\n# Load HQQ-quantized model\nllm = LLM(\n    model=\"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    quantization=\"hqq\",\n    dtype=\"float16\"\n)\n\n# Generate\nsampling_params = SamplingParams(temperature=0.7, max_tokens=100)\noutputs = llm.generate([\"What is machine learning?\"], sampling_params)\n```\n\n### vLLM with custom HQQ config\n\n```python\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"meta-llama\u002FLlama-3.1-8B\",\n    quantization=\"hqq\",\n    quantization_config={\n        \"nbits\": 4,\n        \"group_size\": 64\n    }\n)\n```\n\n## PEFT\u002FLoRA fine-tuning\n\n### Fine-tune quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\nfrom peft import LoraConfig, get_peft_model\n\n# Load quantized model\nquant_config = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quant_config,\n    device_map=\"auto\"\n)\n\n# Apply LoRA\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\nmodel = get_peft_model(model, lora_config)\n\n# Train normally with Trainer or custom loop\n```\n\n### QLoRA-style training\n\n```python\nfrom transformers import TrainingArguments, Trainer\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fhqq-lora-output\",\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    learning_rate=2e-4,\n    num_train_epochs=3,\n    fp16=True,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    data_collator=data_collator\n)\n\ntrainer.train()\n```\n\n## Quantization workflows\n\n### Workflow 1: Quick model compression\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig\n\n# 1. Configure quantization\nconfig = HqqConfig(nbits=4, group_size=64)\n\n# 2. Load and quantize (no calibration needed!)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# 3. Verify quality\nprompt = \"The capital of France is\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=20)\nprint(tokenizer.decode(outputs[0]))\n\n# 4. Save\nmodel.save_pretrained(\".\u002Fllama-8b-hqq\")\ntokenizer.save_pretrained(\".\u002Fllama-8b-hqq\")\n```\n\n### Workflow 2: Optimize for inference speed\n\n```python\nfrom hqq.core.quantize import HQQLinear\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# 1. Quantize with optimal backend\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# 2. Set fast backend\nHQQLinear.set_backend(\"marlin\")  # or \"torchao_int4\"\n\n# 3. Compile for additional speedup\nimport torch\nmodel = torch.compile(model)\n\n# 4. Benchmark\nimport time\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(model.device)\nstart = time.time()\nfor _ in range(10):\n    model.generate(**inputs, max_new_tokens=100)\nprint(f\"Avg time: {(time.time() - start) \u002F 10:.2f}s\")\n```\n\n## Best practices\n\n1. **Start with 4-bit**: Best quality\u002Fsize tradeoff for most models\n2. **Use group_size=64**: Good balance; smaller for extreme quantization\n3. **Choose backend wisely**: Marlin for 4-bit Ampere+, TorchAO for flexibility\n4. **Verify quality**: Always test generation quality after quantization\n5. **Mixed precision**: Keep attention at higher precision, compress MLP more\n6. **PEFT training**: Use LoRA r=16-32 for good fine-tuning results\n\n## Common issues\n\n**Out of memory during quantization:**\n```python\n# Quantize layer-by-layer\nfrom hqq.models.hf.base import AutoHQQHFModel\n\nmodel = AutoHQQHFModel.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"sequential\"  # Load layers sequentially\n)\n```\n\n**Slow inference:**\n```python\n# Switch to optimized backend\nfrom hqq.core.quantize import HQQLinear\nHQQLinear.set_backend(\"marlin\")  # Requires Ampere+ GPU\n\n# Or compile\nmodel = torch.compile(model, mode=\"reduce-overhead\")\n```\n\n**Poor quality at 2-bit:**\n```python\n# Use smaller group size\nconfig = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,  # Smaller groups help at low bits\n    axis=1\n)\n```\n\n## References\n\n- **[Advanced Usage](references\u002Fadvanced-usage.md)** - Custom backends, mixed precision, optimization\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, debugging, benchmarks\n\n## Resources\n\n- **Repository**: https:\u002F\u002Fgithub.com\u002Fmobiusml\u002Fhqq\n- **Paper**: Half-Quadratic Quantization\n- **HuggingFace Models**: https:\u002F\u002Fhuggingface.co\u002Fmobiuslabsgmbh\n- **Version**: 0.2.0+\n- **License**: Apache 2.0\n",{"name":5,"tags":24,"author":31,"license":15,"version":32,"description":7,"dependencies":33},[25,26,27,28,29,30],"Quantization","HQQ","Optimization","Memory Efficiency","Inference","Model Compression","Orchestra Research","1.0.0",[34,35],"hqq>=0.2.0","torch>=2.0.0","main","f4a24bd1ab047411d4ae57ee9a96701a7706e779","claude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-hqq","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive","https:\u002F\u002Fgithub.com\u002Fcocyuhao","Beijing Wuzi University","2026-05-12 10:57:02",{"source":8,"skillName":5,"commands":44,"prompt":54,"zipAvailable":55},[45,48,51],{"tool":46,"command":47},"npx","npx skills add cocyuhao\u002Fai-dev-tooling-research-archive --skill hqq-quantization",{"tool":49,"command":50},"bunx","bunx skills add cocyuhao\u002Fai-dev-tooling-research-archive --skill hqq-quantization",{"tool":52,"command":53},"pnpm","pnpm dlx skills add cocyuhao\u002Fai-dev-tooling-research-archive --skill hqq-quantization","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-hqq\u002FSKILL.md",true,[57,63],{"path":58,"resourceType":59,"resourceTypeDesc":60,"sizeBytes":61,"sourceHtmlUrl":62},"references\u002Fadvanced-usage.md","REFERENCE","参考文档",14077,"https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-hqq\u002Freferences\u002Fadvanced-usage.md",{"path":64,"resourceType":59,"resourceTypeDesc":60,"sizeBytes":65,"sourceHtmlUrl":66},"references\u002Ftroubleshooting.md",11086,"https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-hqq\u002Freferences\u002Ftroubleshooting.md",[68,76,84,91,98,105],{"skillNo":69,"name":70,"nameZh":6,"description":71,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":72,"categories":73,"keywords":74,"publishedAt":75},"01M72JQ593HE3SRJD3446CCKA8","web-access","所有联网操作必须通过此 skill 处理，包括：搜索、网页抓取、登录后操作、网络交互等。 触发场景：用户要求搜索信息、查看网页内容、访问需要登录的网站、操作网页界面、抓取社交媒体内容（小红书、微博、推特等）、读取动态渲染页面、以及任何需要真实浏览器环境的网络任务。","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fweb-access\u002FSKILL.md",[],[],"2026-11-10 22:39:42",{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":80,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":75},"01M72JKXJYFT2RBJ2HP5ZRA853","last30days","Multi-query social research across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web. Intelligent query planning with Gemini\u002FOpenAI fallback.","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Flast30days-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Flast30days-skill\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":75},"01M72JKMRQSD0ZY66KQ863HVY5","writing-plans","Use when you have a spec or requirements for a multi-step task. Creates comprehensive implementation plans with bite-sized tasks, exact file paths, and complete code examples.","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fsoftware-development\u002Fwriting-plans\u002FSKILL.md",[],[],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":75},"01M72JKHCV8DT6VWSD4KS5YWT9","test-driven-development","Use when implementing any feature or bugfix, before writing implementation code. Enforces RED-GREEN-REFACTOR cycle with test-first approach.","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fsoftware-development\u002Ftest-driven-development\u002FSKILL.md",[],[],{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":104,"publishedAt":75},"01M72JKD7VX7EQM38R704F3KT1","systematic-debugging","Use when encountering any bug, test failure, or unexpected behavior. 4-phase root cause investigation — NO fixes without understanding the problem first.","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fsoftware-development\u002Fsystematic-debugging\u002FSKILL.md",[],[],{"skillNo":106,"name":107,"nameZh":6,"description":108,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":109,"categories":110,"keywords":111,"publishedAt":75},"01M72JKA9XXB92FS8JHPCYW8CM","subagent-driven-development","Use when executing implementation plans with independent tasks. Dispatches fresh delegate_task per task with two-stage review (spec compliance then code quality).","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fsoftware-development\u002Fsubagent-driven-development\u002FSKILL.md",[],[]]