[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EWYE868SW9WPCAZ98HXSSA":3,"skill-same:01M6EWYE868SW9WPCAZ98HXSSA":83},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":51,"commitSha":52,"skillDirectory":53,"forksCount":14,"repositoryHtmlUrl":54,"repositoryHomepageUrl":55,"ownerHtmlUrl":56,"ownerBlogUrl":6,"ownerDescription":57,"updatedAt":58,"install":59,"resources":72},"01M6EWYE868SW9WPCAZ98HXSSA","hqq-quantization",null,"Half-Quadratic Quantization for LLMs without calibration data. Use when quantizing models to 4\u002F3\u002F2-bit precision without needing calibration datasets, for fast quantization workflows, or when deploying with vLLM or HuggingFace Transformers.","henriquescastilho\u002Fmy-claude","henriquescastilho","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F153296945?v=4","01M6ESH7EXPXTH9CVQ4K16SRG8","Henrique Scheer","01M6ESH7JHEK2PTNNH4HEH787Z",0,"MIT","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-hqq\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"hqq quantization","HQQ","Quantization","use","with","Model Compression","Memory Efficiency","bit","efficiency","models","llms","model","Transformers","vLLM","data","HuggingFace","Optimization","workflows","research","memory","2026-11-10 10:12:30",false,"---\nname: hqq-quantization\ndescription: Half-Quadratic Quantization for LLMs without calibration data. Use when quantizing models to 4\u002F3\u002F2-bit precision without needing calibration datasets, for fast quantization workflows, or when deploying with vLLM or HuggingFace Transformers.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Quantization, HQQ, Optimization, Memory Efficiency, Inference, Model Compression]\ndependencies: [hqq>=0.2.0, torch>=2.0.0]\n---\n\n# HQQ - Half-Quadratic Quantization\n\nFast, calibration-free weight quantization supporting 8\u002F4\u002F3\u002F2\u002F1-bit precision with multiple optimized backends.\n\n## When to use HQQ\n\n**Use HQQ when:**\n- Quantizing models without calibration data (no dataset needed)\n- Need fast quantization (minutes vs hours for GPTQ\u002FAWQ)\n- Deploying with vLLM or HuggingFace Transformers\n- Fine-tuning quantized models with LoRA\u002FPEFT\n- Experimenting with extreme quantization (2-bit, 1-bit)\n\n**Key advantages:**\n- **No calibration**: Quantize any model instantly without sample data\n- **Multiple backends**: PyTorch, ATEN, TorchAO, Marlin, BitBlas for optimized inference\n- **Flexible precision**: 8\u002F4\u002F3\u002F2\u002F1-bit with configurable group sizes\n- **Framework integration**: Native HuggingFace and vLLM support\n- **PEFT compatible**: Fine-tune quantized models with LoRA\n\n**Use alternatives instead:**\n- **AWQ**: Need calibration-based accuracy, production serving\n- **GPTQ**: Maximum accuracy with calibration data available\n- **bitsandbytes**: Simple 8-bit\u002F4-bit without custom backends\n- **llama.cpp\u002FGGUF**: CPU inference, Apple Silicon deployment\n\n## Quick start\n\n### Installation\n\n```bash\npip install hqq\n\n# With specific backend\npip install hqq[torch]      # PyTorch backend\npip install hqq[torchao]    # TorchAO int4 backend\npip install hqq[bitblas]    # BitBlas backend\npip install hqq[marlin]     # Marlin backend\n```\n\n### Basic quantization\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig, HQQLinear\nimport torch.nn as nn\n\n# Configure quantization\nconfig = BaseQuantizeConfig(\n    nbits=4,           # 4-bit quantization\n    group_size=64,     # Group size for quantization\n    axis=1             # Quantize along output dimension\n)\n\n# Quantize a linear layer\nlinear = nn.Linear(4096, 4096)\nhqq_linear = HQQLinear(linear, config)\n\n# Use normally\noutput = hqq_linear(input_tensor)\n```\n\n### Quantize full model with HuggingFace\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Configure HQQ\nquantization_config = HqqConfig(\n    nbits=4,\n    group_size=64,\n    axis=1\n)\n\n# Load and quantize\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quantization_config,\n    device_map=\"auto\"\n)\n\n# Model is quantized and ready to use\n```\n\n## Core concepts\n\n### Quantization configuration\n\nHQQ uses `BaseQuantizeConfig` to define quantization parameters:\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig\n\n# Standard 4-bit config\nconfig_4bit = BaseQuantizeConfig(\n    nbits=4,           # Bits per weight (1-8)\n    group_size=64,     # Weights per quantization group\n    axis=1             # 0=input dim, 1=output dim\n)\n\n# Aggressive 2-bit config\nconfig_2bit = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,     # Smaller groups for low-bit\n    axis=1\n)\n\n# Mixed precision per layer type\nlayer_configs = {\n    \"self_attn.q_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.k_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.v_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"mlp.gate_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.up_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.down_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n}\n```\n\n### HQQLinear layer\n\nThe core quantized layer that replaces `nn.Linear`:\n\n```python\nfrom hqq.core.quantize import HQQLinear\nimport torch\n\n# Create quantized layer\nlinear = torch.nn.Linear(4096, 4096)\nhqq_layer = HQQLinear(linear, config)\n\n# Access quantized weights\nW_q = hqq_layer.W_q           # Quantized weights\nscale = hqq_layer.scale       # Scale factors\nzero = hqq_layer.zero         # Zero points\n\n# Dequantize for inspection\nW_dequant = hqq_layer.dequantize()\n```\n\n### Backends\n\nHQQ supports multiple inference backends for different hardware:\n\n```python\nfrom hqq.core.quantize import HQQLinear\n\n# Available backends\nbackends = [\n    \"pytorch\",          # Pure PyTorch (default)\n    \"pytorch_compile\",  # torch.compile optimized\n    \"aten\",            # Custom CUDA kernels\n    \"torchao_int4\",    # TorchAO int4 matmul\n    \"gemlite\",         # GemLite CUDA kernels\n    \"bitblas\",         # BitBlas optimized\n    \"marlin\",          # Marlin 4-bit kernels\n]\n\n# Set backend globally\nHQQLinear.set_backend(\"torchao_int4\")\n\n# Or per layer\nhqq_layer.set_backend(\"marlin\")\n```\n\n**Backend selection guide:**\n| Backend | Best For | Requirements |\n|---------|----------|--------------|\n| pytorch | Compatibility | Any GPU |\n| pytorch_compile | Moderate speedup | torch>=2.0 |\n| aten | Good balance | CUDA GPU |\n| torchao_int4 | 4-bit inference | torchao installed |\n| marlin | Maximum 4-bit speed | Ampere+ GPU |\n| bitblas | Flexible bit-widths | bitblas installed |\n\n## HuggingFace integration\n\n### Load pre-quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load HQQ-quantized model from Hub\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# Use normally\ninputs = tokenizer(\"Hello, world!\", return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=50)\n```\n\n### Quantize and save\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Quantize\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Save quantized model\nmodel.save_pretrained(\".\u002Fllama-8b-hqq-4bit\")\n\n# Push to Hub\nmodel.push_to_hub(\"my-org\u002FLlama-3.1-8B-HQQ-4bit\")\n```\n\n### Mixed precision quantization\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Different precision per layer type\nconfig = HqqConfig(\n    nbits=4,\n    group_size=64,\n    # Attention layers: higher precision\n    # MLP layers: lower precision for memory savings\n    dynamic_config={\n        \"attn\": {\"nbits\": 4, \"group_size\": 64},\n        \"mlp\": {\"nbits\": 2, \"group_size\": 32}\n    }\n)\n```\n\n## vLLM integration\n\n### Serve HQQ models with vLLM\n\n```python\nfrom vllm import LLM, SamplingParams\n\n# Load HQQ-quantized model\nllm = LLM(\n    model=\"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    quantization=\"hqq\",\n    dtype=\"float16\"\n)\n\n# Generate\nsampling_params = SamplingParams(temperature=0.7, max_tokens=100)\noutputs = llm.generate([\"What is machine learning?\"], sampling_params)\n```\n\n### vLLM with custom HQQ config\n\n```python\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"meta-llama\u002FLlama-3.1-8B\",\n    quantization=\"hqq\",\n    quantization_config={\n        \"nbits\": 4,\n        \"group_size\": 64\n    }\n)\n```\n\n## PEFT\u002FLoRA fine-tuning\n\n### Fine-tune quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\nfrom peft import LoraConfig, get_peft_model\n\n# Load quantized model\nquant_config = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quant_config,\n    device_map=\"auto\"\n)\n\n# Apply LoRA\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\nmodel = get_peft_model(model, lora_config)\n\n# Train normally with Trainer or custom loop\n```\n\n### QLoRA-style training\n\n```python\nfrom transformers import TrainingArguments, Trainer\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fhqq-lora-output\",\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    learning_rate=2e-4,\n    num_train_epochs=3,\n    fp16=True,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    data_collator=data_collator\n)\n\ntrainer.train()\n```\n\n## Quantization workflows\n\n### Workflow 1: Quick model compression\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig\n\n# 1. Configure quantization\nconfig = HqqConfig(nbits=4, group_size=64)\n\n# 2. Load and quantize (no calibration needed!)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# 3. Verify quality\nprompt = \"The capital of France is\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=20)\nprint(tokenizer.decode(outputs[0]))\n\n# 4. Save\nmodel.save_pretrained(\".\u002Fllama-8b-hqq\")\ntokenizer.save_pretrained(\".\u002Fllama-8b-hqq\")\n```\n\n### Workflow 2: Optimize for inference speed\n\n```python\nfrom hqq.core.quantize import HQQLinear\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# 1. Quantize with optimal backend\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# 2. Set fast backend\nHQQLinear.set_backend(\"marlin\")  # or \"torchao_int4\"\n\n# 3. Compile for additional speedup\nimport torch\nmodel = torch.compile(model)\n\n# 4. Benchmark\nimport time\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(model.device)\nstart = time.time()\nfor _ in range(10):\n    model.generate(**inputs, max_new_tokens=100)\nprint(f\"Avg time: {(time.time() - start) \u002F 10:.2f}s\")\n```\n\n## Best practices\n\n1. **Start with 4-bit**: Best quality\u002Fsize tradeoff for most models\n2. **Use group_size=64**: Good balance; smaller for extreme quantization\n3. **Choose backend wisely**: Marlin for 4-bit Ampere+, TorchAO for flexibility\n4. **Verify quality**: Always test generation quality after quantization\n5. **Mixed precision**: Keep attention at higher precision, compress MLP more\n6. **PEFT training**: Use LoRA r=16-32 for good fine-tuning results\n\n## Common issues\n\n**Out of memory during quantization:**\n```python\n# Quantize layer-by-layer\nfrom hqq.models.hf.base import AutoHQQHFModel\n\nmodel = AutoHQQHFModel.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"sequential\"  # Load layers sequentially\n)\n```\n\n**Slow inference:**\n```python\n# Switch to optimized backend\nfrom hqq.core.quantize import HQQLinear\nHQQLinear.set_backend(\"marlin\")  # Requires Ampere+ GPU\n\n# Or compile\nmodel = torch.compile(model, mode=\"reduce-overhead\")\n```\n\n**Poor quality at 2-bit:**\n```python\n# Use smaller group size\nconfig = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,  # Smaller groups help at low bits\n    axis=1\n)\n```\n\n## References\n\n- **[Advanced Usage](references\u002Fadvanced-usage.md)** - Custom backends, mixed precision, optimization\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, debugging, benchmarks\n\n## Resources\n\n- **Repository**: https:\u002F\u002Fgithub.com\u002Fmobiusml\u002Fhqq\n- **Paper**: Half-Quadratic Quantization\n- **HuggingFace Models**: https:\u002F\u002Fhuggingface.co\u002Fmobiuslabsgmbh\n- **Version**: 0.2.0+\n- **License**: Apache 2.0\n","# HQQ - Half-Quadratic Quantization\n\nFast, calibration-free weight quantization supporting 8\u002F4\u002F3\u002F2\u002F1-bit precision with multiple optimized backends.\n\n## When to use HQQ\n\n**Use HQQ when:**\n- Quantizing models without calibration data (no dataset needed)\n- Need fast quantization (minutes vs hours for GPTQ\u002FAWQ)\n- Deploying with vLLM or HuggingFace Transformers\n- Fine-tuning quantized models with LoRA\u002FPEFT\n- Experimenting with extreme quantization (2-bit, 1-bit)\n\n**Key advantages:**\n- **No calibration**: Quantize any model instantly without sample data\n- **Multiple backends**: PyTorch, ATEN, TorchAO, Marlin, BitBlas for optimized inference\n- **Flexible precision**: 8\u002F4\u002F3\u002F2\u002F1-bit with configurable group sizes\n- **Framework integration**: Native HuggingFace and vLLM support\n- **PEFT compatible**: Fine-tune quantized models with LoRA\n\n**Use alternatives instead:**\n- **AWQ**: Need calibration-based accuracy, production serving\n- **GPTQ**: Maximum accuracy with calibration data available\n- **bitsandbytes**: Simple 8-bit\u002F4-bit without custom backends\n- **llama.cpp\u002FGGUF**: CPU inference, Apple Silicon deployment\n\n## Quick start\n\n### Installation\n\n```bash\npip install hqq\n\n# With specific backend\npip install hqq[torch]      # PyTorch backend\npip install hqq[torchao]    # TorchAO int4 backend\npip install hqq[bitblas]    # BitBlas backend\npip install hqq[marlin]     # Marlin backend\n```\n\n### Basic quantization\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig, HQQLinear\nimport torch.nn as nn\n\n# Configure quantization\nconfig = BaseQuantizeConfig(\n    nbits=4,           # 4-bit quantization\n    group_size=64,     # Group size for quantization\n    axis=1             # Quantize along output dimension\n)\n\n# Quantize a linear layer\nlinear = nn.Linear(4096, 4096)\nhqq_linear = HQQLinear(linear, config)\n\n# Use normally\noutput = hqq_linear(input_tensor)\n```\n\n### Quantize full model with HuggingFace\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Configure HQQ\nquantization_config = HqqConfig(\n    nbits=4,\n    group_size=64,\n    axis=1\n)\n\n# Load and quantize\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quantization_config,\n    device_map=\"auto\"\n)\n\n# Model is quantized and ready to use\n```\n\n## Core concepts\n\n### Quantization configuration\n\nHQQ uses `BaseQuantizeConfig` to define quantization parameters:\n\n```python\nfrom hqq.core.quantize import BaseQuantizeConfig\n\n# Standard 4-bit config\nconfig_4bit = BaseQuantizeConfig(\n    nbits=4,           # Bits per weight (1-8)\n    group_size=64,     # Weights per quantization group\n    axis=1             # 0=input dim, 1=output dim\n)\n\n# Aggressive 2-bit config\nconfig_2bit = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,     # Smaller groups for low-bit\n    axis=1\n)\n\n# Mixed precision per layer type\nlayer_configs = {\n    \"self_attn.q_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.k_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"self_attn.v_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n    \"mlp.gate_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.up_proj\": BaseQuantizeConfig(nbits=2, group_size=32),\n    \"mlp.down_proj\": BaseQuantizeConfig(nbits=4, group_size=64),\n}\n```\n\n### HQQLinear layer\n\nThe core quantized layer that replaces `nn.Linear`:\n\n```python\nfrom hqq.core.quantize import HQQLinear\nimport torch\n\n# Create quantized layer\nlinear = torch.nn.Linear(4096, 4096)\nhqq_layer = HQQLinear(linear, config)\n\n# Access quantized weights\nW_q = hqq_layer.W_q           # Quantized weights\nscale = hqq_layer.scale       # Scale factors\nzero = hqq_layer.zero         # Zero points\n\n# Dequantize for inspection\nW_dequant = hqq_layer.dequantize()\n```\n\n### Backends\n\nHQQ supports multiple inference backends for different hardware:\n\n```python\nfrom hqq.core.quantize import HQQLinear\n\n# Available backends\nbackends = [\n    \"pytorch\",          # Pure PyTorch (default)\n    \"pytorch_compile\",  # torch.compile optimized\n    \"aten\",            # Custom CUDA kernels\n    \"torchao_int4\",    # TorchAO int4 matmul\n    \"gemlite\",         # GemLite CUDA kernels\n    \"bitblas\",         # BitBlas optimized\n    \"marlin\",          # Marlin 4-bit kernels\n]\n\n# Set backend globally\nHQQLinear.set_backend(\"torchao_int4\")\n\n# Or per layer\nhqq_layer.set_backend(\"marlin\")\n```\n\n**Backend selection guide:**\n| Backend | Best For | Requirements |\n|---------|----------|--------------|\n| pytorch | Compatibility | Any GPU |\n| pytorch_compile | Moderate speedup | torch>=2.0 |\n| aten | Good balance | CUDA GPU |\n| torchao_int4 | 4-bit inference | torchao installed |\n| marlin | Maximum 4-bit speed | Ampere+ GPU |\n| bitblas | Flexible bit-widths | bitblas installed |\n\n## HuggingFace integration\n\n### Load pre-quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Load HQQ-quantized model from Hub\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# Use normally\ninputs = tokenizer(\"Hello, world!\", return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=50)\n```\n\n### Quantize and save\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Quantize\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Save quantized model\nmodel.save_pretrained(\".\u002Fllama-8b-hqq-4bit\")\n\n# Push to Hub\nmodel.push_to_hub(\"my-org\u002FLlama-3.1-8B-HQQ-4bit\")\n```\n\n### Mixed precision quantization\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# Different precision per layer type\nconfig = HqqConfig(\n    nbits=4,\n    group_size=64,\n    # Attention layers: higher precision\n    # MLP layers: lower precision for memory savings\n    dynamic_config={\n        \"attn\": {\"nbits\": 4, \"group_size\": 64},\n        \"mlp\": {\"nbits\": 2, \"group_size\": 32}\n    }\n)\n```\n\n## vLLM integration\n\n### Serve HQQ models with vLLM\n\n```python\nfrom vllm import LLM, SamplingParams\n\n# Load HQQ-quantized model\nllm = LLM(\n    model=\"mobiuslabsgmbh\u002FLlama-3.1-8B-HQQ-4bit\",\n    quantization=\"hqq\",\n    dtype=\"float16\"\n)\n\n# Generate\nsampling_params = SamplingParams(temperature=0.7, max_tokens=100)\noutputs = llm.generate([\"What is machine learning?\"], sampling_params)\n```\n\n### vLLM with custom HQQ config\n\n```python\nfrom vllm import LLM\n\nllm = LLM(\n    model=\"meta-llama\u002FLlama-3.1-8B\",\n    quantization=\"hqq\",\n    quantization_config={\n        \"nbits\": 4,\n        \"group_size\": 64\n    }\n)\n```\n\n## PEFT\u002FLoRA fine-tuning\n\n### Fine-tune quantized models\n\n```python\nfrom transformers import AutoModelForCausalLM, HqqConfig\nfrom peft import LoraConfig, get_peft_model\n\n# Load quantized model\nquant_config = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=quant_config,\n    device_map=\"auto\"\n)\n\n# Apply LoRA\nlora_config = LoraConfig(\n    r=16,\n    lora_alpha=32,\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\nmodel = get_peft_model(model, lora_config)\n\n# Train normally with Trainer or custom loop\n```\n\n### QLoRA-style training\n\n```python\nfrom transformers import TrainingArguments, Trainer\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fhqq-lora-output\",\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    learning_rate=2e-4,\n    num_train_epochs=3,\n    fp16=True,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    data_collator=data_collator\n)\n\ntrainer.train()\n```\n\n## Quantization workflows\n\n### Workflow 1: Quick model compression\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig\n\n# 1. Configure quantization\nconfig = HqqConfig(nbits=4, group_size=64)\n\n# 2. Load and quantize (no calibration needed!)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-3.1-8B\")\n\n# 3. Verify quality\nprompt = \"The capital of France is\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=20)\nprint(tokenizer.decode(outputs[0]))\n\n# 4. Save\nmodel.save_pretrained(\".\u002Fllama-8b-hqq\")\ntokenizer.save_pretrained(\".\u002Fllama-8b-hqq\")\n```\n\n### Workflow 2: Optimize for inference speed\n\n```python\nfrom hqq.core.quantize import HQQLinear\nfrom transformers import AutoModelForCausalLM, HqqConfig\n\n# 1. Quantize with optimal backend\nconfig = HqqConfig(nbits=4, group_size=64)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# 2. Set fast backend\nHQQLinear.set_backend(\"marlin\")  # or \"torchao_int4\"\n\n# 3. Compile for additional speedup\nimport torch\nmodel = torch.compile(model)\n\n# 4. Benchmark\nimport time\ninputs = tokenizer(\"Hello\", return_tensors=\"pt\").to(model.device)\nstart = time.time()\nfor _ in range(10):\n    model.generate(**inputs, max_new_tokens=100)\nprint(f\"Avg time: {(time.time() - start) \u002F 10:.2f}s\")\n```\n\n## Best practices\n\n1. **Start with 4-bit**: Best quality\u002Fsize tradeoff for most models\n2. **Use group_size=64**: Good balance; smaller for extreme quantization\n3. **Choose backend wisely**: Marlin for 4-bit Ampere+, TorchAO for flexibility\n4. **Verify quality**: Always test generation quality after quantization\n5. **Mixed precision**: Keep attention at higher precision, compress MLP more\n6. **PEFT training**: Use LoRA r=16-32 for good fine-tuning results\n\n## Common issues\n\n**Out of memory during quantization:**\n```python\n# Quantize layer-by-layer\nfrom hqq.models.hf.base import AutoHQQHFModel\n\nmodel = AutoHQQHFModel.from_pretrained(\n    \"meta-llama\u002FLlama-3.1-8B\",\n    quantization_config=config,\n    device_map=\"sequential\"  # Load layers sequentially\n)\n```\n\n**Slow inference:**\n```python\n# Switch to optimized backend\nfrom hqq.core.quantize import HQQLinear\nHQQLinear.set_backend(\"marlin\")  # Requires Ampere+ GPU\n\n# Or compile\nmodel = torch.compile(model, mode=\"reduce-overhead\")\n```\n\n**Poor quality at 2-bit:**\n```python\n# Use smaller group size\nconfig = BaseQuantizeConfig(\n    nbits=2,\n    group_size=16,  # Smaller groups help at low bits\n    axis=1\n)\n```\n\n## References\n\n- **[Advanced Usage](references\u002Fadvanced-usage.md)** - Custom backends, mixed precision, optimization\n- **[Troubleshooting](references\u002Ftroubleshooting.md)** - Common issues, debugging, benchmarks\n\n## Resources\n\n- **Repository**: https:\u002F\u002Fgithub.com\u002Fmobiusml\u002Fhqq\n- **Paper**: Half-Quadratic Quantization\n- **HuggingFace Models**: https:\u002F\u002Fhuggingface.co\u002Fmobiuslabsgmbh\n- **Version**: 0.2.0+\n- **License**: Apache 2.0\n",{"name":5,"tags":44,"author":46,"license":15,"version":47,"description":7,"dependencies":48},[21,20,35,25,45,24],"Inference","Orchestra Research","1.0.0",[49,50],"hqq>=0.2.0","torch>=2.0.0","main","7925716f065257435fdb12e97f7e52842147d5ad",".codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-hqq","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude","https:\u002F\u002Fclaude.com\u002Fclaude-code","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho","Software · Applied AI · Quantitative Modeling · Building with Claude.","2026-06-10 12:24:46",{"source":8,"skillName":5,"commands":60,"prompt":70,"zipAvailable":71},[61,64,67],{"tool":62,"command":63},"npx","npx skills add henriquescastilho\u002Fmy-claude --skill hqq-quantization",{"tool":65,"command":66},"bunx","bunx skills add henriquescastilho\u002Fmy-claude --skill hqq-quantization",{"tool":68,"command":69},"pnpm","pnpm dlx skills add henriquescastilho\u002Fmy-claude --skill hqq-quantization","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-hqq\u002FSKILL.md",true,[73,79],{"path":74,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":77,"sourceHtmlUrl":78},"references\u002Fadvanced-usage.md","REFERENCE","参考文档",14077,"https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-hqq\u002Freferences\u002Fadvanced-usage.md",{"path":80,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":81,"sourceHtmlUrl":82},"references\u002Ftroubleshooting.md",11086,"https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-hqq\u002Freferences\u002Ftroubleshooting.md",[84,125,134,143,165,193],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":106,"publishedAt":39},"01M6F0B6XPQ65AX2X0DWK8KT3E","markitdown","Convert files and office documents to Markdown. Supports PDF, DOCX, PPTX, XLSX, images (with OCR), audio (with transcription), HTML, CSV, JSON, XML, ZIP, YouTube URLs, EPubs and more.","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-scientific-writer\u002Fskills\u002Fmarkitdown\u002FSKILL.md",[90,95,98,101],{"categoryNo":91,"categoryName":92,"taxonomyType":93,"taxonomyTypeDesc":94},"01JBSEEDCAT000000000000002","文档处理","SCENARIO","使用场景",{"categoryNo":96,"categoryName":97,"taxonomyType":93,"taxonomyTypeDesc":94},"01JBSEEDCAT000000000000003","演示文稿",{"categoryNo":99,"categoryName":100,"taxonomyType":93,"taxonomyTypeDesc":94},"01JBSEEDCAT000000000000004","电子表格",{"categoryNo":102,"categoryName":103,"taxonomyType":104,"taxonomyTypeDesc":105},"01JBSEEDCAT000000000000034","行政与办公人员","PROFESSION","适用职业",[86,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,23],"docx","PDF","pptx","xlsx","documents","csv","ocr","Audio","Bash","markdown","convert","read","write","edit","html","images","files","youtube",{"skillNo":126,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":127,"categories":128,"keywords":133,"publishedAt":39},"01M6F06SBHR9F0SDWE282Y8075","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-scientific-writer\u002Fscientific_writer\u002F.claude\u002Fskills\u002Fmarkitdown\u002FSKILL.md",[129,130,131,132],{"categoryNo":91,"categoryName":92,"taxonomyType":93,"taxonomyTypeDesc":94},{"categoryNo":96,"categoryName":97,"taxonomyType":93,"taxonomyTypeDesc":94},{"categoryNo":99,"categoryName":100,"taxonomyType":93,"taxonomyTypeDesc":94},{"categoryNo":102,"categoryName":103,"taxonomyType":104,"taxonomyTypeDesc":105},[86,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,23],{"skillNo":135,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":136,"categories":137,"keywords":142,"publishedAt":39},"01M6F02D4R14EJAPBWVDF9J9VQ","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-scientific-writer\u002F.claude\u002Fskills\u002Fmarkitdown\u002FSKILL.md",[138,139,140,141],{"categoryNo":91,"categoryName":92,"taxonomyType":93,"taxonomyTypeDesc":94},{"categoryNo":96,"categoryName":97,"taxonomyType":93,"taxonomyTypeDesc":94},{"categoryNo":99,"categoryName":100,"taxonomyType":93,"taxonomyTypeDesc":94},{"categoryNo":102,"categoryName":103,"taxonomyType":104,"taxonomyTypeDesc":105},[86,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,23],{"skillNo":144,"name":145,"nameZh":6,"description":146,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":147,"categories":148,"keywords":149,"publishedAt":39},"01M6F000KS6YB4MKS774G4R8ND","zarr-python","Chunked N-D arrays for cloud storage. Compressed arrays, parallel I\u002FO, S3\u002FGCS integration, NumPy\u002FDask\u002FXarray compatible, for large-scale scientific computing pipelines.","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-scientific-skills\u002Fscientific-skills\u002Fzarr-python\u002FSKILL.md",[],[150,151,152,153,154,155,156,157,158,159,160,161,162,163,164],"python","zarr python","storage","integration","s3","gcs","cloud","dask","parallel","pipelines","arrays","numpy","scientific computing","scale","scientific",{"skillNo":166,"name":167,"nameZh":6,"description":168,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":169,"categories":170,"keywords":174,"publishedAt":39},"01M6EZZJWFXM1GR78KTW6J0DN8","what-if-oracle","Run structured What-If scenario analysis with multi-branch possibility exploration. Use this skill when the user asks speculative questions like \"what if...\", \"what would happen if...\", \"what are the possibilities\", \"explore scenarios\", \"scenario analysis\", \"possibility space\", \"what could go wrong\", \"best case \u002F worst case\", \"risk analysis\", \"contingency planning\", \"strategic options\", or any question about uncertain futures. Also trigger when the user faces a fork-in-the-road decision, wants to stress-test an idea, or needs to think through consequences before committing.","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-scientific-skills\u002Fscientific-skills\u002Fwhat-if-oracle\u002FSKILL.md",[171],{"categoryNo":172,"categoryName":173,"taxonomyType":93,"taxonomyTypeDesc":94},"01JBSEEDCAT000000000000023","Skill与智能体开发",[175,176,177,178,179,180,181,182,183,184,185,118,119,186,187,188,189,190,191,192],"oracle","what if oracle","go","run","ux","futures","analysis","explore","decision","branch","question","idea","exploration","options","stress-test","strategies","scenario","stress",{"skillNo":194,"name":195,"nameZh":6,"description":196,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":197,"categories":198,"keywords":209,"publishedAt":39},"01M6EZZCXAEB49RXGQCJX5E8GB","venue-templates","Access comprehensive LaTeX templates, formatting requirements, and submission guidelines for major scientific publication venues (Nature, Science, PLOS, IEEE, ACM), academic conferences (NeurIPS, ICML, CVPR, CHI), research posters, and grant proposals (NSF, NIH, DOE, DARPA). This skill should be used when preparing manuscripts for journal submission, conference papers, research posters, or grant proposals and need venue-specific formatting requirements and templates.","https:\u002F\u002Fgithub.com\u002Fhenriquescastilho\u002Fmy-claude\u002Fblob\u002Fmain\u002F.codex\u002Fvendor_imports\u002Fclaude\u002Fmarketplaces\u002Fclaude-scientific-skills\u002Fscientific-skills\u002Fvenue-templates\u002FSKILL.md",[199,202,205,206],{"categoryNo":200,"categoryName":201,"taxonomyType":93,"taxonomyTypeDesc":94},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":203,"categoryName":204,"taxonomyType":93,"taxonomyTypeDesc":94},"01JBSEEDCAT000000000000021","资料检索与研究",{"categoryNo":172,"categoryName":173,"taxonomyType":93,"taxonomyTypeDesc":94},{"categoryNo":207,"categoryName":208,"taxonomyType":104,"taxonomyTypeDesc":105},"01JBSEEDCAT000000000000036","内容创作者",[210,211,37,212,213,214,215,216,115,217,118,119,120,218,219,220,221,222,164,223],"templates","venue templates","guidelines","NeurIPS","ICML","LaTeX","science","access","proposals","Academic","requirements","formatting","journal","chi"]