[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M72FMPC907CGKRKB7M46H2MQ":3,"skill-same:01M72FMPC907CGKRKB7M46H2MQ":78},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":43,"commitSha":44,"skillDirectory":45,"forksCount":14,"repositoryHtmlUrl":46,"repositoryHomepageUrl":6,"ownerHtmlUrl":47,"ownerBlogUrl":6,"ownerDescription":48,"updatedAt":49,"install":50,"resources":63},"01M72FMPC907CGKRKB7M46H2MQ","quantizing-models-bitsandbytes",null,"Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss. Use when GPU memory is limited, need to fit larger models, or want faster inference. Supports INT8, NF4, FP4 formats, QLoRA training, and 8-bit optimizers. Works with HuggingFace Transformers.","cocyuhao\u002Fai-dev-tooling-research-archive","cocyuhao","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F269083740?v=4","01M72F908ET5R76KP9NXS6Y2GJ","Hao Yu","01M72F909376KCNJVKG8MEPY2Q",0,"MIT","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes\u002FSKILL.md",[],[],"2026-11-10 18:39:20",false,"---\nname: quantizing-models-bitsandbytes\ndescription: Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss. Use when GPU memory is limited, need to fit larger models, or want faster inference. Supports INT8, NF4, FP4 formats, QLoRA training, and 8-bit optimizers. Works with HuggingFace Transformers.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Optimization, Bitsandbytes, Quantization, 8-Bit, 4-Bit, Memory Optimization, QLoRA, NF4, INT8, HuggingFace, Efficient Inference]\ndependencies: [bitsandbytes, transformers, accelerate, torch]\n---\n\n# bitsandbytes - LLM Quantization\n\n## Quick start\n\nbitsandbytes reduces LLM memory by 50% (8-bit) or 75% (4-bit) with \u003C1% accuracy loss.\n\n**Installation**:\n```bash\npip install bitsandbytes transformers accelerate\n```\n\n**8-bit quantization** (50% memory reduction):\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nconfig = BitsAndBytesConfig(load_in_8bit=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Memory: 14GB → 7GB\n```\n\n**4-bit quantization** (75% memory reduction):\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16\n)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Memory: 14GB → 3.5GB\n```\n\n## Common workflows\n\n### Workflow 1: Load large model in limited GPU memory\n\nCopy this checklist:\n\n```\nQuantization Loading:\n- [ ] Step 1: Calculate memory requirements\n- [ ] Step 2: Choose quantization level (4-bit or 8-bit)\n- [ ] Step 3: Configure quantization\n- [ ] Step 4: Load and verify model\n```\n\n**Step 1: Calculate memory requirements**\n\nEstimate model memory:\n```\nFP16 memory (GB) = Parameters × 2 bytes \u002F 1e9\nINT8 memory (GB) = Parameters × 1 byte \u002F 1e9\nINT4 memory (GB) = Parameters × 0.5 bytes \u002F 1e9\n\nExample (Llama 2 7B):\nFP16: 7B × 2 \u002F 1e9 = 14 GB\nINT8: 7B × 1 \u002F 1e9 = 7 GB\nINT4: 7B × 0.5 \u002F 1e9 = 3.5 GB\n```\n\n**Step 2: Choose quantization level**\n\n| GPU VRAM | Model Size | Recommended |\n|----------|------------|-------------|\n| 8 GB | 3B | 4-bit |\n| 12 GB | 7B | 4-bit |\n| 16 GB | 7B | 8-bit or 4-bit |\n| 24 GB | 13B | 8-bit or 70B 4-bit |\n| 40+ GB | 70B | 8-bit |\n\n**Step 3: Configure quantization**\n\nFor 8-bit (better accuracy):\n```python\nfrom transformers import BitsAndBytesConfig\nimport torch\n\nconfig = BitsAndBytesConfig(\n    load_in_8bit=True,\n    llm_int8_threshold=6.0,  # Outlier threshold\n    llm_int8_has_fp16_weight=False\n)\n```\n\nFor 4-bit (maximum memory savings):\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16,  # Compute in FP16\n    bnb_4bit_quant_type=\"nf4\",  # NormalFloat4 (recommended)\n    bnb_4bit_use_double_quant=True  # Nested quantization\n)\n```\n\n**Step 4: Load and verify model**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-13b-hf\",\n    quantization_config=config,\n    device_map=\"auto\",  # Automatic device placement\n    torch_dtype=torch.float16\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-13b-hf\")\n\n# Test inference\ninputs = tokenizer(\"Hello, how are you?\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_length=50)\nprint(tokenizer.decode(outputs[0]))\n\n# Check memory\nimport torch\nprint(f\"Memory allocated: {torch.cuda.memory_allocated()\u002F1e9:.2f}GB\")\n```\n\n### Workflow 2: Fine-tune with QLoRA (4-bit training)\n\nQLoRA enables fine-tuning large models on consumer GPUs.\n\nCopy this checklist:\n\n```\nQLoRA Fine-tuning:\n- [ ] Step 1: Install dependencies\n- [ ] Step 2: Configure 4-bit base model\n- [ ] Step 3: Add LoRA adapters\n- [ ] Step 4: Train with standard Trainer\n```\n\n**Step 1: Install dependencies**\n\n```bash\npip install bitsandbytes transformers peft accelerate datasets\n```\n\n**Step 2: Configure 4-bit base model**\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nimport torch\n\nbnb_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_use_double_quant=True\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=bnb_config,\n    device_map=\"auto\"\n)\n```\n\n**Step 3: Add LoRA adapters**\n\n```python\nfrom peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training\n\n# Prepare model for training\nmodel = prepare_model_for_kbit_training(model)\n\n# Configure LoRA\nlora_config = LoraConfig(\n    r=16,  # LoRA rank\n    lora_alpha=32,  # LoRA alpha\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\n# Add LoRA adapters\nmodel = get_peft_model(model, lora_config)\nmodel.print_trainable_parameters()\n# Output: trainable params: 4.2M || all params: 6.7B || trainable%: 0.06%\n```\n\n**Step 4: Train with standard Trainer**\n\n```python\nfrom transformers import Trainer, TrainingArguments\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fqlora-output\",\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    num_train_epochs=3,\n    learning_rate=2e-4,\n    fp16=True,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    tokenizer=tokenizer\n)\n\ntrainer.train()\n\n# Save LoRA adapters (only ~20MB)\nmodel.save_pretrained(\".\u002Fqlora-adapters\")\n```\n\n### Workflow 3: 8-bit optimizer for memory-efficient training\n\nUse 8-bit Adam\u002FAdamW to reduce optimizer memory by 75%.\n\n```\n8-bit Optimizer Setup:\n- [ ] Step 1: Replace standard optimizer\n- [ ] Step 2: Configure training\n- [ ] Step 3: Monitor memory savings\n```\n\n**Step 1: Replace standard optimizer**\n\n```python\nimport bitsandbytes as bnb\nfrom transformers import Trainer, TrainingArguments\n\n# Instead of torch.optim.AdamW\nmodel = AutoModelForCausalLM.from_pretrained(\"model-name\")\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Foutput\",\n    per_device_train_batch_size=8,\n    optim=\"paged_adamw_8bit\",  # 8-bit optimizer\n    learning_rate=5e-5\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset\n)\n\ntrainer.train()\n```\n\n**Manual optimizer usage**:\n```python\nimport bitsandbytes as bnb\n\noptimizer = bnb.optim.AdamW8bit(\n    model.parameters(),\n    lr=1e-4,\n    betas=(0.9, 0.999),\n    eps=1e-8\n)\n\n# Training loop\nfor batch in dataloader:\n    loss = model(**batch).loss\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n```\n\n**Step 2: Configure training**\n\nCompare memory:\n```\nStandard AdamW optimizer memory = model_params × 8 bytes (states)\n8-bit AdamW memory = model_params × 2 bytes\nSavings = 75% optimizer memory\n\nExample (Llama 2 7B):\nStandard: 7B × 8 = 56 GB\n8-bit: 7B × 2 = 14 GB\nSavings: 42 GB\n```\n\n**Step 3: Monitor memory savings**\n\n```python\nimport torch\n\nbefore = torch.cuda.memory_allocated()\n\n# Training step\noptimizer.step()\n\nafter = torch.cuda.memory_allocated()\nprint(f\"Memory used: {(after-before)\u002F1e9:.2f}GB\")\n```\n\n## When to use vs alternatives\n\n**Use bitsandbytes when:**\n- GPU memory limited (need to fit larger model)\n- Training with QLoRA (fine-tune 70B on single GPU)\n- Inference only (50-75% memory reduction)\n- Using HuggingFace Transformers\n- Acceptable 0-2% accuracy degradation\n\n**Use alternatives instead:**\n- **GPTQ\u002FAWQ**: Production serving (faster inference than bitsandbytes)\n- **GGUF**: CPU inference (llama.cpp)\n- **FP8**: H100 GPUs (hardware FP8 faster)\n- **Full precision**: Accuracy critical, memory not constrained\n\n## Common issues\n\n**Issue: CUDA error during loading**\n\nInstall matching CUDA version:\n```bash\n# Check CUDA version\nnvcc --version\n\n# Install matching bitsandbytes\npip install bitsandbytes --no-cache-dir\n```\n\n**Issue: Model loading slow**\n\nUse CPU offload for large models:\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"model-name\",\n    quantization_config=config,\n    device_map=\"auto\",\n    max_memory={0: \"20GB\", \"cpu\": \"30GB\"}  # Offload to CPU\n)\n```\n\n**Issue: Lower accuracy than expected**\n\nTry 8-bit instead of 4-bit:\n```python\nconfig = BitsAndBytesConfig(load_in_8bit=True)\n# 8-bit has \u003C0.5% accuracy loss vs 1-2% for 4-bit\n```\n\nOr use NF4 with double quantization:\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",  # Better than fp4\n    bnb_4bit_use_double_quant=True  # Extra accuracy\n)\n```\n\n**Issue: OOM even with 4-bit**\n\nEnable CPU offload:\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"model-name\",\n    quantization_config=config,\n    device_map=\"auto\",\n    offload_folder=\"offload\",  # Disk offload\n    offload_state_dict=True\n)\n```\n\n## Advanced topics\n\n**QLoRA training guide**: See [references\u002Fqlora-training.md](references\u002Fqlora-training.md) for complete fine-tuning workflows, hyperparameter tuning, and multi-GPU training.\n\n**Quantization formats**: See [references\u002Fquantization-formats.md](references\u002Fquantization-formats.md) for INT8, NF4, FP4 comparison, double quantization, and custom quantization configs.\n\n**Memory optimization**: See [references\u002Fmemory-optimization.md](references\u002Fmemory-optimization.md) for CPU offloading strategies, gradient checkpointing, and memory profiling.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA with compute capability 7.0+ (Turing, Ampere, Hopper)\n- **VRAM**: Depends on model and quantization\n  - 4-bit Llama 2 7B: 4GB\n  - 4-bit Llama 2 13B: 8GB\n  - 4-bit Llama 2 70B: 24GB\n- **CUDA**: 11.1+ (12.0+ recommended)\n- **PyTorch**: 2.0+\n\n**Supported platforms**: NVIDIA GPUs (primary), AMD ROCm, Intel GPUs (experimental)\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002Fbitsandbytes-foundation\u002Fbitsandbytes\n- HuggingFace docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftransformers\u002Fquantization\u002Fbitsandbytes\n- QLoRA paper: \"QLoRA: Efficient Finetuning of Quantized LLMs\" (2023)\n- LLM.int8() paper: \"LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale\" (2022)\n\n\n\n","# bitsandbytes - LLM Quantization\n\n## Quick start\n\nbitsandbytes reduces LLM memory by 50% (8-bit) or 75% (4-bit) with \u003C1% accuracy loss.\n\n**Installation**:\n```bash\npip install bitsandbytes transformers accelerate\n```\n\n**8-bit quantization** (50% memory reduction):\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nconfig = BitsAndBytesConfig(load_in_8bit=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Memory: 14GB → 7GB\n```\n\n**4-bit quantization** (75% memory reduction):\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16\n)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Memory: 14GB → 3.5GB\n```\n\n## Common workflows\n\n### Workflow 1: Load large model in limited GPU memory\n\nCopy this checklist:\n\n```\nQuantization Loading:\n- [ ] Step 1: Calculate memory requirements\n- [ ] Step 2: Choose quantization level (4-bit or 8-bit)\n- [ ] Step 3: Configure quantization\n- [ ] Step 4: Load and verify model\n```\n\n**Step 1: Calculate memory requirements**\n\nEstimate model memory:\n```\nFP16 memory (GB) = Parameters × 2 bytes \u002F 1e9\nINT8 memory (GB) = Parameters × 1 byte \u002F 1e9\nINT4 memory (GB) = Parameters × 0.5 bytes \u002F 1e9\n\nExample (Llama 2 7B):\nFP16: 7B × 2 \u002F 1e9 = 14 GB\nINT8: 7B × 1 \u002F 1e9 = 7 GB\nINT4: 7B × 0.5 \u002F 1e9 = 3.5 GB\n```\n\n**Step 2: Choose quantization level**\n\n| GPU VRAM | Model Size | Recommended |\n|----------|------------|-------------|\n| 8 GB | 3B | 4-bit |\n| 12 GB | 7B | 4-bit |\n| 16 GB | 7B | 8-bit or 4-bit |\n| 24 GB | 13B | 8-bit or 70B 4-bit |\n| 40+ GB | 70B | 8-bit |\n\n**Step 3: Configure quantization**\n\nFor 8-bit (better accuracy):\n```python\nfrom transformers import BitsAndBytesConfig\nimport torch\n\nconfig = BitsAndBytesConfig(\n    load_in_8bit=True,\n    llm_int8_threshold=6.0,  # Outlier threshold\n    llm_int8_has_fp16_weight=False\n)\n```\n\nFor 4-bit (maximum memory savings):\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16,  # Compute in FP16\n    bnb_4bit_quant_type=\"nf4\",  # NormalFloat4 (recommended)\n    bnb_4bit_use_double_quant=True  # Nested quantization\n)\n```\n\n**Step 4: Load and verify model**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-13b-hf\",\n    quantization_config=config,\n    device_map=\"auto\",  # Automatic device placement\n    torch_dtype=torch.float16\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-13b-hf\")\n\n# Test inference\ninputs = tokenizer(\"Hello, how are you?\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_length=50)\nprint(tokenizer.decode(outputs[0]))\n\n# Check memory\nimport torch\nprint(f\"Memory allocated: {torch.cuda.memory_allocated()\u002F1e9:.2f}GB\")\n```\n\n### Workflow 2: Fine-tune with QLoRA (4-bit training)\n\nQLoRA enables fine-tuning large models on consumer GPUs.\n\nCopy this checklist:\n\n```\nQLoRA Fine-tuning:\n- [ ] Step 1: Install dependencies\n- [ ] Step 2: Configure 4-bit base model\n- [ ] Step 3: Add LoRA adapters\n- [ ] Step 4: Train with standard Trainer\n```\n\n**Step 1: Install dependencies**\n\n```bash\npip install bitsandbytes transformers peft accelerate datasets\n```\n\n**Step 2: Configure 4-bit base model**\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nimport torch\n\nbnb_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_use_double_quant=True\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=bnb_config,\n    device_map=\"auto\"\n)\n```\n\n**Step 3: Add LoRA adapters**\n\n```python\nfrom peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training\n\n# Prepare model for training\nmodel = prepare_model_for_kbit_training(model)\n\n# Configure LoRA\nlora_config = LoraConfig(\n    r=16,  # LoRA rank\n    lora_alpha=32,  # LoRA alpha\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\n# Add LoRA adapters\nmodel = get_peft_model(model, lora_config)\nmodel.print_trainable_parameters()\n# Output: trainable params: 4.2M || all params: 6.7B || trainable%: 0.06%\n```\n\n**Step 4: Train with standard Trainer**\n\n```python\nfrom transformers import Trainer, TrainingArguments\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fqlora-output\",\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    num_train_epochs=3,\n    learning_rate=2e-4,\n    fp16=True,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    tokenizer=tokenizer\n)\n\ntrainer.train()\n\n# Save LoRA adapters (only ~20MB)\nmodel.save_pretrained(\".\u002Fqlora-adapters\")\n```\n\n### Workflow 3: 8-bit optimizer for memory-efficient training\n\nUse 8-bit Adam\u002FAdamW to reduce optimizer memory by 75%.\n\n```\n8-bit Optimizer Setup:\n- [ ] Step 1: Replace standard optimizer\n- [ ] Step 2: Configure training\n- [ ] Step 3: Monitor memory savings\n```\n\n**Step 1: Replace standard optimizer**\n\n```python\nimport bitsandbytes as bnb\nfrom transformers import Trainer, TrainingArguments\n\n# Instead of torch.optim.AdamW\nmodel = AutoModelForCausalLM.from_pretrained(\"model-name\")\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Foutput\",\n    per_device_train_batch_size=8,\n    optim=\"paged_adamw_8bit\",  # 8-bit optimizer\n    learning_rate=5e-5\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset\n)\n\ntrainer.train()\n```\n\n**Manual optimizer usage**:\n```python\nimport bitsandbytes as bnb\n\noptimizer = bnb.optim.AdamW8bit(\n    model.parameters(),\n    lr=1e-4,\n    betas=(0.9, 0.999),\n    eps=1e-8\n)\n\n# Training loop\nfor batch in dataloader:\n    loss = model(**batch).loss\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n```\n\n**Step 2: Configure training**\n\nCompare memory:\n```\nStandard AdamW optimizer memory = model_params × 8 bytes (states)\n8-bit AdamW memory = model_params × 2 bytes\nSavings = 75% optimizer memory\n\nExample (Llama 2 7B):\nStandard: 7B × 8 = 56 GB\n8-bit: 7B × 2 = 14 GB\nSavings: 42 GB\n```\n\n**Step 3: Monitor memory savings**\n\n```python\nimport torch\n\nbefore = torch.cuda.memory_allocated()\n\n# Training step\noptimizer.step()\n\nafter = torch.cuda.memory_allocated()\nprint(f\"Memory used: {(after-before)\u002F1e9:.2f}GB\")\n```\n\n## When to use vs alternatives\n\n**Use bitsandbytes when:**\n- GPU memory limited (need to fit larger model)\n- Training with QLoRA (fine-tune 70B on single GPU)\n- Inference only (50-75% memory reduction)\n- Using HuggingFace Transformers\n- Acceptable 0-2% accuracy degradation\n\n**Use alternatives instead:**\n- **GPTQ\u002FAWQ**: Production serving (faster inference than bitsandbytes)\n- **GGUF**: CPU inference (llama.cpp)\n- **FP8**: H100 GPUs (hardware FP8 faster)\n- **Full precision**: Accuracy critical, memory not constrained\n\n## Common issues\n\n**Issue: CUDA error during loading**\n\nInstall matching CUDA version:\n```bash\n# Check CUDA version\nnvcc --version\n\n# Install matching bitsandbytes\npip install bitsandbytes --no-cache-dir\n```\n\n**Issue: Model loading slow**\n\nUse CPU offload for large models:\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"model-name\",\n    quantization_config=config,\n    device_map=\"auto\",\n    max_memory={0: \"20GB\", \"cpu\": \"30GB\"}  # Offload to CPU\n)\n```\n\n**Issue: Lower accuracy than expected**\n\nTry 8-bit instead of 4-bit:\n```python\nconfig = BitsAndBytesConfig(load_in_8bit=True)\n# 8-bit has \u003C0.5% accuracy loss vs 1-2% for 4-bit\n```\n\nOr use NF4 with double quantization:\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",  # Better than fp4\n    bnb_4bit_use_double_quant=True  # Extra accuracy\n)\n```\n\n**Issue: OOM even with 4-bit**\n\nEnable CPU offload:\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"model-name\",\n    quantization_config=config,\n    device_map=\"auto\",\n    offload_folder=\"offload\",  # Disk offload\n    offload_state_dict=True\n)\n```\n\n## Advanced topics\n\n**QLoRA training guide**: See [references\u002Fqlora-training.md](references\u002Fqlora-training.md) for complete fine-tuning workflows, hyperparameter tuning, and multi-GPU training.\n\n**Quantization formats**: See [references\u002Fquantization-formats.md](references\u002Fquantization-formats.md) for INT8, NF4, FP4 comparison, double quantization, and custom quantization configs.\n\n**Memory optimization**: See [references\u002Fmemory-optimization.md](references\u002Fmemory-optimization.md) for CPU offloading strategies, gradient checkpointing, and memory profiling.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA with compute capability 7.0+ (Turing, Ampere, Hopper)\n- **VRAM**: Depends on model and quantization\n  - 4-bit Llama 2 7B: 4GB\n  - 4-bit Llama 2 13B: 8GB\n  - 4-bit Llama 2 70B: 24GB\n- **CUDA**: 11.1+ (12.0+ recommended)\n- **PyTorch**: 2.0+\n\n**Supported platforms**: NVIDIA GPUs (primary), AMD ROCm, Intel GPUs (experimental)\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002Fbitsandbytes-foundation\u002Fbitsandbytes\n- HuggingFace docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftransformers\u002Fquantization\u002Fbitsandbytes\n- QLoRA paper: \"QLoRA: Efficient Finetuning of Quantized LLMs\" (2023)\n- LLM.int8() paper: \"LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale\" (2022)\n\n\n\n",{"name":5,"tags":24,"author":36,"license":15,"version":37,"description":7,"dependencies":38},[25,26,27,28,29,30,31,32,33,34,35],"Optimization","Bitsandbytes","Quantization","8-Bit","4-Bit","Memory Optimization","QLoRA","NF4","INT8","HuggingFace","Efficient Inference","Orchestra Research","1.0.0",[39,40,41,42],"bitsandbytes","transformers","accelerate","torch","main","f4a24bd1ab047411d4ae57ee9a96701a7706e779","claude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive","https:\u002F\u002Fgithub.com\u002Fcocyuhao","Beijing Wuzi University","2026-05-12 10:57:02",{"source":8,"skillName":5,"commands":51,"prompt":61,"zipAvailable":62},[52,55,58],{"tool":53,"command":54},"npx","npx skills add cocyuhao\u002Fai-dev-tooling-research-archive --skill quantizing-models-bitsandbytes",{"tool":56,"command":57},"bunx","bunx skills add cocyuhao\u002Fai-dev-tooling-research-archive --skill quantizing-models-bitsandbytes",{"tool":59,"command":60},"pnpm","pnpm dlx skills add cocyuhao\u002Fai-dev-tooling-research-archive --skill quantizing-models-bitsandbytes","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes\u002FSKILL.md",true,[64,70,74],{"path":65,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":68,"sourceHtmlUrl":69},"references\u002Fmemory-optimization.md","REFERENCE","参考文档",12642,"https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes\u002Freferences\u002Fmemory-optimization.md",{"path":71,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":72,"sourceHtmlUrl":73},"references\u002Fqlora-training.md",12015,"https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes\u002Freferences\u002Fqlora-training.md",{"path":75,"resourceType":66,"resourceTypeDesc":67,"sizeBytes":76,"sourceHtmlUrl":77},"references\u002Fquantization-formats.md",10258,"https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fclaude-code-templates\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes\u002Freferences\u002Fquantization-formats.md",[79,87,95,102,109,116],{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":83,"categories":84,"keywords":85,"publishedAt":86},"01M72JQ593HE3SRJD3446CCKA8","web-access","所有联网操作必须通过此 skill 处理，包括：搜索、网页抓取、登录后操作、网络交互等。 触发场景：用户要求搜索信息、查看网页内容、访问需要登录的网站、操作网页界面、抓取社交媒体内容（小红书、微博、推特等）、读取动态渲染页面、以及任何需要真实浏览器环境的网络任务。","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fweb-access\u002FSKILL.md",[],[],"2026-11-10 22:39:42",{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":91,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":86},"01M72JKXJYFT2RBJ2HP5ZRA853","last30days","Multi-query social research across Reddit, X, YouTube, TikTok, Instagram, Hacker News, Polymarket, and the web. Intelligent query planning with Gemini\u002FOpenAI fallback.","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Flast30days-skill\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Flast30days-skill\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":86},"01M72JKMRQSD0ZY66KQ863HVY5","writing-plans","Use when you have a spec or requirements for a multi-step task. Creates comprehensive implementation plans with bite-sized tasks, exact file paths, and complete code examples.","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fsoftware-development\u002Fwriting-plans\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":86},"01M72JKHCV8DT6VWSD4KS5YWT9","test-driven-development","Use when implementing any feature or bugfix, before writing implementation code. Enforces RED-GREEN-REFACTOR cycle with test-first approach.","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fsoftware-development\u002Ftest-driven-development\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":86},"01M72JKD7VX7EQM38R704F3KT1","systematic-debugging","Use when encountering any bug, test failure, or unexpected behavior. 4-phase root cause investigation — NO fixes without understanding the problem first.","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fsoftware-development\u002Fsystematic-debugging\u002FSKILL.md",[],[],{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":86},"01M72JKA9XXB92FS8JHPCYW8CM","subagent-driven-development","Use when executing implementation plans with independent tasks. Dispatches fresh delegate_task per task with two-stage review (spec compliance then code quality).","https:\u002F\u002Fgithub.com\u002Fcocyuhao\u002Fai-dev-tooling-research-archive\u002Fblob\u002Fmain\u002Fhermes-agent\u002Fskills\u002Fsoftware-development\u002Fsubagent-driven-development\u002FSKILL.md",[],[]]