[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7CKMHFS7TDWM39A3SCPNP0Z":3,"skill-same:01M7CKMHFS7TDWM39A3SCPNP0Z":79},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":43,"commitSha":44,"skillDirectory":45,"forksCount":46,"repositoryHtmlUrl":47,"repositoryHomepageUrl":6,"ownerHtmlUrl":48,"ownerBlogUrl":6,"ownerDescription":49,"updatedAt":50,"install":51,"resources":64},"01M7CKMHFS7TDWM39A3SCPNP0Z","quantizing-models-bitsandbytes",null,"Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss. Use when GPU memory is limited, need to fit larger models, or want faster inference. Supports INT8, NF4, FP4 formats, QLoRA training, and 8-bit optimizers. Works with HuggingFace Transformers.","Carchaloneo22\u002FClaude-Code-template","Carchaloneo22","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F159488757?v=4","01M7CK5MQXAFZE3W7SAP69KNYR","Carlos Chaparro","01M7CK5MRFE26CDPEQDN5RSQPK",2,"MIT","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: quantizing-models-bitsandbytes\ndescription: Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss. Use when GPU memory is limited, need to fit larger models, or want faster inference. Supports INT8, NF4, FP4 formats, QLoRA training, and 8-bit optimizers. Works with HuggingFace Transformers.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Optimization, Bitsandbytes, Quantization, 8-Bit, 4-Bit, Memory Optimization, QLoRA, NF4, INT8, HuggingFace, Efficient Inference]\ndependencies: [bitsandbytes, transformers, accelerate, torch]\n---\n\n# bitsandbytes - LLM Quantization\n\n## Quick start\n\nbitsandbytes reduces LLM memory by 50% (8-bit) or 75% (4-bit) with \u003C1% accuracy loss.\n\n**Installation**:\n```bash\npip install bitsandbytes transformers accelerate\n```\n\n**8-bit quantization** (50% memory reduction):\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nconfig = BitsAndBytesConfig(load_in_8bit=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Memory: 14GB → 7GB\n```\n\n**4-bit quantization** (75% memory reduction):\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16\n)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Memory: 14GB → 3.5GB\n```\n\n## Common workflows\n\n### Workflow 1: Load large model in limited GPU memory\n\nCopy this checklist:\n\n```\nQuantization Loading:\n- [ ] Step 1: Calculate memory requirements\n- [ ] Step 2: Choose quantization level (4-bit or 8-bit)\n- [ ] Step 3: Configure quantization\n- [ ] Step 4: Load and verify model\n```\n\n**Step 1: Calculate memory requirements**\n\nEstimate model memory:\n```\nFP16 memory (GB) = Parameters × 2 bytes \u002F 1e9\nINT8 memory (GB) = Parameters × 1 byte \u002F 1e9\nINT4 memory (GB) = Parameters × 0.5 bytes \u002F 1e9\n\nExample (Llama 2 7B):\nFP16: 7B × 2 \u002F 1e9 = 14 GB\nINT8: 7B × 1 \u002F 1e9 = 7 GB\nINT4: 7B × 0.5 \u002F 1e9 = 3.5 GB\n```\n\n**Step 2: Choose quantization level**\n\n| GPU VRAM | Model Size | Recommended |\n|----------|------------|-------------|\n| 8 GB | 3B | 4-bit |\n| 12 GB | 7B | 4-bit |\n| 16 GB | 7B | 8-bit or 4-bit |\n| 24 GB | 13B | 8-bit or 70B 4-bit |\n| 40+ GB | 70B | 8-bit |\n\n**Step 3: Configure quantization**\n\nFor 8-bit (better accuracy):\n```python\nfrom transformers import BitsAndBytesConfig\nimport torch\n\nconfig = BitsAndBytesConfig(\n    load_in_8bit=True,\n    llm_int8_threshold=6.0,  # Outlier threshold\n    llm_int8_has_fp16_weight=False\n)\n```\n\nFor 4-bit (maximum memory savings):\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16,  # Compute in FP16\n    bnb_4bit_quant_type=\"nf4\",  # NormalFloat4 (recommended)\n    bnb_4bit_use_double_quant=True  # Nested quantization\n)\n```\n\n**Step 4: Load and verify model**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-13b-hf\",\n    quantization_config=config,\n    device_map=\"auto\",  # Automatic device placement\n    torch_dtype=torch.float16\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-13b-hf\")\n\n# Test inference\ninputs = tokenizer(\"Hello, how are you?\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_length=50)\nprint(tokenizer.decode(outputs[0]))\n\n# Check memory\nimport torch\nprint(f\"Memory allocated: {torch.cuda.memory_allocated()\u002F1e9:.2f}GB\")\n```\n\n### Workflow 2: Fine-tune with QLoRA (4-bit training)\n\nQLoRA enables fine-tuning large models on consumer GPUs.\n\nCopy this checklist:\n\n```\nQLoRA Fine-tuning:\n- [ ] Step 1: Install dependencies\n- [ ] Step 2: Configure 4-bit base model\n- [ ] Step 3: Add LoRA adapters\n- [ ] Step 4: Train with standard Trainer\n```\n\n**Step 1: Install dependencies**\n\n```bash\npip install bitsandbytes transformers peft accelerate datasets\n```\n\n**Step 2: Configure 4-bit base model**\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nimport torch\n\nbnb_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_use_double_quant=True\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=bnb_config,\n    device_map=\"auto\"\n)\n```\n\n**Step 3: Add LoRA adapters**\n\n```python\nfrom peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training\n\n# Prepare model for training\nmodel = prepare_model_for_kbit_training(model)\n\n# Configure LoRA\nlora_config = LoraConfig(\n    r=16,  # LoRA rank\n    lora_alpha=32,  # LoRA alpha\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\n# Add LoRA adapters\nmodel = get_peft_model(model, lora_config)\nmodel.print_trainable_parameters()\n# Output: trainable params: 4.2M || all params: 6.7B || trainable%: 0.06%\n```\n\n**Step 4: Train with standard Trainer**\n\n```python\nfrom transformers import Trainer, TrainingArguments\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fqlora-output\",\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    num_train_epochs=3,\n    learning_rate=2e-4,\n    fp16=True,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    tokenizer=tokenizer\n)\n\ntrainer.train()\n\n# Save LoRA adapters (only ~20MB)\nmodel.save_pretrained(\".\u002Fqlora-adapters\")\n```\n\n### Workflow 3: 8-bit optimizer for memory-efficient training\n\nUse 8-bit Adam\u002FAdamW to reduce optimizer memory by 75%.\n\n```\n8-bit Optimizer Setup:\n- [ ] Step 1: Replace standard optimizer\n- [ ] Step 2: Configure training\n- [ ] Step 3: Monitor memory savings\n```\n\n**Step 1: Replace standard optimizer**\n\n```python\nimport bitsandbytes as bnb\nfrom transformers import Trainer, TrainingArguments\n\n# Instead of torch.optim.AdamW\nmodel = AutoModelForCausalLM.from_pretrained(\"model-name\")\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Foutput\",\n    per_device_train_batch_size=8,\n    optim=\"paged_adamw_8bit\",  # 8-bit optimizer\n    learning_rate=5e-5\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset\n)\n\ntrainer.train()\n```\n\n**Manual optimizer usage**:\n```python\nimport bitsandbytes as bnb\n\noptimizer = bnb.optim.AdamW8bit(\n    model.parameters(),\n    lr=1e-4,\n    betas=(0.9, 0.999),\n    eps=1e-8\n)\n\n# Training loop\nfor batch in dataloader:\n    loss = model(**batch).loss\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n```\n\n**Step 2: Configure training**\n\nCompare memory:\n```\nStandard AdamW optimizer memory = model_params × 8 bytes (states)\n8-bit AdamW memory = model_params × 2 bytes\nSavings = 75% optimizer memory\n\nExample (Llama 2 7B):\nStandard: 7B × 8 = 56 GB\n8-bit: 7B × 2 = 14 GB\nSavings: 42 GB\n```\n\n**Step 3: Monitor memory savings**\n\n```python\nimport torch\n\nbefore = torch.cuda.memory_allocated()\n\n# Training step\noptimizer.step()\n\nafter = torch.cuda.memory_allocated()\nprint(f\"Memory used: {(after-before)\u002F1e9:.2f}GB\")\n```\n\n## When to use vs alternatives\n\n**Use bitsandbytes when:**\n- GPU memory limited (need to fit larger model)\n- Training with QLoRA (fine-tune 70B on single GPU)\n- Inference only (50-75% memory reduction)\n- Using HuggingFace Transformers\n- Acceptable 0-2% accuracy degradation\n\n**Use alternatives instead:**\n- **GPTQ\u002FAWQ**: Production serving (faster inference than bitsandbytes)\n- **GGUF**: CPU inference (llama.cpp)\n- **FP8**: H100 GPUs (hardware FP8 faster)\n- **Full precision**: Accuracy critical, memory not constrained\n\n## Common issues\n\n**Issue: CUDA error during loading**\n\nInstall matching CUDA version:\n```bash\n# Check CUDA version\nnvcc --version\n\n# Install matching bitsandbytes\npip install bitsandbytes --no-cache-dir\n```\n\n**Issue: Model loading slow**\n\nUse CPU offload for large models:\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"model-name\",\n    quantization_config=config,\n    device_map=\"auto\",\n    max_memory={0: \"20GB\", \"cpu\": \"30GB\"}  # Offload to CPU\n)\n```\n\n**Issue: Lower accuracy than expected**\n\nTry 8-bit instead of 4-bit:\n```python\nconfig = BitsAndBytesConfig(load_in_8bit=True)\n# 8-bit has \u003C0.5% accuracy loss vs 1-2% for 4-bit\n```\n\nOr use NF4 with double quantization:\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",  # Better than fp4\n    bnb_4bit_use_double_quant=True  # Extra accuracy\n)\n```\n\n**Issue: OOM even with 4-bit**\n\nEnable CPU offload:\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"model-name\",\n    quantization_config=config,\n    device_map=\"auto\",\n    offload_folder=\"offload\",  # Disk offload\n    offload_state_dict=True\n)\n```\n\n## Advanced topics\n\n**QLoRA training guide**: See [references\u002Fqlora-training.md](references\u002Fqlora-training.md) for complete fine-tuning workflows, hyperparameter tuning, and multi-GPU training.\n\n**Quantization formats**: See [references\u002Fquantization-formats.md](references\u002Fquantization-formats.md) for INT8, NF4, FP4 comparison, double quantization, and custom quantization configs.\n\n**Memory optimization**: See [references\u002Fmemory-optimization.md](references\u002Fmemory-optimization.md) for CPU offloading strategies, gradient checkpointing, and memory profiling.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA with compute capability 7.0+ (Turing, Ampere, Hopper)\n- **VRAM**: Depends on model and quantization\n  - 4-bit Llama 2 7B: 4GB\n  - 4-bit Llama 2 13B: 8GB\n  - 4-bit Llama 2 70B: 24GB\n- **CUDA**: 11.1+ (12.0+ recommended)\n- **PyTorch**: 2.0+\n\n**Supported platforms**: NVIDIA GPUs (primary), AMD ROCm, Intel GPUs (experimental)\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002Fbitsandbytes-foundation\u002Fbitsandbytes\n- HuggingFace docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftransformers\u002Fquantization\u002Fbitsandbytes\n- QLoRA paper: \"QLoRA: Efficient Finetuning of Quantized LLMs\" (2023)\n- LLM.int8() paper: \"LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale\" (2022)\n\n\n\n","# bitsandbytes - LLM Quantization\n\n## Quick start\n\nbitsandbytes reduces LLM memory by 50% (8-bit) or 75% (4-bit) with \u003C1% accuracy loss.\n\n**Installation**:\n```bash\npip install bitsandbytes transformers accelerate\n```\n\n**8-bit quantization** (50% memory reduction):\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nconfig = BitsAndBytesConfig(load_in_8bit=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Memory: 14GB → 7GB\n```\n\n**4-bit quantization** (75% memory reduction):\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16\n)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Memory: 14GB → 3.5GB\n```\n\n## Common workflows\n\n### Workflow 1: Load large model in limited GPU memory\n\nCopy this checklist:\n\n```\nQuantization Loading:\n- [ ] Step 1: Calculate memory requirements\n- [ ] Step 2: Choose quantization level (4-bit or 8-bit)\n- [ ] Step 3: Configure quantization\n- [ ] Step 4: Load and verify model\n```\n\n**Step 1: Calculate memory requirements**\n\nEstimate model memory:\n```\nFP16 memory (GB) = Parameters × 2 bytes \u002F 1e9\nINT8 memory (GB) = Parameters × 1 byte \u002F 1e9\nINT4 memory (GB) = Parameters × 0.5 bytes \u002F 1e9\n\nExample (Llama 2 7B):\nFP16: 7B × 2 \u002F 1e9 = 14 GB\nINT8: 7B × 1 \u002F 1e9 = 7 GB\nINT4: 7B × 0.5 \u002F 1e9 = 3.5 GB\n```\n\n**Step 2: Choose quantization level**\n\n| GPU VRAM | Model Size | Recommended |\n|----------|------------|-------------|\n| 8 GB | 3B | 4-bit |\n| 12 GB | 7B | 4-bit |\n| 16 GB | 7B | 8-bit or 4-bit |\n| 24 GB | 13B | 8-bit or 70B 4-bit |\n| 40+ GB | 70B | 8-bit |\n\n**Step 3: Configure quantization**\n\nFor 8-bit (better accuracy):\n```python\nfrom transformers import BitsAndBytesConfig\nimport torch\n\nconfig = BitsAndBytesConfig(\n    load_in_8bit=True,\n    llm_int8_threshold=6.0,  # Outlier threshold\n    llm_int8_has_fp16_weight=False\n)\n```\n\nFor 4-bit (maximum memory savings):\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16,  # Compute in FP16\n    bnb_4bit_quant_type=\"nf4\",  # NormalFloat4 (recommended)\n    bnb_4bit_use_double_quant=True  # Nested quantization\n)\n```\n\n**Step 4: Load and verify model**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-13b-hf\",\n    quantization_config=config,\n    device_map=\"auto\",  # Automatic device placement\n    torch_dtype=torch.float16\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-13b-hf\")\n\n# Test inference\ninputs = tokenizer(\"Hello, how are you?\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_length=50)\nprint(tokenizer.decode(outputs[0]))\n\n# Check memory\nimport torch\nprint(f\"Memory allocated: {torch.cuda.memory_allocated()\u002F1e9:.2f}GB\")\n```\n\n### Workflow 2: Fine-tune with QLoRA (4-bit training)\n\nQLoRA enables fine-tuning large models on consumer GPUs.\n\nCopy this checklist:\n\n```\nQLoRA Fine-tuning:\n- [ ] Step 1: Install dependencies\n- [ ] Step 2: Configure 4-bit base model\n- [ ] Step 3: Add LoRA adapters\n- [ ] Step 4: Train with standard Trainer\n```\n\n**Step 1: Install dependencies**\n\n```bash\npip install bitsandbytes transformers peft accelerate datasets\n```\n\n**Step 2: Configure 4-bit base model**\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nimport torch\n\nbnb_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_use_double_quant=True\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=bnb_config,\n    device_map=\"auto\"\n)\n```\n\n**Step 3: Add LoRA adapters**\n\n```python\nfrom peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training\n\n# Prepare model for training\nmodel = prepare_model_for_kbit_training(model)\n\n# Configure LoRA\nlora_config = LoraConfig(\n    r=16,  # LoRA rank\n    lora_alpha=32,  # LoRA alpha\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\n# Add LoRA adapters\nmodel = get_peft_model(model, lora_config)\nmodel.print_trainable_parameters()\n# Output: trainable params: 4.2M || all params: 6.7B || trainable%: 0.06%\n```\n\n**Step 4: Train with standard Trainer**\n\n```python\nfrom transformers import Trainer, TrainingArguments\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fqlora-output\",\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    num_train_epochs=3,\n    learning_rate=2e-4,\n    fp16=True,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    tokenizer=tokenizer\n)\n\ntrainer.train()\n\n# Save LoRA adapters (only ~20MB)\nmodel.save_pretrained(\".\u002Fqlora-adapters\")\n```\n\n### Workflow 3: 8-bit optimizer for memory-efficient training\n\nUse 8-bit Adam\u002FAdamW to reduce optimizer memory by 75%.\n\n```\n8-bit Optimizer Setup:\n- [ ] Step 1: Replace standard optimizer\n- [ ] Step 2: Configure training\n- [ ] Step 3: Monitor memory savings\n```\n\n**Step 1: Replace standard optimizer**\n\n```python\nimport bitsandbytes as bnb\nfrom transformers import Trainer, TrainingArguments\n\n# Instead of torch.optim.AdamW\nmodel = AutoModelForCausalLM.from_pretrained(\"model-name\")\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Foutput\",\n    per_device_train_batch_size=8,\n    optim=\"paged_adamw_8bit\",  # 8-bit optimizer\n    learning_rate=5e-5\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset\n)\n\ntrainer.train()\n```\n\n**Manual optimizer usage**:\n```python\nimport bitsandbytes as bnb\n\noptimizer = bnb.optim.AdamW8bit(\n    model.parameters(),\n    lr=1e-4,\n    betas=(0.9, 0.999),\n    eps=1e-8\n)\n\n# Training loop\nfor batch in dataloader:\n    loss = model(**batch).loss\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n```\n\n**Step 2: Configure training**\n\nCompare memory:\n```\nStandard AdamW optimizer memory = model_params × 8 bytes (states)\n8-bit AdamW memory = model_params × 2 bytes\nSavings = 75% optimizer memory\n\nExample (Llama 2 7B):\nStandard: 7B × 8 = 56 GB\n8-bit: 7B × 2 = 14 GB\nSavings: 42 GB\n```\n\n**Step 3: Monitor memory savings**\n\n```python\nimport torch\n\nbefore = torch.cuda.memory_allocated()\n\n# Training step\noptimizer.step()\n\nafter = torch.cuda.memory_allocated()\nprint(f\"Memory used: {(after-before)\u002F1e9:.2f}GB\")\n```\n\n## When to use vs alternatives\n\n**Use bitsandbytes when:**\n- GPU memory limited (need to fit larger model)\n- Training with QLoRA (fine-tune 70B on single GPU)\n- Inference only (50-75% memory reduction)\n- Using HuggingFace Transformers\n- Acceptable 0-2% accuracy degradation\n\n**Use alternatives instead:**\n- **GPTQ\u002FAWQ**: Production serving (faster inference than bitsandbytes)\n- **GGUF**: CPU inference (llama.cpp)\n- **FP8**: H100 GPUs (hardware FP8 faster)\n- **Full precision**: Accuracy critical, memory not constrained\n\n## Common issues\n\n**Issue: CUDA error during loading**\n\nInstall matching CUDA version:\n```bash\n# Check CUDA version\nnvcc --version\n\n# Install matching bitsandbytes\npip install bitsandbytes --no-cache-dir\n```\n\n**Issue: Model loading slow**\n\nUse CPU offload for large models:\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"model-name\",\n    quantization_config=config,\n    device_map=\"auto\",\n    max_memory={0: \"20GB\", \"cpu\": \"30GB\"}  # Offload to CPU\n)\n```\n\n**Issue: Lower accuracy than expected**\n\nTry 8-bit instead of 4-bit:\n```python\nconfig = BitsAndBytesConfig(load_in_8bit=True)\n# 8-bit has \u003C0.5% accuracy loss vs 1-2% for 4-bit\n```\n\nOr use NF4 with double quantization:\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",  # Better than fp4\n    bnb_4bit_use_double_quant=True  # Extra accuracy\n)\n```\n\n**Issue: OOM even with 4-bit**\n\nEnable CPU offload:\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"model-name\",\n    quantization_config=config,\n    device_map=\"auto\",\n    offload_folder=\"offload\",  # Disk offload\n    offload_state_dict=True\n)\n```\n\n## Advanced topics\n\n**QLoRA training guide**: See [references\u002Fqlora-training.md](references\u002Fqlora-training.md) for complete fine-tuning workflows, hyperparameter tuning, and multi-GPU training.\n\n**Quantization formats**: See [references\u002Fquantization-formats.md](references\u002Fquantization-formats.md) for INT8, NF4, FP4 comparison, double quantization, and custom quantization configs.\n\n**Memory optimization**: See [references\u002Fmemory-optimization.md](references\u002Fmemory-optimization.md) for CPU offloading strategies, gradient checkpointing, and memory profiling.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA with compute capability 7.0+ (Turing, Ampere, Hopper)\n- **VRAM**: Depends on model and quantization\n  - 4-bit Llama 2 7B: 4GB\n  - 4-bit Llama 2 13B: 8GB\n  - 4-bit Llama 2 70B: 24GB\n- **CUDA**: 11.1+ (12.0+ recommended)\n- **PyTorch**: 2.0+\n\n**Supported platforms**: NVIDIA GPUs (primary), AMD ROCm, Intel GPUs (experimental)\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002Fbitsandbytes-foundation\u002Fbitsandbytes\n- HuggingFace docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftransformers\u002Fquantization\u002Fbitsandbytes\n- QLoRA paper: \"QLoRA: Efficient Finetuning of Quantized LLMs\" (2023)\n- LLM.int8() paper: \"LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale\" (2022)\n\n\n\n",{"name":5,"tags":24,"author":36,"license":15,"version":37,"description":7,"dependencies":38},[25,26,27,28,29,30,31,32,33,34,35],"Optimization","Bitsandbytes","Quantization","8-Bit","4-Bit","Memory Optimization","QLoRA","NF4","INT8","HuggingFace","Efficient Inference","Orchestra Research","1.0.0",[39,40,41,42],"bitsandbytes","transformers","accelerate","torch","add-x-twitter-scraper-skill","7bc347b2f7dc2dde2761a965585925e0d0171748","cli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes",0,"https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template","https:\u002F\u002Fgithub.com\u002FCarchaloneo22","Ingeniero Informático | Ingeniero en Desarrollo de Software | Consultor TIC en Gestión Publica | Docente Análisis y Desarrollo de Software. ","2026-09-12 11:10:56",{"source":8,"skillName":5,"commands":52,"prompt":62,"zipAvailable":63},[53,56,59],{"tool":54,"command":55},"npx","npx skills add Carchaloneo22\u002FClaude-Code-template --skill quantizing-models-bitsandbytes",{"tool":57,"command":58},"bunx","bunx skills add Carchaloneo22\u002FClaude-Code-template --skill quantizing-models-bitsandbytes",{"tool":60,"command":61},"pnpm","pnpm dlx skills add Carchaloneo22\u002FClaude-Code-template --skill quantizing-models-bitsandbytes","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes\u002FSKILL.md",true,[65,71,75],{"path":66,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":69,"sourceHtmlUrl":70},"references\u002Fmemory-optimization.md","REFERENCE","参考文档",12642,"https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes\u002Freferences\u002Fmemory-optimization.md",{"path":72,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":73,"sourceHtmlUrl":74},"references\u002Fqlora-training.md",12015,"https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes\u002Freferences\u002Fqlora-training.md",{"path":76,"resourceType":67,"resourceTypeDesc":68,"sizeBytes":77,"sourceHtmlUrl":78},"references\u002Fquantization-formats.md",10258,"https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes\u002Freferences\u002Fquantization-formats.md",[80,87,96,103,110,117],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":19},"01M7CPQFVGKX7H3Q5Y9CN0XAZN","zapier-make-patterns","No-code automation democratizes workflow building. Zapier and Make (formerly Integromat) let non-developers automate business processes without writing code. But no-code doesn't mean no-complexity - these platforms have their own patterns, pitfalls, and breaking points.  This skill covers when to use which platform, how to build reliable automations, and when to graduate to code-based solutions. Key insight: Zapier optimizes for simplicity and integrations (7000+ apps), Make optimizes for power","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fzapier-make-patterns\u002FSKILL.md",[],[],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":91,"licenseFileUrl":92,"sourceHtmlUrl":93,"categories":94,"keywords":95,"publishedAt":19},"01M7CPQDR2HMKAYHAAY99EA4B2","yeet","Use only when the user explicitly asks to stage, commit, push, and open a GitHub pull request in one flow using the GitHub CLI (`gh`).","Apache-2.0","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fyeet\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fyeet\u002FSKILL.md",[],[],{"skillNo":97,"name":98,"nameZh":6,"description":99,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":100,"categories":101,"keywords":102,"publishedAt":19},"01M7CPQ7K5MAQSXEB027HZFM2G","workflow-automation","Workflow automation is the infrastructure that makes AI agents reliable. Without durable execution, a network hiccup during a 10-step payment flow means lost money and angry customers. With it, workflows resume exactly where they left off.  This skill covers the platforms (n8n, Temporal, Inngest) and patterns (sequential, parallel, orchestrator-worker) that turn brittle scripts into production-grade automation.  Key insight: The platforms make different tradeoffs. n8n optimizes for accessibility","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fworkflow-automation\u002FSKILL.md",[],[],{"skillNo":104,"name":105,"nameZh":6,"description":106,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":107,"categories":108,"keywords":109,"publishedAt":19},"01M7CPQ34F18JHMEM8NHQS1N7K","trigger-dev","Trigger.dev expert for background jobs, AI workflows, and reliable async execution with excellent developer experience and TypeScript-first design. Use when: trigger.dev, trigger dev, background task, ai background job, long running task.","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Ftrigger-dev\u002FSKILL.md",[],[],{"skillNo":111,"name":112,"nameZh":6,"description":113,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":114,"categories":115,"keywords":116,"publishedAt":19},"01M7CPPZSXXDB1SAWT5N63ACZT","planning-with-files","Transforms workflow to use Manus-style persistent markdown files for planning, progress tracking, and knowledge storage. Use when starting complex tasks, multi-step projects, research tasks, or when the user mentions planning, organizing work, tracking progress, or wants structured output.","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fplanning-with-files\u002FSKILL.md",[],[],{"skillNo":118,"name":119,"nameZh":6,"description":120,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":121,"categories":122,"keywords":123,"publishedAt":19},"01M7CPPTQSBVWSK0R10FJNMP0F","n8n-workflow-patterns","Proven workflow architectural patterns from real n8n workflows. Use when building new workflows, designing workflow structure, choosing workflow patterns, planning workflow architecture, or asking about webhook processing, HTTP API integration, database operations, AI agent workflows, or scheduled tasks.","https:\u002F\u002Fgithub.com\u002FCarchaloneo22\u002FClaude-Code-template\u002Fblob\u002Fadd-x-twitter-scraper-skill\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fn8n\u002Fn8n-workflow-patterns\u002FSKILL.md",[],[]]