[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BHDHW90CC79PP81DPZ1PJ6":3,"skill-same:01M7BHDHW90CC79PP81DPZ1PJ6":80},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":43,"commitSha":44,"skillDirectory":45,"forksCount":46,"repositoryHtmlUrl":47,"repositoryHomepageUrl":48,"ownerHtmlUrl":49,"ownerBlogUrl":6,"ownerDescription":50,"updatedAt":51,"install":52,"resources":65},"01M7BHDHW90CC79PP81DPZ1PJ6","quantizing-models-bitsandbytes",null,"Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss. Use when GPU memory is limited, need to fit larger models, or want faster inference. Supports INT8, NF4, FP4 formats, QLoRA training, and 8-bit optimizers. Works with HuggingFace Transformers.","YYC-Cube\u002FYYC3-Knowledge-Base","YYC-Cube","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F235006543?v=4","01M79J2PH4SWD2X0825X6PN748","yyc3","01M79J2PHVK3F6ZBYT8K2SCZCW",1,"MIT","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-AI-应用工具\u002FYYC3-AI-应用工具-claude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: quantizing-models-bitsandbytes\ndescription: Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss. Use when GPU memory is limited, need to fit larger models, or want faster inference. Supports INT8, NF4, FP4 formats, QLoRA training, and 8-bit optimizers. Works with HuggingFace Transformers.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Optimization, Bitsandbytes, Quantization, 8-Bit, 4-Bit, Memory Optimization, QLoRA, NF4, INT8, HuggingFace, Efficient Inference]\ndependencies: [bitsandbytes, transformers, accelerate, torch]\n---\n\n# bitsandbytes - LLM Quantization\n\n## Quick start\n\nbitsandbytes reduces LLM memory by 50% (8-bit) or 75% (4-bit) with \u003C1% accuracy loss.\n\n**Installation**:\n```bash\npip install bitsandbytes transformers accelerate\n```\n\n**8-bit quantization** (50% memory reduction):\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nconfig = BitsAndBytesConfig(load_in_8bit=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Memory: 14GB → 7GB\n```\n\n**4-bit quantization** (75% memory reduction):\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16\n)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Memory: 14GB → 3.5GB\n```\n\n## Common workflows\n\n### Workflow 1: Load large model in limited GPU memory\n\nCopy this checklist:\n\n```\nQuantization Loading:\n- [ ] Step 1: Calculate memory requirements\n- [ ] Step 2: Choose quantization level (4-bit or 8-bit)\n- [ ] Step 3: Configure quantization\n- [ ] Step 4: Load and verify model\n```\n\n**Step 1: Calculate memory requirements**\n\nEstimate model memory:\n```\nFP16 memory (GB) = Parameters × 2 bytes \u002F 1e9\nINT8 memory (GB) = Parameters × 1 byte \u002F 1e9\nINT4 memory (GB) = Parameters × 0.5 bytes \u002F 1e9\n\nExample (Llama 2 7B):\nFP16: 7B × 2 \u002F 1e9 = 14 GB\nINT8: 7B × 1 \u002F 1e9 = 7 GB\nINT4: 7B × 0.5 \u002F 1e9 = 3.5 GB\n```\n\n**Step 2: Choose quantization level**\n\n| GPU VRAM | Model Size | Recommended |\n|----------|------------|-------------|\n| 8 GB | 3B | 4-bit |\n| 12 GB | 7B | 4-bit |\n| 16 GB | 7B | 8-bit or 4-bit |\n| 24 GB | 13B | 8-bit or 70B 4-bit |\n| 40+ GB | 70B | 8-bit |\n\n**Step 3: Configure quantization**\n\nFor 8-bit (better accuracy):\n```python\nfrom transformers import BitsAndBytesConfig\nimport torch\n\nconfig = BitsAndBytesConfig(\n    load_in_8bit=True,\n    llm_int8_threshold=6.0,  # Outlier threshold\n    llm_int8_has_fp16_weight=False\n)\n```\n\nFor 4-bit (maximum memory savings):\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16,  # Compute in FP16\n    bnb_4bit_quant_type=\"nf4\",  # NormalFloat4 (recommended)\n    bnb_4bit_use_double_quant=True  # Nested quantization\n)\n```\n\n**Step 4: Load and verify model**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-13b-hf\",\n    quantization_config=config,\n    device_map=\"auto\",  # Automatic device placement\n    torch_dtype=torch.float16\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-13b-hf\")\n\n# Test inference\ninputs = tokenizer(\"Hello, how are you?\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_length=50)\nprint(tokenizer.decode(outputs[0]))\n\n# Check memory\nimport torch\nprint(f\"Memory allocated: {torch.cuda.memory_allocated()\u002F1e9:.2f}GB\")\n```\n\n### Workflow 2: Fine-tune with QLoRA (4-bit training)\n\nQLoRA enables fine-tuning large models on consumer GPUs.\n\nCopy this checklist:\n\n```\nQLoRA Fine-tuning:\n- [ ] Step 1: Install dependencies\n- [ ] Step 2: Configure 4-bit base model\n- [ ] Step 3: Add LoRA adapters\n- [ ] Step 4: Train with standard Trainer\n```\n\n**Step 1: Install dependencies**\n\n```bash\npip install bitsandbytes transformers peft accelerate datasets\n```\n\n**Step 2: Configure 4-bit base model**\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nimport torch\n\nbnb_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_use_double_quant=True\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=bnb_config,\n    device_map=\"auto\"\n)\n```\n\n**Step 3: Add LoRA adapters**\n\n```python\nfrom peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training\n\n# Prepare model for training\nmodel = prepare_model_for_kbit_training(model)\n\n# Configure LoRA\nlora_config = LoraConfig(\n    r=16,  # LoRA rank\n    lora_alpha=32,  # LoRA alpha\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\n# Add LoRA adapters\nmodel = get_peft_model(model, lora_config)\nmodel.print_trainable_parameters()\n# Output: trainable params: 4.2M || all params: 6.7B || trainable%: 0.06%\n```\n\n**Step 4: Train with standard Trainer**\n\n```python\nfrom transformers import Trainer, TrainingArguments\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fqlora-output\",\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    num_train_epochs=3,\n    learning_rate=2e-4,\n    fp16=True,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    tokenizer=tokenizer\n)\n\ntrainer.train()\n\n# Save LoRA adapters (only ~20MB)\nmodel.save_pretrained(\".\u002Fqlora-adapters\")\n```\n\n### Workflow 3: 8-bit optimizer for memory-efficient training\n\nUse 8-bit Adam\u002FAdamW to reduce optimizer memory by 75%.\n\n```\n8-bit Optimizer Setup:\n- [ ] Step 1: Replace standard optimizer\n- [ ] Step 2: Configure training\n- [ ] Step 3: Monitor memory savings\n```\n\n**Step 1: Replace standard optimizer**\n\n```python\nimport bitsandbytes as bnb\nfrom transformers import Trainer, TrainingArguments\n\n# Instead of torch.optim.AdamW\nmodel = AutoModelForCausalLM.from_pretrained(\"model-name\")\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Foutput\",\n    per_device_train_batch_size=8,\n    optim=\"paged_adamw_8bit\",  # 8-bit optimizer\n    learning_rate=5e-5\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset\n)\n\ntrainer.train()\n```\n\n**Manual optimizer usage**:\n```python\nimport bitsandbytes as bnb\n\noptimizer = bnb.optim.AdamW8bit(\n    model.parameters(),\n    lr=1e-4,\n    betas=(0.9, 0.999),\n    eps=1e-8\n)\n\n# Training loop\nfor batch in dataloader:\n    loss = model(**batch).loss\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n```\n\n**Step 2: Configure training**\n\nCompare memory:\n```\nStandard AdamW optimizer memory = model_params × 8 bytes (states)\n8-bit AdamW memory = model_params × 2 bytes\nSavings = 75% optimizer memory\n\nExample (Llama 2 7B):\nStandard: 7B × 8 = 56 GB\n8-bit: 7B × 2 = 14 GB\nSavings: 42 GB\n```\n\n**Step 3: Monitor memory savings**\n\n```python\nimport torch\n\nbefore = torch.cuda.memory_allocated()\n\n# Training step\noptimizer.step()\n\nafter = torch.cuda.memory_allocated()\nprint(f\"Memory used: {(after-before)\u002F1e9:.2f}GB\")\n```\n\n## When to use vs alternatives\n\n**Use bitsandbytes when:**\n- GPU memory limited (need to fit larger model)\n- Training with QLoRA (fine-tune 70B on single GPU)\n- Inference only (50-75% memory reduction)\n- Using HuggingFace Transformers\n- Acceptable 0-2% accuracy degradation\n\n**Use alternatives instead:**\n- **GPTQ\u002FAWQ**: Production serving (faster inference than bitsandbytes)\n- **GGUF**: CPU inference (llama.cpp)\n- **FP8**: H100 GPUs (hardware FP8 faster)\n- **Full precision**: Accuracy critical, memory not constrained\n\n## Common issues\n\n**Issue: CUDA error during loading**\n\nInstall matching CUDA version:\n```bash\n# Check CUDA version\nnvcc --version\n\n# Install matching bitsandbytes\npip install bitsandbytes --no-cache-dir\n```\n\n**Issue: Model loading slow**\n\nUse CPU offload for large models:\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"model-name\",\n    quantization_config=config,\n    device_map=\"auto\",\n    max_memory={0: \"20GB\", \"cpu\": \"30GB\"}  # Offload to CPU\n)\n```\n\n**Issue: Lower accuracy than expected**\n\nTry 8-bit instead of 4-bit:\n```python\nconfig = BitsAndBytesConfig(load_in_8bit=True)\n# 8-bit has \u003C0.5% accuracy loss vs 1-2% for 4-bit\n```\n\nOr use NF4 with double quantization:\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",  # Better than fp4\n    bnb_4bit_use_double_quant=True  # Extra accuracy\n)\n```\n\n**Issue: OOM even with 4-bit**\n\nEnable CPU offload:\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"model-name\",\n    quantization_config=config,\n    device_map=\"auto\",\n    offload_folder=\"offload\",  # Disk offload\n    offload_state_dict=True\n)\n```\n\n## Advanced topics\n\n**QLoRA training guide**: See [references\u002Fqlora-training.md](references\u002Fqlora-training.md) for complete fine-tuning workflows, hyperparameter tuning, and multi-GPU training.\n\n**Quantization formats**: See [references\u002Fquantization-formats.md](references\u002Fquantization-formats.md) for INT8, NF4, FP4 comparison, double quantization, and custom quantization configs.\n\n**Memory optimization**: See [references\u002Fmemory-optimization.md](references\u002Fmemory-optimization.md) for CPU offloading strategies, gradient checkpointing, and memory profiling.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA with compute capability 7.0+ (Turing, Ampere, Hopper)\n- **VRAM**: Depends on model and quantization\n  - 4-bit Llama 2 7B: 4GB\n  - 4-bit Llama 2 13B: 8GB\n  - 4-bit Llama 2 70B: 24GB\n- **CUDA**: 11.1+ (12.0+ recommended)\n- **PyTorch**: 2.0+\n\n**Supported platforms**: NVIDIA GPUs (primary), AMD ROCm, Intel GPUs (experimental)\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002Fbitsandbytes-foundation\u002Fbitsandbytes\n- HuggingFace docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftransformers\u002Fquantization\u002Fbitsandbytes\n- QLoRA paper: \"QLoRA: Efficient Finetuning of Quantized LLMs\" (2023)\n- LLM.int8() paper: \"LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale\" (2022)\n\n\n\n","# bitsandbytes - LLM Quantization\n\n## Quick start\n\nbitsandbytes reduces LLM memory by 50% (8-bit) or 75% (4-bit) with \u003C1% accuracy loss.\n\n**Installation**:\n```bash\npip install bitsandbytes transformers accelerate\n```\n\n**8-bit quantization** (50% memory reduction):\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\n\nconfig = BitsAndBytesConfig(load_in_8bit=True)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Memory: 14GB → 7GB\n```\n\n**4-bit quantization** (75% memory reduction):\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16\n)\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=config,\n    device_map=\"auto\"\n)\n\n# Memory: 14GB → 3.5GB\n```\n\n## Common workflows\n\n### Workflow 1: Load large model in limited GPU memory\n\nCopy this checklist:\n\n```\nQuantization Loading:\n- [ ] Step 1: Calculate memory requirements\n- [ ] Step 2: Choose quantization level (4-bit or 8-bit)\n- [ ] Step 3: Configure quantization\n- [ ] Step 4: Load and verify model\n```\n\n**Step 1: Calculate memory requirements**\n\nEstimate model memory:\n```\nFP16 memory (GB) = Parameters × 2 bytes \u002F 1e9\nINT8 memory (GB) = Parameters × 1 byte \u002F 1e9\nINT4 memory (GB) = Parameters × 0.5 bytes \u002F 1e9\n\nExample (Llama 2 7B):\nFP16: 7B × 2 \u002F 1e9 = 14 GB\nINT8: 7B × 1 \u002F 1e9 = 7 GB\nINT4: 7B × 0.5 \u002F 1e9 = 3.5 GB\n```\n\n**Step 2: Choose quantization level**\n\n| GPU VRAM | Model Size | Recommended |\n|----------|------------|-------------|\n| 8 GB | 3B | 4-bit |\n| 12 GB | 7B | 4-bit |\n| 16 GB | 7B | 8-bit or 4-bit |\n| 24 GB | 13B | 8-bit or 70B 4-bit |\n| 40+ GB | 70B | 8-bit |\n\n**Step 3: Configure quantization**\n\nFor 8-bit (better accuracy):\n```python\nfrom transformers import BitsAndBytesConfig\nimport torch\n\nconfig = BitsAndBytesConfig(\n    load_in_8bit=True,\n    llm_int8_threshold=6.0,  # Outlier threshold\n    llm_int8_has_fp16_weight=False\n)\n```\n\nFor 4-bit (maximum memory savings):\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16,  # Compute in FP16\n    bnb_4bit_quant_type=\"nf4\",  # NormalFloat4 (recommended)\n    bnb_4bit_use_double_quant=True  # Nested quantization\n)\n```\n\n**Step 4: Load and verify model**\n\n```python\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-13b-hf\",\n    quantization_config=config,\n    device_map=\"auto\",  # Automatic device placement\n    torch_dtype=torch.float16\n)\n\ntokenizer = AutoTokenizer.from_pretrained(\"meta-llama\u002FLlama-2-13b-hf\")\n\n# Test inference\ninputs = tokenizer(\"Hello, how are you?\", return_tensors=\"pt\").to(\"cuda\")\noutputs = model.generate(**inputs, max_length=50)\nprint(tokenizer.decode(outputs[0]))\n\n# Check memory\nimport torch\nprint(f\"Memory allocated: {torch.cuda.memory_allocated()\u002F1e9:.2f}GB\")\n```\n\n### Workflow 2: Fine-tune with QLoRA (4-bit training)\n\nQLoRA enables fine-tuning large models on consumer GPUs.\n\nCopy this checklist:\n\n```\nQLoRA Fine-tuning:\n- [ ] Step 1: Install dependencies\n- [ ] Step 2: Configure 4-bit base model\n- [ ] Step 3: Add LoRA adapters\n- [ ] Step 4: Train with standard Trainer\n```\n\n**Step 1: Install dependencies**\n\n```bash\npip install bitsandbytes transformers peft accelerate datasets\n```\n\n**Step 2: Configure 4-bit base model**\n\n```python\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nimport torch\n\nbnb_config = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_compute_dtype=torch.float16,\n    bnb_4bit_quant_type=\"nf4\",\n    bnb_4bit_use_double_quant=True\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-2-7b-hf\",\n    quantization_config=bnb_config,\n    device_map=\"auto\"\n)\n```\n\n**Step 3: Add LoRA adapters**\n\n```python\nfrom peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training\n\n# Prepare model for training\nmodel = prepare_model_for_kbit_training(model)\n\n# Configure LoRA\nlora_config = LoraConfig(\n    r=16,  # LoRA rank\n    lora_alpha=32,  # LoRA alpha\n    target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\"],\n    lora_dropout=0.05,\n    bias=\"none\",\n    task_type=\"CAUSAL_LM\"\n)\n\n# Add LoRA adapters\nmodel = get_peft_model(model, lora_config)\nmodel.print_trainable_parameters()\n# Output: trainable params: 4.2M || all params: 6.7B || trainable%: 0.06%\n```\n\n**Step 4: Train with standard Trainer**\n\n```python\nfrom transformers import Trainer, TrainingArguments\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Fqlora-output\",\n    per_device_train_batch_size=4,\n    gradient_accumulation_steps=4,\n    num_train_epochs=3,\n    learning_rate=2e-4,\n    fp16=True,\n    logging_steps=10,\n    save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset,\n    tokenizer=tokenizer\n)\n\ntrainer.train()\n\n# Save LoRA adapters (only ~20MB)\nmodel.save_pretrained(\".\u002Fqlora-adapters\")\n```\n\n### Workflow 3: 8-bit optimizer for memory-efficient training\n\nUse 8-bit Adam\u002FAdamW to reduce optimizer memory by 75%.\n\n```\n8-bit Optimizer Setup:\n- [ ] Step 1: Replace standard optimizer\n- [ ] Step 2: Configure training\n- [ ] Step 3: Monitor memory savings\n```\n\n**Step 1: Replace standard optimizer**\n\n```python\nimport bitsandbytes as bnb\nfrom transformers import Trainer, TrainingArguments\n\n# Instead of torch.optim.AdamW\nmodel = AutoModelForCausalLM.from_pretrained(\"model-name\")\n\ntraining_args = TrainingArguments(\n    output_dir=\".\u002Foutput\",\n    per_device_train_batch_size=8,\n    optim=\"paged_adamw_8bit\",  # 8-bit optimizer\n    learning_rate=5e-5\n)\n\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=train_dataset\n)\n\ntrainer.train()\n```\n\n**Manual optimizer usage**:\n```python\nimport bitsandbytes as bnb\n\noptimizer = bnb.optim.AdamW8bit(\n    model.parameters(),\n    lr=1e-4,\n    betas=(0.9, 0.999),\n    eps=1e-8\n)\n\n# Training loop\nfor batch in dataloader:\n    loss = model(**batch).loss\n    loss.backward()\n    optimizer.step()\n    optimizer.zero_grad()\n```\n\n**Step 2: Configure training**\n\nCompare memory:\n```\nStandard AdamW optimizer memory = model_params × 8 bytes (states)\n8-bit AdamW memory = model_params × 2 bytes\nSavings = 75% optimizer memory\n\nExample (Llama 2 7B):\nStandard: 7B × 8 = 56 GB\n8-bit: 7B × 2 = 14 GB\nSavings: 42 GB\n```\n\n**Step 3: Monitor memory savings**\n\n```python\nimport torch\n\nbefore = torch.cuda.memory_allocated()\n\n# Training step\noptimizer.step()\n\nafter = torch.cuda.memory_allocated()\nprint(f\"Memory used: {(after-before)\u002F1e9:.2f}GB\")\n```\n\n## When to use vs alternatives\n\n**Use bitsandbytes when:**\n- GPU memory limited (need to fit larger model)\n- Training with QLoRA (fine-tune 70B on single GPU)\n- Inference only (50-75% memory reduction)\n- Using HuggingFace Transformers\n- Acceptable 0-2% accuracy degradation\n\n**Use alternatives instead:**\n- **GPTQ\u002FAWQ**: Production serving (faster inference than bitsandbytes)\n- **GGUF**: CPU inference (llama.cpp)\n- **FP8**: H100 GPUs (hardware FP8 faster)\n- **Full precision**: Accuracy critical, memory not constrained\n\n## Common issues\n\n**Issue: CUDA error during loading**\n\nInstall matching CUDA version:\n```bash\n# Check CUDA version\nnvcc --version\n\n# Install matching bitsandbytes\npip install bitsandbytes --no-cache-dir\n```\n\n**Issue: Model loading slow**\n\nUse CPU offload for large models:\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"model-name\",\n    quantization_config=config,\n    device_map=\"auto\",\n    max_memory={0: \"20GB\", \"cpu\": \"30GB\"}  # Offload to CPU\n)\n```\n\n**Issue: Lower accuracy than expected**\n\nTry 8-bit instead of 4-bit:\n```python\nconfig = BitsAndBytesConfig(load_in_8bit=True)\n# 8-bit has \u003C0.5% accuracy loss vs 1-2% for 4-bit\n```\n\nOr use NF4 with double quantization:\n```python\nconfig = BitsAndBytesConfig(\n    load_in_4bit=True,\n    bnb_4bit_quant_type=\"nf4\",  # Better than fp4\n    bnb_4bit_use_double_quant=True  # Extra accuracy\n)\n```\n\n**Issue: OOM even with 4-bit**\n\nEnable CPU offload:\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"model-name\",\n    quantization_config=config,\n    device_map=\"auto\",\n    offload_folder=\"offload\",  # Disk offload\n    offload_state_dict=True\n)\n```\n\n## Advanced topics\n\n**QLoRA training guide**: See [references\u002Fqlora-training.md](references\u002Fqlora-training.md) for complete fine-tuning workflows, hyperparameter tuning, and multi-GPU training.\n\n**Quantization formats**: See [references\u002Fquantization-formats.md](references\u002Fquantization-formats.md) for INT8, NF4, FP4 comparison, double quantization, and custom quantization configs.\n\n**Memory optimization**: See [references\u002Fmemory-optimization.md](references\u002Fmemory-optimization.md) for CPU offloading strategies, gradient checkpointing, and memory profiling.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA with compute capability 7.0+ (Turing, Ampere, Hopper)\n- **VRAM**: Depends on model and quantization\n  - 4-bit Llama 2 7B: 4GB\n  - 4-bit Llama 2 13B: 8GB\n  - 4-bit Llama 2 70B: 24GB\n- **CUDA**: 11.1+ (12.0+ recommended)\n- **PyTorch**: 2.0+\n\n**Supported platforms**: NVIDIA GPUs (primary), AMD ROCm, Intel GPUs (experimental)\n\n## Resources\n\n- GitHub: https:\u002F\u002Fgithub.com\u002Fbitsandbytes-foundation\u002Fbitsandbytes\n- HuggingFace docs: https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftransformers\u002Fquantization\u002Fbitsandbytes\n- QLoRA paper: \"QLoRA: Efficient Finetuning of Quantized LLMs\" (2023)\n- LLM.int8() paper: \"LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale\" (2022)\n\n\n\n",{"name":5,"tags":24,"author":36,"license":15,"version":37,"description":7,"dependencies":38},[25,26,27,28,29,30,31,32,33,34,35],"Optimization","Bitsandbytes","Quantization","8-Bit","4-Bit","Memory Optimization","QLoRA","NF4","INT8","HuggingFace","Efficient Inference","Orchestra Research","1.0.0",[39,40,41,42],"bitsandbytes","transformers","accelerate","torch","main","87afe6ce96220accf50a345dea46784c8c9f210e","YYC3-AI-应用工具\u002FYYC3-AI-应用工具-claude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes",0,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base","https:\u002F\u002Fkb.yyc3.top\u002F","https:\u002F\u002Fgithub.com\u002FYYC-Cube","万象归元于云枢 | 深栈智启新纪元","2026-11-10 10:37:32",{"source":8,"skillName":5,"commands":53,"prompt":63,"zipAvailable":64},[54,57,60],{"tool":55,"command":56},"npx","npx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill quantizing-models-bitsandbytes",{"tool":58,"command":59},"bunx","bunx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill quantizing-models-bitsandbytes",{"tool":61,"command":62},"pnpm","pnpm dlx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill quantizing-models-bitsandbytes","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-AI-应用工具\u002FYYC3-AI-应用工具-claude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes\u002FSKILL.md",true,[66,72,76],{"path":67,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":70,"sourceHtmlUrl":71},"references\u002Fmemory-optimization.md","REFERENCE","参考文档",12642,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-AI-应用工具\u002FYYC3-AI-应用工具-claude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes\u002Freferences\u002Fmemory-optimization.md",{"path":73,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Fqlora-training.md",12015,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-AI-应用工具\u002FYYC3-AI-应用工具-claude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes\u002Freferences\u002Fqlora-training.md",{"path":77,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":78,"sourceHtmlUrl":79},"references\u002Fquantization-formats.md",10258,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-AI-应用工具\u002FYYC3-AI-应用工具-claude-code-templates-main\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fai-research\u002Foptimization-bitsandbytes\u002Freferences\u002Fquantization-formats.md",[81,88,95,102,109,116],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M7CTDWHERCXPQCP0K03A86AS","vercel-sandbox","Run agent-browser + Chrome inside Vercel Sandbox microVMs for browser automation from any Vercel-deployed app. Use when the user needs browser automation in a Vercel app (Next.js, SvelteKit, Nuxt, Remix, Astro, etc.), wants to run headless Chrome without binary size limits, needs persistent browser sessions across commands, or wants ephemeral isolated browser environments. Triggers include \"Vercel Sandbox browser\", \"microVM Chrome\", \"agent-browser in sandbox\", \"browser automation on Vercel\", or any task requiring Chrome in a Vercel Sandbox.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fvercel-sandbox\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M7CTD8D2BNJY54S65M47HEJA","slack","Interact with Slack workspaces using browser automation. Use when the user needs to check unread channels, navigate Slack, send messages, extract data, find information, search conversations, or automate any Slack task. Triggers include \"check my Slack\", \"what channels have unreads\", \"send a message to\", \"search Slack for\", \"extract from Slack\", \"find who said\", or any task requiring programmatic Slack interaction.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fslack\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":19},"01M7CTCSSB8ZP3Q58C3YZDZK1W","electron","Automate Electron desktop apps (VS Code, Slack, Discord, Figma, Notion, Spotify, etc.) using agent-browser via Chrome DevTools Protocol. Use when the user needs to interact with an Electron app, automate a desktop app, connect to a running app, control a native app, or test an Electron application. Triggers include \"automate Slack app\", \"control VS Code\", \"interact with Discord app\", \"test this Electron app\", \"connect to desktop app\", or any task requiring automation of a native Electron application.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Felectron\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":19},"01M7CTCBJY76688V4DCZGWFHGQ","dogfood","Systematically explore and test a web application to find bugs, UX issues, and other problems. Use when asked to \"dogfood\", \"QA\", \"exploratory test\", \"find issues\", \"bug hunt\", \"test this app\u002Fsite\u002Fplatform\", or review the quality of a web application. Produces a structured report with full reproduction evidence -- step-by-step screenshots, repro videos, and detailed repro steps for every issue -- so findings can be handed directly to the responsible teams.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fdogfood\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":19},"01M7CTC4H0FFGVEAQR9PA4GA1A","agent-browser","Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to \"open a website\", \"fill out a form\", \"click a button\", \"take a screenshot\", \"scrape data from a page\", \"test this web app\", \"login to a site\", \"automate browser actions\", or any task requiring programmatic web interaction.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fagent-browser\u002FSKILL.md",[],[],{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":19},"01M7CTBKVFYBG78FDSABSDJTTQ","shadcn","Manages shadcn components and projects — adding, searching, fixing, debugging, styling, and composing UI, including chat interfaces. Provides project context, component docs, and usage examples. Applies when working with shadcn\u002Fui, component registries, presets, --preset codes, or any project with a components.json file. Also triggers for \"shadcn init\", \"create an app with --preset\", \"switch to --preset\", and for building a design system or applying a DESIGN.md or brand spec to shadcn\u002Fui.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-UI-界面组件\u002FYYC3-UI-界面组件-ui\u002Fskills\u002Fshadcn\u002FSKILL.md",[],[]]