llama-cpp-quantization
暂无简介
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 13
- Forks
- 2
03 SKILL.md 正文
/llama-cpp-quantization
When to Use
- Run open-source LLMs locally without GPU (CPU inference via GGUF)
- Choose quantization level based on quality/memory tradeoff
- Embed llama.cpp HTTP server for OpenAI-compatible local API
- Extract embeddings from local model for [[in-memory-vector-storage]]
Do NOT use for
- Cloud LLM APIs (Anthropic/OpenAI — use [[http-client-auth-patterns]])
- Production GPU clusters (use [[vllm-paged-attention]] or [[tgi-streaming-inference]])
Quantization level guide
Format Bits Size(7B) Quality Use case
──────────────────────────────────────────────
Q2_K 2.6 2.8 GB poor emergency low-RAM only
Q4_K_S 4.4 4.1 GB ok minimum usable
Q4_K_M 4.8 4.5 GB good ← default recommendation
Q5_K_M 5.7 5.3 GB great if RAM allows
Q8_0 8.0 7.2 GB near-lossless pre-production
F16 16.0 13.5 GB lossless fine-tuning only
Rule: Q4_K_M for dev/testing, Q8_0 for production inference gates
Start llama.cpp HTTP server
# Download GGUF model
curl -L -o models/mistral-7b-q4.gguf \
"https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/resolve/main/mistral-7b-v0.1.Q4_K_M.gguf"
# Start server (OpenAI-compatible API)
./llama-server \
--model models/mistral-7b-q4.gguf \
--ctx-size 4096 \ # context window (tokens)
--n-gpu-layers 35 \ # layers on GPU (0 = CPU-only)
--threads 8 \ # CPU threads
--port 8080 \
--host 0.0.0.0 \
--parallel 4 # concurrent request slots
OpenAI-compatible client call
import OpenAI from 'openai'
const client = new OpenAI({
baseURL: 'http://localhost:8080/v1',
apiKey: 'not-needed', // llama.cpp ignores this
})
const completion = await client.chat.completions.create({
model: 'local',
messages: [{ role: 'user', content: 'What is 2+2?' }],
max_tokens: 256,
temperature: 0.1,
stream: true,
})
for await (const chunk of completion) {
process.stdout.write(chunk.choices[0]?.delta?.content ?? '')
}
Quantize a model yourself
# Convert HuggingFace model → GGUF
python3 convert_hf_to_gguf.py \
--outfile models/my-model-f16.gguf \
--model /path/to/hf-model
# Quantize F16 → Q4_K_M
./quantize \
models/my-model-f16.gguf \
models/my-model-q4km.gguf \
Q4_K_M
Extract embeddings
# GET /embedding (llama.cpp server endpoint)
curl -s http://localhost:8080/embedding \
-H "Content-Type: application/json" \
-d '{"content": "agent identity for vector search"}' \
| jq '.embedding | length' # → 4096 (model dim)
Anti-Fake-Pass Checklist
❌ ctx-size > model's trained max → gibberish past context boundary (check model card)
❌ n-gpu-layers too high → VRAM OOM; start at 0 and increment until it fits
❌ Q2_K for code tasks → quality too low; code fails to compile, logic errors
❌ --parallel > available RAM/VRAM → OOM on concurrent requests
❌ F16 for CPU inference → 2× slower than Q8_0 with same quality; use Q8_0
❌ Not setting --threads to nproc → defaults to 1 thread, 10× slower
04 同仓库更多 Skills
openai--sentry--sentry
Use when the user asks to inspect Sentry issues or events, summarize recent production errors, or pull basic Sentry health data via the Sentry API; perform read-only queries with the bundled script and require `SENTRY_AUTH_TOKEN`.
ito-basket-compare
Compare Itô prediction-market baskets against a user's knowledge base, portfolio notes, financial context, watchlist, or research thesis. Use for read-only basket comparison and gap analysis without investment advice or live trading.
implementing-network-traffic-baselining
Build network traffic baselines from NetFlow/IPFIX data using Python pandas for statistical analysis, z-score anomaly detection, and hourly/daily traffic pattern profiling
detecting-evasion-techniques-in-endpoint-logs
Detects defense evasion techniques used by adversaries in endpoint logs including log tampering, timestomping, process injection, and security tool disabling. Use when investigating suspicious endpoint behavior, building detection rules for evasion tactics, or conducting threat hunting for stealthy adversary activity. Activates for requests involving evasion detection, defense evasion analysis, log tampering detection, or MITRE ATT&CK TA0005.
implementing-microsegmentation-with-guardicore
Implementing microsegmentation using Akamai Guardicore Segmentation to map application dependencies, create granular network policies, visualize east-west traffic flows, and enforce least-privilege communication between workloads across data centers and cloud.
ramdisk-tmpfs-patterns
暂无简介