[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6CAP6H2ASFD24E5B9RMFVCP":3,"skill-same:01M6CAP6H2ASFD24E5B9RMFVCP":93},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":35,"linkOnly":36,"markdownContent":37,"markdownBody":38,"frontMatter":39,"refName":53,"commitSha":54,"skillDirectory":55,"forksCount":56,"repositoryHtmlUrl":57,"repositoryHomepageUrl":6,"ownerHtmlUrl":58,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":59,"install":60,"resources":73},"01M6CAP6H2ASFD24E5B9RMFVCP","llm-inference-and-serving",null,"Run, serve, and optimize LLMs: llama.cpp local inference, vLLM high-throughput serving, and abliteration with Obliteratus.","adnichols\u002Fai-configs","adnichols","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F493746?v=4","01M6C33VRY9WF6XWYPT1AFA7JJ","Aaron Nichols","01M6C33VWTNKGRM92E9G643V4W",5,"MIT","https:\u002F\u002Fgithub.com\u002Fadnichols\u002Fai-configs\u002Fblob\u002Fmain\u002F_hermes\u002Fdefault\u002Fprofiles\u002Fwriting\u002Fskills\u002Fmlops\u002Fllm-inference-and-serving\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34],"inference","llm inference","llm inference and serving","macos","llms","linux","windows","cpp","run","HuggingFace","Llama","vLLM","hermes","hermes agent","hub","obliteratus","2026-11-10 10:12:30",false,"---\nname: llm-inference-and-serving\ndescription: \"Run, serve, and optimize LLMs: llama.cpp local inference, vLLM high-throughput serving, and abliteration with Obliteratus.\"\nversion: 1.0.0\nauthor: Hermes Agent\nlicense: MIT\nplatforms: [linux, macos, windows]\nmetadata:\n  hermes:\n    tags: [llm, inference, vllm, llama.cpp, gguf, serving, obliteratus]\n    related_skills: [huggingface-hub]\n---\n\n# LLM Inference and Serving\n\nThree distinct but related domains: local CPU\u002FGPU inference with llama.cpp, high-throughput server deployment with vLLM, and model surgery (abliteration) with Obliteratus.\n\n## llama.cpp (Local GGUF Inference)\n\n### Setup\n```bash\nbrew install llama.cpp          # macOS (Metal)\n# OR clone and build with CUDA \u002F ROCm flags\n```\n\n### Download Models\n```bash\n# Via huggingface-cli (preferred)\nhuggingface-cli download TheBloke\u002FLlama-2-7B-GGUF llama-2-7b.Q4_K_M.gguf --local-dir .\u002Fmodels\n\n# Via curl (direct)\ncurl -L -o .\u002Fmodels\u002Fmodel.gguf \"\u003Curl>\"\n```\n\n### Run Inference\n```bash\n# Interactive chat\nllama-cli -m .\u002Fmodels\u002Fmodel.gguf --color -c 4096 -p \"You are a helpful assistant.\"\n\n# Single completion\nllama-cli -m .\u002Fmodels\u002Fmodel.gguf -p \"The capital of France is\" -n 32\n\n# Server mode (OpenAI-compatible API)\nllama-server -m .\u002Fmodels\u002Fmodel.gguf -c 4096 --port 8080\n# Test: curl http:\u002F\u002Flocalhost:8080\u002Fv1\u002Fchat\u002Fcompletions -d '{\"messages\":[{\"role\":\"user\",\"content\":\"hello\"}]}'\n```\n\n### Key Flags\n| Flag | Meaning |\n|------|---------|\n| `-m` | Model path |\n| `-c` | Context size |\n| `-n` | Tokens to generate |\n| `--color` | Syntax-highlight special tokens |\n| `-ngl` | Number of GPU layers to offload |\n| `--port` | Server port |\n\n### GGUF Quantization\n```bash\n# Convert HF model to GGUF\npython convert_hf_to_gguf.py .\u002Fhf-model\u002F --outfile .\u002Fmodel.gguf --outtype q4_k_m\n```\n\nQuantization types (smaller = faster but less accurate): `Q2_K`, `Q3_K_M`, `Q4_K_M`, `Q5_K_M`, `Q6_K`, `Q8_0`, `FP16`\n\n## vLLM (High-Throughput Serving)\n\n### Setup\n```bash\npip install vllm\n```\n\n### Launch Server\n```bash\npython -m vllm.entrypoints.openai.api_server \\\n  --model meta-llama\u002FLlama-2-7b-hf \\\n  --tensor-parallel-size 1 \\\n  --port 8000\n```\n\n### Client Usage (OpenAI-compatible)\n```python\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http:\u002F\u002Flocalhost:8000\u002Fv1\", api_key=\"dummy\")\nresponse = client.chat.completions.create(model=\"meta-llama\u002FLlama-2-7b-hf\", messages=[...])\n```\n\n### Key Flags\n| Flag | Meaning |\n|------|---------|\n| `--tensor-parallel-size` | GPUs per model replica |\n| `--pipeline-parallel-size` | Model replicas |\n| `--max-model-len` | Max context length |\n| `--quantization` | `awq`, `gptq`, `squeezellm` |\n| `--enable-prefix-caching` | Automatic prompt caching |\n\n### Benchmark\n```bash\npython benchmarks\u002Fbenchmark_throughput.py --model meta-llama\u002FLlama-2-7b-hf --input-len 512 --output-len 128\n```\n\n## Obliteratus (Abliteration)\n\nRemove refusal behaviors from open-weight LLMs via diff-in-means abliteration.\n\n### Setup\n```bash\npip install obliteratus\n```\n\n### Run Abliteration\n```python\nfrom obliteratus import abliterate\nabliterate(\n    model_path=\"meta-llama\u002FLlama-2-7b-hf\",\n    output_path=\".\u002Funhinged-llama-7b\",\n    harm_categories=[\"illegal\", \"hate\", \"harassment\"],\n)\n```\n\n### Post-Abliteration Verification\n- Run a refusal test suite (e.g., `jailbreak` benchmarks)\n- Check that helpfulness wasn't degraded on standard benchmarks\n- Quantize to GGUF and test with llama.cpp if deploying locally\n\n## Common Pitfalls\n- **llama.cpp GPU layers:** `-ngl` too high for your VRAM causes OOM; start at 99 and reduce if crashes\n- **vLLM CUDA version:** vLLM is tightly coupled to PyTorch CUDA version; use the exact wheel for your CUDA\n- **GGUF context size:** `-c` larger than model's training context causes gibberish; respect model card\n- **Obliteratus ethics:** Only abliterate models you own or have explicit license to modify; never redistribute without checking license terms\n","# LLM Inference and Serving\n\nThree distinct but related domains: local CPU\u002FGPU inference with llama.cpp, high-throughput server deployment with vLLM, and model surgery (abliteration) with Obliteratus.\n\n## llama.cpp (Local GGUF Inference)\n\n### Setup\n```bash\nbrew install llama.cpp          # macOS (Metal)\n# OR clone and build with CUDA \u002F ROCm flags\n```\n\n### Download Models\n```bash\n# Via huggingface-cli (preferred)\nhuggingface-cli download TheBloke\u002FLlama-2-7B-GGUF llama-2-7b.Q4_K_M.gguf --local-dir .\u002Fmodels\n\n# Via curl (direct)\ncurl -L -o .\u002Fmodels\u002Fmodel.gguf \"\u003Curl>\"\n```\n\n### Run Inference\n```bash\n# Interactive chat\nllama-cli -m .\u002Fmodels\u002Fmodel.gguf --color -c 4096 -p \"You are a helpful assistant.\"\n\n# Single completion\nllama-cli -m .\u002Fmodels\u002Fmodel.gguf -p \"The capital of France is\" -n 32\n\n# Server mode (OpenAI-compatible API)\nllama-server -m .\u002Fmodels\u002Fmodel.gguf -c 4096 --port 8080\n# Test: curl http:\u002F\u002Flocalhost:8080\u002Fv1\u002Fchat\u002Fcompletions -d '{\"messages\":[{\"role\":\"user\",\"content\":\"hello\"}]}'\n```\n\n### Key Flags\n| Flag | Meaning |\n|------|---------|\n| `-m` | Model path |\n| `-c` | Context size |\n| `-n` | Tokens to generate |\n| `--color` | Syntax-highlight special tokens |\n| `-ngl` | Number of GPU layers to offload |\n| `--port` | Server port |\n\n### GGUF Quantization\n```bash\n# Convert HF model to GGUF\npython convert_hf_to_gguf.py .\u002Fhf-model\u002F --outfile .\u002Fmodel.gguf --outtype q4_k_m\n```\n\nQuantization types (smaller = faster but less accurate): `Q2_K`, `Q3_K_M`, `Q4_K_M`, `Q5_K_M`, `Q6_K`, `Q8_0`, `FP16`\n\n## vLLM (High-Throughput Serving)\n\n### Setup\n```bash\npip install vllm\n```\n\n### Launch Server\n```bash\npython -m vllm.entrypoints.openai.api_server \\\n  --model meta-llama\u002FLlama-2-7b-hf \\\n  --tensor-parallel-size 1 \\\n  --port 8000\n```\n\n### Client Usage (OpenAI-compatible)\n```python\nfrom openai import OpenAI\nclient = OpenAI(base_url=\"http:\u002F\u002Flocalhost:8000\u002Fv1\", api_key=\"dummy\")\nresponse = client.chat.completions.create(model=\"meta-llama\u002FLlama-2-7b-hf\", messages=[...])\n```\n\n### Key Flags\n| Flag | Meaning |\n|------|---------|\n| `--tensor-parallel-size` | GPUs per model replica |\n| `--pipeline-parallel-size` | Model replicas |\n| `--max-model-len` | Max context length |\n| `--quantization` | `awq`, `gptq`, `squeezellm` |\n| `--enable-prefix-caching` | Automatic prompt caching |\n\n### Benchmark\n```bash\npython benchmarks\u002Fbenchmark_throughput.py --model meta-llama\u002FLlama-2-7b-hf --input-len 512 --output-len 128\n```\n\n## Obliteratus (Abliteration)\n\nRemove refusal behaviors from open-weight LLMs via diff-in-means abliteration.\n\n### Setup\n```bash\npip install obliteratus\n```\n\n### Run Abliteration\n```python\nfrom obliteratus import abliterate\nabliterate(\n    model_path=\"meta-llama\u002FLlama-2-7b-hf\",\n    output_path=\".\u002Funhinged-llama-7b\",\n    harm_categories=[\"illegal\", \"hate\", \"harassment\"],\n)\n```\n\n### Post-Abliteration Verification\n- Run a refusal test suite (e.g., `jailbreak` benchmarks)\n- Check that helpfulness wasn't degraded on standard benchmarks\n- Quantize to GGUF and test with llama.cpp if deploying locally\n\n## Common Pitfalls\n- **llama.cpp GPU layers:** `-ngl` too high for your VRAM causes OOM; start at 99 and reduce if crashes\n- **vLLM CUDA version:** vLLM is tightly coupled to PyTorch CUDA version; use the exact wheel for your CUDA\n- **GGUF context size:** `-c` larger than model's training context causes gibberish; respect model card\n- **Obliteratus ethics:** Only abliterate models you own or have explicit license to modify; never redistribute without checking license terms\n",{"name":5,"author":40,"license":15,"version":41,"metadata":42,"platforms":52,"description":7},"Hermes Agent","1.0.0",{"hermes":43},{"tags":44,"related_skills":50},[45,19,46,47,48,49,34],"llm","vllm","llama.cpp","gguf","serving",[51],"huggingface-hub",[24,22,25],"main","64e2fc9afa7bb3da1138eb73399144d0e7e73e9d","_hermes\u002Fdefault\u002Fprofiles\u002Fwriting\u002Fskills\u002Fmlops\u002Fllm-inference-and-serving",4,"https:\u002F\u002Fgithub.com\u002Fadnichols\u002Fai-configs","https:\u002F\u002Fgithub.com\u002Fadnichols","2026-11-06 00:28:38",{"source":8,"skillName":5,"commands":61,"prompt":71,"zipAvailable":72},[62,65,68],{"tool":63,"command":64},"npx","npx skills add adnichols\u002Fai-configs --skill llm-inference-and-serving",{"tool":66,"command":67},"bunx","bunx skills add adnichols\u002Fai-configs --skill llm-inference-and-serving",{"tool":69,"command":70},"pnpm","pnpm dlx skills add adnichols\u002Fai-configs --skill llm-inference-and-serving","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fadnichols\u002Fai-configs\u002Fblob\u002Fmain\u002F_hermes\u002Fdefault\u002Fprofiles\u002Fwriting\u002Fskills\u002Fmlops\u002Fllm-inference-and-serving\u002FSKILL.md",true,[74,79,85,89],{"path":75,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":77,"sourceHtmlUrl":78},"LICENSE","许可证",10769,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fchains\u002Fcardano-foundation-suggest-tooling\u002FLICENSE",{"path":80,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Fecosystem-map.md","REFERENCE","参考文档",14324,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fchains\u002Fcardano-foundation-suggest-tooling\u002Freferences\u002Fecosystem-map.md",{"path":86,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":87,"sourceHtmlUrl":88},"SOURCE.md",386,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fchains\u002Fcardano-foundation-suggest-tooling\u002FSOURCE.md",{"path":90,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":91,"sourceHtmlUrl":92},"TRUST.auto.yaml",2232,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fchains\u002Fcardano-foundation-suggest-tooling\u002FTRUST.auto.yaml",[94,111,145,170,181,197],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":35},"01M6CB7RN791P5RRRCJWWC7DAR","blocked-page-recovery","Use when a fetch fails: 403\u002F429, paywall, WAF, bot wall.","https:\u002F\u002Fgithub.com\u002Fadnichols\u002Fai-configs\u002Fblob\u002Fmain\u002F_hermes\u002Fdefault\u002Fskills\u002Fweb\u002Fblocked-page-recovery\u002FSKILL.md",[],[101,102,103,104,105,106,107,32,31,108,109,25,110,24,22],"recovery","page","blocked page recovery","bot","fallback","use","fetch","Citations","waf","research",{"skillNo":112,"name":113,"nameZh":6,"description":114,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":115,"categories":116,"keywords":130,"publishedAt":35},"01M6CB7NQY2VVYCQJ1QBXACQF4","writing-plans","Use when creating or updating executable software plans. Keep plans as local files unless the user explicitly asks to register them in Doct.","https:\u002F\u002Fgithub.com\u002Fadnichols\u002Fai-configs\u002Fblob\u002Fmain\u002F_hermes\u002Fdefault\u002Fskills\u002Fsoftware-development\u002Fwriting-plans\u002FSKILL.md",[117,122,125],{"categoryNo":118,"categoryName":119,"taxonomyType":120,"taxonomyTypeDesc":121},"01JBSEEDCAT000000000000018","写作与文案","SCENARIO","使用场景",{"categoryNo":123,"categoryName":124,"taxonomyType":120,"taxonomyTypeDesc":121},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":126,"categoryName":127,"taxonomyType":128,"taxonomyTypeDesc":129},"01JBSEEDCAT000000000000036","内容创作者","PROFESSION","适用职业",[131,132,133,134,135,136,137,138,139,140,141,142,106,143,32,31,144],"writing","writing plans","documentation","test","workflow","design","user","development","planning","files","implementation","html","ops","plan",{"skillNo":146,"name":147,"nameZh":6,"description":148,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":149,"categories":150,"keywords":160,"publishedAt":35},"01M6CB7JME246KZQHP73Z2STDF","test-driven-development","TDD: enforce RED-GREEN-REFACTOR, tests before code.","https:\u002F\u002Fgithub.com\u002Fadnichols\u002Fai-configs\u002Fblob\u002Fmain\u002F_hermes\u002Fdefault\u002Fskills\u002Fsoftware-development\u002Ftest-driven-development\u002FSKILL.md",[151,154,157],{"categoryNo":152,"categoryName":153,"taxonomyType":120,"taxonomyTypeDesc":121},"01JBSEEDCAT000000000000007","代码审查",{"categoryNo":155,"categoryName":156,"taxonomyType":120,"taxonomyTypeDesc":121},"01JBSEEDCAT000000000000008","测试与质量",{"categoryNo":158,"categoryName":159,"taxonomyType":128,"taxonomyTypeDesc":129},"01JBSEEDCAT000000000000037","律师与法务",[134,138,161,162,163,32,31,164,144,165,166,167,22,168,24,169,25],"test driven development","red-green-refactor","superpowers","quality","refactor","code","debugging","tdd","testing",{"skillNo":171,"name":172,"nameZh":6,"description":173,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":174,"categories":175,"keywords":177,"publishedAt":35},"01M6CB7FQ407KQTD4Z9VAWKY54","systematic-debugging","4-phase root cause debugging: understand bugs before fixing.","https:\u002F\u002Fgithub.com\u002Fadnichols\u002Fai-configs\u002Fblob\u002Fmain\u002F_hermes\u002Fdefault\u002Fskills\u002Fsoftware-development\u002Fsystematic-debugging\u002FSKILL.md",[176],{"categoryNo":155,"categoryName":156,"taxonomyType":120,"taxonomyTypeDesc":121},[167,178,138,163,32,31,179,144,180,25,134,24,22],"systematic debugging","troubleshooting","investigation",{"skillNo":182,"name":183,"nameZh":6,"description":184,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":185,"categories":186,"keywords":190,"publishedAt":35},"01M6CB7D57MK88HE5YPZHKH68Q","spike","Throwaway experiments to validate an idea before build.","https:\u002F\u002Fgithub.com\u002Fadnichols\u002Fai-configs\u002Fblob\u002Fmain\u002F_hermes\u002Fdefault\u002Fskills\u002Fsoftware-development\u002Fspike\u002FSKILL.md",[187],{"categoryNo":188,"categoryName":189,"taxonomyType":120,"taxonomyTypeDesc":121},"01JBSEEDCAT000000000000021","资料检索与研究",[183,25,191,192,110,193,24,22,194,195,196,32,31,144,139],"validate","experiments","prototype","feasibility","sketch","experiment",{"skillNo":198,"name":199,"nameZh":6,"description":200,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":201,"categories":202,"keywords":207,"publishedAt":35},"01M6CB7APAQ37YE5YE1MAAFT42","simplify-code","Parallel 3-agent cleanup of recent code changes.","https:\u002F\u002Fgithub.com\u002Fadnichols\u002Fai-configs\u002Fblob\u002Fmain\u002F_hermes\u002Fdefault\u002Fskills\u002Fsoftware-development\u002Fsimplify-code\u002FSKILL.md",[203,204],{"categoryNo":123,"categoryName":124,"taxonomyType":120,"taxonomyTypeDesc":121},{"categoryNo":205,"categoryName":206,"taxonomyType":128,"taxonomyTypeDesc":129},"01JBSEEDCAT000000000000030","开发工程师",[208,209,166,210,32,31,211,144,165,212,213,214,138,25,134,24,215,22],"simplify code","simplify","subagent","parallel","code-review","delegation","cleanup","review"]