[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M758TTCZ2AKCFGJC7M3M7P5K":3,"skill-same:01M758TTCZ2AKCFGJC7M3M7P5K":68},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":45,"commitSha":46,"skillDirectory":47,"forksCount":48,"repositoryHtmlUrl":49,"repositoryHomepageUrl":50,"ownerHtmlUrl":51,"ownerBlogUrl":6,"ownerDescription":52,"updatedAt":53,"install":54,"resources":67},"01M758TTCZ2AKCFGJC7M3M7P5K","tensorrt-llm",null,"High-throughput LLM inference on NVIDIA GPUs.","lukemcqueen\u002Fsteadfaste-fleet","lukemcqueen","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F405240?v=4","01M757MHPAGRXTJAT46SKZP98R","Luke McQueen","01M757MHPZ573KP8DT9GB8SBQ7",4,"MIT","https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet\u002Fblob\u002Fmain\u002Fskills\u002Fmlops\u002Ftensorrt-llm\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: tensorrt-llm\ndescription: High-throughput LLM inference on NVIDIA GPUs.\nversion: 1.0.1\nauthor: Orchestra Research\nlicense: MIT\ndependencies: [tensorrt-llm, torch]\nplatforms: [linux, macos]\nmetadata:\n  hermes:\n    tags: [Inference Serving, TensorRT-LLM, NVIDIA, Inference Optimization, High Throughput, Low Latency, Production, FP8, INT4, In-Flight Batching, Multi-GPU]\n\n---\n\n# TensorRT-LLM\n\nNVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs.\n\n## When to use TensorRT-LLM\n\n**Use TensorRT-LLM when:**\n- Deploying on NVIDIA GPUs (A100, H100, GB200)\n- Need maximum throughput (24,000+ tokens\u002Fsec on Llama 3)\n- Require low latency for real-time applications\n- Working with quantized models (FP8, INT4, FP4)\n- Scaling across multiple GPUs or nodes\n\n**Use vLLM instead when:**\n- Need simpler setup and Python-first API\n- Want PagedAttention without TensorRT compilation\n- Working with AMD GPUs or non-NVIDIA hardware\n\n**Use llama.cpp instead when:**\n- Deploying on CPU or Apple Silicon\n- Need edge deployment without NVIDIA GPUs\n- Want simpler GGUF quantization format\n\n## Quick start\n\n### Installation\n\n```bash\n# Docker (recommended) — images are on NGC (nvcr.io), not Docker Hub.\n# Replace x.y.z with the desired version (e.g. 1.2.1). Browse tags on NGC:\n# https:\u002F\u002Fcatalog.ngc.nvidia.com\u002Forgs\u002Fnvidia\u002Fteams\u002Ftensorrt-llm\u002Fcontainers\u002Frelease\u002Ftags\ndocker pull nvcr.io\u002Fnvidia\u002Ftensorrt-llm\u002Frelease:x.y.z\n\n# pip install (current stable GA)\npip install tensorrt_llm\n\n# Requires CUDA 13.2.1, TensorRT 10.x, Python 3.10-3.12\n```\n\n### Basic inference\n\n```python\nfrom tensorrt_llm import LLM, SamplingParams\n\n# Initialize model\nllm = LLM(model=\"meta-llama\u002FMeta-Llama-3-8B\")\n\n# Configure sampling\nsampling_params = SamplingParams(\n    max_tokens=100,\n    temperature=0.7,\n    top_p=0.9\n)\n\n# Generate\nprompts = [\"Explain quantum computing\"]\noutputs = llm.generate(prompts, sampling_params)\n\nfor output in outputs:\n    print(output.text)\n```\n\n### Serving with trtllm-serve\n\n```bash\n# Start server (automatic model download and compilation)\ntrtllm-serve meta-llama\u002FMeta-Llama-3-8B \\\n    --tp_size 4 \\              # Tensor parallelism (4 GPUs)\n    --max_batch_size 256 \\\n    --max_num_tokens 4096\n\n# Client request\ncurl -X POST http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"model\": \"meta-llama\u002FMeta-Llama-3-8B\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}],\n    \"temperature\": 0.7,\n    \"max_tokens\": 100\n  }'\n```\n\n## Key features\n\n### Performance optimizations\n- **In-flight batching**: Dynamic batching during generation\n- **Paged KV cache**: Efficient memory management\n- **Flash Attention**: Optimized attention kernels\n- **Quantization**: FP8, INT4, FP4 for 2-4× faster inference\n- **CUDA graphs**: Reduced kernel launch overhead\n\n### Parallelism\n- **Tensor parallelism (TP)**: Split model across GPUs\n- **Pipeline parallelism (PP)**: Layer-wise distribution\n- **Expert parallelism**: For Mixture-of-Experts models\n- **Multi-node**: Scale beyond single machine\n\n### Advanced features\n- **Speculative decoding**: Faster generation with draft models\n- **LoRA serving**: Efficient multi-adapter deployment\n- **Disaggregated serving**: Separate prefill and generation\n\n## Common patterns\n\n### Quantized model (FP8)\n\n```python\nfrom tensorrt_llm import LLM\n\n# Load FP8 quantized model (2× faster, 50% memory)\nllm = LLM(\n    model=\"meta-llama\u002FMeta-Llama-3-70B\",\n    dtype=\"fp8\",\n    max_num_tokens=8192\n)\n\n# Inference same as before\noutputs = llm.generate([\"Summarize this article...\"])\n```\n\n### Multi-GPU deployment\n\n```python\n# Tensor parallelism across 8 GPUs\nllm = LLM(\n    model=\"meta-llama\u002FMeta-Llama-3-405B\",\n    tensor_parallel_size=8,\n    dtype=\"fp8\"\n)\n```\n\n### Batch inference\n\n```python\n# Process 100 prompts efficiently\nprompts = [f\"Question {i}: ...\" for i in range(100)]\n\noutputs = llm.generate(\n    prompts,\n    sampling_params=SamplingParams(max_tokens=200)\n)\n\n# Automatic in-flight batching for maximum throughput\n```\n\n## Performance benchmarks\n\n**Meta Llama 3-8B** (H100 GPU):\n- Throughput: 24,000 tokens\u002Fsec\n- Latency: ~10ms per token\n- vs PyTorch: **100× faster**\n\n**Llama 3-70B** (8× A100 80GB):\n- FP8 quantization: 2× faster than FP16\n- Memory: 50% reduction with FP8\n\n## Supported models\n\n- **LLaMA family**: Llama 2, Llama 3, CodeLlama\n- **GPT family**: GPT-2, GPT-J, GPT-NeoX\n- **Qwen**: Qwen, Qwen2, QwQ\n- **DeepSeek**: DeepSeek-V2, DeepSeek-V3\n- **Mixtral**: Mixtral-8x7B, Mixtral-8x22B\n- **Vision**: LLaVA, Phi-3-vision\n- **100+ models** on HuggingFace\n\n## References\n\n- **[Optimization Guide](references\u002Foptimization.md)** - Quantization, batching, KV cache tuning\n- **[Multi-GPU Setup](references\u002Fmulti-gpu.md)** - Tensor\u002Fpipeline parallelism, multi-node\n- **[Serving Guide](references\u002Fserving.md)** - Production deployment, monitoring, autoscaling\n\n## Resources\n\n- **Docs**: https:\u002F\u002Fnvidia.github.io\u002FTensorRT-LLM\u002F\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FTensorRT-LLM\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=tensorrt_llm\n\n\n","# TensorRT-LLM\n\nNVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs.\n\n## When to use TensorRT-LLM\n\n**Use TensorRT-LLM when:**\n- Deploying on NVIDIA GPUs (A100, H100, GB200)\n- Need maximum throughput (24,000+ tokens\u002Fsec on Llama 3)\n- Require low latency for real-time applications\n- Working with quantized models (FP8, INT4, FP4)\n- Scaling across multiple GPUs or nodes\n\n**Use vLLM instead when:**\n- Need simpler setup and Python-first API\n- Want PagedAttention without TensorRT compilation\n- Working with AMD GPUs or non-NVIDIA hardware\n\n**Use llama.cpp instead when:**\n- Deploying on CPU or Apple Silicon\n- Need edge deployment without NVIDIA GPUs\n- Want simpler GGUF quantization format\n\n## Quick start\n\n### Installation\n\n```bash\n# Docker (recommended) — images are on NGC (nvcr.io), not Docker Hub.\n# Replace x.y.z with the desired version (e.g. 1.2.1). Browse tags on NGC:\n# https:\u002F\u002Fcatalog.ngc.nvidia.com\u002Forgs\u002Fnvidia\u002Fteams\u002Ftensorrt-llm\u002Fcontainers\u002Frelease\u002Ftags\ndocker pull nvcr.io\u002Fnvidia\u002Ftensorrt-llm\u002Frelease:x.y.z\n\n# pip install (current stable GA)\npip install tensorrt_llm\n\n# Requires CUDA 13.2.1, TensorRT 10.x, Python 3.10-3.12\n```\n\n### Basic inference\n\n```python\nfrom tensorrt_llm import LLM, SamplingParams\n\n# Initialize model\nllm = LLM(model=\"meta-llama\u002FMeta-Llama-3-8B\")\n\n# Configure sampling\nsampling_params = SamplingParams(\n    max_tokens=100,\n    temperature=0.7,\n    top_p=0.9\n)\n\n# Generate\nprompts = [\"Explain quantum computing\"]\noutputs = llm.generate(prompts, sampling_params)\n\nfor output in outputs:\n    print(output.text)\n```\n\n### Serving with trtllm-serve\n\n```bash\n# Start server (automatic model download and compilation)\ntrtllm-serve meta-llama\u002FMeta-Llama-3-8B \\\n    --tp_size 4 \\              # Tensor parallelism (4 GPUs)\n    --max_batch_size 256 \\\n    --max_num_tokens 4096\n\n# Client request\ncurl -X POST http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"model\": \"meta-llama\u002FMeta-Llama-3-8B\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}],\n    \"temperature\": 0.7,\n    \"max_tokens\": 100\n  }'\n```\n\n## Key features\n\n### Performance optimizations\n- **In-flight batching**: Dynamic batching during generation\n- **Paged KV cache**: Efficient memory management\n- **Flash Attention**: Optimized attention kernels\n- **Quantization**: FP8, INT4, FP4 for 2-4× faster inference\n- **CUDA graphs**: Reduced kernel launch overhead\n\n### Parallelism\n- **Tensor parallelism (TP)**: Split model across GPUs\n- **Pipeline parallelism (PP)**: Layer-wise distribution\n- **Expert parallelism**: For Mixture-of-Experts models\n- **Multi-node**: Scale beyond single machine\n\n### Advanced features\n- **Speculative decoding**: Faster generation with draft models\n- **LoRA serving**: Efficient multi-adapter deployment\n- **Disaggregated serving**: Separate prefill and generation\n\n## Common patterns\n\n### Quantized model (FP8)\n\n```python\nfrom tensorrt_llm import LLM\n\n# Load FP8 quantized model (2× faster, 50% memory)\nllm = LLM(\n    model=\"meta-llama\u002FMeta-Llama-3-70B\",\n    dtype=\"fp8\",\n    max_num_tokens=8192\n)\n\n# Inference same as before\noutputs = llm.generate([\"Summarize this article...\"])\n```\n\n### Multi-GPU deployment\n\n```python\n# Tensor parallelism across 8 GPUs\nllm = LLM(\n    model=\"meta-llama\u002FMeta-Llama-3-405B\",\n    tensor_parallel_size=8,\n    dtype=\"fp8\"\n)\n```\n\n### Batch inference\n\n```python\n# Process 100 prompts efficiently\nprompts = [f\"Question {i}: ...\" for i in range(100)]\n\noutputs = llm.generate(\n    prompts,\n    sampling_params=SamplingParams(max_tokens=200)\n)\n\n# Automatic in-flight batching for maximum throughput\n```\n\n## Performance benchmarks\n\n**Meta Llama 3-8B** (H100 GPU):\n- Throughput: 24,000 tokens\u002Fsec\n- Latency: ~10ms per token\n- vs PyTorch: **100× faster**\n\n**Llama 3-70B** (8× A100 80GB):\n- FP8 quantization: 2× faster than FP16\n- Memory: 50% reduction with FP8\n\n## Supported models\n\n- **LLaMA family**: Llama 2, Llama 3, CodeLlama\n- **GPT family**: GPT-2, GPT-J, GPT-NeoX\n- **Qwen**: Qwen, Qwen2, QwQ\n- **DeepSeek**: DeepSeek-V2, DeepSeek-V3\n- **Mixtral**: Mixtral-8x7B, Mixtral-8x22B\n- **Vision**: LLaVA, Phi-3-vision\n- **100+ models** on HuggingFace\n\n## References\n\n- **[Optimization Guide](references\u002Foptimization.md)** - Quantization, batching, KV cache tuning\n- **[Multi-GPU Setup](references\u002Fmulti-gpu.md)** - Tensor\u002Fpipeline parallelism, multi-node\n- **[Serving Guide](references\u002Fserving.md)** - Production deployment, monitoring, autoscaling\n\n## Resources\n\n- **Docs**: https:\u002F\u002Fnvidia.github.io\u002FTensorRT-LLM\u002F\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FTensorRT-LLM\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=tensorrt_llm\n\n\n",{"name":5,"author":24,"license":15,"version":25,"metadata":26,"platforms":40,"description":7,"dependencies":43},"Orchestra Research","1.0.1",{"hermes":27},{"tags":28},[29,30,31,32,33,34,35,36,37,38,39],"Inference Serving","TensorRT-LLM","NVIDIA","Inference Optimization","High Throughput","Low Latency","Production","FP8","INT4","In-Flight Batching","Multi-GPU",[41,42],"linux","macos",[5,44],"torch","main","2e6aa8d15ab0a366c90be54f61bf9c6e97015e83","skills\u002Fmlops\u002Ftensorrt-llm",1,"https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet","steadfaste.com","https:\u002F\u002Fgithub.com\u002Flukemcqueen","AI Architect | Building Hermes Cortex: governed multi-agent AI for autonomous software engineering | Enterprise Tech Consultant","2026-11-14 22:33:34",{"source":8,"skillName":5,"commands":55,"prompt":65,"zipAvailable":66},[56,59,62],{"tool":57,"command":58},"npx","npx skills add lukemcqueen\u002Fsteadfaste-fleet --skill tensorrt-llm",{"tool":60,"command":61},"bunx","bunx skills add lukemcqueen\u002Fsteadfaste-fleet --skill tensorrt-llm",{"tool":63,"command":64},"pnpm","pnpm dlx skills add lukemcqueen\u002Fsteadfaste-fleet --skill tensorrt-llm","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet\u002Fblob\u002Fmain\u002Fskills\u002Fmlops\u002Ftensorrt-llm\u002FSKILL.md",true,[],[69,77,84,91,98,105],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":76},"01M7DXY4G8NSVB59K8B6S2AQNP","macbook-linux-maintenance","Legacy MacBook Linux fixes: GPU wedge, reboot hang, thermal.","https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet\u002Fblob\u002Fmain\u002Fskills\u002Fdevops\u002Fmacbook-linux-maintenance\u002FSKILL.md",[],[],"2026-11-15 14:19:17",{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":76},"01M7DXXYJCXFJ3TGB84ZQ8KFE6","governance-regression-triage","Triage a host's governance issues: what changed.","https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet\u002Fblob\u002Fmain\u002Fskills\u002Fdevops\u002Fgovernance-regression-triage\u002FSKILL.md",[],[],{"skillNo":85,"name":86,"nameZh":6,"description":87,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":76},"01M7DXXSAP1G8FDRJAMXQGBBMK","enforcement-immutability","Use when enforcement files lose their immutable flag.","https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet\u002Fblob\u002Fmain\u002Fskills\u002Fdevops\u002Fenforcement-immutability\u002FSKILL.md",[],[],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":76},"01M78SSDQVRYX64Y16X32TC0NS","steadfaste-fleet-repo-deploy","Use when committing to the steadfaste-fleet repo.","https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet\u002Fblob\u002Fmain\u002Fskills\u002Fdevops\u002Fsteadfaste-fleet-repo-deploy\u002FSKILL.md",[],[],{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":104,"publishedAt":76},"01M78SS928F3R2RJAYVW4E30CE","steadfaste-fleet-maintenance","Maintain an installed Steadfaste Fleet instance — update both the upstream Hermes Agent and the fleet repo layer, merge diverged histories, sync skills safely, and troubleshoot common issues. Covers the dual-update workflow (hermes update + fleet-update.sh), encrypted brain data backup, mycortex import, and recovery techniques.","https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet\u002Fblob\u002Fmain\u002Fskills\u002Fdevops\u002Fsteadfaste-fleet-maintenance\u002FSKILL.md",[],[],{"skillNo":106,"name":107,"nameZh":6,"description":108,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":109,"categories":110,"keywords":111,"publishedAt":76},"01M78SP1CHW0T0BWE5ZT6JBPX8","fleet-preflight","DEPRECATED alias — merged into survey-before-action (2026-08-20). Load survey-before-action instead; its \"Repo-Specific Pre-Flight\" section covers git search, Hermes boundary, and deployment verification.","https:\u002F\u002Fgithub.com\u002Flukemcqueen\u002Fsteadfaste-fleet\u002Fblob\u002Fmain\u002Fskills\u002Fdevops\u002Ffleet-preflight\u002FSKILL.md",[],[]]