[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M74EWP0RNDNC8AECTTRBR9TM":3,"skill-same:01M74EWP0RNDNC8AECTTRBR9TM":81},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":45,"commitSha":46,"skillDirectory":47,"forksCount":14,"repositoryHtmlUrl":48,"repositoryHomepageUrl":6,"ownerHtmlUrl":49,"ownerBlogUrl":50,"ownerDescription":51,"updatedAt":52,"install":53,"resources":66},"01M74EWP0RNDNC8AECTTRBR9TM","tensorrt-llm",null,"Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100\u002FH100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8\u002FINT4), in-flight batching, and multi-GPU scaling.","aniruddhaadak80\u002Fherclew","aniruddhaadak80","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F127435065?v=4","01M74EJ1XW3EF2EYN26PFN1TG8","ANIRUDDHA ADAK","01M74EJ1YFT2GM6APEHR1CTTVF",0,"MIT","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Foptional-skills\u002Fmlops\u002Ftensorrt-llm\u002FSKILL.md",[],[],"2026-11-11 15:11:14",false,"---\nname: tensorrt-llm\ndescription: Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100\u002FH100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8\u002FINT4), in-flight batching, and multi-GPU scaling.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ndependencies: [tensorrt-llm, torch]\nplatforms: [linux, macos]\nmetadata:\n  hermes:\n    tags: [Inference Serving, TensorRT-LLM, NVIDIA, Inference Optimization, High Throughput, Low Latency, Production, FP8, INT4, In-Flight Batching, Multi-GPU]\n\n---\n\n# TensorRT-LLM\n\nNVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs.\n\n## When to use TensorRT-LLM\n\n**Use TensorRT-LLM when:**\n- Deploying on NVIDIA GPUs (A100, H100, GB200)\n- Need maximum throughput (24,000+ tokens\u002Fsec on Llama 3)\n- Require low latency for real-time applications\n- Working with quantized models (FP8, INT4, FP4)\n- Scaling across multiple GPUs or nodes\n\n**Use vLLM instead when:**\n- Need simpler setup and Python-first API\n- Want PagedAttention without TensorRT compilation\n- Working with AMD GPUs or non-NVIDIA hardware\n\n**Use llama.cpp instead when:**\n- Deploying on CPU or Apple Silicon\n- Need edge deployment without NVIDIA GPUs\n- Want simpler GGUF quantization format\n\n## Quick start\n\n### Installation\n\n```bash\n# Docker (recommended)\ndocker pull nvidia\u002Ftensorrt_llm:latest\n\n# pip install\npip install tensorrt_llm==1.2.0rc3\n\n# Requires CUDA 13.0.0, TensorRT 10.13.2, Python 3.10-3.12\n```\n\n### Basic inference\n\n```python\nfrom tensorrt_llm import LLM, SamplingParams\n\n# Initialize model\nllm = LLM(model=\"meta-llama\u002FMeta-Llama-3-8B\")\n\n# Configure sampling\nsampling_params = SamplingParams(\n    max_tokens=100,\n    temperature=0.7,\n    top_p=0.9\n)\n\n# Generate\nprompts = [\"Explain quantum computing\"]\noutputs = llm.generate(prompts, sampling_params)\n\nfor output in outputs:\n    print(output.text)\n```\n\n### Serving with trtllm-serve\n\n```bash\n# Start server (automatic model download and compilation)\ntrtllm-serve meta-llama\u002FMeta-Llama-3-8B \\\n    --tp_size 4 \\              # Tensor parallelism (4 GPUs)\n    --max_batch_size 256 \\\n    --max_num_tokens 4096\n\n# Client request\ncurl -X POST http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"model\": \"meta-llama\u002FMeta-Llama-3-8B\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}],\n    \"temperature\": 0.7,\n    \"max_tokens\": 100\n  }'\n```\n\n## Key features\n\n### Performance optimizations\n- **In-flight batching**: Dynamic batching during generation\n- **Paged KV cache**: Efficient memory management\n- **Flash Attention**: Optimized attention kernels\n- **Quantization**: FP8, INT4, FP4 for 2-4× faster inference\n- **CUDA graphs**: Reduced kernel launch overhead\n\n### Parallelism\n- **Tensor parallelism (TP)**: Split model across GPUs\n- **Pipeline parallelism (PP)**: Layer-wise distribution\n- **Expert parallelism**: For Mixture-of-Experts models\n- **Multi-node**: Scale beyond single machine\n\n### Advanced features\n- **Speculative decoding**: Faster generation with draft models\n- **LoRA serving**: Efficient multi-adapter deployment\n- **Disaggregated serving**: Separate prefill and generation\n\n## Common patterns\n\n### Quantized model (FP8)\n\n```python\nfrom tensorrt_llm import LLM\n\n# Load FP8 quantized model (2× faster, 50% memory)\nllm = LLM(\n    model=\"meta-llama\u002FMeta-Llama-3-70B\",\n    dtype=\"fp8\",\n    max_num_tokens=8192\n)\n\n# Inference same as before\noutputs = llm.generate([\"Summarize this article...\"])\n```\n\n### Multi-GPU deployment\n\n```python\n# Tensor parallelism across 8 GPUs\nllm = LLM(\n    model=\"meta-llama\u002FMeta-Llama-3-405B\",\n    tensor_parallel_size=8,\n    dtype=\"fp8\"\n)\n```\n\n### Batch inference\n\n```python\n# Process 100 prompts efficiently\nprompts = [f\"Question {i}: ...\" for i in range(100)]\n\noutputs = llm.generate(\n    prompts,\n    sampling_params=SamplingParams(max_tokens=200)\n)\n\n# Automatic in-flight batching for maximum throughput\n```\n\n## Performance benchmarks\n\n**Meta Llama 3-8B** (H100 GPU):\n- Throughput: 24,000 tokens\u002Fsec\n- Latency: ~10ms per token\n- vs PyTorch: **100× faster**\n\n**Llama 3-70B** (8× A100 80GB):\n- FP8 quantization: 2× faster than FP16\n- Memory: 50% reduction with FP8\n\n## Supported models\n\n- **LLaMA family**: Llama 2, Llama 3, CodeLlama\n- **GPT family**: GPT-2, GPT-J, GPT-NeoX\n- **Qwen**: Qwen, Qwen2, QwQ\n- **DeepSeek**: DeepSeek-V2, DeepSeek-V3\n- **Mixtral**: Mixtral-8x7B, Mixtral-8x22B\n- **Vision**: LLaVA, Phi-3-vision\n- **100+ models** on HuggingFace\n\n## References\n\n- **[Optimization Guide](references\u002Foptimization.md)** - Quantization, batching, KV cache tuning\n- **[Multi-GPU Setup](references\u002Fmulti-gpu.md)** - Tensor\u002Fpipeline parallelism, multi-node\n- **[Serving Guide](references\u002Fserving.md)** - Production deployment, monitoring, autoscaling\n\n## Resources\n\n- **Docs**: https:\u002F\u002Fnvidia.github.io\u002FTensorRT-LLM\u002F\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FTensorRT-LLM\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=tensorrt_llm\n\n\n","# TensorRT-LLM\n\nNVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs.\n\n## When to use TensorRT-LLM\n\n**Use TensorRT-LLM when:**\n- Deploying on NVIDIA GPUs (A100, H100, GB200)\n- Need maximum throughput (24,000+ tokens\u002Fsec on Llama 3)\n- Require low latency for real-time applications\n- Working with quantized models (FP8, INT4, FP4)\n- Scaling across multiple GPUs or nodes\n\n**Use vLLM instead when:**\n- Need simpler setup and Python-first API\n- Want PagedAttention without TensorRT compilation\n- Working with AMD GPUs or non-NVIDIA hardware\n\n**Use llama.cpp instead when:**\n- Deploying on CPU or Apple Silicon\n- Need edge deployment without NVIDIA GPUs\n- Want simpler GGUF quantization format\n\n## Quick start\n\n### Installation\n\n```bash\n# Docker (recommended)\ndocker pull nvidia\u002Ftensorrt_llm:latest\n\n# pip install\npip install tensorrt_llm==1.2.0rc3\n\n# Requires CUDA 13.0.0, TensorRT 10.13.2, Python 3.10-3.12\n```\n\n### Basic inference\n\n```python\nfrom tensorrt_llm import LLM, SamplingParams\n\n# Initialize model\nllm = LLM(model=\"meta-llama\u002FMeta-Llama-3-8B\")\n\n# Configure sampling\nsampling_params = SamplingParams(\n    max_tokens=100,\n    temperature=0.7,\n    top_p=0.9\n)\n\n# Generate\nprompts = [\"Explain quantum computing\"]\noutputs = llm.generate(prompts, sampling_params)\n\nfor output in outputs:\n    print(output.text)\n```\n\n### Serving with trtllm-serve\n\n```bash\n# Start server (automatic model download and compilation)\ntrtllm-serve meta-llama\u002FMeta-Llama-3-8B \\\n    --tp_size 4 \\              # Tensor parallelism (4 GPUs)\n    --max_batch_size 256 \\\n    --max_num_tokens 4096\n\n# Client request\ncurl -X POST http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"model\": \"meta-llama\u002FMeta-Llama-3-8B\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}],\n    \"temperature\": 0.7,\n    \"max_tokens\": 100\n  }'\n```\n\n## Key features\n\n### Performance optimizations\n- **In-flight batching**: Dynamic batching during generation\n- **Paged KV cache**: Efficient memory management\n- **Flash Attention**: Optimized attention kernels\n- **Quantization**: FP8, INT4, FP4 for 2-4× faster inference\n- **CUDA graphs**: Reduced kernel launch overhead\n\n### Parallelism\n- **Tensor parallelism (TP)**: Split model across GPUs\n- **Pipeline parallelism (PP)**: Layer-wise distribution\n- **Expert parallelism**: For Mixture-of-Experts models\n- **Multi-node**: Scale beyond single machine\n\n### Advanced features\n- **Speculative decoding**: Faster generation with draft models\n- **LoRA serving**: Efficient multi-adapter deployment\n- **Disaggregated serving**: Separate prefill and generation\n\n## Common patterns\n\n### Quantized model (FP8)\n\n```python\nfrom tensorrt_llm import LLM\n\n# Load FP8 quantized model (2× faster, 50% memory)\nllm = LLM(\n    model=\"meta-llama\u002FMeta-Llama-3-70B\",\n    dtype=\"fp8\",\n    max_num_tokens=8192\n)\n\n# Inference same as before\noutputs = llm.generate([\"Summarize this article...\"])\n```\n\n### Multi-GPU deployment\n\n```python\n# Tensor parallelism across 8 GPUs\nllm = LLM(\n    model=\"meta-llama\u002FMeta-Llama-3-405B\",\n    tensor_parallel_size=8,\n    dtype=\"fp8\"\n)\n```\n\n### Batch inference\n\n```python\n# Process 100 prompts efficiently\nprompts = [f\"Question {i}: ...\" for i in range(100)]\n\noutputs = llm.generate(\n    prompts,\n    sampling_params=SamplingParams(max_tokens=200)\n)\n\n# Automatic in-flight batching for maximum throughput\n```\n\n## Performance benchmarks\n\n**Meta Llama 3-8B** (H100 GPU):\n- Throughput: 24,000 tokens\u002Fsec\n- Latency: ~10ms per token\n- vs PyTorch: **100× faster**\n\n**Llama 3-70B** (8× A100 80GB):\n- FP8 quantization: 2× faster than FP16\n- Memory: 50% reduction with FP8\n\n## Supported models\n\n- **LLaMA family**: Llama 2, Llama 3, CodeLlama\n- **GPT family**: GPT-2, GPT-J, GPT-NeoX\n- **Qwen**: Qwen, Qwen2, QwQ\n- **DeepSeek**: DeepSeek-V2, DeepSeek-V3\n- **Mixtral**: Mixtral-8x7B, Mixtral-8x22B\n- **Vision**: LLaVA, Phi-3-vision\n- **100+ models** on HuggingFace\n\n## References\n\n- **[Optimization Guide](references\u002Foptimization.md)** - Quantization, batching, KV cache tuning\n- **[Multi-GPU Setup](references\u002Fmulti-gpu.md)** - Tensor\u002Fpipeline parallelism, multi-node\n- **[Serving Guide](references\u002Fserving.md)** - Production deployment, monitoring, autoscaling\n\n## Resources\n\n- **Docs**: https:\u002F\u002Fnvidia.github.io\u002FTensorRT-LLM\u002F\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FTensorRT-LLM\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=tensorrt_llm\n\n\n",{"name":5,"author":24,"license":15,"version":25,"metadata":26,"platforms":40,"description":7,"dependencies":43},"Orchestra Research","1.0.0",{"hermes":27},{"tags":28},[29,30,31,32,33,34,35,36,37,38,39],"Inference Serving","TensorRT-LLM","NVIDIA","Inference Optimization","High Throughput","Low Latency","Production","FP8","INT4","In-Flight Batching","Multi-GPU",[41,42],"linux","macos",[5,44],"torch","main","ff3476b880e3f7eddb4ffe2b62141766402e972b","core-agent\u002Foptional-skills\u002Fmlops\u002Ftensorrt-llm","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80","https:\u002F\u002Faniruddha-adak.vercel.app","I build agentic harnesses and production-grade agent skills. \r\n\r\nContributor :) @genspark-ai @CopilotKit @openclaw @NousResearch  @synthetic-sciences @PriorLabs","2026-11-14 20:45:37",{"source":8,"skillName":5,"commands":54,"prompt":64,"zipAvailable":65},[55,58,61],{"tool":56,"command":57},"npx","npx skills add aniruddhaadak80\u002Fherclew --skill tensorrt-llm",{"tool":59,"command":60},"bunx","bunx skills add aniruddhaadak80\u002Fherclew --skill tensorrt-llm",{"tool":62,"command":63},"pnpm","pnpm dlx skills add aniruddhaadak80\u002Fherclew --skill tensorrt-llm","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Foptional-skills\u002Fmlops\u002Ftensorrt-llm\u002FSKILL.md",true,[67,73,77],{"path":68,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":71,"sourceHtmlUrl":72},"references\u002Fmulti-gpu.md","REFERENCE","参考文档",6699,"https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Foptional-skills\u002Fmlops\u002Ftensorrt-llm\u002Freferences\u002Fmulti-gpu.md",{"path":74,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":75,"sourceHtmlUrl":76},"references\u002Foptimization.md",5613,"https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Foptional-skills\u002Fmlops\u002Ftensorrt-llm\u002Freferences\u002Foptimization.md",{"path":78,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":79,"sourceHtmlUrl":80},"references\u002Fserving.md",9798,"https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Foptional-skills\u002Fmlops\u002Ftensorrt-llm\u002Freferences\u002Fserving.md",[82,91,99,106,113,120],{"skillNo":83,"name":84,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":86,"licenseFileUrl":87,"sourceHtmlUrl":88,"categories":89,"keywords":90,"publishedAt":19},"01M74FJW4M7QZ5KZHYH1W3SYET","skill-creator","Create, edit, audit, tidy, validate, or restructure AgentSkills and SKILL.md files.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fgateway-node\u002Fskills\u002Fsoftware-development\u002Fskill-creator\u002Flicense.txt","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fgateway-node\u002Fskills\u002Fsoftware-development\u002Fskill-creator\u002FSKILL.md",[],[],{"skillNo":92,"name":93,"nameZh":6,"description":94,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":95,"sourceHtmlUrl":96,"categories":97,"keywords":98,"publishedAt":19},"01M74FDY7EK673NW905BRFR3ZD","prose","OpenProse VM skill pack. Activate on any `prose` command, .prose files, or OpenProse mentions; orchestrates multi-agent workflows.","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fgateway-node\u002Fextensions\u002Fopen-prose\u002Fskills\u002Fprose\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fgateway-node\u002Fextensions\u002Fopen-prose\u002Fskills\u002Fprose\u002FSKILL.md",[],[],{"skillNo":100,"name":101,"nameZh":6,"description":102,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":103,"categories":104,"keywords":105,"publishedAt":19},"01M74FAMG647DMCPE12FZE3PKF","writing-plans","Write implementation plans: bite-sized tasks, paths, code.","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Fskills\u002Fsoftware-development\u002Fwriting-plans\u002FSKILL.md",[],[],{"skillNo":107,"name":108,"nameZh":6,"description":109,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":110,"categories":111,"keywords":112,"publishedAt":19},"01M74FAHVK9XY77HSP589QYQYB","test-driven-development","TDD: enforce RED-GREEN-REFACTOR, tests before code.","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Fskills\u002Fsoftware-development\u002Ftest-driven-development\u002FSKILL.md",[],[],{"skillNo":114,"name":115,"nameZh":6,"description":116,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":117,"categories":118,"keywords":119,"publishedAt":19},"01M74FAECYXZ2DQXQSQ44FHZJZ","systematic-debugging","4-phase root cause debugging: understand bugs before fixing.","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Fskills\u002Fsoftware-development\u002Fsystematic-debugging\u002FSKILL.md",[],[],{"skillNo":121,"name":122,"nameZh":6,"description":123,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":124,"categories":125,"keywords":126,"publishedAt":19},"01M74FABCBB1Z2Y5F67699RPX6","subagent-driven-development","Execute plans via delegate_task subagents (2-stage review).","https:\u002F\u002Fgithub.com\u002Faniruddhaadak80\u002Fherclew\u002Fblob\u002Fmain\u002Fcore-agent\u002Fskills\u002Fsoftware-development\u002Fsubagent-driven-development\u002FSKILL.md",[],[]]