[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M758YYQTM9D2SQX812APBW55":3,"skill-same:01M758YYQTM9D2SQX812APBW55":80},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":40,"commitSha":41,"skillDirectory":42,"forksCount":14,"repositoryHtmlUrl":43,"repositoryHomepageUrl":6,"ownerHtmlUrl":44,"ownerBlogUrl":45,"ownerDescription":46,"updatedAt":47,"install":48,"resources":61},"01M758YYQTM9D2SQX812APBW55","tensorrt-llm",null,"Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100\u002FH100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8\u002FINT4), in-flight batching, and multi-GPU scaling.","zengrong233\u002Fdevin-skills","zengrong233","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F55529920?v=4","01M758DNYKQQCQX87XFVAGX5MW","songzengrong","01M758DNZVEJJQNM71DSRVHFEB",0,"MIT","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Ftensorrt-llm\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: tensorrt-llm\ndescription: Optimizes LLM inference with NVIDIA TensorRT for maximum throughput and lowest latency. Use for production deployment on NVIDIA GPUs (A100\u002FH100), when you need 10-100x faster inference than PyTorch, or for serving models with quantization (FP8\u002FINT4), in-flight batching, and multi-GPU scaling.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Inference Serving, TensorRT-LLM, NVIDIA, Inference Optimization, High Throughput, Low Latency, Production, FP8, INT4, In-Flight Batching, Multi-GPU]\ndependencies: [tensorrt-llm, torch]\n---\n\n# TensorRT-LLM\n\nNVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs.\n\n## When to use TensorRT-LLM\n\n**Use TensorRT-LLM when:**\n- Deploying on NVIDIA GPUs (A100, H100, GB200)\n- Need maximum throughput (24,000+ tokens\u002Fsec on Llama 3)\n- Require low latency for real-time applications\n- Working with quantized models (FP8, INT4, FP4)\n- Scaling across multiple GPUs or nodes\n\n**Use vLLM instead when:**\n- Need simpler setup and Python-first API\n- Want PagedAttention without TensorRT compilation\n- Working with AMD GPUs or non-NVIDIA hardware\n\n**Use llama.cpp instead when:**\n- Deploying on CPU or Apple Silicon\n- Need edge deployment without NVIDIA GPUs\n- Want simpler GGUF quantization format\n\n## Quick start\n\n### Installation\n\n```bash\n# Docker (recommended)\ndocker pull nvidia\u002Ftensorrt_llm:latest\n\n# pip install\npip install tensorrt_llm==1.2.0rc3\n\n# Requires CUDA 13.0.0, TensorRT 10.13.2, Python 3.10-3.12\n```\n\n### Basic inference\n\n```python\nfrom tensorrt_llm import LLM, SamplingParams\n\n# Initialize model\nllm = LLM(model=\"meta-llama\u002FMeta-Llama-3-8B\")\n\n# Configure sampling\nsampling_params = SamplingParams(\n    max_tokens=100,\n    temperature=0.7,\n    top_p=0.9\n)\n\n# Generate\nprompts = [\"Explain quantum computing\"]\noutputs = llm.generate(prompts, sampling_params)\n\nfor output in outputs:\n    print(output.text)\n```\n\n### Serving with trtllm-serve\n\n```bash\n# Start server (automatic model download and compilation)\ntrtllm-serve meta-llama\u002FMeta-Llama-3-8B \\\n    --tp_size 4 \\              # Tensor parallelism (4 GPUs)\n    --max_batch_size 256 \\\n    --max_num_tokens 4096\n\n# Client request\ncurl -X POST http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"model\": \"meta-llama\u002FMeta-Llama-3-8B\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}],\n    \"temperature\": 0.7,\n    \"max_tokens\": 100\n  }'\n```\n\n## Key features\n\n### Performance optimizations\n- **In-flight batching**: Dynamic batching during generation\n- **Paged KV cache**: Efficient memory management\n- **Flash Attention**: Optimized attention kernels\n- **Quantization**: FP8, INT4, FP4 for 2-4× faster inference\n- **CUDA graphs**: Reduced kernel launch overhead\n\n### Parallelism\n- **Tensor parallelism (TP)**: Split model across GPUs\n- **Pipeline parallelism (PP)**: Layer-wise distribution\n- **Expert parallelism**: For Mixture-of-Experts models\n- **Multi-node**: Scale beyond single machine\n\n### Advanced features\n- **Speculative decoding**: Faster generation with draft models\n- **LoRA serving**: Efficient multi-adapter deployment\n- **Disaggregated serving**: Separate prefill and generation\n\n## Common patterns\n\n### Quantized model (FP8)\n\n```python\nfrom tensorrt_llm import LLM\n\n# Load FP8 quantized model (2× faster, 50% memory)\nllm = LLM(\n    model=\"meta-llama\u002FMeta-Llama-3-70B\",\n    dtype=\"fp8\",\n    max_num_tokens=8192\n)\n\n# Inference same as before\noutputs = llm.generate([\"Summarize this article...\"])\n```\n\n### Multi-GPU deployment\n\n```python\n# Tensor parallelism across 8 GPUs\nllm = LLM(\n    model=\"meta-llama\u002FMeta-Llama-3-405B\",\n    tensor_parallel_size=8,\n    dtype=\"fp8\"\n)\n```\n\n### Batch inference\n\n```python\n# Process 100 prompts efficiently\nprompts = [f\"Question {i}: ...\" for i in range(100)]\n\noutputs = llm.generate(\n    prompts,\n    sampling_params=SamplingParams(max_tokens=200)\n)\n\n# Automatic in-flight batching for maximum throughput\n```\n\n## Performance benchmarks\n\n**Meta Llama 3-8B** (H100 GPU):\n- Throughput: 24,000 tokens\u002Fsec\n- Latency: ~10ms per token\n- vs PyTorch: **100× faster**\n\n**Llama 3-70B** (8× A100 80GB):\n- FP8 quantization: 2× faster than FP16\n- Memory: 50% reduction with FP8\n\n## Supported models\n\n- **LLaMA family**: Llama 2, Llama 3, CodeLlama\n- **GPT family**: GPT-2, GPT-J, GPT-NeoX\n- **Qwen**: Qwen, Qwen2, QwQ\n- **DeepSeek**: DeepSeek-V2, DeepSeek-V3\n- **Mixtral**: Mixtral-8x7B, Mixtral-8x22B\n- **Vision**: LLaVA, Phi-3-vision\n- **100+ models** on HuggingFace\n\n## References\n\n- **[Optimization Guide](references\u002Foptimization.md)** - Quantization, batching, KV cache tuning\n- **[Multi-GPU Setup](references\u002Fmulti-gpu.md)** - Tensor\u002Fpipeline parallelism, multi-node\n- **[Serving Guide](references\u002Fserving.md)** - Production deployment, monitoring, autoscaling\n\n## Resources\n\n- **Docs**: https:\u002F\u002Fnvidia.github.io\u002FTensorRT-LLM\u002F\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FTensorRT-LLM\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=tensorrt_llm\n\n\n","# TensorRT-LLM\n\nNVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs.\n\n## When to use TensorRT-LLM\n\n**Use TensorRT-LLM when:**\n- Deploying on NVIDIA GPUs (A100, H100, GB200)\n- Need maximum throughput (24,000+ tokens\u002Fsec on Llama 3)\n- Require low latency for real-time applications\n- Working with quantized models (FP8, INT4, FP4)\n- Scaling across multiple GPUs or nodes\n\n**Use vLLM instead when:**\n- Need simpler setup and Python-first API\n- Want PagedAttention without TensorRT compilation\n- Working with AMD GPUs or non-NVIDIA hardware\n\n**Use llama.cpp instead when:**\n- Deploying on CPU or Apple Silicon\n- Need edge deployment without NVIDIA GPUs\n- Want simpler GGUF quantization format\n\n## Quick start\n\n### Installation\n\n```bash\n# Docker (recommended)\ndocker pull nvidia\u002Ftensorrt_llm:latest\n\n# pip install\npip install tensorrt_llm==1.2.0rc3\n\n# Requires CUDA 13.0.0, TensorRT 10.13.2, Python 3.10-3.12\n```\n\n### Basic inference\n\n```python\nfrom tensorrt_llm import LLM, SamplingParams\n\n# Initialize model\nllm = LLM(model=\"meta-llama\u002FMeta-Llama-3-8B\")\n\n# Configure sampling\nsampling_params = SamplingParams(\n    max_tokens=100,\n    temperature=0.7,\n    top_p=0.9\n)\n\n# Generate\nprompts = [\"Explain quantum computing\"]\noutputs = llm.generate(prompts, sampling_params)\n\nfor output in outputs:\n    print(output.text)\n```\n\n### Serving with trtllm-serve\n\n```bash\n# Start server (automatic model download and compilation)\ntrtllm-serve meta-llama\u002FMeta-Llama-3-8B \\\n    --tp_size 4 \\              # Tensor parallelism (4 GPUs)\n    --max_batch_size 256 \\\n    --max_num_tokens 4096\n\n# Client request\ncurl -X POST http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"model\": \"meta-llama\u002FMeta-Llama-3-8B\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}],\n    \"temperature\": 0.7,\n    \"max_tokens\": 100\n  }'\n```\n\n## Key features\n\n### Performance optimizations\n- **In-flight batching**: Dynamic batching during generation\n- **Paged KV cache**: Efficient memory management\n- **Flash Attention**: Optimized attention kernels\n- **Quantization**: FP8, INT4, FP4 for 2-4× faster inference\n- **CUDA graphs**: Reduced kernel launch overhead\n\n### Parallelism\n- **Tensor parallelism (TP)**: Split model across GPUs\n- **Pipeline parallelism (PP)**: Layer-wise distribution\n- **Expert parallelism**: For Mixture-of-Experts models\n- **Multi-node**: Scale beyond single machine\n\n### Advanced features\n- **Speculative decoding**: Faster generation with draft models\n- **LoRA serving**: Efficient multi-adapter deployment\n- **Disaggregated serving**: Separate prefill and generation\n\n## Common patterns\n\n### Quantized model (FP8)\n\n```python\nfrom tensorrt_llm import LLM\n\n# Load FP8 quantized model (2× faster, 50% memory)\nllm = LLM(\n    model=\"meta-llama\u002FMeta-Llama-3-70B\",\n    dtype=\"fp8\",\n    max_num_tokens=8192\n)\n\n# Inference same as before\noutputs = llm.generate([\"Summarize this article...\"])\n```\n\n### Multi-GPU deployment\n\n```python\n# Tensor parallelism across 8 GPUs\nllm = LLM(\n    model=\"meta-llama\u002FMeta-Llama-3-405B\",\n    tensor_parallel_size=8,\n    dtype=\"fp8\"\n)\n```\n\n### Batch inference\n\n```python\n# Process 100 prompts efficiently\nprompts = [f\"Question {i}: ...\" for i in range(100)]\n\noutputs = llm.generate(\n    prompts,\n    sampling_params=SamplingParams(max_tokens=200)\n)\n\n# Automatic in-flight batching for maximum throughput\n```\n\n## Performance benchmarks\n\n**Meta Llama 3-8B** (H100 GPU):\n- Throughput: 24,000 tokens\u002Fsec\n- Latency: ~10ms per token\n- vs PyTorch: **100× faster**\n\n**Llama 3-70B** (8× A100 80GB):\n- FP8 quantization: 2× faster than FP16\n- Memory: 50% reduction with FP8\n\n## Supported models\n\n- **LLaMA family**: Llama 2, Llama 3, CodeLlama\n- **GPT family**: GPT-2, GPT-J, GPT-NeoX\n- **Qwen**: Qwen, Qwen2, QwQ\n- **DeepSeek**: DeepSeek-V2, DeepSeek-V3\n- **Mixtral**: Mixtral-8x7B, Mixtral-8x22B\n- **Vision**: LLaVA, Phi-3-vision\n- **100+ models** on HuggingFace\n\n## References\n\n- **[Optimization Guide](references\u002Foptimization.md)** - Quantization, batching, KV cache tuning\n- **[Multi-GPU Setup](references\u002Fmulti-gpu.md)** - Tensor\u002Fpipeline parallelism, multi-node\n- **[Serving Guide](references\u002Fserving.md)** - Production deployment, monitoring, autoscaling\n\n## Resources\n\n- **Docs**: https:\u002F\u002Fnvidia.github.io\u002FTensorRT-LLM\u002F\n- **GitHub**: https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FTensorRT-LLM\n- **Models**: https:\u002F\u002Fhuggingface.co\u002Fmodels?library=tensorrt_llm\n\n\n",{"name":5,"tags":24,"author":36,"license":15,"version":37,"description":7,"dependencies":38},[25,26,27,28,29,30,31,32,33,34,35],"Inference Serving","TensorRT-LLM","NVIDIA","Inference Optimization","High Throughput","Low Latency","Production","FP8","INT4","In-Flight Batching","Multi-GPU","Orchestra Research","1.0.0",[5,39],"torch","skills","4fb202b97f2676132ac8713dbd2370cefacacb81",".agents\u002Fskills\u002Ftensorrt-llm","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills","https:\u002F\u002Fgithub.com\u002Fzengrong233","https:\u002F\u002Fwtu.edu.cn\u002F","my name songzengrong\r\n  research interest's:NLP","2026-06-04 10:48:22",{"source":8,"skillName":5,"commands":49,"prompt":59,"zipAvailable":60},[50,53,56],{"tool":51,"command":52},"npx","npx skills add zengrong233\u002Fdevin-skills --skill tensorrt-llm",{"tool":54,"command":55},"bunx","bunx skills add zengrong233\u002Fdevin-skills --skill tensorrt-llm",{"tool":57,"command":58},"pnpm","pnpm dlx skills add zengrong233\u002Fdevin-skills --skill tensorrt-llm","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Ftensorrt-llm\u002FSKILL.md",true,[62,68,72,76],{"path":63,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":66,"sourceHtmlUrl":67},".openskills.json","REFERENCE","参考文档",239,"https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Ftensorrt-llm\u002F.openskills.json",{"path":69,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":70,"sourceHtmlUrl":71},"references\u002Fmulti-gpu.md",6699,"https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Ftensorrt-llm\u002Freferences\u002Fmulti-gpu.md",{"path":73,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Foptimization.md",5613,"https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Ftensorrt-llm\u002Freferences\u002Foptimization.md",{"path":77,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":78,"sourceHtmlUrl":79},"references\u002Fserving.md",9798,"https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Ftensorrt-llm\u002Freferences\u002Fserving.md",[81,88,95,104,112,119],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M759188DV0S9ARY1ETHCEY45","whisper","OpenAI's general-purpose speech recognition model. Supports 99 languages, transcription, translation to English, and language identification. Six model sizes from tiny (39M params) to large (1550M params). Use for speech-to-text, podcast transcription, or multilingual audio processing. Best for robust, multilingual ASR.","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fwhisper\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M75913ZB4EDVWYGYRTRWTKD1","weights-and-biases","Track ML experiments with automatic logging, visualize training in real-time, optimize hyperparameters with sweeps, and manage model registry with W&B - collaborative MLOps platform","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fweights-and-biases\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":99,"licenseFileUrl":100,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":19},"01M75910YNW7H0KZQDR96TGWZ9","webapp-testing","Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fwebapp-testing\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fwebapp-testing\u002FSKILL.md",[],[],{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":99,"licenseFileUrl":108,"sourceHtmlUrl":109,"categories":110,"keywords":111,"publishedAt":19},"01M7590X6XJR78KVSHCVFYB59A","web-artifacts-builder","Suite of tools for creating elaborate, multi-component claude.ai HTML artifacts using modern frontend web technologies (React, Tailwind CSS, shadcn\u002Fui). Use for complex artifacts requiring state management, routing, or shadcn\u002Fui components - not for simple single-file HTML\u002FJSX artifacts.","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fweb-artifacts-builder\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fweb-artifacts-builder\u002FSKILL.md",[],[],{"skillNo":113,"name":114,"nameZh":6,"description":115,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":116,"categories":117,"keywords":118,"publishedAt":19},"01M7590RENMJJTXV44P8G553ZQ","serving-llms-vllm","Serves LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency\u002Fthroughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ\u002FAWQ\u002FFP8), and tensor parallelism.","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fvllm\u002FSKILL.md",[],[],{"skillNo":120,"name":121,"nameZh":6,"description":122,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":123,"categories":124,"keywords":125,"publishedAt":19},"01M7590GA6EZKJA8GAAMMPAXBB","verl-rl-training","Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.","https:\u002F\u002Fgithub.com\u002Fzengrong233\u002Fdevin-skills\u002Fblob\u002Fskills\u002F.agents\u002Fskills\u002Fverl\u002FSKILL.md",[],[]]