[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M73GCJ5F6DF0RYHJEJSBSHM5":3,"skill-same:01M73GCJ5F6DF0RYHJEJSBSHM5":47},{"skillNo":4,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":23,"commitSha":24,"skillDirectory":25,"forksCount":26,"repositoryHtmlUrl":27,"repositoryHomepageUrl":28,"ownerHtmlUrl":29,"ownerBlogUrl":30,"ownerDescription":31,"updatedAt":32,"install":33,"resources":46},"01M73GCJ5F6DF0RYHJEJSBSHM5","tgi-streaming-inference",null,"Horonomos-laps\u002FYana-AI","Horonomos-laps","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F355299122?v=4","01M76B3QPXE88ET2SQPDPVATZT"," Horonomos-laps","01M72ZQRJPCYPSSWT4D00VJR2A",13,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FHoronomos-laps\u002FYana-AI\u002Fblob\u002Fmain\u002Fcore\u002Fskills\u002Ftgi-streaming-inference\u002FSKILL.md",[],[],"2026-11-11 15:11:14",false,"---\nname: tgi-streaming-inference\ndescription: HuggingFace Text Generation Inference (TGI) for production LLM streaming. Continuous batching, SSE token streaming, speculative decoding, quantization (GPTQ\u002FAWQ), and multi-node tensor parallelism. Sources: huggingface\u002Ftext-generation-inference (Apache-2.0).\norigin: yana-ai — synthesized from huggingface\u002Ftext-generation-inference (Apache-2.0)\nlicense: Apache-2.0\nversion: 1.0.0\ncompatibility: yana-ai >= 1.3.51\n---\n\n# \u002Ftgi-streaming-inference\n\n## When to Use\n\n- Production LLM endpoint with token streaming (SSE) and low TTFT\n- Automatic continuous batching: new requests join in-flight batches\n- GPTQ\u002FAWQ quantization: run 70B model on 2×A100 instead of 4×\n- Speculative decoding: draft model speeds up large model generation 2–3×\n\n## Do NOT use for\n\n- CPU inference (use [[llama-cpp-quantization]])\n- Multi-model serving (use [[triton-inference-server]])\n\n---\n\n## Docker deployment\n\n```bash\ndocker run --gpus all \\\n  -p 8080:80 \\\n  -v $HOME\u002Fmodels:\u002Fdata \\\n  ghcr.io\u002Fhuggingface\u002Ftext-generation-inference:latest \\\n    --model-id          meta-llama\u002FLlama-3-8B-Instruct \\\n    --quantize          bitsandbytes-nf4   \\   # or gptq \u002F awq\n    --max-input-length  4096               \\\n    --max-total-tokens  8192               \\\n    --max-batch-prefill-tokens 4096        \\\n    --num-shard         1                      # tensor parallel shards = GPUs\n```\n\n---\n\n## SSE token streaming (Node.js)\n\n```javascript\nasync function* streamTGI(prompt: string, params = {}) {\n  const res = await fetch('http:\u002F\u002Flocalhost:8080\u002Fgenerate_stream', {\n    method:  'POST',\n    headers: { 'Content-Type': 'application\u002Fjson' },\n    body:    JSON.stringify({\n      inputs: prompt,\n      parameters: {\n        max_new_tokens:  512,\n        temperature:     0.1,\n        repetition_penalty: 1.1,\n        do_sample:       true,\n        ...params,\n      },\n    }),\n  })\n\n  if (!res.ok) throw new Error(`[tgi] ${res.status}`)\n\n  const reader = res.body!.getReader()\n  const dec    = new TextDecoder()\n\n  while (true) {\n    const { done, value } = await reader.read()\n    if (done) break\n    const lines = dec.decode(value).split('\\n').filter(l => l.startsWith('data:'))\n    for (const line of lines) {\n      const event = JSON.parse(line.slice(5))\n      if (event.token?.text) yield event.token.text\n      if (event.generated_text) return  \u002F\u002F final event\n    }\n  }\n}\n\n\u002F\u002F Usage\nfor await (const token of streamTGI('Explain PagedAttention in 3 lines:')) {\n  process.stdout.write(token)\n}\n```\n\n---\n\n## Speculative decoding config\n\n```bash\n# Use a small draft model to propose tokens, large model to verify\ndocker run --gpus all \\\n  ghcr.io\u002Fhuggingface\u002Ftext-generation-inference:latest \\\n    --model-id          meta-llama\u002FLlama-3-70B-Instruct \\\n    --speculate         4    \\   # draft 4 tokens at a time\n    --num-shard         4\n# Throughput improvement: ~2.5× for typical instruction-following workloads\n```\n\n---\n\n## Quantization comparison (70B model)\n\n```\nMethod        VRAM     Speed     Quality loss\n──────────────────────────────────────────────\nfp16          140 GB   1.0×      baseline\nbfloat16      140 GB   1.0×      near-zero\nGPTQ-4bit      35 GB   1.3×      ~1–2 perplexity pts\nAWQ-4bit       35 GB   1.4×      ~0.5 perplexity pts ← best 4-bit\nNF4\u002Fbitsandbytes 35 GB 0.9×     ~1 perplexity pt (slower than GPTQ)\n```\n\n---\n\n## Anti-Fake-Pass Checklist\n\n```\n❌ max-total-tokens \u003C max-input-length + expected output → requests rejected\n❌ --num-shard > number of GPUs → TGI fails to start\n❌ Streaming without handling SSE 'error' events → silent truncation on OOM\n❌ temperature: 0 with do_sample: true → nonsensical; use do_sample: false for greedy\n❌ GPTQ model without matching --quantize gptq flag → wrong kernel, wrong output\n❌ No Content-Type: application\u002Fjson → TGI returns 415, not parsed as JSON error\n```\n","# \u002Ftgi-streaming-inference\n\n## When to Use\n\n- Production LLM endpoint with token streaming (SSE) and low TTFT\n- Automatic continuous batching: new requests join in-flight batches\n- GPTQ\u002FAWQ quantization: run 70B model on 2×A100 instead of 4×\n- Speculative decoding: draft model speeds up large model generation 2–3×\n\n## Do NOT use for\n\n- CPU inference (use [[llama-cpp-quantization]])\n- Multi-model serving (use [[triton-inference-server]])\n\n---\n\n## Docker deployment\n\n```bash\ndocker run --gpus all \\\n  -p 8080:80 \\\n  -v $HOME\u002Fmodels:\u002Fdata \\\n  ghcr.io\u002Fhuggingface\u002Ftext-generation-inference:latest \\\n    --model-id          meta-llama\u002FLlama-3-8B-Instruct \\\n    --quantize          bitsandbytes-nf4   \\   # or gptq \u002F awq\n    --max-input-length  4096               \\\n    --max-total-tokens  8192               \\\n    --max-batch-prefill-tokens 4096        \\\n    --num-shard         1                      # tensor parallel shards = GPUs\n```\n\n---\n\n## SSE token streaming (Node.js)\n\n```javascript\nasync function* streamTGI(prompt: string, params = {}) {\n  const res = await fetch('http:\u002F\u002Flocalhost:8080\u002Fgenerate_stream', {\n    method:  'POST',\n    headers: { 'Content-Type': 'application\u002Fjson' },\n    body:    JSON.stringify({\n      inputs: prompt,\n      parameters: {\n        max_new_tokens:  512,\n        temperature:     0.1,\n        repetition_penalty: 1.1,\n        do_sample:       true,\n        ...params,\n      },\n    }),\n  })\n\n  if (!res.ok) throw new Error(`[tgi] ${res.status}`)\n\n  const reader = res.body!.getReader()\n  const dec    = new TextDecoder()\n\n  while (true) {\n    const { done, value } = await reader.read()\n    if (done) break\n    const lines = dec.decode(value).split('\\n').filter(l => l.startsWith('data:'))\n    for (const line of lines) {\n      const event = JSON.parse(line.slice(5))\n      if (event.token?.text) yield event.token.text\n      if (event.generated_text) return  \u002F\u002F final event\n    }\n  }\n}\n\n\u002F\u002F Usage\nfor await (const token of streamTGI('Explain PagedAttention in 3 lines:')) {\n  process.stdout.write(token)\n}\n```\n\n---\n\n## Speculative decoding config\n\n```bash\n# Use a small draft model to propose tokens, large model to verify\ndocker run --gpus all \\\n  ghcr.io\u002Fhuggingface\u002Ftext-generation-inference:latest \\\n    --model-id          meta-llama\u002FLlama-3-70B-Instruct \\\n    --speculate         4    \\   # draft 4 tokens at a time\n    --num-shard         4\n# Throughput improvement: ~2.5× for typical instruction-following workloads\n```\n\n---\n\n## Quantization comparison (70B model)\n\n```\nMethod        VRAM     Speed     Quality loss\n──────────────────────────────────────────────\nfp16          140 GB   1.0×      baseline\nbfloat16      140 GB   1.0×      near-zero\nGPTQ-4bit      35 GB   1.3×      ~1–2 perplexity pts\nAWQ-4bit       35 GB   1.4×      ~0.5 perplexity pts ← best 4-bit\nNF4\u002Fbitsandbytes 35 GB 0.9×     ~1 perplexity pt (slower than GPTQ)\n```\n\n---\n\n## Anti-Fake-Pass Checklist\n\n```\n❌ max-total-tokens \u003C max-input-length + expected output → requests rejected\n❌ --num-shard > number of GPUs → TGI fails to start\n❌ Streaming without handling SSE 'error' events → silent truncation on OOM\n❌ temperature: 0 with do_sample: true → nonsensical; use do_sample: false for greedy\n❌ GPTQ model without matching --quantize gptq flag → wrong kernel, wrong output\n❌ No Content-Type: application\u002Fjson → TGI returns 415, not parsed as JSON error\n```\n",{},"main","481dcd7dd2e288b9d627a2bd017db70e64e6c6fa","core\u002Fskills\u002Ftgi-streaming-inference",1,"https:\u002F\u002Fgithub.com\u002FHoronomos-laps\u002FYana-AI","https:\u002F\u002Fyana.vutam.link\u002F","https:\u002F\u002Fgithub.com\u002FHoronomos-laps","https:\u002F\u002Fyana.vutam.link","An open-source governed runtime & execution environment for multi-agent AI systems.","2026-11-11 23:56:26",{"source":7,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add Horonomos-laps\u002FYana-AI --skill tgi-streaming-inference",{"tool":39,"command":40},"bunx","bunx skills add Horonomos-laps\u002FYana-AI --skill tgi-streaming-inference",{"tool":42,"command":43},"pnpm","pnpm dlx skills add Horonomos-laps\u002FYana-AI --skill tgi-streaming-inference","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FHoronomos-laps\u002FYana-AI\u002Fblob\u002Fmain\u002Fcore\u002Fskills\u002Ftgi-streaming-inference\u002FSKILL.md",true,[],[48,56,63,71,79,87],{"skillNo":49,"name":50,"nameZh":6,"description":51,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":52,"licenseFileUrl":6,"sourceHtmlUrl":53,"categories":54,"keywords":55,"publishedAt":18},"01M735GKET5YVYKAY62W2P5YXB","openai--sentry--sentry","Use when the user asks to inspect Sentry issues or events, summarize recent production errors, or pull basic Sentry health data via the Sentry API; perform read-only queries with the bundled script and require `SENTRY_AUTH_TOKEN`.","MIT","https:\u002F\u002Fgithub.com\u002FHoronomos-laps\u002FYana-AI\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fopenai--sentry--sentry\u002FSKILL.md",[],[],{"skillNo":57,"name":58,"nameZh":6,"description":59,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":60,"categories":61,"keywords":62,"publishedAt":18},"01M73DCAP83T17HY7SMGK7MGFE","ito-basket-compare","Compare Itô prediction-market baskets against a user's knowledge base, portfolio notes, financial context, watchlist, or research thesis. Use for read-only basket comparison and gap analysis without investment advice or live trading.","https:\u002F\u002Fgithub.com\u002FHoronomos-laps\u002FYana-AI\u002Fblob\u002Fmain\u002Fcore\u002Fskills\u002Fito-basket-compare\u002FSKILL.md",[],[],{"skillNo":64,"name":65,"nameZh":6,"description":66,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":67,"sourceHtmlUrl":68,"categories":69,"keywords":70,"publishedAt":18},"01M734CYEMZFDR3KH6SW0JP8TD","implementing-network-traffic-baselining","Build network traffic baselines from NetFlow\u002FIPFIX data using Python pandas for statistical analysis, z-score anomaly detection, and hourly\u002Fdaily traffic pattern profiling","https:\u002F\u002Fgithub.com\u002Fyanacuti1121\u002FYana-AI\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fimplementing-network-traffic-baselining\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FHoronomos-laps\u002FYana-AI\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fimplementing-network-traffic-baselining\u002FSKILL.md",[],[],{"skillNo":72,"name":73,"nameZh":6,"description":74,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":75,"sourceHtmlUrl":76,"categories":77,"keywords":78,"publishedAt":18},"01M73ASZ6NJRJM44Z2P84NGQ3Q","detecting-evasion-techniques-in-endpoint-logs","Detects defense evasion techniques used by adversaries in endpoint logs including log tampering, timestomping, process injection, and security tool disabling. Use when investigating suspicious endpoint behavior, building detection rules for evasion tactics, or conducting threat hunting for stealthy adversary activity. Activates for requests involving evasion detection, defense evasion analysis, log tampering detection, or MITRE ATT&CK TA0005.","https:\u002F\u002Fgithub.com\u002Fyanacuti1121\u002FYana-AI\u002Fblob\u002Fmain\u002Fcore\u002Fskills\u002Fdetecting-evasion-techniques-in-endpoint-logs\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FHoronomos-laps\u002FYana-AI\u002Fblob\u002Fmain\u002Fcore\u002Fskills\u002Fdetecting-evasion-techniques-in-endpoint-logs\u002FSKILL.md",[],[],{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":83,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":18},"01M734AFNXD06Z78QRK0VKGT7Q","implementing-microsegmentation-with-guardicore","Implementing microsegmentation using Akamai Guardicore Segmentation to map application dependencies, create granular network policies, visualize east-west traffic flows, and enforce least-privilege communication between workloads across data centers and cloud.","https:\u002F\u002Fgithub.com\u002Fyanacuti1121\u002FYana-AI\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fimplementing-microsegmentation-with-guardicore\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FHoronomos-laps\u002FYana-AI\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fimplementing-microsegmentation-with-guardicore\u002FSKILL.md",[],[],{"skillNo":88,"name":89,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":18},"01M736TK27QMCXK0DVCH25SJN5","ramdisk-tmpfs-patterns","https:\u002F\u002Fgithub.com\u002FHoronomos-laps\u002FYana-AI\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Framdisk-tmpfs-patterns\u002FSKILL.md",[],[]]