tgi-streaming-inference

/tgi-streaming-inference

When to Use

• Production LLM endpoint with token streaming (SSE) and low TTFT
• Automatic continuous batching: new requests join in-flight batches
• GPTQ/AWQ quantization: run 70B model on 2×A100 instead of 4×
• Speculative decoding: draft model speeds up large model generation 2–3×

Do NOT use for

• CPU inference (use [[llama-cpp-quantization]])
• Multi-model serving (use [[triton-inference-server]])

---

Docker deployment

docker run --gpus all \
  -p 8080:80 \
  -v $HOME/models:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
    --model-id          meta-llama/Llama-3-8B-Instruct \
    --quantize          bitsandbytes-nf4   \   # or gptq / awq
    --max-input-length  4096               \
    --max-total-tokens  8192               \
    --max-batch-prefill-tokens 4096        \
    --num-shard         1                      # tensor parallel shards = GPUs

---

SSE token streaming (Node.js)

async function* streamTGI(prompt: string, params = {}) {
  const res = await fetch('http://localhost:8080/generate_stream', {
    method:  'POST',
    headers: { 'Content-Type': 'application/json' },
    body:    JSON.stringify({
      inputs: prompt,
      parameters: {
        max_new_tokens:  512,
        temperature:     0.1,
        repetition_penalty: 1.1,
        do_sample:       true,
        ...params,
      },
    }),
  })

  if (!res.ok) throw new Error(`[tgi] ${res.status}`)

  const reader = res.body!.getReader()
  const dec    = new TextDecoder()

  while (true) {
    const { done, value } = await reader.read()
    if (done) break
    const lines = dec.decode(value).split('\n').filter(l => l.startsWith('data:'))
    for (const line of lines) {
      const event = JSON.parse(line.slice(5))
      if (event.token?.text) yield event.token.text
      if (event.generated_text) return  // final event
    }
  }
}

// Usage
for await (const token of streamTGI('Explain PagedAttention in 3 lines:')) {
  process.stdout.write(token)
}

---

Speculative decoding config

# Use a small draft model to propose tokens, large model to verify
docker run --gpus all \
  ghcr.io/huggingface/text-generation-inference:latest \
    --model-id          meta-llama/Llama-3-70B-Instruct \
    --speculate         4    \   # draft 4 tokens at a time
    --num-shard         4
# Throughput improvement: ~2.5× for typical instruction-following workloads

---

Quantization comparison (70B model)

Method        VRAM     Speed     Quality loss
──────────────────────────────────────────────
fp16          140 GB   1.0×      baseline
bfloat16      140 GB   1.0×      near-zero
GPTQ-4bit      35 GB   1.3×      ~1–2 perplexity pts
AWQ-4bit       35 GB   1.4×      ~0.5 perplexity pts ← best 4-bit
NF4/bitsandbytes 35 GB 0.9×     ~1 perplexity pt (slower than GPTQ)

---

Anti-Fake-Pass Checklist

❌ max-total-tokens < max-input-length + expected output → requests rejected
❌ --num-shard > number of GPUs → TGI fails to start
❌ Streaming without handling SSE 'error' events → silent truncation on OOM
❌ temperature: 0 with do_sample: true → nonsensical; use do_sample: false for greedy
❌ GPTQ model without matching --quantize gptq flag → wrong kernel, wrong output
❌ No Content-Type: application/json → TGI returns 415, not parsed as JSON error