AGENT SKILL

groq-guide

Ultra-low-latency inference on Groq's LPU hardware — when tokens-per-second is the product requirement.

License 许可
MIT
所属仓库
TanNhatCMS/jarvis
最近更新
2026-11-07
Stars
★ 0
Forks
1

01 前言 Front Matter

name
groq-guide
category
ai-research
description
Ultra-low-latency inference on Groq's LPU hardware — when tokens-per-second is the product requirement.

03 SKILL.md 正文

Overview

Groq runs language models on its own custom silicon — Language Processing Units (LPUs) — delivering exceptionally high tokens-per-second and low latency for supported open models. The value proposition is simple and sharp: when speed is the binding constraint, Groq's hardware-tuned inference is among the fastest available via API.

The tradeoff is model selection: Groq supports a curated set of models compiled for its hardware, not the full open-model universe. You trade breadth for velocity. For applications where latency defines the experience — voice agents, real-time copilots, interactive tools, agents making many sequential calls — that trade is often worth it.

Use Groq when you've measured your latency requirements and they demand hardware-level speed. It's a specialist tool: unmatched at its specialty, narrow outside it.

When to use

  • Real-time applications: voice agents, live copilots, interactive chat where latency is the UX.
  • Agent loops with many sequential model calls — per-call latency compounds.
  • High-throughput batch processing where tokens/sec per dollar wins.
  • Demos and prototypes where responsiveness sells the experience.
  • Any workload where you can use a Groq-supported model and speed is the top requirement.
  • Latency-sensitive evals and benchmarking.

Core concepts

  • LPU hardware: purpose-built chips for LLM inference — deterministic, high-throughput execution. The speed comes from hardware/software co-design, not just optimization tricks.
  • Curated model set: models compiled and supported for LPU execution. Check current availability — the set grows over time but remains curated, not comprehensive.
  • Tokens-per-second leadership: the headline metric — Groq typically leads on raw generation speed for supported models. Validate on your prompt shapes.
  • Low time-to-first-token: fast first tokens matter for interactive UX; Groq's architecture delivers here too. Measure TTFT for your use case.
  • API compatibility: OpenAI-compatible API surface — swapping in Groq is usually a base-URL and key change, making evaluation cheap.
  • Rate limits and tiers: generous free tiers for development; paid tiers for production throughput. Understand the rate limits for your expected concurrency.
  • Deterministic performance: LPU execution is highly deterministic — less latency variance than GPU-based serving. Useful for SLO planning.
  • Model versioning: supported models update over time; pin what's pinnable and re-validate on changes.

Practical workflow

  1. Confirm model availability. Check that a model meeting your quality bar is supported on Groq. If none is, Groq's speed doesn't help — quality bar first.
  2. Benchmark your workload. Measure TTFT, tokens/sec, and end-to-end latency on your actual prompts. Confirm the speed advantage holds for your shapes (long contexts, structured outputs).
  3. Evaluate quality parity. Run your eval set — ensure the Groq-served model meets your quality bar. Hardware-optimized serving should be behavior-preserving; verify, don't assume.
  4. Design for the speed. Ultra-low latency enables new UX: streaming voice, real-time collaboration, rapid agent loops. Don't just port a slow design — exploit the speed.
  5. Plan rate limits. Map your production concurrency to Groq's rate limits and tiers. Load-test at expected peak; arrange capacity before launch.
  6. Build fallbacks. For production, configure fallback providers — speed specialists are still single points of failure.
  7. Monitor latency in production. Track TTFT and tokens/sec on live traffic; alert on regressions. Deterministic hardware should give stable numbers — instability signals a problem.

Checklist for Groq in production:

  • Supported model clears your quality bar on your evals.
  • Latency benchmarked on your prompts (TTFT + throughput).
  • Rate limits mapped to production concurrency.
  • Fallback providers configured.
  • Production latency monitored with alerts.

Common pitfalls

  • Speed without a quality bar. Choosing Groq's fastest model instead of the slowest model that meets your quality needs. Quality gates the choice; speed ranks among qualifiers.
  • Model set mismatch. Designing around a model Groq doesn't support, then compromising quality to get the speed. Check availability first.
  • Ignoring rate limits. Free-tier limits in development masking production needs. Plan capacity for launch traffic, not dev traffic.
  • No fallback. Groq as a single point of failure for a latency-critical product. Incidents happen to everyone.
  • Over-engineering for speed you don't need. Batch/offline workloads paying a premium (in model choice constraints) for latency they can't use. Match the tool to the requirement.
  • Assuming quality parity. Hardware-optimized inference is usually behavior-preserving, but "usually" isn't a guarantee — run your evals.
  • Long-context surprises. Speed advantages can narrow at very long contexts. Benchmark at your actual context lengths.
  • Latency SLOs without monitoring. Assuming deterministic hardware means no monitoring needed. Monitor anyway — networks, queues, and rate limits still vary.

04 同仓库更多 Skills

TanNhatCMS
TanNhatCMS/jarvis

web-scrape

Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open/fetch fails on JS-heavy pages, or when you need clean markdown/structured extraction from a site. Requires FIRECRAWL_API_KEY.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

design-taste-frontend

tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML/CSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, "AI-made", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

test-driven-development

Use when implementing any feature or bugfix, before writing implementation code

★ 0
TanNhatCMS
TanNhatCMS/jarvis

systematic-debugging

Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes

★ 0
TanNhatCMS
TanNhatCMS/jarvis

requesting-code-review

Use when completing tasks, implementing major features, or before merging to verify work meets requirements

★ 0
TanNhatCMS
TanNhatCMS/jarvis

brainstorming

You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).

★ 0