friendli-ai
High-throughput inference on FriendliAI — optimized serving engine for low-latency, cost-efficient LLMs.
- License 许可
- MIT
- 最近更新
- 2026-11-07
- Stars
- ★ 0
- Forks
- 1
01 前言 Front Matter
- name
- friendli-ai
- category
- ai-research
- description
- High-throughput inference on FriendliAI — optimized serving engine for low-latency, cost-efficient LLMs.
03 SKILL.md 正文
Overview
FriendliAI builds inference infrastructure with a focus on serving efficiency: their serving engine (with techniques like iteration-level scheduling and optimized kernels) targets high throughput and low latency per dollar, offered both as a managed inference service and as software for self-hosted deployment. The positioning is infrastructure-grade efficiency — for teams serving LLMs at scale where serving economics dominate the bill.
The distinctive angle is the engine itself: FriendliAI's research in efficient serving (published work on scheduling and optimization) informs both their cloud service and their deployable software. For organizations serving open models at volume — whether on FriendliAI's cloud or their own GPUs with FriendliAI's stack — the value is measured in tokens per dollar at target latency.
Evaluate FriendliAI where serving efficiency is the decision criterion: high volume, latency SLOs, and cost-per-token pressure.
When to use
- High-volume LLM serving where cost-per-token at scale decides.
- Latency-sensitive serving needing efficient scheduling (low TTFT + high throughput).
- Self-hosted inference wanting a high-efficiency serving engine.
- Multi-tenant serving with mixed workload shapes.
- Throughput-heavy batch + online mixed workloads.
- Teams optimizing serving economics as a core competency.
Core concepts
- Serving engine efficiency: iteration-level scheduling, optimized attention kernels, and quantization support — the technical sources of throughput-per-dollar. Understand what's applied in your deployment.
- Managed service vs. software: use their cloud, or deploy their engine on your infrastructure. The choice hinges on your ops capacity and data control needs.
- Throughput per dollar: the headline metric — tokens/sec per unit cost at your latency SLO. Benchmark this, not raw tokens/sec alone.
- Latency SLO adherence: efficiency that violates p99 latency isn't useful. Measure the latency distribution, not just the mean throughput.
- Quantization integration: efficient serving often involves quantization — validate output quality on your task for the quantized configuration.
- Mixed workload handling: batch and interactive traffic on shared infrastructure — scheduling that protects interactive latency while maintaining batch throughput.
- Dedicated capacity: reserved deployments for predictable performance. Match to your SLO requirements.
- Observability: throughput, latency percentiles, GPU utilization, queue depths — the metrics that reveal serving health.
Practical workflow
- Define the efficiency target. Target cost per million tokens at your required p99 latency. This is the benchmark everything else serves.
- Benchmark throughput per dollar. Your models, your prompt shapes, your concurrency: measure tokens/sec per dollar at acceptable latency. Compare against your current serving.
- Validate quantized quality. If the efficient configuration uses quantization: run your eval set. Efficiency that degrades quality below your bar is disqualified.
- Test mixed workloads. If you serve batch + interactive: verify interactive latency holds under batch load. Scheduling quality shows here.
- Decide managed vs. self-hosted. Ops capacity, data control, and economics — model both. Self-hosting the engine trades ops burden for control.
- Load-test at scale. Sustained peak concurrency; watch latency percentiles, queue behavior, and failure modes under overload.
- Monitor serving economics continuously. Track cost per token, latency SLOs, and utilization. Serving efficiency drifts with traffic shape — review regularly.
Checklist for FriendliAI in production:
- Cost per token at required p99 latency benchmarked.
- Quantized configurations quality-validated.
- Mixed-workload latency verified (if applicable).
- Managed vs. self-hosted decision modeled.
- Serving metrics monitored (throughput, latency, utilization, cost).
Common pitfalls
- Throughput without latency SLOs. Optimizing tokens/sec while p99 latency violates UX requirements. Efficiency is throughput per dollar at acceptable latency.
- Unvalidated quantization. Assuming quantized serving preserves quality. Validate on your task — always.
- Single-workload benchmarks. Testing only uniform prompts when production mixes short/long, batch/interactive. Test the mix.
- Ignoring queue behavior. Throughput looks fine until queues build under burst. Test bursts; monitor queue depths.
- Managed-vs-self-hosted on vibes. Choosing without modeling ops cost, data control needs, and economics. Decide on numbers.
- Utilization blindness. Low GPU utilization means wasted money; saturation means latency risk. Right-size from metrics.
- No overload plan. What happens past capacity — graceful degradation or collapse? Define and test the overload behavior.
- Static efficiency assumptions. Traffic shape changes alter serving efficiency. Re-benchmark when workloads shift.
04 同仓库更多 Skills
web-scrape
Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open/fetch fails on JS-heavy pages, or when you need clean markdown/structured extraction from a site. Requires FIRECRAWL_API_KEY.
design-taste-frontend
tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML/CSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, "AI-made", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.
test-driven-development
Use when implementing any feature or bugfix, before writing implementation code
systematic-debugging
Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes
requesting-code-review
Use when completing tasks, implementing major features, or before merging to verify work meets requirements
brainstorming
You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).