AGENT SKILL

baseten-guide

Deploy ML models to production with Baseten — high-performance inference with autoscaling and custom runtimes.

License 许可
MIT
所属仓库
TanNhatCMS/jarvis
最近更新
2026-11-07
Stars
★ 0
Forks
1

01 前言 Front Matter

name
baseten-guide
category
ai-research
description
Deploy ML models to production with Baseten — high-performance inference with autoscaling and custom runtimes.

03 SKILL.md 正文

Overview

Baseten is a model-deployment platform focused on production inference performance: deploy open models (or your own) behind autoscaled, low-latency APIs with strong performance engineering (custom inference engines, optimized serving stacks). The pitch is "self-hosted performance without self-hosting" — you get serious inference optimization managed for you, with dedicated capacity options for strict SLOs.

Where serverless GPU platforms optimize for convenience and scale-to-zero, Baseten optimizes for the production middle: sustained traffic, latency SLOs, and cost-per-token at volume. If your workload has graduated from "run this model sometimes" to "serve this model reliably at scale," Baseten is built for that transition.

The evaluation lens: measure p99 latency and cost per million tokens at your actual traffic shape. Baseten's value shows up in those numbers, not in feature lists.

When to use

  • Production LLM serving with latency SLOs (p99 matters, not just averages).
  • High-volume open-model inference where cost-per-token at scale is the metric.
  • Deploying fine-tuned models behind production-grade endpoints.
  • Multi-modal model serving (the platform covers more than text).
  • Teams that need inference performance engineering without hiring for it.
  • Dedicated capacity for isolation and predictable performance.

Core concepts

  • Model deployments: versioned deployments of models behind API endpoints. Deploy from popular open models or your own weights/fine-tunes. Versioning lets you roll forward and back safely.
  • Performance-optimized serving: the platform applies inference optimizations (efficient attention kernels, batching, quantization options) so you don't have to. Understand what's applied to your deployment — it affects both latency and output behavior.
  • Autoscaling: capacity scales with traffic within configured bounds. Configure min/max replicas deliberately: min for baseline latency (warm capacity), max for cost protection.
  • Dedicated deployments: reserved GPUs for your workload — no noisy neighbors, predictable latency. The right choice when shared capacity variance violates SLOs.
  • Custom models and fine-tunes: bring your own weights or train on the platform; serve the result on the same optimized stack. Keeps the customize→serve loop tight.
  • Development vs. production tiers: lighter/cheaper options for dev and staging, performance tiers for production. Don't benchmark dev-tier latency and extrapolate to production.
  • Observability: request logging, latency metrics, and GPU utilization. Wire these into your monitoring — inference issues show up here before users report them.
  • Cost model: pay for the capacity you reserve plus usage. Model your expected cost at baseline and peak traffic before committing.

Practical workflow

  1. Define SLOs first. Target p50/p99 latency, throughput, and availability. Everything — deployment tier, replica counts, dedicated vs. shared — follows from these numbers.
  2. Deploy and benchmark. Deploy the model, then load-test at realistic traffic shape (not just steady-state — include bursts). Measure p99 latency and tokens/sec; tune replica bounds and instance types.
  3. Validate output quality. Optimized serving stacks (quantization, custom kernels) can subtly change outputs. Run your eval set against the deployed endpoint, not just the reference model.
  4. Configure autoscaling deliberately. Set min replicas for warm baseline capacity (cold scaling hurts p99), max replicas as a cost circuit breaker. Test scale-up behavior under a traffic spike.
  5. Set up staging. Mirror production config in staging for safe rollout testing. Canary new model versions through staging before production.
  6. Version and roll back safely. Deploy new versions alongside old; shift traffic gradually; keep the previous version deployable for instant rollback.
  7. Monitor continuously. Track latency percentiles, error rates, GPU utilization, and cost. Alert on SLO breaches and cost anomalies.

Checklist for a Baseten production deployment:

  • SLOs defined; load test passed at realistic traffic shape.
  • Output quality validated on the deployed (optimized) stack.
  • Autoscaling min/max set and spike-tested.
  • Staging mirrors production; canary process defined.
  • Latency, error, utilization, and cost monitoring in place.

Common pitfalls

  • Benchmarking the wrong tier. Dev-tier numbers don't predict production latency. Benchmark the tier you'll actually run.
  • Skipping output validation. Assuming the optimized stack is behavior-identical to the reference model. Quantization and custom kernels can shift outputs — verify.
  • Min replicas at zero for latency-sensitive APIs. Scale-from-zero saves money and destroys p99. Warm baseline capacity is the cost of latency SLOs.
  • No max-replica cost guard. Autoscaling without an upper bound during a traffic anomaly or retry storm. Set the max; alert approaching it.
  • Traffic shape mismatch. Load-testing steady state when production has bursts. Test bursts, cold traffic patterns, and sustained peaks.
  • Big-bang version upgrades. Swapping model versions without canarying. Inference behavior changes are user-facing — roll out gradually.
  • Ignoring GPU utilization. Low utilization means you're overprovisioned (wasting money); saturated means no headroom (latency risk). Right-size from the metrics.
  • No rollback plan. A bad deploy with no quick revert path. Keep the previous version warm and the rollback procedure rehearsed.

04 同仓库更多 Skills

TanNhatCMS
TanNhatCMS/jarvis

web-scrape

Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open/fetch fails on JS-heavy pages, or when you need clean markdown/structured extraction from a site. Requires FIRECRAWL_API_KEY.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

design-taste-frontend

tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML/CSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, "AI-made", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.

★ 0
TanNhatCMS
TanNhatCMS/jarvis

test-driven-development

Use when implementing any feature or bugfix, before writing implementation code

★ 0
TanNhatCMS
TanNhatCMS/jarvis

systematic-debugging

Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes

★ 0
TanNhatCMS
TanNhatCMS/jarvis

requesting-code-review

Use when completing tasks, implementing major features, or before merging to verify work meets requirements

★ 0
TanNhatCMS
TanNhatCMS/jarvis

brainstorming

You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).

★ 0