AGENT SKILL

serving-cli

Use the Transformers CLI for download, chat, local serving, OpenAI-compatible endpoints, server options, model warmup, continuous batching flags, CORS, reasoning, and safe preflight checks.

License 许可
Apache-2.0
最近更新
2026-09-03
Stars
★ 351
Forks
27

01 前言 Front Matter

name
serving-cli
license
Apache 2.0
metadata
{ "disco-role": "operating" }
description
Use the Transformers CLI for download, chat, local serving, OpenAI-compatible endpoints, server options, model warmup, continuous batching flags, CORS, reasoning, and safe preflight checks.
disable-model-invocation
true

02 资源文件 5

03 SKILL.md 正文

Serving CLI

Use this sub-skill when an agent needs transformers command-line workflows: download, chat, serve, local OpenAI-compatible HTTP APIs, model warmup, server options, CORS, reasoning flags, or a safe preflight before starting long-lived services.

Route First

  • Use this sub-skill for transformers --help, transformers download, transformers chat, transformers serve, /v1/chat/completions, /v1/completions, /v1/responses, /v1/audio/transcriptions, /v1/models, /load_model, and server startup diagnostics.
  • Use sibling inference-pipelines for in-process pipeline(...) task selection, batching, dtype/device choices, and no-server inference.
  • Use sibling generation for GenerationConfig, generate(), streamers, decoding strategy, chat templates, and continuous batching internals outside the CLI server.
  • Use sibling tokenizers-processors for tokenizer/processor loading, multimodal preprocessing, chat templates, and local asset validation.
  • Use sibling training for Trainer, TrainingArguments, fine-tuning, distributed training, and Hub pushing from training jobs.
  • Use sibling quantization-integrations for bitsandbytes, GPTQ/AWQ/Quanto, Accelerate placement, and backend-specific memory reductions.
  • Use sibling model-extension for new architectures, custom modeling files, dynamic modules, and trust_remote_code review.

Core CLI Map

The inspected package exposes the console script:

transformers --help

It dispatches to the CLI app at transformers.cli.transformers:main with commands including:

  • transformers download MODEL_ID [--cache-dir DIR] [--force-download] [--trust-remote-code]
  • transformers chat MODEL_ID [BASE_URL] [--system-prompt TEXT] [--save-folder DIR] [generate_flag=value ...]
  • transformers serve [MODEL_ID] [server/model/continuous-batching options]
  • transformers env and transformers version for environment reporting.

For option groups, command examples, and expected signals, use references/cli-reference.md.

Safe Preflight

Before asking an agent to start a server, run the bundled dry-run helper:

python sub-skills/serving-cli/scripts/cli_preflight.py --check-serving --check-clients

Expected success signal: OK serving CLI preflight passed with imported modules and optional package availability. Expected failure signal: non-zero exit with ERROR lines naming missing hard requirements or requested extras. The script does not download models, bind ports, or start uvicorn unless a future caller edits it; by default it only imports modules, checks optional packages, validates argument combinations, and optionally probes a health URL.

Use --port 8000 to test whether a local port appears available before starting transformers serve. Use --base-url http://localhost:8000 --probe-health only when a server is expected to already be running.

Starting A Server Safely

Default transformers serve starts a long-lived FastAPI/Uvicorn server on localhost:8000. Do not launch it by default in automated verification; prefer preflight and command construction unless the user explicitly asks to run a service.

Minimal command:

transformers serve --host localhost --port 8000 --log-level info

Common production-like local command:

transformers serve Qwen/Qwen2.5-0.5B-Instruct \
  --host localhost \
  --port 8000 \
  --dtype auto \
  --device auto \
  --model-timeout 300

Decision points:

  • Add positional MODEL_ID to force one preloaded model and prevent idle unload for that model.
  • Omit positional MODEL_ID for on-demand model loading driven by request bodies.
  • Keep --trust-remote-code off unless custom repository code was reviewed.
  • Use --enable-cors only for browser-based clients that need it; it enables permissive CORS.
  • Use --model-timeout SECONDS to unload idle on-demand models; it is ignored when a force model is provided.
  • Use --default-seed N when reproducibility matters and the backend supports deterministic behavior.

Full serving workflows are in references/serving-workflows.md.

Chat CLI

transformers chat is an interactive client for an already running transformers serve server. It calls /load_model to show model download/loading progress, then streams chat completions through the server.

transformers chat Qwen/Qwen2.5-0.5B-Instruct http://localhost:8000 max_new_tokens=128 do_sample=False

Inside the interactive prompt:

  • !help prints commands.
  • !status prints model and generation settings.
  • !clear resets conversation history.
  • !set temperature=0.7 max_new_tokens=256 updates generation settings.
  • !save NAME saves conversation and settings.
  • !exit ends the chat client.

If chat fails immediately, verify that the server is reachable at the positional BASE_URL, /health returns a response, and /load_model can stream a ready event for the chosen model.

OpenAI-Compatible APIs

transformers serve implements local endpoints compatible with common OpenAI client patterns:

  • POST /v1/chat/completions for chat, multimodal messages, streaming, tools where supported, and reasoning fields.
  • POST /v1/completions for legacy prompt completion and choices[].text outputs.
  • POST /v1/responses for Responses API-style text, image, audio, video, multi-turn, and streaming workflows.
  • POST /v1/audio/transcriptions for multipart audio transcription.
  • GET /v1/models for locally cached generative model discovery.
  • POST /load_model for SSE model warmup and download/loading progress.
  • GET /health for server liveness.

Use references/openai-api.md for request bodies, Python client snippets, streaming expectations, unsupported-field behavior, and response checks.

Continuous Batching And Optimizations

Enable continuous batching only after confirming backend and model compatibility:

transformers serve \
  --continuous-batching \
  --attn-implementation sdpa \
  --dtype bfloat16 \
  --cb-block-size 16 \
  --cb-max-batch-tokens 4096

Important constraints:

  • --compile improves some decode loops but is documented as incompatible with continuous batching.
  • --quantization bnb-4bit and --quantization bnb-8bit request runtime bitsandbytes quantization; pre-quantized models may not need a server flag.
  • --attn-implementation flash_attention_2 or sdpa must match installed kernels and hardware.
  • --cb-use-cuda-graph, --cb-num-blocks, --cb-max-memory-percent, and related KV cache sizing flags are performance tuning knobs, not correctness fixes.

For deeper generation and continuous batching concepts, route to generation. For quantization package choices, route to quantization-integrations.

Optional Dependency Boundary

A minimal Transformers import is not enough for all CLI workflows. Known boundaries:

  • CLI app import can fail if base CLI dependencies such as requests are missing, because transformers chat imports HTTP clients.
  • transformers serve requires serving extras such as fastapi, uvicorn, pydantic, and openai; install with pip install "transformers[serving]" when available.
  • transformers chat uses requests, httpx, huggingface_hub, optional rich, and server-side /load_model.
  • Model-backed serving requires backend packages such as torch; PyTorch-dependent classes raise optional dependency ImportError when torch is absent.
  • Audio and multimodal serving may need additional packages for audio/video/image decoding beyond the base serving stack.

Troubleshooting paths and minimal install suggestions are in references/troubleshooting.md.

Fast Validation Checklist

  1. Run python sub-skills/serving-cli/scripts/cli_preflight.py --check-serving --check-clients --port 8000.
  2. Check transformers --help only after requests and Typer-related CLI dependencies import successfully.
  3. Construct the intended transformers serve command without running it.
  4. Confirm port availability, host binding, CORS need, model source, trust_remote_code, dtype/device, and timeout policy.
  5. If a server is already running, probe GET /health and optionally GET /v1/models before sending generation requests.
  6. Warm up with POST /load_model only when network/model downloads and backend memory are acceptable.
  7. Send a tiny non-streaming request before enabling streaming, continuous batching, quantization, or browser clients.

Evidence Base

This sub-skill distills Transformers 5.13.0.dev0 behavior from CLI entrypoints, chat/download/serve command implementations, serving handlers, serve CLI docs, conversation docs, optimization docs, and CLI tests. Runtime guidance is self-contained and does not require opening source repository files.

04 同仓库更多 Skills

VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-triton-opt

Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-solve

END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-skill

PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-pattern-fusion

Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-orchestrate

Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-feedback

Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).

★ 351