serving-cli
Use the Transformers CLI for download, chat, local serving, OpenAI-compatible endpoints, server options, model warmup, continuous batching flags, CORS, reasoning, and safe preflight checks.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- serving-cli
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Use the Transformers CLI for download, chat, local serving, OpenAI-compatible endpoints, server options, model warmup, continuous batching flags, CORS, reasoning, and safe preflight checks.
- disable-model-invocation
- true
02 资源文件 5
- 参考文档references/cli-reference.md5.0 KB
- 参考文档references/openai-api.md5.2 KB
- 参考文档references/serving-workflows.md5.0 KB
- 参考文档references/troubleshooting.md5.7 KB
- 脚本scripts/cli_preflight.py6.6 KB
03 SKILL.md 正文
Serving CLI
Use this sub-skill when an agent needs transformers command-line workflows: download, chat, serve, local OpenAI-compatible HTTP APIs, model warmup, server options, CORS, reasoning flags, or a safe preflight before starting long-lived services.
Route First
- Use this sub-skill for
transformers --help,transformers download,transformers chat,transformers serve,/v1/chat/completions,/v1/completions,/v1/responses,/v1/audio/transcriptions,/v1/models,/load_model, and server startup diagnostics. - Use sibling inference-pipelines for in-process
pipeline(...)task selection, batching, dtype/device choices, and no-server inference. - Use sibling generation for
GenerationConfig,generate(), streamers, decoding strategy, chat templates, and continuous batching internals outside the CLI server. - Use sibling tokenizers-processors for tokenizer/processor loading, multimodal preprocessing, chat templates, and local asset validation.
- Use sibling training for
Trainer,TrainingArguments, fine-tuning, distributed training, and Hub pushing from training jobs. - Use sibling quantization-integrations for bitsandbytes, GPTQ/AWQ/Quanto, Accelerate placement, and backend-specific memory reductions.
- Use sibling model-extension for new architectures, custom modeling files, dynamic modules, and
trust_remote_codereview.
Core CLI Map
The inspected package exposes the console script:
transformers --help
It dispatches to the CLI app at transformers.cli.transformers:main with commands including:
transformers download MODEL_ID [--cache-dir DIR] [--force-download] [--trust-remote-code]transformers chat MODEL_ID [BASE_URL] [--system-prompt TEXT] [--save-folder DIR] [generate_flag=value ...]transformers serve [MODEL_ID] [server/model/continuous-batching options]transformers envandtransformers versionfor environment reporting.
For option groups, command examples, and expected signals, use references/cli-reference.md.
Safe Preflight
Before asking an agent to start a server, run the bundled dry-run helper:
python sub-skills/serving-cli/scripts/cli_preflight.py --check-serving --check-clients
Expected success signal: OK serving CLI preflight passed with imported modules and optional package availability. Expected failure signal: non-zero exit with ERROR lines naming missing hard requirements or requested extras. The script does not download models, bind ports, or start uvicorn unless a future caller edits it; by default it only imports modules, checks optional packages, validates argument combinations, and optionally probes a health URL.
Use --port 8000 to test whether a local port appears available before starting transformers serve. Use --base-url http://localhost:8000 --probe-health only when a server is expected to already be running.
Starting A Server Safely
Default transformers serve starts a long-lived FastAPI/Uvicorn server on localhost:8000. Do not launch it by default in automated verification; prefer preflight and command construction unless the user explicitly asks to run a service.
Minimal command:
transformers serve --host localhost --port 8000 --log-level info
Common production-like local command:
transformers serve Qwen/Qwen2.5-0.5B-Instruct \
--host localhost \
--port 8000 \
--dtype auto \
--device auto \
--model-timeout 300
Decision points:
- Add positional
MODEL_IDto force one preloaded model and prevent idle unload for that model. - Omit positional
MODEL_IDfor on-demand model loading driven by request bodies. - Keep
--trust-remote-codeoff unless custom repository code was reviewed. - Use
--enable-corsonly for browser-based clients that need it; it enables permissive CORS. - Use
--model-timeout SECONDSto unload idle on-demand models; it is ignored when a force model is provided. - Use
--default-seed Nwhen reproducibility matters and the backend supports deterministic behavior.
Full serving workflows are in references/serving-workflows.md.
Chat CLI
transformers chat is an interactive client for an already running transformers serve server. It calls /load_model to show model download/loading progress, then streams chat completions through the server.
transformers chat Qwen/Qwen2.5-0.5B-Instruct http://localhost:8000 max_new_tokens=128 do_sample=False
Inside the interactive prompt:
!helpprints commands.!statusprints model and generation settings.!clearresets conversation history.!set temperature=0.7 max_new_tokens=256updates generation settings.!save NAMEsaves conversation and settings.!exitends the chat client.
If chat fails immediately, verify that the server is reachable at the positional BASE_URL, /health returns a response, and /load_model can stream a ready event for the chosen model.
OpenAI-Compatible APIs
transformers serve implements local endpoints compatible with common OpenAI client patterns:
POST /v1/chat/completionsfor chat, multimodal messages, streaming, tools where supported, and reasoning fields.POST /v1/completionsfor legacy prompt completion andchoices[].textoutputs.POST /v1/responsesfor Responses API-style text, image, audio, video, multi-turn, and streaming workflows.POST /v1/audio/transcriptionsfor multipart audio transcription.GET /v1/modelsfor locally cached generative model discovery.POST /load_modelfor SSE model warmup and download/loading progress.GET /healthfor server liveness.
Use references/openai-api.md for request bodies, Python client snippets, streaming expectations, unsupported-field behavior, and response checks.
Continuous Batching And Optimizations
Enable continuous batching only after confirming backend and model compatibility:
transformers serve \
--continuous-batching \
--attn-implementation sdpa \
--dtype bfloat16 \
--cb-block-size 16 \
--cb-max-batch-tokens 4096
Important constraints:
--compileimproves some decode loops but is documented as incompatible with continuous batching.--quantization bnb-4bitand--quantization bnb-8bitrequest runtime bitsandbytes quantization; pre-quantized models may not need a server flag.--attn-implementation flash_attention_2orsdpamust match installed kernels and hardware.--cb-use-cuda-graph,--cb-num-blocks,--cb-max-memory-percent, and related KV cache sizing flags are performance tuning knobs, not correctness fixes.
For deeper generation and continuous batching concepts, route to generation. For quantization package choices, route to quantization-integrations.
Optional Dependency Boundary
A minimal Transformers import is not enough for all CLI workflows. Known boundaries:
- CLI app import can fail if base CLI dependencies such as
requestsare missing, becausetransformers chatimports HTTP clients. transformers serverequires serving extras such asfastapi,uvicorn,pydantic, andopenai; install withpip install "transformers[serving]"when available.transformers chatusesrequests,httpx,huggingface_hub, optionalrich, and server-side/load_model.- Model-backed serving requires backend packages such as
torch; PyTorch-dependent classes raise optional dependencyImportErrorwhen torch is absent. - Audio and multimodal serving may need additional packages for audio/video/image decoding beyond the base serving stack.
Troubleshooting paths and minimal install suggestions are in references/troubleshooting.md.
Fast Validation Checklist
- Run
python sub-skills/serving-cli/scripts/cli_preflight.py --check-serving --check-clients --port 8000. - Check
transformers --helponly afterrequestsand Typer-related CLI dependencies import successfully. - Construct the intended
transformers servecommand without running it. - Confirm port availability, host binding, CORS need, model source,
trust_remote_code, dtype/device, and timeout policy. - If a server is already running, probe
GET /healthand optionallyGET /v1/modelsbefore sending generation requests. - Warm up with
POST /load_modelonly when network/model downloads and backend memory are acceptable. - Send a tiny non-streaming request before enabling streaming, continuous batching, quantization, or browser clients.
Evidence Base
This sub-skill distills Transformers 5.13.0.dev0 behavior from CLI entrypoints, chat/download/serve command implementations, serving handlers, serve CLI docs, conversation docs, optimization docs, and CLI tests. Runtime guidance is self-contained and does not require opening source repository files.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).