models-and-serving
Configure DB-GPT 0.8.1 model providers and local backends, validate model TOML, operate model controller/worker/API-server roles, use model-management CLI commands, and diagnose serving failures without mistaking an import or registry record for a live deployment.
- License 许可
- MIT
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- models-and-serving
- license
- MIT
- metadata
- { "disco-role": "operating" }
- description
- Configure DB-GPT 0.8.1 model providers and local backends, validate model TOML, operate model controller/worker/API-server roles, use model-management CLI commands, and diagnose serving failures without mistaking an import or registry record for a live deployment.
- disable-model-invocation
- true
02 资源文件 6
- 参考文档references/backends.md4.8 KB
- 参考文档references/model-api-reference.md7.6 KB
- 参考文档references/provider-and-backend-matrix.md6.2 KB
- 参考文档references/serving-workflows.md7.4 KB
- 参考文档references/troubleshooting.md7.0 KB
- 脚本scripts/model_config_check.py10.1 KB
03 SKILL.md 正文
Models and serving
Use this route when a DB-GPT task mentions model/provider TOML, models.llms,
embeddings or rerankers, OpenAI-compatible endpoints, Ollama, Hugging Face,
vLLM, llama.cpp, MLX, model workers, controller, model API server, model CLI,
model registration, health/heartbeat, GPU memory, quantization, or serving logs.
This route owns model selection and the model-service topology. Route general
installation, profile creation, workspace paths, and generic start webserver
setup to setup-and-cli; route agents/AWEL to agents-and-awel; route public
client CRUD and sandbox calls to apis-client-and-sandbox; route document,
embedding-pipeline, vector-store, and knowledge workflows to data-and-rag.
Operating contract
- Work against the public DB-GPT 0.8.1 package interface, not a source checkout.
- Treat a TOML parse, Python import, controller registry row, or
healthyvalue observed before a real request as configuration evidence only. Deployment is successful only after the intended service is reachable and a suitable model operation succeeds. - Keep provider credentials in environment variables. Never print resolved secrets, copy them into generated files, or put them in shell history.
- Choose one model role explicitly:
llm,text2vec(embedding), orreranker. A chat model cannot silently substitute for an embedding model. - CPU is the verified baseline for this route. CUDA/local-model, vLLM, bitsandbytes, llama.cpp acceleration, and MLX are optional backend paths; do not claim they work from a CPU import or from host GPU visibility alone.
Read the focused contracts before making changes:
- model-api-reference.md — configuration dataclasses, model/provider fields, worker types, and OpenAI-compatible API surface.
- provider-and-backend-matrix.md — provider credentials, endpoint conventions, embedding pairing, optional dependencies, and backend boundaries.
- serving-workflows.md — proxy/local TOML, unified webserver, split controller/worker/API-server deployment, and CLI operations.
- backends.md — local-model, quantization, VRAM, and hardware-specific decision rules.
- troubleshooting.md — symptom-to-check diagnosis and recovery boundaries.
Standard workflow
-
Identify the topology and model roles. Decide whether the user needs a unified webserver process, a split cluster, a standalone model API server, or only a provider configuration. Record the LLM, embedding, and optional reranker names separately.
-
Select the provider/backend. Prefer a remote proxy for a CPU-first smoke test. For a local model, require a real model path or an explicitly allowed model download and select
hf,vllm,llama.cpp,llama.cpp.server, ormlxdeliberately. Check the matrix for the provider-specific key and endpoint. -
Write a redacted TOML. Put model entries under
[models]using repeated[[models.llms]],[[models.embeddings]], and[[models.rerankers]]tables. Use${env:NAME}or${env:NAME:-default}placeholders. Setdefault_llm,default_embedding, ordefault_rerankeronly when multiple entries make selection ambiguous. Keep web/database configuration owned by the setup route unless the serving topology requires the related model service section. -
Validate without network or model loading. Run the bundled checker:
python scripts/model_config_check.py path/to/config.tomlIt parses TOML, checks model-table shape, provider/role requirements, duplicate names, placeholder syntax, endpoint URLs, and obvious service-port collisions. It never resolves secrets, calls a provider, contacts a controller, downloads a model, or starts a process. Use
--jsonfor a machine-readable report and--allow-missing-embeddingsonly when the configuration is intentionally chat-only; RAG/knowledge workflows must not use that exception. -
Run a safe package-level check. Use
dbgpt --version, the relevant static--help, and import/config-parser checks. Do not usedbgpt model start --helpor another dynamic model command as an offline gate: model command discovery can query a controller and return a timeout/502 when no controller is reachable. -
Start only the chosen topology. Use the exact commands in serving-workflows.md. Do not start both a unified model manager and an independent worker on the same ports. For split mode, start and verify the controller before workers, then verify workers before the API/web layer.
-
Verify reachability and model capability. Check the controller health and registry, then issue a minimal request through the intended API or provider.
dbgpt model listis a registry/heartbeat view, not proof that generation or embedding works. An unreachable controller, failed heartbeat, missing model, authentication failure, or failed embedding request is a deployment failure; report it as such with the exact next check. -
Diagnose from the right boundary. Inspect the command's resolved config path, service logs, bind address, port, controller address, provider base URL, model name, and optional-backend imports in that order. Redact API keys and authorization headers in reports.
CLI routing facts
The DB-GPT CLI registers model commands under dbgpt model and service commands
under dbgpt start / dbgpt stop:
dbgpt model [--address CONTROLLER_URL] list|start|stop|restart|chat
dbgpt start controller --config FILE [--daemon]
dbgpt start worker --config FILE [--daemon]
dbgpt start apiserver --config FILE [--daemon]
dbgpt start webserver [web options]
dbgpt stop controller|worker|apiserver [--port PORT]
CONTROLLER_ADDRESS can supply the controller address used by model-management
commands when --address is omitted. The dynamic model start command obtains
supported-model metadata from the controller before constructing some options;
therefore a failed remote request during help or invocation is not a successful
start. For model-service launch commands, the configuration file is the
source of worker/model parameters; do not rely on stale examples that show
model flags which are not present in the installed command tree.
Validation and boundaries
A valid result for this route includes:
- provider/model/role entries parse with no secret resolution or network call;
- an embedding entry exists and is compatible with the intended RAG path;
- each selected local backend's dependency, path, device, and VRAM assumptions are explicit;
- service roles have distinct, reachable addresses and non-conflicting ports;
- controller registry and heartbeat checks are separated from generation, completion, embedding, or reranking checks;
- optional CUDA/GPU/provider/service dependencies are labeled unverified when they were not executed.
Do not claim that dbgpt model list proves deployment, that --daemon proves a
healthy child process, that a provider key exists merely because a placeholder
parsed, or that an A100-visible host proves CUDA/torch/vLLM/bitsandbytes support.
Do not add API CRUD, agent orchestration, or RAG chunking instructions here.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).