model-deployment
Guides LazyLLM model modules, online providers, local serving, fine-tuning, multimodal examples, and backend-aware deployment diagnostics.
- License 许可
- Apache-2.0
- 最近更新
- 2026-10-08
- Stars
- ★ 11
- Forks
- 3
01 前言 Front Matter
- name
- model-deployment
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Guides LazyLLM model modules, online providers, local serving, fine-tuning, multimodal examples, and backend-aware deployment diagnostics.
- disable-model-invocation
- true
02 资源文件 3
- 参考文档references/backend-matrix.md2.7 KB
- 参考文档references/model-workflows.md3.7 KB
- 参考文档references/troubleshooting.md2.4 KB
03 SKILL.md 正文
LazyLLM Model Deployment
Use this sub-skill for LazyLLM tasks about TrainableModule, OnlineModule, OnlineChatModule, ServerModule, ActionModule, model type mapping, model deployment, fine-tuning, distillation, local serving, provider calls, streaming tool calls, or multimodal examples.
Start here when
- The user asks how to wrap a local model or online provider in LazyLLM.
- The task mentions
lazyllm deploy, vLLM, LMDeploy, LightLLM, Infinity, LLaMA-Factory, model caches, CUDA, or serving ports. - A provider/chat module fails due to model type, API key, timeout, message inspection, streaming tool calls, or provider response format.
- The user wants to adapt chatbot, online chatbot, multimodal, painting, TTS, STT, OCR, fine-tuning, or distillation examples.
Files to read
- model-workflows.md for class selection, online/local/multimodal recipes, and safe provider-free checks.
- backend-matrix.md for optional dependency and backend classification.
- troubleshooting.md for provider, CUDA, model path, and serving failures.
- scripts/model_surface_smoke.py for a no-network check of model type inference and online chat message utilities.
Safe workflow
- Classify the module type.
OnlineModule/OnlineChatModule: provider or endpoint backed.TrainableModule: local model/fine-tune/deploy abstraction.ServerModule: wraps functions/modules/services and may start/bind servers.ActionModule: wraps deterministic actions in a LazyLLM module chain.
- Classify backend requirements. Do not run a model call until provider credentials or local model/GPU/backend are confirmed.
- Perform no-network checks first.
python scripts/model_surface_smoke.py python ../../scripts/inspect_lazyllm_surface.py --include-optional - Route composed apps to their owner. If model modules are only nodes inside a
pipeline, read flow-orchestration. If they serve RAG or agents, read the RAG or agent sub-skill too. - Record optional verification clearly. Online/GPU/model-download tests are optional unless the user explicitly asks to execute them and supplies resources.
Verified signatures to remember
TrainableModule(base_model='', target_path='', *, stream=False, return_trace=False, trust_remote_code=True, type=None, source=None, use_model_map=True)OnlineModule(model=None, source=None, *, type=None, url=None, **kwargs)OnlineChatModule(model=None, source=None, url=None, stream=True, return_trace=False, skip_auth=False, type=None, api_key=None, static_params=None, id=None, name=None, group_id=None, dynamic_auth=False, timeout=180, **kwargs)ServerModule(m=None, pre=None, post=None, stream=False, return_trace=False, port=None, pythonpath=None, launcher=None, url=None, num_replicas=1, security_key=None)ActionModule(*action, return_trace=False)
Model type and streaming facts
LazyLLM has provider model-name mapping helpers that classify names into categories including llm, vlm, stt, tts, embed, sd, text2video, and cross_modal_embed. Tests also verify online chat helper behavior that removes prior tool traces and merges streamed tool-call chunks by index.
Use these pure helpers to debug provider message formatting without calling the provider.
Backend boundaries
- Safe CPU checks: signature inspection, model type mapping, chat history sanitization, stream chunk merge utility.
- Provider-required: online chat/completion, online multimodal, web-search-backed flows, writer pipeline with cloud LLM.
- GPU/local backend: vLLM/LMDeploy/LightLLM, local TrainableModule inference, fine-tuning, distillation, painting, speech, OCR.
- External service: deployed server endpoints, k8s/slurm/SCO launcher tests.
Handoff outputs
When you finish a model-deployment task, leave:
- chosen LazyLLM module class and constructor parameters,
- exact optional extra/backend/provider requirements,
- safe smoke result or reason it was not executable,
- explicit status of credentials, model path/cache, GPU, server port, and budget,
- routes to flow/RAG/agent/writer sub-skills for composed apps.
04 同仓库更多 Skills
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
python-api
Programmatic Object-Detection-Metrics API guidance for BoundingBox, BoundingBoxes, Evaluator, enums, coordinate conversion, AP/mAP semantics, and source-style import diagnostics.
file-evaluation
Folder-based PASCAL VOC AP/mAP evaluation from ground-truth and detection text files.
object-detection-metrics
Guides legacy Object-Detection-Metrics PASCAL VOC AP/mAP evaluation from detection text folders and direct Python API classes.
nx-workspace-patterns
Configure and optimize Nx monorepo workspaces. Use when setting up Nx, configuring project boundaries, optimizing build caching, or implementing affected commands.