inference-deployment

Inference and Deployment Router

Use this sub-skill when a user has, or is preparing, Chinese-LLaMA-Alpaca model assets for local generation, batch prediction, interactive instruction mode, Gradio chat, an OpenAI-compatible FastAPI server, embeddings, or LangChain QA/summarization. The commands below assume the current working directory is this sub-skill directory.

Do not launch a long-running server, download model weights, or load a large 7B+ model unless the user confirms model paths, backend use, port/network exposure, and runtime budget. If the user only has LoRA adapters and no loadable base/merged model, route first to ../model-reconstruction/.

Fast Route

1. Pick the model family. Chinese LLaMA is base/continuation-oriented; Chinese Alpaca is instruction/chat-oriented. Use references/hf-inference.md for prompt and flag details.
2. Run batch or interactive HF inference with scripts/inference_hf.py when a HF model/base+LoRA path is available.
3. Use Gradio or API serving only with explicit launch approval. See references/gradio-and-api-server.md before running scripts/gradio_demo.py or scripts/openai_api_server.py.
4. Use LangChain guidance for optional app demos. See references/langchain-integrations.md; these scripts require extra packages and model/embedding paths.
5. Debug failures with exact symptoms. Use references/troubleshooting.md for tokenizer mismatch, CPU/GPU, 8-bit, NTK, xFormers, Gradio, FastAPI request schema, and optional dependency errors.

Bundled Runtime Files

• scripts/inference_hf.py: HF batch and single-turn interactive inference.
• scripts/gradio_demo.py: Gradio chat UI.
• scripts/openai_api_server.py and scripts/openai_api_protocol.py: OpenAI-compatible completions/chat/embeddings server and schemas.
• scripts/patches.py: local attention and NTK scaling patch module used by inference/server scripts.
• scripts/langchain_qa.py, scripts/langchain_sum.py, and scripts/doc.txt: optional LangChain examples.

Scope Boundaries

• Model conversion, tokenizer extension, SHA256 checks, and LoRA merging belong to ../model-reconstruction/.
• Training or SFT of new adapters belongs to ../training-finetuning/.
• C-Eval scoring and example benchmark interpretation belong to ../evaluation-benchmarks/.
• This sub-skill can plan CPU-only checks, but CPU generation for 7B+ models is often very slow and is not a substitute for GPU performance validation.