inference
Load and smoke-check FlagEmbedding embedders, M3 models, and rerankers for retrieval inference.
- License 许可
- MIT
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- inference
- license
- MIT
- metadata
- { "disco-role": "operating" }
- description
- Load and smoke-check FlagEmbedding embedders, M3 models, and rerankers for retrieval inference.
- disable-model-invocation
- true
02 资源文件 4
- 参考文档references/api-reference.md10.2 KB
- 参考文档references/troubleshooting.md6.6 KB
- 参考文档references/workflows.md13.5 KB
- 脚本scripts/smoke_inference_api.py14.1 KB
03 SKILL.md 正文
Inference
Use this sub-skill when the task is to load a FlagEmbedding model for inference, encode queries or corpus passages, compute embedding similarity, compute BGE-M3 dense/sparse/ColBERT scores, rerank query-passage pairs, or smoke-check the inference API before a retrieval run.
Do not use this sub-skill for fine-tuning data formats, fine-tuning launchers,
training hyperparameters, or checkpoint production. Route those tasks to the
sibling fine-tuning sub-skill. Do not use it for benchmark suites, leaderboard
evaluation, or metric scripts. Route those tasks to the sibling evaluation
sub-skill. For package installation, import failures, and global optional
dependency repair, route to the root troubleshooting reference at
../../references/troubleshooting.md.
Start Here
- Decide whether the model is an embedder, an M3 embedder, or a reranker.
- Prefer
FlagAutoModel.from_finetuned(...)for embedders andFlagAutoReranker.from_finetuned(...)for rerankers when the checkpoint name is in the built-in mapping. - For a custom local checkpoint or an unmapped remote checkpoint, pass
model_classexplicitly and choose the concrete class behavior from references/api-reference.md. - For retrieval, use
encode_queries(...)for queries andencode_corpus(...)for passages. Query instructions are applied byencode_queries; passages normally should not receive the query instruction. - For reranking, pass a pair as
("query", "passage")or a batch as a list of pairs tocompute_score(...); validate the returned list or nested list shape before sorting results. - For a safe local check, run
scripts/smoke_inference_api.py --helpfirst. Running the script with no--model-nameperforms import and signature probing only. Loading or encoding happens only when--model-nameis passed.
Public Entry Points
Embedder auto loader and concrete classes:
FlagAutoModelFlagModelBGEM3FlagModelFlagLLMModelFlagICLModelFlagPseudoMoEModel
Reranker auto loader and concrete classes:
FlagAutoRerankerFlagRerankerFlagLLMRerankerLayerWiseFlagLLMRerankerLightWeightFlagLLMReranker
Embedder model_class ids:
encoder-only-base: encoder-only dense embedder, normallyFlagModel.encoder-only-m3: BGE-M3 dense/sparse/ColBERT embedder,BGEM3FlagModel.decoder-only-base: decoder-only LLM embedder,FlagLLMModel.decoder-only-icl: few-shot ICL LLM embedder,FlagICLModel.decoder-only-pseudo_moe: pseudo-MoE LLM embedder,FlagPseudoMoEModel.
Reranker model_class ids:
encoder-only-base: encoder cross-encoder reranker,FlagReranker.decoder-only-base: decoder-only LLM reranker,FlagLLMReranker.decoder-only-layerwise: layer-selectable LLM reranker,LayerWiseFlagLLMReranker.decoder-only-lightweight: token-compression LLM reranker,LightWeightFlagLLMReranker.
Loader Choice
Use auto loaders when the checkpoint's effective model name is in the built-in
mapping. The effective name is the basename of model_name_or_path; if the
basename starts with checkpoint-, the parent directory basename is used. This
means a fine-tuned checkpoint directory such as .../my-bge/checkpoint-1000
is mapped as my-bge, not checkpoint-1000.
If auto mapping raises a model-not-found error, do not keep retrying with the
same call. Pick the correct model_class, and for embedders also set
pooling_method, query_instruction_format, and trust_remote_code rather
than relying on mapping defaults. Custom decoder-only and remote-code
checkpoints are especially likely to need explicit trust_remote_code=True
after code review.
Use concrete classes directly when the architecture is already known or when you need class-specific options such as M3 return modes, ICL examples, pseudo-MoE domains, layerwise cutoff layers, or lightweight compression.
Core Parameters
Use references/api-reference.md for verified signatures and parameter defaults. The parameters most likely to affect inference behavior are:
normalize_embeddings: normalize dense embeddings before similarity.use_fp16anduse_bf16: inference dtype. Use CPU with full precision for conservative smoke checks.query_instruction_for_retrievalandquery_instruction_format: query-side retrieval instructions for embedders.devices:Noneauto-selects available accelerators then CPU; a string such as"cpu"or"cuda:0"pins one device; a list enables multi-process inference across target devices.pooling_method: useclsormeanfor compatible encoder-only models; decoder-only embedders requirelast_token.trust_remote_code: keepFalseunless the checkpoint requires custom model code and that code has been reviewed.truncate_dim: truncate output vectors for Matryoshka-style embeddings.batch_size,query_max_length, andpassage_max_length: embedder throughput and truncation controls.- Reranker
query_max_length,max_length, andnormalize: pair scoring truncation and optional sigmoid normalization. - Layerwise/lightweight reranker
cutoff_layers,compress_ratio, andcompress_layers: score layer selection and token compression controls.
Expected Outputs
Base embedders return a NumPy array by default. A single input returns a 1-D
vector; a list returns a 2-D array shaped like (num_texts, embedding_dim).
With convert_to_numpy=False, expect a torch tensor instead.
BGEM3FlagModel returns a dictionary with dense_vecs, lexical_weights, and
colbert_vecs. Keys whose return flag is false are present with None values.
Use dense_vecs @ dense_vecs.T for dense similarity, use
compute_lexical_matching_score(...) for sparse lexical weights, and use
compute_score(...) for per-pair M3 mode scores.
Standard rerankers return a list of float scores for a batch. Layerwise and lightweight rerankers return one score list when a single cutoff layer is used, or a list of score lists when multiple cutoff layers are requested. Do not sort documents until this shape is validated.
Workflows
Use references/workflows.md for these recipes:
- Auto and concrete embedder loading.
- Query/corpus encoding and similarity computation.
- M3 dense, sparse, ColBERT, and
compute_scoreworkflows. - Decoder-only, ICL, and pseudo-MoE embedder options.
- Encoder, decoder, layerwise, and lightweight reranker workflows.
- Batch sizing, CPU fallback, and multi-device inference.
Use references/troubleshooting.md when:
- Auto mapping cannot find a model.
- Remote-code or local custom-code loading fails.
- Network/cache behavior is surprising.
- CPU/GPU fallback, dtype, or OOM behavior changes results or fails.
- Output shapes or score modes are misused.
Smoke Checks
Run the bundled helper before writing a larger pipeline:
python sub-skills/inference/scripts/smoke_inference_api.py
The default command imports public inference symbols and prints signatures only.
It does not instantiate a model and should not trigger downloads. From the
flag-embedding skill directory, load and encode only when an explicit local
path or model id is supplied:
python sub-skills/inference/scripts/smoke_inference_api.py \
--model-kind embedder \
--model-name MODEL_NAME_OR_PATH \
--devices cpu \
--batch-size 2
For a custom local checkpoint, add --model-class, --pooling-method, and
--query-instruction-format as needed. For a reranker, use
--model-kind reranker; for BGE-M3-specific probes, use --model-kind m3.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).