prompt-and-inference
Build and debug OpenCompass prompt templates, PromptList dialogue prompts, retrievers, Gen/PPL inferencers, reader_cfg answer masking, in-context examples, and meta_template role routing.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- prompt-and-inference
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Build and debug OpenCompass prompt templates, PromptList dialogue prompts, retrievers, Gen/PPL inferencers, reader_cfg answer masking, in-context examples, and meta_template role routing.
- disable-model-invocation
- true
02 资源文件 4
- 参考文档references/prompt-templates.md6.5 KB
- 参考文档references/retrievers-and-inferencers.md5.4 KB
- 参考文档references/troubleshooting.md5.6 KB
- 脚本scripts/render_prompt_preview.py7.7 KB
03 SKILL.md 正文
OpenCompass Prompt and Inference
Use this sub-skill when an OpenCompass task involves prompt construction or inference config behavior: PromptTemplate, RawPromptTemplate, PromptList dialogue turns, ice_template, prompt_template, retrievers, GenInferencer, PPLInferencer, output-column masking, or meta_template role mapping.
Route the Task
- For prompt syntax, answer masking, missing literal fields, dialogue
PromptList, raw messages, or CoT wording, usereferences/prompt-templates.md. - For choosing
ZeroRetriever/few-shot retrievers and wiringGenInferencervsPPLInferencer, usereferences/retrievers-and-inferencers.md. - For symptoms such as literal
{answer}, leaked labels, missing system role handling, unsupported PPL candidate templates, or train split failures, usereferences/troubleshooting.md. - For a no-download local preview of tiny samples and templates, run
python scripts/render_prompt_preview.py --helpfrom this sub-skill directory.
High-Value Defaults
- Use
GenInferencerfor free-form answers and keep the dataset answer column inreader_cfg.output_columnso it is replaced bygen_field_replace_tokeninstead of leaking into the generation prompt. - Use
PPLInferencerfor multiple-choice/discriminative tasks and provide candidate templates keyed by label ("A","B", etc.) inprompt_template.templateorice_template.template. - Use
ZeroRetrieverfor 0-shot; useFixKRetrieveronly when the selected in-context examples exist in the dataset train/index split. - Put model-specific chat wrappers in the model
meta_template; keep dataset prompt templates model-agnostic and role-based (HUMAN,BOT,SYSTEM).
Boundaries
- Route dataset class creation,
reader_cfgsource columns, and config file structure toconfiguration-and-datasets. - Route tokenizer behavior, model chat templates, backend acceleration, and real HF/API inference issues to
model-backends. - Do not claim real model inference was verified from this sub-skill; the bundled script renders prompt construction only.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).