repo-development
Maintain a Pyserini source checkout, build developer resources, and choose focused tests without triggering heavyweight experiments.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- repo-development
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Maintain a Pyserini source checkout, build developer resources, and choose focused tests without triggering heavyweight experiments.
- disable-model-invocation
- true
02 资源文件 4
- 参考文档references/development-workflows.md6.2 KB
- 参考文档references/test-selection.md6.3 KB
- 参考文档references/troubleshooting.md6.0 KB
- 脚本scripts/select_safe_tests.py20.1 KB
03 SKILL.md 正文
Pyserini Repo Development
Use this sub-skill when the user is contributing to Pyserini from a source checkout, fixing tests, preparing an editable install, initializing submodules, building Anserini/evaluation resources, updating docs or prebuilt-index metadata, or deciding which tests are safe for a focused change.
Do not use this sub-skill for normal PyPI installation, user-facing indexing/search/fetch, dense encoding/Faiss workflows, run evaluation/fusion usage, REST/MCP deployment, or public API recipes. Route those tasks to ../install-and-runtime/SKILL.md, ../index-search-fetch/SKILL.md, ../dense-encoding/SKILL.md, ../evaluation-and-fusion/SKILL.md, or ../serving-and-agent-tools/SKILL.md.
Quick Development Flow
-
Confirm the checkout shape before installing:
git status --short git submodule status tools python --version java -version -
If
toolsis missing or uninitialized, initialize only the expected submodule:git submodule update --init --recursive tools -
Install in editable mode inside a Python 3.12 environment with JDK 21 available:
python -m pip install --upgrade pip python -m pip install -e . python -m pip check python -c "import pyserini; print('editable import ok')" -
Build local developer resources only when the target tests need them:
(cd tools/eval && tar xvfz trec_eval.9.0.4.tar.gz && cd trec_eval.9.0.4 && make) (cd tools/eval/ndeval && make) mkdir -p pyserini/resources/jars cp ../anserini/target/anserini-*-fatjar.jar pyserini/resources/jars/The fatjar copy assumes a sibling Anserini checkout that has already been built. If the user keeps the fatjar elsewhere, set
ANSERINI_CLASSPATHfor the current process before any PyJNIus/JVM import starts. -
Select tests with the bundled helper before running native tests:
python scripts/select_safe_tests.py --help python scripts/select_safe_tests.py --paths pyserini/server/config.py tests/core/test_server_config.py python scripts/select_safe_tests.py --category checkout --include-bounded -
Run the smallest matching commands first. Do not run broad
tests,integrations,scripts/jobs*.txt, prebuilt-index downloads, model encoders, or two-click reproduction matrices unless the user explicitly approves heavyweight/network work.
Maintainer Decision Rules
- Source checkout vs PyPI: Use editable source installs only for maintainer work or unreleased code. If a user just needs retrieval, prefer the normal runtime install flow.
- Submodule:
tools/is a Git submodule for evaluation tools and topics/qrels resources. Missing files undertools/evalortools/topics-and-qrelsusually mean the submodule was not initialized. - Fatjar: Lucene-backed imports need an
anserini-*-fatjar.jarin Pyserini package resources or onANSERINI_CLASSPATHbefore the JVM starts. - Eval tools: Native eval tests may need built
trec_evalandndevalbinaries from thetoolssubmodule; Python import success alone is not enough. - Docs/resources: Updating generated docs or prebuilt-index metadata can intentionally rewrite files. Run those checks only when the task is about generated docs and review the resulting diff.
- Jobs manifests: Treat
scripts/jobs.docs-all.txt,scripts/jobs.integrations-all.txt, andscripts/jobs.regressions-all.txtas maintainer manifests, not default validation commands.
What to Read Next
references/development-workflows.mdfor source checkout, editable install, submodule, eval-tool, fatjar, docs, and job-manifest workflows.references/test-selection.mdfor safe test categories, native candidate commands, and skip rules.references/troubleshooting.mdfor missing submodules, missing jars, Java/Python mismatches, eval-tool failures, and accidental network/download tests.scripts/select_safe_tests.py --helpfor a deterministic command selector that is safe by default.
Handoff Checklist
- Confirm Python 3.12, JDK 21, editable install, and
pip checkbefore blaming Pyserini code. - Check
toolssubmodule state before running eval, qrels, or topic tests. - Check the Anserini fatjar before running Lucene-backed imports or tests.
- Use exact test classes/methods instead of whole directories when the change touches one API surface.
- Record skipped heavyweight tests separately with the reason: network download, model download, optional GPU/Faiss dependency, broad integration, or mutating generated docs.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).