AGENT SKILL

repo-development

Maintain a Pyserini source checkout, build developer resources, and choose focused tests without triggering heavyweight experiments.

License 许可
Apache-2.0
最近更新
2026-09-03
Stars
★ 351
Forks
27

01 前言 Front Matter

name
repo-development
license
Apache 2.0
metadata
{ "disco-role": "operating" }
description
Maintain a Pyserini source checkout, build developer resources, and choose focused tests without triggering heavyweight experiments.
disable-model-invocation
true

02 资源文件 4

03 SKILL.md 正文

Pyserini Repo Development

Use this sub-skill when the user is contributing to Pyserini from a source checkout, fixing tests, preparing an editable install, initializing submodules, building Anserini/evaluation resources, updating docs or prebuilt-index metadata, or deciding which tests are safe for a focused change.

Do not use this sub-skill for normal PyPI installation, user-facing indexing/search/fetch, dense encoding/Faiss workflows, run evaluation/fusion usage, REST/MCP deployment, or public API recipes. Route those tasks to ../install-and-runtime/SKILL.md, ../index-search-fetch/SKILL.md, ../dense-encoding/SKILL.md, ../evaluation-and-fusion/SKILL.md, or ../serving-and-agent-tools/SKILL.md.

Quick Development Flow

  1. Confirm the checkout shape before installing:

    git status --short
    git submodule status tools
    python --version
    java -version
    
  2. If tools is missing or uninitialized, initialize only the expected submodule:

    git submodule update --init --recursive tools
    
  3. Install in editable mode inside a Python 3.12 environment with JDK 21 available:

    python -m pip install --upgrade pip
    python -m pip install -e .
    python -m pip check
    python -c "import pyserini; print('editable import ok')"
    
  4. Build local developer resources only when the target tests need them:

    (cd tools/eval && tar xvfz trec_eval.9.0.4.tar.gz && cd trec_eval.9.0.4 && make)
    (cd tools/eval/ndeval && make)
    mkdir -p pyserini/resources/jars
    cp ../anserini/target/anserini-*-fatjar.jar pyserini/resources/jars/
    

    The fatjar copy assumes a sibling Anserini checkout that has already been built. If the user keeps the fatjar elsewhere, set ANSERINI_CLASSPATH for the current process before any PyJNIus/JVM import starts.

  5. Select tests with the bundled helper before running native tests:

    python scripts/select_safe_tests.py --help
    python scripts/select_safe_tests.py --paths pyserini/server/config.py tests/core/test_server_config.py
    python scripts/select_safe_tests.py --category checkout --include-bounded
    
  6. Run the smallest matching commands first. Do not run broad tests, integrations, scripts/jobs*.txt, prebuilt-index downloads, model encoders, or two-click reproduction matrices unless the user explicitly approves heavyweight/network work.

Maintainer Decision Rules

  • Source checkout vs PyPI: Use editable source installs only for maintainer work or unreleased code. If a user just needs retrieval, prefer the normal runtime install flow.
  • Submodule: tools/ is a Git submodule for evaluation tools and topics/qrels resources. Missing files under tools/eval or tools/topics-and-qrels usually mean the submodule was not initialized.
  • Fatjar: Lucene-backed imports need an anserini-*-fatjar.jar in Pyserini package resources or on ANSERINI_CLASSPATH before the JVM starts.
  • Eval tools: Native eval tests may need built trec_eval and ndeval binaries from the tools submodule; Python import success alone is not enough.
  • Docs/resources: Updating generated docs or prebuilt-index metadata can intentionally rewrite files. Run those checks only when the task is about generated docs and review the resulting diff.
  • Jobs manifests: Treat scripts/jobs.docs-all.txt, scripts/jobs.integrations-all.txt, and scripts/jobs.regressions-all.txt as maintainer manifests, not default validation commands.

What to Read Next

  • references/development-workflows.md for source checkout, editable install, submodule, eval-tool, fatjar, docs, and job-manifest workflows.
  • references/test-selection.md for safe test categories, native candidate commands, and skip rules.
  • references/troubleshooting.md for missing submodules, missing jars, Java/Python mismatches, eval-tool failures, and accidental network/download tests.
  • scripts/select_safe_tests.py --help for a deterministic command selector that is safe by default.

Handoff Checklist

  • Confirm Python 3.12, JDK 21, editable install, and pip check before blaming Pyserini code.
  • Check tools submodule state before running eval, qrels, or topic tests.
  • Check the Anserini fatjar before running Lucene-backed imports or tests.
  • Use exact test classes/methods instead of whole directories when the change touches one API surface.
  • Record skipped heavyweight tests separately with the reason: network download, model download, optional GPU/Faiss dependency, broad integration, or mutating generated docs.

04 同仓库更多 Skills

VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-triton-opt

Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-solve

END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-skill

PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-pattern-fusion

Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-orchestrate

Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-feedback

Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).

★ 351