AGENT SKILL

search-indexing

Operate and troubleshoot Gerev search, indexing, parsers, SQLite queues, ranking, and startup/status behavior.

License 许可
MIT
最近更新
2026-09-03
Stars
★ 351
Forks
27

01 前言 Front Matter

name
search-indexing
license
MIT
metadata
{ "disco-role": "operating" }
description
Operate and troubleshoot Gerev search, indexing, parsers, SQLite queues, ranking, and startup/status behavior.
disable-model-invocation
true

02 资源文件 3

03 SKILL.md 正文

Gerev Search and Indexing

Use this sub-skill when a task involves Gerev's search API, index maintenance, ranking stack, document parser helpers, SQLite-backed queues, or the startup/status behavior that makes search usable.

Best-fit tasks

  • Explain or debug GET /api/v1/search results, empty result sets, slow ranking, answer snippets, parent/child result nesting, or score interpretation.
  • Reason about BM25, Faiss, SentenceTransformer, CrossEncoder, and Transformers QA model interactions without opening the source checkout.
  • Inspect index persistence, SQLite queues, database/index file placement, and status counters.
  • Diagnose startup failures caused by model cache downloads, NLTK data, CUDA availability, storage permissions, singleton indexes, or the current PDF parser import defect.
  • Understand safe parser helper behavior for TXT, HTML, DOCX, PPTX, and PDF text extraction.

Start here

  1. Read search-indexing-stack.md for the end-to-end flow, data model, parser map, queue behavior, index persistence, and startup/status route semantics.

  2. For failures, use troubleshooting.md first. It covers cold Hugging Face caches, NLTK punkt/punkt_tab, CUDA fallback, storage path issues, stale indexes, queue stalls, and the known split_PDF_into_paragraphs import defect.

  3. When you have access to a Gerev checkout or storage directory, run the bundled inspect_search_indexing.py read-only helper from this sub-skill directory or with the equivalent path to the bundled script:

    python scripts/inspect_search_indexing.py --app-dir <checkout>/app --storage-dir <storage-dir> --sqlite-details
    

    Omit --storage-dir to let the helper infer Docker versus home storage. Add --json for machine-readable output or --strict when a CI-style nonzero exit is useful.

High-signal routing checks

  • Search import/startup touches model loading at import time through models.py; do not import the full app casually on a cold or offline host unless model caches are expected to exist.
  • search_logic.py calls nltk.download('punkt') at import time. A downloader warning can be nonfatal when tokenizer data is already present, but missing punkt or punkt_tab will break BM25 tokenization on modern NLTK.
  • app/main.py startup must create FaissIndex and Bm25Index singletons before search is used; direct calls to search_documents() without startup can raise "Index is not initialized".
  • The current source contains an app-startup blocker: indexing/index_documents.py imports split_PDF_into_paragraphs from parsers/pdf.py, but pdf.py defines only pdf_to_text and pdf_to_textV2. Treat this as a source defect to document or fix only when the user authorizes code changes.
  • Search/index storage is under /opt/storage in Docker-like mode and under a user home storage directory otherwise. Mismatched environment, user, or permissions can make the app appear to have no documents or no indexes.

Boundaries

Included here: search query flow, BM25/Faiss ranking, transformer model loading, NLTK data, parser helpers, SQLite index/task queues, indexing background loop, SQLAlchemy search schemas, and the search/status/startup parts of the FastAPI app.

Do not use this sub-skill for connector credential setup, connector-specific API behavior, Docker image build/release automation, or frontend UX changes except where those surfaces directly affect search/status interpretation.

04 同仓库更多 Skills

VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-triton-opt

Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-solve

END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-skill

PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-pattern-fusion

Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-orchestrate

Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-feedback

Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).

★ 351