search-indexing
Operate and troubleshoot Gerev search, indexing, parsers, SQLite queues, ranking, and startup/status behavior.
- License 许可
- MIT
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- search-indexing
- license
- MIT
- metadata
- { "disco-role": "operating" }
- description
- Operate and troubleshoot Gerev search, indexing, parsers, SQLite queues, ranking, and startup/status behavior.
- disable-model-invocation
- true
02 资源文件 3
- 参考文档references/search-indexing-stack.md11.1 KB
- 参考文档references/troubleshooting.md7.6 KB
- 脚本scripts/inspect_search_indexing.py12.8 KB
03 SKILL.md 正文
Gerev Search and Indexing
Use this sub-skill when a task involves Gerev's search API, index maintenance, ranking stack, document parser helpers, SQLite-backed queues, or the startup/status behavior that makes search usable.
Best-fit tasks
- Explain or debug
GET /api/v1/searchresults, empty result sets, slow ranking, answer snippets, parent/child result nesting, or score interpretation. - Reason about BM25, Faiss, SentenceTransformer, CrossEncoder, and Transformers QA model interactions without opening the source checkout.
- Inspect index persistence, SQLite queues, database/index file placement, and status counters.
- Diagnose startup failures caused by model cache downloads, NLTK data, CUDA availability, storage permissions, singleton indexes, or the current PDF parser import defect.
- Understand safe parser helper behavior for TXT, HTML, DOCX, PPTX, and PDF text extraction.
Start here
-
Read
search-indexing-stack.mdfor the end-to-end flow, data model, parser map, queue behavior, index persistence, and startup/status route semantics. -
For failures, use
troubleshooting.mdfirst. It covers cold Hugging Face caches, NLTKpunkt/punkt_tab, CUDA fallback, storage path issues, stale indexes, queue stalls, and the knownsplit_PDF_into_paragraphsimport defect. -
When you have access to a Gerev checkout or storage directory, run the bundled
inspect_search_indexing.pyread-only helper from this sub-skill directory or with the equivalent path to the bundled script:python scripts/inspect_search_indexing.py --app-dir <checkout>/app --storage-dir <storage-dir> --sqlite-detailsOmit
--storage-dirto let the helper infer Docker versus home storage. Add--jsonfor machine-readable output or--strictwhen a CI-style nonzero exit is useful.
High-signal routing checks
- Search import/startup touches model loading at import time through
models.py; do not import the full app casually on a cold or offline host unless model caches are expected to exist. search_logic.pycallsnltk.download('punkt')at import time. A downloader warning can be nonfatal when tokenizer data is already present, but missingpunktorpunkt_tabwill break BM25 tokenization on modern NLTK.app/main.pystartup must createFaissIndexandBm25Indexsingletons before search is used; direct calls tosearch_documents()without startup can raise "Index is not initialized".- The current source contains an app-startup blocker:
indexing/index_documents.pyimportssplit_PDF_into_paragraphsfromparsers/pdf.py, butpdf.pydefines onlypdf_to_textandpdf_to_textV2. Treat this as a source defect to document or fix only when the user authorizes code changes. - Search/index storage is under
/opt/storagein Docker-like mode and under a user home storage directory otherwise. Mismatched environment, user, or permissions can make the app appear to have no documents or no indexes.
Boundaries
Included here: search query flow, BM25/Faiss ranking, transformer model loading, NLTK data, parser helpers, SQLite index/task queues, indexing background loop, SQLAlchemy search schemas, and the search/status/startup parts of the FastAPI app.
Do not use this sub-skill for connector credential setup, connector-specific API behavior, Docker image build/release automation, or frontend UX changes except where those surfaces directly affect search/status interpretation.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).