AGENT SKILL

leann-index

Build, update, rebuild, watch, or list LEANN indexes. Use for `leann build`, `build-jsonl`, `rebuild`, `watch`, source readers, backend choice, incremental behavior, metadata stamping, folder filters, embedding cache, and AST-aware code indexing.

License 许可
MIT
所属仓库
zm2231/LEANN-fork
最近更新
2026-07-21
Stars
★ 0
Forks
0

01 前言 Front Matter

name
leann-index
description
Build, update, rebuild, watch, or list LEANN indexes. Use for `leann build`, `build-jsonl`, `rebuild`, `watch`, source readers, backend choice, incremental behavior, metadata stamping, folder filters, embedding cache, and AST-aware code indexing.

03 SKILL.md 正文

LEANN Indexing

Use this when creating or maintaining indexes under .leann/indexes/<name>/.

First Decision: Backend

Choose by corpus size and whether content is edited in place.

Backend Best use Incremental --docs Incremental build-jsonl --incremental-by-id
flat Small/medium edited corpora, exact recall Add/remove/modify in place Add/remove/modify in place
ivf Larger edited corpora, approximate FAISS search Add/remove/modify in place Add/remove/modify in place
hnsw Large mostly static corpus that fits in RAM Any docs change triggers full rebuild Add-only for non-compact indexes; changed/removed rows rebuild
diskann Very large disk-backed corpus Full rebuild Full rebuild
compact indexes Archived/read-only indexes No incremental No incremental

Default to flat --no-recompute for actively edited markdown/wiki/CRM/tool-doc corpora. Default to hnsw for large mostly static corpora. Use ivf when flat is too slow but remove/modify still matters.

Embeddings

Pick the embedding provider explicitly for reproducible builds. Examples:

--embedding-mode sentence-transformers \
--embedding-model facebook/contriever

For OpenAI-compatible embedding servers:

--embedding-mode openai \
--embedding-model <model-name> \
--embedding-api-base <base-url> \
--embedding-api-key <api-key>

If the server requires a non-empty placeholder key, pass one explicitly.

Build From Files

leann build docs --docs ./documents \
  --backend-name flat \
  --no-recompute \
  --embedding-mode sentence-transformers \
  --embedding-model facebook/contriever

build --docs preserves file/path metadata and stamps:

  • indexed_at
  • created_at
  • modified_at
  • source_root
  • relative_path
  • folder_path
  • top_folder

Folder Filters

Use --top-folder-depth when the first path segment is only a wrapper folder.

leann build docs --docs /path/to/docs \
  --top-folder-depth 2 \
  --backend-name flat \
  --no-recompute \
  --embedding-mode sentence-transformers \
  --embedding-model facebook/contriever

For Clients/BD/proposal.md with --top-folder-depth 2, metadata is:

  • relative_path = Clients/BD/proposal.md
  • folder_path = Clients/BD
  • source_root = <docs root>
  • top_folder = BD

Then search with:

leann search dropbox "proposal" \
  --metadata-filters '{"top_folder":{"==":"BD"}}'

Do not call this category in core LEANN. top_folder is factual filesystem metadata; app-specific category semantics belong in JSONL metadata or a source reader.

Build From JSONL

Use build-jsonl when the caller already has one row per searchable object and needs exact metadata preserved.

{"id":"workon","text":"workon switch project repo context","metadata":{"name":"workon","group":"project","tier":"hot"}}
leann build-jsonl code-mode-tools \
  --input .pi/code-mode-leann/tools.jsonl \
  --text-field text \
  --metadata-field metadata \
  --id-field id \
  --incremental-by-id \
  --backend-name flat \
  --no-recompute \
  --embedding-mode sentence-transformers \
  --embedding-model facebook/contriever

Rules:

  • Every row needs a unique stable --id-field.
  • If metadata.id exists, it must match the top-level ID.
  • LEANN writes the stable ID into metadata as id and source_document_id.
  • metadata.indexed_at is ignored for rowhashes, so timestamp refreshes do not re-embed unchanged rows.

Rebuild And Watch

leann rebuild docs
leann rebuild docs --force
leann watch docs --interval 30

rebuild replays stored build_config, including docs paths, JSONL input, chunking fields, backend, embedding flags, --top-folder-depth, and --incremental-by-id.

Use --force for a full rebuild. The content-addressed embedding cache keeps force rebuilds cheap when text is unchanged.

Code Indexes

leann build code --docs ./src --use-ast-chunking \
  --backend-name flat \
  --no-recompute \
  --embedding-mode sentence-transformers \
  --embedding-model facebook/contriever

AST chunking supports Python, Java, C#, TS/TSX/JS. Other languages fall back to token chunking. Verify by reading meta.json after build.

Source Readers

Specialized readers emit richer source metadata, including source_type, event_time, author, and source-specific IDs when available.

leann index-browser chrome
leann index-email
leann index-calendar --max-count 1000
leann index-imessage
leann index-chatgpt --export-path ~/chatgpt-export.zip
leann index-claude --export-path ~/claude-export.zip
leann index-wechat --export-dir ~/wechat-export

All index-* commands share the same embedding flags as leann build.

Important Flags

Input:

  • --docs PATH [PATH ...]
  • --file-types .md,.pdf,.txt
  • --include-hidden / --no-include-hidden
  • --top-folder-depth N

Backend:

  • --backend-name {hnsw,diskann,ivf,flat}
  • --compact / --no-compact
  • --recompute / --no-recompute
  • --force

Chunking:

  • --doc-chunk-size N, --doc-chunk-overlap N
  • --code-chunk-size N, --code-chunk-overlap N
  • --use-ast-chunking
  • --ast-chunk-size N, --ast-chunk-overlap N

JSONL:

  • --input FILE.jsonl
  • --text-field FIELD
  • --metadata-field FIELD
  • --id-field FIELD
  • --incremental-by-id

Default --docs extensions include documents (.txt, .md, .pdf, .docx, .pptx), common code/config/markup files, and .ipynb. PDFs get PyMuPDF/pdfplumber first, then generic fallback.

Embedding Cache

Build-time embeddings use ~/.leann/embed-cache.sqlite by default.

  • Override: LEANN_EMBED_CACHE_PATH=/path/cache.sqlite
  • Disable: LEANN_EMBED_CACHE=0
  • Force model revision key: LEANN_EMBED_MODEL_REV=<stable-revision>

Cache keys include canonicalization version, model, provider/mode, dimensions, normalization/pooling, prompt template, truncation settings, and model revision when known. Batch size and timeouts do not affect cache keys.

BM25/FTS5 is built and maintained by default. Do not pass a --prebuild-bm25 flag in this fork.

Shipped: --top-folder-depth

SHIP for the requested folder-filter change:

  • Added leann build --top-folder-depth in packages/leann-core/src/leann/cli.py.
  • Default is 1, preserving current behavior.
  • With --top-folder-depth 2, Clients/BD/proposal.md yields top_folder = BD.
  • Wired through build defaults, explicit option tracking, persisted build_config, rebuild, and watch replay.
  • Documented in docs/configuration-guide.md and docs/CHANGELOG.md.
  • Verified with uv run pytest tests/test_cli_build_config.py tests/test_indexed_at.py -q: 22 passed.
  • Verified with uv run ruff check packages/leann-core/src/leann/cli.py tests/test_cli_build_config.py tests/test_indexed_at.py.
  • Real CLI smoke confirmed --metadata-filters '{"top_folder":{"==":"BD"}}' returned only the BD document.

Common Pitfalls

  • HNSW is not the backend for edited docs. Use flat or ivf.
  • Compact indexes are read-only.
  • build --docs cannot preserve arbitrary app metadata; use build-jsonl for that.
  • Filesystem event_time and source_type are not present unless a source reader or JSONL producer emits them.
  • For Qwen/E5-style asymmetric models, pass both document and query prompt templates.

04 同仓库更多 Skills