rag-document-processing
Guides LazyLLM RAG, document ingestion, readers, transforms, retrievers, rerankers, stores, BM25, and parser-service workflows.
- License 许可
- Apache-2.0
- 最近更新
- 2026-10-08
- Stars
- ★ 11
- Forks
- 3
01 前言 Front Matter
- name
- rag-document-processing
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Guides LazyLLM RAG, document ingestion, readers, transforms, retrievers, rerankers, stores, BM25, and parser-service workflows.
- disable-model-invocation
- true
02 资源文件 3
- 参考文档references/data-and-service-formats.md2.6 KB
- 参考文档references/rag-workflows.md3.1 KB
- 参考文档references/troubleshooting.md2.3 KB
03 SKILL.md 正文
LazyLLM RAG and Document Processing
Use this sub-skill for LazyLLM tasks involving Document, DocNode, document readers, transforms, node groups, retrievers, rerankers, BM25, stores/indexes, document service metadata, parser service, Milvus/OpenSearch, or RAG examples.
Start here when
- The task mentions
lazyllm.tools.rag,Document,Retriever,Reranker,DocNode, node groups, chunking, embedding, reranking, or BM25. - An import error says to run
lazyllm install rag. - The user wants a local RAG app, parser-service architecture, URL/file retriever, map store, Milvus/OpenSearch index, or RAG with online/local model modules.
- You need to separate local text retrieval from optional embedding/vector database/model backends.
Files to read
- rag-workflows.md for local RAG recipes, object relationships, and backend escalation.
- data-and-service-formats.md for document/node/service request shapes and parser-service boundaries.
- troubleshooting.md for missing dependencies, parser service, vector DB, and retrieval issues.
- scripts/rag_bm25_smoke.py for a no-network BM25 retrieval check.
Safe RAG workflow
- Install/check the RAG extra when importing RAG objects:
lazyllm install rag python ../../scripts/check_lazyllm_env.py --require-rag - Start local and text-only. Use
DocNodeand BM25 or local readers before adding embeddings, rerankers, vector DBs, or model modules. - Decide ownership of model nodes. Model modules used for embeddings/reranking/chat belong to model-deployment for backend checks.
- Decide service boundaries. Parser service, document service, Milvus/OpenSearch/Redis, object storage, and external URLs require service credentials or local fixtures.
- Run local smoke.
python scripts/rag_bm25_smoke.py
Key public surfaces
Document(*args, **kw)is the main RAG container and index manager.Retriever(doc, group_name, similarity=None, similarity_cut_off=-inf, index='default', topk=6, embed_keys=None, target=None, output_format=None, join=False, weight=None, priority=None, **kwargs)pulls contexts from a document/index.Reranker(name='ModuleReranker', *args, **kwargs)reranks retrieved contexts and may depend on a model/backend.DocNoderepresents text chunks with metadata and relationships.- BM25 supports local English/Chinese retrieval with no embedding server.
Backend posture
- Required for local guidance: base LazyLLM plus
ragextra. - Safe CPU candidates: BM25, document/node transforms, reader utilities, retriever logic, local doc-service manager with SQLite/mocked parser client.
- Optional external services: Milvus, OpenSearch/Elasticsearch, Redis/RedisVL, parser service workers, object storage, remote URLs.
- Optional model/GPU: embedding models, rerankers, local LLM answer generation, multimodal OCR/audio ingestion.
Handoff checklist
When you complete a RAG task, report:
- document source type and data format,
- node/chunking/grouping plan,
- retrieval method and index/store backend,
- model/embedding/reranker backend status,
- local smoke command or reason it is optional/skipped,
- clear separation between local artifacts and external services.
04 同仓库更多 Skills
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
obsidian-clipper-template-creator
Guide for creating templates for the Obsidian Web Clipper. Use when you want to create a new clipping template, understand available variables, or format clipped content.
python-api
Programmatic Object-Detection-Metrics API guidance for BoundingBox, BoundingBoxes, Evaluator, enums, coordinate conversion, AP/mAP semantics, and source-style import diagnostics.
file-evaluation
Folder-based PASCAL VOC AP/mAP evaluation from ground-truth and detection text files.
object-detection-metrics
Guides legacy Object-Detection-Metrics PASCAL VOC AP/mAP evaluation from detection text folders and direct Python API classes.
nx-workspace-patterns
Configure and optimize Nx monorepo workspaces. Use when setting up Nx, configuring project boundaries, optimizing build caching, or implementing affected commands.