AGENT SKILL

rag-document-processing

Guides LazyLLM RAG, document ingestion, readers, transforms, retrievers, rerankers, stores, BM25, and parser-service workflows.

License 许可
Apache-2.0
最近更新
2026-10-08
Stars
★ 11
Forks
3

01 前言 Front Matter

name
rag-document-processing
license
Apache 2.0
metadata
{ "disco-role": "operating" }
description
Guides LazyLLM RAG, document ingestion, readers, transforms, retrievers, rerankers, stores, BM25, and parser-service workflows.
disable-model-invocation
true

02 资源文件 3

03 SKILL.md 正文

LazyLLM RAG and Document Processing

Use this sub-skill for LazyLLM tasks involving Document, DocNode, document readers, transforms, node groups, retrievers, rerankers, BM25, stores/indexes, document service metadata, parser service, Milvus/OpenSearch, or RAG examples.

Start here when

  • The task mentions lazyllm.tools.rag, Document, Retriever, Reranker, DocNode, node groups, chunking, embedding, reranking, or BM25.
  • An import error says to run lazyllm install rag.
  • The user wants a local RAG app, parser-service architecture, URL/file retriever, map store, Milvus/OpenSearch index, or RAG with online/local model modules.
  • You need to separate local text retrieval from optional embedding/vector database/model backends.

Files to read

Safe RAG workflow

  1. Install/check the RAG extra when importing RAG objects:
    lazyllm install rag
    python ../../scripts/check_lazyllm_env.py --require-rag
    
  2. Start local and text-only. Use DocNode and BM25 or local readers before adding embeddings, rerankers, vector DBs, or model modules.
  3. Decide ownership of model nodes. Model modules used for embeddings/reranking/chat belong to model-deployment for backend checks.
  4. Decide service boundaries. Parser service, document service, Milvus/OpenSearch/Redis, object storage, and external URLs require service credentials or local fixtures.
  5. Run local smoke.
    python scripts/rag_bm25_smoke.py
    

Key public surfaces

  • Document(*args, **kw) is the main RAG container and index manager.
  • Retriever(doc, group_name, similarity=None, similarity_cut_off=-inf, index='default', topk=6, embed_keys=None, target=None, output_format=None, join=False, weight=None, priority=None, **kwargs) pulls contexts from a document/index.
  • Reranker(name='ModuleReranker', *args, **kwargs) reranks retrieved contexts and may depend on a model/backend.
  • DocNode represents text chunks with metadata and relationships.
  • BM25 supports local English/Chinese retrieval with no embedding server.

Backend posture

  • Required for local guidance: base LazyLLM plus rag extra.
  • Safe CPU candidates: BM25, document/node transforms, reader utilities, retriever logic, local doc-service manager with SQLite/mocked parser client.
  • Optional external services: Milvus, OpenSearch/Elasticsearch, Redis/RedisVL, parser service workers, object storage, remote URLs.
  • Optional model/GPU: embedding models, rerankers, local LLM answer generation, multimodal OCR/audio ingestion.

Handoff checklist

When you complete a RAG task, report:

  • document source type and data format,
  • node/chunking/grouping plan,
  • retrieval method and index/store backend,
  • model/embedding/reranker backend status,
  • local smoke command or reason it is optional/skipped,
  • clear separation between local artifacts and external services.

04 同仓库更多 Skills