[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EM1DC7FXH6ZQCF9XYJDFA8":3,"skill-same:01M6EM1DC7FXH6ZQCF9XYJDFA8":87},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":44,"commitSha":45,"skillDirectory":46,"forksCount":47,"repositoryHtmlUrl":48,"repositoryHomepageUrl":49,"ownerHtmlUrl":50,"ownerBlogUrl":6,"ownerDescription":51,"updatedAt":52,"install":53,"resources":66},"01M6EM1DC7FXH6ZQCF9XYJDFA8","building-rag-systems",null,"Build production RAG systems with semantic chunking, incremental indexing, and filtered retrieval.\nUse when implementing document ingestion pipelines, vector search with Qdrant, or context-aware\nretrieval. Covers chunking strategies, change detection, payload indexing, and context expansion.\nNOT when doing simple similarity search without production requirements.","ComeOnOliver\u002Fskillshub","ComeOnOliver","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F43215937?v=4","01M6E5JNFECJAAVFE0W7SMWWD2","Haoran Shu","01M6E5JNNDQS7AJZDYFQ8TPZCK",67,"MIT","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fasmayaseen\u002Fbuilding-rag-systems\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"rag","building rag systems","systems","strategies","change","Document Ingestion","change detection","similarity","search","indexing","requirements","chunking","vector","build","document","simple","ingestion","payload","use","context","2026-11-10 10:12:30",false,"---\r\nname: building-rag-systems\r\ndescription: |\r\n  Build production RAG systems with semantic chunking, incremental indexing, and filtered retrieval.\r\n  Use when implementing document ingestion pipelines, vector search with Qdrant, or context-aware\r\n  retrieval. Covers chunking strategies, change detection, payload indexing, and context expansion.\r\n  NOT when doing simple similarity search without production requirements.\r\n---\r\n\r\n# Building RAG Systems\r\n\r\nProduction-grade RAG with semantic chunking, incremental updates, and filtered retrieval.\r\n\r\n## Quick Start\r\n\r\n```bash\r\n# Dependencies\r\npip install qdrant-client openai pydantic python-frontmatter\r\n\r\n# Core components\r\n# 1. Crawler → discovers files, extracts path metadata\r\n# 2. Parser → extracts frontmatter, computes file hash\r\n# 3. Chunker → semantic split on ## headers, 400 tokens, 15% overlap\r\n# 4. Embedder → batched OpenAI embeddings\r\n# 5. Uploader → Qdrant upsert with indexed payloads\r\n```\r\n\r\n---\r\n\r\n## Ingestion Pipeline\r\n\r\n### Architecture\r\n\r\n```\r\n┌──────────┐    ┌────────┐    ┌─────────┐    ┌──────────┐    ┌──────────┐\r\n│ Crawler  │ -> │ Parser │ -> │ Chunker │ -> │ Embedder │ -> │ Uploader │\r\n└──────────┘    └────────┘    └─────────┘    └──────────┘    └──────────┘\r\n     │              │              │              │              │\r\nDiscovers      Extracts       Splits by     Generates      Upserts to\r\nfiles          frontmatter    semantic      vectors        Qdrant\r\n               + file hash    boundaries    (batched)      (batched)\r\n```\r\n\r\n### Semantic Chunking (NOT Fixed-Size)\r\n\r\n```python\r\nclass SemanticChunker:\r\n    \"\"\"\r\n    Production chunking:\r\n    - Split on ## headers (semantic boundaries)\r\n    - Target 400 tokens (NVIDIA benchmark optimal)\r\n    - 15% overlap for context continuity\r\n    - Track prev\u002Fnext for context expansion\r\n    \"\"\"\r\n    SECTION_PATTERN = re.compile(r\"(?=^## )\", re.MULTILINE)\r\n    TOKENS_PER_WORD = 1.3\r\n\r\n    def __init__(\r\n        self,\r\n        target_tokens: int = 400,\r\n        max_tokens: int = 512,\r\n        overlap_percent: float = 0.15,\r\n    ):\r\n        self.target_words = int(target_tokens \u002F self.TOKENS_PER_WORD)\r\n        self.overlap_words = int(self.target_words * overlap_percent)\r\n\r\n    def chunk(self, content: str, file_hash: str) -> list[Chunk]:\r\n        sections = self.SECTION_PATTERN.split(content)\r\n        chunks = []\r\n\r\n        for idx, section in enumerate(sections):\r\n            content_hash = hashlib.sha256(section.encode()).hexdigest()[:16]\r\n            chunk_id = f\"{file_hash[:8]}_{content_hash}_{idx}\"\r\n\r\n            chunks.append(Chunk(\r\n                id=chunk_id,\r\n                text=section,\r\n                chunk_index=idx,\r\n                total_chunks=len(sections),\r\n                prev_chunk_id=chunks[-1].id if chunks else None,\r\n                content_hash=content_hash,\r\n                source_file_hash=file_hash,\r\n            ))\r\n\r\n            # Set next_chunk_id on previous\r\n            if len(chunks) > 1:\r\n                chunks[-2].next_chunk_id = chunk_id\r\n\r\n        return chunks\r\n```\r\n\r\n### Change Detection (Incremental Updates)\r\n\r\n```python\r\ndef compute_file_hash(file_path: str) -> str:\r\n    \"\"\"SHA-256 for change detection.\"\"\"\r\n    with open(file_path, 'rb') as f:\r\n        return hashlib.sha256(f.read()).hexdigest()\r\n\r\nclass QdrantStateTracker:\r\n    \"\"\"Query Qdrant payloads directly - no external state DB needed.\"\"\"\r\n\r\n    def get_indexed_files(self, book_id: str) -> dict[str, str]:\r\n        \"\"\"Returns {file_path: file_hash} from Qdrant.\"\"\"\r\n        indexed = {}\r\n        offset = None\r\n\r\n        while True:\r\n            points, next_offset = self.client.scroll(\r\n                collection_name=self.collection,\r\n                scroll_filter=Filter(must=[\r\n                    FieldCondition(key=\"book_id\", match=MatchValue(value=book_id))\r\n                ]),\r\n                limit=100,\r\n                offset=offset,\r\n                with_payload=[\"source_file\", \"source_file_hash\"],\r\n                with_vectors=False,\r\n            )\r\n\r\n            for point in points:\r\n                indexed[point.payload[\"source_file\"]] = point.payload[\"source_file_hash\"]\r\n\r\n            if next_offset is None:\r\n                break\r\n            offset = next_offset\r\n\r\n        return indexed\r\n\r\n    def detect_changes(self, current: dict[str, str], indexed: dict[str, str]):\r\n        \"\"\"Compare filesystem vs index.\"\"\"\r\n        new = [p for p in current if p not in indexed]\r\n        deleted = [p for p in indexed if p not in current]\r\n        modified = [p for p in current if p in indexed and current[p] != indexed[p]]\r\n        return new, modified, deleted\r\n```\r\n\r\n### Batched Embeddings\r\n\r\n```python\r\nclass OpenAIEmbedder:\r\n    def __init__(self, model: str = \"text-embedding-3-small\", batch_size: int = 20):\r\n        self.client = OpenAI()\r\n        self.model = model\r\n        self.batch_size = batch_size  # OpenAI recommendation\r\n\r\n    def embed_chunks(self, chunks: list[Chunk]) -> list[EmbeddedChunk]:\r\n        embedded = []\r\n        for i in range(0, len(chunks), self.batch_size):\r\n            batch = chunks[i:i + self.batch_size]\r\n            response = self.client.embeddings.create(\r\n                input=[c.text for c in batch],\r\n                model=self.model,\r\n            )\r\n            for chunk, data in zip(batch, response.data):\r\n                embedded.append(EmbeddedChunk(**chunk.dict(), embedding=data.embedding))\r\n        return embedded\r\n```\r\n\r\n### Qdrant Collection with Payload Indexes\r\n\r\n```python\r\ndef create_collection(self, recreate: bool = False):\r\n    \"\"\"Create collection with proper indexes for filtered retrieval.\"\"\"\r\n    self.client.create_collection(\r\n        collection_name=self.collection,\r\n        vectors_config=VectorParams(size=1536, distance=Distance.COSINE),\r\n    )\r\n\r\n    # Index ALL fields you filter by\r\n    indexes = [\r\n        (\"book_id\", PayloadSchemaType.KEYWORD),      # Tenant isolation\r\n        (\"module\", PayloadSchemaType.KEYWORD),       # Content filter\r\n        (\"chapter\", PayloadSchemaType.INTEGER),      # Range filter\r\n        (\"hardware_tier\", PayloadSchemaType.INTEGER),# Personalization\r\n        (\"proficiency_level\", PayloadSchemaType.KEYWORD),\r\n        (\"parent_doc_id\", PayloadSchemaType.KEYWORD),# Context expansion\r\n        (\"source_file_hash\", PayloadSchemaType.KEYWORD),  # Change detection\r\n    ]\r\n\r\n    for field, schema in indexes:\r\n        self.client.create_payload_index(\r\n            collection_name=self.collection,\r\n            field_name=field,\r\n            field_schema=schema,\r\n        )\r\n```\r\n\r\n---\r\n\r\n## Retrieval Patterns\r\n\r\n### Comprehensive Filter Builder\r\n\r\n```python\r\ndef build_filter(self, query: SearchQuery) -> Filter:\r\n    \"\"\"Build Qdrant filter with all conditions (AND logic).\"\"\"\r\n    conditions = []\r\n\r\n    # Required: Tenant isolation\r\n    conditions.append(FieldCondition(\r\n        key=\"book_id\", match=MatchValue(value=query.book_id)\r\n    ))\r\n\r\n    # Required: Hardware tier (lte = \"tier X or lower\")\r\n    conditions.append(FieldCondition(\r\n        key=\"hardware_tier\", range=Range(lte=query.hardware_tier)\r\n    ))\r\n\r\n    # Optional: Module exact match\r\n    if query.module:\r\n        conditions.append(FieldCondition(\r\n            key=\"module\", match=MatchValue(value=query.module)\r\n        ))\r\n\r\n    # Optional: Chapter range\r\n    if query.chapter_min or query.chapter_max:\r\n        chapter_range = Range()\r\n        if query.chapter_min:\r\n            chapter_range.gte = query.chapter_min\r\n        if query.chapter_max:\r\n            chapter_range.lte = query.chapter_max\r\n        conditions.append(FieldCondition(key=\"chapter\", range=chapter_range))\r\n\r\n    # Optional: Proficiency OR logic\r\n    if query.proficiency_levels:\r\n        conditions.append(FieldCondition(\r\n            key=\"proficiency_level\",\r\n            match=MatchAny(any=query.proficiency_levels),\r\n        ))\r\n\r\n    return Filter(must=conditions)\r\n```\r\n\r\n### Context Expansion (Walk Chunk Chain)\r\n\r\n```python\r\ndef expand_context(self, chunk_id: str, prev: int = 1, next: int = 1) -> list[Chunk]:\r\n    \"\"\"Walk prev_chunk_id\u002Fnext_chunk_id chain for surrounding context.\"\"\"\r\n    current = self.get_chunk_by_id(chunk_id)\r\n    if not current:\r\n        return []\r\n\r\n    # Walk backwards\r\n    prev_chunks = []\r\n    prev_id = current.prev_chunk_id\r\n    for _ in range(prev):\r\n        if not prev_id:\r\n            break\r\n        chunk = self.get_chunk_by_id(prev_id)\r\n        if not chunk:\r\n            break\r\n        prev_chunks.insert(0, chunk)\r\n        prev_id = chunk.prev_chunk_id\r\n\r\n    # Walk forwards\r\n    next_chunks = []\r\n    next_id = current.next_chunk_id\r\n    for _ in range(next):\r\n        if not next_id:\r\n            break\r\n        chunk = self.get_chunk_by_id(next_id)\r\n        if not chunk:\r\n            break\r\n        next_chunks.append(chunk)\r\n        next_id = chunk.next_chunk_id\r\n\r\n    return prev_chunks + [current] + next_chunks\r\n```\r\n\r\n### Full Document Retrieval\r\n\r\n```python\r\ndef get_document_chunks(self, parent_doc_id: str) -> list[Chunk]:\r\n    \"\"\"Get all chunks for a document, ordered by chunk_index.\"\"\"\r\n    points, _ = self.client.scroll(\r\n        collection_name=self.collection,\r\n        scroll_filter=Filter(must=[\r\n            FieldCondition(key=\"parent_doc_id\", match=MatchValue(value=parent_doc_id))\r\n        ]),\r\n        limit=100,\r\n        with_payload=True,\r\n        with_vectors=False,\r\n    )\r\n\r\n    chunks = [self._to_chunk(p) for p in points]\r\n    chunks.sort(key=lambda c: c.chunk_index)\r\n    return chunks\r\n```\r\n\r\n---\r\n\r\n## Payload Schema\r\n\r\n```python\r\nclass ChunkPayload(BaseModel):\r\n    \"\"\"Complete payload for filtered retrieval and context expansion.\"\"\"\r\n\r\n    # Tenant isolation\r\n    book_id: str\r\n\r\n    # Content filters (all indexed)\r\n    module: str\r\n    chapter: int\r\n    lesson: int\r\n    hardware_tier: int\r\n    proficiency_level: str\r\n\r\n    # Display content\r\n    text: str\r\n    section_title: Optional[str]\r\n    source_file: str\r\n\r\n    # Context expansion\r\n    parent_doc_id: str\r\n    chunk_index: int\r\n    total_chunks: int\r\n    prev_chunk_id: Optional[str]\r\n    next_chunk_id: Optional[str]\r\n\r\n    # Change detection\r\n    content_hash: str\r\n    source_file_hash: str\r\n```\r\n\r\n---\r\n\r\n## Anti-Patterns\r\n\r\n| Don't | Do Instead |\r\n|-------|------------|\r\n| Fixed character chunking | Semantic boundaries (## headers) |\r\n| Position-based chunk IDs | Content hash for stable IDs |\r\n| No overlap between chunks | 10-20% overlap for continuity |\r\n| Full re-index on every change | Incremental with file hash detection |\r\n| Missing payload indexes | Index every field you filter by |\r\n| Synchronous embedding | Batch with background jobs |\r\n| External state database | Qdrant-native state tracking |\r\n\r\n---\r\n\r\n## Verification\r\n\r\nRun: `python scripts\u002Fverify.py`\r\n\r\n## Related Skills\r\n\r\n- `scaffolding-fastapi-dapr` - API patterns for search endpoints\r\n- `streaming-llm-responses` - Streaming RAG responses\r\n\r\n## References\r\n\r\n- [references\u002Fingestion-patterns.md](references\u002Fingestion-patterns.md) - Full ingestion pipeline\r\n- [references\u002Fretrieval-patterns.md](references\u002Fretrieval-patterns.md) - Filter strategies, context expansion","# Building RAG Systems\r\n\r\nProduction-grade RAG with semantic chunking, incremental updates, and filtered retrieval.\r\n\r\n## Quick Start\r\n\r\n```bash\r\n# Dependencies\r\npip install qdrant-client openai pydantic python-frontmatter\r\n\r\n# Core components\r\n# 1. Crawler → discovers files, extracts path metadata\r\n# 2. Parser → extracts frontmatter, computes file hash\r\n# 3. Chunker → semantic split on ## headers, 400 tokens, 15% overlap\r\n# 4. Embedder → batched OpenAI embeddings\r\n# 5. Uploader → Qdrant upsert with indexed payloads\r\n```\r\n\r\n---\r\n\r\n## Ingestion Pipeline\r\n\r\n### Architecture\r\n\r\n```\r\n┌──────────┐    ┌────────┐    ┌─────────┐    ┌──────────┐    ┌──────────┐\r\n│ Crawler  │ -> │ Parser │ -> │ Chunker │ -> │ Embedder │ -> │ Uploader │\r\n└──────────┘    └────────┘    └─────────┘    └──────────┘    └──────────┘\r\n     │              │              │              │              │\r\nDiscovers      Extracts       Splits by     Generates      Upserts to\r\nfiles          frontmatter    semantic      vectors        Qdrant\r\n               + file hash    boundaries    (batched)      (batched)\r\n```\r\n\r\n### Semantic Chunking (NOT Fixed-Size)\r\n\r\n```python\r\nclass SemanticChunker:\r\n    \"\"\"\r\n    Production chunking:\r\n    - Split on ## headers (semantic boundaries)\r\n    - Target 400 tokens (NVIDIA benchmark optimal)\r\n    - 15% overlap for context continuity\r\n    - Track prev\u002Fnext for context expansion\r\n    \"\"\"\r\n    SECTION_PATTERN = re.compile(r\"(?=^## )\", re.MULTILINE)\r\n    TOKENS_PER_WORD = 1.3\r\n\r\n    def __init__(\r\n        self,\r\n        target_tokens: int = 400,\r\n        max_tokens: int = 512,\r\n        overlap_percent: float = 0.15,\r\n    ):\r\n        self.target_words = int(target_tokens \u002F self.TOKENS_PER_WORD)\r\n        self.overlap_words = int(self.target_words * overlap_percent)\r\n\r\n    def chunk(self, content: str, file_hash: str) -> list[Chunk]:\r\n        sections = self.SECTION_PATTERN.split(content)\r\n        chunks = []\r\n\r\n        for idx, section in enumerate(sections):\r\n            content_hash = hashlib.sha256(section.encode()).hexdigest()[:16]\r\n            chunk_id = f\"{file_hash[:8]}_{content_hash}_{idx}\"\r\n\r\n            chunks.append(Chunk(\r\n                id=chunk_id,\r\n                text=section,\r\n                chunk_index=idx,\r\n                total_chunks=len(sections),\r\n                prev_chunk_id=chunks[-1].id if chunks else None,\r\n                content_hash=content_hash,\r\n                source_file_hash=file_hash,\r\n            ))\r\n\r\n            # Set next_chunk_id on previous\r\n            if len(chunks) > 1:\r\n                chunks[-2].next_chunk_id = chunk_id\r\n\r\n        return chunks\r\n```\r\n\r\n### Change Detection (Incremental Updates)\r\n\r\n```python\r\ndef compute_file_hash(file_path: str) -> str:\r\n    \"\"\"SHA-256 for change detection.\"\"\"\r\n    with open(file_path, 'rb') as f:\r\n        return hashlib.sha256(f.read()).hexdigest()\r\n\r\nclass QdrantStateTracker:\r\n    \"\"\"Query Qdrant payloads directly - no external state DB needed.\"\"\"\r\n\r\n    def get_indexed_files(self, book_id: str) -> dict[str, str]:\r\n        \"\"\"Returns {file_path: file_hash} from Qdrant.\"\"\"\r\n        indexed = {}\r\n        offset = None\r\n\r\n        while True:\r\n            points, next_offset = self.client.scroll(\r\n                collection_name=self.collection,\r\n                scroll_filter=Filter(must=[\r\n                    FieldCondition(key=\"book_id\", match=MatchValue(value=book_id))\r\n                ]),\r\n                limit=100,\r\n                offset=offset,\r\n                with_payload=[\"source_file\", \"source_file_hash\"],\r\n                with_vectors=False,\r\n            )\r\n\r\n            for point in points:\r\n                indexed[point.payload[\"source_file\"]] = point.payload[\"source_file_hash\"]\r\n\r\n            if next_offset is None:\r\n                break\r\n            offset = next_offset\r\n\r\n        return indexed\r\n\r\n    def detect_changes(self, current: dict[str, str], indexed: dict[str, str]):\r\n        \"\"\"Compare filesystem vs index.\"\"\"\r\n        new = [p for p in current if p not in indexed]\r\n        deleted = [p for p in indexed if p not in current]\r\n        modified = [p for p in current if p in indexed and current[p] != indexed[p]]\r\n        return new, modified, deleted\r\n```\r\n\r\n### Batched Embeddings\r\n\r\n```python\r\nclass OpenAIEmbedder:\r\n    def __init__(self, model: str = \"text-embedding-3-small\", batch_size: int = 20):\r\n        self.client = OpenAI()\r\n        self.model = model\r\n        self.batch_size = batch_size  # OpenAI recommendation\r\n\r\n    def embed_chunks(self, chunks: list[Chunk]) -> list[EmbeddedChunk]:\r\n        embedded = []\r\n        for i in range(0, len(chunks), self.batch_size):\r\n            batch = chunks[i:i + self.batch_size]\r\n            response = self.client.embeddings.create(\r\n                input=[c.text for c in batch],\r\n                model=self.model,\r\n            )\r\n            for chunk, data in zip(batch, response.data):\r\n                embedded.append(EmbeddedChunk(**chunk.dict(), embedding=data.embedding))\r\n        return embedded\r\n```\r\n\r\n### Qdrant Collection with Payload Indexes\r\n\r\n```python\r\ndef create_collection(self, recreate: bool = False):\r\n    \"\"\"Create collection with proper indexes for filtered retrieval.\"\"\"\r\n    self.client.create_collection(\r\n        collection_name=self.collection,\r\n        vectors_config=VectorParams(size=1536, distance=Distance.COSINE),\r\n    )\r\n\r\n    # Index ALL fields you filter by\r\n    indexes = [\r\n        (\"book_id\", PayloadSchemaType.KEYWORD),      # Tenant isolation\r\n        (\"module\", PayloadSchemaType.KEYWORD),       # Content filter\r\n        (\"chapter\", PayloadSchemaType.INTEGER),      # Range filter\r\n        (\"hardware_tier\", PayloadSchemaType.INTEGER),# Personalization\r\n        (\"proficiency_level\", PayloadSchemaType.KEYWORD),\r\n        (\"parent_doc_id\", PayloadSchemaType.KEYWORD),# Context expansion\r\n        (\"source_file_hash\", PayloadSchemaType.KEYWORD),  # Change detection\r\n    ]\r\n\r\n    for field, schema in indexes:\r\n        self.client.create_payload_index(\r\n            collection_name=self.collection,\r\n            field_name=field,\r\n            field_schema=schema,\r\n        )\r\n```\r\n\r\n---\r\n\r\n## Retrieval Patterns\r\n\r\n### Comprehensive Filter Builder\r\n\r\n```python\r\ndef build_filter(self, query: SearchQuery) -> Filter:\r\n    \"\"\"Build Qdrant filter with all conditions (AND logic).\"\"\"\r\n    conditions = []\r\n\r\n    # Required: Tenant isolation\r\n    conditions.append(FieldCondition(\r\n        key=\"book_id\", match=MatchValue(value=query.book_id)\r\n    ))\r\n\r\n    # Required: Hardware tier (lte = \"tier X or lower\")\r\n    conditions.append(FieldCondition(\r\n        key=\"hardware_tier\", range=Range(lte=query.hardware_tier)\r\n    ))\r\n\r\n    # Optional: Module exact match\r\n    if query.module:\r\n        conditions.append(FieldCondition(\r\n            key=\"module\", match=MatchValue(value=query.module)\r\n        ))\r\n\r\n    # Optional: Chapter range\r\n    if query.chapter_min or query.chapter_max:\r\n        chapter_range = Range()\r\n        if query.chapter_min:\r\n            chapter_range.gte = query.chapter_min\r\n        if query.chapter_max:\r\n            chapter_range.lte = query.chapter_max\r\n        conditions.append(FieldCondition(key=\"chapter\", range=chapter_range))\r\n\r\n    # Optional: Proficiency OR logic\r\n    if query.proficiency_levels:\r\n        conditions.append(FieldCondition(\r\n            key=\"proficiency_level\",\r\n            match=MatchAny(any=query.proficiency_levels),\r\n        ))\r\n\r\n    return Filter(must=conditions)\r\n```\r\n\r\n### Context Expansion (Walk Chunk Chain)\r\n\r\n```python\r\ndef expand_context(self, chunk_id: str, prev: int = 1, next: int = 1) -> list[Chunk]:\r\n    \"\"\"Walk prev_chunk_id\u002Fnext_chunk_id chain for surrounding context.\"\"\"\r\n    current = self.get_chunk_by_id(chunk_id)\r\n    if not current:\r\n        return []\r\n\r\n    # Walk backwards\r\n    prev_chunks = []\r\n    prev_id = current.prev_chunk_id\r\n    for _ in range(prev):\r\n        if not prev_id:\r\n            break\r\n        chunk = self.get_chunk_by_id(prev_id)\r\n        if not chunk:\r\n            break\r\n        prev_chunks.insert(0, chunk)\r\n        prev_id = chunk.prev_chunk_id\r\n\r\n    # Walk forwards\r\n    next_chunks = []\r\n    next_id = current.next_chunk_id\r\n    for _ in range(next):\r\n        if not next_id:\r\n            break\r\n        chunk = self.get_chunk_by_id(next_id)\r\n        if not chunk:\r\n            break\r\n        next_chunks.append(chunk)\r\n        next_id = chunk.next_chunk_id\r\n\r\n    return prev_chunks + [current] + next_chunks\r\n```\r\n\r\n### Full Document Retrieval\r\n\r\n```python\r\ndef get_document_chunks(self, parent_doc_id: str) -> list[Chunk]:\r\n    \"\"\"Get all chunks for a document, ordered by chunk_index.\"\"\"\r\n    points, _ = self.client.scroll(\r\n        collection_name=self.collection,\r\n        scroll_filter=Filter(must=[\r\n            FieldCondition(key=\"parent_doc_id\", match=MatchValue(value=parent_doc_id))\r\n        ]),\r\n        limit=100,\r\n        with_payload=True,\r\n        with_vectors=False,\r\n    )\r\n\r\n    chunks = [self._to_chunk(p) for p in points]\r\n    chunks.sort(key=lambda c: c.chunk_index)\r\n    return chunks\r\n```\r\n\r\n---\r\n\r\n## Payload Schema\r\n\r\n```python\r\nclass ChunkPayload(BaseModel):\r\n    \"\"\"Complete payload for filtered retrieval and context expansion.\"\"\"\r\n\r\n    # Tenant isolation\r\n    book_id: str\r\n\r\n    # Content filters (all indexed)\r\n    module: str\r\n    chapter: int\r\n    lesson: int\r\n    hardware_tier: int\r\n    proficiency_level: str\r\n\r\n    # Display content\r\n    text: str\r\n    section_title: Optional[str]\r\n    source_file: str\r\n\r\n    # Context expansion\r\n    parent_doc_id: str\r\n    chunk_index: int\r\n    total_chunks: int\r\n    prev_chunk_id: Optional[str]\r\n    next_chunk_id: Optional[str]\r\n\r\n    # Change detection\r\n    content_hash: str\r\n    source_file_hash: str\r\n```\r\n\r\n---\r\n\r\n## Anti-Patterns\r\n\r\n| Don't | Do Instead |\r\n|-------|------------|\r\n| Fixed character chunking | Semantic boundaries (## headers) |\r\n| Position-based chunk IDs | Content hash for stable IDs |\r\n| No overlap between chunks | 10-20% overlap for continuity |\r\n| Full re-index on every change | Incremental with file hash detection |\r\n| Missing payload indexes | Index every field you filter by |\r\n| Synchronous embedding | Batch with background jobs |\r\n| External state database | Qdrant-native state tracking |\r\n\r\n---\r\n\r\n## Verification\r\n\r\nRun: `python scripts\u002Fverify.py`\r\n\r\n## Related Skills\r\n\r\n- `scaffolding-fastapi-dapr` - API patterns for search endpoints\r\n- `streaming-llm-responses` - Streaming RAG responses\r\n\r\n## References\r\n\r\n- [references\u002Fingestion-patterns.md](references\u002Fingestion-patterns.md) - Full ingestion pipeline\r\n- [references\u002Fretrieval-patterns.md](references\u002Fretrieval-patterns.md) - Filter strategies, context expansion",{"name":5,"description":7},"main","def8531e65114c0fca8fb8551c1871ee0eed705c","skills\u002Faiskillstore\u002Fmarketplace\u002Fasmayaseen\u002Fbuilding-rag-systems",24,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub","https:\u002F\u002Fskillshub.wtf","https:\u002F\u002Fgithub.com\u002FComeOnOliver","AI & Blockchain Engineer\r\n","2026-06-24 22:07:19",{"source":8,"skillName":5,"commands":54,"prompt":64,"zipAvailable":65},[55,58,61],{"tool":56,"command":57},"npx","npx skills add ComeOnOliver\u002Fskillshub --skill building-rag-systems",{"tool":59,"command":60},"bunx","bunx skills add ComeOnOliver\u002Fskillshub --skill building-rag-systems",{"tool":62,"command":63},"pnpm","pnpm dlx skills add ComeOnOliver\u002Fskillshub --skill building-rag-systems","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fasmayaseen\u002Fbuilding-rag-systems\u002FSKILL.md",true,[67,73,77,83],{"path":68,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":71,"sourceHtmlUrl":72},"references\u002Fingestion-patterns.md","REFERENCE","参考文档",13452,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fasmayaseen\u002Fbuilding-rag-systems\u002Freferences\u002Fingestion-patterns.md",{"path":74,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":75,"sourceHtmlUrl":76},"references\u002Fretrieval-patterns.md",12090,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fasmayaseen\u002Fbuilding-rag-systems\u002Freferences\u002Fretrieval-patterns.md",{"path":78,"resourceType":79,"resourceTypeDesc":80,"sizeBytes":81,"sourceHtmlUrl":82},"scripts\u002Fverify.py","SCRIPT","脚本",621,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fasmayaseen\u002Fbuilding-rag-systems\u002Fscripts\u002Fverify.py",{"path":84,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":85,"sourceHtmlUrl":86},"skill-report.json",27702,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fasmayaseen\u002Fbuilding-rag-systems\u002Fskill-report.json",[88,102,112,121,127,134],{"skillNo":89,"name":90,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":98,"publishedAt":39},"01M6FKJN13GMG4KY57J3QD8GY5","model-hierarchy-skill","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzscole\u002Fmodel-hierarchy-skill\u002Fmodel-hierarchy-skill\u002FSKILL.md",[93],{"categoryNo":94,"categoryName":95,"taxonomyType":96,"taxonomyTypeDesc":97},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[99,100,101],"model","model hierarchy","model hierarchy skill",{"skillNo":103,"name":104,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":108,"publishedAt":39},"01M6FKJJJ6GCAMR4PPPRD12M5X","zoom-mcp","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-mcp\u002FSKILL.md",[107],{"categoryNo":94,"categoryName":95,"taxonomyType":96,"taxonomyTypeDesc":97},[109,110,111],"MCP","zoom","zoom mcp",{"skillNo":113,"name":114,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":115,"categories":116,"keywords":117,"publishedAt":39},"01M6FKJE3KYT3KPEZMEEHYKA6K","zoom-apps-sdk","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-apps-sdk\u002FSKILL.md",[],[118,119,120,110],"sdk","zoom apps sdk","apps",{"skillNo":122,"name":123,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":124,"categories":125,"keywords":126,"publishedAt":39},"01M6FKJ9EDH9Q5VWDF8HAJRSP0","windows","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwindows\u002FSKILL.md",[],[123],{"skillNo":128,"name":129,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":130,"categories":131,"keywords":133,"publishedAt":39},"01M6FKJ666BEEFC3AE9GJ72Q93","whiteboard","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwhiteboard\u002FSKILL.md",[132],{"categoryNo":94,"categoryName":95,"taxonomyType":96,"taxonomyTypeDesc":97},[129],{"skillNo":135,"name":136,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":137,"categories":138,"keywords":139,"publishedAt":39},"01M6FKJ21WFCDGTJQ2Q91T9DVW","websockets","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwebsockets\u002FSKILL.md",[],[136]]