[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6WZXF7J5H6PXX48CNJDP0MF":3,"skill-same:01M6WZXF7J5H6PXX48CNJDP0MF":87},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":52,"commitSha":53,"skillDirectory":54,"forksCount":14,"repositoryHtmlUrl":55,"repositoryHomepageUrl":56,"ownerHtmlUrl":57,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":58,"install":59,"resources":72},"01M6WZXF7J5H6PXX48CNJDP0MF","rag-engineering",null,"Retrieval-Augmented Generation pipelines — ingestion, chunking, embedding, vector stores, retrieval, evaluation. Use when building a RAG pipeline, choosing chunking strategies or embedding models, debugging retrieval quality or hallucinations, evaluating an existing RAG system, or scaling\u002Fmigrating vector stores.","projectious-work\u002Fprocesskit","projectious-work","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F263889726?v=4","01M6N97TJJ118WQ35CHRV8RD0E","projectious.work","01M6N97TPWJPF0BMKYQ4PBY5W8",0,"MIT","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Farchive\u002Fprocesskit-install-20261101\u002Fcontext\u002Ftemplates\u002Fprocesskit\u002Fv0.18.2\u002Fcontext\u002Fskills\u002Fdata-ai\u002Frag-engineering\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"rag engineering","engineering","rag","strategies","when","Existing","v1","rag pipeline","embedding","scaling","generation","chunking","ingestion","use","work","vector stores","system","pipeline","vector","debugging","2026-11-10 10:12:30",false,"---\nname: rag-engineering\ndescription: |\n  Retrieval-Augmented Generation pipelines — ingestion, chunking, embedding, vector stores, retrieval, evaluation. Use when building a RAG pipeline, choosing chunking strategies or embedding models, debugging retrieval quality or hallucinations, evaluating an existing RAG system, or scaling\u002Fmigrating vector stores.\nmetadata:\n  processkit:\n    apiVersion: processkit.projectious.work\u002Fv1\n    id: SKILL-rag-engineering\n    version: \"1.1.0\"\n    created: 2026-04-06T00:00:00Z\n    category: data-ai\n---\n\n# RAG Engineering\n\n## Intro\n\nA RAG pipeline has two phases: indexing (offline) and\nretrieval+generation (online). Decisions cascade — bad chunking\nruins retrieval no matter how good your embeddings are, and bad\nretrieval ruins generation no matter how strong your LLM is. Start\nfrom the data and work forward, and measure with a golden eval set\nfrom day one.\n\n## Overview\n\n### Architecture\n\n```\nIndexing:  Documents -> Parse -> Chunk -> Embed -> Vector DB\nQuery:     Question  -> Embed -> Retrieve top-k -> Prompt -> LLM\n```\n\n### Document ingestion\n\nParse source documents into clean text with metadata preserved.\n\n- **PDF** — `pymupdf4llm` or `unstructured`. Preserve headings,\n  tables, page numbers. OCR fallback for scanned docs via\n  `tesseract`.\n- **HTML** — strip boilerplate with `trafilatura` or\n  `readability-lxml`; keep headings, lists, code blocks.\n- **Code** — tree-sitter for AST-aware parsing. Keep function\u002Fclass\n  boundaries intact. Preserve file paths and language metadata.\n- **Always capture** source URL\u002Fpath, document title, section\n  heading, page number, last-modified date. Metadata enables\n  filtering at retrieval time.\n\n### Chunking strategies\n\nPick based on document structure and query patterns. See\n`references\u002Fchunking-strategies.md` for code and trade-offs.\n\n- **Fixed-size with overlap** — simple baseline; 256–1024 tokens,\n  10–20% overlap.\n- **Sentence-based** — group 3–5 sentences; preserves boundaries.\n- **Semantic chunking** — split where embedding similarity drops;\n  best for heterogeneous docs but slow.\n- **Recursive character** — LangChain default; splits on `\\n\\n`,\n  then `\\n`, then space.\n- **Document-structure-aware** — split on headings, code blocks,\n  sections; best when structure is reliable.\n\nRule of thumb: chunk size should match expected answer granularity.\nFAQ -> small chunks. Long technical explanations -> larger chunks.\n\n### Embedding models\n\nPick on domain match, dimensionality, speed, and cost.\n\n| Model | Dims | Context | Notes |\n|---|---|---|---|\n| `text-embedding-3-small` | 1536 | 8191 | Good default, cheap |\n| `text-embedding-3-large` | 3072 | 8191 | Better quality, 2x cost |\n| `nomic-embed-text` | 768 | 8192 | Open source, strong MTEB |\n| `bge-large-en-v1.5` | 1024 | 512 | Open source, good for code |\n| `voyage-code-2` | 1536 | 16000 | Best for code retrieval |\n\nAlways benchmark on **your** queries before committing. MTEB\nleaderboard scores don't predict domain-specific performance. For\nsignificant gains on a tight domain, fine-tune with synthetic\nquery\u002Fpassage pairs and contrastive loss — even 1000 pairs helps.\n\n### Vector stores\n\n- **FAISS** — in-memory, fastest for \u003C10M vectors, no filtering;\n  prototyping or read-heavy workloads.\n- **pgvector** — Postgres extension; use when you already have\n  Postgres and need joins\u002Ffiltering. HNSW index for performance.\n- **Chroma** — embedded DB, easy local dev; \u003C1M vectors practical.\n- **Qdrant** — Rust-based, strong filtering, good hybrid search;\n  self-host or cloud; production-ready.\n- **Pinecone** — fully managed, scales well; expensive at scale.\n- **Weaviate** — built-in vectorization, multimodal, feature-rich.\n\nFor most projects: start with Chroma locally, move to Qdrant or\npgvector for production.\n\n### Retrieval strategies\n\n- **Dense** — embed query, find nearest neighbors. Default.\n- **Sparse (BM25)** — term matching. Better for exact terms,\n  acronyms, IDs.\n- **Hybrid** — combine dense + sparse with reciprocal rank fusion.\n  Almost always better than either alone.\n- **Reranking** — retrieve top-50 with a fast method, rerank to\n  top-5 with a cross-encoder (`bge-reranker-v2-m3`, Cohere Rerank).\n- **MMR (Maximal Marginal Relevance)** — diversify results;\n  lambda 0.5–0.7.\n- **Parent-document retrieval** — embed small chunks, return the\n  parent section; combines precision with context.\n- **Multi-query** — LLM generates 3–5 query variants, retrieve for\n  each, deduplicate. Helps with ambiguous queries.\n\nSee `references\u002Fretrieval-patterns.md` for the architecture\ndiagram and per-pattern code.\n\n### Prompt construction\n\nThe retrieval-to-generation handoff is where many RAG systems\nfail.\n\n- **Context ordering** — most relevant chunks first; LLMs attend\n  more to the start and end of context.\n- **Context budget** — reserve 30–40% of context window for\n  retrieved content.\n- **Citations** — number each chunk `[1]`, `[2]`; instruct the LLM\n  to cite; verify citations in post-processing.\n- **Deduplication** — remove near-duplicates (cosine > 0.95)\n  before inserting.\n- **No-context handling** — if retrieval scores are below threshold,\n  tell the LLM explicitly that no relevant info was found rather\n  than stuffing bad context.\n\n### Evaluation\n\nKey metrics: **context precision** (are retrieved chunks\nrelevant?), **context recall** (are all needed chunks retrieved?),\n**faithfulness** (is the answer grounded in context?), **answer\nrelevance** (does it address the question?).\n\nUse the RAGAS framework. Build a golden dataset of 50–100\nquestion\u002Fideal-context\u002Freference-answer triples from your real\ndocuments. See `references\u002Fevaluation.md` for metric details and\ndebugging workflow.\n\nCommon failure modes:\n\n- **Hallucination** — low faithfulness. Better retrieval, explicit\n  grounding instructions.\n- **Incomplete answers** — low context recall. Smaller chunks,\n  multi-query.\n- **Wrong documents** — low context precision. Hybrid search,\n  metadata filtering, reranking.\n- **Right docs, bad answer** — model issue. Better prompt,\n  stronger model.\n\n## Gotchas\n\nAgent-specific failure modes — provider-neutral pause-and-self-check items:\n\n- **Tuning the prompt to fix a retrieval problem.** If the retrieved chunks are wrong, rewriting the generation prompt will not fix the underlying retrieval failure — it just masks symptoms. Isolate which stage is failing (retrieval vs. generation) before making changes. Never tune the prompt to compensate for poor retrieval.\n- **Chunks too large relative to query granularity.** A 2,000-token chunk contains many facts. The embedding represents the average of all of them, so it will be a mediocre match for any specific fact inside the chunk. Match chunk size to the granularity of expected queries — FAQ answers need small chunks; technical explanations need larger ones.\n- **Pure dense retrieval for catalogs with exact identifiers.** Dense semantic search fails on product SKUs, version numbers, proper nouns, and acronyms. These need exact term matching via sparse (BM25) retrieval. Hybrid dense + sparse is the correct default for production.\n- **Stuffing low-scoring retrieved chunks into the context.** When no relevant content exists, returning the top-k chunks regardless of relevance score causes hallucinations — the model will try to construct an answer from irrelevant content. Set a similarity threshold; if no chunk passes it, tell the LLM explicitly that no relevant information was found.\n- **No faithfulness evaluation before production.** A RAG system that produces fluent, confident answers that contradict the retrieved context is worse than one that says \"I don't know.\" Measure faithfulness (answer grounded in context?) from the first prototype, not as an afterthought.\n- **Re-indexing for every iteration when debugging chunking.** Rebuilding the full vector index to test every chunking change is expensive and slow. Maintain a small representative test corpus that re-indexes in seconds; use the full corpus only for final validation.\n- **Building without a golden eval set.** Without a ground-truth set of queries and expected context passages, \"does the retrieval work?\" is a feeling, not a measurement. Build the eval set from real user questions before the first production deployment and run it after every change.\n\n## Full reference\n\n### Hybrid search and RRF\n\nReciprocal Rank Fusion is the standard combiner for hybrid\nretrieval — no score-weight tuning needed:\n\n```\nRRF(d) = sum( 1 \u002F (k + rank_i(d)) ) for each retrieval method i\n                                                 (k typically 60)\n```\n\nHybrid + RRF outperforms either dense or sparse alone on nearly\nevery benchmark and should be the default for production. Vector\nstores with native hybrid support: Qdrant, Weaviate, Elasticsearch.\nFor others, run both searches and fuse in application code.\n\n### Reranking\n\nBi-encoders (embedding models) score query and document\nindependently. Cross-encoders score `(query, document)` jointly —\nmuch more accurate but too slow for full-corpus search.\n\n```\nStage 1: Retrieve top-50 with fast method (dense, hybrid)\nStage 2: Rerank to top-5 with cross-encoder\n```\n\nModels: `bge-reranker-v2-m3` (open, strong), Cohere Rerank (API,\neasy), `ms-marco-MiniLM-L-12-v2` (fast, lighter). Latency is\ntypically 100–500ms for 50 candidates.\n\n### MMR\n\n```\nMMR = argmax[ lambda * sim(q, d) - (1 - lambda) * max(sim(d, d_selected)) ]\n```\n\nGreedy: pick the highest-scoring candidate, repeat. Lambda 1.0 =\npure relevance, 0.0 = pure diversity. Start at 0.6. Lower lambda\nwhen chunks are highly redundant.\n\n### Parent-document retrieval\n\nEmbed at 256 tokens (precise matching) but store a mapping to a\n1024-token parent (richer context). Match on child embeddings,\nreturn the parent. Small chunks find precise matches; large chunks\ngive the LLM enough context to answer well.\n\n### Multi-query retrieval\n\nUse a small\u002Ffast LLM (Haiku, GPT-4o-mini) to rewrite the query into\n3–5 variants, retrieve top-10 for each, deduplicate by document ID,\nfuse with RRF. Adds ~200–500ms of latency. Best for ambiguous or\nbroad queries.\n\n### Metadata filtering\n\nPre-filter candidates before or during vector search to reduce\nsearch space and improve precision.\n\n```python\nresults = vector_store.similarity_search(\n    query_embedding, k=10,\n    filter={\"source\": \"api-docs\", \"version\": \">=3.0\"},\n)\n```\n\nUseful filters: document type, date range, language, author,\nproduct version, access control. Pre-filter > post-filter when the\nindex supports it.\n\n### Chunking guidelines\n\n1. **Always measure.** Chunk strategy changes retrieval quality —\n   re-run eval after every change.\n2. **Embed metadata in the chunk.** Prepend\n   `\"Title: {title}\\nSection: {heading}\\n\\n\"` so retrieval matches\n   queries that reference titles or sections.\n3. **Track provenance.** Source file, page or line number, heading\n   path with each chunk — essential for citations and debugging.\n4. **Small chunks for retrieval, large for generation.** Use\n   parent-document retrieval.\n5. **Deduplicate.** Cosine > 0.95 = duplicate. Common with\n   overlapping chunks and repeated boilerplate.\n\n### RAGAS targets\n\n| Metric | Target |\n|---|---|\n| Context precision | > 0.8 |\n| Context recall | > 0.7 |\n| Faithfulness | > 0.9 (non-negotiable for production) |\n| Answer relevancy | > 0.8 |\n\n### Failure-mode debugging\n\n| Symptom | Likely cause | Fix |\n|---|---|---|\n| Wrong docs retrieved | Vocabulary mismatch | Add hybrid search |\n| Right doc exists, not retrieved | Chunk too large, answer buried | Smaller chunks, parent-doc retrieval |\n| Top-1 right, rest noise | No reranking | Add cross-encoder reranker |\n| Same info repeated | Overlapping chunks, no dedup | MMR or dedup pass |\n| Answer contradicts context | Hallucination \u002F model prior | Stronger grounding prompt, better model |\n| Vague generic answer | Context not used effectively | Reorder context, improve template |\n| Refuses despite good context | Over-cautious system prompt | Relax prompt, check conflicts |\n| Answers part of question | Low context recall | Multi-query, smaller chunks |\n\nGolden rule: never tune the prompt to fix a retrieval problem, and\nnever tune retrieval to fix a prompt problem. Isolate the variable.\n\n### Worked scenarios\n\n**Basic RAG over internal docs.** Recursive chunking at 512\ntokens, `text-embedding-3-small`, Chroma. Dense retrieval, top-5.\nBuild an eval set from real user questions. Iterate from this\nbaseline.\n\n**RAG misses relevant info.** Diagnose context recall — what was\nretrieved vs. what should have been? Add hybrid search (BM25 +\ndense), add cross-encoder reranking, try smaller chunk sizes. Test\neach change against the eval set.\n\n**Benchmarking a RAG system.** Build a golden dataset: 50\nquestions, manually identified ideal context passages, reference\nanswers. Run RAGAS. Identify the weakest metric and focus there.\nSet up automated eval in CI to catch regressions.\n","# RAG Engineering\n\n## Intro\n\nA RAG pipeline has two phases: indexing (offline) and\nretrieval+generation (online). Decisions cascade — bad chunking\nruins retrieval no matter how good your embeddings are, and bad\nretrieval ruins generation no matter how strong your LLM is. Start\nfrom the data and work forward, and measure with a golden eval set\nfrom day one.\n\n## Overview\n\n### Architecture\n\n```\nIndexing:  Documents -> Parse -> Chunk -> Embed -> Vector DB\nQuery:     Question  -> Embed -> Retrieve top-k -> Prompt -> LLM\n```\n\n### Document ingestion\n\nParse source documents into clean text with metadata preserved.\n\n- **PDF** — `pymupdf4llm` or `unstructured`. Preserve headings,\n  tables, page numbers. OCR fallback for scanned docs via\n  `tesseract`.\n- **HTML** — strip boilerplate with `trafilatura` or\n  `readability-lxml`; keep headings, lists, code blocks.\n- **Code** — tree-sitter for AST-aware parsing. Keep function\u002Fclass\n  boundaries intact. Preserve file paths and language metadata.\n- **Always capture** source URL\u002Fpath, document title, section\n  heading, page number, last-modified date. Metadata enables\n  filtering at retrieval time.\n\n### Chunking strategies\n\nPick based on document structure and query patterns. See\n`references\u002Fchunking-strategies.md` for code and trade-offs.\n\n- **Fixed-size with overlap** — simple baseline; 256–1024 tokens,\n  10–20% overlap.\n- **Sentence-based** — group 3–5 sentences; preserves boundaries.\n- **Semantic chunking** — split where embedding similarity drops;\n  best for heterogeneous docs but slow.\n- **Recursive character** — LangChain default; splits on `\\n\\n`,\n  then `\\n`, then space.\n- **Document-structure-aware** — split on headings, code blocks,\n  sections; best when structure is reliable.\n\nRule of thumb: chunk size should match expected answer granularity.\nFAQ -> small chunks. Long technical explanations -> larger chunks.\n\n### Embedding models\n\nPick on domain match, dimensionality, speed, and cost.\n\n| Model | Dims | Context | Notes |\n|---|---|---|---|\n| `text-embedding-3-small` | 1536 | 8191 | Good default, cheap |\n| `text-embedding-3-large` | 3072 | 8191 | Better quality, 2x cost |\n| `nomic-embed-text` | 768 | 8192 | Open source, strong MTEB |\n| `bge-large-en-v1.5` | 1024 | 512 | Open source, good for code |\n| `voyage-code-2` | 1536 | 16000 | Best for code retrieval |\n\nAlways benchmark on **your** queries before committing. MTEB\nleaderboard scores don't predict domain-specific performance. For\nsignificant gains on a tight domain, fine-tune with synthetic\nquery\u002Fpassage pairs and contrastive loss — even 1000 pairs helps.\n\n### Vector stores\n\n- **FAISS** — in-memory, fastest for \u003C10M vectors, no filtering;\n  prototyping or read-heavy workloads.\n- **pgvector** — Postgres extension; use when you already have\n  Postgres and need joins\u002Ffiltering. HNSW index for performance.\n- **Chroma** — embedded DB, easy local dev; \u003C1M vectors practical.\n- **Qdrant** — Rust-based, strong filtering, good hybrid search;\n  self-host or cloud; production-ready.\n- **Pinecone** — fully managed, scales well; expensive at scale.\n- **Weaviate** — built-in vectorization, multimodal, feature-rich.\n\nFor most projects: start with Chroma locally, move to Qdrant or\npgvector for production.\n\n### Retrieval strategies\n\n- **Dense** — embed query, find nearest neighbors. Default.\n- **Sparse (BM25)** — term matching. Better for exact terms,\n  acronyms, IDs.\n- **Hybrid** — combine dense + sparse with reciprocal rank fusion.\n  Almost always better than either alone.\n- **Reranking** — retrieve top-50 with a fast method, rerank to\n  top-5 with a cross-encoder (`bge-reranker-v2-m3`, Cohere Rerank).\n- **MMR (Maximal Marginal Relevance)** — diversify results;\n  lambda 0.5–0.7.\n- **Parent-document retrieval** — embed small chunks, return the\n  parent section; combines precision with context.\n- **Multi-query** — LLM generates 3–5 query variants, retrieve for\n  each, deduplicate. Helps with ambiguous queries.\n\nSee `references\u002Fretrieval-patterns.md` for the architecture\ndiagram and per-pattern code.\n\n### Prompt construction\n\nThe retrieval-to-generation handoff is where many RAG systems\nfail.\n\n- **Context ordering** — most relevant chunks first; LLMs attend\n  more to the start and end of context.\n- **Context budget** — reserve 30–40% of context window for\n  retrieved content.\n- **Citations** — number each chunk `[1]`, `[2]`; instruct the LLM\n  to cite; verify citations in post-processing.\n- **Deduplication** — remove near-duplicates (cosine > 0.95)\n  before inserting.\n- **No-context handling** — if retrieval scores are below threshold,\n  tell the LLM explicitly that no relevant info was found rather\n  than stuffing bad context.\n\n### Evaluation\n\nKey metrics: **context precision** (are retrieved chunks\nrelevant?), **context recall** (are all needed chunks retrieved?),\n**faithfulness** (is the answer grounded in context?), **answer\nrelevance** (does it address the question?).\n\nUse the RAGAS framework. Build a golden dataset of 50–100\nquestion\u002Fideal-context\u002Freference-answer triples from your real\ndocuments. See `references\u002Fevaluation.md` for metric details and\ndebugging workflow.\n\nCommon failure modes:\n\n- **Hallucination** — low faithfulness. Better retrieval, explicit\n  grounding instructions.\n- **Incomplete answers** — low context recall. Smaller chunks,\n  multi-query.\n- **Wrong documents** — low context precision. Hybrid search,\n  metadata filtering, reranking.\n- **Right docs, bad answer** — model issue. Better prompt,\n  stronger model.\n\n## Gotchas\n\nAgent-specific failure modes — provider-neutral pause-and-self-check items:\n\n- **Tuning the prompt to fix a retrieval problem.** If the retrieved chunks are wrong, rewriting the generation prompt will not fix the underlying retrieval failure — it just masks symptoms. Isolate which stage is failing (retrieval vs. generation) before making changes. Never tune the prompt to compensate for poor retrieval.\n- **Chunks too large relative to query granularity.** A 2,000-token chunk contains many facts. The embedding represents the average of all of them, so it will be a mediocre match for any specific fact inside the chunk. Match chunk size to the granularity of expected queries — FAQ answers need small chunks; technical explanations need larger ones.\n- **Pure dense retrieval for catalogs with exact identifiers.** Dense semantic search fails on product SKUs, version numbers, proper nouns, and acronyms. These need exact term matching via sparse (BM25) retrieval. Hybrid dense + sparse is the correct default for production.\n- **Stuffing low-scoring retrieved chunks into the context.** When no relevant content exists, returning the top-k chunks regardless of relevance score causes hallucinations — the model will try to construct an answer from irrelevant content. Set a similarity threshold; if no chunk passes it, tell the LLM explicitly that no relevant information was found.\n- **No faithfulness evaluation before production.** A RAG system that produces fluent, confident answers that contradict the retrieved context is worse than one that says \"I don't know.\" Measure faithfulness (answer grounded in context?) from the first prototype, not as an afterthought.\n- **Re-indexing for every iteration when debugging chunking.** Rebuilding the full vector index to test every chunking change is expensive and slow. Maintain a small representative test corpus that re-indexes in seconds; use the full corpus only for final validation.\n- **Building without a golden eval set.** Without a ground-truth set of queries and expected context passages, \"does the retrieval work?\" is a feeling, not a measurement. Build the eval set from real user questions before the first production deployment and run it after every change.\n\n## Full reference\n\n### Hybrid search and RRF\n\nReciprocal Rank Fusion is the standard combiner for hybrid\nretrieval — no score-weight tuning needed:\n\n```\nRRF(d) = sum( 1 \u002F (k + rank_i(d)) ) for each retrieval method i\n                                                 (k typically 60)\n```\n\nHybrid + RRF outperforms either dense or sparse alone on nearly\nevery benchmark and should be the default for production. Vector\nstores with native hybrid support: Qdrant, Weaviate, Elasticsearch.\nFor others, run both searches and fuse in application code.\n\n### Reranking\n\nBi-encoders (embedding models) score query and document\nindependently. Cross-encoders score `(query, document)` jointly —\nmuch more accurate but too slow for full-corpus search.\n\n```\nStage 1: Retrieve top-50 with fast method (dense, hybrid)\nStage 2: Rerank to top-5 with cross-encoder\n```\n\nModels: `bge-reranker-v2-m3` (open, strong), Cohere Rerank (API,\neasy), `ms-marco-MiniLM-L-12-v2` (fast, lighter). Latency is\ntypically 100–500ms for 50 candidates.\n\n### MMR\n\n```\nMMR = argmax[ lambda * sim(q, d) - (1 - lambda) * max(sim(d, d_selected)) ]\n```\n\nGreedy: pick the highest-scoring candidate, repeat. Lambda 1.0 =\npure relevance, 0.0 = pure diversity. Start at 0.6. Lower lambda\nwhen chunks are highly redundant.\n\n### Parent-document retrieval\n\nEmbed at 256 tokens (precise matching) but store a mapping to a\n1024-token parent (richer context). Match on child embeddings,\nreturn the parent. Small chunks find precise matches; large chunks\ngive the LLM enough context to answer well.\n\n### Multi-query retrieval\n\nUse a small\u002Ffast LLM (Haiku, GPT-4o-mini) to rewrite the query into\n3–5 variants, retrieve top-10 for each, deduplicate by document ID,\nfuse with RRF. Adds ~200–500ms of latency. Best for ambiguous or\nbroad queries.\n\n### Metadata filtering\n\nPre-filter candidates before or during vector search to reduce\nsearch space and improve precision.\n\n```python\nresults = vector_store.similarity_search(\n    query_embedding, k=10,\n    filter={\"source\": \"api-docs\", \"version\": \">=3.0\"},\n)\n```\n\nUseful filters: document type, date range, language, author,\nproduct version, access control. Pre-filter > post-filter when the\nindex supports it.\n\n### Chunking guidelines\n\n1. **Always measure.** Chunk strategy changes retrieval quality —\n   re-run eval after every change.\n2. **Embed metadata in the chunk.** Prepend\n   `\"Title: {title}\\nSection: {heading}\\n\\n\"` so retrieval matches\n   queries that reference titles or sections.\n3. **Track provenance.** Source file, page or line number, heading\n   path with each chunk — essential for citations and debugging.\n4. **Small chunks for retrieval, large for generation.** Use\n   parent-document retrieval.\n5. **Deduplicate.** Cosine > 0.95 = duplicate. Common with\n   overlapping chunks and repeated boilerplate.\n\n### RAGAS targets\n\n| Metric | Target |\n|---|---|\n| Context precision | > 0.8 |\n| Context recall | > 0.7 |\n| Faithfulness | > 0.9 (non-negotiable for production) |\n| Answer relevancy | > 0.8 |\n\n### Failure-mode debugging\n\n| Symptom | Likely cause | Fix |\n|---|---|---|\n| Wrong docs retrieved | Vocabulary mismatch | Add hybrid search |\n| Right doc exists, not retrieved | Chunk too large, answer buried | Smaller chunks, parent-doc retrieval |\n| Top-1 right, rest noise | No reranking | Add cross-encoder reranker |\n| Same info repeated | Overlapping chunks, no dedup | MMR or dedup pass |\n| Answer contradicts context | Hallucination \u002F model prior | Stronger grounding prompt, better model |\n| Vague generic answer | Context not used effectively | Reorder context, improve template |\n| Refuses despite good context | Over-cautious system prompt | Relax prompt, check conflicts |\n| Answers part of question | Low context recall | Multi-query, smaller chunks |\n\nGolden rule: never tune the prompt to fix a retrieval problem, and\nnever tune retrieval to fix a prompt problem. Isolate the variable.\n\n### Worked scenarios\n\n**Basic RAG over internal docs.** Recursive chunking at 512\ntokens, `text-embedding-3-small`, Chroma. Dense retrieval, top-5.\nBuild an eval set from real user questions. Iterate from this\nbaseline.\n\n**RAG misses relevant info.** Diagnose context recall — what was\nretrieved vs. what should have been? Add hybrid search (BM25 +\ndense), add cross-encoder reranking, try smaller chunk sizes. Test\neach change against the eval set.\n\n**Benchmarking a RAG system.** Build a golden dataset: 50\nquestions, manually identified ideal context passages, reference\nanswers. Run RAGAS. Identify the weakest metric and focus there.\nSet up automated eval in CI to catch regressions.\n",{"name":5,"metadata":44,"description":51},{"processkit":45},{"id":46,"created":47,"version":48,"category":49,"apiVersion":50},"SKILL-rag-engineering",1775433600000,"1.1.0","data-ai","processkit.projectious.work\u002Fv1","Retrieval-Augmented Generation pipelines — ingestion, chunking, embedding, vector stores, retrieval, evaluation. Use when building a RAG pipeline, choosing chunking strategies or embedding models, debugging retrieval quality or hallucinations, evaluating an existing RAG system, or scaling\u002Fmigrating vector stores.\n","main","4876a92620ee792aac2ac9a1a806322d8a572367","archive\u002Fprocesskit-install-20261101\u002Fcontext\u002Ftemplates\u002Fprocesskit\u002Fv0.18.2\u002Fcontext\u002Fskills\u002Fdata-ai\u002Frag-engineering","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit","https:\u002F\u002Fprojectious-work.github.io\u002Fprocesskit\u002F","https:\u002F\u002Fgithub.com\u002Fprojectious-work","2026-11-11 23:24:18",{"source":8,"skillName":5,"commands":60,"prompt":70,"zipAvailable":71},[61,64,67],{"tool":62,"command":63},"npx","npx skills add projectious-work\u002Fprocesskit --skill rag-engineering",{"tool":65,"command":66},"bunx","bunx skills add projectious-work\u002Fprocesskit --skill rag-engineering",{"tool":68,"command":69},"pnpm","pnpm dlx skills add projectious-work\u002Fprocesskit --skill rag-engineering","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Farchive\u002Fprocesskit-install-20261101\u002Fcontext\u002Ftemplates\u002Fprocesskit\u002Fv0.18.2\u002Fcontext\u002Fskills\u002Fdata-ai\u002Frag-engineering\u002FSKILL.md",true,[73,79,83],{"path":74,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":77,"sourceHtmlUrl":78},"references\u002Fchunking-strategies.md","REFERENCE","参考文档",6221,"https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Farchive\u002Fprocesskit-install-20261101\u002Fcontext\u002Ftemplates\u002Fprocesskit\u002Fv0.18.2\u002Fcontext\u002Fskills\u002Fdata-ai\u002Frag-engineering\u002Freferences\u002Fchunking-strategies.md",{"path":80,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":81,"sourceHtmlUrl":82},"references\u002Fevaluation.md",7105,"https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Farchive\u002Fprocesskit-install-20261101\u002Fcontext\u002Ftemplates\u002Fprocesskit\u002Fv0.18.2\u002Fcontext\u002Fskills\u002Fdata-ai\u002Frag-engineering\u002Freferences\u002Fevaluation.md",{"path":84,"resourceType":75,"resourceTypeDesc":76,"sizeBytes":85,"sourceHtmlUrl":86},"references\u002Fretrieval-patterns.md",6246,"https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Farchive\u002Fprocesskit-install-20261101\u002Fcontext\u002Ftemplates\u002Fprocesskit\u002Fv0.18.2\u002Fcontext\u002Fskills\u002Fdata-ai\u002Frag-engineering\u002Freferences\u002Fretrieval-patterns.md",[88,114,134,155,172,190],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":99,"publishedAt":39},"01M6X5MTC89B4ZGXW3JVSQCPF8","skill-management","Manage Skill package manifests as v1 ontology entities.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fskill-management\u002FSKILL.md",[94],{"categoryNo":95,"categoryName":96,"taxonomyType":97,"taxonomyTypeDesc":98},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[100,101,102,103,104,105,33,106,107,108,90,109,110,111,25,112,113],"management","skill management","create","update","package","alpha","list","manage","ontology","get","as","manifests","skill package","transition",{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":118,"categories":119,"keywords":121,"publishedAt":39},"01M6X5MCV74NHW5MEDNHZRF2C8","schema-management","Manage generated processkit schema contracts through deterministic MCP rebuild and inspection workflows.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fschema-management\u002FSKILL.md",[120],{"categoryNo":95,"categoryName":96,"taxonomyType":97,"taxonomyTypeDesc":98},[100,122,123,124,125,126,105,33,127,128,107,129,130,131,109,132,133],"schema","schema management","MCP","workflows","validation","generated","rebuild","contracts","wed","deterministic","contract","mode",{"skillNo":135,"name":136,"nameZh":6,"description":137,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":138,"categories":139,"keywords":140,"publishedAt":39},"01M6X5M5G6YJ96AMZGSCZJ1D13","proposition-management","Record claims, risks, beliefs, world facts, WSJF estimates, and assumptions through the Proposition interface.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fproposition-management\u002FSKILL.md",[],[100,141,142,102,103,143,105,33,144,145,146,147,109,148,149,150,151,152,153,154],"proposition","proposition management","index","log","world","record","query","creation","event","interface","updates","facts","claims","WSJF",{"skillNo":156,"name":157,"nameZh":6,"description":158,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":159,"categories":160,"keywords":161,"publishedAt":39},"01M6X5KMQ83QN9M6MPWZM7344F","ontology-management","Manage generated v1 ontology entities that do not require a dedicated domain server.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fontology-management\u002FSKILL.md",[],[100,108,162,102,105,163,164,127,165,144,107,166,167,109,168,169,149,170,25,171],"ontology management","lifecycle","archive","roadmap","do","domain","iteration","keep","measurement","generic",{"skillNo":173,"name":174,"nameZh":6,"description":175,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":176,"categories":177,"keywords":178,"publishedAt":39},"01M6X5KCY2DN3SM21RERSFNEBT","okf-compatibility","Import, export, and validate conformant Open Knowledge Format bundles.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fokf-compatibility\u002FSKILL.md",[],[179,180,181,182,105,183,184,185,33,186,187,188,189],"okf","compatibility","okf compatibility","validate","open","knowledge","format","export","bundles","bundle","import",{"skillNo":191,"name":192,"nameZh":6,"description":193,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":194,"categories":195,"keywords":196,"publishedAt":39},"01M6X5JA8S5PNGJVFW4EQY8T83","capability-management","Manage versioned Capability entities and their governed lifecycle.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fcapability-management\u002FSKILL.md",[],[100,197,198,102,103,143,105,163,33,106,144,199,107,146,109,169,149,150,200,113],"capability","capability management","queries","changes"]