[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6CSZ787KNGCFPJ5SVSXV900":3,"skill-same:01M6CSZ787KNGCFPJ5SVSXV900":124},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":27,"linkOnly":28,"markdownContent":29,"markdownBody":30,"frontMatter":31,"refName":34,"commitSha":35,"skillDirectory":36,"forksCount":37,"repositoryHtmlUrl":38,"repositoryHomepageUrl":38,"ownerHtmlUrl":39,"ownerBlogUrl":6,"ownerDescription":40,"updatedAt":41,"install":42,"resources":55},"01M6CSZ787KNGCFPJ5SVSXV900","embedding-strategies",null,"Guide to selecting and optimizing embedding models for vector search applications.","Sam-06060\u002Fjarvis-assistant","Sam-06060","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F208600848?v=4","01M6CRKT7C7FXD21V6HXF293S0","Samson Frank Ganta","01M6CRKTBEKYFD8YX5Z1GT99EE",6,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FSam-06060\u002Fjarvis-assistant\u002Fblob\u002Fmain\u002Fmodules\u002Fagent_skills\u002Fawesome-skills\u002Fplaybooks\u002Fembedding-strategies\u002FSKILL.md",[],[19,20,21,22,23,24,25,26],"embedding strategies","search","vector","Vector Search","applications","guide","models","community","2026-11-10 10:12:30",false,"---\nname: embedding-strategies\ndescription: \"Guide to selecting and optimizing embedding models for vector search applications.\"\nrisk: unknown\nsource: community\ndate_added: \"2026-02-27\"\n---\n\n# Embedding Strategies\n\nGuide to selecting and optimizing embedding models for vector search applications.\n\n## Do not use this skill when\n\n- The task is unrelated to embedding strategies\n- You need a different domain or tool outside this scope\n\n## Instructions\n\n- Clarify goals, constraints, and required inputs.\n- Apply relevant best practices and validate outcomes.\n- Provide actionable steps and verification.\n- If detailed examples are required, open `resources\u002Fimplementation-playbook.md`.\n\n## Use this skill when\n\n- Choosing embedding models for RAG\n- Optimizing chunking strategies\n- Fine-tuning embeddings for domains\n- Comparing embedding model performance\n- Reducing embedding dimensions\n- Handling multilingual content\n\n## Core Concepts\n\n### 1. Embedding Model Comparison\n\n| Model | Dimensions | Max Tokens | Best For |\n|-------|------------|------------|----------|\n| **text-embedding-3-large** | 3072 | 8191 | High accuracy |\n| **text-embedding-3-small** | 1536 | 8191 | Cost-effective |\n| **voyage-2** | 1024 | 4000 | Code, legal |\n| **bge-large-en-v1.5** | 1024 | 512 | Open source |\n| **all-MiniLM-L6-v2** | 384 | 256 | Fast, lightweight |\n| **multilingual-e5-large** | 1024 | 512 | Multi-language |\n\n### 2. Embedding Pipeline\n\n```\nDocument → Chunking → Preprocessing → Embedding Model → Vector\n                ↓\n        [Overlap, Size]  [Clean, Normalize]  [API\u002FLocal]\n```\n\n## Templates\n\n### Template 1: OpenAI Embeddings\n\n```python\nfrom openai import OpenAI\nfrom typing import List\nimport numpy as np\n\nclient = OpenAI()\n\ndef get_embeddings(\n    texts: List[str],\n    model: str = \"text-embedding-3-small\",\n    dimensions: int = None\n) -> List[List[float]]:\n    \"\"\"Get embeddings from OpenAI.\"\"\"\n    # Handle batching for large lists\n    batch_size = 100\n    all_embeddings = []\n\n    for i in range(0, len(texts), batch_size):\n        batch = texts[i:i + batch_size]\n\n        kwargs = {\"input\": batch, \"model\": model}\n        if dimensions:\n            kwargs[\"dimensions\"] = dimensions\n\n        response = client.embeddings.create(**kwargs)\n        embeddings = [item.embedding for item in response.data]\n        all_embeddings.extend(embeddings)\n\n    return all_embeddings\n\n\ndef get_embedding(text: str, **kwargs) -> List[float]:\n    \"\"\"Get single embedding.\"\"\"\n    return get_embeddings([text], **kwargs)[0]\n\n\n# Dimension reduction with OpenAI\ndef get_reduced_embedding(text: str, dimensions: int = 512) -> List[float]:\n    \"\"\"Get embedding with reduced dimensions (Matryoshka).\"\"\"\n    return get_embedding(\n        text,\n        model=\"text-embedding-3-small\",\n        dimensions=dimensions\n    )\n```\n\n### Template 2: Local Embeddings with Sentence Transformers\n\n```python\nfrom sentence_transformers import SentenceTransformer\nfrom typing import List, Optional\nimport numpy as np\n\nclass LocalEmbedder:\n    \"\"\"Local embedding with sentence-transformers.\"\"\"\n\n    def __init__(\n        self,\n        model_name: str = \"BAAI\u002Fbge-large-en-v1.5\",\n        device: str = \"cuda\"\n    ):\n        self.model = SentenceTransformer(model_name, device=device)\n\n    def embed(\n        self,\n        texts: List[str],\n        normalize: bool = True,\n        show_progress: bool = False\n    ) -> np.ndarray:\n        \"\"\"Embed texts with optional normalization.\"\"\"\n        embeddings = self.model.encode(\n            texts,\n            normalize_embeddings=normalize,\n            show_progress_bar=show_progress,\n            convert_to_numpy=True\n        )\n        return embeddings\n\n    def embed_query(self, query: str) -> np.ndarray:\n        \"\"\"Embed a query with BGE-style prefix.\"\"\"\n        # BGE models benefit from query prefix\n        if \"bge\" in self.model.get_sentence_embedding_dimension():\n            query = f\"Represent this sentence for searching relevant passages: {query}\"\n        return self.embed([query])[0]\n\n    def embed_documents(self, documents: List[str]) -> np.ndarray:\n        \"\"\"Embed documents for indexing.\"\"\"\n        return self.embed(documents)\n\n\n# E5 model with instructions\nclass E5Embedder:\n    def __init__(self, model_name: str = \"intfloat\u002Fmultilingual-e5-large\"):\n        self.model = SentenceTransformer(model_name)\n\n    def embed_query(self, query: str) -> np.ndarray:\n        return self.model.encode(f\"query: {query}\")\n\n    def embed_document(self, document: str) -> np.ndarray:\n        return self.model.encode(f\"passage: {document}\")\n```\n\n### Template 3: Chunking Strategies\n\n```python\nfrom typing import List, Tuple\nimport re\n\ndef chunk_by_tokens(\n    text: str,\n    chunk_size: int = 512,\n    chunk_overlap: int = 50,\n    tokenizer=None\n) -> List[str]:\n    \"\"\"Chunk text by token count.\"\"\"\n    import tiktoken\n    tokenizer = tokenizer or tiktoken.get_encoding(\"cl100k_base\")\n\n    tokens = tokenizer.encode(text)\n    chunks = []\n\n    start = 0\n    while start \u003C len(tokens):\n        end = start + chunk_size\n        chunk_tokens = tokens[start:end]\n        chunk_text = tokenizer.decode(chunk_tokens)\n        chunks.append(chunk_text)\n        start = end - chunk_overlap\n\n    return chunks\n\n\ndef chunk_by_sentences(\n    text: str,\n    max_chunk_size: int = 1000,\n    min_chunk_size: int = 100\n) -> List[str]:\n    \"\"\"Chunk text by sentences, respecting size limits.\"\"\"\n    import nltk\n    sentences = nltk.sent_tokenize(text)\n\n    chunks = []\n    current_chunk = []\n    current_size = 0\n\n    for sentence in sentences:\n        sentence_size = len(sentence)\n\n        if current_size + sentence_size > max_chunk_size and current_chunk:\n            chunks.append(\" \".join(current_chunk))\n            current_chunk = []\n            current_size = 0\n\n        current_chunk.append(sentence)\n        current_size += sentence_size\n\n    if current_chunk:\n        chunks.append(\" \".join(current_chunk))\n\n    return chunks\n\n\ndef chunk_by_semantic_sections(\n    text: str,\n    headers_pattern: str = r'^#{1,3}\\s+.+$'\n) -> List[Tuple[str, str]]:\n    \"\"\"Chunk markdown by headers, preserving hierarchy.\"\"\"\n    lines = text.split('\\n')\n    chunks = []\n    current_header = \"\"\n    current_content = []\n\n    for line in lines:\n        if re.match(headers_pattern, line, re.MULTILINE):\n            if current_content:\n                chunks.append((current_header, '\\n'.join(current_content)))\n            current_header = line\n            current_content = []\n        else:\n            current_content.append(line)\n\n    if current_content:\n        chunks.append((current_header, '\\n'.join(current_content)))\n\n    return chunks\n\n\ndef recursive_character_splitter(\n    text: str,\n    chunk_size: int = 1000,\n    chunk_overlap: int = 200,\n    separators: List[str] = None\n) -> List[str]:\n    \"\"\"LangChain-style recursive splitter.\"\"\"\n    separators = separators or [\"\\n\\n\", \"\\n\", \". \", \" \", \"\"]\n\n    def split_text(text: str, separators: List[str]) -> List[str]:\n        if not text:\n            return []\n\n        separator = separators[0]\n        remaining_separators = separators[1:]\n\n        if separator == \"\":\n            # Character-level split\n            return [text[i:i+chunk_size] for i in range(0, len(text), chunk_size - chunk_overlap)]\n\n        splits = text.split(separator)\n        chunks = []\n        current_chunk = []\n        current_length = 0\n\n        for split in splits:\n            split_length = len(split) + len(separator)\n\n            if current_length + split_length > chunk_size and current_chunk:\n                chunk_text = separator.join(current_chunk)\n\n                # Recursively split if still too large\n                if len(chunk_text) > chunk_size and remaining_separators:\n                    chunks.extend(split_text(chunk_text, remaining_separators))\n                else:\n                    chunks.append(chunk_text)\n\n                # Start new chunk with overlap\n                overlap_splits = []\n                overlap_length = 0\n                for s in reversed(current_chunk):\n                    if overlap_length + len(s) \u003C= chunk_overlap:\n                        overlap_splits.insert(0, s)\n                        overlap_length += len(s)\n                    else:\n                        break\n                current_chunk = overlap_splits\n                current_length = overlap_length\n\n            current_chunk.append(split)\n            current_length += split_length\n\n        if current_chunk:\n            chunks.append(separator.join(current_chunk))\n\n        return chunks\n\n    return split_text(text, separators)\n```\n\n### Template 4: Domain-Specific Embedding Pipeline\n\n```python\nclass DomainEmbeddingPipeline:\n    \"\"\"Pipeline for domain-specific embeddings.\"\"\"\n\n    def __init__(\n        self,\n        embedding_model: str = \"text-embedding-3-small\",\n        chunk_size: int = 512,\n        chunk_overlap: int = 50,\n        preprocessing_fn=None\n    ):\n        self.embedding_model = embedding_model\n        self.chunk_size = chunk_size\n        self.chunk_overlap = chunk_overlap\n        self.preprocess = preprocessing_fn or self._default_preprocess\n\n    def _default_preprocess(self, text: str) -> str:\n        \"\"\"Default preprocessing.\"\"\"\n        # Remove excessive whitespace\n        text = re.sub(r'\\s+', ' ', text)\n        # Remove special characters\n        text = re.sub(r'[^\\w\\s.,!?-]', '', text)\n        return text.strip()\n\n    async def process_documents(\n        self,\n        documents: List[dict],\n        id_field: str = \"id\",\n        content_field: str = \"content\",\n        metadata_fields: List[str] = None\n    ) -> List[dict]:\n        \"\"\"Process documents for vector storage.\"\"\"\n        processed = []\n\n        for doc in documents:\n            content = doc[content_field]\n            doc_id = doc[id_field]\n\n            # Preprocess\n            cleaned = self.preprocess(content)\n\n            # Chunk\n            chunks = chunk_by_tokens(\n                cleaned,\n                self.chunk_size,\n                self.chunk_overlap\n            )\n\n            # Create embeddings\n            embeddings = get_embeddings(chunks, self.embedding_model)\n\n            # Create records\n            for i, (chunk, embedding) in enumerate(zip(chunks, embeddings)):\n                record = {\n                    \"id\": f\"{doc_id}_chunk_{i}\",\n                    \"document_id\": doc_id,\n                    \"chunk_index\": i,\n                    \"text\": chunk,\n                    \"embedding\": embedding\n                }\n\n                # Add metadata\n                if metadata_fields:\n                    for field in metadata_fields:\n                        if field in doc:\n                            record[field] = doc[field]\n\n                processed.append(record)\n\n        return processed\n\n\n# Code-specific pipeline\nclass CodeEmbeddingPipeline:\n    \"\"\"Specialized pipeline for code embeddings.\"\"\"\n\n    def __init__(self, model: str = \"voyage-code-2\"):\n        self.model = model\n\n    def chunk_code(self, code: str, language: str) -> List[dict]:\n        \"\"\"Chunk code by functions\u002Fclasses.\"\"\"\n        import tree_sitter\n\n        # Parse with tree-sitter\n        # Extract functions, classes, methods\n        # Return chunks with context\n        pass\n\n    def embed_with_context(self, chunk: str, context: str) -> List[float]:\n        \"\"\"Embed code with surrounding context.\"\"\"\n        combined = f\"Context: {context}\\n\\nCode:\\n{chunk}\"\n        return get_embedding(combined, model=self.model)\n```\n\n### Template 5: Embedding Quality Evaluation\n\n```python\nimport numpy as np\nfrom typing import List, Tuple\n\ndef evaluate_retrieval_quality(\n    queries: List[str],\n    relevant_docs: List[List[str]],  # List of relevant doc IDs per query\n    retrieved_docs: List[List[str]],  # List of retrieved doc IDs per query\n    k: int = 10\n) -> dict:\n    \"\"\"Evaluate embedding quality for retrieval.\"\"\"\n\n    def precision_at_k(relevant: set, retrieved: List[str], k: int) -> float:\n        retrieved_k = retrieved[:k]\n        relevant_retrieved = len(set(retrieved_k) & relevant)\n        return relevant_retrieved \u002F k\n\n    def recall_at_k(relevant: set, retrieved: List[str], k: int) -> float:\n        retrieved_k = retrieved[:k]\n        relevant_retrieved = len(set(retrieved_k) & relevant)\n        return relevant_retrieved \u002F len(relevant) if relevant else 0\n\n    def mrr(relevant: set, retrieved: List[str]) -> float:\n        for i, doc in enumerate(retrieved):\n            if doc in relevant:\n                return 1 \u002F (i + 1)\n        return 0\n\n    def ndcg_at_k(relevant: set, retrieved: List[str], k: int) -> float:\n        dcg = sum(\n            1 \u002F np.log2(i + 2) if doc in relevant else 0\n            for i, doc in enumerate(retrieved[:k])\n        )\n        ideal_dcg = sum(1 \u002F np.log2(i + 2) for i in range(min(len(relevant), k)))\n        return dcg \u002F ideal_dcg if ideal_dcg > 0 else 0\n\n    metrics = {\n        f\"precision@{k}\": [],\n        f\"recall@{k}\": [],\n        \"mrr\": [],\n        f\"ndcg@{k}\": []\n    }\n\n    for relevant, retrieved in zip(relevant_docs, retrieved_docs):\n        relevant_set = set(relevant)\n        metrics[f\"precision@{k}\"].append(precision_at_k(relevant_set, retrieved, k))\n        metrics[f\"recall@{k}\"].append(recall_at_k(relevant_set, retrieved, k))\n        metrics[\"mrr\"].append(mrr(relevant_set, retrieved))\n        metrics[f\"ndcg@{k}\"].append(ndcg_at_k(relevant_set, retrieved, k))\n\n    return {name: np.mean(values) for name, values in metrics.items()}\n\n\ndef compute_embedding_similarity(\n    embeddings1: np.ndarray,\n    embeddings2: np.ndarray,\n    metric: str = \"cosine\"\n) -> np.ndarray:\n    \"\"\"Compute similarity matrix between embedding sets.\"\"\"\n    if metric == \"cosine\":\n        # Normalize\n        norm1 = embeddings1 \u002F np.linalg.norm(embeddings1, axis=1, keepdims=True)\n        norm2 = embeddings2 \u002F np.linalg.norm(embeddings2, axis=1, keepdims=True)\n        return norm1 @ norm2.T\n    elif metric == \"euclidean\":\n        from scipy.spatial.distance import cdist\n        return -cdist(embeddings1, embeddings2, metric='euclidean')\n    elif metric == \"dot\":\n        return embeddings1 @ embeddings2.T\n```\n\n## Best Practices\n\n### Do's\n- **Match model to use case** - Code vs prose vs multilingual\n- **Chunk thoughtfully** - Preserve semantic boundaries\n- **Normalize embeddings** - For cosine similarity\n- **Batch requests** - More efficient than one-by-one\n- **Cache embeddings** - Avoid recomputing\n\n### Don'ts\n- **Don't ignore token limits** - Truncation loses info\n- **Don't mix embedding models** - Incompatible spaces\n- **Don't skip preprocessing** - Garbage in, garbage out\n- **Don't over-chunk** - Lose context\n\n## Resources\n\n- [OpenAI Embeddings](https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Fembeddings)\n- [Sentence Transformers](https:\u002F\u002Fwww.sbert.net\u002F)\n- [MTEB Benchmark](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fmteb\u002Fleaderboard)\n","# Embedding Strategies\n\nGuide to selecting and optimizing embedding models for vector search applications.\n\n## Do not use this skill when\n\n- The task is unrelated to embedding strategies\n- You need a different domain or tool outside this scope\n\n## Instructions\n\n- Clarify goals, constraints, and required inputs.\n- Apply relevant best practices and validate outcomes.\n- Provide actionable steps and verification.\n- If detailed examples are required, open `resources\u002Fimplementation-playbook.md`.\n\n## Use this skill when\n\n- Choosing embedding models for RAG\n- Optimizing chunking strategies\n- Fine-tuning embeddings for domains\n- Comparing embedding model performance\n- Reducing embedding dimensions\n- Handling multilingual content\n\n## Core Concepts\n\n### 1. Embedding Model Comparison\n\n| Model | Dimensions | Max Tokens | Best For |\n|-------|------------|------------|----------|\n| **text-embedding-3-large** | 3072 | 8191 | High accuracy |\n| **text-embedding-3-small** | 1536 | 8191 | Cost-effective |\n| **voyage-2** | 1024 | 4000 | Code, legal |\n| **bge-large-en-v1.5** | 1024 | 512 | Open source |\n| **all-MiniLM-L6-v2** | 384 | 256 | Fast, lightweight |\n| **multilingual-e5-large** | 1024 | 512 | Multi-language |\n\n### 2. Embedding Pipeline\n\n```\nDocument → Chunking → Preprocessing → Embedding Model → Vector\n                ↓\n        [Overlap, Size]  [Clean, Normalize]  [API\u002FLocal]\n```\n\n## Templates\n\n### Template 1: OpenAI Embeddings\n\n```python\nfrom openai import OpenAI\nfrom typing import List\nimport numpy as np\n\nclient = OpenAI()\n\ndef get_embeddings(\n    texts: List[str],\n    model: str = \"text-embedding-3-small\",\n    dimensions: int = None\n) -> List[List[float]]:\n    \"\"\"Get embeddings from OpenAI.\"\"\"\n    # Handle batching for large lists\n    batch_size = 100\n    all_embeddings = []\n\n    for i in range(0, len(texts), batch_size):\n        batch = texts[i:i + batch_size]\n\n        kwargs = {\"input\": batch, \"model\": model}\n        if dimensions:\n            kwargs[\"dimensions\"] = dimensions\n\n        response = client.embeddings.create(**kwargs)\n        embeddings = [item.embedding for item in response.data]\n        all_embeddings.extend(embeddings)\n\n    return all_embeddings\n\n\ndef get_embedding(text: str, **kwargs) -> List[float]:\n    \"\"\"Get single embedding.\"\"\"\n    return get_embeddings([text], **kwargs)[0]\n\n\n# Dimension reduction with OpenAI\ndef get_reduced_embedding(text: str, dimensions: int = 512) -> List[float]:\n    \"\"\"Get embedding with reduced dimensions (Matryoshka).\"\"\"\n    return get_embedding(\n        text,\n        model=\"text-embedding-3-small\",\n        dimensions=dimensions\n    )\n```\n\n### Template 2: Local Embeddings with Sentence Transformers\n\n```python\nfrom sentence_transformers import SentenceTransformer\nfrom typing import List, Optional\nimport numpy as np\n\nclass LocalEmbedder:\n    \"\"\"Local embedding with sentence-transformers.\"\"\"\n\n    def __init__(\n        self,\n        model_name: str = \"BAAI\u002Fbge-large-en-v1.5\",\n        device: str = \"cuda\"\n    ):\n        self.model = SentenceTransformer(model_name, device=device)\n\n    def embed(\n        self,\n        texts: List[str],\n        normalize: bool = True,\n        show_progress: bool = False\n    ) -> np.ndarray:\n        \"\"\"Embed texts with optional normalization.\"\"\"\n        embeddings = self.model.encode(\n            texts,\n            normalize_embeddings=normalize,\n            show_progress_bar=show_progress,\n            convert_to_numpy=True\n        )\n        return embeddings\n\n    def embed_query(self, query: str) -> np.ndarray:\n        \"\"\"Embed a query with BGE-style prefix.\"\"\"\n        # BGE models benefit from query prefix\n        if \"bge\" in self.model.get_sentence_embedding_dimension():\n            query = f\"Represent this sentence for searching relevant passages: {query}\"\n        return self.embed([query])[0]\n\n    def embed_documents(self, documents: List[str]) -> np.ndarray:\n        \"\"\"Embed documents for indexing.\"\"\"\n        return self.embed(documents)\n\n\n# E5 model with instructions\nclass E5Embedder:\n    def __init__(self, model_name: str = \"intfloat\u002Fmultilingual-e5-large\"):\n        self.model = SentenceTransformer(model_name)\n\n    def embed_query(self, query: str) -> np.ndarray:\n        return self.model.encode(f\"query: {query}\")\n\n    def embed_document(self, document: str) -> np.ndarray:\n        return self.model.encode(f\"passage: {document}\")\n```\n\n### Template 3: Chunking Strategies\n\n```python\nfrom typing import List, Tuple\nimport re\n\ndef chunk_by_tokens(\n    text: str,\n    chunk_size: int = 512,\n    chunk_overlap: int = 50,\n    tokenizer=None\n) -> List[str]:\n    \"\"\"Chunk text by token count.\"\"\"\n    import tiktoken\n    tokenizer = tokenizer or tiktoken.get_encoding(\"cl100k_base\")\n\n    tokens = tokenizer.encode(text)\n    chunks = []\n\n    start = 0\n    while start \u003C len(tokens):\n        end = start + chunk_size\n        chunk_tokens = tokens[start:end]\n        chunk_text = tokenizer.decode(chunk_tokens)\n        chunks.append(chunk_text)\n        start = end - chunk_overlap\n\n    return chunks\n\n\ndef chunk_by_sentences(\n    text: str,\n    max_chunk_size: int = 1000,\n    min_chunk_size: int = 100\n) -> List[str]:\n    \"\"\"Chunk text by sentences, respecting size limits.\"\"\"\n    import nltk\n    sentences = nltk.sent_tokenize(text)\n\n    chunks = []\n    current_chunk = []\n    current_size = 0\n\n    for sentence in sentences:\n        sentence_size = len(sentence)\n\n        if current_size + sentence_size > max_chunk_size and current_chunk:\n            chunks.append(\" \".join(current_chunk))\n            current_chunk = []\n            current_size = 0\n\n        current_chunk.append(sentence)\n        current_size += sentence_size\n\n    if current_chunk:\n        chunks.append(\" \".join(current_chunk))\n\n    return chunks\n\n\ndef chunk_by_semantic_sections(\n    text: str,\n    headers_pattern: str = r'^#{1,3}\\s+.+$'\n) -> List[Tuple[str, str]]:\n    \"\"\"Chunk markdown by headers, preserving hierarchy.\"\"\"\n    lines = text.split('\\n')\n    chunks = []\n    current_header = \"\"\n    current_content = []\n\n    for line in lines:\n        if re.match(headers_pattern, line, re.MULTILINE):\n            if current_content:\n                chunks.append((current_header, '\\n'.join(current_content)))\n            current_header = line\n            current_content = []\n        else:\n            current_content.append(line)\n\n    if current_content:\n        chunks.append((current_header, '\\n'.join(current_content)))\n\n    return chunks\n\n\ndef recursive_character_splitter(\n    text: str,\n    chunk_size: int = 1000,\n    chunk_overlap: int = 200,\n    separators: List[str] = None\n) -> List[str]:\n    \"\"\"LangChain-style recursive splitter.\"\"\"\n    separators = separators or [\"\\n\\n\", \"\\n\", \". \", \" \", \"\"]\n\n    def split_text(text: str, separators: List[str]) -> List[str]:\n        if not text:\n            return []\n\n        separator = separators[0]\n        remaining_separators = separators[1:]\n\n        if separator == \"\":\n            # Character-level split\n            return [text[i:i+chunk_size] for i in range(0, len(text), chunk_size - chunk_overlap)]\n\n        splits = text.split(separator)\n        chunks = []\n        current_chunk = []\n        current_length = 0\n\n        for split in splits:\n            split_length = len(split) + len(separator)\n\n            if current_length + split_length > chunk_size and current_chunk:\n                chunk_text = separator.join(current_chunk)\n\n                # Recursively split if still too large\n                if len(chunk_text) > chunk_size and remaining_separators:\n                    chunks.extend(split_text(chunk_text, remaining_separators))\n                else:\n                    chunks.append(chunk_text)\n\n                # Start new chunk with overlap\n                overlap_splits = []\n                overlap_length = 0\n                for s in reversed(current_chunk):\n                    if overlap_length + len(s) \u003C= chunk_overlap:\n                        overlap_splits.insert(0, s)\n                        overlap_length += len(s)\n                    else:\n                        break\n                current_chunk = overlap_splits\n                current_length = overlap_length\n\n            current_chunk.append(split)\n            current_length += split_length\n\n        if current_chunk:\n            chunks.append(separator.join(current_chunk))\n\n        return chunks\n\n    return split_text(text, separators)\n```\n\n### Template 4: Domain-Specific Embedding Pipeline\n\n```python\nclass DomainEmbeddingPipeline:\n    \"\"\"Pipeline for domain-specific embeddings.\"\"\"\n\n    def __init__(\n        self,\n        embedding_model: str = \"text-embedding-3-small\",\n        chunk_size: int = 512,\n        chunk_overlap: int = 50,\n        preprocessing_fn=None\n    ):\n        self.embedding_model = embedding_model\n        self.chunk_size = chunk_size\n        self.chunk_overlap = chunk_overlap\n        self.preprocess = preprocessing_fn or self._default_preprocess\n\n    def _default_preprocess(self, text: str) -> str:\n        \"\"\"Default preprocessing.\"\"\"\n        # Remove excessive whitespace\n        text = re.sub(r'\\s+', ' ', text)\n        # Remove special characters\n        text = re.sub(r'[^\\w\\s.,!?-]', '', text)\n        return text.strip()\n\n    async def process_documents(\n        self,\n        documents: List[dict],\n        id_field: str = \"id\",\n        content_field: str = \"content\",\n        metadata_fields: List[str] = None\n    ) -> List[dict]:\n        \"\"\"Process documents for vector storage.\"\"\"\n        processed = []\n\n        for doc in documents:\n            content = doc[content_field]\n            doc_id = doc[id_field]\n\n            # Preprocess\n            cleaned = self.preprocess(content)\n\n            # Chunk\n            chunks = chunk_by_tokens(\n                cleaned,\n                self.chunk_size,\n                self.chunk_overlap\n            )\n\n            # Create embeddings\n            embeddings = get_embeddings(chunks, self.embedding_model)\n\n            # Create records\n            for i, (chunk, embedding) in enumerate(zip(chunks, embeddings)):\n                record = {\n                    \"id\": f\"{doc_id}_chunk_{i}\",\n                    \"document_id\": doc_id,\n                    \"chunk_index\": i,\n                    \"text\": chunk,\n                    \"embedding\": embedding\n                }\n\n                # Add metadata\n                if metadata_fields:\n                    for field in metadata_fields:\n                        if field in doc:\n                            record[field] = doc[field]\n\n                processed.append(record)\n\n        return processed\n\n\n# Code-specific pipeline\nclass CodeEmbeddingPipeline:\n    \"\"\"Specialized pipeline for code embeddings.\"\"\"\n\n    def __init__(self, model: str = \"voyage-code-2\"):\n        self.model = model\n\n    def chunk_code(self, code: str, language: str) -> List[dict]:\n        \"\"\"Chunk code by functions\u002Fclasses.\"\"\"\n        import tree_sitter\n\n        # Parse with tree-sitter\n        # Extract functions, classes, methods\n        # Return chunks with context\n        pass\n\n    def embed_with_context(self, chunk: str, context: str) -> List[float]:\n        \"\"\"Embed code with surrounding context.\"\"\"\n        combined = f\"Context: {context}\\n\\nCode:\\n{chunk}\"\n        return get_embedding(combined, model=self.model)\n```\n\n### Template 5: Embedding Quality Evaluation\n\n```python\nimport numpy as np\nfrom typing import List, Tuple\n\ndef evaluate_retrieval_quality(\n    queries: List[str],\n    relevant_docs: List[List[str]],  # List of relevant doc IDs per query\n    retrieved_docs: List[List[str]],  # List of retrieved doc IDs per query\n    k: int = 10\n) -> dict:\n    \"\"\"Evaluate embedding quality for retrieval.\"\"\"\n\n    def precision_at_k(relevant: set, retrieved: List[str], k: int) -> float:\n        retrieved_k = retrieved[:k]\n        relevant_retrieved = len(set(retrieved_k) & relevant)\n        return relevant_retrieved \u002F k\n\n    def recall_at_k(relevant: set, retrieved: List[str], k: int) -> float:\n        retrieved_k = retrieved[:k]\n        relevant_retrieved = len(set(retrieved_k) & relevant)\n        return relevant_retrieved \u002F len(relevant) if relevant else 0\n\n    def mrr(relevant: set, retrieved: List[str]) -> float:\n        for i, doc in enumerate(retrieved):\n            if doc in relevant:\n                return 1 \u002F (i + 1)\n        return 0\n\n    def ndcg_at_k(relevant: set, retrieved: List[str], k: int) -> float:\n        dcg = sum(\n            1 \u002F np.log2(i + 2) if doc in relevant else 0\n            for i, doc in enumerate(retrieved[:k])\n        )\n        ideal_dcg = sum(1 \u002F np.log2(i + 2) for i in range(min(len(relevant), k)))\n        return dcg \u002F ideal_dcg if ideal_dcg > 0 else 0\n\n    metrics = {\n        f\"precision@{k}\": [],\n        f\"recall@{k}\": [],\n        \"mrr\": [],\n        f\"ndcg@{k}\": []\n    }\n\n    for relevant, retrieved in zip(relevant_docs, retrieved_docs):\n        relevant_set = set(relevant)\n        metrics[f\"precision@{k}\"].append(precision_at_k(relevant_set, retrieved, k))\n        metrics[f\"recall@{k}\"].append(recall_at_k(relevant_set, retrieved, k))\n        metrics[\"mrr\"].append(mrr(relevant_set, retrieved))\n        metrics[f\"ndcg@{k}\"].append(ndcg_at_k(relevant_set, retrieved, k))\n\n    return {name: np.mean(values) for name, values in metrics.items()}\n\n\ndef compute_embedding_similarity(\n    embeddings1: np.ndarray,\n    embeddings2: np.ndarray,\n    metric: str = \"cosine\"\n) -> np.ndarray:\n    \"\"\"Compute similarity matrix between embedding sets.\"\"\"\n    if metric == \"cosine\":\n        # Normalize\n        norm1 = embeddings1 \u002F np.linalg.norm(embeddings1, axis=1, keepdims=True)\n        norm2 = embeddings2 \u002F np.linalg.norm(embeddings2, axis=1, keepdims=True)\n        return norm1 @ norm2.T\n    elif metric == \"euclidean\":\n        from scipy.spatial.distance import cdist\n        return -cdist(embeddings1, embeddings2, metric='euclidean')\n    elif metric == \"dot\":\n        return embeddings1 @ embeddings2.T\n```\n\n## Best Practices\n\n### Do's\n- **Match model to use case** - Code vs prose vs multilingual\n- **Chunk thoughtfully** - Preserve semantic boundaries\n- **Normalize embeddings** - For cosine similarity\n- **Batch requests** - More efficient than one-by-one\n- **Cache embeddings** - Avoid recomputing\n\n### Don'ts\n- **Don't ignore token limits** - Truncation loses info\n- **Don't mix embedding models** - Incompatible spaces\n- **Don't skip preprocessing** - Garbage in, garbage out\n- **Don't over-chunk** - Lose context\n\n## Resources\n\n- [OpenAI Embeddings](https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Fembeddings)\n- [Sentence Transformers](https:\u002F\u002Fwww.sbert.net\u002F)\n- [MTEB Benchmark](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fmteb\u002Fleaderboard)\n",{"name":5,"risk":32,"source":26,"date_added":33,"description":7},"unknown","2026-02-27","main","a50a3aa6b39fb0e1b7db88bef1bfc9c93606e126","modules\u002Fagent_skills\u002Fawesome-skills\u002Fplaybooks\u002Fembedding-strategies",0,"https:\u002F\u002Fgithub.com\u002FSam-06060\u002Fjarvis-assistant","https:\u002F\u002Fgithub.com\u002FSam-06060","Mobile Application Developer | Kotlin | Flutter, KMP, Swfit, Android,Swift UI | Building at Saar It Resources | Creator of Jarvis (AI Assistant)","2026-06-23 05:13:38",{"source":8,"skillName":5,"commands":43,"prompt":53,"zipAvailable":54},[44,47,50],{"tool":45,"command":46},"npx","npx skills add Sam-06060\u002Fjarvis-assistant --skill embedding-strategies",{"tool":48,"command":49},"bunx","bunx skills add Sam-06060\u002Fjarvis-assistant --skill embedding-strategies",{"tool":51,"command":52},"pnpm","pnpm dlx skills add Sam-06060\u002Fjarvis-assistant --skill embedding-strategies","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FSam-06060\u002Fjarvis-assistant\u002Fblob\u002Fmain\u002Fmodules\u002Fagent_skills\u002Fawesome-skills\u002Fplaybooks\u002Fembedding-strategies\u002FSKILL.md",true,[56,62,68,72,78,82,86,90,94,98,102,106,110,114,118],{"path":57,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":60,"sourceHtmlUrl":61},"agents\u002Fopenai.yaml","REFERENCE","参考文档",300,"https:\u002F\u002Fgithub.com\u002FMishit18\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fmedia\u002Fspeech\u002Fagents\u002Fopenai.yaml",{"path":63,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":66,"sourceHtmlUrl":67},"assets\u002Fspeech-small.svg","ASSET","资产文件",742,"https:\u002F\u002Fgithub.com\u002FMishit18\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fmedia\u002Fspeech\u002Fassets\u002Fspeech-small.svg",{"path":69,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":70,"sourceHtmlUrl":71},"assets\u002Fspeech.png",1234,"https:\u002F\u002Fgithub.com\u002FMishit18\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fmedia\u002Fspeech\u002Fassets\u002Fspeech.png",{"path":73,"resourceType":74,"resourceTypeDesc":75,"sizeBytes":76,"sourceHtmlUrl":77},"LICENSE.txt","LICENSE","许可证",10777,"https:\u002F\u002Fgithub.com\u002FMishit18\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fmedia\u002Fspeech\u002FLICENSE.txt",{"path":79,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Faccessibility.md",697,"https:\u002F\u002Fgithub.com\u002FMishit18\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fmedia\u002Fspeech\u002Freferences\u002Faccessibility.md",{"path":83,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":84,"sourceHtmlUrl":85},"references\u002Faudio-api.md",903,"https:\u002F\u002Fgithub.com\u002FMishit18\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fmedia\u002Fspeech\u002Freferences\u002Faudio-api.md",{"path":87,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":88,"sourceHtmlUrl":89},"references\u002Fcli.md",3276,"https:\u002F\u002Fgithub.com\u002FMishit18\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fmedia\u002Fspeech\u002Freferences\u002Fcli.md",{"path":91,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":92,"sourceHtmlUrl":93},"references\u002Fcodex-network.md",1149,"https:\u002F\u002Fgithub.com\u002FMishit18\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fmedia\u002Fspeech\u002Freferences\u002Fcodex-network.md",{"path":95,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":96,"sourceHtmlUrl":97},"references\u002Fivr.md",709,"https:\u002F\u002Fgithub.com\u002FMishit18\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fmedia\u002Fspeech\u002Freferences\u002Fivr.md",{"path":99,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":100,"sourceHtmlUrl":101},"references\u002Fnarration.md",672,"https:\u002F\u002Fgithub.com\u002FMishit18\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fmedia\u002Fspeech\u002Freferences\u002Fnarration.md",{"path":103,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":104,"sourceHtmlUrl":105},"references\u002Fprompting.md",1554,"https:\u002F\u002Fgithub.com\u002FMishit18\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fmedia\u002Fspeech\u002Freferences\u002Fprompting.md",{"path":107,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":108,"sourceHtmlUrl":109},"references\u002Fsample-prompts.md",992,"https:\u002F\u002Fgithub.com\u002FMishit18\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fmedia\u002Fspeech\u002Freferences\u002Fsample-prompts.md",{"path":111,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":112,"sourceHtmlUrl":113},"references\u002Fvoice-directions.md",2156,"https:\u002F\u002Fgithub.com\u002FMishit18\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fmedia\u002Fspeech\u002Freferences\u002Fvoice-directions.md",{"path":115,"resourceType":58,"resourceTypeDesc":59,"sizeBytes":116,"sourceHtmlUrl":117},"references\u002Fvoiceover.md",791,"https:\u002F\u002Fgithub.com\u002FMishit18\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fmedia\u002Fspeech\u002Freferences\u002Fvoiceover.md",{"path":119,"resourceType":120,"resourceTypeDesc":121,"sizeBytes":122,"sourceHtmlUrl":123},"scripts\u002Ftext_to_speech.py","SCRIPT","脚本",15593,"https:\u002F\u002Fgithub.com\u002FMishit18\u002Fclaude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fmedia\u002Fspeech\u002Fscripts\u002Ftext_to_speech.py",[125,138,154,180,199,226],{"skillNo":126,"name":127,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":128,"categories":129,"keywords":135,"publishedAt":27},"01M6GDRVS6FXX3ESH0C9DW01C0","test_skill","https:\u002F\u002Fgithub.com\u002FSam-06060\u002Fjarvis-assistant\u002Fblob\u002Fmain\u002Fscratch\u002Ftest_skill\u002FSKILL.md",[130],{"categoryNo":131,"categoryName":132,"taxonomyType":133,"taxonomyTypeDesc":134},"01JBSEEDCAT000000000000008","测试与质量","SCENARIO","使用场景",[136,137],"Test Skill","test",{"skillNo":139,"name":140,"nameZh":6,"description":141,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":142,"categories":143,"keywords":144,"publishedAt":27},"01M6GDRQ40HX22MA6XWC31RHHK","zustand-store-ts","Create Zustand stores following established patterns with proper TypeScript types and middleware.","https:\u002F\u002Fgithub.com\u002FSam-06060\u002Fjarvis-assistant\u002Fblob\u002Fmain\u002Fmodules\u002Fagent_skills\u002Fawesome-skills\u002Fplaybooks\u002Fzustand-store-ts\u002FSKILL.md",[],[145,146,147,148,149,150,151,26,152,153],"zustand store ts","zustand","store","with","types","middleware","patterns","create","typescript",{"skillNo":155,"name":156,"nameZh":6,"description":157,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":158,"categories":159,"keywords":163,"publishedAt":27},"01M6GDRKEA0WE670SPFWX5GKCN","zoom-automation","Automate Zoom meeting creation, management, recordings, webinars, and participant tracking via Rube MCP (Composio). Always search tools first for current schemas.","https:\u002F\u002Fgithub.com\u002FSam-06060\u002Fjarvis-assistant\u002Fblob\u002Fmain\u002Fmodules\u002Fagent_skills\u002Fawesome-skills\u002Fplaybooks\u002Fzoom-automation\u002FSKILL.md",[160],{"categoryNo":161,"categoryName":162,"taxonomyType":133,"taxonomyTypeDesc":134},"01JBSEEDCAT000000000000023","Skill与智能体开发",[164,165,166,167,168,20,169,170,171,172,173,174,175,176,26,177,178,179],"automation","zoom automation","zoom","first","Tools","MCP","via","meeting","automate","critical","creation","webinars","composio","recordings","management","tracking",{"skillNo":181,"name":182,"nameZh":6,"description":183,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":184,"categories":185,"keywords":187,"publishedAt":27},"01M6GDRFS1452PC71JQ38S24HR","zoho-crm-automation","Automate Zoho CRM tasks via Rube MCP (Composio): create\u002Fupdate records, search contacts, manage leads, and convert leads. Always search tools first for current schemas.","https:\u002F\u002Fgithub.com\u002FSam-06060\u002Fjarvis-assistant\u002Fblob\u002Fmain\u002Fmodules\u002Fagent_skills\u002Fawesome-skills\u002Fplaybooks\u002Fzoho-crm-automation\u002FSKILL.md",[186],{"categoryNo":161,"categoryName":162,"taxonomyType":133,"taxonomyTypeDesc":134},[188,164,189,190,191,192,193,26,20,194,176,195,173,172,168,196,152,197,169,198],"crm automation","zoho crm","crm","zoho crm automation","zoho","leads","contacts","records","update","tasks","manage",{"skillNo":200,"name":201,"nameZh":6,"description":202,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":203,"categories":204,"keywords":208,"publishedAt":27},"01M6GDRCKQBAVTGHX5XBH26KPZ","zod-validation-expert","Expert in Zod — TypeScript-first schema validation. Covers parsing, custom errors, refinements, type inference, and integration with React Hook Form, Next.js, and tRPC.","https:\u002F\u002Fgithub.com\u002FSam-06060\u002Fjarvis-assistant\u002Fblob\u002Fmain\u002Fmodules\u002Fagent_skills\u002Fawesome-skills\u002Fplaybooks\u002Fzod-validation-expert\u002FSKILL.md",[205],{"categoryNo":206,"categoryName":207,"taxonomyType":133,"taxonomyTypeDesc":134},"01JBSEEDCAT000000000000009","前端开发",[209,210,211,212,213,153,214,215,216,217,218,219,148,220,221,222,223,26,224,225],"expert","zod","zod validation expert","validation","react","safe","parsing","errors","schema validation","react hook form","hook","trpc","form","type","next","schema","Next.js",{"skillNo":227,"name":228,"nameZh":6,"description":229,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":230,"categories":231,"keywords":233,"publishedAt":27},"01M6GDR658B08K4MY0RWD4A6EH","zeroize-audit","Detects missing zeroization of sensitive data in source code and identifies zeroization removed by compiler optimizations, with assembly-level analysis, and control-flow verification. Use for auditing C\u002FC++\u002FRust code handling secrets, keys, passwords, or other sensitive data.","https:\u002F\u002Fgithub.com\u002FSam-06060\u002Fjarvis-assistant\u002Fblob\u002Fmain\u002Fmodules\u002Fagent_skills\u002Fawesome-skills\u002Fplaybooks\u002Fzeroize-audit\u002FSKILL.md",[232],{"categoryNo":161,"categoryName":162,"taxonomyType":133,"taxonomyTypeDesc":134},[234,235,236,237,238,239,240,241,26,242,243,244,245,246,247,248,249,250,251,169],"zeroize audit","audit","write","passwords","get","project","c++","activate","assembly","read","Bash","secrets","analysis","data","flow","verification","code","rust"]