[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M70NV44V7R3NFVF7VG7GY9WH":3,"skill-same:01M70NV44V7R3NFVF7VG7GY9WH":66},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":37,"linkOnly":38,"markdownContent":39,"markdownBody":40,"frontMatter":41,"refName":42,"commitSha":43,"skillDirectory":44,"forksCount":45,"repositoryHtmlUrl":46,"repositoryHomepageUrl":47,"ownerHtmlUrl":48,"ownerBlogUrl":49,"ownerDescription":50,"updatedAt":51,"install":52,"resources":65},"01M70NV44V7R3NFVF7VG7GY9WH","embedding-strategies",null,"Select and optimize embedding models for semantic search and RAG applications. Use when choosing embedding models, implementing chunking strategies, or optimizing embedding quality for specific domains.","shengdabai\u002FTony-Claude-Code-Skills","shengdabai","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F116417419?v=4","01M70F6D2WY5PQV5A92BKYJQEW","Tony Sheng","01M70F6D3GAAAYZ5H4CC1G8VXM",3,"MIT","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fwshobson-agents\u002Fplugins\u002Fllm-application-dev\u002Fskills\u002Fembedding-strategies\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36],"embedding strategies","strategies","embedding","semantic","semantic search","quality","rag","models","optimize","when","use","domains","implementing","applications","search","select","for","chunking","2026-11-10 10:12:30",false,"---\nname: embedding-strategies\ndescription: Select and optimize embedding models for semantic search and RAG applications. Use when choosing embedding models, implementing chunking strategies, or optimizing embedding quality for specific domains.\n---\n\n# Embedding Strategies\n\nGuide to selecting and optimizing embedding models for vector search applications.\n\n## When to Use This Skill\n\n- Choosing embedding models for RAG\n- Optimizing chunking strategies\n- Fine-tuning embeddings for domains\n- Comparing embedding model performance\n- Reducing embedding dimensions\n- Handling multilingual content\n\n## Core Concepts\n\n### 1. Embedding Model Comparison\n\n| Model | Dimensions | Max Tokens | Best For |\n|-------|------------|------------|----------|\n| **text-embedding-3-large** | 3072 | 8191 | High accuracy |\n| **text-embedding-3-small** | 1536 | 8191 | Cost-effective |\n| **voyage-2** | 1024 | 4000 | Code, legal |\n| **bge-large-en-v1.5** | 1024 | 512 | Open source |\n| **all-MiniLM-L6-v2** | 384 | 256 | Fast, lightweight |\n| **multilingual-e5-large** | 1024 | 512 | Multi-language |\n\n### 2. Embedding Pipeline\n\n```\nDocument → Chunking → Preprocessing → Embedding Model → Vector\n                ↓\n        [Overlap, Size]  [Clean, Normalize]  [API\u002FLocal]\n```\n\n## Templates\n\n### Template 1: OpenAI Embeddings\n\n```python\nfrom openai import OpenAI\nfrom typing import List\nimport numpy as np\n\nclient = OpenAI()\n\ndef get_embeddings(\n    texts: List[str],\n    model: str = \"text-embedding-3-small\",\n    dimensions: int = None\n) -> List[List[float]]:\n    \"\"\"Get embeddings from OpenAI.\"\"\"\n    # Handle batching for large lists\n    batch_size = 100\n    all_embeddings = []\n\n    for i in range(0, len(texts), batch_size):\n        batch = texts[i:i + batch_size]\n\n        kwargs = {\"input\": batch, \"model\": model}\n        if dimensions:\n            kwargs[\"dimensions\"] = dimensions\n\n        response = client.embeddings.create(**kwargs)\n        embeddings = [item.embedding for item in response.data]\n        all_embeddings.extend(embeddings)\n\n    return all_embeddings\n\n\ndef get_embedding(text: str, **kwargs) -> List[float]:\n    \"\"\"Get single embedding.\"\"\"\n    return get_embeddings([text], **kwargs)[0]\n\n\n# Dimension reduction with OpenAI\ndef get_reduced_embedding(text: str, dimensions: int = 512) -> List[float]:\n    \"\"\"Get embedding with reduced dimensions (Matryoshka).\"\"\"\n    return get_embedding(\n        text,\n        model=\"text-embedding-3-small\",\n        dimensions=dimensions\n    )\n```\n\n### Template 2: Local Embeddings with Sentence Transformers\n\n```python\nfrom sentence_transformers import SentenceTransformer\nfrom typing import List, Optional\nimport numpy as np\n\nclass LocalEmbedder:\n    \"\"\"Local embedding with sentence-transformers.\"\"\"\n\n    def __init__(\n        self,\n        model_name: str = \"BAAI\u002Fbge-large-en-v1.5\",\n        device: str = \"cuda\"\n    ):\n        self.model = SentenceTransformer(model_name, device=device)\n\n    def embed(\n        self,\n        texts: List[str],\n        normalize: bool = True,\n        show_progress: bool = False\n    ) -> np.ndarray:\n        \"\"\"Embed texts with optional normalization.\"\"\"\n        embeddings = self.model.encode(\n            texts,\n            normalize_embeddings=normalize,\n            show_progress_bar=show_progress,\n            convert_to_numpy=True\n        )\n        return embeddings\n\n    def embed_query(self, query: str) -> np.ndarray:\n        \"\"\"Embed a query with BGE-style prefix.\"\"\"\n        # BGE models benefit from query prefix\n        if \"bge\" in self.model.get_sentence_embedding_dimension():\n            query = f\"Represent this sentence for searching relevant passages: {query}\"\n        return self.embed([query])[0]\n\n    def embed_documents(self, documents: List[str]) -> np.ndarray:\n        \"\"\"Embed documents for indexing.\"\"\"\n        return self.embed(documents)\n\n\n# E5 model with instructions\nclass E5Embedder:\n    def __init__(self, model_name: str = \"intfloat\u002Fmultilingual-e5-large\"):\n        self.model = SentenceTransformer(model_name)\n\n    def embed_query(self, query: str) -> np.ndarray:\n        return self.model.encode(f\"query: {query}\")\n\n    def embed_document(self, document: str) -> np.ndarray:\n        return self.model.encode(f\"passage: {document}\")\n```\n\n### Template 3: Chunking Strategies\n\n```python\nfrom typing import List, Tuple\nimport re\n\ndef chunk_by_tokens(\n    text: str,\n    chunk_size: int = 512,\n    chunk_overlap: int = 50,\n    tokenizer=None\n) -> List[str]:\n    \"\"\"Chunk text by token count.\"\"\"\n    import tiktoken\n    tokenizer = tokenizer or tiktoken.get_encoding(\"cl100k_base\")\n\n    tokens = tokenizer.encode(text)\n    chunks = []\n\n    start = 0\n    while start \u003C len(tokens):\n        end = start + chunk_size\n        chunk_tokens = tokens[start:end]\n        chunk_text = tokenizer.decode(chunk_tokens)\n        chunks.append(chunk_text)\n        start = end - chunk_overlap\n\n    return chunks\n\n\ndef chunk_by_sentences(\n    text: str,\n    max_chunk_size: int = 1000,\n    min_chunk_size: int = 100\n) -> List[str]:\n    \"\"\"Chunk text by sentences, respecting size limits.\"\"\"\n    import nltk\n    sentences = nltk.sent_tokenize(text)\n\n    chunks = []\n    current_chunk = []\n    current_size = 0\n\n    for sentence in sentences:\n        sentence_size = len(sentence)\n\n        if current_size + sentence_size > max_chunk_size and current_chunk:\n            chunks.append(\" \".join(current_chunk))\n            current_chunk = []\n            current_size = 0\n\n        current_chunk.append(sentence)\n        current_size += sentence_size\n\n    if current_chunk:\n        chunks.append(\" \".join(current_chunk))\n\n    return chunks\n\n\ndef chunk_by_semantic_sections(\n    text: str,\n    headers_pattern: str = r'^#{1,3}\\s+.+$'\n) -> List[Tuple[str, str]]:\n    \"\"\"Chunk markdown by headers, preserving hierarchy.\"\"\"\n    lines = text.split('\\n')\n    chunks = []\n    current_header = \"\"\n    current_content = []\n\n    for line in lines:\n        if re.match(headers_pattern, line, re.MULTILINE):\n            if current_content:\n                chunks.append((current_header, '\\n'.join(current_content)))\n            current_header = line\n            current_content = []\n        else:\n            current_content.append(line)\n\n    if current_content:\n        chunks.append((current_header, '\\n'.join(current_content)))\n\n    return chunks\n\n\ndef recursive_character_splitter(\n    text: str,\n    chunk_size: int = 1000,\n    chunk_overlap: int = 200,\n    separators: List[str] = None\n) -> List[str]:\n    \"\"\"LangChain-style recursive splitter.\"\"\"\n    separators = separators or [\"\\n\\n\", \"\\n\", \". \", \" \", \"\"]\n\n    def split_text(text: str, separators: List[str]) -> List[str]:\n        if not text:\n            return []\n\n        separator = separators[0]\n        remaining_separators = separators[1:]\n\n        if separator == \"\":\n            # Character-level split\n            return [text[i:i+chunk_size] for i in range(0, len(text), chunk_size - chunk_overlap)]\n\n        splits = text.split(separator)\n        chunks = []\n        current_chunk = []\n        current_length = 0\n\n        for split in splits:\n            split_length = len(split) + len(separator)\n\n            if current_length + split_length > chunk_size and current_chunk:\n                chunk_text = separator.join(current_chunk)\n\n                # Recursively split if still too large\n                if len(chunk_text) > chunk_size and remaining_separators:\n                    chunks.extend(split_text(chunk_text, remaining_separators))\n                else:\n                    chunks.append(chunk_text)\n\n                # Start new chunk with overlap\n                overlap_splits = []\n                overlap_length = 0\n                for s in reversed(current_chunk):\n                    if overlap_length + len(s) \u003C= chunk_overlap:\n                        overlap_splits.insert(0, s)\n                        overlap_length += len(s)\n                    else:\n                        break\n                current_chunk = overlap_splits\n                current_length = overlap_length\n\n            current_chunk.append(split)\n            current_length += split_length\n\n        if current_chunk:\n            chunks.append(separator.join(current_chunk))\n\n        return chunks\n\n    return split_text(text, separators)\n```\n\n### Template 4: Domain-Specific Embedding Pipeline\n\n```python\nclass DomainEmbeddingPipeline:\n    \"\"\"Pipeline for domain-specific embeddings.\"\"\"\n\n    def __init__(\n        self,\n        embedding_model: str = \"text-embedding-3-small\",\n        chunk_size: int = 512,\n        chunk_overlap: int = 50,\n        preprocessing_fn=None\n    ):\n        self.embedding_model = embedding_model\n        self.chunk_size = chunk_size\n        self.chunk_overlap = chunk_overlap\n        self.preprocess = preprocessing_fn or self._default_preprocess\n\n    def _default_preprocess(self, text: str) -> str:\n        \"\"\"Default preprocessing.\"\"\"\n        # Remove excessive whitespace\n        text = re.sub(r'\\s+', ' ', text)\n        # Remove special characters\n        text = re.sub(r'[^\\w\\s.,!?-]', '', text)\n        return text.strip()\n\n    async def process_documents(\n        self,\n        documents: List[dict],\n        id_field: str = \"id\",\n        content_field: str = \"content\",\n        metadata_fields: List[str] = None\n    ) -> List[dict]:\n        \"\"\"Process documents for vector storage.\"\"\"\n        processed = []\n\n        for doc in documents:\n            content = doc[content_field]\n            doc_id = doc[id_field]\n\n            # Preprocess\n            cleaned = self.preprocess(content)\n\n            # Chunk\n            chunks = chunk_by_tokens(\n                cleaned,\n                self.chunk_size,\n                self.chunk_overlap\n            )\n\n            # Create embeddings\n            embeddings = get_embeddings(chunks, self.embedding_model)\n\n            # Create records\n            for i, (chunk, embedding) in enumerate(zip(chunks, embeddings)):\n                record = {\n                    \"id\": f\"{doc_id}_chunk_{i}\",\n                    \"document_id\": doc_id,\n                    \"chunk_index\": i,\n                    \"text\": chunk,\n                    \"embedding\": embedding\n                }\n\n                # Add metadata\n                if metadata_fields:\n                    for field in metadata_fields:\n                        if field in doc:\n                            record[field] = doc[field]\n\n                processed.append(record)\n\n        return processed\n\n\n# Code-specific pipeline\nclass CodeEmbeddingPipeline:\n    \"\"\"Specialized pipeline for code embeddings.\"\"\"\n\n    def __init__(self, model: str = \"voyage-code-2\"):\n        self.model = model\n\n    def chunk_code(self, code: str, language: str) -> List[dict]:\n        \"\"\"Chunk code by functions\u002Fclasses.\"\"\"\n        import tree_sitter\n\n        # Parse with tree-sitter\n        # Extract functions, classes, methods\n        # Return chunks with context\n        pass\n\n    def embed_with_context(self, chunk: str, context: str) -> List[float]:\n        \"\"\"Embed code with surrounding context.\"\"\"\n        combined = f\"Context: {context}\\n\\nCode:\\n{chunk}\"\n        return get_embedding(combined, model=self.model)\n```\n\n### Template 5: Embedding Quality Evaluation\n\n```python\nimport numpy as np\nfrom typing import List, Tuple\n\ndef evaluate_retrieval_quality(\n    queries: List[str],\n    relevant_docs: List[List[str]],  # List of relevant doc IDs per query\n    retrieved_docs: List[List[str]],  # List of retrieved doc IDs per query\n    k: int = 10\n) -> dict:\n    \"\"\"Evaluate embedding quality for retrieval.\"\"\"\n\n    def precision_at_k(relevant: set, retrieved: List[str], k: int) -> float:\n        retrieved_k = retrieved[:k]\n        relevant_retrieved = len(set(retrieved_k) & relevant)\n        return relevant_retrieved \u002F k\n\n    def recall_at_k(relevant: set, retrieved: List[str], k: int) -> float:\n        retrieved_k = retrieved[:k]\n        relevant_retrieved = len(set(retrieved_k) & relevant)\n        return relevant_retrieved \u002F len(relevant) if relevant else 0\n\n    def mrr(relevant: set, retrieved: List[str]) -> float:\n        for i, doc in enumerate(retrieved):\n            if doc in relevant:\n                return 1 \u002F (i + 1)\n        return 0\n\n    def ndcg_at_k(relevant: set, retrieved: List[str], k: int) -> float:\n        dcg = sum(\n            1 \u002F np.log2(i + 2) if doc in relevant else 0\n            for i, doc in enumerate(retrieved[:k])\n        )\n        ideal_dcg = sum(1 \u002F np.log2(i + 2) for i in range(min(len(relevant), k)))\n        return dcg \u002F ideal_dcg if ideal_dcg > 0 else 0\n\n    metrics = {\n        f\"precision@{k}\": [],\n        f\"recall@{k}\": [],\n        \"mrr\": [],\n        f\"ndcg@{k}\": []\n    }\n\n    for relevant, retrieved in zip(relevant_docs, retrieved_docs):\n        relevant_set = set(relevant)\n        metrics[f\"precision@{k}\"].append(precision_at_k(relevant_set, retrieved, k))\n        metrics[f\"recall@{k}\"].append(recall_at_k(relevant_set, retrieved, k))\n        metrics[\"mrr\"].append(mrr(relevant_set, retrieved))\n        metrics[f\"ndcg@{k}\"].append(ndcg_at_k(relevant_set, retrieved, k))\n\n    return {name: np.mean(values) for name, values in metrics.items()}\n\n\ndef compute_embedding_similarity(\n    embeddings1: np.ndarray,\n    embeddings2: np.ndarray,\n    metric: str = \"cosine\"\n) -> np.ndarray:\n    \"\"\"Compute similarity matrix between embedding sets.\"\"\"\n    if metric == \"cosine\":\n        # Normalize\n        norm1 = embeddings1 \u002F np.linalg.norm(embeddings1, axis=1, keepdims=True)\n        norm2 = embeddings2 \u002F np.linalg.norm(embeddings2, axis=1, keepdims=True)\n        return norm1 @ norm2.T\n    elif metric == \"euclidean\":\n        from scipy.spatial.distance import cdist\n        return -cdist(embeddings1, embeddings2, metric='euclidean')\n    elif metric == \"dot\":\n        return embeddings1 @ embeddings2.T\n```\n\n## Best Practices\n\n### Do's\n- **Match model to use case** - Code vs prose vs multilingual\n- **Chunk thoughtfully** - Preserve semantic boundaries\n- **Normalize embeddings** - For cosine similarity\n- **Batch requests** - More efficient than one-by-one\n- **Cache embeddings** - Avoid recomputing\n\n### Don'ts\n- **Don't ignore token limits** - Truncation loses info\n- **Don't mix embedding models** - Incompatible spaces\n- **Don't skip preprocessing** - Garbage in, garbage out\n- **Don't over-chunk** - Lose context\n\n## Resources\n\n- [OpenAI Embeddings](https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Fembeddings)\n- [Sentence Transformers](https:\u002F\u002Fwww.sbert.net\u002F)\n- [MTEB Benchmark](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fmteb\u002Fleaderboard)\n","# Embedding Strategies\n\nGuide to selecting and optimizing embedding models for vector search applications.\n\n## When to Use This Skill\n\n- Choosing embedding models for RAG\n- Optimizing chunking strategies\n- Fine-tuning embeddings for domains\n- Comparing embedding model performance\n- Reducing embedding dimensions\n- Handling multilingual content\n\n## Core Concepts\n\n### 1. Embedding Model Comparison\n\n| Model | Dimensions | Max Tokens | Best For |\n|-------|------------|------------|----------|\n| **text-embedding-3-large** | 3072 | 8191 | High accuracy |\n| **text-embedding-3-small** | 1536 | 8191 | Cost-effective |\n| **voyage-2** | 1024 | 4000 | Code, legal |\n| **bge-large-en-v1.5** | 1024 | 512 | Open source |\n| **all-MiniLM-L6-v2** | 384 | 256 | Fast, lightweight |\n| **multilingual-e5-large** | 1024 | 512 | Multi-language |\n\n### 2. Embedding Pipeline\n\n```\nDocument → Chunking → Preprocessing → Embedding Model → Vector\n                ↓\n        [Overlap, Size]  [Clean, Normalize]  [API\u002FLocal]\n```\n\n## Templates\n\n### Template 1: OpenAI Embeddings\n\n```python\nfrom openai import OpenAI\nfrom typing import List\nimport numpy as np\n\nclient = OpenAI()\n\ndef get_embeddings(\n    texts: List[str],\n    model: str = \"text-embedding-3-small\",\n    dimensions: int = None\n) -> List[List[float]]:\n    \"\"\"Get embeddings from OpenAI.\"\"\"\n    # Handle batching for large lists\n    batch_size = 100\n    all_embeddings = []\n\n    for i in range(0, len(texts), batch_size):\n        batch = texts[i:i + batch_size]\n\n        kwargs = {\"input\": batch, \"model\": model}\n        if dimensions:\n            kwargs[\"dimensions\"] = dimensions\n\n        response = client.embeddings.create(**kwargs)\n        embeddings = [item.embedding for item in response.data]\n        all_embeddings.extend(embeddings)\n\n    return all_embeddings\n\n\ndef get_embedding(text: str, **kwargs) -> List[float]:\n    \"\"\"Get single embedding.\"\"\"\n    return get_embeddings([text], **kwargs)[0]\n\n\n# Dimension reduction with OpenAI\ndef get_reduced_embedding(text: str, dimensions: int = 512) -> List[float]:\n    \"\"\"Get embedding with reduced dimensions (Matryoshka).\"\"\"\n    return get_embedding(\n        text,\n        model=\"text-embedding-3-small\",\n        dimensions=dimensions\n    )\n```\n\n### Template 2: Local Embeddings with Sentence Transformers\n\n```python\nfrom sentence_transformers import SentenceTransformer\nfrom typing import List, Optional\nimport numpy as np\n\nclass LocalEmbedder:\n    \"\"\"Local embedding with sentence-transformers.\"\"\"\n\n    def __init__(\n        self,\n        model_name: str = \"BAAI\u002Fbge-large-en-v1.5\",\n        device: str = \"cuda\"\n    ):\n        self.model = SentenceTransformer(model_name, device=device)\n\n    def embed(\n        self,\n        texts: List[str],\n        normalize: bool = True,\n        show_progress: bool = False\n    ) -> np.ndarray:\n        \"\"\"Embed texts with optional normalization.\"\"\"\n        embeddings = self.model.encode(\n            texts,\n            normalize_embeddings=normalize,\n            show_progress_bar=show_progress,\n            convert_to_numpy=True\n        )\n        return embeddings\n\n    def embed_query(self, query: str) -> np.ndarray:\n        \"\"\"Embed a query with BGE-style prefix.\"\"\"\n        # BGE models benefit from query prefix\n        if \"bge\" in self.model.get_sentence_embedding_dimension():\n            query = f\"Represent this sentence for searching relevant passages: {query}\"\n        return self.embed([query])[0]\n\n    def embed_documents(self, documents: List[str]) -> np.ndarray:\n        \"\"\"Embed documents for indexing.\"\"\"\n        return self.embed(documents)\n\n\n# E5 model with instructions\nclass E5Embedder:\n    def __init__(self, model_name: str = \"intfloat\u002Fmultilingual-e5-large\"):\n        self.model = SentenceTransformer(model_name)\n\n    def embed_query(self, query: str) -> np.ndarray:\n        return self.model.encode(f\"query: {query}\")\n\n    def embed_document(self, document: str) -> np.ndarray:\n        return self.model.encode(f\"passage: {document}\")\n```\n\n### Template 3: Chunking Strategies\n\n```python\nfrom typing import List, Tuple\nimport re\n\ndef chunk_by_tokens(\n    text: str,\n    chunk_size: int = 512,\n    chunk_overlap: int = 50,\n    tokenizer=None\n) -> List[str]:\n    \"\"\"Chunk text by token count.\"\"\"\n    import tiktoken\n    tokenizer = tokenizer or tiktoken.get_encoding(\"cl100k_base\")\n\n    tokens = tokenizer.encode(text)\n    chunks = []\n\n    start = 0\n    while start \u003C len(tokens):\n        end = start + chunk_size\n        chunk_tokens = tokens[start:end]\n        chunk_text = tokenizer.decode(chunk_tokens)\n        chunks.append(chunk_text)\n        start = end - chunk_overlap\n\n    return chunks\n\n\ndef chunk_by_sentences(\n    text: str,\n    max_chunk_size: int = 1000,\n    min_chunk_size: int = 100\n) -> List[str]:\n    \"\"\"Chunk text by sentences, respecting size limits.\"\"\"\n    import nltk\n    sentences = nltk.sent_tokenize(text)\n\n    chunks = []\n    current_chunk = []\n    current_size = 0\n\n    for sentence in sentences:\n        sentence_size = len(sentence)\n\n        if current_size + sentence_size > max_chunk_size and current_chunk:\n            chunks.append(\" \".join(current_chunk))\n            current_chunk = []\n            current_size = 0\n\n        current_chunk.append(sentence)\n        current_size += sentence_size\n\n    if current_chunk:\n        chunks.append(\" \".join(current_chunk))\n\n    return chunks\n\n\ndef chunk_by_semantic_sections(\n    text: str,\n    headers_pattern: str = r'^#{1,3}\\s+.+$'\n) -> List[Tuple[str, str]]:\n    \"\"\"Chunk markdown by headers, preserving hierarchy.\"\"\"\n    lines = text.split('\\n')\n    chunks = []\n    current_header = \"\"\n    current_content = []\n\n    for line in lines:\n        if re.match(headers_pattern, line, re.MULTILINE):\n            if current_content:\n                chunks.append((current_header, '\\n'.join(current_content)))\n            current_header = line\n            current_content = []\n        else:\n            current_content.append(line)\n\n    if current_content:\n        chunks.append((current_header, '\\n'.join(current_content)))\n\n    return chunks\n\n\ndef recursive_character_splitter(\n    text: str,\n    chunk_size: int = 1000,\n    chunk_overlap: int = 200,\n    separators: List[str] = None\n) -> List[str]:\n    \"\"\"LangChain-style recursive splitter.\"\"\"\n    separators = separators or [\"\\n\\n\", \"\\n\", \". \", \" \", \"\"]\n\n    def split_text(text: str, separators: List[str]) -> List[str]:\n        if not text:\n            return []\n\n        separator = separators[0]\n        remaining_separators = separators[1:]\n\n        if separator == \"\":\n            # Character-level split\n            return [text[i:i+chunk_size] for i in range(0, len(text), chunk_size - chunk_overlap)]\n\n        splits = text.split(separator)\n        chunks = []\n        current_chunk = []\n        current_length = 0\n\n        for split in splits:\n            split_length = len(split) + len(separator)\n\n            if current_length + split_length > chunk_size and current_chunk:\n                chunk_text = separator.join(current_chunk)\n\n                # Recursively split if still too large\n                if len(chunk_text) > chunk_size and remaining_separators:\n                    chunks.extend(split_text(chunk_text, remaining_separators))\n                else:\n                    chunks.append(chunk_text)\n\n                # Start new chunk with overlap\n                overlap_splits = []\n                overlap_length = 0\n                for s in reversed(current_chunk):\n                    if overlap_length + len(s) \u003C= chunk_overlap:\n                        overlap_splits.insert(0, s)\n                        overlap_length += len(s)\n                    else:\n                        break\n                current_chunk = overlap_splits\n                current_length = overlap_length\n\n            current_chunk.append(split)\n            current_length += split_length\n\n        if current_chunk:\n            chunks.append(separator.join(current_chunk))\n\n        return chunks\n\n    return split_text(text, separators)\n```\n\n### Template 4: Domain-Specific Embedding Pipeline\n\n```python\nclass DomainEmbeddingPipeline:\n    \"\"\"Pipeline for domain-specific embeddings.\"\"\"\n\n    def __init__(\n        self,\n        embedding_model: str = \"text-embedding-3-small\",\n        chunk_size: int = 512,\n        chunk_overlap: int = 50,\n        preprocessing_fn=None\n    ):\n        self.embedding_model = embedding_model\n        self.chunk_size = chunk_size\n        self.chunk_overlap = chunk_overlap\n        self.preprocess = preprocessing_fn or self._default_preprocess\n\n    def _default_preprocess(self, text: str) -> str:\n        \"\"\"Default preprocessing.\"\"\"\n        # Remove excessive whitespace\n        text = re.sub(r'\\s+', ' ', text)\n        # Remove special characters\n        text = re.sub(r'[^\\w\\s.,!?-]', '', text)\n        return text.strip()\n\n    async def process_documents(\n        self,\n        documents: List[dict],\n        id_field: str = \"id\",\n        content_field: str = \"content\",\n        metadata_fields: List[str] = None\n    ) -> List[dict]:\n        \"\"\"Process documents for vector storage.\"\"\"\n        processed = []\n\n        for doc in documents:\n            content = doc[content_field]\n            doc_id = doc[id_field]\n\n            # Preprocess\n            cleaned = self.preprocess(content)\n\n            # Chunk\n            chunks = chunk_by_tokens(\n                cleaned,\n                self.chunk_size,\n                self.chunk_overlap\n            )\n\n            # Create embeddings\n            embeddings = get_embeddings(chunks, self.embedding_model)\n\n            # Create records\n            for i, (chunk, embedding) in enumerate(zip(chunks, embeddings)):\n                record = {\n                    \"id\": f\"{doc_id}_chunk_{i}\",\n                    \"document_id\": doc_id,\n                    \"chunk_index\": i,\n                    \"text\": chunk,\n                    \"embedding\": embedding\n                }\n\n                # Add metadata\n                if metadata_fields:\n                    for field in metadata_fields:\n                        if field in doc:\n                            record[field] = doc[field]\n\n                processed.append(record)\n\n        return processed\n\n\n# Code-specific pipeline\nclass CodeEmbeddingPipeline:\n    \"\"\"Specialized pipeline for code embeddings.\"\"\"\n\n    def __init__(self, model: str = \"voyage-code-2\"):\n        self.model = model\n\n    def chunk_code(self, code: str, language: str) -> List[dict]:\n        \"\"\"Chunk code by functions\u002Fclasses.\"\"\"\n        import tree_sitter\n\n        # Parse with tree-sitter\n        # Extract functions, classes, methods\n        # Return chunks with context\n        pass\n\n    def embed_with_context(self, chunk: str, context: str) -> List[float]:\n        \"\"\"Embed code with surrounding context.\"\"\"\n        combined = f\"Context: {context}\\n\\nCode:\\n{chunk}\"\n        return get_embedding(combined, model=self.model)\n```\n\n### Template 5: Embedding Quality Evaluation\n\n```python\nimport numpy as np\nfrom typing import List, Tuple\n\ndef evaluate_retrieval_quality(\n    queries: List[str],\n    relevant_docs: List[List[str]],  # List of relevant doc IDs per query\n    retrieved_docs: List[List[str]],  # List of retrieved doc IDs per query\n    k: int = 10\n) -> dict:\n    \"\"\"Evaluate embedding quality for retrieval.\"\"\"\n\n    def precision_at_k(relevant: set, retrieved: List[str], k: int) -> float:\n        retrieved_k = retrieved[:k]\n        relevant_retrieved = len(set(retrieved_k) & relevant)\n        return relevant_retrieved \u002F k\n\n    def recall_at_k(relevant: set, retrieved: List[str], k: int) -> float:\n        retrieved_k = retrieved[:k]\n        relevant_retrieved = len(set(retrieved_k) & relevant)\n        return relevant_retrieved \u002F len(relevant) if relevant else 0\n\n    def mrr(relevant: set, retrieved: List[str]) -> float:\n        for i, doc in enumerate(retrieved):\n            if doc in relevant:\n                return 1 \u002F (i + 1)\n        return 0\n\n    def ndcg_at_k(relevant: set, retrieved: List[str], k: int) -> float:\n        dcg = sum(\n            1 \u002F np.log2(i + 2) if doc in relevant else 0\n            for i, doc in enumerate(retrieved[:k])\n        )\n        ideal_dcg = sum(1 \u002F np.log2(i + 2) for i in range(min(len(relevant), k)))\n        return dcg \u002F ideal_dcg if ideal_dcg > 0 else 0\n\n    metrics = {\n        f\"precision@{k}\": [],\n        f\"recall@{k}\": [],\n        \"mrr\": [],\n        f\"ndcg@{k}\": []\n    }\n\n    for relevant, retrieved in zip(relevant_docs, retrieved_docs):\n        relevant_set = set(relevant)\n        metrics[f\"precision@{k}\"].append(precision_at_k(relevant_set, retrieved, k))\n        metrics[f\"recall@{k}\"].append(recall_at_k(relevant_set, retrieved, k))\n        metrics[\"mrr\"].append(mrr(relevant_set, retrieved))\n        metrics[f\"ndcg@{k}\"].append(ndcg_at_k(relevant_set, retrieved, k))\n\n    return {name: np.mean(values) for name, values in metrics.items()}\n\n\ndef compute_embedding_similarity(\n    embeddings1: np.ndarray,\n    embeddings2: np.ndarray,\n    metric: str = \"cosine\"\n) -> np.ndarray:\n    \"\"\"Compute similarity matrix between embedding sets.\"\"\"\n    if metric == \"cosine\":\n        # Normalize\n        norm1 = embeddings1 \u002F np.linalg.norm(embeddings1, axis=1, keepdims=True)\n        norm2 = embeddings2 \u002F np.linalg.norm(embeddings2, axis=1, keepdims=True)\n        return norm1 @ norm2.T\n    elif metric == \"euclidean\":\n        from scipy.spatial.distance import cdist\n        return -cdist(embeddings1, embeddings2, metric='euclidean')\n    elif metric == \"dot\":\n        return embeddings1 @ embeddings2.T\n```\n\n## Best Practices\n\n### Do's\n- **Match model to use case** - Code vs prose vs multilingual\n- **Chunk thoughtfully** - Preserve semantic boundaries\n- **Normalize embeddings** - For cosine similarity\n- **Batch requests** - More efficient than one-by-one\n- **Cache embeddings** - Avoid recomputing\n\n### Don'ts\n- **Don't ignore token limits** - Truncation loses info\n- **Don't mix embedding models** - Incompatible spaces\n- **Don't skip preprocessing** - Garbage in, garbage out\n- **Don't over-chunk** - Lose context\n\n## Resources\n\n- [OpenAI Embeddings](https:\u002F\u002Fplatform.openai.com\u002Fdocs\u002Fguides\u002Fembeddings)\n- [Sentence Transformers](https:\u002F\u002Fwww.sbert.net\u002F)\n- [MTEB Benchmark](https:\u002F\u002Fhuggingface.co\u002Fspaces\u002Fmteb\u002Fleaderboard)\n",{"name":5,"description":7},"main","7850740bd837adb32f0a109ab2af82482196c6f2","skills\u002Fwshobson-agents\u002Fplugins\u002Fllm-application-dev\u002Fskills\u002Fembedding-strategies",1,"https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills","https:\u002F\u002Fzturnsgo.com","https:\u002F\u002Fgithub.com\u002Fshengdabai","https:\u002F\u002Fzturnsgo.com\u002F","AI × education creator — producer, not consumer. Taught 6,000+ learners; shipping learning tools & daily essays with Claude Code, practicing @xiaolai's way 🥋","2026-11-01 21:28:23",{"source":8,"skillName":5,"commands":53,"prompt":63,"zipAvailable":64},[54,57,60],{"tool":55,"command":56},"npx","npx skills add shengdabai\u002FTony-Claude-Code-Skills --skill embedding-strategies",{"tool":58,"command":59},"bunx","bunx skills add shengdabai\u002FTony-Claude-Code-Skills --skill embedding-strategies",{"tool":61,"command":62},"pnpm","pnpm dlx skills add shengdabai\u002FTony-Claude-Code-Skills --skill embedding-strategies","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fwshobson-agents\u002Fplugins\u002Fllm-application-dev\u002Fskills\u002Fembedding-strategies\u002FSKILL.md",true,[],[67,86,120,145,166,186],{"skillNo":68,"name":69,"nameZh":6,"description":70,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":71,"categories":72,"keywords":78,"publishedAt":37},"01M70P17DAD4C1C47NPH6A96EM","xiaolai-methodology","以李笑来(Li Xiaolai)的思考方式、方法论与表达风格,为用户做全方位 1-on-1 教练指导,覆盖创业、赚钱、投资、学习、自学、读书写作、用 AI、以及人生心态与自我管理。当用户提到 \"李笑来\"、\"笑来\"、\"笑来老师\"、\"扮演李笑来\"、\"找笑来聊聊\"、\"笑来怎么看\"、\"xiaolai\"、\"把时间当作朋友\"、\"通往财富自由之路\"、\"定投改变命运\"、\"韭菜的自我修养\"、\"自学是门手艺\"、\"让时间陪你慢慢变富\"、\"七年就是一辈子\"、\"元认知\"、\"活在未来\"、\"复利\u002F长期主义\"、\"践行\"、\"痴迷改进\"、\"个人商业模式\u002F出售时间\"、\"睡后收入\"、\"定投\"、\"财富自由\"、\"操作系统升级\"、\"最少必要知识\"、\"北极星提示词\"、\"用 AI 该怎么学\u002F思考\" 等任一触发词,或当用户带着关于创业\u002F赚钱\u002F投资\u002F学习\u002F职业\u002F重大人生决策的困惑、希望被人用笑来式的「逻辑 + 大白话 + 类比 + 重新定义 + 践行导向」来点醒和指导时,自动进入李笑来教练模式。","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fxiaolai-methodology\u002FSKILL.md",[73],{"categoryNo":74,"categoryName":75,"taxonomyType":76,"taxonomyTypeDesc":77},"01JBSEEDCAT000000000000018","写作与文案","SCENARIO","使用场景",[79,80,81,82,83,84,85],"methodology","xiaolai","xiaolai methodology","朋友","投资","writing","收入",{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":100,"publishedAt":37},"01M70P13NB9YP7ACJR1S861CDS","zread","Produce and consume a wiki-style knowledge base for a code repository via the `zread` CLI and its on-disk output under `.\u002F.zread\u002Fwiki\u002F`. Use this skill whenever the user wants to understand, onboard onto, explore, summarize, map, or get an overview of an unfamiliar codebase; asks for architecture docs, a project wiki, a repo walkthrough, module\u002Fpackage explanations, or \"what does this repo do\"; wants to generate, regenerate, resume, browse, or serve code documentation locally; or mentions zread \u002F zread.ai directly. Also use it proactively before diving into a large unknown repo — if `.\u002F.zread\u002Fwiki\u002Fcurrent` exists, read the generated pages instead of crawling source file-by-file; if it doesn't, consider offering to run `zread generate`. The trigger is the intent (understand a codebase through generated docs), not the literal word \"zread\".","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fzread\u002FSKILL.md",[92,95],{"categoryNo":93,"categoryName":94,"taxonomyType":76,"taxonomyTypeDesc":77},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":96,"categoryName":97,"taxonomyType":98,"taxonomyTypeDesc":99},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[88,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119],"knowledge","wiki","explore","walkthrough","project","get","knowledge base","run","generate","intent","cli","map","documentation","project wiki","disk","architecture","code","unknown","generated",{"skillNo":121,"name":122,"nameZh":6,"description":123,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":124,"categories":125,"keywords":130,"publishedAt":37},"01M70P10S4SWMZQ8BJAFBFDRW3","youtube","Launch the YouTube Creator Agent to auto-generate videos.\nUse when: user types \u002Fyoutube, wants to create YouTube videos,\ngenerate video content, or upload to YouTube.\nKeywords: youtube, video, 视频, 创作视频, 发布视频","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fyoutube\u002FSKILL.md",[126,127],{"categoryNo":93,"categoryName":94,"taxonomyType":76,"taxonomyTypeDesc":77},{"categoryNo":128,"categoryName":129,"taxonomyType":98,"taxonomyTypeDesc":99},"01JBSEEDCAT000000000000036","内容创作者",[122,131,132,133,134,135,136,137,109,138,139,140,28,29,141,142,143,144],"user","content","视频","creator","generate video","videos","launch","video content","create","upload","auto","types","Video","创作",{"skillNo":146,"name":147,"nameZh":6,"description":148,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":149,"categories":150,"keywords":152,"publishedAt":37},"01M70P0YE95D8BACZCGV8BK90K","youtube-script-sync","Sync, update, or optimize Tony's GetNote knowledge base `Youtube视频逐字稿` by turning each parent recording or first-person idea note into an idempotent child note containing a directly recordable YouTube long-video or Shorts script. Use when the user says “更新\u002F同步\u002F优化 Youtube视频逐字稿知识库”, or says “更新\u002F同步\u002F优化该知识库” while this knowledge base is the active context.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fyoutube-script-sync\u002FSKILL.md",[151],{"categoryNo":93,"categoryName":94,"taxonomyType":76,"taxonomyTypeDesc":77},[153,154,122,155,156,143,157,158,159,160,161,162,163,27,107,164,28,29,101,165],"sync","script","youtube script sync","recording","shorts","first","base","person","update","active","child","idempotent","long",{"skillNo":167,"name":168,"nameZh":6,"description":169,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":170,"categories":171,"keywords":176,"publishedAt":37},"01M70P0WMV5XT0EEC0N445PW07","writing-skills","How to write SKILL.md files that trigger reliably and teach effectively. Use when creating, improving, or reviewing Claude Code skills.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fxiaolai\u002Fnlpm\u002Fskills\u002Fnlpm\u002Fwriting-skills\u002FSKILL.md",[172,173,174,175],{"categoryNo":74,"categoryName":75,"taxonomyType":76,"taxonomyTypeDesc":77},{"categoryNo":93,"categoryName":94,"taxonomyType":76,"taxonomyTypeDesc":77},{"categoryNo":96,"categoryName":97,"taxonomyType":98,"taxonomyTypeDesc":99},{"categoryNo":128,"categoryName":129,"taxonomyType":98,"taxonomyTypeDesc":99},[177,84,178,179,180,181,117,28,29,182,183,184,185],"writing skills","how","teach","write skill","write","claude code skills","files","how to","trigger",{"skillNo":187,"name":188,"nameZh":6,"description":189,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":190,"categories":191,"keywords":194,"publishedAt":37},"01M70P0SJFR4YVJQBGMHKGSKXT","writing-rules","How to write .claude\u002Frules\u002F files that Claude actually follows. Use when creating, improving, or reviewing project rules.","https:\u002F\u002Fgithub.com\u002Fshengdabai\u002FTony-Claude-Code-Skills\u002Fblob\u002Fmain\u002Fskills\u002Fxiaolai\u002Fnlpm\u002Fskills\u002Fnlpm\u002Fwriting-rules\u002FSKILL.md",[192,193],{"categoryNo":74,"categoryName":75,"taxonomyType":76,"taxonomyTypeDesc":77},{"categoryNo":128,"categoryName":129,"taxonomyType":98,"taxonomyTypeDesc":99},[195,84,196,105,197,183,28,178,181,29,184],"Rules","writing rules","project rules"]