[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6C6P0H9PW5PD7XGFKY8JSQA":3,"skill-same:01M6C6P0H9PW5PD7XGFKY8JSQA":103},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":41,"linkOnly":42,"markdownContent":43,"markdownBody":44,"frontMatter":45,"refName":46,"commitSha":47,"skillDirectory":48,"forksCount":14,"repositoryHtmlUrl":49,"repositoryHomepageUrl":6,"ownerHtmlUrl":50,"ownerBlogUrl":6,"ownerDescription":51,"updatedAt":52,"install":53,"resources":66},"01M6C6P0H9PW5PD7XGFKY8JSQA","cocoindex",null,"Comprehensive toolkit for developing with the CocoIndex library. Use when users need to create data transformation pipelines (flows), write custom functions, or operate flows via CLI or API. Covers building ETL workflows for AI data processing, including embedding documents into vector databases, building knowledge graphs, creating search indexes, or processing data streams with incremental updates.","swesmith\u002Fdavila7__claude-code-templates.734b8a50","swesmith","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F189315905?v=4","01M6C6MQ65WXKTCSMZA2V1MVHH","SWE-smith","01M6C6MQAR55RCJYERH7BG0DYN",2,"MIT","https:\u002F\u002Fgithub.com\u002Fswesmith\u002Fdavila7__claude-code-templates.734b8a50\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fdevelopment\u002Fcocoindex\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",[5,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40],"data","pipelines","databases","use","write","data processing","knowledge","api","cli","library","vector","search","etl","create","toolkit","documents","workflows","2026-11-10 10:12:30",false,"---\nname: cocoindex\ndescription: Comprehensive toolkit for developing with the CocoIndex library. Use when users need to create data transformation pipelines (flows), write custom functions, or operate flows via CLI or API. Covers building ETL workflows for AI data processing, including embedding documents into vector databases, building knowledge graphs, creating search indexes, or processing data streams with incremental updates.\n---\n\n# CocoIndex\n\n## Overview\n\nCocoIndex is an ultra-performant real-time data transformation framework for AI with incremental processing. This skill enables building **indexing flows** that extract data from sources, apply transformations (chunking, embedding, LLM extraction), and export to targets (vector databases, graph databases, relational databases).\n\n**Core capabilities:**\n\n1. **Write indexing flows** - Define ETL pipelines using Python\n2. **Create custom functions** - Build reusable transformation logic\n3. **Operate flows** - Run and manage flows using CLI or Python API\n\n**Key features:**\n\n- Incremental processing (only processes changed data)\n- Live updates (continuously sync source changes to targets)\n- Built-in functions (text chunking, embeddings, LLM extraction)\n- Multiple data sources (local files, S3, Azure Blob, Google Drive, Postgres)\n- Multiple targets (Postgres+pgvector, Qdrant, LanceDB, Neo4j, Kuzu)\n\n**For detailed documentation:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002F>\n**Search documentation:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fsearch?q=url%20encoded%20keyword>\n\n## When to Use This Skill\n\nUse when users request:\n\n- \"Build a vector search index for my documents\"\n- \"Create an embedding pipeline for code\u002FPDFs\u002Fimages\"\n- \"Extract structured information using LLMs\"\n- \"Build a knowledge graph from documents\"\n- \"Set up live document indexing\"\n- \"Create custom transformation functions\"\n- \"Run\u002Fupdate my CocoIndex flow\"\n\n## Flow Writing Workflow\n\n### Step 1: Understand Requirements\n\nAsk clarifying questions to understand:\n\n**Data source:**\n\n- Where is the data? (local files, S3, database, etc.)\n- What file types? (text, PDF, JSON, images, code, etc.)\n- How often does it change? (one-time, periodic, continuous)\n\n**Transformations:**\n\n- What processing is needed? (chunking, embedding, extraction, etc.)\n- Which embedding model? (SentenceTransformer, OpenAI, custom)\n- Any custom logic? (filtering, parsing, enrichment)\n\n**Target:**\n\n- Where should results go? (Postgres, Qdrant, Neo4j, etc.)\n- What schema? (fields, primary keys, indexes)\n- Vector search needed? (specify similarity metric)\n\n### Step 2: Set Up Dependencies\n\nGuide user to add CocoIndex with appropriate extras to their project based on their needs:\n\n**Required dependency:**\n\n- `cocoindex` - Core functionality, CLI, and most built-in functions\n\n**Optional extras (add as needed):**\n\n- `cocoindex[embeddings]` - For SentenceTransformer embeddings (when using `SentenceTransformerEmbed`)\n- `cocoindex[colpali]` - For ColPali image\u002Fdocument embeddings (when using `ColPaliEmbedImage` or `ColPaliEmbedQuery`)\n- `cocoindex[lancedb]` - For LanceDB target (when exporting to LanceDB)\n- `cocoindex[embeddings,lancedb]` - Multiple extras can be combined\n\n**What's included:**\n\n- Base package: Core functionality, CLI, most built-in functions, Postgres\u002FQdrant\u002FNeo4j\u002FKuzu targets\n- `embeddings` extra: SentenceTransformers library for local embedding models\n- `colpali` extra: ColPali engine for multimodal document\u002Fimage embeddings\n- `lancedb` extra: LanceDB client library for LanceDB vector database support\n\nUsers can install using their preferred package manager (pip, uv, poetry, etc.) or add to `pyproject.toml`.\n\n**For installation details:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fgetting_started\u002Finstallation>\n\n### Step 3: Set Up Environment\n\n**Check existing environment first:**\n\n1. Check if `COCOINDEX_DATABASE_URL` exists in environment variables\n   - If not found, use default: `postgres:\u002F\u002Fcocoindex:cocoindex@localhost\u002Fcocoindex`\n\n2. **For flows requiring LLM APIs** (embeddings, extraction):\n   - Ask user which LLM provider they want to use:\n     - **OpenAI** - Both generation and embeddings\n     - **Anthropic** - Generation only\n     - **Gemini** - Both generation and embeddings\n     - **Voyage** - Embeddings only\n     - **Ollama** - Local models (generation and embeddings)\n   - Check if the corresponding API key exists in environment variables\n   - If not found, **ask user to provide the API key value**\n   - **Never create simplified examples without LLM** - always get the proper API key and use the real LLM functions\n\n**Guide user to create `.env` file:**\n\n```bash\n# Database connection (required - internal storage)\nCOCOINDEX_DATABASE_URL=postgres:\u002F\u002Fcocoindex:cocoindex@localhost\u002Fcocoindex\n\n# LLM API keys (add the ones you need)\nOPENAI_API_KEY=sk-...          # For OpenAI (generation + embeddings)\nANTHROPIC_API_KEY=sk-ant-...   # For Anthropic (generation only)\nGOOGLE_API_KEY=...             # For Gemini (generation + embeddings)\nVOYAGE_API_KEY=pa-...          # For Voyage (embeddings only)\n# Ollama requires no API key (local)\n```\n\n**For more LLM options:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fai\u002Fllm>\n\nCreate basic project structure:\n\n```python\n# main.py\nfrom dotenv import load_dotenv\nimport cocoindex\n\n@cocoindex.flow_def(name=\"FlowName\")\ndef my_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):\n    # Flow definition here\n    pass\n\nif __name__ == \"__main__\":\n    load_dotenv()\n    cocoindex.init()\n    my_flow.update()\n```\n\n### Step 4: Write the Flow\n\nFollow this structure:\n\n```python\n@cocoindex.flow_def(name=\"DescriptiveName\")\ndef flow_name(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):\n    # 1. Import source data\n    data_scope[\"source_name\"] = flow_builder.add_source(\n        cocoindex.sources.SourceType(...)\n    )\n\n    # 2. Create collector(s) for outputs\n    collector = data_scope.add_collector()\n\n    # 3. Transform data (iterate through rows)\n    with data_scope[\"source_name\"].row() as item:\n        # Apply transformations\n        item[\"new_field\"] = item[\"existing_field\"].transform(\n            cocoindex.functions.FunctionName(...)\n        )\n\n        ...\n\n        # Nested iteration (e.g., chunks within documents)\n        with item[\"nested_table\"].row() as nested_item:\n            # More transformations\n            nested_item[\"embedding\"] = nested_item[\"text\"].transform(...)\n\n            # Collect data for export\n            collector.collect(\n                field1=nested_item[\"field1\"],\n                field2=item[\"field2\"],\n                generated_id=cocoindex.GeneratedField.UUID\n            )\n\n    # 4. Export to target\n    collector.export(\n        \"target_name\",\n        cocoindex.targets.TargetType(...),\n        primary_key_fields=[\"field1\"],\n        vector_indexes=[...]  # If needed\n    )\n```\n\n**Key principles:**\n\n- Each source creates a field in the top-level data scope\n- Use `.row()` to iterate through table data\n- **CRITICAL: Always assign transformed data to row fields** - Use `item[\"new_field\"] = item[\"existing_field\"].transform(...)`, NOT local variables like `new_field = item[\"existing_field\"].transform(...)`\n- Transformations create new fields without mutating existing data\n- Collectors gather data from any scope level\n- Export must happen at top level (not within row iterations)\n\n**Common mistakes to avoid:**\n\n❌ **Wrong:** Using local variables for transformations\n\n```python\nwith data_scope[\"files\"].row() as file:\n    summary = file[\"content\"].transform(...)  # ❌ Local variable\n    summaries_collector.collect(filename=file[\"filename\"], summary=summary)\n```\n\n✅ **Correct:** Assigning to row fields\n\n```python\nwith data_scope[\"files\"].row() as file:\n    file[\"summary\"] = file[\"content\"].transform(...)  # ✅ Field assignment\n    summaries_collector.collect(filename=file[\"filename\"], summary=file[\"summary\"])\n```\n\n❌ **Wrong:** Creating unnecessary dataclasses to mirror flow fields\n\n```python\nfrom dataclasses import dataclass\n\n@dataclass\nclass FileSummary:  # ❌ Unnecessary - CocoIndex manages fields automatically\n    filename: str\n    summary: str\n    embedding: list[float]\n\n# This dataclass is never used in the flow!\n```\n\n### Step 5: Design the Flow Solution\n\n**IMPORTANT:** The patterns listed below are common starting points, but **you cannot exhaustively enumerate all possible scenarios**. When user requirements don't match existing patterns:\n\n1. **Combine elements from multiple patterns** - Mix and match sources, transformations, and targets creatively\n2. **Review additional examples** - See \u003Chttps:\u002F\u002Fgithub.com\u002Fcocoindex-io\u002Fcocoindex?tab=readme-ov-file#-examples-and-demo> for diverse real-world use cases (face recognition, multimodal search, product recommendations, patient form extraction, etc.)\n3. **Think from first principles** - Use the core APIs (sources, transforms, collectors, exports) and apply common sense to solve novel problems\n4. **Be creative** - CocoIndex is flexible; unique combinations of components can solve unique problems\n\n**Common starting patterns (use references for detailed examples):**\n\n**For text embedding:** Load `references\u002Fflow_patterns.md` and refer to \"Pattern 1: Simple Text Embedding\"\n\n**For code embedding:** Load `references\u002Fflow_patterns.md` and refer to \"Pattern 2: Code Embedding with Language Detection\"\n\n**For LLM extraction + knowledge graph:** Load `references\u002Fflow_patterns.md` and refer to \"Pattern 3: LLM-based Extraction to Knowledge Graph\"\n\n**For live updates:** Load `references\u002Fflow_patterns.md` and refer to \"Pattern 4: Live Updates with Refresh Interval\"\n\n**For custom functions:** Load `references\u002Fflow_patterns.md` and refer to \"Pattern 5: Custom Transform Function\"\n\n**For reusable query logic:** Load `references\u002Fflow_patterns.md` and refer to \"Pattern 6: Transform Flow for Reusable Logic\"\n\n**For concurrency control:** Load `references\u002Fflow_patterns.md` and refer to \"Pattern 7: Concurrency Control\"\n\n**Example of pattern composition:**\n\nIf a user asks to \"index images from S3, generate captions with a vision API, and store in Qdrant\", combine:\n\n- AmazonS3 source (from S3 examples)\n- Custom function for vision API calls (from custom functions pattern)\n- EmbedText to embed the captions (from embedding patterns)\n- Qdrant target (from target examples)\n\nNo single pattern covers this exact scenario, but the building blocks are composable.\n\n### Step 6: Test and Run\n\nGuide user through testing:\n\n```bash\n# 1. Run with setup\ncocoindex update --setup -f main   # -f force setup without confirmation prompts\n\n\n# 2. Start a server and redirect users to CocoInsight\ncocoindex server -ci main\n# Then open CocoInsight at https:\u002F\u002Fcocoindex.io\u002Fcocoinsight\n\n```\n\n## Data Types\n\nCocoIndex has a type system independent of programming languages. All data types are determined at flow definition time, making schemas clear and predictable.\n\n**IMPORTANT: When to define types:**\n\n- **Custom functions**: Type annotations are **required** for return values (these are the source of truth for type inference)\n- **Flow fields**: Type annotations are **NOT needed** - CocoIndex automatically infers types from sources, functions, and transformations\n- **Dataclasses\u002FPydantic models**: Only create them when they're **actually used** (as function parameters\u002Freturns or ExtractByLlm output_type), NOT to mirror flow field schemas\n\n**Type annotation requirements:**\n\n- **Return values of custom functions**: Must use **specific type annotations** - these are the source of truth for type inference\n- **Arguments of custom functions**: Relaxed - can use `Any`, `dict[str, Any]`, or omit annotations; engine already knows the types\n- **Flow definitions**: No explicit type annotations needed - CocoIndex automatically infers types from sources and functions\n\n**Why specific return types matter:** Custom function return types let CocoIndex infer field types throughout the flow without processing real data. This enables creating proper target schemas (e.g., vector indexes with fixed dimensions).\n\n**Common type categories:**\n\n1. **Primitive types**: `str`, `int`, `float`, `bool`, `bytes`, `datetime.date`, `datetime.datetime`, `uuid.UUID`\n\n2. **Vector types** (embeddings): Specify dimension in return type if you plan to export as vectors to targets, as most targets require a fixed vector dimension\n   - `cocoindex.Vector[cocoindex.Float32, typing.Literal[768]]` - 768-dim float32 vector (recommended)\n   - `list[float]` without dimension also works\n\n3. **Struct types**: Dataclass, NamedTuple, or Pydantic model\n   - Return type: Must use specific class (e.g., `Person`)\n   - Argument: Can use `dict[str, Any]` or `Any`\n\n4. **Table types**:\n   - **KTable** (keyed): `dict[K, V]` where K = key type (primitive or frozen struct), V = Struct type\n   - **LTable** (ordered): `list[R]` where R = Struct type\n   - Arguments: Can use `dict[Any, Any]` or `list[Any]`\n\n5. **Json type**: `cocoindex.Json` for unstructured\u002Fdynamic data\n\n6. **Optional types**: `T | None` for nullable values\n\n**Examples:**\n\n```python\nfrom dataclasses import dataclass\nfrom typing import Literal\nimport cocoindex\n\n@dataclass\nclass Person:\n    name: str\n    age: int\n\n# ✅ Vector with dimension (recommended for vector search)\n@cocoindex.op.function(behavior_version=1)\ndef embed_text(text: str) -> cocoindex.Vector[cocoindex.Float32, Literal[768]]:\n    \"\"\"Generate 768-dim embedding - dimension needed for vector index.\"\"\"\n    # ... embedding logic ...\n    return embedding  # numpy array or list of 768 floats\n\n# ✅ Struct return type, relaxed argument\n@cocoindex.op.function(behavior_version=1)\ndef process_person(person: dict[str, Any]) -> Person:\n    \"\"\"Argument can be dict[str, Any], return must be specific Struct.\"\"\"\n    return Person(name=person[\"name\"], age=person[\"age\"])\n\n# ✅ LTable return type\n@cocoindex.op.function(behavior_version=1)\ndef filter_people(people: list[Any]) -> list[Person]:\n    \"\"\"Return type specifies list of specific Struct.\"\"\"\n    return [p for p in people if p.age >= 18]\n\n# ❌ Wrong: dict[str, str] is not a valid specific CocoIndex type\n# @cocoindex.op.function(...)\n# def bad_example(person: Person) -> dict[str, str]:\n#     return {\"name\": person.name}\n```\n\n**For comprehensive data types documentation:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fcore\u002Fdata_types>\n\n## Custom Functions\n\nWhen users need custom transformation logic, create custom functions.\n\n### Decision: Standalone vs Spec+Executor\n\n**Use standalone function when:**\n\n- Simple transformation\n- No configuration needed\n- No setup\u002Finitialization required\n\n**Use spec+executor when:**\n\n- Needs configuration (model names, API endpoints, parameters)\n- Requires setup (loading models, establishing connections)\n- Complex multi-step processing\n\n### Creating Standalone Functions\n\n```python\n@cocoindex.op.function(behavior_version=1)\ndef my_function(input_arg: str, optional_arg: int | None = None) -> dict:\n    \"\"\"\n    Function description.\n\n    Args:\n        input_arg: Description\n        optional_arg: Optional description\n    \"\"\"\n    # Transformation logic\n    return {\"result\": f\"processed-{input_arg}\"}\n```\n\n**Requirements:**\n\n- Decorator: `@cocoindex.op.function()`\n- Type annotations on all arguments and return value\n- Optional parameters: `cache=True` for expensive ops, `behavior_version` (required with cache)\n\n### Creating Spec+Executor Functions\n\n```python\n# 1. Define configuration spec\nclass MyFunction(cocoindex.op.FunctionSpec):\n    \"\"\"Configuration for MyFunction.\"\"\"\n    model_name: str\n    threshold: float = 0.5\n\n# 2. Define executor\n@cocoindex.op.executor_class(cache=True, behavior_version=1)\nclass MyFunctionExecutor:\n    spec: MyFunction  # Required: link to spec\n    model = None      # Instance variables for state\n\n    def prepare(self) -> None:\n        \"\"\"Optional: run once before execution.\"\"\"\n        # Load model, setup connections, etc.\n        self.model = load_model(self.spec.model_name)\n\n    def __call__(self, text: str) -> dict:\n        \"\"\"Required: execute for each data row.\"\"\"\n        # Use self.spec for configuration\n        # Use self.model for loaded resources\n        result = self.model.process(text)\n        return {\"result\": result}\n```\n\n**When to enable cache:**\n\n- LLM API calls\n- Model inference\n- External API calls\n- Computationally expensive operations\n\n**Important:** Increment `behavior_version` when function logic changes to invalidate cache.\n\nFor detailed examples and patterns, load `references\u002Fcustom_functions.md`.\n\n**For more on custom functions:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fcustom_ops\u002Fcustom_functions>\n\n## Operating Flows\n\n### CLI Operations\n\n**Setup flow (create resources):**\n\n```bash\ncocoindex setup main\n```\n\n**One-time update:**\n\n```bash\ncocoindex update main\n\n# With auto-setup\ncocoindex update --setup main\n\n# Force reset everything before setup and update\ncocoindex update --reset main\n```\n\n**Live update (continuous monitoring):**\n\n```bash\ncocoindex update main.py -L\n\n# Requires refresh_interval on source or source-specific change capture\n```\n\n**Drop flow (remove all resources):**\n\n```bash\ncocoindex drop main.py\n```\n\n**Inspect flow:**\n\n```bash\ncocoindex show main.py:FlowName\n```\n\n**Test without side effects:**\n\n```bash\ncocoindex evaluate main.py:FlowName --output-dir .\u002Ftest_output\n```\n\nFor complete CLI reference, load `references\u002Fcli_operations.md`.\n\n**For CLI documentation:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fcore\u002Fcli>\n\n### API Operations\n\n**Basic setup:**\n\n```python\nfrom dotenv import load_dotenv\nimport cocoindex\n\nload_dotenv()\ncocoindex.init()\n\n@cocoindex.flow_def(name=\"MyFlow\")\ndef my_flow(flow_builder, data_scope):\n    # ... flow definition ...\n    pass\n```\n\n**One-time update:**\n\n```python\nstats = my_flow.update()\nprint(f\"Processed {stats.total_rows} rows\")\n\n# Async\nstats = await my_flow.update_async()\n```\n\n**Live update:**\n\n```python\n# As context manager\nwith cocoindex.FlowLiveUpdater(my_flow) as updater:\n    # Updater runs in background\n    # Your application logic here\n    pass\n\n# Manual control\nupdater = cocoindex.FlowLiveUpdater(\n    my_flow,\n    cocoindex.FlowLiveUpdaterOptions(\n        live_mode=True,\n        print_stats=True\n    )\n)\nupdater.start()\n# ... application logic ...\nupdater.wait()\n```\n\n**Setup\u002Fdrop:**\n\n```python\nmy_flow.setup(report_to_stdout=True)\nmy_flow.drop(report_to_stdout=True)\ncocoindex.setup_all_flows()\ncocoindex.drop_all_flows()\n```\n\n**Query with transform flows:**\n\n```python\n@cocoindex.transform_flow()\ndef text_to_embedding(text: cocoindex.DataSlice[str]) -> cocoindex.DataSlice[list[float]]:\n    return text.transform(\n        cocoindex.functions.SentenceTransformerEmbed(model=\"...\")\n    )\n\n# Use in flow for indexing\ndoc[\"embedding\"] = text_to_embedding(doc[\"content\"])\n\n# Use for querying\nquery_embedding = text_to_embedding.eval(\"search query\")\n```\n\nFor complete API reference and patterns, load `references\u002Fapi_operations.md`.\n\n**For API documentation:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fcore\u002Fflow_methods>\n\n## Built-in Functions\n\n### Text Processing\n\n**SplitRecursively** - Chunk text intelligently\n\n```python\ndoc[\"chunks\"] = doc[\"content\"].transform(\n    cocoindex.functions.SplitRecursively(),\n    language=\"markdown\",  # or \"python\", \"javascript\", etc.\n    chunk_size=2000,\n    chunk_overlap=500\n)\n```\n\n**ParseJson** - Parse JSON strings\n\n```python\ndata = json_string.transform(cocoindex.functions.ParseJson())\n```\n\n**DetectProgrammingLanguage** - Detect language from filename\n\n```python\nfile[\"language\"] = file[\"filename\"].transform(\n    cocoindex.functions.DetectProgrammingLanguage()\n)\n```\n\n### Embeddings\n\n**SentenceTransformerEmbed** - Local embedding model\n\n```python\n# Requires: cocoindex[embeddings]\nchunk[\"embedding\"] = chunk[\"text\"].transform(\n    cocoindex.functions.SentenceTransformerEmbed(\n        model=\"sentence-transformers\u002Fall-MiniLM-L6-v2\"\n    )\n)\n```\n\n**EmbedText** - LLM API embeddings\n\nThis is the **recommended way** to generate embeddings using LLM APIs (OpenAI, Voyage, etc.).\n\n```python\nchunk[\"embedding\"] = chunk[\"text\"].transform(\n    cocoindex.functions.EmbedText(\n        api_type=cocoindex.LlmApiType.OPENAI,\n        model=\"text-embedding-3-small\",\n    )\n)\n```\n\n**ColPaliEmbedImage** - Multimodal image embeddings\n\n```python\n# Requires: cocoindex[colpali]\nimage[\"embedding\"] = image[\"img_bytes\"].transform(\n    cocoindex.functions.ColPaliEmbedImage(model=\"vidore\u002Fcolpali-v1.2\")\n)\n```\n\n### LLM Extraction\n\n**ExtractByLlm** - Extract structured data with LLM\n\nThis is the **recommended way** to use LLMs for extraction and summarization tasks. It supports both structured outputs (dataclasses, Pydantic models) and simple text outputs (str).\n\n```python\nimport dataclasses\n\n# For structured extraction\n@dataclasses.dataclass\nclass ProductInfo:\n    name: str\n    price: float\n    category: str\n\nitem[\"product_info\"] = item[\"text\"].transform(\n    cocoindex.functions.ExtractByLlm(\n        llm_spec=cocoindex.LlmSpec(\n            api_type=cocoindex.LlmApiType.OPENAI,\n            model=\"gpt-4o-mini\"\n        ),\n        output_type=ProductInfo,\n        instruction=\"Extract product information\"\n    )\n)\n\n# For text summarization\u002Fgeneration\nfile[\"summary\"] = file[\"content\"].transform(\n    cocoindex.functions.ExtractByLlm(\n        llm_spec=cocoindex.LlmSpec(\n            api_type=cocoindex.LlmApiType.OPENAI,\n            model=\"gpt-4o-mini\"\n        ),\n        output_type=str,\n        instruction=\"Summarize this document in one paragraph\"\n    )\n)\n```\n\n## Common Sources and Targets\n\n**Browse all sources:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fsources\u002F>\n**Browse all targets:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Ftargets\u002F>\n\n### Sources\n\n**LocalFile:**\n\n```python\ncocoindex.sources.LocalFile(\n    path=\"documents\",\n    included_patterns=[\"*.md\", \"*.txt\"],\n    excluded_patterns=[\"**\u002F.*\", \"node_modules\"]\n)\n```\n\n**AmazonS3:**\n\n```python\ncocoindex.sources.AmazonS3(\n    bucket=\"my-bucket\",\n    prefix=\"documents\u002F\",\n    aws_access_key_id=cocoindex.add_transient_auth_entry(\"...\"),\n    aws_secret_access_key=cocoindex.add_transient_auth_entry(\"...\")\n)\n```\n\n**Postgres:**\n\n```python\ncocoindex.sources.Postgres(\n    connection=cocoindex.add_auth_entry(\"conn\", cocoindex.sources.PostgresConnection(...)),\n    query=\"SELECT id, content FROM documents\"\n)\n```\n\n### Targets\n\n**Postgres (with vector support):**\n\n```python\ncollector.export(\n    \"target_name\",\n    cocoindex.targets.Postgres(),\n    primary_key_fields=[\"id\"],\n    vector_indexes=[\n        cocoindex.VectorIndexDef(\n            field_name=\"embedding\",\n            metric=cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY\n        )\n    ]\n)\n```\n\n**Qdrant:**\n\n```python\ncollector.export(\n    \"target_name\",\n    cocoindex.targets.Qdrant(collection_name=\"my_collection\"),\n    primary_key_fields=[\"id\"]\n)\n```\n\n**LanceDB:**\n\n```python\n# Requires: cocoindex[lancedb]\ncollector.export(\n    \"target_name\",\n    cocoindex.targets.LanceDB(uri=\"lancedb_data\", table_name=\"my_table\"),\n    primary_key_fields=[\"id\"]\n)\n```\n\n**Neo4j (nodes):**\n\n```python\ncollector.export(\n    \"nodes\",\n    cocoindex.targets.Neo4j(\n        connection=neo4j_conn,\n        mapping=cocoindex.targets.Nodes(label=\"Entity\")\n    ),\n    primary_key_fields=[\"id\"]\n)\n```\n\n**Neo4j (relationships):**\n\n```python\ncollector.export(\n    \"relationships\",\n    cocoindex.targets.Neo4j(\n        connection=neo4j_conn,\n        mapping=cocoindex.targets.Relationships(\n            rel_type=\"RELATES_TO\",\n            source=cocoindex.targets.NodeFromFields(\n                label=\"Entity\",\n                fields=[cocoindex.targets.TargetFieldMapping(source=\"source_id\", target=\"id\")]\n            ),\n            target=cocoindex.targets.NodeFromFields(\n                label=\"Entity\",\n                fields=[cocoindex.targets.TargetFieldMapping(source=\"target_id\", target=\"id\")]\n            )\n        )\n    ),\n    primary_key_fields=[\"id\"]\n)\n```\n\n## Common Issues and Solutions\n\n### \"Flow not found\"\n\n- Check APP_TARGET format: `cocoindex show main.py`\n- Use `--app-dir` if not in project root\n- Verify flow name matches decorator\n\n### \"Database connection failed\"\n\n- Check `.env` has `COCOINDEX_DATABASE_URL`\n- Test connection: `psql $COCOINDEX_DATABASE_URL`\n- Use `--env-file` to specify custom location\n\n### \"Schema mismatch\"\n\n- Re-run setup: `cocoindex setup main.py`\n- Drop and recreate: `cocoindex drop main.py && cocoindex setup main.py`\n\n### \"Live update exits immediately\"\n\n- Add `refresh_interval` to source\n- Or use source-specific change capture (Postgres notifications, S3 events)\n\n### \"Out of memory\"\n\n- Add concurrency limits on sources: `max_inflight_rows`, `max_inflight_bytes`\n- Set global limits in `.env`: `COCOINDEX_SOURCE_MAX_INFLIGHT_ROWS`\n\n## Reference Documentation\n\nThis skill includes comprehensive reference documentation for common patterns and operations:\n\n- **references\u002Fflow_patterns.md** - Complete examples of common flow patterns (text embedding, code embedding, knowledge graphs, live updates, concurrency control, etc.)\n- **references\u002Fcustom_functions.md** - Detailed guide for creating custom functions with examples (standalone functions, spec+executor pattern, LLM calls, external APIs, caching)\n- **references\u002Fcli_operations.md** - Complete CLI reference with all commands, options, and workflows\n- **references\u002Fapi_operations.md** - Python API reference with examples for programmatic flow control, live updates, queries, and application integration patterns\n\nLoad these references when users need:\n\n- Detailed examples of specific patterns\n- Complete API documentation\n- Advanced usage scenarios\n- Troubleshooting guidance\n\n**For comprehensive documentation:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002F>\n**Search specific topics:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fsearch?q=url%20encoded%20keyword>\n","# CocoIndex\n\n## Overview\n\nCocoIndex is an ultra-performant real-time data transformation framework for AI with incremental processing. This skill enables building **indexing flows** that extract data from sources, apply transformations (chunking, embedding, LLM extraction), and export to targets (vector databases, graph databases, relational databases).\n\n**Core capabilities:**\n\n1. **Write indexing flows** - Define ETL pipelines using Python\n2. **Create custom functions** - Build reusable transformation logic\n3. **Operate flows** - Run and manage flows using CLI or Python API\n\n**Key features:**\n\n- Incremental processing (only processes changed data)\n- Live updates (continuously sync source changes to targets)\n- Built-in functions (text chunking, embeddings, LLM extraction)\n- Multiple data sources (local files, S3, Azure Blob, Google Drive, Postgres)\n- Multiple targets (Postgres+pgvector, Qdrant, LanceDB, Neo4j, Kuzu)\n\n**For detailed documentation:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002F>\n**Search documentation:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fsearch?q=url%20encoded%20keyword>\n\n## When to Use This Skill\n\nUse when users request:\n\n- \"Build a vector search index for my documents\"\n- \"Create an embedding pipeline for code\u002FPDFs\u002Fimages\"\n- \"Extract structured information using LLMs\"\n- \"Build a knowledge graph from documents\"\n- \"Set up live document indexing\"\n- \"Create custom transformation functions\"\n- \"Run\u002Fupdate my CocoIndex flow\"\n\n## Flow Writing Workflow\n\n### Step 1: Understand Requirements\n\nAsk clarifying questions to understand:\n\n**Data source:**\n\n- Where is the data? (local files, S3, database, etc.)\n- What file types? (text, PDF, JSON, images, code, etc.)\n- How often does it change? (one-time, periodic, continuous)\n\n**Transformations:**\n\n- What processing is needed? (chunking, embedding, extraction, etc.)\n- Which embedding model? (SentenceTransformer, OpenAI, custom)\n- Any custom logic? (filtering, parsing, enrichment)\n\n**Target:**\n\n- Where should results go? (Postgres, Qdrant, Neo4j, etc.)\n- What schema? (fields, primary keys, indexes)\n- Vector search needed? (specify similarity metric)\n\n### Step 2: Set Up Dependencies\n\nGuide user to add CocoIndex with appropriate extras to their project based on their needs:\n\n**Required dependency:**\n\n- `cocoindex` - Core functionality, CLI, and most built-in functions\n\n**Optional extras (add as needed):**\n\n- `cocoindex[embeddings]` - For SentenceTransformer embeddings (when using `SentenceTransformerEmbed`)\n- `cocoindex[colpali]` - For ColPali image\u002Fdocument embeddings (when using `ColPaliEmbedImage` or `ColPaliEmbedQuery`)\n- `cocoindex[lancedb]` - For LanceDB target (when exporting to LanceDB)\n- `cocoindex[embeddings,lancedb]` - Multiple extras can be combined\n\n**What's included:**\n\n- Base package: Core functionality, CLI, most built-in functions, Postgres\u002FQdrant\u002FNeo4j\u002FKuzu targets\n- `embeddings` extra: SentenceTransformers library for local embedding models\n- `colpali` extra: ColPali engine for multimodal document\u002Fimage embeddings\n- `lancedb` extra: LanceDB client library for LanceDB vector database support\n\nUsers can install using their preferred package manager (pip, uv, poetry, etc.) or add to `pyproject.toml`.\n\n**For installation details:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fgetting_started\u002Finstallation>\n\n### Step 3: Set Up Environment\n\n**Check existing environment first:**\n\n1. Check if `COCOINDEX_DATABASE_URL` exists in environment variables\n   - If not found, use default: `postgres:\u002F\u002Fcocoindex:cocoindex@localhost\u002Fcocoindex`\n\n2. **For flows requiring LLM APIs** (embeddings, extraction):\n   - Ask user which LLM provider they want to use:\n     - **OpenAI** - Both generation and embeddings\n     - **Anthropic** - Generation only\n     - **Gemini** - Both generation and embeddings\n     - **Voyage** - Embeddings only\n     - **Ollama** - Local models (generation and embeddings)\n   - Check if the corresponding API key exists in environment variables\n   - If not found, **ask user to provide the API key value**\n   - **Never create simplified examples without LLM** - always get the proper API key and use the real LLM functions\n\n**Guide user to create `.env` file:**\n\n```bash\n# Database connection (required - internal storage)\nCOCOINDEX_DATABASE_URL=postgres:\u002F\u002Fcocoindex:cocoindex@localhost\u002Fcocoindex\n\n# LLM API keys (add the ones you need)\nOPENAI_API_KEY=sk-...          # For OpenAI (generation + embeddings)\nANTHROPIC_API_KEY=sk-ant-...   # For Anthropic (generation only)\nGOOGLE_API_KEY=...             # For Gemini (generation + embeddings)\nVOYAGE_API_KEY=pa-...          # For Voyage (embeddings only)\n# Ollama requires no API key (local)\n```\n\n**For more LLM options:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fai\u002Fllm>\n\nCreate basic project structure:\n\n```python\n# main.py\nfrom dotenv import load_dotenv\nimport cocoindex\n\n@cocoindex.flow_def(name=\"FlowName\")\ndef my_flow(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):\n    # Flow definition here\n    pass\n\nif __name__ == \"__main__\":\n    load_dotenv()\n    cocoindex.init()\n    my_flow.update()\n```\n\n### Step 4: Write the Flow\n\nFollow this structure:\n\n```python\n@cocoindex.flow_def(name=\"DescriptiveName\")\ndef flow_name(flow_builder: cocoindex.FlowBuilder, data_scope: cocoindex.DataScope):\n    # 1. Import source data\n    data_scope[\"source_name\"] = flow_builder.add_source(\n        cocoindex.sources.SourceType(...)\n    )\n\n    # 2. Create collector(s) for outputs\n    collector = data_scope.add_collector()\n\n    # 3. Transform data (iterate through rows)\n    with data_scope[\"source_name\"].row() as item:\n        # Apply transformations\n        item[\"new_field\"] = item[\"existing_field\"].transform(\n            cocoindex.functions.FunctionName(...)\n        )\n\n        ...\n\n        # Nested iteration (e.g., chunks within documents)\n        with item[\"nested_table\"].row() as nested_item:\n            # More transformations\n            nested_item[\"embedding\"] = nested_item[\"text\"].transform(...)\n\n            # Collect data for export\n            collector.collect(\n                field1=nested_item[\"field1\"],\n                field2=item[\"field2\"],\n                generated_id=cocoindex.GeneratedField.UUID\n            )\n\n    # 4. Export to target\n    collector.export(\n        \"target_name\",\n        cocoindex.targets.TargetType(...),\n        primary_key_fields=[\"field1\"],\n        vector_indexes=[...]  # If needed\n    )\n```\n\n**Key principles:**\n\n- Each source creates a field in the top-level data scope\n- Use `.row()` to iterate through table data\n- **CRITICAL: Always assign transformed data to row fields** - Use `item[\"new_field\"] = item[\"existing_field\"].transform(...)`, NOT local variables like `new_field = item[\"existing_field\"].transform(...)`\n- Transformations create new fields without mutating existing data\n- Collectors gather data from any scope level\n- Export must happen at top level (not within row iterations)\n\n**Common mistakes to avoid:**\n\n❌ **Wrong:** Using local variables for transformations\n\n```python\nwith data_scope[\"files\"].row() as file:\n    summary = file[\"content\"].transform(...)  # ❌ Local variable\n    summaries_collector.collect(filename=file[\"filename\"], summary=summary)\n```\n\n✅ **Correct:** Assigning to row fields\n\n```python\nwith data_scope[\"files\"].row() as file:\n    file[\"summary\"] = file[\"content\"].transform(...)  # ✅ Field assignment\n    summaries_collector.collect(filename=file[\"filename\"], summary=file[\"summary\"])\n```\n\n❌ **Wrong:** Creating unnecessary dataclasses to mirror flow fields\n\n```python\nfrom dataclasses import dataclass\n\n@dataclass\nclass FileSummary:  # ❌ Unnecessary - CocoIndex manages fields automatically\n    filename: str\n    summary: str\n    embedding: list[float]\n\n# This dataclass is never used in the flow!\n```\n\n### Step 5: Design the Flow Solution\n\n**IMPORTANT:** The patterns listed below are common starting points, but **you cannot exhaustively enumerate all possible scenarios**. When user requirements don't match existing patterns:\n\n1. **Combine elements from multiple patterns** - Mix and match sources, transformations, and targets creatively\n2. **Review additional examples** - See \u003Chttps:\u002F\u002Fgithub.com\u002Fcocoindex-io\u002Fcocoindex?tab=readme-ov-file#-examples-and-demo> for diverse real-world use cases (face recognition, multimodal search, product recommendations, patient form extraction, etc.)\n3. **Think from first principles** - Use the core APIs (sources, transforms, collectors, exports) and apply common sense to solve novel problems\n4. **Be creative** - CocoIndex is flexible; unique combinations of components can solve unique problems\n\n**Common starting patterns (use references for detailed examples):**\n\n**For text embedding:** Load `references\u002Fflow_patterns.md` and refer to \"Pattern 1: Simple Text Embedding\"\n\n**For code embedding:** Load `references\u002Fflow_patterns.md` and refer to \"Pattern 2: Code Embedding with Language Detection\"\n\n**For LLM extraction + knowledge graph:** Load `references\u002Fflow_patterns.md` and refer to \"Pattern 3: LLM-based Extraction to Knowledge Graph\"\n\n**For live updates:** Load `references\u002Fflow_patterns.md` and refer to \"Pattern 4: Live Updates with Refresh Interval\"\n\n**For custom functions:** Load `references\u002Fflow_patterns.md` and refer to \"Pattern 5: Custom Transform Function\"\n\n**For reusable query logic:** Load `references\u002Fflow_patterns.md` and refer to \"Pattern 6: Transform Flow for Reusable Logic\"\n\n**For concurrency control:** Load `references\u002Fflow_patterns.md` and refer to \"Pattern 7: Concurrency Control\"\n\n**Example of pattern composition:**\n\nIf a user asks to \"index images from S3, generate captions with a vision API, and store in Qdrant\", combine:\n\n- AmazonS3 source (from S3 examples)\n- Custom function for vision API calls (from custom functions pattern)\n- EmbedText to embed the captions (from embedding patterns)\n- Qdrant target (from target examples)\n\nNo single pattern covers this exact scenario, but the building blocks are composable.\n\n### Step 6: Test and Run\n\nGuide user through testing:\n\n```bash\n# 1. Run with setup\ncocoindex update --setup -f main   # -f force setup without confirmation prompts\n\n\n# 2. Start a server and redirect users to CocoInsight\ncocoindex server -ci main\n# Then open CocoInsight at https:\u002F\u002Fcocoindex.io\u002Fcocoinsight\n\n```\n\n## Data Types\n\nCocoIndex has a type system independent of programming languages. All data types are determined at flow definition time, making schemas clear and predictable.\n\n**IMPORTANT: When to define types:**\n\n- **Custom functions**: Type annotations are **required** for return values (these are the source of truth for type inference)\n- **Flow fields**: Type annotations are **NOT needed** - CocoIndex automatically infers types from sources, functions, and transformations\n- **Dataclasses\u002FPydantic models**: Only create them when they're **actually used** (as function parameters\u002Freturns or ExtractByLlm output_type), NOT to mirror flow field schemas\n\n**Type annotation requirements:**\n\n- **Return values of custom functions**: Must use **specific type annotations** - these are the source of truth for type inference\n- **Arguments of custom functions**: Relaxed - can use `Any`, `dict[str, Any]`, or omit annotations; engine already knows the types\n- **Flow definitions**: No explicit type annotations needed - CocoIndex automatically infers types from sources and functions\n\n**Why specific return types matter:** Custom function return types let CocoIndex infer field types throughout the flow without processing real data. This enables creating proper target schemas (e.g., vector indexes with fixed dimensions).\n\n**Common type categories:**\n\n1. **Primitive types**: `str`, `int`, `float`, `bool`, `bytes`, `datetime.date`, `datetime.datetime`, `uuid.UUID`\n\n2. **Vector types** (embeddings): Specify dimension in return type if you plan to export as vectors to targets, as most targets require a fixed vector dimension\n   - `cocoindex.Vector[cocoindex.Float32, typing.Literal[768]]` - 768-dim float32 vector (recommended)\n   - `list[float]` without dimension also works\n\n3. **Struct types**: Dataclass, NamedTuple, or Pydantic model\n   - Return type: Must use specific class (e.g., `Person`)\n   - Argument: Can use `dict[str, Any]` or `Any`\n\n4. **Table types**:\n   - **KTable** (keyed): `dict[K, V]` where K = key type (primitive or frozen struct), V = Struct type\n   - **LTable** (ordered): `list[R]` where R = Struct type\n   - Arguments: Can use `dict[Any, Any]` or `list[Any]`\n\n5. **Json type**: `cocoindex.Json` for unstructured\u002Fdynamic data\n\n6. **Optional types**: `T | None` for nullable values\n\n**Examples:**\n\n```python\nfrom dataclasses import dataclass\nfrom typing import Literal\nimport cocoindex\n\n@dataclass\nclass Person:\n    name: str\n    age: int\n\n# ✅ Vector with dimension (recommended for vector search)\n@cocoindex.op.function(behavior_version=1)\ndef embed_text(text: str) -> cocoindex.Vector[cocoindex.Float32, Literal[768]]:\n    \"\"\"Generate 768-dim embedding - dimension needed for vector index.\"\"\"\n    # ... embedding logic ...\n    return embedding  # numpy array or list of 768 floats\n\n# ✅ Struct return type, relaxed argument\n@cocoindex.op.function(behavior_version=1)\ndef process_person(person: dict[str, Any]) -> Person:\n    \"\"\"Argument can be dict[str, Any], return must be specific Struct.\"\"\"\n    return Person(name=person[\"name\"], age=person[\"age\"])\n\n# ✅ LTable return type\n@cocoindex.op.function(behavior_version=1)\ndef filter_people(people: list[Any]) -> list[Person]:\n    \"\"\"Return type specifies list of specific Struct.\"\"\"\n    return [p for p in people if p.age >= 18]\n\n# ❌ Wrong: dict[str, str] is not a valid specific CocoIndex type\n# @cocoindex.op.function(...)\n# def bad_example(person: Person) -> dict[str, str]:\n#     return {\"name\": person.name}\n```\n\n**For comprehensive data types documentation:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fcore\u002Fdata_types>\n\n## Custom Functions\n\nWhen users need custom transformation logic, create custom functions.\n\n### Decision: Standalone vs Spec+Executor\n\n**Use standalone function when:**\n\n- Simple transformation\n- No configuration needed\n- No setup\u002Finitialization required\n\n**Use spec+executor when:**\n\n- Needs configuration (model names, API endpoints, parameters)\n- Requires setup (loading models, establishing connections)\n- Complex multi-step processing\n\n### Creating Standalone Functions\n\n```python\n@cocoindex.op.function(behavior_version=1)\ndef my_function(input_arg: str, optional_arg: int | None = None) -> dict:\n    \"\"\"\n    Function description.\n\n    Args:\n        input_arg: Description\n        optional_arg: Optional description\n    \"\"\"\n    # Transformation logic\n    return {\"result\": f\"processed-{input_arg}\"}\n```\n\n**Requirements:**\n\n- Decorator: `@cocoindex.op.function()`\n- Type annotations on all arguments and return value\n- Optional parameters: `cache=True` for expensive ops, `behavior_version` (required with cache)\n\n### Creating Spec+Executor Functions\n\n```python\n# 1. Define configuration spec\nclass MyFunction(cocoindex.op.FunctionSpec):\n    \"\"\"Configuration for MyFunction.\"\"\"\n    model_name: str\n    threshold: float = 0.5\n\n# 2. Define executor\n@cocoindex.op.executor_class(cache=True, behavior_version=1)\nclass MyFunctionExecutor:\n    spec: MyFunction  # Required: link to spec\n    model = None      # Instance variables for state\n\n    def prepare(self) -> None:\n        \"\"\"Optional: run once before execution.\"\"\"\n        # Load model, setup connections, etc.\n        self.model = load_model(self.spec.model_name)\n\n    def __call__(self, text: str) -> dict:\n        \"\"\"Required: execute for each data row.\"\"\"\n        # Use self.spec for configuration\n        # Use self.model for loaded resources\n        result = self.model.process(text)\n        return {\"result\": result}\n```\n\n**When to enable cache:**\n\n- LLM API calls\n- Model inference\n- External API calls\n- Computationally expensive operations\n\n**Important:** Increment `behavior_version` when function logic changes to invalidate cache.\n\nFor detailed examples and patterns, load `references\u002Fcustom_functions.md`.\n\n**For more on custom functions:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fcustom_ops\u002Fcustom_functions>\n\n## Operating Flows\n\n### CLI Operations\n\n**Setup flow (create resources):**\n\n```bash\ncocoindex setup main\n```\n\n**One-time update:**\n\n```bash\ncocoindex update main\n\n# With auto-setup\ncocoindex update --setup main\n\n# Force reset everything before setup and update\ncocoindex update --reset main\n```\n\n**Live update (continuous monitoring):**\n\n```bash\ncocoindex update main.py -L\n\n# Requires refresh_interval on source or source-specific change capture\n```\n\n**Drop flow (remove all resources):**\n\n```bash\ncocoindex drop main.py\n```\n\n**Inspect flow:**\n\n```bash\ncocoindex show main.py:FlowName\n```\n\n**Test without side effects:**\n\n```bash\ncocoindex evaluate main.py:FlowName --output-dir .\u002Ftest_output\n```\n\nFor complete CLI reference, load `references\u002Fcli_operations.md`.\n\n**For CLI documentation:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fcore\u002Fcli>\n\n### API Operations\n\n**Basic setup:**\n\n```python\nfrom dotenv import load_dotenv\nimport cocoindex\n\nload_dotenv()\ncocoindex.init()\n\n@cocoindex.flow_def(name=\"MyFlow\")\ndef my_flow(flow_builder, data_scope):\n    # ... flow definition ...\n    pass\n```\n\n**One-time update:**\n\n```python\nstats = my_flow.update()\nprint(f\"Processed {stats.total_rows} rows\")\n\n# Async\nstats = await my_flow.update_async()\n```\n\n**Live update:**\n\n```python\n# As context manager\nwith cocoindex.FlowLiveUpdater(my_flow) as updater:\n    # Updater runs in background\n    # Your application logic here\n    pass\n\n# Manual control\nupdater = cocoindex.FlowLiveUpdater(\n    my_flow,\n    cocoindex.FlowLiveUpdaterOptions(\n        live_mode=True,\n        print_stats=True\n    )\n)\nupdater.start()\n# ... application logic ...\nupdater.wait()\n```\n\n**Setup\u002Fdrop:**\n\n```python\nmy_flow.setup(report_to_stdout=True)\nmy_flow.drop(report_to_stdout=True)\ncocoindex.setup_all_flows()\ncocoindex.drop_all_flows()\n```\n\n**Query with transform flows:**\n\n```python\n@cocoindex.transform_flow()\ndef text_to_embedding(text: cocoindex.DataSlice[str]) -> cocoindex.DataSlice[list[float]]:\n    return text.transform(\n        cocoindex.functions.SentenceTransformerEmbed(model=\"...\")\n    )\n\n# Use in flow for indexing\ndoc[\"embedding\"] = text_to_embedding(doc[\"content\"])\n\n# Use for querying\nquery_embedding = text_to_embedding.eval(\"search query\")\n```\n\nFor complete API reference and patterns, load `references\u002Fapi_operations.md`.\n\n**For API documentation:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fcore\u002Fflow_methods>\n\n## Built-in Functions\n\n### Text Processing\n\n**SplitRecursively** - Chunk text intelligently\n\n```python\ndoc[\"chunks\"] = doc[\"content\"].transform(\n    cocoindex.functions.SplitRecursively(),\n    language=\"markdown\",  # or \"python\", \"javascript\", etc.\n    chunk_size=2000,\n    chunk_overlap=500\n)\n```\n\n**ParseJson** - Parse JSON strings\n\n```python\ndata = json_string.transform(cocoindex.functions.ParseJson())\n```\n\n**DetectProgrammingLanguage** - Detect language from filename\n\n```python\nfile[\"language\"] = file[\"filename\"].transform(\n    cocoindex.functions.DetectProgrammingLanguage()\n)\n```\n\n### Embeddings\n\n**SentenceTransformerEmbed** - Local embedding model\n\n```python\n# Requires: cocoindex[embeddings]\nchunk[\"embedding\"] = chunk[\"text\"].transform(\n    cocoindex.functions.SentenceTransformerEmbed(\n        model=\"sentence-transformers\u002Fall-MiniLM-L6-v2\"\n    )\n)\n```\n\n**EmbedText** - LLM API embeddings\n\nThis is the **recommended way** to generate embeddings using LLM APIs (OpenAI, Voyage, etc.).\n\n```python\nchunk[\"embedding\"] = chunk[\"text\"].transform(\n    cocoindex.functions.EmbedText(\n        api_type=cocoindex.LlmApiType.OPENAI,\n        model=\"text-embedding-3-small\",\n    )\n)\n```\n\n**ColPaliEmbedImage** - Multimodal image embeddings\n\n```python\n# Requires: cocoindex[colpali]\nimage[\"embedding\"] = image[\"img_bytes\"].transform(\n    cocoindex.functions.ColPaliEmbedImage(model=\"vidore\u002Fcolpali-v1.2\")\n)\n```\n\n### LLM Extraction\n\n**ExtractByLlm** - Extract structured data with LLM\n\nThis is the **recommended way** to use LLMs for extraction and summarization tasks. It supports both structured outputs (dataclasses, Pydantic models) and simple text outputs (str).\n\n```python\nimport dataclasses\n\n# For structured extraction\n@dataclasses.dataclass\nclass ProductInfo:\n    name: str\n    price: float\n    category: str\n\nitem[\"product_info\"] = item[\"text\"].transform(\n    cocoindex.functions.ExtractByLlm(\n        llm_spec=cocoindex.LlmSpec(\n            api_type=cocoindex.LlmApiType.OPENAI,\n            model=\"gpt-4o-mini\"\n        ),\n        output_type=ProductInfo,\n        instruction=\"Extract product information\"\n    )\n)\n\n# For text summarization\u002Fgeneration\nfile[\"summary\"] = file[\"content\"].transform(\n    cocoindex.functions.ExtractByLlm(\n        llm_spec=cocoindex.LlmSpec(\n            api_type=cocoindex.LlmApiType.OPENAI,\n            model=\"gpt-4o-mini\"\n        ),\n        output_type=str,\n        instruction=\"Summarize this document in one paragraph\"\n    )\n)\n```\n\n## Common Sources and Targets\n\n**Browse all sources:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fsources\u002F>\n**Browse all targets:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Ftargets\u002F>\n\n### Sources\n\n**LocalFile:**\n\n```python\ncocoindex.sources.LocalFile(\n    path=\"documents\",\n    included_patterns=[\"*.md\", \"*.txt\"],\n    excluded_patterns=[\"**\u002F.*\", \"node_modules\"]\n)\n```\n\n**AmazonS3:**\n\n```python\ncocoindex.sources.AmazonS3(\n    bucket=\"my-bucket\",\n    prefix=\"documents\u002F\",\n    aws_access_key_id=cocoindex.add_transient_auth_entry(\"...\"),\n    aws_secret_access_key=cocoindex.add_transient_auth_entry(\"...\")\n)\n```\n\n**Postgres:**\n\n```python\ncocoindex.sources.Postgres(\n    connection=cocoindex.add_auth_entry(\"conn\", cocoindex.sources.PostgresConnection(...)),\n    query=\"SELECT id, content FROM documents\"\n)\n```\n\n### Targets\n\n**Postgres (with vector support):**\n\n```python\ncollector.export(\n    \"target_name\",\n    cocoindex.targets.Postgres(),\n    primary_key_fields=[\"id\"],\n    vector_indexes=[\n        cocoindex.VectorIndexDef(\n            field_name=\"embedding\",\n            metric=cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY\n        )\n    ]\n)\n```\n\n**Qdrant:**\n\n```python\ncollector.export(\n    \"target_name\",\n    cocoindex.targets.Qdrant(collection_name=\"my_collection\"),\n    primary_key_fields=[\"id\"]\n)\n```\n\n**LanceDB:**\n\n```python\n# Requires: cocoindex[lancedb]\ncollector.export(\n    \"target_name\",\n    cocoindex.targets.LanceDB(uri=\"lancedb_data\", table_name=\"my_table\"),\n    primary_key_fields=[\"id\"]\n)\n```\n\n**Neo4j (nodes):**\n\n```python\ncollector.export(\n    \"nodes\",\n    cocoindex.targets.Neo4j(\n        connection=neo4j_conn,\n        mapping=cocoindex.targets.Nodes(label=\"Entity\")\n    ),\n    primary_key_fields=[\"id\"]\n)\n```\n\n**Neo4j (relationships):**\n\n```python\ncollector.export(\n    \"relationships\",\n    cocoindex.targets.Neo4j(\n        connection=neo4j_conn,\n        mapping=cocoindex.targets.Relationships(\n            rel_type=\"RELATES_TO\",\n            source=cocoindex.targets.NodeFromFields(\n                label=\"Entity\",\n                fields=[cocoindex.targets.TargetFieldMapping(source=\"source_id\", target=\"id\")]\n            ),\n            target=cocoindex.targets.NodeFromFields(\n                label=\"Entity\",\n                fields=[cocoindex.targets.TargetFieldMapping(source=\"target_id\", target=\"id\")]\n            )\n        )\n    ),\n    primary_key_fields=[\"id\"]\n)\n```\n\n## Common Issues and Solutions\n\n### \"Flow not found\"\n\n- Check APP_TARGET format: `cocoindex show main.py`\n- Use `--app-dir` if not in project root\n- Verify flow name matches decorator\n\n### \"Database connection failed\"\n\n- Check `.env` has `COCOINDEX_DATABASE_URL`\n- Test connection: `psql $COCOINDEX_DATABASE_URL`\n- Use `--env-file` to specify custom location\n\n### \"Schema mismatch\"\n\n- Re-run setup: `cocoindex setup main.py`\n- Drop and recreate: `cocoindex drop main.py && cocoindex setup main.py`\n\n### \"Live update exits immediately\"\n\n- Add `refresh_interval` to source\n- Or use source-specific change capture (Postgres notifications, S3 events)\n\n### \"Out of memory\"\n\n- Add concurrency limits on sources: `max_inflight_rows`, `max_inflight_bytes`\n- Set global limits in `.env`: `COCOINDEX_SOURCE_MAX_INFLIGHT_ROWS`\n\n## Reference Documentation\n\nThis skill includes comprehensive reference documentation for common patterns and operations:\n\n- **references\u002Fflow_patterns.md** - Complete examples of common flow patterns (text embedding, code embedding, knowledge graphs, live updates, concurrency control, etc.)\n- **references\u002Fcustom_functions.md** - Detailed guide for creating custom functions with examples (standalone functions, spec+executor pattern, LLM calls, external APIs, caching)\n- **references\u002Fcli_operations.md** - Complete CLI reference with all commands, options, and workflows\n- **references\u002Fapi_operations.md** - Python API reference with examples for programmatic flow control, live updates, queries, and application integration patterns\n\nLoad these references when users need:\n\n- Detailed examples of specific patterns\n- Complete API documentation\n- Advanced usage scenarios\n- Troubleshooting guidance\n\n**For comprehensive documentation:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002F>\n**Search specific topics:** \u003Chttps:\u002F\u002Fcocoindex.io\u002Fdocs\u002Fsearch?q=url%20encoded%20keyword>\n",{"name":5,"description":7},"main","4c3286f35fe531acca1cd0f79fb6175473425b3b","cli-tool\u002Fcomponents\u002Fskills\u002Fdevelopment\u002Fcocoindex","https:\u002F\u002Fgithub.com\u002Fswesmith\u002Fdavila7__claude-code-templates.734b8a50","https:\u002F\u002Fgithub.com\u002Fswesmith","Scaling Data for Software Engineering Agents","2025-12-20 08:10:01",{"source":8,"skillName":5,"commands":54,"prompt":64,"zipAvailable":65},[55,58,61],{"tool":56,"command":57},"npx","npx skills add swesmith\u002Fdavila7__claude-code-templates.734b8a50 --skill cocoindex",{"tool":59,"command":60},"bunx","bunx skills add swesmith\u002Fdavila7__claude-code-templates.734b8a50 --skill cocoindex",{"tool":62,"command":63},"pnpm","pnpm dlx skills add swesmith\u002Fdavila7__claude-code-templates.734b8a50 --skill cocoindex","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fswesmith\u002Fdavila7__claude-code-templates.734b8a50\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fdevelopment\u002Fcocoindex\u002FSKILL.md",true,[67,73,77,81,85,89,95,99],{"path":68,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":71,"sourceHtmlUrl":72},"README.md","REFERENCE","参考文档",1111,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fai-crypto\u002Fxspoonai-official-gas-oracle\u002FREADME.md",{"path":74,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":75,"sourceHtmlUrl":76},"references\u002Fapi_operations.md",13488,"https:\u002F\u002Fgithub.com\u002Fswesmith\u002Fdavila7__claude-code-templates.734b8a50\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fdevelopment\u002Fcocoindex\u002Freferences\u002Fapi_operations.md",{"path":78,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":79,"sourceHtmlUrl":80},"references\u002Fcli_operations.md",8897,"https:\u002F\u002Fgithub.com\u002Fswesmith\u002Fdavila7__claude-code-templates.734b8a50\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fdevelopment\u002Fcocoindex\u002Freferences\u002Fcli_operations.md",{"path":82,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Fcustom_functions.md",13690,"https:\u002F\u002Fgithub.com\u002Fswesmith\u002Fdavila7__claude-code-templates.734b8a50\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fdevelopment\u002Fcocoindex\u002Freferences\u002Fcustom_functions.md",{"path":86,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":87,"sourceHtmlUrl":88},"references\u002Fflow_patterns.md",13857,"https:\u002F\u002Fgithub.com\u002Fswesmith\u002Fdavila7__claude-code-templates.734b8a50\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fdevelopment\u002Fcocoindex\u002Freferences\u002Fflow_patterns.md",{"path":90,"resourceType":91,"resourceTypeDesc":92,"sizeBytes":93,"sourceHtmlUrl":94},"scripts\u002Fgas_tracker.py","SCRIPT","脚本",4139,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fai-crypto\u002Fxspoonai-official-gas-oracle\u002Fscripts\u002Fgas_tracker.py",{"path":96,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":97,"sourceHtmlUrl":98},"SOURCE.md",369,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fai-crypto\u002Fxspoonai-official-gas-oracle\u002FSOURCE.md",{"path":100,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":101,"sourceHtmlUrl":102},"TRUST.auto.yaml",2211,"https:\u002F\u002Fgithub.com\u002Fjiayaoqijia\u002Fcryptoskill\u002Fblob\u002Fmain\u002Fskills\u002Fai-crypto\u002Fxspoonai-official-gas-oracle\u002FTRUST.auto.yaml",[104,116,127,138,157,170],{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":108,"categories":109,"keywords":113,"publishedAt":41},"01M6C6RMEKTWGQ7K4RBMW457T5","template-skill","Replace with description of the skill and when Claude should use it.","https:\u002F\u002Fgithub.com\u002Fswesmith\u002Fdavila7__claude-code-templates.734b8a50\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Futilities\u002Ftemplate-skill\u002FSKILL.md",[110],{"categoryNo":111,"categoryName":112,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发",[114,115,27],"template","template skill",{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":41},"01M6C6RFC6VDGR5WJ35S3MP8PC","domain-name-brainstormer","Generates creative domain name ideas for your project and checks availability across multiple TLDs (.com, .io, .dev, .ai, etc.). Saves hours of brainstorming and manual checking.","https:\u002F\u002Fgithub.com\u002Fswesmith\u002Fdavila7__claude-code-templates.734b8a50\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Futilities\u002Fdomain-name-brainstormer\u002FSKILL.md",[],[123,124,125,126],"domain name brainstormer","brainstorming","dev","creative",{"skillNo":128,"name":129,"nameZh":6,"description":130,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":131,"categories":132,"keywords":133,"publishedAt":41},"01M6C6RCTQSBTTY796KBKKYPRD","raffle-winner-picker","Picks random winners from lists, spreadsheets, or Google Sheets for giveaways, raffles, and contests. Ensures fair, unbiased selection with transparency.","https:\u002F\u002Fgithub.com\u002Fswesmith\u002Fdavila7__claude-code-templates.734b8a50\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Fraffle-winner-picker\u002FSKILL.md",[],[134,135,136,137],"raffle winner picker","google","spreadsheets","google sheets",{"skillNo":139,"name":140,"nameZh":6,"description":141,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":142,"categories":143,"keywords":149,"publishedAt":41},"01M6C6R9VMFAT9Q481MRSTHH18","meeting-insights-analyzer","Analyzes meeting transcripts and recordings to uncover behavioral patterns, communication insights, and actionable feedback. Identifies when you avoid conflict, use filler words, dominate conversations, or miss opportunities to listen. Perfect for professionals seeking to improve their communication and leadership skills.","https:\u002F\u002Fgithub.com\u002Fswesmith\u002Fdavila7__claude-code-templates.734b8a50\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Fmeeting-insights-analyzer\u002FSKILL.md",[144],{"categoryNo":145,"categoryName":146,"taxonomyType":147,"taxonomyTypeDesc":148},"01JBSEEDCAT000000000000034","行政与办公人员","PROFESSION","适用职业",[150,151,152,153,154,27,155,156],"meeting insights analyzer","listen","feedback","recordings","behavioral","improve","communication",{"skillNo":158,"name":159,"nameZh":6,"description":160,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":161,"categories":162,"keywords":166,"publishedAt":41},"01M6C6R6XQMJQ9XDXHSM1QSDG3","invoice-organizer","Automatically organizes invoices and receipts for tax preparation by reading messy files, extracting key information, renaming them consistently, and sorting them into logical folders. Turns hours of manual bookkeeping into minutes of automated organization.","https:\u002F\u002Fgithub.com\u002Fswesmith\u002Fdavila7__claude-code-templates.734b8a50\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Finvoice-organizer\u002FSKILL.md",[163],{"categoryNo":164,"categoryName":165,"taxonomyType":147,"taxonomyTypeDesc":148},"01JBSEEDCAT000000000000040","财务",[167,168,169],"invoice organizer","tax","receipts",{"skillNo":171,"name":172,"nameZh":6,"description":173,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":174,"categories":175,"keywords":176,"publishedAt":41},"01M6C6R3RXFPKV39WYHWTDSNZP","file-organizer","Intelligently organizes your files and folders across your computer by understanding context, finding duplicates, suggesting better structures, and automating cleanup tasks. Reduces cognitive load and keeps your digital workspace tidy without manual effort.","https:\u002F\u002Fgithub.com\u002Fswesmith\u002Fdavila7__claude-code-templates.734b8a50\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fproductivity\u002Ffile-organizer\u002FSKILL.md",[],[177,178,179,180,181,182,183,184,185],"file","file organizer","tasks","duplicates","context","workspace","computer","tidy","cleanup"]