[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6DXQZ2HSP548JH2V3FQ7H2V":3,"skill-same:01M6DXQZ2HSP548JH2V3FQ7H2V":68},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":34,"linkOnly":35,"markdownContent":36,"markdownBody":37,"frontMatter":38,"refName":39,"commitSha":40,"skillDirectory":41,"forksCount":42,"repositoryHtmlUrl":43,"repositoryHomepageUrl":6,"ownerHtmlUrl":44,"ownerBlogUrl":45,"ownerDescription":46,"updatedAt":47,"install":48,"resources":61},"01M6DXQZ2HSP548JH2V3FQ7H2V","create-custom-dagster-component",null,"Create a custom Dagster Component with demo mode support, realistic asset structure, and optional custom scaffolder using the dg CLI. Use this skill if there is no Component included in an existing integration or if Dagster does not have the integration.","cianfhoghlaim\u002Fcianfhoghlaim","cianfhoghlaim","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F57019547?v=4","01M6DWR3GAV495PYDTZSR1FZNX","Cian Mac (an Déisigh Uí) Liatháin || Deacy-Lyons","01M6DWR3N3Z768BB64XVJN0AY2",4,"MIT","https:\u002F\u002Fgithub.com\u002Fcianfhoghlaim\u002Fcianfhoghlaim\u002Fblob\u002Fmain\u002Fdocs\u002Fresearch\u002Fdagster\u002Fdagster-dspy\u002F.claude\u002Fskills\u002Fcreate-custom-dagster-component\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[24,25,26,27,28,29,30,31,32,33],"component","create","dagster","create custom dagster component","cli","support","demo","integration","use","structure","2026-11-10 10:12:30",false,"---\nname: create-custom-dagster-component\ndescription: Create a custom Dagster Component with demo mode support, realistic asset structure, and optional custom scaffolder using the dg CLI. Use this skill if there is no Component included in an existing integration or if Dagster does not have the integration.\nlicense: MIT\n---\n\n# Create a custom Component\n\n## Overview\n\nThis skill automates the creation and validation of a new custom Dagster component using the `dg` CLI tool with uv as a package manager. It incorporates demo mode functionality for creating realistic demonstrations that can run locally without external dependencies. The documentation for creating good components can be found here https:\u002F\u002Fdocs.dagster.io\u002Fguides\u002Fbuild\u002Fcomponents\u002Fcreating-new-components\u002Fcreating-and-registering-a-component and here https:\u002F\u002Fgithub.com\u002Fdagster-io\u002Fdagster\u002Fblob\u002Fmaster\u002Fpython_modules\u002Flibraries\u002Fdagster-dbt\u002Fdagster_dbt\u002Fcomponents\u002Fdbt_project\u002Fcomponent.py for a complex example of a component.\n\n## What This Skill Does\n\nWhen invoked, this skill will:\n\n1. ✅ Create a new Dagster Component project using `dg scaffold component ComponentName`\n2. ✅ Fills in the component logic in the `build_defs()` function with both real and demo mode implementations\n3. ✅ Implement a `demo_mode` boolean flag in the component YAML for toggling between real and local demo implementations\n4. ✅ Create 3-5 realistic assets with proper dependencies and technology kinds\n5. ✅ Instantiate a component YAML and fill it in using the `dg scaffold defs my_module.components.ComponentName my_component` command\n6. ✅ Optionally create a custom scaffolder if requested\n7. ✅ Validate that the component loaded correctly using `dg check defs` and `dg list defs` to ensure that the expected component instances are all loaded.\n8. ✅ Provide clear next steps for development\n\n## Prerequisites\n\nBefore running this skill, ensure:\n\n- `uv` is installed (check with `uv --version`)\n- You have a component name in mind (or will use the default)\n- You're in a Dagster project directory with the dg CLI available\n- You have inspected and understand how to create multi-asset integrations (see this guide: https:\u002F\u002Fdocs.dagster.io\u002Fintegrations\u002Fguides\u002Fmulti-asset-integration)\n\n## Skill Workflow\n\n### Step 1: Get Component Name and Demo Mode Preference\n\nAsk the user for:\n\n1. A component name, or use a sensible default like `MyDagsterComponent`. Validate that:\n   - The name starts with a letter\n   - Contains only alphanumeric characters, hyphens, or underscores\n   - The component doesn't already exist (or ask to overwrite)\n2. Whether they want demo mode support (default: yes for demonstration projects)\n3. Whether they want to create a custom scaffolder (see Step 5)\n\n### Step 2: Create Component\n\nUse `dg` to create the component\n\n```bash\nuv run dg scaffold component \u003CComponentName>\n```\n\nThis will:\n\n- Scaffold a new Dagster Component in `defs\u002Fcomponents`\n- Create a `component_name.py` file\n\n### Step 3: Implement Component Logic with Demo Mode\n\nFill in the `build_defs()` function in the component file. The component should:\n\n1. **Accept a `demo_mode` parameter** in the component params (default: False)\n2. **Create 3-5 realistic assets** based on the chosen technologies\n3. **Implement dual logic paths:**\n   - Real implementation: connects to actual systems (the body should include connections to real systems)\n   - Demo mode: uses local data\u002Fmocked behavior for demonstrations\n   - Important Configuration in Pydantic + YAML fields: All configuration (a new pipeline or asset) should be configurable in the Component and _not_ hard coded in the Component Python. **Demo mode assets are the exception that _should_ be hard coded.**\n   - All Resources should be configured outside of the Component in the `defs\u002F` folder in a `resources.py` file, using `dg scaffold defs dagster.resource resources.py`\n   - All Components that invoke a pipeline or API that might trigger multiple assets should allow for an `assets` field in the YAML that describes what assets are used in the underlying component. See https:\u002F\u002Fdagster.io\u002Fblog\u002Fdsls-to-the-rescue for best practices in how to design a good DSL. Refer to https:\u002F\u002Fgithub.com\u002Fdagster-io\u002Fdagster\u002Fblob\u002Fmaster\u002Fpython_modules\u002Flibraries\u002Fdagster-dbt\u002Fdagster_dbt\u002Fcomponents\u002Fdbt_project\u002Fcomponent.py and https:\u002F\u002Fgithub.com\u002Fdagster-io\u002Fdagster\u002Fblob\u002Fmaster\u002Fpython_modules\u002Flibraries\u002Fdagster-fivetran\u002Fdagster_fivetran\u002Fcomponents\u002Fworkspace_component\u002Fcomponent.py for two reference architectures for good component design with mutli-assets.\n4. **Set proper asset metadata:**\n   - Use the `kinds` argument to indicate technologies in use\n   - Add descriptive names and documentation\n   - Establish proper dependencies between assets\n5. **Design asset keys for downstream integration** (CRITICAL):\n   - Consider what components will consume your assets\n   - Choose key structures that minimize downstream configuration\n   - See \"Design Asset Keys for Integration\" section below\n\nExample asset structure:\n\n- Raw data ingestion asset\n- Data transformation\u002Fcleaning asset\n- Business logic\u002Faggregation asset\n- ML model or analytics asset (if applicable)\n- Output\u002Fexport asset\n\n### Design Asset Keys for Integration\n\n**CRITICAL:** When creating a custom component, consider **what will consume** your component's assets. The asset keys you generate should align with downstream component expectations to avoid requiring per-asset configuration.\n\n#### Key Principle: Upstream Defines, Downstream Consumes\n\nYour component (upstream) should generate asset keys in a structure that downstream components naturally reference. This eliminates the need for `meta.dagster.asset_key` or complex translation configuration.\n\n#### Common Downstream Consumers\n\n**If dbt will consume your assets:**\n- Use pattern: `[\"\u003Csource_name>\", \"\u003Ctable_name>\"]`\n- Example: `[\"fivetran_raw\", \"customers\"]` or `[\"api_raw\", \"users\"]`\n- This allows dbt sources to reference naturally: `source('fivetran_raw', 'customers')`\n\n**If custom Dagster assets will consume them:**\n- Match the key structure those assets expect in their `deps`\n- Minimize nesting when possible (prefer 2 levels: `[\"category\", \"name\"]`)\n- Avoid deeply nested keys like `[\"system\", \"subsystem\", \"type\", \"name\"]` unless necessary\n\n**If another integration component will consume them:**\n- Check that component's expected input key structure\n- Align your keys to match, or provide clear mapping documentation\n\n**If your assets are intermediate and consumed by your own component:**\n- Use clear, hierarchical keys that reflect data flow\n- Example: `[\"raw\", \"table\"]` → `[\"processed\", \"table\"]` → `[\"enriched\", \"table\"]`\n\n#### Example: Creating an API Ingestion Component\n\n```python\nimport dagster as dg\n\nclass APIIngestionComponent(dg.Component, dg.Model, dg.Resolvable):\n    \"\"\"Ingests data from REST APIs.\"\"\"\n\n    api_endpoint: str\n    tables: list[str]\n    demo_mode: bool = False\n\n    def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:\n        assets = []\n\n        for table in self.tables:\n            # Design key for dbt consumption: [\"api_raw\", \"table_name\"]\n            # NOT: [\"api\", \"ingestion\", \"raw\", \"table_name\"]\n            @dg.asset(\n                key=dg.AssetKey([\"api_raw\", table]),  # ← Flattened for easy downstream reference\n                kinds={\"api\", \"python\"},\n            )\n            def ingest_table(context: dg.AssetExecutionContext):\n                if self.demo_mode:\n                    context.log.info(f\"Demo mode: Mocking API call for {table}\")\n                    return {\"status\": \"demo\", \"rows\": 100}\n                else:\n                    # Real API call\n                    pass\n\n            assets.append(ingest_table)\n\n        return dg.Definitions(assets=assets)\n```\n\n**Result:** dbt can reference these assets naturally:\n```yaml\n# sources.yml\nsources:\n  - name: api_raw\n    tables:\n      - name: customers  # Matches [\"api_raw\", \"customers\"]\n```\n\n#### Verification\n\nAlways verify asset keys align with downstream dependencies:\n\n```bash\n# Check asset keys and their dependencies\nuv run dg list defs --json | uv run python -c \"\nimport sys, json\nassets = json.load(sys.stdin)['assets']\nprint('\\\\n'.join([f\\\"{a['key']}: deps={a.get('deps', [])}\\\" for a in assets]))\n\"\n```\n\n**What to verify:**\n- Downstream assets list your assets in their `deps` array\n- No duplicate keys with different structures\n- Keys are simple and descriptive (typically 2 levels: `[\"category\", \"name\"]`)\n\n#### Anti-Patterns to Avoid\n\n❌ **Too deeply nested:** `[\"company\", \"team\", \"project\", \"environment\", \"table\"]`\n- Hard for downstream to reference\n- Requires complex mapping\n\n❌ **Inconsistent structure:** Some assets with 2 levels, others with 4\n- Confusing for consumers\n- Unpredictable references\n\n❌ **Generic names:** `[\"data\", \"table1\"]`, `[\"output\", \"result\"]`\n- Not clear what system they're from\n- Conflicts with other components\n\n✅ **Good patterns:**\n- `[\"source_system\", \"entity\"]`: `[\"fivetran_raw\", \"customers\"]`\n- `[\"integration\", \"object\"]`: `[\"salesforce\", \"accounts\"]`\n- `[\"stage\", \"table\"]`: `[\"staging\", \"orders\"]`\n\n#### Critical: Asset Keys Must Be Identical in Demo and Production Mode\n\n**IMPORTANT:** Asset keys should be **exactly the same** whether `demo_mode` is True or False. Only the asset implementation (the function body) should differ between modes.\n\n**Why this matters:**\n- Downstream components reference assets by key\n- Dependencies are established based on keys\n- If keys differ between modes, dependencies break when switching modes\n- Testing in demo mode won't accurately reflect production behavior\n\n**Example - CORRECT approach:**\n\n```python\ndef build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:\n    @dg.asset(\n        key=dg.AssetKey([\"fivetran_raw\", \"customers\"]),  # ← Same key in both modes\n        kinds={\"fivetran\"},\n    )\n    def customers_sync(context: dg.AssetExecutionContext):\n        if self.demo_mode:\n            # Demo implementation - mock data\n            context.log.info(\"Demo mode: Creating empty table\")\n            # ... create mock table\n        else:\n            # Production implementation - real Fivetran sync\n            context.log.info(\"Production: Syncing from Fivetran\")\n            # ... call Fivetran API\n\n    return dg.Definitions(assets=[customers_sync])\n```\n\n**Example - INCORRECT approach:**\n\n```python\ndef build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:\n    if self.demo_mode:\n        @dg.asset(\n            key=dg.AssetKey([\"demo\", \"customers\"]),  # ❌ Different key!\n        )\n        def demo_customers():\n            pass\n        return dg.Definitions(assets=[demo_customers])\n    else:\n        @dg.asset(\n            key=dg.AssetKey([\"fivetran_raw\", \"customers\"]),  # ❌ Different key!\n        )\n        def prod_customers():\n            pass\n        return dg.Definitions(assets=[prod_customers])\n```\n\n**Reference Documentation:**\n\n- Cross-reference https:\u002F\u002Fdocs.dagster.io\u002Fllms.txt for up-to-date titles and descriptions\n- Use https:\u002F\u002Fdocs.dagster.io\u002Fllms-full.txt for full API details\n- Check available integrations with:\n\n```bash\nuv run dg docs integrations --json\n```\n\n## Important: Always Add Kinds to Assets\n\nWhen creating assets in your component, **ALWAYS add the `kinds` parameter** to properly categorize assets by their technology\u002Fintegration type. This helps with:\n- Filtering and organizing assets in the Dagster UI\n- Understanding the technology stack at a glance\n- Grouping assets by integration type\n\n**Common integration kinds:**\n- `kinds={\"fivetran\"}` for Fivetran assets\n- `kinds={\"dbt\"}` for dbt assets\n- `kinds={\"census\"}` for Census assets\n- `kinds={\"sling\"}` for Sling assets\n- `kinds={\"powerbi\"}` for PowerBI assets\n- `kinds={\"looker\"}` for Looker assets\n- `kinds={\"airbyte\"}` for Airbyte assets\n- `kinds={\"python\"}` for custom Python processing\n- `kinds={\"snowflake\"}` for Snowflake assets\n\nYou can verify kinds are showing correctly by running:\n```bash\nuv run dg list defs\n```\nThe \"Kinds\" column should show the integration type for each asset.\n\n**Example Component Structure:**\n\n```python\nfrom dagster import asset, Definitions, AssetExecutionContext\nfrom pydantic import BaseModel\n\nclass MyComponentParams(BaseModel):\n    demo_mode: bool = False\n    # ... other params\n\nclass MyComponent(Component):\n    params_schema = MyComponentParams\n\n    def build_defs(self, context: ComponentLoadContext) -> Definitions:\n        params = self.params\n\n        @asset(\n            kinds={\"fivetran\"},  # ← REQUIRED: Add the integration kind\n        )\n        def raw_data(context: AssetExecutionContext):\n            if params.demo_mode:\n                # Demo implementation - local\u002Fmocked data\n                context.log.info(\"Running in demo mode with local data\")\n                pass\n            else:\n                # Real implementation - connect to actual systems\n                context.log.info(\"Running with real data source\")\n                pass\n\n        @asset(\n            deps=[raw_data],\n            kinds={\"dbt\"},  # ← REQUIRED: Add the integration kind\n        )\n        def processed_data(context: AssetExecutionContext):\n            if params.demo_mode:\n                context.log.info(\"Processing demo data\")\n                pass\n            else:\n                context.log.info(\"Processing real data\")\n                pass\n\n        # ... more assets\n\n        return Definitions(assets=[raw_data, processed_data, ...])\n```\n\n### Step 4: Create Component Instance YAML\n\nUse `dg scaffold defs` to create the component instance:\n\n```bash\nuv run dg scaffold defs my_module.components.ComponentName my_component\n```\n\nThis creates a YAML file that should include the `demo_mode` parameter:\n\n```yaml\ntype: my_module.components.ComponentName\nattributes:\n  demo_mode: true  # Set to true for local demos, false for real deployments\n  # ... other params\n```\n\n### Step 5: Create Custom Scaffolder (Optional)\n\nIf the user requested a custom scaffolder in Step 1, follow the directions here:\nhttps:\u002F\u002Fdocs.dagster.io\u002Fguides\u002Fbuild\u002Fcomponents\u002Fcreating-new-components\u002Fcomponent-customization#customizing-scaffolding-behavior\n\nCustomize the scaffolder to provide a better developer experience for creating instances of this component.\n\n### Step 6: Validate Setup and Asset Key Alignment\n\nRun these commands to ensure everything works:\n\n```bash\n# Check that definitions load without errors\nuv run dg check defs\n\n# List all assets to verify they were created\nuv run dg list defs\n```\n\nVerify that:\n\n- ✅ All expected assets are listed\n- ✅ The component instance is properly configured\n- ✅ No errors or warnings are shown\n- ✅ The `demo_mode` flag toggles between implementations correctly\n- ✅ The `demo_mode: false` implementation uses realistic resources and is a production implementation\n\n**CRITICAL: Verify Asset Key Alignment**\n\nCheck that asset dependencies are correct by running:\n\n```bash\nuv run dg list defs --json | uv run python -c \"\nimport sys, json\ndata = json.load(sys.stdin)\nassets = data.get('assets', [])\nprint('Asset Dependencies:\\n')\nfor asset in assets:\n    key = asset.get('key', 'unknown')\n    deps = asset.get('deps', [])\n    if deps:\n        print(f'{key}')\n        for dep in deps:\n            print(f'  ← {dep}')\n    else:\n        print(f'{key} (no dependencies)')\n    print()\n\"\n```\n\n**What to verify:**\n- ✅ Downstream assets list upstream assets in their `deps` array\n- ✅ No missing dependencies\n- ✅ Asset keys are simple and descriptive (typically 2 levels: `[\"category\", \"name\"]`)\n- ✅ Asset keys work consistently in both demo mode and production mode\n\n**Key Principle:** Asset keys should be **identical** between demo mode and production mode. Only the asset implementation (the function body) should differ. This ensures:\n- Dependencies work the same in both modes\n- You can switch between modes without reconfiguring downstream components\n- Testing in demo mode accurately reflects production behavior\n\n### Step 7: Test Demo Mode\n\nIf demo mode was implemented:\n\n1. Ensure the component YAML has `demo_mode: true`\n2. Run `dg check defs` to verify it works locally\n3. Document how to switch between demo and real modes\n\n## Success Criteria\n\nThe component is complete when:\n\n- ✅ Component scaffolding is created\n- ✅ `build_defs()` is implemented with proper asset logic\n- ✅ Demo mode flag is working (if applicable)\n- ✅ Non demo mode has realistic connections to the database or APIs implemented\n- ✅ 3-5 realistic assets are created with proper dependencies\n- ✅ Assets have appropriate `kinds` metadata\n- ✅ Component YAML instance is created and configured\n- ✅ Custom scaffolder is implemented (if requested)\n- ✅ `dg check defs` passes without errors\n- ✅ `dg list defs` shows all expected assets\n\n## Next Steps\n\nAfter completion, inform the user:\n\n1. The component has been created and validated\n2. Location of the component files\n3. How to toggle demo mode (if applicable)\n4. How to customize the component further\n5. How to create additional instances using the scaffolder","# Create a custom Component\n\n## Overview\n\nThis skill automates the creation and validation of a new custom Dagster component using the `dg` CLI tool with uv as a package manager. It incorporates demo mode functionality for creating realistic demonstrations that can run locally without external dependencies. The documentation for creating good components can be found here https:\u002F\u002Fdocs.dagster.io\u002Fguides\u002Fbuild\u002Fcomponents\u002Fcreating-new-components\u002Fcreating-and-registering-a-component and here https:\u002F\u002Fgithub.com\u002Fdagster-io\u002Fdagster\u002Fblob\u002Fmaster\u002Fpython_modules\u002Flibraries\u002Fdagster-dbt\u002Fdagster_dbt\u002Fcomponents\u002Fdbt_project\u002Fcomponent.py for a complex example of a component.\n\n## What This Skill Does\n\nWhen invoked, this skill will:\n\n1. ✅ Create a new Dagster Component project using `dg scaffold component ComponentName`\n2. ✅ Fills in the component logic in the `build_defs()` function with both real and demo mode implementations\n3. ✅ Implement a `demo_mode` boolean flag in the component YAML for toggling between real and local demo implementations\n4. ✅ Create 3-5 realistic assets with proper dependencies and technology kinds\n5. ✅ Instantiate a component YAML and fill it in using the `dg scaffold defs my_module.components.ComponentName my_component` command\n6. ✅ Optionally create a custom scaffolder if requested\n7. ✅ Validate that the component loaded correctly using `dg check defs` and `dg list defs` to ensure that the expected component instances are all loaded.\n8. ✅ Provide clear next steps for development\n\n## Prerequisites\n\nBefore running this skill, ensure:\n\n- `uv` is installed (check with `uv --version`)\n- You have a component name in mind (or will use the default)\n- You're in a Dagster project directory with the dg CLI available\n- You have inspected and understand how to create multi-asset integrations (see this guide: https:\u002F\u002Fdocs.dagster.io\u002Fintegrations\u002Fguides\u002Fmulti-asset-integration)\n\n## Skill Workflow\n\n### Step 1: Get Component Name and Demo Mode Preference\n\nAsk the user for:\n\n1. A component name, or use a sensible default like `MyDagsterComponent`. Validate that:\n   - The name starts with a letter\n   - Contains only alphanumeric characters, hyphens, or underscores\n   - The component doesn't already exist (or ask to overwrite)\n2. Whether they want demo mode support (default: yes for demonstration projects)\n3. Whether they want to create a custom scaffolder (see Step 5)\n\n### Step 2: Create Component\n\nUse `dg` to create the component\n\n```bash\nuv run dg scaffold component \u003CComponentName>\n```\n\nThis will:\n\n- Scaffold a new Dagster Component in `defs\u002Fcomponents`\n- Create a `component_name.py` file\n\n### Step 3: Implement Component Logic with Demo Mode\n\nFill in the `build_defs()` function in the component file. The component should:\n\n1. **Accept a `demo_mode` parameter** in the component params (default: False)\n2. **Create 3-5 realistic assets** based on the chosen technologies\n3. **Implement dual logic paths:**\n   - Real implementation: connects to actual systems (the body should include connections to real systems)\n   - Demo mode: uses local data\u002Fmocked behavior for demonstrations\n   - Important Configuration in Pydantic + YAML fields: All configuration (a new pipeline or asset) should be configurable in the Component and _not_ hard coded in the Component Python. **Demo mode assets are the exception that _should_ be hard coded.**\n   - All Resources should be configured outside of the Component in the `defs\u002F` folder in a `resources.py` file, using `dg scaffold defs dagster.resource resources.py`\n   - All Components that invoke a pipeline or API that might trigger multiple assets should allow for an `assets` field in the YAML that describes what assets are used in the underlying component. See https:\u002F\u002Fdagster.io\u002Fblog\u002Fdsls-to-the-rescue for best practices in how to design a good DSL. Refer to https:\u002F\u002Fgithub.com\u002Fdagster-io\u002Fdagster\u002Fblob\u002Fmaster\u002Fpython_modules\u002Flibraries\u002Fdagster-dbt\u002Fdagster_dbt\u002Fcomponents\u002Fdbt_project\u002Fcomponent.py and https:\u002F\u002Fgithub.com\u002Fdagster-io\u002Fdagster\u002Fblob\u002Fmaster\u002Fpython_modules\u002Flibraries\u002Fdagster-fivetran\u002Fdagster_fivetran\u002Fcomponents\u002Fworkspace_component\u002Fcomponent.py for two reference architectures for good component design with mutli-assets.\n4. **Set proper asset metadata:**\n   - Use the `kinds` argument to indicate technologies in use\n   - Add descriptive names and documentation\n   - Establish proper dependencies between assets\n5. **Design asset keys for downstream integration** (CRITICAL):\n   - Consider what components will consume your assets\n   - Choose key structures that minimize downstream configuration\n   - See \"Design Asset Keys for Integration\" section below\n\nExample asset structure:\n\n- Raw data ingestion asset\n- Data transformation\u002Fcleaning asset\n- Business logic\u002Faggregation asset\n- ML model or analytics asset (if applicable)\n- Output\u002Fexport asset\n\n### Design Asset Keys for Integration\n\n**CRITICAL:** When creating a custom component, consider **what will consume** your component's assets. The asset keys you generate should align with downstream component expectations to avoid requiring per-asset configuration.\n\n#### Key Principle: Upstream Defines, Downstream Consumes\n\nYour component (upstream) should generate asset keys in a structure that downstream components naturally reference. This eliminates the need for `meta.dagster.asset_key` or complex translation configuration.\n\n#### Common Downstream Consumers\n\n**If dbt will consume your assets:**\n- Use pattern: `[\"\u003Csource_name>\", \"\u003Ctable_name>\"]`\n- Example: `[\"fivetran_raw\", \"customers\"]` or `[\"api_raw\", \"users\"]`\n- This allows dbt sources to reference naturally: `source('fivetran_raw', 'customers')`\n\n**If custom Dagster assets will consume them:**\n- Match the key structure those assets expect in their `deps`\n- Minimize nesting when possible (prefer 2 levels: `[\"category\", \"name\"]`)\n- Avoid deeply nested keys like `[\"system\", \"subsystem\", \"type\", \"name\"]` unless necessary\n\n**If another integration component will consume them:**\n- Check that component's expected input key structure\n- Align your keys to match, or provide clear mapping documentation\n\n**If your assets are intermediate and consumed by your own component:**\n- Use clear, hierarchical keys that reflect data flow\n- Example: `[\"raw\", \"table\"]` → `[\"processed\", \"table\"]` → `[\"enriched\", \"table\"]`\n\n#### Example: Creating an API Ingestion Component\n\n```python\nimport dagster as dg\n\nclass APIIngestionComponent(dg.Component, dg.Model, dg.Resolvable):\n    \"\"\"Ingests data from REST APIs.\"\"\"\n\n    api_endpoint: str\n    tables: list[str]\n    demo_mode: bool = False\n\n    def build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:\n        assets = []\n\n        for table in self.tables:\n            # Design key for dbt consumption: [\"api_raw\", \"table_name\"]\n            # NOT: [\"api\", \"ingestion\", \"raw\", \"table_name\"]\n            @dg.asset(\n                key=dg.AssetKey([\"api_raw\", table]),  # ← Flattened for easy downstream reference\n                kinds={\"api\", \"python\"},\n            )\n            def ingest_table(context: dg.AssetExecutionContext):\n                if self.demo_mode:\n                    context.log.info(f\"Demo mode: Mocking API call for {table}\")\n                    return {\"status\": \"demo\", \"rows\": 100}\n                else:\n                    # Real API call\n                    pass\n\n            assets.append(ingest_table)\n\n        return dg.Definitions(assets=assets)\n```\n\n**Result:** dbt can reference these assets naturally:\n```yaml\n# sources.yml\nsources:\n  - name: api_raw\n    tables:\n      - name: customers  # Matches [\"api_raw\", \"customers\"]\n```\n\n#### Verification\n\nAlways verify asset keys align with downstream dependencies:\n\n```bash\n# Check asset keys and their dependencies\nuv run dg list defs --json | uv run python -c \"\nimport sys, json\nassets = json.load(sys.stdin)['assets']\nprint('\\\\n'.join([f\\\"{a['key']}: deps={a.get('deps', [])}\\\" for a in assets]))\n\"\n```\n\n**What to verify:**\n- Downstream assets list your assets in their `deps` array\n- No duplicate keys with different structures\n- Keys are simple and descriptive (typically 2 levels: `[\"category\", \"name\"]`)\n\n#### Anti-Patterns to Avoid\n\n❌ **Too deeply nested:** `[\"company\", \"team\", \"project\", \"environment\", \"table\"]`\n- Hard for downstream to reference\n- Requires complex mapping\n\n❌ **Inconsistent structure:** Some assets with 2 levels, others with 4\n- Confusing for consumers\n- Unpredictable references\n\n❌ **Generic names:** `[\"data\", \"table1\"]`, `[\"output\", \"result\"]`\n- Not clear what system they're from\n- Conflicts with other components\n\n✅ **Good patterns:**\n- `[\"source_system\", \"entity\"]`: `[\"fivetran_raw\", \"customers\"]`\n- `[\"integration\", \"object\"]`: `[\"salesforce\", \"accounts\"]`\n- `[\"stage\", \"table\"]`: `[\"staging\", \"orders\"]`\n\n#### Critical: Asset Keys Must Be Identical in Demo and Production Mode\n\n**IMPORTANT:** Asset keys should be **exactly the same** whether `demo_mode` is True or False. Only the asset implementation (the function body) should differ between modes.\n\n**Why this matters:**\n- Downstream components reference assets by key\n- Dependencies are established based on keys\n- If keys differ between modes, dependencies break when switching modes\n- Testing in demo mode won't accurately reflect production behavior\n\n**Example - CORRECT approach:**\n\n```python\ndef build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:\n    @dg.asset(\n        key=dg.AssetKey([\"fivetran_raw\", \"customers\"]),  # ← Same key in both modes\n        kinds={\"fivetran\"},\n    )\n    def customers_sync(context: dg.AssetExecutionContext):\n        if self.demo_mode:\n            # Demo implementation - mock data\n            context.log.info(\"Demo mode: Creating empty table\")\n            # ... create mock table\n        else:\n            # Production implementation - real Fivetran sync\n            context.log.info(\"Production: Syncing from Fivetran\")\n            # ... call Fivetran API\n\n    return dg.Definitions(assets=[customers_sync])\n```\n\n**Example - INCORRECT approach:**\n\n```python\ndef build_defs(self, context: dg.ComponentLoadContext) -> dg.Definitions:\n    if self.demo_mode:\n        @dg.asset(\n            key=dg.AssetKey([\"demo\", \"customers\"]),  # ❌ Different key!\n        )\n        def demo_customers():\n            pass\n        return dg.Definitions(assets=[demo_customers])\n    else:\n        @dg.asset(\n            key=dg.AssetKey([\"fivetran_raw\", \"customers\"]),  # ❌ Different key!\n        )\n        def prod_customers():\n            pass\n        return dg.Definitions(assets=[prod_customers])\n```\n\n**Reference Documentation:**\n\n- Cross-reference https:\u002F\u002Fdocs.dagster.io\u002Fllms.txt for up-to-date titles and descriptions\n- Use https:\u002F\u002Fdocs.dagster.io\u002Fllms-full.txt for full API details\n- Check available integrations with:\n\n```bash\nuv run dg docs integrations --json\n```\n\n## Important: Always Add Kinds to Assets\n\nWhen creating assets in your component, **ALWAYS add the `kinds` parameter** to properly categorize assets by their technology\u002Fintegration type. This helps with:\n- Filtering and organizing assets in the Dagster UI\n- Understanding the technology stack at a glance\n- Grouping assets by integration type\n\n**Common integration kinds:**\n- `kinds={\"fivetran\"}` for Fivetran assets\n- `kinds={\"dbt\"}` for dbt assets\n- `kinds={\"census\"}` for Census assets\n- `kinds={\"sling\"}` for Sling assets\n- `kinds={\"powerbi\"}` for PowerBI assets\n- `kinds={\"looker\"}` for Looker assets\n- `kinds={\"airbyte\"}` for Airbyte assets\n- `kinds={\"python\"}` for custom Python processing\n- `kinds={\"snowflake\"}` for Snowflake assets\n\nYou can verify kinds are showing correctly by running:\n```bash\nuv run dg list defs\n```\nThe \"Kinds\" column should show the integration type for each asset.\n\n**Example Component Structure:**\n\n```python\nfrom dagster import asset, Definitions, AssetExecutionContext\nfrom pydantic import BaseModel\n\nclass MyComponentParams(BaseModel):\n    demo_mode: bool = False\n    # ... other params\n\nclass MyComponent(Component):\n    params_schema = MyComponentParams\n\n    def build_defs(self, context: ComponentLoadContext) -> Definitions:\n        params = self.params\n\n        @asset(\n            kinds={\"fivetran\"},  # ← REQUIRED: Add the integration kind\n        )\n        def raw_data(context: AssetExecutionContext):\n            if params.demo_mode:\n                # Demo implementation - local\u002Fmocked data\n                context.log.info(\"Running in demo mode with local data\")\n                pass\n            else:\n                # Real implementation - connect to actual systems\n                context.log.info(\"Running with real data source\")\n                pass\n\n        @asset(\n            deps=[raw_data],\n            kinds={\"dbt\"},  # ← REQUIRED: Add the integration kind\n        )\n        def processed_data(context: AssetExecutionContext):\n            if params.demo_mode:\n                context.log.info(\"Processing demo data\")\n                pass\n            else:\n                context.log.info(\"Processing real data\")\n                pass\n\n        # ... more assets\n\n        return Definitions(assets=[raw_data, processed_data, ...])\n```\n\n### Step 4: Create Component Instance YAML\n\nUse `dg scaffold defs` to create the component instance:\n\n```bash\nuv run dg scaffold defs my_module.components.ComponentName my_component\n```\n\nThis creates a YAML file that should include the `demo_mode` parameter:\n\n```yaml\ntype: my_module.components.ComponentName\nattributes:\n  demo_mode: true  # Set to true for local demos, false for real deployments\n  # ... other params\n```\n\n### Step 5: Create Custom Scaffolder (Optional)\n\nIf the user requested a custom scaffolder in Step 1, follow the directions here:\nhttps:\u002F\u002Fdocs.dagster.io\u002Fguides\u002Fbuild\u002Fcomponents\u002Fcreating-new-components\u002Fcomponent-customization#customizing-scaffolding-behavior\n\nCustomize the scaffolder to provide a better developer experience for creating instances of this component.\n\n### Step 6: Validate Setup and Asset Key Alignment\n\nRun these commands to ensure everything works:\n\n```bash\n# Check that definitions load without errors\nuv run dg check defs\n\n# List all assets to verify they were created\nuv run dg list defs\n```\n\nVerify that:\n\n- ✅ All expected assets are listed\n- ✅ The component instance is properly configured\n- ✅ No errors or warnings are shown\n- ✅ The `demo_mode` flag toggles between implementations correctly\n- ✅ The `demo_mode: false` implementation uses realistic resources and is a production implementation\n\n**CRITICAL: Verify Asset Key Alignment**\n\nCheck that asset dependencies are correct by running:\n\n```bash\nuv run dg list defs --json | uv run python -c \"\nimport sys, json\ndata = json.load(sys.stdin)\nassets = data.get('assets', [])\nprint('Asset Dependencies:\\n')\nfor asset in assets:\n    key = asset.get('key', 'unknown')\n    deps = asset.get('deps', [])\n    if deps:\n        print(f'{key}')\n        for dep in deps:\n            print(f'  ← {dep}')\n    else:\n        print(f'{key} (no dependencies)')\n    print()\n\"\n```\n\n**What to verify:**\n- ✅ Downstream assets list upstream assets in their `deps` array\n- ✅ No missing dependencies\n- ✅ Asset keys are simple and descriptive (typically 2 levels: `[\"category\", \"name\"]`)\n- ✅ Asset keys work consistently in both demo mode and production mode\n\n**Key Principle:** Asset keys should be **identical** between demo mode and production mode. Only the asset implementation (the function body) should differ. This ensures:\n- Dependencies work the same in both modes\n- You can switch between modes without reconfiguring downstream components\n- Testing in demo mode accurately reflects production behavior\n\n### Step 7: Test Demo Mode\n\nIf demo mode was implemented:\n\n1. Ensure the component YAML has `demo_mode: true`\n2. Run `dg check defs` to verify it works locally\n3. Document how to switch between demo and real modes\n\n## Success Criteria\n\nThe component is complete when:\n\n- ✅ Component scaffolding is created\n- ✅ `build_defs()` is implemented with proper asset logic\n- ✅ Demo mode flag is working (if applicable)\n- ✅ Non demo mode has realistic connections to the database or APIs implemented\n- ✅ 3-5 realistic assets are created with proper dependencies\n- ✅ Assets have appropriate `kinds` metadata\n- ✅ Component YAML instance is created and configured\n- ✅ Custom scaffolder is implemented (if requested)\n- ✅ `dg check defs` passes without errors\n- ✅ `dg list defs` shows all expected assets\n\n## Next Steps\n\nAfter completion, inform the user:\n\n1. The component has been created and validated\n2. Location of the component files\n3. How to toggle demo mode (if applicable)\n4. How to customize the component further\n5. How to create additional instances using the scaffolder",{"name":5,"license":15,"description":7},"main","5ace051462d0639f59bcc60912f4ec08c1a96772","docs\u002Fresearch\u002Fdagster\u002Fdagster-dspy\u002F.claude\u002Fskills\u002Fcreate-custom-dagster-component",1,"https:\u002F\u002Fgithub.com\u002Fcianfhoghlaim\u002Fcianfhoghlaim","https:\u002F\u002Fgithub.com\u002Fcianfhoghlaim","cianfhoghlaim.ie","Irish-British teacher, mathematician, foghlaimeoir || minority-language AI researcher || Lucht Siúil ||\r\nFine Gael + Alliance Party + Liberal Democrats Member","2026-11-14 03:06:52",{"source":8,"skillName":5,"commands":49,"prompt":59,"zipAvailable":60},[50,53,56],{"tool":51,"command":52},"npx","npx skills add cianfhoghlaim\u002Fcianfhoghlaim --skill create-custom-dagster-component",{"tool":54,"command":55},"bunx","bunx skills add cianfhoghlaim\u002Fcianfhoghlaim --skill create-custom-dagster-component",{"tool":57,"command":58},"pnpm","pnpm dlx skills add cianfhoghlaim\u002Fcianfhoghlaim --skill create-custom-dagster-component","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fcianfhoghlaim\u002Fcianfhoghlaim\u002Fblob\u002Fmain\u002Fdocs\u002Fresearch\u002Fdagster\u002Fdagster-dspy\u002F.claude\u002Fskills\u002Fcreate-custom-dagster-component\u002FSKILL.md",true,[62],{"path":63,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":66,"sourceHtmlUrl":67},"index.json","REFERENCE","参考文档",690,"https:\u002F\u002Fgithub.com\u002FDCx7C5\u002Fai-marketplace\u002Fblob\u002Fmain\u002Fskills\u002Fnetwork\u002Flayer3\u002Fbgp\u002Fhijack\u002Fexploit\u002Findex.json",[69,87,108,132,160,183],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":34},"01M6DXR6F7GB4JSD67XTX6WY4A","dagster-init","Initialize a dagster project using the create-dagster cli. Create a dagster project, uv virtual environment, and everything needed for a user to run dg dev or dg check defs successfully. (project)","https:\u002F\u002Fgithub.com\u002Fcianfhoghlaim\u002Fcianfhoghlaim\u002Fblob\u002Fmain\u002Fdocs\u002Fresearch\u002Fdagster\u002Fdagster-dspy\u002F.claude\u002Fskills\u002Fdagster-init\u002FSKILL.md",[],[76,26,77,28,25,78,79,80,81,82,83,84,85,86],"init","dagster init","run","dev","virtual","check","initialize","user","environment","uv","project",{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":34},"01M6DXHPZMQ4VAXH9PGBK2PNNP","pydantic-ai-harness","Extend Pydantic AI agents with batteries-included capabilities from pydantic-ai-harness -- currently Code Mode, which collapses many tool calls into one sandboxed Python execution. Use when the user mentions pydantic-ai-harness, CodeMode, Monty, code mode, or tool sandboxing, when they want an agent to run agent-written Python, or when a Pydantic AI agent would benefit from orchestrating multiple tool calls in a single sandboxed script.","https:\u002F\u002Fgithub.com\u002Fcianfhoghlaim\u002Fcianfhoghlaim\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fpydantic\u002Fpydantic-ai-harness\u002FSKILL.md",[],[94,95,96,97,98,99,100,78,101,102,103,104,105,83,32,106,107],"harness","Pydantic","pydantic ai","pydantic ai harness","python","ai agents","ai agent","code","capabilities","execution","sandboxing","tool","pydantic-ai","batteries",{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":112,"categories":113,"keywords":118,"publishedAt":34},"01M6DXH7JE54YEHM58JY8KY2AK","building-pydantic-ai-agents","Build AI agents with Pydantic AI — tools, capabilities (including on-demand loading), structured output, streaming, testing, and multi-agent patterns. Use when the user mentions Pydantic AI, imports pydantic_ai, or asks to build an AI agent, add tools\u002Fcapabilities, defer capability loading, stream output, define agents from YAML, or test agent behavior.","https:\u002F\u002Fgithub.com\u002Fcianfhoghlaim\u002Fcianfhoghlaim\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fpydantic\u002Fbuilding-pydantic-ai-agents\u002FSKILL.md",[114,117],{"categoryNo":115,"categoryName":116,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000008","测试与质量",{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[99,95,96,119,120,98,100,121,102,122,123,124,125,126,32,127,128,129,130,131],"building pydantic ai agents","testing","test","streaming","yaml","Tools","multi-agent","behavior","build","patterns","agent patterns","define","structured output",{"skillNo":133,"name":134,"nameZh":6,"description":135,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":136,"categories":137,"keywords":143,"publishedAt":34},"01M6DXGCVB0WSHBQ7BSA56A12C","motherduck-share-data","Create and manage MotherDuck data shares for zero-copy, read-only data distribution. Use whenever someone wants to share a database with team members, another organization, or the public — covers CREATE SHARE, access\u002Fvisibility\u002Fupdate modes, GRANT READ ON SHARE, attaching share URLs, UPDATE SHARE, and REFRESH DATABASE.","https:\u002F\u002Fgithub.com\u002Fcianfhoghlaim\u002Fcianfhoghlaim\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fmotherduck\u002Fmotherduck-share-data\u002FSKILL.md",[138],{"categoryNo":139,"categoryName":140,"taxonomyType":141,"taxonomyTypeDesc":142},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[144,145,146,147,25,148,149,150,151,152,153,154,155,32,156,157,158,159],"share","data","motherduck","motherduck share data","update","database","ZeRO","team","access","public","read","refresh","only","manage","organization","distribution",{"skillNo":161,"name":162,"nameZh":6,"description":163,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":164,"categories":165,"keywords":166,"publishedAt":34},"01M6DXG9NMFYC72KJWQVJ1D6BZ","motherduck-security-governance","Explain MotherDuck security, governance, and access-control patterns. Use for any question about SOC 2, GDPR, compliance, data residency, regions, SSO, service accounts, token handling, tenant isolation, sharing boundaries, snapshots and recovery, or governance posture — including when a security_compliance_owner, technical_owner, or application_builder is evaluating MotherDuck.","https:\u002F\u002Fgithub.com\u002Fcianfhoghlaim\u002Fcianfhoghlaim\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fmotherduck\u002Fmotherduck-security-governance\u002FSKILL.md",[],[167,168,146,169,170,171,172,145,173,174,175,152,176,177,178,32,179,180,181,182],"security","governance","motherduck security governance","compliance","soc","gdpr","sso","sharing","recovery","explain","technical","question","builder","snapshots","token","control",{"skillNo":184,"name":185,"nameZh":6,"description":186,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":187,"categories":188,"keywords":195,"publishedAt":34},"01M6DXG6DFVSAPB6ATMWR6QGMV","motherduck-rest-api","MotherDuck REST API control-plane reference. Use when calling api.motherduck.com to provision service accounts, create, list, rotate, or revoke access tokens, configure Duckling instance sizes and read scaling, inspect active accounts, or mint Dive embed sessions. Not for SQL or data-plane query work.","https:\u002F\u002Fgithub.com\u002Fcianfhoghlaim\u002Fcianfhoghlaim\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fmotherduck\u002Fmotherduck-rest-api\u002FSKILL.md",[189,192],{"categoryNo":190,"categoryName":191,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000010","后端与接口",{"categoryNo":193,"categoryName":194,"taxonomyType":141,"taxonomyTypeDesc":142},"01JBSEEDCAT000000000000030","开发工程师",[196,197,198,146,199,200,201,25,152,202,203,154,32,204,205,206,207,208,209,182],"REST","api","rest api","motherduck rest api","sessions","sql","configure","work","mint","calling","query","active","tokens","scaling"]