[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6CKPF3QCDRCHQDHEVXM40AM":3,"skill-same:01M6CKPF3QCDRCHQDHEVXM40AM":171},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":22,"publishedAt":43,"linkOnly":44,"markdownContent":45,"markdownBody":46,"frontMatter":47,"refName":48,"commitSha":49,"skillDirectory":50,"forksCount":51,"repositoryHtmlUrl":52,"repositoryHomepageUrl":6,"ownerHtmlUrl":53,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":54,"install":55,"resources":68},"01M6CKPF3QCDRCHQDHEVXM40AM","senior-data-engineer",null,"Data engineering skill for building scalable data pipelines, ETL\u002FELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when designing data architectures, building data pipelines, optimizing data workflows, implementing data governance, or troubleshooting data issues.","kbarbel640-del\u002Fskills","kbarbel640-del","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F255315490?v=4","01M6CJPPQGCCESXMG3K7BBGQ5J","01M6CJPPX2PB2BCZERFVM2C2B3",1,"MIT","https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Falirezarezvani\u002Fsenior-data-engineer\u002FSKILL.md",[17],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42],"engineer","senior data engineer","data","data engineer","data pipelines","stack","pipeline","kafka","etl","elt","python","data engineering","airflow","issues","data governance","infrastructure","dataops","sql","workflows","orchestration","2026-11-10 10:12:30",false,"---\nname: senior-data-engineer\ndescription: Data engineering skill for building scalable data pipelines, ETL\u002FELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when designing data architectures, building data pipelines, optimizing data workflows, implementing data governance, or troubleshooting data issues.\n---\n\n# Senior Data Engineer\n\nProduction-grade data engineering skill for building scalable, reliable data systems.\n\n## Table of Contents\n\n1. [Trigger Phrases](#trigger-phrases)\n2. [Quick Start](#quick-start)\n3. [Workflows](#workflows)\n   - [Building a Batch ETL Pipeline](#workflow-1-building-a-batch-etl-pipeline)\n   - [Implementing Real-Time Streaming](#workflow-2-implementing-real-time-streaming)\n   - [Data Quality Framework Setup](#workflow-3-data-quality-framework-setup)\n4. [Architecture Decision Framework](#architecture-decision-framework)\n5. [Tech Stack](#tech-stack)\n6. [Reference Documentation](#reference-documentation)\n7. [Troubleshooting](#troubleshooting)\n\n---\n\n## Trigger Phrases\n\nActivate this skill when you see:\n\n**Pipeline Design:**\n- \"Design a data pipeline for...\"\n- \"Build an ETL\u002FELT process...\"\n- \"How should I ingest data from...\"\n- \"Set up data extraction from...\"\n\n**Architecture:**\n- \"Should I use batch or streaming?\"\n- \"Lambda vs Kappa architecture\"\n- \"How to handle late-arriving data\"\n- \"Design a data lakehouse\"\n\n**Data Modeling:**\n- \"Create a dimensional model...\"\n- \"Star schema vs snowflake\"\n- \"Implement slowly changing dimensions\"\n- \"Design a data vault\"\n\n**Data Quality:**\n- \"Add data validation to...\"\n- \"Set up data quality checks\"\n- \"Monitor data freshness\"\n- \"Implement data contracts\"\n\n**Performance:**\n- \"Optimize this Spark job\"\n- \"Query is running slow\"\n- \"Reduce pipeline execution time\"\n- \"Tune Airflow DAG\"\n\n---\n\n## Quick Start\n\n### Core Tools\n\n```bash\n# Generate pipeline orchestration config\npython scripts\u002Fpipeline_orchestrator.py generate \\\n  --type airflow \\\n  --source postgres \\\n  --destination snowflake \\\n  --schedule \"0 5 * * *\"\n\n# Validate data quality\npython scripts\u002Fdata_quality_validator.py validate \\\n  --input data\u002Fsales.parquet \\\n  --schema schemas\u002Fsales.json \\\n  --checks freshness,completeness,uniqueness\n\n# Optimize ETL performance\npython scripts\u002Fetl_performance_optimizer.py analyze \\\n  --query queries\u002Fdaily_aggregation.sql \\\n  --engine spark \\\n  --recommend\n```\n\n---\n\n## Workflows\n\n### Workflow 1: Building a Batch ETL Pipeline\n\n**Scenario:** Extract data from PostgreSQL, transform with dbt, load to Snowflake.\n\n#### Step 1: Define Source Schema\n\n```sql\n-- Document source tables\nSELECT\n    table_name,\n    column_name,\n    data_type,\n    is_nullable\nFROM information_schema.columns\nWHERE table_schema = 'source_schema'\nORDER BY table_name, ordinal_position;\n```\n\n#### Step 2: Generate Extraction Config\n\n```bash\npython scripts\u002Fpipeline_orchestrator.py generate \\\n  --type airflow \\\n  --source postgres \\\n  --tables orders,customers,products \\\n  --mode incremental \\\n  --watermark updated_at \\\n  --output dags\u002Fextract_source.py\n```\n\n#### Step 3: Create dbt Models\n\n```sql\n-- models\u002Fstaging\u002Fstg_orders.sql\nWITH source AS (\n    SELECT * FROM {{ source('postgres', 'orders') }}\n),\n\nrenamed AS (\n    SELECT\n        order_id,\n        customer_id,\n        order_date,\n        total_amount,\n        status,\n        _extracted_at\n    FROM source\n    WHERE order_date >= DATEADD(day, -3, CURRENT_DATE)\n)\n\nSELECT * FROM renamed\n```\n\n```sql\n-- models\u002Fmarts\u002Ffct_orders.sql\n{{\n    config(\n        materialized='incremental',\n        unique_key='order_id',\n        cluster_by=['order_date']\n    )\n}}\n\nSELECT\n    o.order_id,\n    o.customer_id,\n    c.customer_segment,\n    o.order_date,\n    o.total_amount,\n    o.status\nFROM {{ ref('stg_orders') }} o\nLEFT JOIN {{ ref('dim_customers') }} c\n    ON o.customer_id = c.customer_id\n\n{% if is_incremental() %}\nWHERE o._extracted_at > (SELECT MAX(_extracted_at) FROM {{ this }})\n{% endif %}\n```\n\n#### Step 4: Configure Data Quality Tests\n\n```yaml\n# models\u002Fmarts\u002Fschema.yml\nversion: 2\n\nmodels:\n  - name: fct_orders\n    description: \"Order fact table\"\n    columns:\n      - name: order_id\n        tests:\n          - unique\n          - not_null\n      - name: total_amount\n        tests:\n          - not_null\n          - dbt_utils.accepted_range:\n              min_value: 0\n              max_value: 1000000\n      - name: order_date\n        tests:\n          - not_null\n          - dbt_utils.recency:\n              datepart: day\n              field: order_date\n              interval: 1\n```\n\n#### Step 5: Create Airflow DAG\n\n```python\n# dags\u002Fdaily_etl.py\nfrom airflow import DAG\nfrom airflow.providers.postgres.operators.postgres import PostgresOperator\nfrom airflow.operators.bash import BashOperator\nfrom airflow.utils.dates import days_ago\nfrom datetime import timedelta\n\ndefault_args = {\n    'owner': 'data-team',\n    'depends_on_past': False,\n    'email_on_failure': True,\n    'email': ['data-alerts@company.com'],\n    'retries': 2,\n    'retry_delay': timedelta(minutes=5),\n}\n\nwith DAG(\n    'daily_etl_pipeline',\n    default_args=default_args,\n    description='Daily ETL from PostgreSQL to Snowflake',\n    schedule_interval='0 5 * * *',\n    start_date=days_ago(1),\n    catchup=False,\n    tags=['etl', 'daily'],\n) as dag:\n\n    extract = BashOperator(\n        task_id='extract_source_data',\n        bash_command='python \u002Fopt\u002Fairflow\u002Fscripts\u002Fextract.py --date {{ ds }}',\n    )\n\n    transform = BashOperator(\n        task_id='run_dbt_models',\n        bash_command='cd \u002Fopt\u002Fairflow\u002Fdbt && dbt run --select marts.*',\n    )\n\n    test = BashOperator(\n        task_id='run_dbt_tests',\n        bash_command='cd \u002Fopt\u002Fairflow\u002Fdbt && dbt test --select marts.*',\n    )\n\n    notify = BashOperator(\n        task_id='send_notification',\n        bash_command='python \u002Fopt\u002Fairflow\u002Fscripts\u002Fnotify.py --status success',\n        trigger_rule='all_success',\n    )\n\n    extract >> transform >> test >> notify\n```\n\n#### Step 6: Validate Pipeline\n\n```bash\n# Test locally\ndbt run --select stg_orders fct_orders\ndbt test --select fct_orders\n\n# Validate data quality\npython scripts\u002Fdata_quality_validator.py validate \\\n  --table fct_orders \\\n  --checks all \\\n  --output reports\u002Fquality_report.json\n```\n\n---\n\n### Workflow 2: Implementing Real-Time Streaming\n\n**Scenario:** Stream events from Kafka, process with Flink\u002FSpark Streaming, sink to data lake.\n\n#### Step 1: Define Event Schema\n\n```json\n{\n  \"$schema\": \"http:\u002F\u002Fjson-schema.org\u002Fdraft-07\u002Fschema#\",\n  \"title\": \"UserEvent\",\n  \"type\": \"object\",\n  \"required\": [\"event_id\", \"user_id\", \"event_type\", \"timestamp\"],\n  \"properties\": {\n    \"event_id\": {\"type\": \"string\", \"format\": \"uuid\"},\n    \"user_id\": {\"type\": \"string\"},\n    \"event_type\": {\"type\": \"string\", \"enum\": [\"page_view\", \"click\", \"purchase\"]},\n    \"timestamp\": {\"type\": \"string\", \"format\": \"date-time\"},\n    \"properties\": {\"type\": \"object\"}\n  }\n}\n```\n\n#### Step 2: Create Kafka Topic\n\n```bash\n# Create topic with appropriate partitions\nkafka-topics.sh --create \\\n  --bootstrap-server localhost:9092 \\\n  --topic user-events \\\n  --partitions 12 \\\n  --replication-factor 3 \\\n  --config retention.ms=604800000 \\\n  --config cleanup.policy=delete\n\n# Verify topic\nkafka-topics.sh --describe \\\n  --bootstrap-server localhost:9092 \\\n  --topic user-events\n```\n\n#### Step 3: Implement Spark Streaming Job\n\n```python\n# streaming\u002Fuser_events_processor.py\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.functions import (\n    from_json, col, window, count, avg,\n    to_timestamp, current_timestamp\n)\nfrom pyspark.sql.types import (\n    StructType, StructField, StringType,\n    TimestampType, MapType\n)\n\n# Initialize Spark\nspark = SparkSession.builder \\\n    .appName(\"UserEventsProcessor\") \\\n    .config(\"spark.sql.streaming.checkpointLocation\", \"\u002Fcheckpoints\u002Fuser-events\") \\\n    .config(\"spark.sql.shuffle.partitions\", \"12\") \\\n    .getOrCreate()\n\n# Define schema\nevent_schema = StructType([\n    StructField(\"event_id\", StringType(), False),\n    StructField(\"user_id\", StringType(), False),\n    StructField(\"event_type\", StringType(), False),\n    StructField(\"timestamp\", StringType(), False),\n    StructField(\"properties\", MapType(StringType(), StringType()), True)\n])\n\n# Read from Kafka\nevents_df = spark.readStream \\\n    .format(\"kafka\") \\\n    .option(\"kafka.bootstrap.servers\", \"localhost:9092\") \\\n    .option(\"subscribe\", \"user-events\") \\\n    .option(\"startingOffsets\", \"latest\") \\\n    .option(\"failOnDataLoss\", \"false\") \\\n    .load()\n\n# Parse JSON\nparsed_df = events_df \\\n    .select(from_json(col(\"value\").cast(\"string\"), event_schema).alias(\"data\")) \\\n    .select(\"data.*\") \\\n    .withColumn(\"event_timestamp\", to_timestamp(col(\"timestamp\")))\n\n# Windowed aggregation\naggregated_df = parsed_df \\\n    .withWatermark(\"event_timestamp\", \"10 minutes\") \\\n    .groupBy(\n        window(col(\"event_timestamp\"), \"5 minutes\"),\n        col(\"event_type\")\n    ) \\\n    .agg(\n        count(\"*\").alias(\"event_count\"),\n        approx_count_distinct(\"user_id\").alias(\"unique_users\")\n    )\n\n# Write to Delta Lake\nquery = aggregated_df.writeStream \\\n    .format(\"delta\") \\\n    .outputMode(\"append\") \\\n    .option(\"checkpointLocation\", \"\u002Fcheckpoints\u002Fuser-events-aggregated\") \\\n    .option(\"path\", \"\u002Fdata\u002Flake\u002Fuser_events_aggregated\") \\\n    .trigger(processingTime=\"1 minute\") \\\n    .start()\n\nquery.awaitTermination()\n```\n\n#### Step 4: Handle Late Data and Errors\n\n```python\n# Dead letter queue for failed records\nfrom pyspark.sql.functions import current_timestamp, lit\n\ndef process_with_error_handling(batch_df, batch_id):\n    try:\n        # Attempt processing\n        valid_df = batch_df.filter(col(\"event_id\").isNotNull())\n        invalid_df = batch_df.filter(col(\"event_id\").isNull())\n\n        # Write valid records\n        valid_df.write \\\n            .format(\"delta\") \\\n            .mode(\"append\") \\\n            .save(\"\u002Fdata\u002Flake\u002Fuser_events\")\n\n        # Write invalid to DLQ\n        if invalid_df.count() > 0:\n            invalid_df \\\n                .withColumn(\"error_timestamp\", current_timestamp()) \\\n                .withColumn(\"error_reason\", lit(\"missing_event_id\")) \\\n                .write \\\n                .format(\"delta\") \\\n                .mode(\"append\") \\\n                .save(\"\u002Fdata\u002Flake\u002Fdlq\u002Fuser_events\")\n\n    except Exception as e:\n        # Log error, alert, continue\n        logger.error(f\"Batch {batch_id} failed: {e}\")\n        raise\n\n# Use foreachBatch for custom processing\nquery = parsed_df.writeStream \\\n    .foreachBatch(process_with_error_handling) \\\n    .option(\"checkpointLocation\", \"\u002Fcheckpoints\u002Fuser-events\") \\\n    .start()\n```\n\n#### Step 5: Monitor Stream Health\n\n```python\n# monitoring\u002Fstream_metrics.py\nfrom prometheus_client import Gauge, Counter, start_http_server\n\n# Define metrics\nRECORDS_PROCESSED = Counter(\n    'stream_records_processed_total',\n    'Total records processed',\n    ['stream_name', 'status']\n)\n\nPROCESSING_LAG = Gauge(\n    'stream_processing_lag_seconds',\n    'Current processing lag',\n    ['stream_name']\n)\n\nBATCH_DURATION = Gauge(\n    'stream_batch_duration_seconds',\n    'Last batch processing duration',\n    ['stream_name']\n)\n\ndef emit_metrics(query):\n    \"\"\"Emit Prometheus metrics from streaming query.\"\"\"\n    progress = query.lastProgress\n    if progress:\n        RECORDS_PROCESSED.labels(\n            stream_name='user-events',\n            status='success'\n        ).inc(progress['numInputRows'])\n\n        if progress['sources']:\n            # Calculate lag from latest offset\n            for source in progress['sources']:\n                end_offset = source.get('endOffset', {})\n                # Parse Kafka offsets and calculate lag\n```\n\n---\n\n### Workflow 3: Data Quality Framework Setup\n\n**Scenario:** Implement comprehensive data quality monitoring with Great Expectations.\n\n#### Step 1: Initialize Great Expectations\n\n```bash\n# Install and initialize\npip install great_expectations\n\ngreat_expectations init\n\n# Connect to data source\ngreat_expectations datasource new\n```\n\n#### Step 2: Create Expectation Suite\n\n```python\n# expectations\u002Forders_suite.py\nimport great_expectations as gx\n\ncontext = gx.get_context()\n\n# Create expectation suite\nsuite = context.add_expectation_suite(\"orders_quality_suite\")\n\n# Add expectations\nvalidator = context.get_validator(\n    batch_request={\n        \"datasource_name\": \"warehouse\",\n        \"data_asset_name\": \"orders\",\n    },\n    expectation_suite_name=\"orders_quality_suite\"\n)\n\n# Schema expectations\nvalidator.expect_table_columns_to_match_ordered_list(\n    column_list=[\n        \"order_id\", \"customer_id\", \"order_date\",\n        \"total_amount\", \"status\", \"created_at\"\n    ]\n)\n\n# Completeness expectations\nvalidator.expect_column_values_to_not_be_null(\"order_id\")\nvalidator.expect_column_values_to_not_be_null(\"customer_id\")\nvalidator.expect_column_values_to_not_be_null(\"order_date\")\n\n# Uniqueness expectations\nvalidator.expect_column_values_to_be_unique(\"order_id\")\n\n# Range expectations\nvalidator.expect_column_values_to_be_between(\n    \"total_amount\",\n    min_value=0,\n    max_value=1000000\n)\n\n# Categorical expectations\nvalidator.expect_column_values_to_be_in_set(\n    \"status\",\n    [\"pending\", \"confirmed\", \"shipped\", \"delivered\", \"cancelled\"]\n)\n\n# Freshness expectation\nvalidator.expect_column_max_to_be_between(\n    \"order_date\",\n    min_value={\"$PARAMETER\": \"now - timedelta(days=1)\"},\n    max_value={\"$PARAMETER\": \"now\"}\n)\n\n# Referential integrity\nvalidator.expect_column_values_to_be_in_set(\n    \"customer_id\",\n    value_set={\"$PARAMETER\": \"valid_customer_ids\"}\n)\n\nvalidator.save_expectation_suite(discard_failed_expectations=False)\n```\n\n#### Step 3: Create Data Quality Checks with dbt\n\n```yaml\n# models\u002Fmarts\u002Fschema.yml\nversion: 2\n\nmodels:\n  - name: fct_orders\n    description: \"Order fact table with data quality checks\"\n\n    tests:\n      # Row count check\n      - dbt_utils.equal_rowcount:\n          compare_model: ref('stg_orders')\n\n      # Freshness check\n      - dbt_utils.recency:\n          datepart: hour\n          field: created_at\n          interval: 24\n\n    columns:\n      - name: order_id\n        description: \"Unique order identifier\"\n        tests:\n          - unique\n          - not_null\n          - relationships:\n              to: ref('dim_orders')\n              field: order_id\n\n      - name: total_amount\n        tests:\n          - not_null\n          - dbt_utils.accepted_range:\n              min_value: 0\n              max_value: 1000000\n              inclusive: true\n          - dbt_expectations.expect_column_values_to_be_between:\n              min_value: 0\n              row_condition: \"status != 'cancelled'\"\n\n      - name: customer_id\n        tests:\n          - not_null\n          - relationships:\n              to: ref('dim_customers')\n              field: customer_id\n              severity: warn\n```\n\n#### Step 4: Implement Data Contracts\n\n```yaml\n# contracts\u002Forders_contract.yaml\ncontract:\n  name: orders_data_contract\n  version: \"1.0.0\"\n  owner: data-team@company.com\n\nschema:\n  type: object\n  properties:\n    order_id:\n      type: string\n      format: uuid\n      description: \"Unique order identifier\"\n    customer_id:\n      type: string\n      not_null: true\n    order_date:\n      type: date\n      not_null: true\n    total_amount:\n      type: decimal\n      precision: 10\n      scale: 2\n      minimum: 0\n    status:\n      type: string\n      enum: [\"pending\", \"confirmed\", \"shipped\", \"delivered\", \"cancelled\"]\n\nsla:\n  freshness:\n    max_delay_hours: 1\n  completeness:\n    min_percentage: 99.9\n  accuracy:\n    duplicate_tolerance: 0.01\n\nconsumers:\n  - name: analytics-team\n    usage: \"Daily reporting dashboards\"\n  - name: ml-team\n    usage: \"Churn prediction model\"\n```\n\n#### Step 5: Set Up Quality Monitoring Dashboard\n\n```python\n# monitoring\u002Fquality_dashboard.py\nfrom datetime import datetime, timedelta\nimport pandas as pd\n\ndef generate_quality_report(connection, table_name: str) -> dict:\n    \"\"\"Generate comprehensive data quality report.\"\"\"\n\n    report = {\n        \"table\": table_name,\n        \"timestamp\": datetime.now().isoformat(),\n        \"checks\": {}\n    }\n\n    # Row count check\n    row_count = connection.execute(\n        f\"SELECT COUNT(*) FROM {table_name}\"\n    ).fetchone()[0]\n    report[\"checks\"][\"row_count\"] = {\n        \"value\": row_count,\n        \"status\": \"pass\" if row_count > 0 else \"fail\"\n    }\n\n    # Freshness check\n    max_date = connection.execute(\n        f\"SELECT MAX(created_at) FROM {table_name}\"\n    ).fetchone()[0]\n    hours_old = (datetime.now() - max_date).total_seconds() \u002F 3600\n    report[\"checks\"][\"freshness\"] = {\n        \"max_timestamp\": max_date.isoformat(),\n        \"hours_old\": round(hours_old, 2),\n        \"status\": \"pass\" if hours_old \u003C 24 else \"fail\"\n    }\n\n    # Null rate check\n    null_query = f\"\"\"\n    SELECT\n        SUM(CASE WHEN order_id IS NULL THEN 1 ELSE 0 END) as null_order_id,\n        SUM(CASE WHEN customer_id IS NULL THEN 1 ELSE 0 END) as null_customer_id,\n        COUNT(*) as total\n    FROM {table_name}\n    \"\"\"\n    null_result = connection.execute(null_query).fetchone()\n    report[\"checks\"][\"null_rates\"] = {\n        \"order_id\": null_result[0] \u002F null_result[2] if null_result[2] > 0 else 0,\n        \"customer_id\": null_result[1] \u002F null_result[2] if null_result[2] > 0 else 0,\n        \"status\": \"pass\" if null_result[0] == 0 and null_result[1] == 0 else \"fail\"\n    }\n\n    # Duplicate check\n    dup_query = f\"\"\"\n    SELECT COUNT(*) - COUNT(DISTINCT order_id) as duplicates\n    FROM {table_name}\n    \"\"\"\n    duplicates = connection.execute(dup_query).fetchone()[0]\n    report[\"checks\"][\"duplicates\"] = {\n        \"count\": duplicates,\n        \"status\": \"pass\" if duplicates == 0 else \"fail\"\n    }\n\n    # Overall status\n    all_passed = all(\n        check[\"status\"] == \"pass\"\n        for check in report[\"checks\"].values()\n    )\n    report[\"overall_status\"] = \"pass\" if all_passed else \"fail\"\n\n    return report\n```\n\n---\n\n## Architecture Decision Framework\n\nUse this framework to choose the right approach for your data pipeline.\n\n### Batch vs Streaming\n\n| Criteria | Batch | Streaming |\n|----------|-------|-----------|\n| **Latency requirement** | Hours to days | Seconds to minutes |\n| **Data volume** | Large historical datasets | Continuous event streams |\n| **Processing complexity** | Complex transformations, ML | Simple aggregations, filtering |\n| **Cost sensitivity** | More cost-effective | Higher infrastructure cost |\n| **Error handling** | Easier to reprocess | Requires careful design |\n\n**Decision Tree:**\n```\nIs real-time insight required?\n├── Yes → Use streaming\n│   └── Is exactly-once semantics needed?\n│       ├── Yes → Kafka + Flink\u002FSpark Structured Streaming\n│       └── No → Kafka + consumer groups\n└── No → Use batch\n    └── Is data volume > 1TB daily?\n        ├── Yes → Spark\u002FDatabricks\n        └── No → dbt + warehouse compute\n```\n\n### Lambda vs Kappa Architecture\n\n| Aspect | Lambda | Kappa |\n|--------|--------|-------|\n| **Complexity** | Two codebases (batch + stream) | Single codebase |\n| **Maintenance** | Higher (sync batch\u002Fstream logic) | Lower |\n| **Reprocessing** | Native batch layer | Replay from source |\n| **Use case** | ML training + real-time serving | Pure event-driven |\n\n**When to choose Lambda:**\n- Need to train ML models on historical data\n- Complex batch transformations not feasible in streaming\n- Existing batch infrastructure\n\n**When to choose Kappa:**\n- Event-sourced architecture\n- All processing can be expressed as stream operations\n- Starting fresh without legacy systems\n\n### Data Warehouse vs Data Lakehouse\n\n| Feature | Warehouse (Snowflake\u002FBigQuery) | Lakehouse (Delta\u002FIceberg) |\n|---------|-------------------------------|---------------------------|\n| **Best for** | BI, SQL analytics | ML, unstructured data |\n| **Storage cost** | Higher (proprietary format) | Lower (open formats) |\n| **Flexibility** | Schema-on-write | Schema-on-read |\n| **Performance** | Excellent for SQL | Good, improving |\n| **Ecosystem** | Mature BI tools | Growing ML tooling |\n\n---\n\n## Tech Stack\n\n| Category | Technologies |\n|----------|--------------|\n| **Languages** | Python, SQL, Scala |\n| **Orchestration** | Airflow, Prefect, Dagster |\n| **Transformation** | dbt, Spark, Flink |\n| **Streaming** | Kafka, Kinesis, Pub\u002FSub |\n| **Storage** | S3, GCS, Delta Lake, Iceberg |\n| **Warehouses** | Snowflake, BigQuery, Redshift, Databricks |\n| **Quality** | Great Expectations, dbt tests, Monte Carlo |\n| **Monitoring** | Prometheus, Grafana, Datadog |\n\n---\n\n## Reference Documentation\n\n### 1. Data Pipeline Architecture\nSee `references\u002Fdata_pipeline_architecture.md` for:\n- Lambda vs Kappa architecture patterns\n- Batch processing with Spark and Airflow\n- Stream processing with Kafka and Flink\n- Exactly-once semantics implementation\n- Error handling and dead letter queues\n\n### 2. Data Modeling Patterns\nSee `references\u002Fdata_modeling_patterns.md` for:\n- Dimensional modeling (Star\u002FSnowflake)\n- Slowly Changing Dimensions (SCD Types 1-6)\n- Data Vault modeling\n- dbt best practices\n- Partitioning and clustering\n\n### 3. DataOps Best Practices\nSee `references\u002Fdataops_best_practices.md` for:\n- Data testing frameworks\n- Data contracts and schema validation\n- CI\u002FCD for data pipelines\n- Observability and lineage\n- Incident response\n\n---\n\n## Troubleshooting\n\n### Pipeline Failures\n\n**Symptom:** Airflow DAG fails with timeout\n```\nTask exceeded max execution time\n```\n\n**Solution:**\n1. Check resource allocation\n2. Profile slow operations\n3. Add incremental processing\n```python\n# Increase timeout\ndefault_args = {\n    'execution_timeout': timedelta(hours=2),\n}\n\n# Or use incremental loads\nWHERE updated_at > '{{ prev_ds }}'\n```\n\n---\n\n**Symptom:** Spark job OOM\n```\njava.lang.OutOfMemoryError: Java heap space\n```\n\n**Solution:**\n1. Increase executor memory\n2. Reduce partition size\n3. Use disk spill\n```python\nspark.conf.set(\"spark.executor.memory\", \"8g\")\nspark.conf.set(\"spark.sql.shuffle.partitions\", \"200\")\nspark.conf.set(\"spark.memory.fraction\", \"0.8\")\n```\n\n---\n\n**Symptom:** Kafka consumer lag increasing\n```\nConsumer lag: 1000000 messages\n```\n\n**Solution:**\n1. Increase consumer parallelism\n2. Optimize processing logic\n3. Scale consumer group\n```bash\n# Add more partitions\nkafka-topics.sh --alter \\\n  --bootstrap-server localhost:9092 \\\n  --topic user-events \\\n  --partitions 24\n```\n\n---\n\n### Data Quality Issues\n\n**Symptom:** Duplicate records appearing\n```\nExpected unique, found 150 duplicates\n```\n\n**Solution:**\n1. Add deduplication logic\n2. Use merge\u002Fupsert operations\n```sql\n-- dbt incremental with dedup\n{{\n    config(\n        materialized='incremental',\n        unique_key='order_id'\n    )\n}}\n\nSELECT * FROM (\n    SELECT\n        *,\n        ROW_NUMBER() OVER (\n            PARTITION BY order_id\n            ORDER BY updated_at DESC\n        ) as rn\n    FROM {{ source('raw', 'orders') }}\n) WHERE rn = 1\n```\n\n---\n\n**Symptom:** Stale data in tables\n```\nLast update: 3 days ago\n```\n\n**Solution:**\n1. Check upstream pipeline status\n2. Verify source availability\n3. Add freshness monitoring\n```yaml\n# dbt freshness check\nsources:\n  - name: raw\n    freshness:\n      warn_after: {count: 12, period: hour}\n      error_after: {count: 24, period: hour}\n    loaded_at_field: _loaded_at\n```\n\n---\n\n**Symptom:** Schema drift detected\n```\nColumn 'new_field' not in expected schema\n```\n\n**Solution:**\n1. Update data contract\n2. Modify transformations\n3. Communicate with producers\n```python\n# Handle schema evolution\ndf = spark.read.format(\"delta\") \\\n    .option(\"mergeSchema\", \"true\") \\\n    .load(\"\u002Fdata\u002Forders\")\n```\n\n---\n\n### Performance Issues\n\n**Symptom:** Query takes hours\n```\nQuery runtime: 4 hours (expected: 30 minutes)\n```\n\n**Solution:**\n1. Check query plan\n2. Add proper partitioning\n3. Optimize joins\n```sql\n-- Before: Full table scan\nSELECT * FROM orders WHERE order_date = '2024-01-15';\n\n-- After: Partition pruning\n-- Table partitioned by order_date\nSELECT * FROM orders WHERE order_date = '2024-01-15';\n\n-- Add clustering for frequent filters\nALTER TABLE orders CLUSTER BY (customer_id);\n```\n\n---\n\n**Symptom:** dbt model takes too long\n```\nModel fct_orders completed in 45 minutes\n```\n\n**Solution:**\n1. Use incremental materialization\n2. Reduce upstream dependencies\n3. Pre-aggregate where possible\n```sql\n-- Convert to incremental\n{{\n    config(\n        materialized='incremental',\n        unique_key='order_id',\n        on_schema_change='sync_all_columns'\n    )\n}}\n\nSELECT * FROM {{ ref('stg_orders') }}\n{% if is_incremental() %}\nWHERE _loaded_at > (SELECT MAX(_loaded_at) FROM {{ this }})\n{% endif %}\n```\n","# Senior Data Engineer\n\nProduction-grade data engineering skill for building scalable, reliable data systems.\n\n## Table of Contents\n\n1. [Trigger Phrases](#trigger-phrases)\n2. [Quick Start](#quick-start)\n3. [Workflows](#workflows)\n   - [Building a Batch ETL Pipeline](#workflow-1-building-a-batch-etl-pipeline)\n   - [Implementing Real-Time Streaming](#workflow-2-implementing-real-time-streaming)\n   - [Data Quality Framework Setup](#workflow-3-data-quality-framework-setup)\n4. [Architecture Decision Framework](#architecture-decision-framework)\n5. [Tech Stack](#tech-stack)\n6. [Reference Documentation](#reference-documentation)\n7. [Troubleshooting](#troubleshooting)\n\n---\n\n## Trigger Phrases\n\nActivate this skill when you see:\n\n**Pipeline Design:**\n- \"Design a data pipeline for...\"\n- \"Build an ETL\u002FELT process...\"\n- \"How should I ingest data from...\"\n- \"Set up data extraction from...\"\n\n**Architecture:**\n- \"Should I use batch or streaming?\"\n- \"Lambda vs Kappa architecture\"\n- \"How to handle late-arriving data\"\n- \"Design a data lakehouse\"\n\n**Data Modeling:**\n- \"Create a dimensional model...\"\n- \"Star schema vs snowflake\"\n- \"Implement slowly changing dimensions\"\n- \"Design a data vault\"\n\n**Data Quality:**\n- \"Add data validation to...\"\n- \"Set up data quality checks\"\n- \"Monitor data freshness\"\n- \"Implement data contracts\"\n\n**Performance:**\n- \"Optimize this Spark job\"\n- \"Query is running slow\"\n- \"Reduce pipeline execution time\"\n- \"Tune Airflow DAG\"\n\n---\n\n## Quick Start\n\n### Core Tools\n\n```bash\n# Generate pipeline orchestration config\npython scripts\u002Fpipeline_orchestrator.py generate \\\n  --type airflow \\\n  --source postgres \\\n  --destination snowflake \\\n  --schedule \"0 5 * * *\"\n\n# Validate data quality\npython scripts\u002Fdata_quality_validator.py validate \\\n  --input data\u002Fsales.parquet \\\n  --schema schemas\u002Fsales.json \\\n  --checks freshness,completeness,uniqueness\n\n# Optimize ETL performance\npython scripts\u002Fetl_performance_optimizer.py analyze \\\n  --query queries\u002Fdaily_aggregation.sql \\\n  --engine spark \\\n  --recommend\n```\n\n---\n\n## Workflows\n\n### Workflow 1: Building a Batch ETL Pipeline\n\n**Scenario:** Extract data from PostgreSQL, transform with dbt, load to Snowflake.\n\n#### Step 1: Define Source Schema\n\n```sql\n-- Document source tables\nSELECT\n    table_name,\n    column_name,\n    data_type,\n    is_nullable\nFROM information_schema.columns\nWHERE table_schema = 'source_schema'\nORDER BY table_name, ordinal_position;\n```\n\n#### Step 2: Generate Extraction Config\n\n```bash\npython scripts\u002Fpipeline_orchestrator.py generate \\\n  --type airflow \\\n  --source postgres \\\n  --tables orders,customers,products \\\n  --mode incremental \\\n  --watermark updated_at \\\n  --output dags\u002Fextract_source.py\n```\n\n#### Step 3: Create dbt Models\n\n```sql\n-- models\u002Fstaging\u002Fstg_orders.sql\nWITH source AS (\n    SELECT * FROM {{ source('postgres', 'orders') }}\n),\n\nrenamed AS (\n    SELECT\n        order_id,\n        customer_id,\n        order_date,\n        total_amount,\n        status,\n        _extracted_at\n    FROM source\n    WHERE order_date >= DATEADD(day, -3, CURRENT_DATE)\n)\n\nSELECT * FROM renamed\n```\n\n```sql\n-- models\u002Fmarts\u002Ffct_orders.sql\n{{\n    config(\n        materialized='incremental',\n        unique_key='order_id',\n        cluster_by=['order_date']\n    )\n}}\n\nSELECT\n    o.order_id,\n    o.customer_id,\n    c.customer_segment,\n    o.order_date,\n    o.total_amount,\n    o.status\nFROM {{ ref('stg_orders') }} o\nLEFT JOIN {{ ref('dim_customers') }} c\n    ON o.customer_id = c.customer_id\n\n{% if is_incremental() %}\nWHERE o._extracted_at > (SELECT MAX(_extracted_at) FROM {{ this }})\n{% endif %}\n```\n\n#### Step 4: Configure Data Quality Tests\n\n```yaml\n# models\u002Fmarts\u002Fschema.yml\nversion: 2\n\nmodels:\n  - name: fct_orders\n    description: \"Order fact table\"\n    columns:\n      - name: order_id\n        tests:\n          - unique\n          - not_null\n      - name: total_amount\n        tests:\n          - not_null\n          - dbt_utils.accepted_range:\n              min_value: 0\n              max_value: 1000000\n      - name: order_date\n        tests:\n          - not_null\n          - dbt_utils.recency:\n              datepart: day\n              field: order_date\n              interval: 1\n```\n\n#### Step 5: Create Airflow DAG\n\n```python\n# dags\u002Fdaily_etl.py\nfrom airflow import DAG\nfrom airflow.providers.postgres.operators.postgres import PostgresOperator\nfrom airflow.operators.bash import BashOperator\nfrom airflow.utils.dates import days_ago\nfrom datetime import timedelta\n\ndefault_args = {\n    'owner': 'data-team',\n    'depends_on_past': False,\n    'email_on_failure': True,\n    'email': ['data-alerts@company.com'],\n    'retries': 2,\n    'retry_delay': timedelta(minutes=5),\n}\n\nwith DAG(\n    'daily_etl_pipeline',\n    default_args=default_args,\n    description='Daily ETL from PostgreSQL to Snowflake',\n    schedule_interval='0 5 * * *',\n    start_date=days_ago(1),\n    catchup=False,\n    tags=['etl', 'daily'],\n) as dag:\n\n    extract = BashOperator(\n        task_id='extract_source_data',\n        bash_command='python \u002Fopt\u002Fairflow\u002Fscripts\u002Fextract.py --date {{ ds }}',\n    )\n\n    transform = BashOperator(\n        task_id='run_dbt_models',\n        bash_command='cd \u002Fopt\u002Fairflow\u002Fdbt && dbt run --select marts.*',\n    )\n\n    test = BashOperator(\n        task_id='run_dbt_tests',\n        bash_command='cd \u002Fopt\u002Fairflow\u002Fdbt && dbt test --select marts.*',\n    )\n\n    notify = BashOperator(\n        task_id='send_notification',\n        bash_command='python \u002Fopt\u002Fairflow\u002Fscripts\u002Fnotify.py --status success',\n        trigger_rule='all_success',\n    )\n\n    extract >> transform >> test >> notify\n```\n\n#### Step 6: Validate Pipeline\n\n```bash\n# Test locally\ndbt run --select stg_orders fct_orders\ndbt test --select fct_orders\n\n# Validate data quality\npython scripts\u002Fdata_quality_validator.py validate \\\n  --table fct_orders \\\n  --checks all \\\n  --output reports\u002Fquality_report.json\n```\n\n---\n\n### Workflow 2: Implementing Real-Time Streaming\n\n**Scenario:** Stream events from Kafka, process with Flink\u002FSpark Streaming, sink to data lake.\n\n#### Step 1: Define Event Schema\n\n```json\n{\n  \"$schema\": \"http:\u002F\u002Fjson-schema.org\u002Fdraft-07\u002Fschema#\",\n  \"title\": \"UserEvent\",\n  \"type\": \"object\",\n  \"required\": [\"event_id\", \"user_id\", \"event_type\", \"timestamp\"],\n  \"properties\": {\n    \"event_id\": {\"type\": \"string\", \"format\": \"uuid\"},\n    \"user_id\": {\"type\": \"string\"},\n    \"event_type\": {\"type\": \"string\", \"enum\": [\"page_view\", \"click\", \"purchase\"]},\n    \"timestamp\": {\"type\": \"string\", \"format\": \"date-time\"},\n    \"properties\": {\"type\": \"object\"}\n  }\n}\n```\n\n#### Step 2: Create Kafka Topic\n\n```bash\n# Create topic with appropriate partitions\nkafka-topics.sh --create \\\n  --bootstrap-server localhost:9092 \\\n  --topic user-events \\\n  --partitions 12 \\\n  --replication-factor 3 \\\n  --config retention.ms=604800000 \\\n  --config cleanup.policy=delete\n\n# Verify topic\nkafka-topics.sh --describe \\\n  --bootstrap-server localhost:9092 \\\n  --topic user-events\n```\n\n#### Step 3: Implement Spark Streaming Job\n\n```python\n# streaming\u002Fuser_events_processor.py\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.functions import (\n    from_json, col, window, count, avg,\n    to_timestamp, current_timestamp\n)\nfrom pyspark.sql.types import (\n    StructType, StructField, StringType,\n    TimestampType, MapType\n)\n\n# Initialize Spark\nspark = SparkSession.builder \\\n    .appName(\"UserEventsProcessor\") \\\n    .config(\"spark.sql.streaming.checkpointLocation\", \"\u002Fcheckpoints\u002Fuser-events\") \\\n    .config(\"spark.sql.shuffle.partitions\", \"12\") \\\n    .getOrCreate()\n\n# Define schema\nevent_schema = StructType([\n    StructField(\"event_id\", StringType(), False),\n    StructField(\"user_id\", StringType(), False),\n    StructField(\"event_type\", StringType(), False),\n    StructField(\"timestamp\", StringType(), False),\n    StructField(\"properties\", MapType(StringType(), StringType()), True)\n])\n\n# Read from Kafka\nevents_df = spark.readStream \\\n    .format(\"kafka\") \\\n    .option(\"kafka.bootstrap.servers\", \"localhost:9092\") \\\n    .option(\"subscribe\", \"user-events\") \\\n    .option(\"startingOffsets\", \"latest\") \\\n    .option(\"failOnDataLoss\", \"false\") \\\n    .load()\n\n# Parse JSON\nparsed_df = events_df \\\n    .select(from_json(col(\"value\").cast(\"string\"), event_schema).alias(\"data\")) \\\n    .select(\"data.*\") \\\n    .withColumn(\"event_timestamp\", to_timestamp(col(\"timestamp\")))\n\n# Windowed aggregation\naggregated_df = parsed_df \\\n    .withWatermark(\"event_timestamp\", \"10 minutes\") \\\n    .groupBy(\n        window(col(\"event_timestamp\"), \"5 minutes\"),\n        col(\"event_type\")\n    ) \\\n    .agg(\n        count(\"*\").alias(\"event_count\"),\n        approx_count_distinct(\"user_id\").alias(\"unique_users\")\n    )\n\n# Write to Delta Lake\nquery = aggregated_df.writeStream \\\n    .format(\"delta\") \\\n    .outputMode(\"append\") \\\n    .option(\"checkpointLocation\", \"\u002Fcheckpoints\u002Fuser-events-aggregated\") \\\n    .option(\"path\", \"\u002Fdata\u002Flake\u002Fuser_events_aggregated\") \\\n    .trigger(processingTime=\"1 minute\") \\\n    .start()\n\nquery.awaitTermination()\n```\n\n#### Step 4: Handle Late Data and Errors\n\n```python\n# Dead letter queue for failed records\nfrom pyspark.sql.functions import current_timestamp, lit\n\ndef process_with_error_handling(batch_df, batch_id):\n    try:\n        # Attempt processing\n        valid_df = batch_df.filter(col(\"event_id\").isNotNull())\n        invalid_df = batch_df.filter(col(\"event_id\").isNull())\n\n        # Write valid records\n        valid_df.write \\\n            .format(\"delta\") \\\n            .mode(\"append\") \\\n            .save(\"\u002Fdata\u002Flake\u002Fuser_events\")\n\n        # Write invalid to DLQ\n        if invalid_df.count() > 0:\n            invalid_df \\\n                .withColumn(\"error_timestamp\", current_timestamp()) \\\n                .withColumn(\"error_reason\", lit(\"missing_event_id\")) \\\n                .write \\\n                .format(\"delta\") \\\n                .mode(\"append\") \\\n                .save(\"\u002Fdata\u002Flake\u002Fdlq\u002Fuser_events\")\n\n    except Exception as e:\n        # Log error, alert, continue\n        logger.error(f\"Batch {batch_id} failed: {e}\")\n        raise\n\n# Use foreachBatch for custom processing\nquery = parsed_df.writeStream \\\n    .foreachBatch(process_with_error_handling) \\\n    .option(\"checkpointLocation\", \"\u002Fcheckpoints\u002Fuser-events\") \\\n    .start()\n```\n\n#### Step 5: Monitor Stream Health\n\n```python\n# monitoring\u002Fstream_metrics.py\nfrom prometheus_client import Gauge, Counter, start_http_server\n\n# Define metrics\nRECORDS_PROCESSED = Counter(\n    'stream_records_processed_total',\n    'Total records processed',\n    ['stream_name', 'status']\n)\n\nPROCESSING_LAG = Gauge(\n    'stream_processing_lag_seconds',\n    'Current processing lag',\n    ['stream_name']\n)\n\nBATCH_DURATION = Gauge(\n    'stream_batch_duration_seconds',\n    'Last batch processing duration',\n    ['stream_name']\n)\n\ndef emit_metrics(query):\n    \"\"\"Emit Prometheus metrics from streaming query.\"\"\"\n    progress = query.lastProgress\n    if progress:\n        RECORDS_PROCESSED.labels(\n            stream_name='user-events',\n            status='success'\n        ).inc(progress['numInputRows'])\n\n        if progress['sources']:\n            # Calculate lag from latest offset\n            for source in progress['sources']:\n                end_offset = source.get('endOffset', {})\n                # Parse Kafka offsets and calculate lag\n```\n\n---\n\n### Workflow 3: Data Quality Framework Setup\n\n**Scenario:** Implement comprehensive data quality monitoring with Great Expectations.\n\n#### Step 1: Initialize Great Expectations\n\n```bash\n# Install and initialize\npip install great_expectations\n\ngreat_expectations init\n\n# Connect to data source\ngreat_expectations datasource new\n```\n\n#### Step 2: Create Expectation Suite\n\n```python\n# expectations\u002Forders_suite.py\nimport great_expectations as gx\n\ncontext = gx.get_context()\n\n# Create expectation suite\nsuite = context.add_expectation_suite(\"orders_quality_suite\")\n\n# Add expectations\nvalidator = context.get_validator(\n    batch_request={\n        \"datasource_name\": \"warehouse\",\n        \"data_asset_name\": \"orders\",\n    },\n    expectation_suite_name=\"orders_quality_suite\"\n)\n\n# Schema expectations\nvalidator.expect_table_columns_to_match_ordered_list(\n    column_list=[\n        \"order_id\", \"customer_id\", \"order_date\",\n        \"total_amount\", \"status\", \"created_at\"\n    ]\n)\n\n# Completeness expectations\nvalidator.expect_column_values_to_not_be_null(\"order_id\")\nvalidator.expect_column_values_to_not_be_null(\"customer_id\")\nvalidator.expect_column_values_to_not_be_null(\"order_date\")\n\n# Uniqueness expectations\nvalidator.expect_column_values_to_be_unique(\"order_id\")\n\n# Range expectations\nvalidator.expect_column_values_to_be_between(\n    \"total_amount\",\n    min_value=0,\n    max_value=1000000\n)\n\n# Categorical expectations\nvalidator.expect_column_values_to_be_in_set(\n    \"status\",\n    [\"pending\", \"confirmed\", \"shipped\", \"delivered\", \"cancelled\"]\n)\n\n# Freshness expectation\nvalidator.expect_column_max_to_be_between(\n    \"order_date\",\n    min_value={\"$PARAMETER\": \"now - timedelta(days=1)\"},\n    max_value={\"$PARAMETER\": \"now\"}\n)\n\n# Referential integrity\nvalidator.expect_column_values_to_be_in_set(\n    \"customer_id\",\n    value_set={\"$PARAMETER\": \"valid_customer_ids\"}\n)\n\nvalidator.save_expectation_suite(discard_failed_expectations=False)\n```\n\n#### Step 3: Create Data Quality Checks with dbt\n\n```yaml\n# models\u002Fmarts\u002Fschema.yml\nversion: 2\n\nmodels:\n  - name: fct_orders\n    description: \"Order fact table with data quality checks\"\n\n    tests:\n      # Row count check\n      - dbt_utils.equal_rowcount:\n          compare_model: ref('stg_orders')\n\n      # Freshness check\n      - dbt_utils.recency:\n          datepart: hour\n          field: created_at\n          interval: 24\n\n    columns:\n      - name: order_id\n        description: \"Unique order identifier\"\n        tests:\n          - unique\n          - not_null\n          - relationships:\n              to: ref('dim_orders')\n              field: order_id\n\n      - name: total_amount\n        tests:\n          - not_null\n          - dbt_utils.accepted_range:\n              min_value: 0\n              max_value: 1000000\n              inclusive: true\n          - dbt_expectations.expect_column_values_to_be_between:\n              min_value: 0\n              row_condition: \"status != 'cancelled'\"\n\n      - name: customer_id\n        tests:\n          - not_null\n          - relationships:\n              to: ref('dim_customers')\n              field: customer_id\n              severity: warn\n```\n\n#### Step 4: Implement Data Contracts\n\n```yaml\n# contracts\u002Forders_contract.yaml\ncontract:\n  name: orders_data_contract\n  version: \"1.0.0\"\n  owner: data-team@company.com\n\nschema:\n  type: object\n  properties:\n    order_id:\n      type: string\n      format: uuid\n      description: \"Unique order identifier\"\n    customer_id:\n      type: string\n      not_null: true\n    order_date:\n      type: date\n      not_null: true\n    total_amount:\n      type: decimal\n      precision: 10\n      scale: 2\n      minimum: 0\n    status:\n      type: string\n      enum: [\"pending\", \"confirmed\", \"shipped\", \"delivered\", \"cancelled\"]\n\nsla:\n  freshness:\n    max_delay_hours: 1\n  completeness:\n    min_percentage: 99.9\n  accuracy:\n    duplicate_tolerance: 0.01\n\nconsumers:\n  - name: analytics-team\n    usage: \"Daily reporting dashboards\"\n  - name: ml-team\n    usage: \"Churn prediction model\"\n```\n\n#### Step 5: Set Up Quality Monitoring Dashboard\n\n```python\n# monitoring\u002Fquality_dashboard.py\nfrom datetime import datetime, timedelta\nimport pandas as pd\n\ndef generate_quality_report(connection, table_name: str) -> dict:\n    \"\"\"Generate comprehensive data quality report.\"\"\"\n\n    report = {\n        \"table\": table_name,\n        \"timestamp\": datetime.now().isoformat(),\n        \"checks\": {}\n    }\n\n    # Row count check\n    row_count = connection.execute(\n        f\"SELECT COUNT(*) FROM {table_name}\"\n    ).fetchone()[0]\n    report[\"checks\"][\"row_count\"] = {\n        \"value\": row_count,\n        \"status\": \"pass\" if row_count > 0 else \"fail\"\n    }\n\n    # Freshness check\n    max_date = connection.execute(\n        f\"SELECT MAX(created_at) FROM {table_name}\"\n    ).fetchone()[0]\n    hours_old = (datetime.now() - max_date).total_seconds() \u002F 3600\n    report[\"checks\"][\"freshness\"] = {\n        \"max_timestamp\": max_date.isoformat(),\n        \"hours_old\": round(hours_old, 2),\n        \"status\": \"pass\" if hours_old \u003C 24 else \"fail\"\n    }\n\n    # Null rate check\n    null_query = f\"\"\"\n    SELECT\n        SUM(CASE WHEN order_id IS NULL THEN 1 ELSE 0 END) as null_order_id,\n        SUM(CASE WHEN customer_id IS NULL THEN 1 ELSE 0 END) as null_customer_id,\n        COUNT(*) as total\n    FROM {table_name}\n    \"\"\"\n    null_result = connection.execute(null_query).fetchone()\n    report[\"checks\"][\"null_rates\"] = {\n        \"order_id\": null_result[0] \u002F null_result[2] if null_result[2] > 0 else 0,\n        \"customer_id\": null_result[1] \u002F null_result[2] if null_result[2] > 0 else 0,\n        \"status\": \"pass\" if null_result[0] == 0 and null_result[1] == 0 else \"fail\"\n    }\n\n    # Duplicate check\n    dup_query = f\"\"\"\n    SELECT COUNT(*) - COUNT(DISTINCT order_id) as duplicates\n    FROM {table_name}\n    \"\"\"\n    duplicates = connection.execute(dup_query).fetchone()[0]\n    report[\"checks\"][\"duplicates\"] = {\n        \"count\": duplicates,\n        \"status\": \"pass\" if duplicates == 0 else \"fail\"\n    }\n\n    # Overall status\n    all_passed = all(\n        check[\"status\"] == \"pass\"\n        for check in report[\"checks\"].values()\n    )\n    report[\"overall_status\"] = \"pass\" if all_passed else \"fail\"\n\n    return report\n```\n\n---\n\n## Architecture Decision Framework\n\nUse this framework to choose the right approach for your data pipeline.\n\n### Batch vs Streaming\n\n| Criteria | Batch | Streaming |\n|----------|-------|-----------|\n| **Latency requirement** | Hours to days | Seconds to minutes |\n| **Data volume** | Large historical datasets | Continuous event streams |\n| **Processing complexity** | Complex transformations, ML | Simple aggregations, filtering |\n| **Cost sensitivity** | More cost-effective | Higher infrastructure cost |\n| **Error handling** | Easier to reprocess | Requires careful design |\n\n**Decision Tree:**\n```\nIs real-time insight required?\n├── Yes → Use streaming\n│   └── Is exactly-once semantics needed?\n│       ├── Yes → Kafka + Flink\u002FSpark Structured Streaming\n│       └── No → Kafka + consumer groups\n└── No → Use batch\n    └── Is data volume > 1TB daily?\n        ├── Yes → Spark\u002FDatabricks\n        └── No → dbt + warehouse compute\n```\n\n### Lambda vs Kappa Architecture\n\n| Aspect | Lambda | Kappa |\n|--------|--------|-------|\n| **Complexity** | Two codebases (batch + stream) | Single codebase |\n| **Maintenance** | Higher (sync batch\u002Fstream logic) | Lower |\n| **Reprocessing** | Native batch layer | Replay from source |\n| **Use case** | ML training + real-time serving | Pure event-driven |\n\n**When to choose Lambda:**\n- Need to train ML models on historical data\n- Complex batch transformations not feasible in streaming\n- Existing batch infrastructure\n\n**When to choose Kappa:**\n- Event-sourced architecture\n- All processing can be expressed as stream operations\n- Starting fresh without legacy systems\n\n### Data Warehouse vs Data Lakehouse\n\n| Feature | Warehouse (Snowflake\u002FBigQuery) | Lakehouse (Delta\u002FIceberg) |\n|---------|-------------------------------|---------------------------|\n| **Best for** | BI, SQL analytics | ML, unstructured data |\n| **Storage cost** | Higher (proprietary format) | Lower (open formats) |\n| **Flexibility** | Schema-on-write | Schema-on-read |\n| **Performance** | Excellent for SQL | Good, improving |\n| **Ecosystem** | Mature BI tools | Growing ML tooling |\n\n---\n\n## Tech Stack\n\n| Category | Technologies |\n|----------|--------------|\n| **Languages** | Python, SQL, Scala |\n| **Orchestration** | Airflow, Prefect, Dagster |\n| **Transformation** | dbt, Spark, Flink |\n| **Streaming** | Kafka, Kinesis, Pub\u002FSub |\n| **Storage** | S3, GCS, Delta Lake, Iceberg |\n| **Warehouses** | Snowflake, BigQuery, Redshift, Databricks |\n| **Quality** | Great Expectations, dbt tests, Monte Carlo |\n| **Monitoring** | Prometheus, Grafana, Datadog |\n\n---\n\n## Reference Documentation\n\n### 1. Data Pipeline Architecture\nSee `references\u002Fdata_pipeline_architecture.md` for:\n- Lambda vs Kappa architecture patterns\n- Batch processing with Spark and Airflow\n- Stream processing with Kafka and Flink\n- Exactly-once semantics implementation\n- Error handling and dead letter queues\n\n### 2. Data Modeling Patterns\nSee `references\u002Fdata_modeling_patterns.md` for:\n- Dimensional modeling (Star\u002FSnowflake)\n- Slowly Changing Dimensions (SCD Types 1-6)\n- Data Vault modeling\n- dbt best practices\n- Partitioning and clustering\n\n### 3. DataOps Best Practices\nSee `references\u002Fdataops_best_practices.md` for:\n- Data testing frameworks\n- Data contracts and schema validation\n- CI\u002FCD for data pipelines\n- Observability and lineage\n- Incident response\n\n---\n\n## Troubleshooting\n\n### Pipeline Failures\n\n**Symptom:** Airflow DAG fails with timeout\n```\nTask exceeded max execution time\n```\n\n**Solution:**\n1. Check resource allocation\n2. Profile slow operations\n3. Add incremental processing\n```python\n# Increase timeout\ndefault_args = {\n    'execution_timeout': timedelta(hours=2),\n}\n\n# Or use incremental loads\nWHERE updated_at > '{{ prev_ds }}'\n```\n\n---\n\n**Symptom:** Spark job OOM\n```\njava.lang.OutOfMemoryError: Java heap space\n```\n\n**Solution:**\n1. Increase executor memory\n2. Reduce partition size\n3. Use disk spill\n```python\nspark.conf.set(\"spark.executor.memory\", \"8g\")\nspark.conf.set(\"spark.sql.shuffle.partitions\", \"200\")\nspark.conf.set(\"spark.memory.fraction\", \"0.8\")\n```\n\n---\n\n**Symptom:** Kafka consumer lag increasing\n```\nConsumer lag: 1000000 messages\n```\n\n**Solution:**\n1. Increase consumer parallelism\n2. Optimize processing logic\n3. Scale consumer group\n```bash\n# Add more partitions\nkafka-topics.sh --alter \\\n  --bootstrap-server localhost:9092 \\\n  --topic user-events \\\n  --partitions 24\n```\n\n---\n\n### Data Quality Issues\n\n**Symptom:** Duplicate records appearing\n```\nExpected unique, found 150 duplicates\n```\n\n**Solution:**\n1. Add deduplication logic\n2. Use merge\u002Fupsert operations\n```sql\n-- dbt incremental with dedup\n{{\n    config(\n        materialized='incremental',\n        unique_key='order_id'\n    )\n}}\n\nSELECT * FROM (\n    SELECT\n        *,\n        ROW_NUMBER() OVER (\n            PARTITION BY order_id\n            ORDER BY updated_at DESC\n        ) as rn\n    FROM {{ source('raw', 'orders') }}\n) WHERE rn = 1\n```\n\n---\n\n**Symptom:** Stale data in tables\n```\nLast update: 3 days ago\n```\n\n**Solution:**\n1. Check upstream pipeline status\n2. Verify source availability\n3. Add freshness monitoring\n```yaml\n# dbt freshness check\nsources:\n  - name: raw\n    freshness:\n      warn_after: {count: 12, period: hour}\n      error_after: {count: 24, period: hour}\n    loaded_at_field: _loaded_at\n```\n\n---\n\n**Symptom:** Schema drift detected\n```\nColumn 'new_field' not in expected schema\n```\n\n**Solution:**\n1. Update data contract\n2. Modify transformations\n3. Communicate with producers\n```python\n# Handle schema evolution\ndf = spark.read.format(\"delta\") \\\n    .option(\"mergeSchema\", \"true\") \\\n    .load(\"\u002Fdata\u002Forders\")\n```\n\n---\n\n### Performance Issues\n\n**Symptom:** Query takes hours\n```\nQuery runtime: 4 hours (expected: 30 minutes)\n```\n\n**Solution:**\n1. Check query plan\n2. Add proper partitioning\n3. Optimize joins\n```sql\n-- Before: Full table scan\nSELECT * FROM orders WHERE order_date = '2024-01-15';\n\n-- After: Partition pruning\n-- Table partitioned by order_date\nSELECT * FROM orders WHERE order_date = '2024-01-15';\n\n-- Add clustering for frequent filters\nALTER TABLE orders CLUSTER BY (customer_id);\n```\n\n---\n\n**Symptom:** dbt model takes too long\n```\nModel fct_orders completed in 45 minutes\n```\n\n**Solution:**\n1. Use incremental materialization\n2. Reduce upstream dependencies\n3. Pre-aggregate where possible\n```sql\n-- Convert to incremental\n{{\n    config(\n        materialized='incremental',\n        unique_key='order_id',\n        on_schema_change='sync_all_columns'\n    )\n}}\n\nSELECT * FROM {{ ref('stg_orders') }}\n{% if is_incremental() %}\nWHERE _loaded_at > (SELECT MAX(_loaded_at) FROM {{ this }})\n{% endif %}\n```\n",{"name":5,"description":7},"main","e7e336ac628c3e3bd771aff1aaa469c93778f94a","skills\u002Falirezarezvani\u002Fsenior-data-engineer",0,"https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills","https:\u002F\u002Fgithub.com\u002Fkbarbel640-del","2026-08-23 22:39:37",{"source":8,"skillName":5,"commands":56,"prompt":66,"zipAvailable":67},[57,60,63],{"tool":58,"command":59},"npx","npx skills add kbarbel640-del\u002Fskills --skill senior-data-engineer",{"tool":61,"command":62},"bunx","bunx skills add kbarbel640-del\u002Fskills --skill senior-data-engineer",{"tool":64,"command":65},"pnpm","pnpm dlx skills add kbarbel640-del\u002Fskills --skill senior-data-engineer","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Falirezarezvani\u002Fsenior-data-engineer\u002FSKILL.md",true,[69,75,81,85,89,93,97,101,105,109,113,117,121,125,129,135,139,143,147,151,155,159,163,167],{"path":70,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":73,"sourceHtmlUrl":74},"_meta.json","REFERENCE","参考文档",300,"https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Falirezarezvani\u002Fsenior-data-engineer\u002F_meta.json",{"path":76,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":79,"sourceHtmlUrl":80},"assets\u002Fdataset_manifest.example.json","ASSET","资产文件",874,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Fassets\u002Fdataset_manifest.example.json",{"path":82,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":83,"sourceHtmlUrl":84},"assets\u002Fresult.example.json",448,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Fassets\u002Fresult.example.json",{"path":86,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":87,"sourceHtmlUrl":88},"assets\u002Frun_config.example.json",1227,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Fassets\u002Frun_config.example.json",{"path":90,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":91,"sourceHtmlUrl":92},"assets\u002Ftask_config.example.yaml",1325,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Fassets\u002Ftask_config.example.yaml",{"path":94,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":95,"sourceHtmlUrl":96},"references\u002Fconfiguration.md",5138,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Freferences\u002Fconfiguration.md",{"path":98,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":99,"sourceHtmlUrl":100},"references\u002Fdata_modeling_patterns.md",26737,"https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Falirezarezvani\u002Fsenior-data-engineer\u002Freferences\u002Fdata_modeling_patterns.md",{"path":102,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":103,"sourceHtmlUrl":104},"references\u002Fdata_pipeline_architecture.md",34584,"https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Falirezarezvani\u002Fsenior-data-engineer\u002Freferences\u002Fdata_pipeline_architecture.md",{"path":106,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":107,"sourceHtmlUrl":108},"references\u002Fdataops_best_practices.md",43421,"https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Falirezarezvani\u002Fsenior-data-engineer\u002Freferences\u002Fdataops_best_practices.md",{"path":110,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":111,"sourceHtmlUrl":112},"references\u002Fdatasets.md",6214,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Freferences\u002Fdatasets.md",{"path":114,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":115,"sourceHtmlUrl":116},"references\u002Fevaluation.md",5277,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Freferences\u002Fevaluation.md",{"path":118,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":119,"sourceHtmlUrl":120},"references\u002Fsecurity.md",7430,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Freferences\u002Fsecurity.md",{"path":122,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":123,"sourceHtmlUrl":124},"references\u002Fsources.md",7530,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Freferences\u002Fsources.md",{"path":126,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":127,"sourceHtmlUrl":128},"references\u002Fupstream.md",7998,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Freferences\u002Fupstream.md",{"path":130,"resourceType":131,"resourceTypeDesc":132,"sizeBytes":133,"sourceHtmlUrl":134},"scripts\u002F__init__.py","SCRIPT","脚本",64,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Fscripts\u002F__init__.py",{"path":136,"resourceType":131,"resourceTypeDesc":132,"sizeBytes":137,"sourceHtmlUrl":138},"scripts\u002F_common.py",46561,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Fscripts\u002F_common.py",{"path":140,"resourceType":131,"resourceTypeDesc":132,"sizeBytes":141,"sourceHtmlUrl":142},"scripts\u002Faudit_dataset.py",13848,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Fscripts\u002Faudit_dataset.py",{"path":144,"resourceType":131,"resourceTypeDesc":132,"sizeBytes":145,"sourceHtmlUrl":146},"scripts\u002Fdata_quality_validator.py",60660,"https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Falirezarezvani\u002Fsenior-data-engineer\u002Fscripts\u002Fdata_quality_validator.py",{"path":148,"resourceType":131,"resourceTypeDesc":132,"sizeBytes":149,"sourceHtmlUrl":150},"scripts\u002Fetl_performance_optimizer.py",66066,"https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Falirezarezvani\u002Fsenior-data-engineer\u002Fscripts\u002Fetl_performance_optimizer.py",{"path":152,"resourceType":131,"resourceTypeDesc":132,"sizeBytes":153,"sourceHtmlUrl":154},"scripts\u002Fevaluate_local.py",8148,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Fscripts\u002Fevaluate_local.py",{"path":156,"resourceType":131,"resourceTypeDesc":132,"sizeBytes":157,"sourceHtmlUrl":158},"scripts\u002Finspect_outputs.py",5908,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Fscripts\u002Finspect_outputs.py",{"path":160,"resourceType":131,"resourceTypeDesc":132,"sizeBytes":161,"sourceHtmlUrl":162},"scripts\u002Fpipeline_orchestrator.py",28587,"https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Falirezarezvani\u002Fsenior-data-engineer\u002Fscripts\u002Fpipeline_orchestrator.py",{"path":164,"resourceType":131,"resourceTypeDesc":132,"sizeBytes":165,"sourceHtmlUrl":166},"scripts\u002Fplan_run.py",8582,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Fscripts\u002Fplan_run.py",{"path":168,"resourceType":131,"resourceTypeDesc":132,"sizeBytes":169,"sourceHtmlUrl":170},"scripts\u002Fvalidate_config.py",6054,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Fhypogenic\u002Fscripts\u002Fvalidate_config.py",[172,201,230,251,257,277],{"skillNo":173,"name":174,"nameZh":6,"description":175,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":176,"categories":177,"keywords":181,"publishedAt":43},"01M6G8J55967D6281ACMKAKTEQ","crowd-prompting","A marketplace where AI agents improve prompts, system instructions, tool descriptions, and other text-based content with domain expertise from real-world operations — and earn tokens for valuable contributions.","https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fzzadrian\u002Fcrowd-prompting\u002FSKILL.md",[178],{"categoryNo":179,"categoryName":180,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000036","内容创作者",[182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200],"crowd prompting","prompting","api","with","tokens","domain","world","prompts","operations","instructions","text","tool","system","improve","productivity","marketplace","https","content","ai agents",{"skillNo":202,"name":203,"nameZh":6,"description":204,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":205,"categories":206,"keywords":212,"publishedAt":43},"01M6G8J0PKPAYVET6BWE8W4C26","dilemm-ai","Compete in DilemmAI, the prisoner's dilemma AI arena at dilemm.ai. Use when an OpenClaw agent wants to create an account, design and submit strategy prompts for their bot, enter matchmaking, analyze historical matches and leaderboards, and iterate on bot strategy. Requires headless browser automation (no API available yet).","https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fzyori\u002Fdilemm-ai\u002FSKILL.md",[207],{"categoryNo":208,"categoryName":209,"taxonomyType":210,"taxonomyTypeDesc":211},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[213,184,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,189,229],"dilemm ai","arena","analyze","browser","openclaw agent","create","automation","browser automation","bot","openclaw","design","strategy","headless","compete","submit","matchmaking","use",{"skillNo":231,"name":232,"nameZh":6,"description":233,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":234,"categories":235,"keywords":239,"publishedAt":43},"01M6G8HWAF3K9SW7NP3Z83CPJ6","arxiv-paper-reviews","Interact with arXiv Crawler API to fetch papers, read reviews, and submit comments. Use when working with arXiv papers, fetching paper lists by date\u002Fcategory\u002Finterest, viewing paper details with comments, or submitting paper reviews via API at http:\u002F\u002F150.158.152.82:8000.","https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fzxrys\u002Fweak-accept\u002FSKILL.md",[236],{"categoryNo":237,"categoryName":238,"taxonomyType":210,"taxonomyTypeDesc":211},"01JBSEEDCAT000000000000021","资料检索与研究",[240,241,242,243,244,245,246,185,247,227,229,248,184,249,250],"arxiv paper reviews","reviews","arxiv","paper","crawler","papers","comments","fetch","read","http","via",{"skillNo":252,"name":232,"nameZh":6,"description":233,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":253,"categories":254,"keywords":256,"publishedAt":43},"01M6G8HQJ7N7WMY6YV7D7SSJ16","https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fzxrys\u002Farxiv-paper-reviews\u002FSKILL.md",[255],{"categoryNo":237,"categoryName":238,"taxonomyType":210,"taxonomyTypeDesc":211},[243,242,241,240,250,245,246,185,244,227,229,247,248,184,249],{"skillNo":258,"name":259,"nameZh":6,"description":260,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":261,"categories":262,"keywords":263,"publishedAt":43},"01M6G8HM95Y64JNVFC2HN4E2BG","git-summary","Get a quick summary of the current Git repository including status, recent commits, branches, and contributors.","https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fzweack\u002Fgit-summary\u002FSKILL.md",[],[264,265,266,267,268,269,270,271,272,273,274,275,276],"git","summary","git summary","commits","repository","win32","status","linux","contributors","true","quick","get","branches",{"skillNo":278,"name":279,"nameZh":6,"description":280,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":281,"categories":282,"keywords":287,"publishedAt":43},"01M6G8HGEGF05CNEJCGVYFSC4R","agent-constitution","Interact with AgentConstitution governance contracts on Base Sepolia. Check compliance, read rules, log actions, query governance state.","https:\u002F\u002Fgithub.com\u002Fkbarbel640-del\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fztsalexey\u002Fagent-constitution\u002FSKILL.md",[283,284],{"categoryNo":208,"categoryName":209,"taxonomyType":210,"taxonomyTypeDesc":211},{"categoryNo":285,"categoryName":286,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000037","律师与法务",[288,289,290,291,185,292,293,48,294,248,295,296,297,298,299,198,300,301,302],"agent constitution","constitution","contracts","tree","query","base","log","check","Rules","projects","compliance","actions","state","governance","github"]