[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BWB1NEQ9BDFWY8P9BP8HEW":3,"skill-same:01M7BWB1NEQ9BDFWY8P9BP8HEW":47},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":29,"ownerHtmlUrl":30,"ownerBlogUrl":6,"ownerDescription":31,"updatedAt":32,"install":33,"resources":46},"01M7BWB1NEQ9BDFWY8P9BP8HEW","data-quality-frameworks",null,"Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.","YYC-Cube\u002FYYC3-Knowledge-Base","YYC-Cube","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F235006543?v=4","01M79J2PH4SWD2X0825X6PN748","yyc3","01M79J2PHVK3F6ZBYT8K2SCZCW",1,"MIT","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Plugin-插件协议\u002FYYC3-Plugin-插件协议-plugins-claude\u002Fdata-engineering\u002Fskills\u002Fdata-quality-frameworks\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: data-quality-frameworks\ndescription: Implement data quality validation with Great Expectations, dbt tests, and data contracts. Use when building data quality pipelines, implementing validation rules, or establishing data contracts.\n---\n\n# Data Quality Frameworks\n\nProduction patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.\n\n## When to Use This Skill\n\n- Implementing data quality checks in pipelines\n- Setting up Great Expectations validation\n- Building comprehensive dbt test suites\n- Establishing data contracts between teams\n- Monitoring data quality metrics\n- Automating data validation in CI\u002FCD\n\n## Core Concepts\n\n### 1. Data Quality Dimensions\n\n| Dimension        | Description              | Example Check                                      |\n| ---------------- | ------------------------ | -------------------------------------------------- |\n| **Completeness** | No missing values        | `expect_column_values_to_not_be_null`              |\n| **Uniqueness**   | No duplicates            | `expect_column_values_to_be_unique`                |\n| **Validity**     | Values in expected range | `expect_column_values_to_be_in_set`                |\n| **Accuracy**     | Data matches reality     | Cross-reference validation                         |\n| **Consistency**  | No contradictions        | `expect_column_pair_values_A_to_be_greater_than_B` |\n| **Timeliness**   | Data is recent           | `expect_column_max_to_be_between`                  |\n\n### 2. Testing Pyramid for Data\n\n```\n          \u002F\\\n         \u002F  \\     Integration Tests (cross-table)\n        \u002F────\\\n       \u002F      \\   Unit Tests (single column)\n      \u002F────────\\\n     \u002F          \\ Schema Tests (structure)\n    \u002F────────────\\\n```\n\n## Quick Start\n\n### Great Expectations Setup\n\n```bash\n# Install\npip install great_expectations\n\n# Initialize project\ngreat_expectations init\n\n# Create datasource\ngreat_expectations datasource new\n```\n\n```python\n# great_expectations\u002Fcheckpoints\u002Fdaily_validation.yml\nimport great_expectations as gx\n\n# Create context\ncontext = gx.get_context()\n\n# Create expectation suite\nsuite = context.add_expectation_suite(\"orders_suite\")\n\n# Add expectations\nsuite.add_expectation(\n    gx.expectations.ExpectColumnValuesToNotBeNull(column=\"order_id\")\n)\nsuite.add_expectation(\n    gx.expectations.ExpectColumnValuesToBeUnique(column=\"order_id\")\n)\n\n# Validate\nresults = context.run_checkpoint(checkpoint_name=\"daily_orders\")\n```\n\n## Patterns\n\n### Pattern 1: Great Expectations Suite\n\n```python\n# expectations\u002Forders_suite.py\nimport great_expectations as gx\nfrom great_expectations.core import ExpectationSuite\nfrom great_expectations.core.expectation_configuration import ExpectationConfiguration\n\ndef build_orders_suite() -> ExpectationSuite:\n    \"\"\"Build comprehensive orders expectation suite\"\"\"\n\n    suite = ExpectationSuite(expectation_suite_name=\"orders_suite\")\n\n    # Schema expectations\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_table_columns_to_match_set\",\n        kwargs={\n            \"column_set\": [\"order_id\", \"customer_id\", \"amount\", \"status\", \"created_at\"],\n            \"exact_match\": False  # Allow additional columns\n        }\n    ))\n\n    # Primary key\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_not_be_null\",\n        kwargs={\"column\": \"order_id\"}\n    ))\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_be_unique\",\n        kwargs={\"column\": \"order_id\"}\n    ))\n\n    # Foreign key\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_not_be_null\",\n        kwargs={\"column\": \"customer_id\"}\n    ))\n\n    # Categorical values\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_be_in_set\",\n        kwargs={\n            \"column\": \"status\",\n            \"value_set\": [\"pending\", \"processing\", \"shipped\", \"delivered\", \"cancelled\"]\n        }\n    ))\n\n    # Numeric ranges\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_be_between\",\n        kwargs={\n            \"column\": \"amount\",\n            \"min_value\": 0,\n            \"max_value\": 100000,\n            \"strict_min\": True  # amount > 0\n        }\n    ))\n\n    # Date validity\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_be_dateutil_parseable\",\n        kwargs={\"column\": \"created_at\"}\n    ))\n\n    # Freshness - data should be recent\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_max_to_be_between\",\n        kwargs={\n            \"column\": \"created_at\",\n            \"min_value\": {\"$PARAMETER\": \"now - timedelta(days=1)\"},\n            \"max_value\": {\"$PARAMETER\": \"now\"}\n        }\n    ))\n\n    # Row count sanity\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_table_row_count_to_be_between\",\n        kwargs={\n            \"min_value\": 1000,  # Expect at least 1000 rows\n            \"max_value\": 10000000\n        }\n    ))\n\n    # Statistical expectations\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_mean_to_be_between\",\n        kwargs={\n            \"column\": \"amount\",\n            \"min_value\": 50,\n            \"max_value\": 500\n        }\n    ))\n\n    return suite\n```\n\n### Pattern 2: Great Expectations Checkpoint\n\n```yaml\n# great_expectations\u002Fcheckpoints\u002Forders_checkpoint.yml\nname: orders_checkpoint\nconfig_version: 1.0\nclass_name: Checkpoint\nrun_name_template: \"%Y%m%d-%H%M%S-orders-validation\"\n\nvalidations:\n  - batch_request:\n      datasource_name: warehouse\n      data_connector_name: default_inferred_data_connector_name\n      data_asset_name: orders\n      data_connector_query:\n        index: -1 # Latest batch\n    expectation_suite_name: orders_suite\n\naction_list:\n  - name: store_validation_result\n    action:\n      class_name: StoreValidationResultAction\n\n  - name: store_evaluation_parameters\n    action:\n      class_name: StoreEvaluationParametersAction\n\n  - name: update_data_docs\n    action:\n      class_name: UpdateDataDocsAction\n\n  # Slack notification on failure\n  - name: send_slack_notification\n    action:\n      class_name: SlackNotificationAction\n      slack_webhook: ${SLACK_WEBHOOK}\n      notify_on: failure\n      renderer:\n        module_name: great_expectations.render.renderer.slack_renderer\n        class_name: SlackRenderer\n```\n\n```python\n# Run checkpoint\nimport great_expectations as gx\n\ncontext = gx.get_context()\nresult = context.run_checkpoint(checkpoint_name=\"orders_checkpoint\")\n\nif not result.success:\n    failed_expectations = [\n        r for r in result.run_results.values()\n        if not r.success\n    ]\n    raise ValueError(f\"Data quality check failed: {failed_expectations}\")\n```\n\n### Pattern 3: dbt Data Tests\n\n```yaml\n# models\u002Fmarts\u002Fcore\u002F_core__models.yml\nversion: 2\n\nmodels:\n  - name: fct_orders\n    description: Order fact table\n    tests:\n      # Table-level tests\n      - dbt_utils.recency:\n          datepart: day\n          field: created_at\n          interval: 1\n      - dbt_utils.at_least_one\n      - dbt_utils.expression_is_true:\n          expression: \"total_amount >= 0\"\n\n    columns:\n      - name: order_id\n        description: Primary key\n        tests:\n          - unique\n          - not_null\n\n      - name: customer_id\n        description: Foreign key to dim_customers\n        tests:\n          - not_null\n          - relationships:\n              to: ref('dim_customers')\n              field: customer_id\n\n      - name: order_status\n        tests:\n          - accepted_values:\n              values:\n                [\"pending\", \"processing\", \"shipped\", \"delivered\", \"cancelled\"]\n\n      - name: total_amount\n        tests:\n          - not_null\n          - dbt_utils.expression_is_true:\n              expression: \">= 0\"\n\n      - name: created_at\n        tests:\n          - not_null\n          - dbt_utils.expression_is_true:\n              expression: \"\u003C= current_timestamp\"\n\n  - name: dim_customers\n    columns:\n      - name: customer_id\n        tests:\n          - unique\n          - not_null\n\n      - name: email\n        tests:\n          - unique\n          - not_null\n          # Custom regex test\n          - dbt_utils.expression_is_true:\n              expression: \"email ~ '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\\\.[A-Za-z]{2,}$'\"\n```\n\n### Pattern 4: Custom dbt Tests\n\n```sql\n-- tests\u002Fgeneric\u002Ftest_row_count_in_range.sql\n{% test row_count_in_range(model, min_count, max_count) %}\n\nwith row_count as (\n    select count(*) as cnt from {{ model }}\n)\n\nselect cnt\nfrom row_count\nwhere cnt \u003C {{ min_count }} or cnt > {{ max_count }}\n\n{% endtest %}\n\n-- Usage in schema.yml:\n-- tests:\n--   - row_count_in_range:\n--       min_count: 1000\n--       max_count: 10000000\n```\n\n```sql\n-- tests\u002Fgeneric\u002Ftest_sequential_values.sql\n{% test sequential_values(model, column_name, interval=1) %}\n\nwith lagged as (\n    select\n        {{ column_name }},\n        lag({{ column_name }}) over (order by {{ column_name }}) as prev_value\n    from {{ model }}\n)\n\nselect *\nfrom lagged\nwhere {{ column_name }} - prev_value != {{ interval }}\n  and prev_value is not null\n\n{% endtest %}\n```\n\n```sql\n-- tests\u002Fsingular\u002Fassert_orders_customers_match.sql\n-- Singular test: specific business rule\n\nwith orders_customers as (\n    select distinct customer_id from {{ ref('fct_orders') }}\n),\n\ndim_customers as (\n    select customer_id from {{ ref('dim_customers') }}\n),\n\norphaned_orders as (\n    select o.customer_id\n    from orders_customers o\n    left join dim_customers c using (customer_id)\n    where c.customer_id is null\n)\n\nselect * from orphaned_orders\n-- Test passes if this returns 0 rows\n```\n\n### Pattern 5: Data Contracts\n\n```yaml\n# contracts\u002Forders_contract.yaml\napiVersion: datacontract.com\u002Fv1.0.0\nkind: DataContract\nmetadata:\n  name: orders\n  version: 1.0.0\n  owner: data-platform-team\n  contact: data-team@company.com\n\ninfo:\n  title: Orders Data Contract\n  description: Contract for order event data from the ecommerce platform\n  purpose: Analytics, reporting, and ML features\n\nservers:\n  production:\n    type: snowflake\n    account: company.us-east-1\n    database: ANALYTICS\n    schema: CORE\n\nterms:\n  usage: Internal analytics only\n  limitations: PII must not be exposed in downstream marts\n  billing: Charged per query TB scanned\n\nschema:\n  type: object\n  properties:\n    order_id:\n      type: string\n      format: uuid\n      description: Unique order identifier\n      required: true\n      unique: true\n      pii: false\n\n    customer_id:\n      type: string\n      format: uuid\n      description: Customer identifier\n      required: true\n      pii: true\n      piiClassification: indirect\n\n    total_amount:\n      type: number\n      minimum: 0\n      maximum: 100000\n      description: Order total in USD\n\n    created_at:\n      type: string\n      format: date-time\n      description: Order creation timestamp\n      required: true\n\n    status:\n      type: string\n      enum: [pending, processing, shipped, delivered, cancelled]\n      description: Current order status\n\nquality:\n  type: SodaCL\n  specification:\n    checks for orders:\n      - row_count > 0\n      - missing_count(order_id) = 0\n      - duplicate_count(order_id) = 0\n      - invalid_count(status) = 0:\n          valid values: [pending, processing, shipped, delivered, cancelled]\n      - freshness(created_at) \u003C 24h\n\nsla:\n  availability: 99.9%\n  freshness: 1 hour\n  latency: 5 minutes\n```\n\n### Pattern 6: Automated Quality Pipeline\n\n```python\n# quality_pipeline.py\nfrom dataclasses import dataclass\nfrom typing import List, Dict, Any\nimport great_expectations as gx\nfrom datetime import datetime\n\n@dataclass\nclass QualityResult:\n    table: str\n    passed: bool\n    total_expectations: int\n    failed_expectations: int\n    details: List[Dict[str, Any]]\n    timestamp: datetime\n\nclass DataQualityPipeline:\n    \"\"\"Orchestrate data quality checks across tables\"\"\"\n\n    def __init__(self, context: gx.DataContext):\n        self.context = context\n        self.results: List[QualityResult] = []\n\n    def validate_table(self, table: str, suite: str) -> QualityResult:\n        \"\"\"Validate a single table against expectation suite\"\"\"\n\n        checkpoint_config = {\n            \"name\": f\"{table}_validation\",\n            \"config_version\": 1.0,\n            \"class_name\": \"Checkpoint\",\n            \"validations\": [{\n                \"batch_request\": {\n                    \"datasource_name\": \"warehouse\",\n                    \"data_asset_name\": table,\n                },\n                \"expectation_suite_name\": suite,\n            }],\n        }\n\n        result = self.context.run_checkpoint(**checkpoint_config)\n\n        # Parse results\n        validation_result = list(result.run_results.values())[0]\n        results = validation_result.results\n\n        failed = [r for r in results if not r.success]\n\n        return QualityResult(\n            table=table,\n            passed=result.success,\n            total_expectations=len(results),\n            failed_expectations=len(failed),\n            details=[{\n                \"expectation\": r.expectation_config.expectation_type,\n                \"success\": r.success,\n                \"observed_value\": r.result.get(\"observed_value\"),\n            } for r in results],\n            timestamp=datetime.now()\n        )\n\n    def run_all(self, tables: Dict[str, str]) -> Dict[str, QualityResult]:\n        \"\"\"Run validation for all tables\"\"\"\n        results = {}\n\n        for table, suite in tables.items():\n            print(f\"Validating {table}...\")\n            results[table] = self.validate_table(table, suite)\n\n        return results\n\n    def generate_report(self, results: Dict[str, QualityResult]) -> str:\n        \"\"\"Generate quality report\"\"\"\n        report = [\"# Data Quality Report\", f\"Generated: {datetime.now()}\", \"\"]\n\n        total_passed = sum(1 for r in results.values() if r.passed)\n        total_tables = len(results)\n\n        report.append(f\"## Summary: {total_passed}\u002F{total_tables} tables passed\")\n        report.append(\"\")\n\n        for table, result in results.items():\n            status = \"✅\" if result.passed else \"❌\"\n            report.append(f\"### {status} {table}\")\n            report.append(f\"- Expectations: {result.total_expectations}\")\n            report.append(f\"- Failed: {result.failed_expectations}\")\n\n            if not result.passed:\n                report.append(\"- Failed checks:\")\n                for detail in result.details:\n                    if not detail[\"success\"]:\n                        report.append(f\"  - {detail['expectation']}: {detail['observed_value']}\")\n            report.append(\"\")\n\n        return \"\\n\".join(report)\n\n# Usage\ncontext = gx.get_context()\npipeline = DataQualityPipeline(context)\n\ntables_to_validate = {\n    \"orders\": \"orders_suite\",\n    \"customers\": \"customers_suite\",\n    \"products\": \"products_suite\",\n}\n\nresults = pipeline.run_all(tables_to_validate)\nreport = pipeline.generate_report(results)\n\n# Fail pipeline if any table failed\nif not all(r.passed for r in results.values()):\n    print(report)\n    raise ValueError(\"Data quality checks failed!\")\n```\n\n## Best Practices\n\n### Do's\n\n- **Test early** - Validate source data before transformations\n- **Test incrementally** - Add tests as you find issues\n- **Document expectations** - Clear descriptions for each test\n- **Alert on failures** - Integrate with monitoring\n- **Version contracts** - Track schema changes\n\n### Don'ts\n\n- **Don't test everything** - Focus on critical columns\n- **Don't ignore warnings** - They often precede failures\n- **Don't skip freshness** - Stale data is bad data\n- **Don't hardcode thresholds** - Use dynamic baselines\n- **Don't test in isolation** - Test relationships too\n","# Data Quality Frameworks\n\nProduction patterns for implementing data quality with Great Expectations, dbt tests, and data contracts to ensure reliable data pipelines.\n\n## When to Use This Skill\n\n- Implementing data quality checks in pipelines\n- Setting up Great Expectations validation\n- Building comprehensive dbt test suites\n- Establishing data contracts between teams\n- Monitoring data quality metrics\n- Automating data validation in CI\u002FCD\n\n## Core Concepts\n\n### 1. Data Quality Dimensions\n\n| Dimension        | Description              | Example Check                                      |\n| ---------------- | ------------------------ | -------------------------------------------------- |\n| **Completeness** | No missing values        | `expect_column_values_to_not_be_null`              |\n| **Uniqueness**   | No duplicates            | `expect_column_values_to_be_unique`                |\n| **Validity**     | Values in expected range | `expect_column_values_to_be_in_set`                |\n| **Accuracy**     | Data matches reality     | Cross-reference validation                         |\n| **Consistency**  | No contradictions        | `expect_column_pair_values_A_to_be_greater_than_B` |\n| **Timeliness**   | Data is recent           | `expect_column_max_to_be_between`                  |\n\n### 2. Testing Pyramid for Data\n\n```\n          \u002F\\\n         \u002F  \\     Integration Tests (cross-table)\n        \u002F────\\\n       \u002F      \\   Unit Tests (single column)\n      \u002F────────\\\n     \u002F          \\ Schema Tests (structure)\n    \u002F────────────\\\n```\n\n## Quick Start\n\n### Great Expectations Setup\n\n```bash\n# Install\npip install great_expectations\n\n# Initialize project\ngreat_expectations init\n\n# Create datasource\ngreat_expectations datasource new\n```\n\n```python\n# great_expectations\u002Fcheckpoints\u002Fdaily_validation.yml\nimport great_expectations as gx\n\n# Create context\ncontext = gx.get_context()\n\n# Create expectation suite\nsuite = context.add_expectation_suite(\"orders_suite\")\n\n# Add expectations\nsuite.add_expectation(\n    gx.expectations.ExpectColumnValuesToNotBeNull(column=\"order_id\")\n)\nsuite.add_expectation(\n    gx.expectations.ExpectColumnValuesToBeUnique(column=\"order_id\")\n)\n\n# Validate\nresults = context.run_checkpoint(checkpoint_name=\"daily_orders\")\n```\n\n## Patterns\n\n### Pattern 1: Great Expectations Suite\n\n```python\n# expectations\u002Forders_suite.py\nimport great_expectations as gx\nfrom great_expectations.core import ExpectationSuite\nfrom great_expectations.core.expectation_configuration import ExpectationConfiguration\n\ndef build_orders_suite() -> ExpectationSuite:\n    \"\"\"Build comprehensive orders expectation suite\"\"\"\n\n    suite = ExpectationSuite(expectation_suite_name=\"orders_suite\")\n\n    # Schema expectations\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_table_columns_to_match_set\",\n        kwargs={\n            \"column_set\": [\"order_id\", \"customer_id\", \"amount\", \"status\", \"created_at\"],\n            \"exact_match\": False  # Allow additional columns\n        }\n    ))\n\n    # Primary key\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_not_be_null\",\n        kwargs={\"column\": \"order_id\"}\n    ))\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_be_unique\",\n        kwargs={\"column\": \"order_id\"}\n    ))\n\n    # Foreign key\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_not_be_null\",\n        kwargs={\"column\": \"customer_id\"}\n    ))\n\n    # Categorical values\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_be_in_set\",\n        kwargs={\n            \"column\": \"status\",\n            \"value_set\": [\"pending\", \"processing\", \"shipped\", \"delivered\", \"cancelled\"]\n        }\n    ))\n\n    # Numeric ranges\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_be_between\",\n        kwargs={\n            \"column\": \"amount\",\n            \"min_value\": 0,\n            \"max_value\": 100000,\n            \"strict_min\": True  # amount > 0\n        }\n    ))\n\n    # Date validity\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_values_to_be_dateutil_parseable\",\n        kwargs={\"column\": \"created_at\"}\n    ))\n\n    # Freshness - data should be recent\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_max_to_be_between\",\n        kwargs={\n            \"column\": \"created_at\",\n            \"min_value\": {\"$PARAMETER\": \"now - timedelta(days=1)\"},\n            \"max_value\": {\"$PARAMETER\": \"now\"}\n        }\n    ))\n\n    # Row count sanity\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_table_row_count_to_be_between\",\n        kwargs={\n            \"min_value\": 1000,  # Expect at least 1000 rows\n            \"max_value\": 10000000\n        }\n    ))\n\n    # Statistical expectations\n    suite.add_expectation(ExpectationConfiguration(\n        expectation_type=\"expect_column_mean_to_be_between\",\n        kwargs={\n            \"column\": \"amount\",\n            \"min_value\": 50,\n            \"max_value\": 500\n        }\n    ))\n\n    return suite\n```\n\n### Pattern 2: Great Expectations Checkpoint\n\n```yaml\n# great_expectations\u002Fcheckpoints\u002Forders_checkpoint.yml\nname: orders_checkpoint\nconfig_version: 1.0\nclass_name: Checkpoint\nrun_name_template: \"%Y%m%d-%H%M%S-orders-validation\"\n\nvalidations:\n  - batch_request:\n      datasource_name: warehouse\n      data_connector_name: default_inferred_data_connector_name\n      data_asset_name: orders\n      data_connector_query:\n        index: -1 # Latest batch\n    expectation_suite_name: orders_suite\n\naction_list:\n  - name: store_validation_result\n    action:\n      class_name: StoreValidationResultAction\n\n  - name: store_evaluation_parameters\n    action:\n      class_name: StoreEvaluationParametersAction\n\n  - name: update_data_docs\n    action:\n      class_name: UpdateDataDocsAction\n\n  # Slack notification on failure\n  - name: send_slack_notification\n    action:\n      class_name: SlackNotificationAction\n      slack_webhook: ${SLACK_WEBHOOK}\n      notify_on: failure\n      renderer:\n        module_name: great_expectations.render.renderer.slack_renderer\n        class_name: SlackRenderer\n```\n\n```python\n# Run checkpoint\nimport great_expectations as gx\n\ncontext = gx.get_context()\nresult = context.run_checkpoint(checkpoint_name=\"orders_checkpoint\")\n\nif not result.success:\n    failed_expectations = [\n        r for r in result.run_results.values()\n        if not r.success\n    ]\n    raise ValueError(f\"Data quality check failed: {failed_expectations}\")\n```\n\n### Pattern 3: dbt Data Tests\n\n```yaml\n# models\u002Fmarts\u002Fcore\u002F_core__models.yml\nversion: 2\n\nmodels:\n  - name: fct_orders\n    description: Order fact table\n    tests:\n      # Table-level tests\n      - dbt_utils.recency:\n          datepart: day\n          field: created_at\n          interval: 1\n      - dbt_utils.at_least_one\n      - dbt_utils.expression_is_true:\n          expression: \"total_amount >= 0\"\n\n    columns:\n      - name: order_id\n        description: Primary key\n        tests:\n          - unique\n          - not_null\n\n      - name: customer_id\n        description: Foreign key to dim_customers\n        tests:\n          - not_null\n          - relationships:\n              to: ref('dim_customers')\n              field: customer_id\n\n      - name: order_status\n        tests:\n          - accepted_values:\n              values:\n                [\"pending\", \"processing\", \"shipped\", \"delivered\", \"cancelled\"]\n\n      - name: total_amount\n        tests:\n          - not_null\n          - dbt_utils.expression_is_true:\n              expression: \">= 0\"\n\n      - name: created_at\n        tests:\n          - not_null\n          - dbt_utils.expression_is_true:\n              expression: \"\u003C= current_timestamp\"\n\n  - name: dim_customers\n    columns:\n      - name: customer_id\n        tests:\n          - unique\n          - not_null\n\n      - name: email\n        tests:\n          - unique\n          - not_null\n          # Custom regex test\n          - dbt_utils.expression_is_true:\n              expression: \"email ~ '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\\\.[A-Za-z]{2,}$'\"\n```\n\n### Pattern 4: Custom dbt Tests\n\n```sql\n-- tests\u002Fgeneric\u002Ftest_row_count_in_range.sql\n{% test row_count_in_range(model, min_count, max_count) %}\n\nwith row_count as (\n    select count(*) as cnt from {{ model }}\n)\n\nselect cnt\nfrom row_count\nwhere cnt \u003C {{ min_count }} or cnt > {{ max_count }}\n\n{% endtest %}\n\n-- Usage in schema.yml:\n-- tests:\n--   - row_count_in_range:\n--       min_count: 1000\n--       max_count: 10000000\n```\n\n```sql\n-- tests\u002Fgeneric\u002Ftest_sequential_values.sql\n{% test sequential_values(model, column_name, interval=1) %}\n\nwith lagged as (\n    select\n        {{ column_name }},\n        lag({{ column_name }}) over (order by {{ column_name }}) as prev_value\n    from {{ model }}\n)\n\nselect *\nfrom lagged\nwhere {{ column_name }} - prev_value != {{ interval }}\n  and prev_value is not null\n\n{% endtest %}\n```\n\n```sql\n-- tests\u002Fsingular\u002Fassert_orders_customers_match.sql\n-- Singular test: specific business rule\n\nwith orders_customers as (\n    select distinct customer_id from {{ ref('fct_orders') }}\n),\n\ndim_customers as (\n    select customer_id from {{ ref('dim_customers') }}\n),\n\norphaned_orders as (\n    select o.customer_id\n    from orders_customers o\n    left join dim_customers c using (customer_id)\n    where c.customer_id is null\n)\n\nselect * from orphaned_orders\n-- Test passes if this returns 0 rows\n```\n\n### Pattern 5: Data Contracts\n\n```yaml\n# contracts\u002Forders_contract.yaml\napiVersion: datacontract.com\u002Fv1.0.0\nkind: DataContract\nmetadata:\n  name: orders\n  version: 1.0.0\n  owner: data-platform-team\n  contact: data-team@company.com\n\ninfo:\n  title: Orders Data Contract\n  description: Contract for order event data from the ecommerce platform\n  purpose: Analytics, reporting, and ML features\n\nservers:\n  production:\n    type: snowflake\n    account: company.us-east-1\n    database: ANALYTICS\n    schema: CORE\n\nterms:\n  usage: Internal analytics only\n  limitations: PII must not be exposed in downstream marts\n  billing: Charged per query TB scanned\n\nschema:\n  type: object\n  properties:\n    order_id:\n      type: string\n      format: uuid\n      description: Unique order identifier\n      required: true\n      unique: true\n      pii: false\n\n    customer_id:\n      type: string\n      format: uuid\n      description: Customer identifier\n      required: true\n      pii: true\n      piiClassification: indirect\n\n    total_amount:\n      type: number\n      minimum: 0\n      maximum: 100000\n      description: Order total in USD\n\n    created_at:\n      type: string\n      format: date-time\n      description: Order creation timestamp\n      required: true\n\n    status:\n      type: string\n      enum: [pending, processing, shipped, delivered, cancelled]\n      description: Current order status\n\nquality:\n  type: SodaCL\n  specification:\n    checks for orders:\n      - row_count > 0\n      - missing_count(order_id) = 0\n      - duplicate_count(order_id) = 0\n      - invalid_count(status) = 0:\n          valid values: [pending, processing, shipped, delivered, cancelled]\n      - freshness(created_at) \u003C 24h\n\nsla:\n  availability: 99.9%\n  freshness: 1 hour\n  latency: 5 minutes\n```\n\n### Pattern 6: Automated Quality Pipeline\n\n```python\n# quality_pipeline.py\nfrom dataclasses import dataclass\nfrom typing import List, Dict, Any\nimport great_expectations as gx\nfrom datetime import datetime\n\n@dataclass\nclass QualityResult:\n    table: str\n    passed: bool\n    total_expectations: int\n    failed_expectations: int\n    details: List[Dict[str, Any]]\n    timestamp: datetime\n\nclass DataQualityPipeline:\n    \"\"\"Orchestrate data quality checks across tables\"\"\"\n\n    def __init__(self, context: gx.DataContext):\n        self.context = context\n        self.results: List[QualityResult] = []\n\n    def validate_table(self, table: str, suite: str) -> QualityResult:\n        \"\"\"Validate a single table against expectation suite\"\"\"\n\n        checkpoint_config = {\n            \"name\": f\"{table}_validation\",\n            \"config_version\": 1.0,\n            \"class_name\": \"Checkpoint\",\n            \"validations\": [{\n                \"batch_request\": {\n                    \"datasource_name\": \"warehouse\",\n                    \"data_asset_name\": table,\n                },\n                \"expectation_suite_name\": suite,\n            }],\n        }\n\n        result = self.context.run_checkpoint(**checkpoint_config)\n\n        # Parse results\n        validation_result = list(result.run_results.values())[0]\n        results = validation_result.results\n\n        failed = [r for r in results if not r.success]\n\n        return QualityResult(\n            table=table,\n            passed=result.success,\n            total_expectations=len(results),\n            failed_expectations=len(failed),\n            details=[{\n                \"expectation\": r.expectation_config.expectation_type,\n                \"success\": r.success,\n                \"observed_value\": r.result.get(\"observed_value\"),\n            } for r in results],\n            timestamp=datetime.now()\n        )\n\n    def run_all(self, tables: Dict[str, str]) -> Dict[str, QualityResult]:\n        \"\"\"Run validation for all tables\"\"\"\n        results = {}\n\n        for table, suite in tables.items():\n            print(f\"Validating {table}...\")\n            results[table] = self.validate_table(table, suite)\n\n        return results\n\n    def generate_report(self, results: Dict[str, QualityResult]) -> str:\n        \"\"\"Generate quality report\"\"\"\n        report = [\"# Data Quality Report\", f\"Generated: {datetime.now()}\", \"\"]\n\n        total_passed = sum(1 for r in results.values() if r.passed)\n        total_tables = len(results)\n\n        report.append(f\"## Summary: {total_passed}\u002F{total_tables} tables passed\")\n        report.append(\"\")\n\n        for table, result in results.items():\n            status = \"✅\" if result.passed else \"❌\"\n            report.append(f\"### {status} {table}\")\n            report.append(f\"- Expectations: {result.total_expectations}\")\n            report.append(f\"- Failed: {result.failed_expectations}\")\n\n            if not result.passed:\n                report.append(\"- Failed checks:\")\n                for detail in result.details:\n                    if not detail[\"success\"]:\n                        report.append(f\"  - {detail['expectation']}: {detail['observed_value']}\")\n            report.append(\"\")\n\n        return \"\\n\".join(report)\n\n# Usage\ncontext = gx.get_context()\npipeline = DataQualityPipeline(context)\n\ntables_to_validate = {\n    \"orders\": \"orders_suite\",\n    \"customers\": \"customers_suite\",\n    \"products\": \"products_suite\",\n}\n\nresults = pipeline.run_all(tables_to_validate)\nreport = pipeline.generate_report(results)\n\n# Fail pipeline if any table failed\nif not all(r.passed for r in results.values()):\n    print(report)\n    raise ValueError(\"Data quality checks failed!\")\n```\n\n## Best Practices\n\n### Do's\n\n- **Test early** - Validate source data before transformations\n- **Test incrementally** - Add tests as you find issues\n- **Document expectations** - Clear descriptions for each test\n- **Alert on failures** - Integrate with monitoring\n- **Version contracts** - Track schema changes\n\n### Don'ts\n\n- **Don't test everything** - Focus on critical columns\n- **Don't ignore warnings** - They often precede failures\n- **Don't skip freshness** - Stale data is bad data\n- **Don't hardcode thresholds** - Use dynamic baselines\n- **Don't test in isolation** - Test relationships too\n",{"name":5,"description":7},"main","87afe6ce96220accf50a345dea46784c8c9f210e","YYC3-Plugin-插件协议\u002FYYC3-Plugin-插件协议-plugins-claude\u002Fdata-engineering\u002Fskills\u002Fdata-quality-frameworks",0,"https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base","https:\u002F\u002Fkb.yyc3.top\u002F","https:\u002F\u002Fgithub.com\u002FYYC-Cube","万象归元于云枢 | 深栈智启新纪元","2026-11-10 10:37:32",{"source":8,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill data-quality-frameworks",{"tool":39,"command":40},"bunx","bunx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill data-quality-frameworks",{"tool":42,"command":43},"pnpm","pnpm dlx skills add YYC-Cube\u002FYYC3-Knowledge-Base --skill data-quality-frameworks","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-Plugin-插件协议\u002FYYC3-Plugin-插件协议-plugins-claude\u002Fdata-engineering\u002Fskills\u002Fdata-quality-frameworks\u002FSKILL.md",true,[],[48,55,62,69,76,83],{"skillNo":49,"name":50,"nameZh":6,"description":51,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":52,"categories":53,"keywords":54,"publishedAt":19},"01M7CTDWHERCXPQCP0K03A86AS","vercel-sandbox","Run agent-browser + Chrome inside Vercel Sandbox microVMs for browser automation from any Vercel-deployed app. Use when the user needs browser automation in a Vercel app (Next.js, SvelteKit, Nuxt, Remix, Astro, etc.), wants to run headless Chrome without binary size limits, needs persistent browser sessions across commands, or wants ephemeral isolated browser environments. Triggers include \"Vercel Sandbox browser\", \"microVM Chrome\", \"agent-browser in sandbox\", \"browser automation on Vercel\", or any task requiring Chrome in a Vercel Sandbox.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fvercel-sandbox\u002FSKILL.md",[],[],{"skillNo":56,"name":57,"nameZh":6,"description":58,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":59,"categories":60,"keywords":61,"publishedAt":19},"01M7CTD8D2BNJY54S65M47HEJA","slack","Interact with Slack workspaces using browser automation. Use when the user needs to check unread channels, navigate Slack, send messages, extract data, find information, search conversations, or automate any Slack task. Triggers include \"check my Slack\", \"what channels have unreads\", \"send a message to\", \"search Slack for\", \"extract from Slack\", \"find who said\", or any task requiring programmatic Slack interaction.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fslack\u002FSKILL.md",[],[],{"skillNo":63,"name":64,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":68,"publishedAt":19},"01M7CTCSSB8ZP3Q58C3YZDZK1W","electron","Automate Electron desktop apps (VS Code, Slack, Discord, Figma, Notion, Spotify, etc.) using agent-browser via Chrome DevTools Protocol. Use when the user needs to interact with an Electron app, automate a desktop app, connect to a running app, control a native app, or test an Electron application. Triggers include \"automate Slack app\", \"control VS Code\", \"interact with Discord app\", \"test this Electron app\", \"connect to desktop app\", or any task requiring automation of a native Electron application.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Felectron\u002FSKILL.md",[],[],{"skillNo":70,"name":71,"nameZh":6,"description":72,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":19},"01M7CTCBJY76688V4DCZGWFHGQ","dogfood","Systematically explore and test a web application to find bugs, UX issues, and other problems. Use when asked to \"dogfood\", \"QA\", \"exploratory test\", \"find issues\", \"bug hunt\", \"test this app\u002Fsite\u002Fplatform\", or review the quality of a web application. Produces a structured report with full reproduction evidence -- step-by-step screenshots, repro videos, and detailed repro steps for every issue -- so findings can be handed directly to the responsible teams.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fdogfood\u002FSKILL.md",[],[],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":19},"01M7CTC4H0FFGVEAQR9PA4GA1A","agent-browser","Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to \"open a website\", \"fill out a form\", \"click a button\", \"take a screenshot\", \"scrape data from a page\", \"test this web app\", \"login to a site\", \"automate browser actions\", or any task requiring programmatic web interaction.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-WebSpec-Web标准\u002FYYC3-WebSpec-Web标准-standards\u002Fbrowser-agent\u002Fskills\u002Fagent-browser\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":19},"01M7CTBKVFYBG78FDSABSDJTTQ","shadcn","Manages shadcn components and projects — adding, searching, fixing, debugging, styling, and composing UI, including chat interfaces. Provides project context, component docs, and usage examples. Applies when working with shadcn\u002Fui, component registries, presets, --preset codes, or any project with a components.json file. Also triggers for \"shadcn init\", \"create an app with --preset\", \"switch to --preset\", and for building a design system or applying a DESIGN.md or brand spec to shadcn\u002Fui.","https:\u002F\u002Fgithub.com\u002FYYC-Cube\u002FYYC3-Knowledge-Base\u002Fblob\u002Fmain\u002FYYC3-UI-界面组件\u002FYYC3-UI-界面组件-ui\u002Fskills\u002Fshadcn\u002FSKILL.md",[],[]]