[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7DM7FPXD5DQ28MS4M0RVC3X":3,"skill-same:01M7DM7FPXD5DQ28MS4M0RVC3X":46},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":6,"ownerHtmlUrl":29,"ownerBlogUrl":30,"ownerDescription":6,"updatedAt":31,"install":32,"resources":45},"01M7DM7FPXD5DQ28MS4M0RVC3X","data-quality-framework",null,"Guide to designing validation rules for each data quality dimension (accuracy, completeness, timeliness, consistency and others) and to using tools such as Great Expectations and dbt tests. Use this skill for data quality work: 'data quality', 'validation rules', 'Great Expectations', 'dbt test', 'data profiling', 'anomaly detection', 'data contract'. It strengthens the quality validation capability of data-quality-manager. Pipeline scheduling and overall architecture design are outside the scope of this skill.","danielnguyenfinhub\u002Ffinhub-harness","danielnguyenfinhub","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F264356854?v=4","01M7DM55XETDR9HF92FWC28SBN","Daniel Nguyen","01M7DM55YGJYBWW488JZSQYEJ4",1,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness\u002Fblob\u002Fmain\u002Fpacks\u002Ffinhub-ops-pack\u002Fskills\u002Fdata-quality-framework\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: data-quality-framework\ndescription: \"Guide to designing validation rules for each data quality dimension (accuracy, completeness, timeliness, consistency and others) and to using tools such as Great Expectations and dbt tests. Use this skill for data quality work: 'data quality', 'validation rules', 'Great Expectations', 'dbt test', 'data profiling', 'anomaly detection', 'data contract'. It strengthens the quality validation capability of data-quality-manager. Pipeline scheduling and overall architecture design are outside the scope of this skill.\"\n---\n\n# Data Quality Framework — Data Quality Framework Guide\n\nA framework for systematically defining, measuring and monitoring data quality.\n\n## The Six Data Quality Dimensions\n\n| Dimension | Definition | Measurement method | Example threshold |\n|------|------|----------|-----------|\n| **Accuracy** | Correctly reflects reality | Source reconciliation, business rule validation | Accuracy > 99.9% |\n| **Completeness** | Required data is present | NULL rate, required fields filled | NULL \u003C 1% |\n| **Timeliness** | Arrives within the expected time | Latency, data freshness | Latency \u003C 30 minutes |\n| **Consistency** | Matches across systems | Cross-validation, referential integrity | Mismatches = 0 |\n| **Uniqueness** | No duplicates | Duplicate row\u002Fkey rate | Duplicates = 0% |\n| **Validity** | Conforms to format and range | Regular expressions, range checks | Valid rate > 99% |\n\n## Validation Rule Design Patterns\n\n### P0 (Mandatory — pipeline stops on failure)\n\n```yaml\nrules:\n  - name: pk_uniqueness\n    type: uniqueness\n    column: order_id\n    threshold: 0  # zero duplicates\n\n  - name: not_null_critical\n    type: completeness\n    columns: [order_id, customer_id, total_amount]\n    max_null_rate: 0\n\n  - name: row_count_sanity\n    type: volume\n    min_rows: 1000  # minimum daily order count\n    max_deviation: 0.5  # warn if volume changes by 50% or more from the previous day\n\n  - name: referential_integrity\n    type: consistency\n    source: orders.customer_id\n    reference: customers.id\n    match_rate: 1.0\n```\n\n### P1 (Important — warn, then proceed)\n\n```yaml\nrules:\n  - name: email_format\n    type: validity\n    column: email\n    pattern: \"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\\\.[a-zA-Z]{2,}$\"\n    threshold: 0.99\n\n  - name: amount_range\n    type: accuracy\n    column: total_amount\n    min: 0\n    max: 100000000  # orders above 100 million are suspect\n\n  - name: freshness\n    type: timeliness\n    column: created_at\n    max_age_hours: 24\n```\n\n## Great Expectations Implementation\n\n```python\nimport great_expectations as gx\n\n# Define expectations\nsuite = context.add_expectation_suite(\"orders_quality\")\n\n# Completeness\nsuite.add_expectation(\n    gx.expectations.ExpectColumnValuesToNotBeNull(column=\"order_id\")\n)\n\n# Uniqueness\nsuite.add_expectation(\n    gx.expectations.ExpectColumnValuesToBeUnique(column=\"order_id\")\n)\n\n# Validity\nsuite.add_expectation(\n    gx.expectations.ExpectColumnValuesToBeBetween(\n        column=\"total_amount\", min_value=0, max_value=100000000\n    )\n)\n\n# Volume\nsuite.add_expectation(\n    gx.expectations.ExpectTableRowCountToBeBetween(\n        min_value=1000, max_value=1000000\n    )\n)\n```\n\n## dbt Tests Implementation\n\n```yaml\n# schema.yml\nmodels:\n  - name: orders\n    columns:\n      - name: order_id\n        tests:\n          - unique\n          - not_null\n      - name: customer_id\n        tests:\n          - not_null\n          - relationships:\n              to: ref('customers')\n              field: id\n      - name: total_amount\n        tests:\n          - not_null\n          - dbt_utils.accepted_range:\n              min_value: 0\n              max_value: 100000000\n    tests:\n      - dbt_utils.recency:\n          datepart: hour\n          field: created_at\n          interval: 24\n```\n\n## Data Profiling Checklist\n\n```\nPer-column profile:\n├── Type: actual type vs declared type\n├── Cardinality: number of distinct values\n├── NULL rate: missing-value patterns\n├── Distribution: histogram, skewness, kurtosis\n├── Outliers: IQR-based outliers\n├── Patterns: match rate against regular expressions for dates, emails, phone numbers, etc.\n└── Dependencies: functional dependencies\n\nPer-table profile:\n├── Row count: expected range vs actual\n├── Duplicate rate: duplicates among all rows\n├── Referential integrity: number of FK violations\n└── Time distribution: record creation time patterns\n```\n\n## Anomaly Detection Techniques\n\n| Technique | Applies to | Formula \u002F method |\n|------|------|----------|\n| Z-Score | Normally distributed data | \\|x - μ\\| \u002F σ > 3 |\n| IQR | Non-normal distributions | x \u003C Q1-1.5*IQR or x > Q3+1.5*IQR |\n| Moving average | Time-series volume | Deviation of more than 2σ from the 7-day moving average |\n| Day-over-day | Daily loads | \\|today - yesterday\\| \u002F yesterday > 0.5 |\n\n## Data Contract\n\n```yaml\n# data-contract.yml\nname: orders\nversion: \"2.0.0\"\nowner: order-team\ndescription: \"Orders data contract\"\n\nschema:\n  - name: order_id\n    type: string\n    required: true\n    unique: true\n  - name: total_amount\n    type: decimal(10,2)\n    required: true\n    min: 0\n\nsla:\n  freshness: 1h\n  availability: 99.9%\n\nquality:\n  completeness: 99.9%\n  accuracy: 99.99%\n```\n","# Data Quality Framework — Data Quality Framework Guide\n\nA framework for systematically defining, measuring and monitoring data quality.\n\n## The Six Data Quality Dimensions\n\n| Dimension | Definition | Measurement method | Example threshold |\n|------|------|----------|-----------|\n| **Accuracy** | Correctly reflects reality | Source reconciliation, business rule validation | Accuracy > 99.9% |\n| **Completeness** | Required data is present | NULL rate, required fields filled | NULL \u003C 1% |\n| **Timeliness** | Arrives within the expected time | Latency, data freshness | Latency \u003C 30 minutes |\n| **Consistency** | Matches across systems | Cross-validation, referential integrity | Mismatches = 0 |\n| **Uniqueness** | No duplicates | Duplicate row\u002Fkey rate | Duplicates = 0% |\n| **Validity** | Conforms to format and range | Regular expressions, range checks | Valid rate > 99% |\n\n## Validation Rule Design Patterns\n\n### P0 (Mandatory — pipeline stops on failure)\n\n```yaml\nrules:\n  - name: pk_uniqueness\n    type: uniqueness\n    column: order_id\n    threshold: 0  # zero duplicates\n\n  - name: not_null_critical\n    type: completeness\n    columns: [order_id, customer_id, total_amount]\n    max_null_rate: 0\n\n  - name: row_count_sanity\n    type: volume\n    min_rows: 1000  # minimum daily order count\n    max_deviation: 0.5  # warn if volume changes by 50% or more from the previous day\n\n  - name: referential_integrity\n    type: consistency\n    source: orders.customer_id\n    reference: customers.id\n    match_rate: 1.0\n```\n\n### P1 (Important — warn, then proceed)\n\n```yaml\nrules:\n  - name: email_format\n    type: validity\n    column: email\n    pattern: \"^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\\\.[a-zA-Z]{2,}$\"\n    threshold: 0.99\n\n  - name: amount_range\n    type: accuracy\n    column: total_amount\n    min: 0\n    max: 100000000  # orders above 100 million are suspect\n\n  - name: freshness\n    type: timeliness\n    column: created_at\n    max_age_hours: 24\n```\n\n## Great Expectations Implementation\n\n```python\nimport great_expectations as gx\n\n# Define expectations\nsuite = context.add_expectation_suite(\"orders_quality\")\n\n# Completeness\nsuite.add_expectation(\n    gx.expectations.ExpectColumnValuesToNotBeNull(column=\"order_id\")\n)\n\n# Uniqueness\nsuite.add_expectation(\n    gx.expectations.ExpectColumnValuesToBeUnique(column=\"order_id\")\n)\n\n# Validity\nsuite.add_expectation(\n    gx.expectations.ExpectColumnValuesToBeBetween(\n        column=\"total_amount\", min_value=0, max_value=100000000\n    )\n)\n\n# Volume\nsuite.add_expectation(\n    gx.expectations.ExpectTableRowCountToBeBetween(\n        min_value=1000, max_value=1000000\n    )\n)\n```\n\n## dbt Tests Implementation\n\n```yaml\n# schema.yml\nmodels:\n  - name: orders\n    columns:\n      - name: order_id\n        tests:\n          - unique\n          - not_null\n      - name: customer_id\n        tests:\n          - not_null\n          - relationships:\n              to: ref('customers')\n              field: id\n      - name: total_amount\n        tests:\n          - not_null\n          - dbt_utils.accepted_range:\n              min_value: 0\n              max_value: 100000000\n    tests:\n      - dbt_utils.recency:\n          datepart: hour\n          field: created_at\n          interval: 24\n```\n\n## Data Profiling Checklist\n\n```\nPer-column profile:\n├── Type: actual type vs declared type\n├── Cardinality: number of distinct values\n├── NULL rate: missing-value patterns\n├── Distribution: histogram, skewness, kurtosis\n├── Outliers: IQR-based outliers\n├── Patterns: match rate against regular expressions for dates, emails, phone numbers, etc.\n└── Dependencies: functional dependencies\n\nPer-table profile:\n├── Row count: expected range vs actual\n├── Duplicate rate: duplicates among all rows\n├── Referential integrity: number of FK violations\n└── Time distribution: record creation time patterns\n```\n\n## Anomaly Detection Techniques\n\n| Technique | Applies to | Formula \u002F method |\n|------|------|----------|\n| Z-Score | Normally distributed data | \\|x - μ\\| \u002F σ > 3 |\n| IQR | Non-normal distributions | x \u003C Q1-1.5*IQR or x > Q3+1.5*IQR |\n| Moving average | Time-series volume | Deviation of more than 2σ from the 7-day moving average |\n| Day-over-day | Daily loads | \\|today - yesterday\\| \u002F yesterday > 0.5 |\n\n## Data Contract\n\n```yaml\n# data-contract.yml\nname: orders\nversion: \"2.0.0\"\nowner: order-team\ndescription: \"Orders data contract\"\n\nschema:\n  - name: order_id\n    type: string\n    required: true\n    unique: true\n  - name: total_amount\n    type: decimal(10,2)\n    required: true\n    min: 0\n\nsla:\n  freshness: 1h\n  availability: 99.9%\n\nquality:\n  completeness: 99.9%\n  accuracy: 99.99%\n```\n",{"name":5,"description":7},"main","dda161ff9daf0d9bccbdba26e34bbee0438a18b0","packs\u002Ffinhub-ops-pack\u002Fskills\u002Fdata-quality-framework",0,"https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness","https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub","www.finhub.net.au","2026-11-12 17:07:32",{"source":8,"skillName":5,"commands":33,"prompt":43,"zipAvailable":44},[34,37,40],{"tool":35,"command":36},"npx","npx skills add danielnguyenfinhub\u002Ffinhub-harness --skill data-quality-framework",{"tool":38,"command":39},"bunx","bunx skills add danielnguyenfinhub\u002Ffinhub-harness --skill data-quality-framework",{"tool":41,"command":42},"pnpm","pnpm dlx skills add danielnguyenfinhub\u002Ffinhub-harness --skill data-quality-framework","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness\u002Fblob\u002Fmain\u002Fpacks\u002Ffinhub-ops-pack\u002Fskills\u002Fdata-quality-framework\u002FSKILL.md",true,[],[47,54,61,67,73,80],{"skillNo":48,"name":49,"nameZh":6,"description":50,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":51,"categories":52,"keywords":53,"publishedAt":19},"01M7DMBP1MQ9X39BGJ9VF62M9P","linky","Fixture whose links and bundled-file mentions all resolve. Use to prove no false errors.","https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fharness_links_ok\u002Fskills\u002Flinky\u002FSKILL.md",[],[],{"skillNo":55,"name":56,"nameZh":6,"description":57,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":58,"categories":59,"keywords":60,"publishedAt":19},"01M7DMBK70KXF6A3V1VDRHYGVT","dup-one","Second owner of the name. Use for the duplicate fixture.","https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fharness_links_bad\u002Fskills\u002Fdup-two\u002FSKILL.md",[],[],{"skillNo":62,"name":56,"nameZh":6,"description":63,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":64,"categories":65,"keywords":66,"publishedAt":19},"01M7DMBFWK8HD4AWNK89KGE89P","First owner of the name. Use for the duplicate fixture.","https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fharness_links_bad\u002Fskills\u002Fdup-one\u002FSKILL.md",[],[],{"skillNo":68,"name":69,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":70,"categories":71,"keywords":72,"publishedAt":19},"01M7DMBETX7172TFNBAA3BVREQ","nested","https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fharness_links_bad\u002Fskills\u002Fbroken\u002Fnested\u002FSKILL.md",[],[],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":77,"categories":78,"keywords":79,"publishedAt":19},"01M7DMBDCH5G61QVAB1QFMQA55","broken","Fixture with one defect per rule. Use to prove each rule fires.","https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fharness_links_bad\u002Fskills\u002Fbroken\u002FSKILL.md",[],[],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":19},"01M7DMBAQHY0Z2PTYGCPM8H7MT","notes-style","House style for client notes. Use when writing or editing a client note.","https:\u002F\u002Fgithub.com\u002Fdanielnguyenfinhub\u002Ffinhub-harness\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fharness_good\u002Fskills\u002Fnotes-style\u002FSKILL.md",[],[]]