[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6WZ80ESNEYZAGC8HTPD2G9Q":3,"skill-same:01M6WZ80ESNEYZAGC8HTPD2G9Q":77},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":43,"linkOnly":44,"markdownContent":45,"markdownBody":46,"frontMatter":47,"refName":56,"commitSha":57,"skillDirectory":58,"forksCount":14,"repositoryHtmlUrl":59,"repositoryHomepageUrl":60,"ownerHtmlUrl":61,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":62,"install":63,"resources":76},"01M6WZ80ESNEYZAGC8HTPD2G9Q","data-quality",null,"Data quality framework — completeness, accuracy, consistency, validation, and contracts. Use when implementing data validation, setting up quality checks for a pipeline, defining data contracts between teams, or investigating data anomalies.","projectious-work\u002Fprocesskit","projectious-work","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F263889726?v=4","01M6N97TJJ118WQ35CHRV8RD0E","projectious.work","01M6N97TPWJPF0BMKYQ4PBY5W8",0,"MIT","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Farchive\u002Fprocesskit-install-20261101\u002Fcontext\u002Ftemplates\u002Fprocesskit\u002Fv0.18.1\u002Fcontext\u002Fskills\u002Fdata-ai\u002Fdata-quality\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[24,25,26,5,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42],"data","data quality","quality","when","data contracts","accuracy","for","implementing","v1","consistency","data quality framework","completeness","teams","contracts","up","use","work","pipeline","validation","2026-11-10 10:12:30",false,"---\nname: data-quality\ndescription: |\n  Data quality framework — completeness, accuracy, consistency, validation, and contracts. Use when implementing data validation, setting up quality checks for a pipeline, defining data contracts between teams, or investigating data anomalies.\nmetadata:\n  processkit:\n    apiVersion: processkit.projectious.work\u002Fv1\n    id: SKILL-data-quality\n    version: \"1.1.0\"\n    created: 2026-04-06T00:00:00Z\n    category: data-ai\n---\n\n# Data Quality\n\n## Intro\n\nData quality is measurable across six dimensions and enforced with\nexplicit validation rules at every pipeline boundary. Codify producer\nand consumer expectations as data contracts and run them as automated\nchecks rather than relying on manual review.\n\n## Overview\n\n### The six dimensions\n\nEvaluate data against:\n\n- **Completeness** — null rate per column, row count vs expected\n- **Accuracy** — spot checks against a source of truth or reference\n- **Consistency** — cross-field validation (end >= start, city matches\n  zip)\n- **Timeliness** — freshness since last update, SLA adherence\n- **Uniqueness** — distinct count of natural keys vs total rows\n- **Validity** — regex matches, range checks, enum membership\n\n### Validation rules\n\nSchema validation enforces column names, types, and nullability. Range\nchecks bound numerics (age 0–150, price >= 0). Format checks enforce\nISO 8601 dates, email regexes, normalized phone numbers. Referential\nintegrity confirms foreign keys resolve. Business rules express domain\ntruths (order total = sum of line items). Freshness checks confirm the\nlatest record is within the expected window. Categorize each rule by\nseverity: `error` blocks the pipeline, `warning` logs and continues.\n\n### Anomaly detection\n\nTrack key metrics over time: row counts, null rates, distinct value\ncounts, mean and median. Alert when metrics deviate beyond a threshold\n(row count drops > 20% vs previous run). Use z-score for normally\ndistributed metrics, IQR for skewed ones. Monitor distribution shifts\nand schema changes (new columns, type changes, dropped columns). Watch\nfor sudden spikes in null rates or default values.\n\n### Data contracts\n\nA data contract is a formal agreement between producer and consumer\nspecifying schema, SLAs, quality thresholds, and ownership. Make\ncontracts machine-readable (JSON Schema, protobuf, YAML) and enforce\nthem automatically. Version them — breaking changes require a new\nversion with a migration period. Include who to contact when violated\nand the escalation path. Review quarterly or when business\nrequirements change.\n\n### Implementing checks\n\nPlace checks at pipeline boundaries: after extraction, after\ntransformation, after loading. Compare source to destination row\ncounts. Store results in a quality metadata table (timestamp, check\nname, result, details). Start with the five critical checks — row\ncount, null rate on key columns, duplicate rate, freshness, schema\nmatch — and add more as issues arise. Keep thresholds in config files,\nnot hardcoded.\n\n## Gotchas\n\nAgent-specific failure modes — provider-neutral pause-and-self-check items:\n\n- **Checks only at the end of the pipeline.** Validating data after full transformation means a bad upstream record travels the entire pipeline before failing, corrupting intermediate outputs along the way. Add checks at ingestion on raw data and at each transformation boundary so failures are caught and quarantined early.\n- **Hardcoded thresholds disconnected from historical baselines.** A null rate threshold of 5% chosen arbitrarily will either fire constantly on a column that naturally has 4% nulls, or miss a regression on a column that usually has 0%. Derive thresholds from historical distributions — mean ± N standard deviations — and update them as the data evolves.\n- **Soft warnings that nobody monitors.** A check that logs a warning but does not block the pipeline or alert anyone is not a data quality gate — it is documentation that the data is wrong. Quality checks must have a defined consequence: block the run, quarantine the records, or page on-call. Unmonitored warnings become noise within a week.\n- **Data contracts in a wiki instead of enforceable code.** A contract documented in Confluence can drift silently from the actual schema for months before anyone notices. Enforce contracts as code — schema validation in tests or a contract-testing framework — so that upstream changes that violate the contract fail the pipeline immediately.\n- **Schema drift detection without alerting the producer.** Detecting that an upstream schema changed is only half the job. If the alert goes only to the pipeline owner, the producer never learns they broke a downstream consumer. Schema drift alerts should notify both the pipeline owner and the upstream team so the source of the change is aware.\n- **Validating only schema, not values.** A column typed as INTEGER passes schema validation even when it contains -1 where only positive values are valid, or 0 where null is the intended sentinel. Value-level checks (range, enum membership, referential integrity) catch a different class of errors than schema checks and are equally important.\n- **No lineage tracking for quarantined records.** When records fail quality checks and are quarantined, there must be a way to trace which source records were rejected, why, and what downstream impact the rejection had. Without lineage, quarantined records are invisible holes in the data that are nearly impossible to audit or recover.\n\n## Full reference\n\n### Great Expectations patterns\n\nOrganize expectations into suites — one per table or pipeline stage.\nAlways include:\n\n- `expect_table_row_count_to_be_between` — catch empty or exploding\n  tables\n- `expect_column_values_to_not_be_null` — for required columns\n- `expect_column_values_to_be_unique` — for natural keys\n- `expect_column_values_to_be_in_set` — for enum columns\n\nRun validation as a pipeline step and fail on critical expectation\nfailures. Store validation results for trend analysis and audit\ntrails.\n\n### Data observability\n\nInstrument pipelines with metadata: row counts, schema snapshots,\nexecution time. Build a lineage graph so you know where each dataset\ncomes from and what depends on it. Dashboard freshness, volume trends,\nand per-table error rates. Alert on stale data, volume anomalies,\nschema drift, and quality-rule failures. When quality drops, automate\nroot-cause traversal back through lineage to find the source.\n\n### Worked examples\n\n- **Customer table checks:** Completeness on email, name, created_at;\n  uniqueness on customer_id; validity on email regex and status enum;\n  consistency between country and postal-code format; timeliness on\n  most-recent created_at within 24 hours; accuracy via spot check\n  against a CRM export. Store results with severity labels.\n- **Daily report numbers look wrong but pipeline succeeded:** Add\n  anomaly detection that tracks row counts and key aggregates and\n  alerts on > 2σ deviation from the 30-day rolling average. Add\n  cross-field consistency checks and a freshness check before the\n  report runs.\n- **Events-to-analytics contract:** Versioned YAML specifying schema,\n  null rate \u003C 1% for user_id, row count > 1000\u002Fday, freshness SLA\n  06:00 UTC, ownership and escalation contacts. Automated validation\n  on each delivery, notifications to both teams on violations.\n\n### Anti-patterns\n\n- Quality checks only at the end of the pipeline\n- Hardcoded thresholds that nobody can tune without a deploy\n- \"Soft\" warnings that nobody monitors — equivalent to silence\n- Contracts that live in a wiki page instead of as enforceable code\n- Schema drift detection without an alert path to the producer\n","# Data Quality\n\n## Intro\n\nData quality is measurable across six dimensions and enforced with\nexplicit validation rules at every pipeline boundary. Codify producer\nand consumer expectations as data contracts and run them as automated\nchecks rather than relying on manual review.\n\n## Overview\n\n### The six dimensions\n\nEvaluate data against:\n\n- **Completeness** — null rate per column, row count vs expected\n- **Accuracy** — spot checks against a source of truth or reference\n- **Consistency** — cross-field validation (end >= start, city matches\n  zip)\n- **Timeliness** — freshness since last update, SLA adherence\n- **Uniqueness** — distinct count of natural keys vs total rows\n- **Validity** — regex matches, range checks, enum membership\n\n### Validation rules\n\nSchema validation enforces column names, types, and nullability. Range\nchecks bound numerics (age 0–150, price >= 0). Format checks enforce\nISO 8601 dates, email regexes, normalized phone numbers. Referential\nintegrity confirms foreign keys resolve. Business rules express domain\ntruths (order total = sum of line items). Freshness checks confirm the\nlatest record is within the expected window. Categorize each rule by\nseverity: `error` blocks the pipeline, `warning` logs and continues.\n\n### Anomaly detection\n\nTrack key metrics over time: row counts, null rates, distinct value\ncounts, mean and median. Alert when metrics deviate beyond a threshold\n(row count drops > 20% vs previous run). Use z-score for normally\ndistributed metrics, IQR for skewed ones. Monitor distribution shifts\nand schema changes (new columns, type changes, dropped columns). Watch\nfor sudden spikes in null rates or default values.\n\n### Data contracts\n\nA data contract is a formal agreement between producer and consumer\nspecifying schema, SLAs, quality thresholds, and ownership. Make\ncontracts machine-readable (JSON Schema, protobuf, YAML) and enforce\nthem automatically. Version them — breaking changes require a new\nversion with a migration period. Include who to contact when violated\nand the escalation path. Review quarterly or when business\nrequirements change.\n\n### Implementing checks\n\nPlace checks at pipeline boundaries: after extraction, after\ntransformation, after loading. Compare source to destination row\ncounts. Store results in a quality metadata table (timestamp, check\nname, result, details). Start with the five critical checks — row\ncount, null rate on key columns, duplicate rate, freshness, schema\nmatch — and add more as issues arise. Keep thresholds in config files,\nnot hardcoded.\n\n## Gotchas\n\nAgent-specific failure modes — provider-neutral pause-and-self-check items:\n\n- **Checks only at the end of the pipeline.** Validating data after full transformation means a bad upstream record travels the entire pipeline before failing, corrupting intermediate outputs along the way. Add checks at ingestion on raw data and at each transformation boundary so failures are caught and quarantined early.\n- **Hardcoded thresholds disconnected from historical baselines.** A null rate threshold of 5% chosen arbitrarily will either fire constantly on a column that naturally has 4% nulls, or miss a regression on a column that usually has 0%. Derive thresholds from historical distributions — mean ± N standard deviations — and update them as the data evolves.\n- **Soft warnings that nobody monitors.** A check that logs a warning but does not block the pipeline or alert anyone is not a data quality gate — it is documentation that the data is wrong. Quality checks must have a defined consequence: block the run, quarantine the records, or page on-call. Unmonitored warnings become noise within a week.\n- **Data contracts in a wiki instead of enforceable code.** A contract documented in Confluence can drift silently from the actual schema for months before anyone notices. Enforce contracts as code — schema validation in tests or a contract-testing framework — so that upstream changes that violate the contract fail the pipeline immediately.\n- **Schema drift detection without alerting the producer.** Detecting that an upstream schema changed is only half the job. If the alert goes only to the pipeline owner, the producer never learns they broke a downstream consumer. Schema drift alerts should notify both the pipeline owner and the upstream team so the source of the change is aware.\n- **Validating only schema, not values.** A column typed as INTEGER passes schema validation even when it contains -1 where only positive values are valid, or 0 where null is the intended sentinel. Value-level checks (range, enum membership, referential integrity) catch a different class of errors than schema checks and are equally important.\n- **No lineage tracking for quarantined records.** When records fail quality checks and are quarantined, there must be a way to trace which source records were rejected, why, and what downstream impact the rejection had. Without lineage, quarantined records are invisible holes in the data that are nearly impossible to audit or recover.\n\n## Full reference\n\n### Great Expectations patterns\n\nOrganize expectations into suites — one per table or pipeline stage.\nAlways include:\n\n- `expect_table_row_count_to_be_between` — catch empty or exploding\n  tables\n- `expect_column_values_to_not_be_null` — for required columns\n- `expect_column_values_to_be_unique` — for natural keys\n- `expect_column_values_to_be_in_set` — for enum columns\n\nRun validation as a pipeline step and fail on critical expectation\nfailures. Store validation results for trend analysis and audit\ntrails.\n\n### Data observability\n\nInstrument pipelines with metadata: row counts, schema snapshots,\nexecution time. Build a lineage graph so you know where each dataset\ncomes from and what depends on it. Dashboard freshness, volume trends,\nand per-table error rates. Alert on stale data, volume anomalies,\nschema drift, and quality-rule failures. When quality drops, automate\nroot-cause traversal back through lineage to find the source.\n\n### Worked examples\n\n- **Customer table checks:** Completeness on email, name, created_at;\n  uniqueness on customer_id; validity on email regex and status enum;\n  consistency between country and postal-code format; timeliness on\n  most-recent created_at within 24 hours; accuracy via spot check\n  against a CRM export. Store results with severity labels.\n- **Daily report numbers look wrong but pipeline succeeded:** Add\n  anomaly detection that tracks row counts and key aggregates and\n  alerts on > 2σ deviation from the 30-day rolling average. Add\n  cross-field consistency checks and a freshness check before the\n  report runs.\n- **Events-to-analytics contract:** Versioned YAML specifying schema,\n  null rate \u003C 1% for user_id, row count > 1000\u002Fday, freshness SLA\n  06:00 UTC, ownership and escalation contacts. Automated validation\n  on each delivery, notifications to both teams on violations.\n\n### Anti-patterns\n\n- Quality checks only at the end of the pipeline\n- Hardcoded thresholds that nobody can tune without a deploy\n- \"Soft\" warnings that nobody monitors — equivalent to silence\n- Contracts that live in a wiki page instead of as enforceable code\n- Schema drift detection without an alert path to the producer\n",{"name":5,"metadata":48,"description":55},{"processkit":49},{"id":50,"created":51,"version":52,"category":53,"apiVersion":54},"SKILL-data-quality",1775433600000,"1.1.0","data-ai","processkit.projectious.work\u002Fv1","Data quality framework — completeness, accuracy, consistency, validation, and contracts. Use when implementing data validation, setting up quality checks for a pipeline, defining data contracts between teams, or investigating data anomalies.\n","main","4876a92620ee792aac2ac9a1a806322d8a572367","archive\u002Fprocesskit-install-20261101\u002Fcontext\u002Ftemplates\u002Fprocesskit\u002Fv0.18.1\u002Fcontext\u002Fskills\u002Fdata-ai\u002Fdata-quality","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit","https:\u002F\u002Fprojectious-work.github.io\u002Fprocesskit\u002F","https:\u002F\u002Fgithub.com\u002Fprojectious-work","2026-11-11 23:24:18",{"source":8,"skillName":5,"commands":64,"prompt":74,"zipAvailable":75},[65,68,71],{"tool":66,"command":67},"npx","npx skills add projectious-work\u002Fprocesskit --skill data-quality",{"tool":69,"command":70},"bunx","bunx skills add projectious-work\u002Fprocesskit --skill data-quality",{"tool":72,"command":73},"pnpm","pnpm dlx skills add projectious-work\u002Fprocesskit --skill data-quality","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Farchive\u002Fprocesskit-install-20261101\u002Fcontext\u002Ftemplates\u002Fprocesskit\u002Fv0.18.1\u002Fcontext\u002Fskills\u002Fdata-ai\u002Fdata-quality\u002FSKILL.md",true,[],[78,104,122,143,160,178],{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":89,"publishedAt":43},"01M6X5MTC89B4ZGXW3JVSQCPF8","skill-management","Manage Skill package manifests as v1 ontology entities.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fskill-management\u002FSKILL.md",[84],{"categoryNo":85,"categoryName":86,"taxonomyType":87,"taxonomyTypeDesc":88},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[90,91,92,93,94,95,40,96,97,98,80,99,100,101,32,102,103],"management","skill management","create","update","package","alpha","list","manage","ontology","get","as","manifests","skill package","transition",{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":108,"categories":109,"keywords":111,"publishedAt":43},"01M6X5MCV74NHW5MEDNHZRF2C8","schema-management","Manage generated processkit schema contracts through deterministic MCP rebuild and inspection workflows.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fschema-management\u002FSKILL.md",[110],{"categoryNo":85,"categoryName":86,"taxonomyType":87,"taxonomyTypeDesc":88},[90,112,113,114,115,42,95,40,116,117,97,37,118,119,99,120,121],"schema","schema management","MCP","workflows","generated","rebuild","wed","deterministic","contract","mode",{"skillNo":123,"name":124,"nameZh":6,"description":125,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":126,"categories":127,"keywords":128,"publishedAt":43},"01M6X5M5G6YJ96AMZGSCZJ1D13","proposition-management","Record claims, risks, beliefs, world facts, WSJF estimates, and assumptions through the Proposition interface.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fproposition-management\u002FSKILL.md",[],[90,129,130,92,93,131,95,40,132,133,134,135,99,136,137,138,139,140,141,142],"proposition","proposition management","index","log","world","record","query","creation","event","interface","updates","facts","claims","WSJF",{"skillNo":144,"name":145,"nameZh":6,"description":146,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":147,"categories":148,"keywords":149,"publishedAt":43},"01M6X5KMQ83QN9M6MPWZM7344F","ontology-management","Manage generated v1 ontology entities that do not require a dedicated domain server.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fontology-management\u002FSKILL.md",[],[90,98,150,92,95,151,152,116,153,132,97,154,155,99,156,157,137,158,32,159],"ontology management","lifecycle","archive","roadmap","do","domain","iteration","keep","measurement","generic",{"skillNo":161,"name":162,"nameZh":6,"description":163,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":164,"categories":165,"keywords":166,"publishedAt":43},"01M6X5KCY2DN3SM21RERSFNEBT","okf-compatibility","Import, export, and validate conformant Open Knowledge Format bundles.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fokf-compatibility\u002FSKILL.md",[],[167,168,169,170,95,171,172,173,40,174,175,176,177],"okf","compatibility","okf compatibility","validate","open","knowledge","format","export","bundles","bundle","import",{"skillNo":179,"name":180,"nameZh":6,"description":181,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":182,"categories":183,"keywords":184,"publishedAt":43},"01M6X5JA8S5PNGJVFW4EQY8T83","capability-management","Manage versioned Capability entities and their governed lifecycle.","https:\u002F\u002Fgithub.com\u002Fprojectious-work\u002Fprocesskit\u002Fblob\u002Fmain\u002Fsrc\u002Fcontext\u002Fskills\u002Fprocesskit\u002Fcapability-management\u002FSKILL.md",[],[90,185,186,92,93,131,95,151,40,96,132,187,97,134,99,157,137,138,188,103],"capability","capability management","queries","changes"]