[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EKXTSV98FZ3CGSS949P46G":3,"skill-same:01M6EKXTSV98FZ3CGSS949P46G":72},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":30,"linkOnly":31,"markdownContent":32,"markdownBody":33,"frontMatter":34,"refName":49,"commitSha":50,"skillDirectory":51,"forksCount":52,"repositoryHtmlUrl":53,"repositoryHomepageUrl":6,"ownerHtmlUrl":54,"ownerBlogUrl":55,"ownerDescription":56,"updatedAt":57,"install":58,"resources":71},"01M6EKXTSV98FZ3CGSS949P46G","missingness-tiered-protocol",null,"Tiered decision tree for handling predictor missingness, with a hard floor on complete-case retention.","cgpan\u002Fedm-ars-public","cgpan","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F118746117?v=4","01M6EKT5TG36Q3GH340QN8RH08","Chenguang Pan","01M6EKT5YTQVRARGJF0VPA1SY7",5,"MIT","https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public\u002Fblob\u002Fmain\u002Fskills\u002Fmethodology\u002Fmissingness-tiered-protocol\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29],"protocol","missingness tiered protocol","methodology","critic","decision","complete","missing","median","imputation","case","retention","2026-11-10 10:12:30",false,"---\nname: missingness-tiered-protocol\nlayer: methodology\ndescription: Tiered decision tree for handling predictor missingness, with a hard floor on complete-case retention.\ntrigger_keywords:\n  - missingness\n  - imputation\n  - missing\n  - mice\n  - median\n  - iterativeimputer\napplicable_task_types: []\napplicable_datasets: []\napplicable_stages:\n  - DataEngineer\n  - Critic\npriority: 1\nreferences_skills: []\nresources: []\nversion: \"1.0\"\nrule_severity: mandatory\n---\n\n# Missingness Tiered Protocol\n\nApply this decision tree to **each predictor variable independently** after\ndropping rows with missing outcomes. Compute missingness percentages on the\nanalytic sample (post-outcome-drop), not on the full raw dataset.\n\n| Missingness in variable | Method |\n|---|---|\n| \u003C 5% | Median imputation (continuous) or mode imputation (categorical) |\n| 5% – 20% | Multiple imputation via `IterativeImputer(max_iter=5, random_state=42)` |\n| > 20% | Impute with `IterativeImputer` AND add a warning to `data_report.warnings` |\n| Complete-case analysis \u003C 60% of original n | ABORT (`validation_passed: false`), unless the outcome has structural\u002FMNAR missingness — see dataset-specific extension |\n\n## Hard rules\n\n1. **NEVER impute the outcome variable.** Drop rows where the outcome is\n   missing FIRST, before computing predictor missingness.\n2. **NEVER fit imputers, scalers, or encoders on the full dataset.** Fit on\n   the training set, then transform both train and test.\n3. **Categorical predictors** must be label-encoded to integers before\n   `IterativeImputer` (which only operates on numeric data); do not\n   coerce text labels with `pd.to_numeric(errors='coerce')` — that\n   silently nulls every categorical value.\n4. The complete-case retention floor (60% of original n) is a hard abort\n   trigger for typical datasets. Datasets with documented structural\n   missingness in the outcome (e.g., postsecondary outcomes only defined\n   for college attendees) override this floor — see the dataset-layer\n   skill for that extension.\n\n## Reference implementation sketch\n\n```python\nfrom sklearn.experimental import enable_iterative_imputer  # noqa\nfrom sklearn.impute import IterativeImputer, SimpleImputer\n\n# Step 1: drop missing outcomes first.\ndf = df.dropna(subset=[outcome_variable])\n\n# Step 2: per-column missingness on the analytic sample.\npct_missing = df[predictor_columns].isna().mean() * 100\n\n# Step 3a: low-missingness columns get a per-column simple imputer.\n# A median or mode IS a column-wise statistic, so this loop is correct.\nlow = [c for c in predictor_columns if pct_missing[c] \u003C 5]\nfor col in low:\n    imputer = SimpleImputer(strategy=\"median\")  # or \"most_frequent\"\n    train_X[[col]] = imputer.fit_transform(train_X[[col]])\n    test_X[[col]] = imputer.transform(test_X[[col]])\n\n# Step 3b: everything at or above 5% goes through ONE IterativeImputer,\n# fitted on the WHOLE numeric block at once, on training rows only.\nhigh = [c for c in predictor_columns if pct_missing[c] >= 5]\nif high:\n    numeric_block = [c for c in predictor_columns\n                     if pd.api.types.is_numeric_dtype(train_X[c])]\n    imputer = IterativeImputer(max_iter=5, random_state=42)\n    train_X[numeric_block] = imputer.fit_transform(train_X[numeric_block])\n    test_X[numeric_block] = imputer.transform(test_X[numeric_block])\n\nfor col in predictor_columns:\n    if pct_missing[col] > 20:\n        data_report[\"warnings\"].append(\n            f\"High missingness: {col} has {pct_missing[col]:.1f}% missing \"\n            \"values; imputed but flagged as a limitation.\"\n        )\n```\n\n**MANDATORY RULE — never fit `IterativeImputer` on a single column.**\n\n```python\n# WRONG. This SHIPPED, and the papers reported the wrong method.\nimputer = IterativeImputer(max_iter=5, random_state=RANDOM_STATE)\ntrain_imputed = imputer.fit_transform(train_X_raw[[col]])   # ONE column\n```\n\n`IterativeImputer` models each feature from the *other* features. Given a\nsingle column there are no other features, so it silently degenerates to\n**mean imputation**. The code then recorded\n`missingness_summary[col][\"imputation_method\"] = \"IterativeImputer\"` and\nthe manuscript repeated it — so five papers named a multivariate imputer\nand performed mean-fill, on variables missing 28–36% (`X1PAREDEXPCT`\n36.1%, `X1PAREDU` 28.6%).\n\nThis is a reporting-accuracy defect, not a preference. Either fit on the\nfull matrix so the imputer can borrow strength across columns, or record\nwhat actually ran. **The recorded `imputation_method` must name the\nmethod that executed.**\n\nFit on TRAINING ROWS ONLY — that discipline is already correct across\nruns (`fit_on_all = 0` in all five audited papers). Preserve it.\n\n## What to record in `data_report.json`\n\n```json\n{\n  \"missingness_summary\": {\n    \"X1TXMTSC\": {\"pct_missing\": 1.2, \"imputation_method\": \"median\"},\n    \"X1MTHEFF\": {\"pct_missing\": 14.7, \"imputation_method\": \"IterativeImputer\"}\n  },\n  \"warnings\": [\"High missingness: X1MTHEFF has 27.3% ...\"]\n}\n```\n\nThe Critic verifies the recorded `imputation_method` matches the protocol\nfor the recorded `pct_missing` and flags any mismatch as a major issue.\n\n## Source provenance\n\nCanonical source: `agent_prompts\u002Fdata_engineer.yaml` §\"Missing Data Protocol\"\n(most complete prose with thresholds and abort condition).\n\nMerged content from:\n- `data_registry\u002Ftask_templates\u002Fprediction.yaml` substep `dp_06` (workflow embedding of the same rules)\n- `data_registry\u002Fevaluation_rubrics\u002Fmethodological_checklist.yaml` item `dp_02` (verification language used by the Critic)\n\nDataset-layer extension (structural MNAR exception for postsecondary\noutcomes) lives in the HSLS dataset skill `hsls09-structural-mnar-outcomes`\nto keep this skill task- and dataset-agnostic.\n","# Missingness Tiered Protocol\n\nApply this decision tree to **each predictor variable independently** after\ndropping rows with missing outcomes. Compute missingness percentages on the\nanalytic sample (post-outcome-drop), not on the full raw dataset.\n\n| Missingness in variable | Method |\n|---|---|\n| \u003C 5% | Median imputation (continuous) or mode imputation (categorical) |\n| 5% – 20% | Multiple imputation via `IterativeImputer(max_iter=5, random_state=42)` |\n| > 20% | Impute with `IterativeImputer` AND add a warning to `data_report.warnings` |\n| Complete-case analysis \u003C 60% of original n | ABORT (`validation_passed: false`), unless the outcome has structural\u002FMNAR missingness — see dataset-specific extension |\n\n## Hard rules\n\n1. **NEVER impute the outcome variable.** Drop rows where the outcome is\n   missing FIRST, before computing predictor missingness.\n2. **NEVER fit imputers, scalers, or encoders on the full dataset.** Fit on\n   the training set, then transform both train and test.\n3. **Categorical predictors** must be label-encoded to integers before\n   `IterativeImputer` (which only operates on numeric data); do not\n   coerce text labels with `pd.to_numeric(errors='coerce')` — that\n   silently nulls every categorical value.\n4. The complete-case retention floor (60% of original n) is a hard abort\n   trigger for typical datasets. Datasets with documented structural\n   missingness in the outcome (e.g., postsecondary outcomes only defined\n   for college attendees) override this floor — see the dataset-layer\n   skill for that extension.\n\n## Reference implementation sketch\n\n```python\nfrom sklearn.experimental import enable_iterative_imputer  # noqa\nfrom sklearn.impute import IterativeImputer, SimpleImputer\n\n# Step 1: drop missing outcomes first.\ndf = df.dropna(subset=[outcome_variable])\n\n# Step 2: per-column missingness on the analytic sample.\npct_missing = df[predictor_columns].isna().mean() * 100\n\n# Step 3a: low-missingness columns get a per-column simple imputer.\n# A median or mode IS a column-wise statistic, so this loop is correct.\nlow = [c for c in predictor_columns if pct_missing[c] \u003C 5]\nfor col in low:\n    imputer = SimpleImputer(strategy=\"median\")  # or \"most_frequent\"\n    train_X[[col]] = imputer.fit_transform(train_X[[col]])\n    test_X[[col]] = imputer.transform(test_X[[col]])\n\n# Step 3b: everything at or above 5% goes through ONE IterativeImputer,\n# fitted on the WHOLE numeric block at once, on training rows only.\nhigh = [c for c in predictor_columns if pct_missing[c] >= 5]\nif high:\n    numeric_block = [c for c in predictor_columns\n                     if pd.api.types.is_numeric_dtype(train_X[c])]\n    imputer = IterativeImputer(max_iter=5, random_state=42)\n    train_X[numeric_block] = imputer.fit_transform(train_X[numeric_block])\n    test_X[numeric_block] = imputer.transform(test_X[numeric_block])\n\nfor col in predictor_columns:\n    if pct_missing[col] > 20:\n        data_report[\"warnings\"].append(\n            f\"High missingness: {col} has {pct_missing[col]:.1f}% missing \"\n            \"values; imputed but flagged as a limitation.\"\n        )\n```\n\n**MANDATORY RULE — never fit `IterativeImputer` on a single column.**\n\n```python\n# WRONG. This SHIPPED, and the papers reported the wrong method.\nimputer = IterativeImputer(max_iter=5, random_state=RANDOM_STATE)\ntrain_imputed = imputer.fit_transform(train_X_raw[[col]])   # ONE column\n```\n\n`IterativeImputer` models each feature from the *other* features. Given a\nsingle column there are no other features, so it silently degenerates to\n**mean imputation**. The code then recorded\n`missingness_summary[col][\"imputation_method\"] = \"IterativeImputer\"` and\nthe manuscript repeated it — so five papers named a multivariate imputer\nand performed mean-fill, on variables missing 28–36% (`X1PAREDEXPCT`\n36.1%, `X1PAREDU` 28.6%).\n\nThis is a reporting-accuracy defect, not a preference. Either fit on the\nfull matrix so the imputer can borrow strength across columns, or record\nwhat actually ran. **The recorded `imputation_method` must name the\nmethod that executed.**\n\nFit on TRAINING ROWS ONLY — that discipline is already correct across\nruns (`fit_on_all = 0` in all five audited papers). Preserve it.\n\n## What to record in `data_report.json`\n\n```json\n{\n  \"missingness_summary\": {\n    \"X1TXMTSC\": {\"pct_missing\": 1.2, \"imputation_method\": \"median\"},\n    \"X1MTHEFF\": {\"pct_missing\": 14.7, \"imputation_method\": \"IterativeImputer\"}\n  },\n  \"warnings\": [\"High missingness: X1MTHEFF has 27.3% ...\"]\n}\n```\n\nThe Critic verifies the recorded `imputation_method` matches the protocol\nfor the recorded `pct_missing` and flags any mismatch as a major issue.\n\n## Source provenance\n\nCanonical source: `agent_prompts\u002Fdata_engineer.yaml` §\"Missing Data Protocol\"\n(most complete prose with thresholds and abort condition).\n\nMerged content from:\n- `data_registry\u002Ftask_templates\u002Fprediction.yaml` substep `dp_06` (workflow embedding of the same rules)\n- `data_registry\u002Fevaluation_rubrics\u002Fmethodological_checklist.yaml` item `dp_02` (verification language used by the Critic)\n\nDataset-layer extension (structural MNAR exception for postsecondary\noutcomes) lives in the HSLS dataset skill `hsls09-structural-mnar-outcomes`\nto keep this skill task- and dataset-agnostic.\n",{"name":5,"layer":21,"version":35,"priority":36,"resources":37,"description":7,"rule_severity":38,"trigger_keywords":39,"applicable_stages":43,"references_skills":46,"applicable_datasets":47,"applicable_task_types":48},"1.0",1,[],"mandatory",[40,27,25,41,26,42],"missingness","mice","iterativeimputer",[44,45],"DataEngineer","Critic",[],[],[],"main","1aeaead66d2bf876109f3b101005d2eed54041d9","skills\u002Fmethodology\u002Fmissingness-tiered-protocol",0,"https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public","https:\u002F\u002Fgithub.com\u002Fcgpan","https:\u002F\u002Fcpan.ai","Ph.D. candidate in Measurement, Evaluation, and Statistics.","2026-09-28 20:39:06",{"source":8,"skillName":5,"commands":59,"prompt":69,"zipAvailable":70},[60,63,66],{"tool":61,"command":62},"npx","npx skills add cgpan\u002Fedm-ars-public --skill missingness-tiered-protocol",{"tool":64,"command":65},"bunx","bunx skills add cgpan\u002Fedm-ars-public --skill missingness-tiered-protocol",{"tool":67,"command":68},"pnpm","pnpm dlx skills add cgpan\u002Fedm-ars-public --skill missingness-tiered-protocol","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public\u002Fblob\u002Fmain\u002Fskills\u002Fmethodology\u002Fmissingness-tiered-protocol\u002FSKILL.md",true,[],[73,96,110,119,131,142],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":77,"categories":78,"keywords":89,"publishedAt":30},"01M6EM1ZTQZVAP90DZTVBHCCFB","acm-fixture","Fixture writing skill for ACM acmart sigconf paper output.","https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskills\u002Fwriting\u002Facm-fixture\u002FSKILL.md",[79,84],{"categoryNo":80,"categoryName":81,"taxonomyType":82,"taxonomyTypeDesc":83},"01JBSEEDCAT000000000000018","写作与文案","SCENARIO","使用场景",{"categoryNo":85,"categoryName":86,"taxonomyType":87,"taxonomyTypeDesc":88},"01JBSEEDCAT000000000000036","内容创作者","PROFESSION","适用职业",[90,91,92,93,94,95],"acm fixture","fixture","writing","LaTeX","writer","paper",{"skillNo":97,"name":98,"nameZh":6,"description":99,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":100,"categories":101,"keywords":102,"publishedAt":30},"01M6EM1XNFETAZFGPHQPAXKBYN","prediction-fixture","Fixture skill for the prediction task type; references shap-fixture.","https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskills\u002Ftask-type\u002Fprediction-fixture\u002FSKILL.md",[],[91,103,104,105,106,107,108,109],"prediction fixture","task","type","references","fixture skill","shap","analyst",{"skillNo":111,"name":112,"nameZh":6,"description":113,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":114,"categories":115,"keywords":116,"publishedAt":30},"01M6EM1V4MQXYGQ7K2ZDJRAPSM","causal-fixture","Fixture skill for the causal_inference task type.","https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskills\u002Ftask-type\u002Fcausal-fixture\u002FSKILL.md",[],[91,117,105,104,118,107],"causal fixture","inference",{"skillNo":120,"name":121,"nameZh":6,"description":122,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":123,"categories":124,"keywords":125,"publishedAt":30},"01M6EM1SCWFZZ2ACSAP65GNWK9","shap-fixture","Fixture methodology for SHAP feature importance analysis.","https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskills\u002Fmethodology\u002Fshap-fixture\u002FSKILL.md",[],[126,91,108,127,21,128,129,130],"shap fixture","feature importance","analysis","explainer","feature",{"skillNo":132,"name":133,"nameZh":6,"description":134,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":135,"categories":136,"keywords":137,"publishedAt":30},"01M6EM1PZC1MK1G0D1YFVYAQ9W","cluster-recon-fixture","Fixture methodology for reconstructing cluster IDs from fingerprint variables.","https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskills\u002Fmethodology\u002Fcluster-recon-fixture\u002FSKILL.md",[],[138,139,140,91,141,21],"recon","cluster recon fixture","cluster","ids",{"skillNo":143,"name":144,"nameZh":6,"description":145,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":146,"categories":147,"keywords":148,"publishedAt":30},"01M6EM1MZVKMSWEXZ7TJ45DS04","hsls09-fixture","Fixture skill for HSLS:09 with a bundled YAML resource and one cross-reference.","https:\u002F\u002Fgithub.com\u002Fcgpan\u002Fedm-ars-public\u002Fblob\u002Fmain\u002Ftests\u002Ffixtures\u002Fskills\u002Fdataset\u002Fhsls09-fixture\u002FSKILL.md",[],[91,149,150,151,152,107,153,138,140,154],"hsls09 fixture","yaml","public","reference","dataset","registry"]