[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7F1Q4WSMB973WE3XCM33B68":3,"skill-same:01M7F1Q4WSMB973WE3XCM33B68":85},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":31,"commitSha":32,"skillDirectory":33,"forksCount":14,"repositoryHtmlUrl":34,"repositoryHomepageUrl":34,"ownerHtmlUrl":35,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":36,"install":37,"resources":50},"01M7F1Q4WSMB973WE3XCM33B68","working-with-data",null,"> Foundational data-discipline skill for econometrics work in any language. Enforces the discipline that every other data, analysis, and writing skill depends on: define the unit of analysis before anything else; treat IDs as a contract; merge with explicit cardinality and validation; never silently","Mr-Nobody-Anonymous\u002FAll-skills","Mr-Nobody-Anonymous","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F184253119?v=4","01M7AEXFJHKTYQW5R0ZCQQTWFV","anonymous","01M7C00M3GQR4QKP9GZA42GFAK",0,"MIT","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Feconomics\u002Fworking-with-data\u002FSKILL.md",[],[],"2026-11-16 10:05:34",false,"---\nname: working-with-data\ndescription: \"> Foundational data-discipline skill for econometrics work in any language. Enforces the discipline that every other data, analysis, and writing skill depends on: define the unit of analysis before anything else; treat IDs as a contract; merge with explicit cardinality and validation; never silently\"\ncategory: economics\nversion: 1.0.0\ndisable-model-invocation: false\nrisk: low\nsource: \"https:\u002F\u002Fgithub.com\u002FJonasWeinert\u002FEconAgentSkills\"\nsource_repository: \"JonasWeinert\u002FEconAgentSkills\"\nsource_path: \"_skills\u002Fdata\u002Fworking-with-data\u002FSKILL.md\"\nlicense: \"MIT\"\nimported_at: \"2026-09-20\"\n---\n# Working With Data\n\nThe foundational data-discipline skill. Every other skill in this repo (`api-data-fetcher`, `stata-data-cleaning`, the analysis skills, the writing skills) assumes the data was prepared with this discipline. Use this skill before, between, or alongside the others — not after.\n\n## Operating Principles\n\n1. **Unit of analysis first, code second.** Before any merge, regression, or figure, state the unit of analysis explicitly (firm-year, household-month, country-quarter, individual). Assert it with `isid` \u002F `df.duplicated().sum() == 0` \u002F `stopifnot(!duplicated(...))`. Half of empirical mistakes come from a silent change of unit between two operations.\n2. **IDs are a contract.** A unique identifier is unique, non-missing, immutable, type-stable, and documented. If your ID can be missing, it isn't an ID. If it can change between waves, it isn't an ID.\n3. **Merging is cardinality first.** Always declare 1:1, m:1, 1:m, or m:m before the merge. Always require `match` (or its language equivalent) and inspect anything that doesn't match. There is no such thing as a \"safe\" `m:m` merge; if you wrote one, redesign.\n4. **Raw is immutable.** `data\u002Fraw\u002F` is write-once. Every transformation produces a new file in `data\u002Fintermediate\u002F` or `data\u002Fprocessed\u002F`. Re-running a pipeline from scratch reproduces the analysis-ready file byte-for-byte.\n5. **PII never leaves the encrypted folder.** Names, GPS, dates of birth, contact info, national IDs, photos: encrypted at rest, env-var keys, `.gitignore`d, never echoed to logs. The de-identified version is what the analysis sees.\n6. **Drops are logged.** Every filter has a reason in a comment and an N before\u002Fafter.\n\n## Decision Policy\n\nThis skill follows the repo-wide [Agent Policy](..\u002F..\u002FAGENT_POLICY.md).\n\n**ASK before proceeding** (blocking):\n\n1. What is the unit of analysis (single ID or composite key)?\n2. Is the dataset a panel (balanced, unbalanced, rotating) or a cross-section?\n3. Does the raw data contain PII? If yes, is encryption already set up?\n4. Before any merge: what is the cardinality (1:1, m:1, 1:m), what is the match key, and what should happen to non-matches (drop, keep with `_merge` flag, error)?\n5. When changing the unit of analysis (collapse, expand, reshape): which observations should aggregate, and how?\n\n**DEFAULT + flag** (use this default; tell the user how to override):\n\n- DataWork folder layout: `data\u002F{raw, intermediate, processed}` + `data\u002Fencrypted\u002F` for PII (see [DIME DataWork Folder](https:\u002F\u002Fdimewiki.worldbank.org\u002FDataWork_Folder)). Override with explicit paths.\n- Merge non-matches: drop master-only and using-only with a count printed, unless the user specifies `keep` or `keepall`.\n- Master dataset per unit of observation, in `data\u002Fprocessed\u002Fmaster\u002F`.\n- Numeric IDs over string IDs when both are available.\n- Long format for analysis-ready panels; wide only when needed for a specific output.\n\n**DOCUMENT and proceed** (write into the decisions log):\n\n- Inferred unit of analysis when not stated, with the assertion that confirms it.\n- Each merge: cardinality, key, N matched \u002F master-only \u002F using-only.\n- Each filter: reason and N before\u002Fafter.\n- Each derived variable: source columns and transformation.\n- Choice of de-identification thresholds (k-anonymity level, age coarsening width).\n\n`PROCEED` items follow the cross-cutting rules in `AGENT_POLICY.md` (raw immutable, no PII in git, dynamic absolute paths, `set seed`, `ieboilstart` \u002F `pyproject.toml` \u002F `Project.toml`).\n\n## Pre-flight Checklist\n\nBefore generating code, confirm with the user — and write the answers into the script header:\n\n- **Unit of analysis.** Single ID (`hhid`) or composite (`hhid year`)? What is \"one observation\"?\n- **Panel structure.** Cross-section \u002F balanced panel \u002F unbalanced \u002F rotating panel?\n- **ID provenance.** Where do IDs come from? Are they stable across waves? Across data sources?\n- **PII status.** Does the file contain personally identifying information? If yes, is the workspace already configured for encryption?\n- **Goal of this script.** Build master dataset \u002F merge for analysis \u002F reshape \u002F sample construction \u002F harmonize across waves \u002F de-identify for release?\n\n## Decision Tree: Combining Datasets\n\n```\nSame unit of observation, complementary variables\n├── Cross-section: 1:1 merge on the unit ID\n│   Stata:  merge 1:1 hhid using \"B.dta\", assert(match)\n│   Python: pd.merge(A, B, on=\"hhid\", how=\"inner\", validate=\"1:1\")\n│   R:      dplyr::inner_join(A, B, by = \"hhid\")\n│           stopifnot(nrow(A) == nrow(result))\n└── Panel: 1:1 on the composite key\n    Stata:  merge 1:1 hhid year using \"B.dta\", assert(match)\n    Python: pd.merge(A, B, on=[\"hhid\",\"year\"], validate=\"1:1\")\n\nMultiple obs in one, one in the other (e.g. household + member info)\n└── m:1 from the many side, attach the household covariates\n    Stata:  merge m:1 hhid using \"household.dta\", assert(match master)\n    Python: pd.merge(members, household, on=\"hhid\", validate=\"m:1\")\n\nBoth sides multi (e.g. transactions x people, with same key)\n└── DO NOT m:m merge. Either:\n    1. Aggregate one side first to make it 1:m, OR\n    2. Use a join that explicitly handles the cross product\n       (rarely the right answer; reconsider the question).\n\nDifferent units of observation but you need a panel\n└── First decide the unit of the result, then aggregate one side.\n    \"Firm-year × employee-quarter\" -> aggregate employees to\n    firm-year (means, sums, counts) before merging.\n\nCross-walks \u002F fuzzy keys (names, addresses)\n└── Use a deterministic crosswalk file when possible. If not, use\n    a probabilistic match (R: fuzzyjoin; Python: rapidfuzz; Stata:\n    matchit \u002F reclink2). Always validate on a labeled subset and\n    report match rates.\n```\n\n## ID Variable Contract\n\n(Following DIME's [ID Variable Properties](https:\u002F\u002Fdimewiki.worldbank.org\u002FID_Variable_Properties).)\n\nA field is an ID variable iff it satisfies *all* of:\n\n1. **Unique** within the dataset (or with the rest of the composite key).\n2. **Non-missing** for every observation.\n3. **Immutable** within and across data sources. Same firm = same `firm_id` everywhere.\n4. **Type-stable.** Either always numeric or always string. Strings preserve leading zeros and very long codes; numerics are smaller and faster. Choose once.\n5. **Documented** in the codebook with: source, format (digits, with\u002Fwithout leading zeros), domain, and any known issues (mergers, reassignments).\n\nStandard assertion (run after import, after merge, before save):\n\n```stata\nisid hhid year                       \u002F\u002F Stata\nduplicates report hhid year          \u002F\u002F explicit count\nassert !missing(hhid)\n```\n\n```python\nassert df.duplicated([\"hhid\", \"year\"]).sum() == 0       # Python\nassert df[[\"hhid\", \"year\"]].notna().all().all()\n```\n\n```r\nstopifnot(!anyDuplicated(df[, c(\"hhid\", \"year\")]))      # R\nstopifnot(!any(is.na(df$hhid)))\n```\n\n## Merging Without Surprises\n\nBefore any merge, declare:\n\n1. **Cardinality.** 1:1, m:1, 1:m. (Reject m:m.)\n2. **Key.** Single column or composite, same name on both sides.\n3. **Expected match status.** All match? Master-only allowed? Using-only allowed?\n4. **What happens to non-matches.** Drop, flag, error.\n\nValidation pattern:\n\n```stata\n* Stata\nuse \"master.dta\", clear\nisid hhid\nmerge 1:1 hhid using \"using.dta\"\nassert _merge == 3                     \u002F\u002F require all match; or:\n* tabulate _merge\n* drop if _merge == 2                  \u002F\u002F drop using-only\ndrop _merge\nisid hhid\n```\n\n```python\n# Python (pandas)\nmaster = pd.read_parquet(\"master.parquet\")\nusing  = pd.read_parquet(\"using.parquet\")\nassert master[\"hhid\"].is_unique\nassert using[\"hhid\"].is_unique\nout = master.merge(using, on=\"hhid\", how=\"left\",\n                   validate=\"1:1\", indicator=True)\nn_only_master = (out[\"_merge\"] == \"left_only\").sum()\nn_only_using  = (out[\"_merge\"] == \"right_only\").sum()\nprint(f\"only_master={n_only_master}, only_using={n_only_using}\")\nout = out.drop(columns=\"_merge\")\n```\n\n```r\n# R (dplyr \u002F inner_join, with explicit anti-joins to inspect)\nlibrary(dplyr)\nmaster \u003C- arrow::read_parquet(\"master.parquet\")\nusing  \u003C- arrow::read_parquet(\"using.parquet\")\nstopifnot(!anyDuplicated(master$hhid), !anyDuplicated(using$hhid))\n\nonly_master \u003C- anti_join(master, using, by = \"hhid\")\nonly_using  \u003C- anti_join(using, master, by = \"hhid\")\nmessage(sprintf(\"only_master=%d, only_using=%d\",\n                nrow(only_master), nrow(only_using)))\n\nout \u003C- inner_join(master, using, by = \"hhid\")\nstopifnot(!anyDuplicated(out$hhid))\n```\n\nIf `n_only_master` or `n_only_using` is non-zero, **stop and ask** (or document if the agent has been told what to do).\n\n## Master Dataset\n\nA master dataset is the single source of truth for time-invariant information about each unit:\n\n- one master per unit of observation (household, individual, firm, school);\n- de-identified version in `data\u002Fprocessed\u002Fmaster\u002F` for analysis;\n- encrypted version in `data\u002Fencrypted\u002Fmaster\u002F` if PII is involved;\n- updated through one reviewed script, never ad-hoc.\n\nA master dataset typically holds:\n\n- The ID and any aliases \u002F crosswalks.\n- Sample frame (was this unit in the sampling universe?).\n- Sampling status and stratum.\n- Treatment assignment (and date, for staggered designs).\n- Time-invariant characteristics (region, founding year, baseline values used in matching).\n\nWhen in doubt: store time-invariant fields in master, time-varying fields in survey-round files, and `merge m:1` from the round file to master at analysis time.\n\n## Sample Construction Discipline\n\nEvery cleaning script logs N at every filter. Pattern:\n\n```python\ndef log_filter(df, label):\n    print(f\"[{label}] N = {len(df):,}, units = {df['unit'].nunique():,}\")\n    return df\n\nclean = (raw\n         .pipe(log_filter, \"raw\")\n         .query(\"year >= 2010\").pipe(log_filter, \"year >= 2010\")\n         .dropna(subset=[\"outcome\"]).pipe(log_filter, \"outcome non-missing\")\n         .query(\"revenue > 0\").pipe(log_filter, \"revenue > 0\"))\n```\n\n```r\nlog_filter \u003C- function(df, label) {\n  message(sprintf(\"[%s] N = %d, units = %d\",\n                  label, nrow(df), dplyr::n_distinct(df$unit)))\n  df\n}\n\nclean \u003C- raw |>\n  log_filter(\"raw\") |>\n  filter(year >= 2010) |> log_filter(\"year >= 2010\") |>\n  filter(!is.na(outcome)) |> log_filter(\"outcome non-missing\")\n```\n\nIn Stata, prefer `iedropone` over bare `drop`:\n\n```stata\nieduplicates ...                              \u002F\u002F resolve dups first\niedropone if year \u003C 2010, error               \u002F\u002F errors if count is unexpected\niedropone if missing(outcome), error\n```\n\nThe output of these logs goes into the decisions log block at the top of the script.\n\n## PII Discipline\n\nTreat PII like radioactive material:\n\n- Common PII (always): names, GPS coordinates, dates of birth, contact info, national IDs, photos.\n- Context-dependent PII: small geographic units, rare conditions, employer x role for high-profile people.\n- Never commit PII to git. Add `data\u002Fencrypted\u002F`, `*.dta` containing PII, and `.env` to `.gitignore`.\n- Encrypt at rest (VeraCrypt for Stata-native; age \u002F gpg for plain files).\n- Coarsen quasi-identifiers before sharing (age in 5-year bands; collapse small geographies; suppress small cells).\n- Use [J-PAL `pii_scan`](https:\u002F\u002Fgithub.com\u002FJ-PAL\u002Fstata_PII_scan) to flag candidates the agent might miss.\n- `stata-data-cleaning` skill has a full `deidentify_for_release.do` example.\n\n## Common Pitfalls\n\n- Merging without checking `_merge` codes (Stata) or `validate=` (pandas) — silently produces a Cartesian-style result when the cardinality is wrong.\n- Reshape that silently drops rows because variable names don't match the stub.\n- `m:m` merge that turns 1,000 + 1,000 rows into 50,000.\n- Using a string ID like `\"01234\"` and a numeric ID like `1234` interchangeably across files — they look the same, never match.\n- Filtering on a derived variable, then later regenerating the variable with a different definition.\n- Storing dates as strings in some files and as Stata dates \u002F pandas datetime in others.\n- Saving a \"cleaned\" file that still contains observations the team agreed to drop, because the drop was applied in interactive mode but not in the script.\n- Hand-typing N in the paper because the cleaning log is in a notebook.\n\n## Additional Resources\n\n- `reference.md` — extended patterns: composite keys, hierarchical data (households \u002F individuals), reshape, survey weights, crosswalks, time-zone discipline, encoding gotchas, codebooks.\n- `examples\u002F` — runnable cross-language patterns:\n  - `examples\u002Fmerge_validate.py` — pandas merges with explicit `validate=` and `_merge`-equivalent inspection\n  - `examples\u002Fmerge_validate.do` — Stata `merge 1:1` \u002F `m:1` with `assert(match)` and `_merge` discipline\n  - `examples\u002Fmerge_validate.R` — dplyr joins with anti-join inspection\n  - `examples\u002Fmaster_dataset_workflow.do` — building a master HH dataset DIME-style\n  - `examples\u002Fsample_construction_logging.py` — the `log_filter` pattern in pandas\n  - `examples\u002Fdecisions_log.txt` — annotated example of a filled-in decisions log\n\n## Cross-Skill Routing\n\n- For API data acquisition (FRED, World Bank, BLS, etc.) → `api-data-fetcher`.\n- For Stata-specific cleaning (`iecodebook`, `ieduplicates`, extended missing values, harmonization across waves) → `stata-data-cleaning`.\n- For panel analysis after data is ready → `r-econometrics` \u002F `stata-regression` \u002F `python-panel-data`.\n\n## References\n\n### DIME (the conceptual backbone)\n\n- DIME Analytics, [Data Cleaning](https:\u002F\u002Fdimewiki.worldbank.org\u002FData_Cleaning).\n- DIME Analytics, [ID Variable Properties](https:\u002F\u002Fdimewiki.worldbank.org\u002FID_Variable_Properties).\n- DIME Analytics, [DataWork Folder](https:\u002F\u002Fdimewiki.worldbank.org\u002FDataWork_Folder).\n- DIME Analytics, [Master Do-files](https:\u002F\u002Fdimewiki.worldbank.org\u002FMaster_Do-files).\n- DIME Analytics, [Personally Identifiable Information (PII)](https:\u002F\u002Fdimewiki.worldbank.org\u002FPersonally_Identifiable_Information_(PII)).\n- DIME Analytics, [Reproducible Research](https:\u002F\u002Fdimewiki.worldbank.org\u002FReproducible_Research).\n\n### Style\n\n- Wickham (2014), *Tidy Data*.\n- Gentzkow & Shapiro (2014), *Code and Data for the Social Sciences*.\n- IPA, *Reproducible Research: Best Practices for Data and Code Management*.\n- Quartz, [Bad Data Guide](https:\u002F\u002Fgithub.com\u002FQuartz\u002Fbad-data-guide).\n","# Working With Data\n\nThe foundational data-discipline skill. Every other skill in this repo (`api-data-fetcher`, `stata-data-cleaning`, the analysis skills, the writing skills) assumes the data was prepared with this discipline. Use this skill before, between, or alongside the others — not after.\n\n## Operating Principles\n\n1. **Unit of analysis first, code second.** Before any merge, regression, or figure, state the unit of analysis explicitly (firm-year, household-month, country-quarter, individual). Assert it with `isid` \u002F `df.duplicated().sum() == 0` \u002F `stopifnot(!duplicated(...))`. Half of empirical mistakes come from a silent change of unit between two operations.\n2. **IDs are a contract.** A unique identifier is unique, non-missing, immutable, type-stable, and documented. If your ID can be missing, it isn't an ID. If it can change between waves, it isn't an ID.\n3. **Merging is cardinality first.** Always declare 1:1, m:1, 1:m, or m:m before the merge. Always require `match` (or its language equivalent) and inspect anything that doesn't match. There is no such thing as a \"safe\" `m:m` merge; if you wrote one, redesign.\n4. **Raw is immutable.** `data\u002Fraw\u002F` is write-once. Every transformation produces a new file in `data\u002Fintermediate\u002F` or `data\u002Fprocessed\u002F`. Re-running a pipeline from scratch reproduces the analysis-ready file byte-for-byte.\n5. **PII never leaves the encrypted folder.** Names, GPS, dates of birth, contact info, national IDs, photos: encrypted at rest, env-var keys, `.gitignore`d, never echoed to logs. The de-identified version is what the analysis sees.\n6. **Drops are logged.** Every filter has a reason in a comment and an N before\u002Fafter.\n\n## Decision Policy\n\nThis skill follows the repo-wide [Agent Policy](..\u002F..\u002FAGENT_POLICY.md).\n\n**ASK before proceeding** (blocking):\n\n1. What is the unit of analysis (single ID or composite key)?\n2. Is the dataset a panel (balanced, unbalanced, rotating) or a cross-section?\n3. Does the raw data contain PII? If yes, is encryption already set up?\n4. Before any merge: what is the cardinality (1:1, m:1, 1:m), what is the match key, and what should happen to non-matches (drop, keep with `_merge` flag, error)?\n5. When changing the unit of analysis (collapse, expand, reshape): which observations should aggregate, and how?\n\n**DEFAULT + flag** (use this default; tell the user how to override):\n\n- DataWork folder layout: `data\u002F{raw, intermediate, processed}` + `data\u002Fencrypted\u002F` for PII (see [DIME DataWork Folder](https:\u002F\u002Fdimewiki.worldbank.org\u002FDataWork_Folder)). Override with explicit paths.\n- Merge non-matches: drop master-only and using-only with a count printed, unless the user specifies `keep` or `keepall`.\n- Master dataset per unit of observation, in `data\u002Fprocessed\u002Fmaster\u002F`.\n- Numeric IDs over string IDs when both are available.\n- Long format for analysis-ready panels; wide only when needed for a specific output.\n\n**DOCUMENT and proceed** (write into the decisions log):\n\n- Inferred unit of analysis when not stated, with the assertion that confirms it.\n- Each merge: cardinality, key, N matched \u002F master-only \u002F using-only.\n- Each filter: reason and N before\u002Fafter.\n- Each derived variable: source columns and transformation.\n- Choice of de-identification thresholds (k-anonymity level, age coarsening width).\n\n`PROCEED` items follow the cross-cutting rules in `AGENT_POLICY.md` (raw immutable, no PII in git, dynamic absolute paths, `set seed`, `ieboilstart` \u002F `pyproject.toml` \u002F `Project.toml`).\n\n## Pre-flight Checklist\n\nBefore generating code, confirm with the user — and write the answers into the script header:\n\n- **Unit of analysis.** Single ID (`hhid`) or composite (`hhid year`)? What is \"one observation\"?\n- **Panel structure.** Cross-section \u002F balanced panel \u002F unbalanced \u002F rotating panel?\n- **ID provenance.** Where do IDs come from? Are they stable across waves? Across data sources?\n- **PII status.** Does the file contain personally identifying information? If yes, is the workspace already configured for encryption?\n- **Goal of this script.** Build master dataset \u002F merge for analysis \u002F reshape \u002F sample construction \u002F harmonize across waves \u002F de-identify for release?\n\n## Decision Tree: Combining Datasets\n\n```\nSame unit of observation, complementary variables\n├── Cross-section: 1:1 merge on the unit ID\n│   Stata:  merge 1:1 hhid using \"B.dta\", assert(match)\n│   Python: pd.merge(A, B, on=\"hhid\", how=\"inner\", validate=\"1:1\")\n│   R:      dplyr::inner_join(A, B, by = \"hhid\")\n│           stopifnot(nrow(A) == nrow(result))\n└── Panel: 1:1 on the composite key\n    Stata:  merge 1:1 hhid year using \"B.dta\", assert(match)\n    Python: pd.merge(A, B, on=[\"hhid\",\"year\"], validate=\"1:1\")\n\nMultiple obs in one, one in the other (e.g. household + member info)\n└── m:1 from the many side, attach the household covariates\n    Stata:  merge m:1 hhid using \"household.dta\", assert(match master)\n    Python: pd.merge(members, household, on=\"hhid\", validate=\"m:1\")\n\nBoth sides multi (e.g. transactions x people, with same key)\n└── DO NOT m:m merge. Either:\n    1. Aggregate one side first to make it 1:m, OR\n    2. Use a join that explicitly handles the cross product\n       (rarely the right answer; reconsider the question).\n\nDifferent units of observation but you need a panel\n└── First decide the unit of the result, then aggregate one side.\n    \"Firm-year × employee-quarter\" -> aggregate employees to\n    firm-year (means, sums, counts) before merging.\n\nCross-walks \u002F fuzzy keys (names, addresses)\n└── Use a deterministic crosswalk file when possible. If not, use\n    a probabilistic match (R: fuzzyjoin; Python: rapidfuzz; Stata:\n    matchit \u002F reclink2). Always validate on a labeled subset and\n    report match rates.\n```\n\n## ID Variable Contract\n\n(Following DIME's [ID Variable Properties](https:\u002F\u002Fdimewiki.worldbank.org\u002FID_Variable_Properties).)\n\nA field is an ID variable iff it satisfies *all* of:\n\n1. **Unique** within the dataset (or with the rest of the composite key).\n2. **Non-missing** for every observation.\n3. **Immutable** within and across data sources. Same firm = same `firm_id` everywhere.\n4. **Type-stable.** Either always numeric or always string. Strings preserve leading zeros and very long codes; numerics are smaller and faster. Choose once.\n5. **Documented** in the codebook with: source, format (digits, with\u002Fwithout leading zeros), domain, and any known issues (mergers, reassignments).\n\nStandard assertion (run after import, after merge, before save):\n\n```stata\nisid hhid year                       \u002F\u002F Stata\nduplicates report hhid year          \u002F\u002F explicit count\nassert !missing(hhid)\n```\n\n```python\nassert df.duplicated([\"hhid\", \"year\"]).sum() == 0       # Python\nassert df[[\"hhid\", \"year\"]].notna().all().all()\n```\n\n```r\nstopifnot(!anyDuplicated(df[, c(\"hhid\", \"year\")]))      # R\nstopifnot(!any(is.na(df$hhid)))\n```\n\n## Merging Without Surprises\n\nBefore any merge, declare:\n\n1. **Cardinality.** 1:1, m:1, 1:m. (Reject m:m.)\n2. **Key.** Single column or composite, same name on both sides.\n3. **Expected match status.** All match? Master-only allowed? Using-only allowed?\n4. **What happens to non-matches.** Drop, flag, error.\n\nValidation pattern:\n\n```stata\n* Stata\nuse \"master.dta\", clear\nisid hhid\nmerge 1:1 hhid using \"using.dta\"\nassert _merge == 3                     \u002F\u002F require all match; or:\n* tabulate _merge\n* drop if _merge == 2                  \u002F\u002F drop using-only\ndrop _merge\nisid hhid\n```\n\n```python\n# Python (pandas)\nmaster = pd.read_parquet(\"master.parquet\")\nusing  = pd.read_parquet(\"using.parquet\")\nassert master[\"hhid\"].is_unique\nassert using[\"hhid\"].is_unique\nout = master.merge(using, on=\"hhid\", how=\"left\",\n                   validate=\"1:1\", indicator=True)\nn_only_master = (out[\"_merge\"] == \"left_only\").sum()\nn_only_using  = (out[\"_merge\"] == \"right_only\").sum()\nprint(f\"only_master={n_only_master}, only_using={n_only_using}\")\nout = out.drop(columns=\"_merge\")\n```\n\n```r\n# R (dplyr \u002F inner_join, with explicit anti-joins to inspect)\nlibrary(dplyr)\nmaster \u003C- arrow::read_parquet(\"master.parquet\")\nusing  \u003C- arrow::read_parquet(\"using.parquet\")\nstopifnot(!anyDuplicated(master$hhid), !anyDuplicated(using$hhid))\n\nonly_master \u003C- anti_join(master, using, by = \"hhid\")\nonly_using  \u003C- anti_join(using, master, by = \"hhid\")\nmessage(sprintf(\"only_master=%d, only_using=%d\",\n                nrow(only_master), nrow(only_using)))\n\nout \u003C- inner_join(master, using, by = \"hhid\")\nstopifnot(!anyDuplicated(out$hhid))\n```\n\nIf `n_only_master` or `n_only_using` is non-zero, **stop and ask** (or document if the agent has been told what to do).\n\n## Master Dataset\n\nA master dataset is the single source of truth for time-invariant information about each unit:\n\n- one master per unit of observation (household, individual, firm, school);\n- de-identified version in `data\u002Fprocessed\u002Fmaster\u002F` for analysis;\n- encrypted version in `data\u002Fencrypted\u002Fmaster\u002F` if PII is involved;\n- updated through one reviewed script, never ad-hoc.\n\nA master dataset typically holds:\n\n- The ID and any aliases \u002F crosswalks.\n- Sample frame (was this unit in the sampling universe?).\n- Sampling status and stratum.\n- Treatment assignment (and date, for staggered designs).\n- Time-invariant characteristics (region, founding year, baseline values used in matching).\n\nWhen in doubt: store time-invariant fields in master, time-varying fields in survey-round files, and `merge m:1` from the round file to master at analysis time.\n\n## Sample Construction Discipline\n\nEvery cleaning script logs N at every filter. Pattern:\n\n```python\ndef log_filter(df, label):\n    print(f\"[{label}] N = {len(df):,}, units = {df['unit'].nunique():,}\")\n    return df\n\nclean = (raw\n         .pipe(log_filter, \"raw\")\n         .query(\"year >= 2010\").pipe(log_filter, \"year >= 2010\")\n         .dropna(subset=[\"outcome\"]).pipe(log_filter, \"outcome non-missing\")\n         .query(\"revenue > 0\").pipe(log_filter, \"revenue > 0\"))\n```\n\n```r\nlog_filter \u003C- function(df, label) {\n  message(sprintf(\"[%s] N = %d, units = %d\",\n                  label, nrow(df), dplyr::n_distinct(df$unit)))\n  df\n}\n\nclean \u003C- raw |>\n  log_filter(\"raw\") |>\n  filter(year >= 2010) |> log_filter(\"year >= 2010\") |>\n  filter(!is.na(outcome)) |> log_filter(\"outcome non-missing\")\n```\n\nIn Stata, prefer `iedropone` over bare `drop`:\n\n```stata\nieduplicates ...                              \u002F\u002F resolve dups first\niedropone if year \u003C 2010, error               \u002F\u002F errors if count is unexpected\niedropone if missing(outcome), error\n```\n\nThe output of these logs goes into the decisions log block at the top of the script.\n\n## PII Discipline\n\nTreat PII like radioactive material:\n\n- Common PII (always): names, GPS coordinates, dates of birth, contact info, national IDs, photos.\n- Context-dependent PII: small geographic units, rare conditions, employer x role for high-profile people.\n- Never commit PII to git. Add `data\u002Fencrypted\u002F`, `*.dta` containing PII, and `.env` to `.gitignore`.\n- Encrypt at rest (VeraCrypt for Stata-native; age \u002F gpg for plain files).\n- Coarsen quasi-identifiers before sharing (age in 5-year bands; collapse small geographies; suppress small cells).\n- Use [J-PAL `pii_scan`](https:\u002F\u002Fgithub.com\u002FJ-PAL\u002Fstata_PII_scan) to flag candidates the agent might miss.\n- `stata-data-cleaning` skill has a full `deidentify_for_release.do` example.\n\n## Common Pitfalls\n\n- Merging without checking `_merge` codes (Stata) or `validate=` (pandas) — silently produces a Cartesian-style result when the cardinality is wrong.\n- Reshape that silently drops rows because variable names don't match the stub.\n- `m:m` merge that turns 1,000 + 1,000 rows into 50,000.\n- Using a string ID like `\"01234\"` and a numeric ID like `1234` interchangeably across files — they look the same, never match.\n- Filtering on a derived variable, then later regenerating the variable with a different definition.\n- Storing dates as strings in some files and as Stata dates \u002F pandas datetime in others.\n- Saving a \"cleaned\" file that still contains observations the team agreed to drop, because the drop was applied in interactive mode but not in the script.\n- Hand-typing N in the paper because the cleaning log is in a notebook.\n\n## Additional Resources\n\n- `reference.md` — extended patterns: composite keys, hierarchical data (households \u002F individuals), reshape, survey weights, crosswalks, time-zone discipline, encoding gotchas, codebooks.\n- `examples\u002F` — runnable cross-language patterns:\n  - `examples\u002Fmerge_validate.py` — pandas merges with explicit `validate=` and `_merge`-equivalent inspection\n  - `examples\u002Fmerge_validate.do` — Stata `merge 1:1` \u002F `m:1` with `assert(match)` and `_merge` discipline\n  - `examples\u002Fmerge_validate.R` — dplyr joins with anti-join inspection\n  - `examples\u002Fmaster_dataset_workflow.do` — building a master HH dataset DIME-style\n  - `examples\u002Fsample_construction_logging.py` — the `log_filter` pattern in pandas\n  - `examples\u002Fdecisions_log.txt` — annotated example of a filled-in decisions log\n\n## Cross-Skill Routing\n\n- For API data acquisition (FRED, World Bank, BLS, etc.) → `api-data-fetcher`.\n- For Stata-specific cleaning (`iecodebook`, `ieduplicates`, extended missing values, harmonization across waves) → `stata-data-cleaning`.\n- For panel analysis after data is ready → `r-econometrics` \u002F `stata-regression` \u002F `python-panel-data`.\n\n## References\n\n### DIME (the conceptual backbone)\n\n- DIME Analytics, [Data Cleaning](https:\u002F\u002Fdimewiki.worldbank.org\u002FData_Cleaning).\n- DIME Analytics, [ID Variable Properties](https:\u002F\u002Fdimewiki.worldbank.org\u002FID_Variable_Properties).\n- DIME Analytics, [DataWork Folder](https:\u002F\u002Fdimewiki.worldbank.org\u002FDataWork_Folder).\n- DIME Analytics, [Master Do-files](https:\u002F\u002Fdimewiki.worldbank.org\u002FMaster_Do-files).\n- DIME Analytics, [Personally Identifiable Information (PII)](https:\u002F\u002Fdimewiki.worldbank.org\u002FPersonally_Identifiable_Information_(PII)).\n- DIME Analytics, [Reproducible Research](https:\u002F\u002Fdimewiki.worldbank.org\u002FReproducible_Research).\n\n### Style\n\n- Wickham (2014), *Tidy Data*.\n- Gentzkow & Shapiro (2014), *Code and Data for the Social Sciences*.\n- IPA, *Reproducible Research: Best Practices for Data and Code Management*.\n- Quartz, [Bad Data Guide](https:\u002F\u002Fgithub.com\u002FQuartz\u002Fbad-data-guide).\n",{"name":5,"risk":24,"source":25,"license":15,"version":26,"category":27,"description":7,"imported_at":28,"source_path":29,"source_repository":30,"disable-model-invocation":20},"low","https:\u002F\u002Fgithub.com\u002FJonasWeinert\u002FEconAgentSkills","1.0.0","economics","2026-09-20","_skills\u002Fdata\u002Fworking-with-data\u002FSKILL.md","JonasWeinert\u002FEconAgentSkills","main","2d1750717747097b602acf30bd09a7cbd4b4c6ea","awesome_skills\u002Feconomics\u002Fworking-with-data","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous","2026-11-09 11:45:36",{"source":8,"skillName":5,"commands":38,"prompt":48,"zipAvailable":49},[39,42,45],{"tool":40,"command":41},"npx","npx skills add Mr-Nobody-Anonymous\u002FAll-skills --skill working-with-data",{"tool":43,"command":44},"bunx","bunx skills add Mr-Nobody-Anonymous\u002FAll-skills --skill working-with-data",{"tool":46,"command":47},"pnpm","pnpm dlx skills add Mr-Nobody-Anonymous\u002FAll-skills --skill working-with-data","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Feconomics\u002Fworking-with-data\u002FSKILL.md",true,[51,57,63,67,73,77,81],{"path":52,"resourceType":53,"resourceTypeDesc":54,"sizeBytes":55,"sourceHtmlUrl":56},"examples\u002Fdecisions_log.txt","REFERENCE","参考文档",2785,"https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Feconomics\u002Fworking-with-data\u002Fexamples\u002Fdecisions_log.txt",{"path":58,"resourceType":59,"resourceTypeDesc":60,"sizeBytes":61,"sourceHtmlUrl":62},"examples\u002Fmaster_dataset_workflow.do","OTHER","其他",2473,"https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Feconomics\u002Fworking-with-data\u002Fexamples\u002Fmaster_dataset_workflow.do",{"path":64,"resourceType":59,"resourceTypeDesc":60,"sizeBytes":65,"sourceHtmlUrl":66},"examples\u002Fmerge_validate.do",1866,"https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Feconomics\u002Fworking-with-data\u002Fexamples\u002Fmerge_validate.do",{"path":68,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":71,"sourceHtmlUrl":72},"examples\u002Fmerge_validate.py","SCRIPT","脚本",2113,"https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Feconomics\u002Fworking-with-data\u002Fexamples\u002Fmerge_validate.py",{"path":74,"resourceType":59,"resourceTypeDesc":60,"sizeBytes":75,"sourceHtmlUrl":76},"examples\u002Fmerge_validate.R",1613,"https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Feconomics\u002Fworking-with-data\u002Fexamples\u002Fmerge_validate.R",{"path":78,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":79,"sourceHtmlUrl":80},"examples\u002Fsample_construction_logging.py",1576,"https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Feconomics\u002Fworking-with-data\u002Fexamples\u002Fsample_construction_logging.py",{"path":82,"resourceType":53,"resourceTypeDesc":54,"sizeBytes":83,"sourceHtmlUrl":84},"reference.md",8199,"https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Feconomics\u002Fworking-with-data\u002Freference.md",[86,93,100,107,114,121],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":19},"01M7FFFZPC2025JQQHPHDX154J","dei-assessment","Assess diversity, equity, and inclusion with metrics. TRIGGERS - Use when user needs help with dei-assessment related tasks.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fdei-assessment\u002FSKILL.md",[],[],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":19},"01M7FFFWN8H4WA7A6Y398QKB6J","dei-action-plan","Create DEI action plans with goals and initiatives. TRIGGERS - Use when user needs help with dei-action-plan related tasks.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fdei-action-plan\u002FSKILL.md",[],[],{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":19},"01M7FFFSHTM0QZRVZ2NRJTJ0PY","culture-fit-assessment","Design culture add assessment processes. TRIGGERS - Use when user needs help with culture-fit-assessment related tasks.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fculture-fit-assessment\u002FSKILL.md",[],[],{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":111,"categories":112,"keywords":113,"publishedAt":19},"01M7FFFMA12RKEQA38KHJSXXPS","cowork-to-code-bridge","Use an already-installed, independently verified cowork-to-code bridge to run narrowly approved actions on the user's own macOS, Linux, or WSL2 machine through a local file queue.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fcowork-to-code-bridge\u002FSKILL.md",[],[],{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":19},"01M7FFFH1F6M8YRGW5V487GH1K","counter-offer-response","Draft counter-offer responses for retention. TRIGGERS - Use when user needs help with counter-offer-response related tasks.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fcounter-offer-response\u002FSKILL.md",[],[],{"skillNo":122,"name":123,"nameZh":6,"description":124,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":125,"categories":126,"keywords":127,"publishedAt":19},"01M7FFFDGEHV8Y7J9N40HZXW52","continuous-feedback-system","Design continuous feedback systems. TRIGGERS - Use when user needs help with continuous-feedback-system related tasks.","https:\u002F\u002Fgithub.com\u002FMr-Nobody-Anonymous\u002FAll-skills\u002Fblob\u002Fmain\u002Fawesome_skills\u002Fhr-people\u002Fcontinuous-feedback-system\u002FSKILL.md",[],[]]