[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M69BNNR7HC8MMMB1W3DS371C":3,"skill-same:01M69BNNR7HC8MMMB1W3DS371C":124},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":28,"publishedAt":41,"linkOnly":42,"markdownContent":43,"markdownBody":44,"frontMatter":45,"refName":47,"commitSha":48,"skillDirectory":49,"forksCount":50,"repositoryHtmlUrl":51,"repositoryHomepageUrl":6,"ownerHtmlUrl":52,"ownerBlogUrl":53,"ownerDescription":54,"updatedAt":55,"install":56,"resources":69},"01M69BNNR7HC8MMMB1W3DS371C","fabric-data-cleaner",null,"Algorithm reference and self-contained PySpark notebook templates for cleaning Microsoft Fabric lakehouse tables. Covers profiling, duplicate detection, null analysis, type validation, statistics, IQR outlier detection, date format validation, Spanish DNI\u002FNIE checksum validation, and email\u002Fphone format checks. Each notebook can be uploaded and run independently in Fabric.","microsoft\u002Fgh-copilot-fabric-agents","microsoft","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F6154722?v=4","01M698VYDXZ6JW5WH3CRV650R8","Microsoft","01M69BNKRQPFJ5Z6ECX2S8E17F",3,"MIT","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-data-cleaner\u002FSKILL.md",[18,23],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",{"categoryNo":24,"categoryName":25,"taxonomyType":26,"taxonomyTypeDesc":27},"01JBSEEDCAT000000000000033","数据分析师","PROFESSION","适用职业",[29,30,31,32,33,9,34,35,36,37,38,39,40],"fabric","fabric data","fabric data cleaner","MCP","mcp server","vscode","microsoft fabric","pyspark","run","code","lakehouse","validation","2026-11-10 10:12:30",false,"---\nname: fabric-data-cleaner\ndescription: \"Algorithm reference and self-contained PySpark notebook templates for cleaning Microsoft Fabric lakehouse tables. Covers profiling, duplicate detection, null analysis, type validation, statistics, IQR outlier detection, date format validation, Spanish DNI\u002FNIE checksum validation, and email\u002Fphone format checks. Each notebook can be uploaded and run independently in Fabric.\"\ncompatibility: \"Requires the Fabric MCP Server VS Code extension (fabric.vscode-fabric-mcp-server) for workspace and table discovery. Notebooks run on Microsoft Fabric's PySpark environment.\"\n---\n\n# Fabric Data Cleaner — Algorithm Reference\n\nSelf-contained PySpark `.ipynb` notebook templates for cleaning Microsoft Fabric lakehouse tables. Each notebook can be uploaded and run independently — configure `{{TABLE_NAME}}` and `{{LAKEHOUSE_NAME}}` in the first cell, run all cells to analyze, then optionally run the fix cell.\n\n## Column Classification Rules\n\nClassify the table's columns to select which notebooks to use:\n\n| Category | Detection Rule | Which Notebook |\n|----------|---------------|----------------|\n| **Numeric** | Type is `int`, `long`, `float`, `double`, `decimal` | [type_validation.ipynb](.\u002Fnotebooks\u002Ftype_validation.ipynb), [statistics.ipynb](.\u002Fnotebooks\u002Fstatistics.ipynb), [outliers.ipynb](.\u002Fnotebooks\u002Foutliers.ipynb) |\n| **Date\u002FTimestamp** | Type is `date` or `timestamp`, or column name contains `date`, `fecha`, `time` | [date_validation.ipynb](.\u002Fnotebooks\u002Fdate_validation.ipynb) |\n| **DNI\u002FPersonal ID** | Column name contains `dni`, `nie`, `nif`, `documento`, `identidad`, `id_personal` | [dni_validation.ipynb](.\u002Fnotebooks\u002Fdni_validation.ipynb) |\n| **Email** | Column name contains `email`, `correo`, `mail` | [contact_validation.ipynb](.\u002Fnotebooks\u002Fcontact_validation.ipynb) |\n| **Phone** | Column name contains `phone`, `tel`, `movil`, `telefono` | [contact_validation.ipynb](.\u002Fnotebooks\u002Fcontact_validation.ipynb) |\n| **Any column** | Always applicable | [profiling.ipynb](.\u002Fnotebooks\u002Fprofiling.ipynb), [duplicates.ipynb](.\u002Fnotebooks\u002Fduplicates.ipynb), [nulls.ipynb](.\u002Fnotebooks\u002Fnulls.ipynb) |\n\n## Gotchas\n\n- **Notebook chaining**: Fix-cell notebooks read from `{TABLE_NAME}_cleaned` if it exists, otherwise from the original table. This means running multiple notebooks in sequence chains their fixes — each one builds on the previous. Profiling and statistics always read the original for baseline comparison.\n- **Lakehouse binding**: Notebooks deployed via REST API must include lakehouse metadata in their JSON (`metadata.dependencies.lakehouse`) or `spark.table()` will fail. The deployment script handles this automatically when you pass the `lakehouse_id` argument. Without the binding, the notebook fails during session init (~30-50s) with no detail error — do NOT misdiagnose this as a capacity issue. The MCP server may report `capacityId: null` even when a capacity is assigned; ignore that field.\n- **Table references use just the table name, not `lakehouse.table`.** In Fabric, when the default lakehouse is attached, `spark.table(\"my_table\")` works. Using `spark.table(\"my_lakehouse.my_table\")` fails with `TABLE_OR_VIEW_NOT_FOUND`. The `LAKEHOUSE_NAME` config variable is for display\u002Flogging only — never use it as a Spark schema qualifier.\n- In Spanish-locale CSV exports, commas are decimal separators (`1.234,56` not `1234.56`). The `type_validation` notebook detects and fixes this, but if you skip it, numeric casts will silently produce nulls.\n- The date `01\u002F02\u002F2024` means February 1st in Spain (`dd\u002FMM\u002Fyyyy`), not January 2nd. Always default to `dd\u002FMM\u002Fyyyy` unless the user specifies otherwise. The `date_validation` notebook auto-detects the best format.\n- Whitespace-only strings (`\"   \"`) pass `isNotNull()` checks in Spark. The `nulls` notebook catches these; without it, they silently survive filters.\n- DNI validation requires the exact mod-23 lookup table `TRWAGMYFPDXBNJZSQVHLCKE`. Do not attempt to compute the letter from scratch — use the UDF in the notebook.\n- NIE prefix replacement is `X=0`, `Y=1`, `Z=2` — **not** their ASCII values. This is a common implementation error.\n- Fabric notebooks use `display()` instead of `df.show()` for rich rendering. All templates use `display()`.\n- Fix cells write to `{TABLE_NAME}_cleaned` as Delta tables. The original table is never modified. All notebooks write to the same `_cleaned` table — run them in order so each builds on the previous result.\n- **Delta schema mismatch**: Notebooks that add flag columns (outlier flags, DNI valid, email\u002Fphone valid) change the schema of the `_cleaned` table. All fix cells use `.option(\"overwriteSchema\", \"true\")` to handle this. Without it, the second notebook in the chain would fail with `_LEGACY_ERROR_TEMP_DELTA_0007`.\n\n## Notebooks — Always Applicable\n\n### 1. Data Profiling — [profiling.ipynb](.\u002Fnotebooks\u002Fprofiling.ipynb)\n\nRun first on any table. Shows schema, sample rows, and per-column null\u002Fdistinct counts.\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`\n\n### 2. Duplicate Detection — [duplicates.ipynb](.\u002Fnotebooks\u002Fduplicates.ipynb)\n\nFinds exact duplicates (window function over all columns) and near-duplicates (same natural key, different values). Fix cell removes exact duplicates via `dropDuplicates()` → writes `{TABLE_NAME}_cleaned`.\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`, `{{KEY_COLUMNS}}` (Python list, e.g., `[\"id\", \"fecha\"]`; use `[]` to skip near-duplicate check)\n\n### 3. Null & Missing Value Analysis — [nulls.ipynb](.\u002Fnotebooks\u002Fnulls.ipynb)\n\nCounts `NULL`, empty string `\"\"`, and whitespace-only values per column. Flags columns with >5% missing. Fix cell trims whitespace and replaces empties with `NULL` → writes `{TABLE_NAME}_cleaned`.\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`\n\n## Notebooks — Numeric Columns\n\n### 4. Type Validation — [type_validation.ipynb](.\u002Fnotebooks\u002Ftype_validation.ipynb)\n\nDetects non-numeric strings in columns that should be numeric. Also catches comma decimal separators (Spanish locale: `^\\d+,\\d+$`). Fix cell replaces commas with dots and casts to `DoubleType` → writes `{TABLE_NAME}_cleaned`.\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`, `{{NUMERIC_COLUMNS}}` (Python list, e.g., `[\"precio\", \"cantidad\"]`)\n\n### 5. Statistical Summary — [statistics.ipynb](.\u002Fnotebooks\u002Fstatistics.ipynb)\n\nComputes count, min, max, mean, median (`percentile_approx`), and stddev per numeric column. Read-only — no fix cell.\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`, `{{NUMERIC_COLUMNS}}`\n\n### 6. Outlier Detection — [outliers.ipynb](.\u002Fnotebooks\u002Foutliers.ipynb)\n\nIQR method: bounds are `[Q1 - k×IQR, Q3 + k×IQR]`. Default `k=1.5` (standard), use `3.0` for extreme-only. Fix cell adds boolean `{column}_outlier_flag` (does NOT delete) → writes `{TABLE_NAME}_cleaned`.\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`, `{{NUMERIC_COLUMNS}}`, `{{IQR_MULTIPLIER}}` (default `1.5`)\n\n## Notebooks — Date Columns\n\n### 7. Date Format Validation — [date_validation.ipynb](.\u002Fnotebooks\u002Fdate_validation.ipynb)\n\nFor typed dates: checks for suspicious ranges (before 1900, future dates). For string dates: tries 8 common formats starting with the expected one, auto-detects the best match. Fix cell parses using the detected format → writes `{TABLE_NAME}_cleaned`.\n\nFormats tested: `yyyy-MM-dd`, `dd\u002FMM\u002Fyyyy`, `dd-MM-yyyy`, `yyyy-MM-dd HH:mm:ss`, `dd\u002FMM\u002Fyyyy HH:mm:ss`, `d\u002FM\u002Fyyyy`, `yyyy\u002FMM\u002Fdd`, `yyyyMMdd`\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`, `{{DATE_COLUMNS}}` (Python list), `{{EXPECTED_DATE_FORMAT}}` (default `\"dd\u002FMM\u002Fyyyy\"`)\n\n## Notebooks — Personal Data (Spanish)\n\n### 8. DNI\u002FNIE Validation — [dni_validation.ipynb](.\u002Fnotebooks\u002Fdni_validation.ipynb)\n\nValidates Spanish national IDs using the mod-23 checksum algorithm:\n\n- **DNI** (8 digits + letter): `number % 23` → lookup in `TRWAGMYFPDXBNJZSQVHLCKE`\n- **NIE** (X\u002FY\u002FZ + 7 digits + letter): replace prefix (`X=0`, `Y=1`, `Z=2`), then same algorithm\n- Pre-processing: uppercase, strip spaces\u002Fdashes\u002Fdots\n\nUDF returns `(is_valid, reason)` with reasons: `valid_dni`, `valid_nie`, `empty`, `invalid_format`, `dni_bad_letter_expected_X`, `nie_bad_letter_expected_X`. Fix cell adds `{column}_valid` flag → writes `{TABLE_NAME}_cleaned`.\n\nSee [.\u002Freferences\u002Fdni-validation-rules.md](.\u002Freferences\u002Fdni-validation-rules.md) for the full algorithm reference.\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`, `{{DNI_COLUMNS}}` (Python list, e.g., `[\"dni\", \"documento_identidad\"]`)\n\n### 9. Contact Validation — [contact_validation.ipynb](.\u002Fnotebooks\u002Fcontact_validation.ipynb)\n\n- **Email**: regex `^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$`\n- **Spanish phone**: 9 digits starting with `6`\u002F`7`\u002F`8`\u002F`9`, optionally prefixed with `+34`. Pre-processing strips spaces, dashes, dots, parentheses.\n\nFix cell adds `{column}_valid_email` \u002F `{column}_valid_phone` flags → writes `{TABLE_NAME}_cleaned`.\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`, `{{EMAIL_COLUMNS}}` (use `[]` if none), `{{PHONE_COLUMNS}}` (use `[]` if none)\n\n## Notebook Deployment, Execution & Cleanup\n\nUse [.\u002Fscripts\u002Ffabric_notebook.py](.\u002Fscripts\u002Ffabric_notebook.py) to manage notebooks in Fabric via the REST API. The script uses `DefaultAzureCredential` — it picks up the logged-in user's Azure CLI session (`az login`), so no `.env` or secrets file is needed.\n\n\n**Install dependencies** (one-time):\n```bash\npip install azure-identity requests\n```\n\n### Deploy a notebook\n\nReads a local `.ipynb` file, injects Fabric lakehouse metadata (kernel info + default lakehouse binding), base64-encodes it with a `.platform` file, and creates a Notebook item via `POST \u002Fv1\u002Fworkspaces\u002F{id}\u002Fnotebooks`. The `lakehouse_id` is **required** — without it, the notebook's Spark session has no lakehouse context and `spark.table()` calls fail.\n\n```bash\npython scripts\u002Ffabric_notebook.py deploy \u003Cworkspace_id> \u003Clakehouse_id> \u003Cnotebook_name> \u003Cpath_to.ipynb>\n```\n\nReturns the notebook ID on success. If the API returns 202 (provisioning), the script polls the operation URL until completion.\n\n### Run a notebook\n\nTriggers an on-demand execution via `POST \u002Fv1\u002Fworkspaces\u002F{id}\u002Fitems\u002F{notebook_id}\u002Fjobs\u002FRunNotebook\u002Finstances`. The notebook runs on the Fabric Spark cluster attached to the workspace.\n\n```bash\npython scripts\u002Ffabric_notebook.py run \u003Cworkspace_id> \u003Cnotebook_id>\n```\n\nReturns the job instance ID. The notebook runs asynchronously — use `status` to poll.\n\n### Check job status\n\nPolls the job instance to check if the run completed, failed, or is still running.\n\n```bash\npython scripts\u002Ffabric_notebook.py status \u003Cworkspace_id> \u003Cnotebook_id> \u003Cjob_instance_id>\n```\n\n### List notebooks\n\nLists all notebooks in the workspace, or checks if a specific notebook exists by display name. Returns the notebook ID if found.\n\n```bash\npython scripts\u002Ffabric_notebook.py list \u003Cworkspace_id>                # list all\npython scripts\u002Ffabric_notebook.py list \u003Cworkspace_id> \u003Cnotebook_name> # find by name\n```\n\nUse `list` to check if a notebook already exists before deploying, or to retrieve the notebook ID after a 202 deploy.\n\n### Delete a notebook\n\nRemoves the notebook from the workspace via `DELETE \u002Fv1\u002Fworkspaces\u002F{id}\u002Fnotebooks\u002F{notebook_id}`. Use this to clean up after the cleaning notebooks have been run.\n\n```bash\npython scripts\u002Ffabric_notebook.py delete \u003Cworkspace_id> \u003Cnotebook_id>\n```\n\n### Deployment gotchas\n\n- **Lakehouse binding is mandatory.** The `deploy` command requires `lakehouse_id` because it injects `metadata.dependencies.lakehouse` into the notebook JSON and includes a `.platform` file. Without this, the Spark session starts with no lakehouse context and `spark.table()` fails immediately. The lakehouse ID can be obtained from Phase 1 discovery via MCP `onelake_list_items`.\n- The create API may return **202 instead of 201** for notebooks with definitions. Always handle both — the script does this automatically.\n- Notebook `displayName` must be **unique within the workspace**. If a notebook with the same name exists, the API returns `ItemDisplayNameAlreadyInUse`. Delete the old one first or use a different name.\n- The `RunNotebook` job type is the correct one for notebooks. Do not use `DefaultJob` — that's for other item types.\n- Job status polling: the `Location` header from the run response is the poll URL. Respect the `Retry-After` header (usually 30-60s).\n- `DefaultAzureCredential` tries Azure CLI first, then managed identity, then environment variables. If `az login` is active, it just works.\n- **Idempotency — do not call deploy twice for the same notebook.** The `deploy` command is not idempotent. If a deploy returns 202 (provisioning), wait for the operation to complete — do NOT call `deploy` again with the same name. The second call will fail with `ItemDisplayNameAlreadyInUse`. Similarly, after deleting a notebook, the name may take up to 90 seconds to become available; the API returns `ItemDisplayNameNotAvailableYet` (409, retriable) during this window. The correct pattern is: deploy → wait for completion → `list` to get the ID → run. Never fire-and-forget a deploy and immediately retry.\n- **One operation at a time.** Do not deploy multiple notebooks in parallel — the Fabric API may throttle or return intermittent errors. Deploy, confirm success, then deploy the next one sequentially.\n- **Use `list` to verify state.** Before deploying, run `list \u003Cworkspace_id> \u003Cnotebook_name>` to check if the notebook already exists. After deploying, use `list` to confirm it was created and retrieve its ID. This avoids duplicate deploys and guessing IDs.\n\n## Notebook Validation\n\nUse [.\u002Fscripts\u002Fvalidate_notebooks.py](.\u002Fscripts\u002Fvalidate_notebooks.py) to verify generated notebooks before deployment. Run it against the output directory after generating customized notebooks.\n\n```bash\npython scripts\u002Fvalidate_notebooks.py \u003Cdirectory>\n```\n\nThe script checks every `.ipynb` file in the directory for:\n\n1. **Valid JSON**: `json.load()` succeeds without exceptions.\n2. **No remaining placeholders**: no `{{...}}` strings anywhere in the file.\n3. **Notebook structure**: top-level `cells` and `metadata` keys exist; every cell has `cell_type` and `source`; every code cell has `outputs`.\n\nExit code `0` means all notebooks passed. Exit code `1` means at least one check failed — fix the notebook and re-run. Exit code `2` means invalid usage (bad directory or no `.ipynb` files found).\n\n**Always validate before deploying.** A notebook with broken JSON will fail to upload; leftover placeholders will cause runtime `NameError`s in Spark.\n\n## Global Rules\n\n- **PySpark only.** Use `pyspark.sql.functions`, never pandas for processing.\n- **Non-destructive.** Fix cells write to a new `{TABLE_NAME}_cleaned` table. The original is never modified. All notebooks target the same `_cleaned` suffix.\n- **Fix cells require manual execution.** Every fix cell is preceded by a markdown warning.\n- **Self-contained.** Each notebook has its own imports, config, SparkSession, and table loading. No cross-notebook dependencies.\n","# Fabric Data Cleaner — Algorithm Reference\n\nSelf-contained PySpark `.ipynb` notebook templates for cleaning Microsoft Fabric lakehouse tables. Each notebook can be uploaded and run independently — configure `{{TABLE_NAME}}` and `{{LAKEHOUSE_NAME}}` in the first cell, run all cells to analyze, then optionally run the fix cell.\n\n## Column Classification Rules\n\nClassify the table's columns to select which notebooks to use:\n\n| Category | Detection Rule | Which Notebook |\n|----------|---------------|----------------|\n| **Numeric** | Type is `int`, `long`, `float`, `double`, `decimal` | [type_validation.ipynb](.\u002Fnotebooks\u002Ftype_validation.ipynb), [statistics.ipynb](.\u002Fnotebooks\u002Fstatistics.ipynb), [outliers.ipynb](.\u002Fnotebooks\u002Foutliers.ipynb) |\n| **Date\u002FTimestamp** | Type is `date` or `timestamp`, or column name contains `date`, `fecha`, `time` | [date_validation.ipynb](.\u002Fnotebooks\u002Fdate_validation.ipynb) |\n| **DNI\u002FPersonal ID** | Column name contains `dni`, `nie`, `nif`, `documento`, `identidad`, `id_personal` | [dni_validation.ipynb](.\u002Fnotebooks\u002Fdni_validation.ipynb) |\n| **Email** | Column name contains `email`, `correo`, `mail` | [contact_validation.ipynb](.\u002Fnotebooks\u002Fcontact_validation.ipynb) |\n| **Phone** | Column name contains `phone`, `tel`, `movil`, `telefono` | [contact_validation.ipynb](.\u002Fnotebooks\u002Fcontact_validation.ipynb) |\n| **Any column** | Always applicable | [profiling.ipynb](.\u002Fnotebooks\u002Fprofiling.ipynb), [duplicates.ipynb](.\u002Fnotebooks\u002Fduplicates.ipynb), [nulls.ipynb](.\u002Fnotebooks\u002Fnulls.ipynb) |\n\n## Gotchas\n\n- **Notebook chaining**: Fix-cell notebooks read from `{TABLE_NAME}_cleaned` if it exists, otherwise from the original table. This means running multiple notebooks in sequence chains their fixes — each one builds on the previous. Profiling and statistics always read the original for baseline comparison.\n- **Lakehouse binding**: Notebooks deployed via REST API must include lakehouse metadata in their JSON (`metadata.dependencies.lakehouse`) or `spark.table()` will fail. The deployment script handles this automatically when you pass the `lakehouse_id` argument. Without the binding, the notebook fails during session init (~30-50s) with no detail error — do NOT misdiagnose this as a capacity issue. The MCP server may report `capacityId: null` even when a capacity is assigned; ignore that field.\n- **Table references use just the table name, not `lakehouse.table`.** In Fabric, when the default lakehouse is attached, `spark.table(\"my_table\")` works. Using `spark.table(\"my_lakehouse.my_table\")` fails with `TABLE_OR_VIEW_NOT_FOUND`. The `LAKEHOUSE_NAME` config variable is for display\u002Flogging only — never use it as a Spark schema qualifier.\n- In Spanish-locale CSV exports, commas are decimal separators (`1.234,56` not `1234.56`). The `type_validation` notebook detects and fixes this, but if you skip it, numeric casts will silently produce nulls.\n- The date `01\u002F02\u002F2024` means February 1st in Spain (`dd\u002FMM\u002Fyyyy`), not January 2nd. Always default to `dd\u002FMM\u002Fyyyy` unless the user specifies otherwise. The `date_validation` notebook auto-detects the best format.\n- Whitespace-only strings (`\"   \"`) pass `isNotNull()` checks in Spark. The `nulls` notebook catches these; without it, they silently survive filters.\n- DNI validation requires the exact mod-23 lookup table `TRWAGMYFPDXBNJZSQVHLCKE`. Do not attempt to compute the letter from scratch — use the UDF in the notebook.\n- NIE prefix replacement is `X=0`, `Y=1`, `Z=2` — **not** their ASCII values. This is a common implementation error.\n- Fabric notebooks use `display()` instead of `df.show()` for rich rendering. All templates use `display()`.\n- Fix cells write to `{TABLE_NAME}_cleaned` as Delta tables. The original table is never modified. All notebooks write to the same `_cleaned` table — run them in order so each builds on the previous result.\n- **Delta schema mismatch**: Notebooks that add flag columns (outlier flags, DNI valid, email\u002Fphone valid) change the schema of the `_cleaned` table. All fix cells use `.option(\"overwriteSchema\", \"true\")` to handle this. Without it, the second notebook in the chain would fail with `_LEGACY_ERROR_TEMP_DELTA_0007`.\n\n## Notebooks — Always Applicable\n\n### 1. Data Profiling — [profiling.ipynb](.\u002Fnotebooks\u002Fprofiling.ipynb)\n\nRun first on any table. Shows schema, sample rows, and per-column null\u002Fdistinct counts.\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`\n\n### 2. Duplicate Detection — [duplicates.ipynb](.\u002Fnotebooks\u002Fduplicates.ipynb)\n\nFinds exact duplicates (window function over all columns) and near-duplicates (same natural key, different values). Fix cell removes exact duplicates via `dropDuplicates()` → writes `{TABLE_NAME}_cleaned`.\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`, `{{KEY_COLUMNS}}` (Python list, e.g., `[\"id\", \"fecha\"]`; use `[]` to skip near-duplicate check)\n\n### 3. Null & Missing Value Analysis — [nulls.ipynb](.\u002Fnotebooks\u002Fnulls.ipynb)\n\nCounts `NULL`, empty string `\"\"`, and whitespace-only values per column. Flags columns with >5% missing. Fix cell trims whitespace and replaces empties with `NULL` → writes `{TABLE_NAME}_cleaned`.\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`\n\n## Notebooks — Numeric Columns\n\n### 4. Type Validation — [type_validation.ipynb](.\u002Fnotebooks\u002Ftype_validation.ipynb)\n\nDetects non-numeric strings in columns that should be numeric. Also catches comma decimal separators (Spanish locale: `^\\d+,\\d+$`). Fix cell replaces commas with dots and casts to `DoubleType` → writes `{TABLE_NAME}_cleaned`.\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`, `{{NUMERIC_COLUMNS}}` (Python list, e.g., `[\"precio\", \"cantidad\"]`)\n\n### 5. Statistical Summary — [statistics.ipynb](.\u002Fnotebooks\u002Fstatistics.ipynb)\n\nComputes count, min, max, mean, median (`percentile_approx`), and stddev per numeric column. Read-only — no fix cell.\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`, `{{NUMERIC_COLUMNS}}`\n\n### 6. Outlier Detection — [outliers.ipynb](.\u002Fnotebooks\u002Foutliers.ipynb)\n\nIQR method: bounds are `[Q1 - k×IQR, Q3 + k×IQR]`. Default `k=1.5` (standard), use `3.0` for extreme-only. Fix cell adds boolean `{column}_outlier_flag` (does NOT delete) → writes `{TABLE_NAME}_cleaned`.\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`, `{{NUMERIC_COLUMNS}}`, `{{IQR_MULTIPLIER}}` (default `1.5`)\n\n## Notebooks — Date Columns\n\n### 7. Date Format Validation — [date_validation.ipynb](.\u002Fnotebooks\u002Fdate_validation.ipynb)\n\nFor typed dates: checks for suspicious ranges (before 1900, future dates). For string dates: tries 8 common formats starting with the expected one, auto-detects the best match. Fix cell parses using the detected format → writes `{TABLE_NAME}_cleaned`.\n\nFormats tested: `yyyy-MM-dd`, `dd\u002FMM\u002Fyyyy`, `dd-MM-yyyy`, `yyyy-MM-dd HH:mm:ss`, `dd\u002FMM\u002Fyyyy HH:mm:ss`, `d\u002FM\u002Fyyyy`, `yyyy\u002FMM\u002Fdd`, `yyyyMMdd`\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`, `{{DATE_COLUMNS}}` (Python list), `{{EXPECTED_DATE_FORMAT}}` (default `\"dd\u002FMM\u002Fyyyy\"`)\n\n## Notebooks — Personal Data (Spanish)\n\n### 8. DNI\u002FNIE Validation — [dni_validation.ipynb](.\u002Fnotebooks\u002Fdni_validation.ipynb)\n\nValidates Spanish national IDs using the mod-23 checksum algorithm:\n\n- **DNI** (8 digits + letter): `number % 23` → lookup in `TRWAGMYFPDXBNJZSQVHLCKE`\n- **NIE** (X\u002FY\u002FZ + 7 digits + letter): replace prefix (`X=0`, `Y=1`, `Z=2`), then same algorithm\n- Pre-processing: uppercase, strip spaces\u002Fdashes\u002Fdots\n\nUDF returns `(is_valid, reason)` with reasons: `valid_dni`, `valid_nie`, `empty`, `invalid_format`, `dni_bad_letter_expected_X`, `nie_bad_letter_expected_X`. Fix cell adds `{column}_valid` flag → writes `{TABLE_NAME}_cleaned`.\n\nSee [.\u002Freferences\u002Fdni-validation-rules.md](.\u002Freferences\u002Fdni-validation-rules.md) for the full algorithm reference.\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`, `{{DNI_COLUMNS}}` (Python list, e.g., `[\"dni\", \"documento_identidad\"]`)\n\n### 9. Contact Validation — [contact_validation.ipynb](.\u002Fnotebooks\u002Fcontact_validation.ipynb)\n\n- **Email**: regex `^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$`\n- **Spanish phone**: 9 digits starting with `6`\u002F`7`\u002F`8`\u002F`9`, optionally prefixed with `+34`. Pre-processing strips spaces, dashes, dots, parentheses.\n\nFix cell adds `{column}_valid_email` \u002F `{column}_valid_phone` flags → writes `{TABLE_NAME}_cleaned`.\n\n**Placeholders**: `{{TABLE_NAME}}`, `{{LAKEHOUSE_NAME}}`, `{{EMAIL_COLUMNS}}` (use `[]` if none), `{{PHONE_COLUMNS}}` (use `[]` if none)\n\n## Notebook Deployment, Execution & Cleanup\n\nUse [.\u002Fscripts\u002Ffabric_notebook.py](.\u002Fscripts\u002Ffabric_notebook.py) to manage notebooks in Fabric via the REST API. The script uses `DefaultAzureCredential` — it picks up the logged-in user's Azure CLI session (`az login`), so no `.env` or secrets file is needed.\n\n\n**Install dependencies** (one-time):\n```bash\npip install azure-identity requests\n```\n\n### Deploy a notebook\n\nReads a local `.ipynb` file, injects Fabric lakehouse metadata (kernel info + default lakehouse binding), base64-encodes it with a `.platform` file, and creates a Notebook item via `POST \u002Fv1\u002Fworkspaces\u002F{id}\u002Fnotebooks`. The `lakehouse_id` is **required** — without it, the notebook's Spark session has no lakehouse context and `spark.table()` calls fail.\n\n```bash\npython scripts\u002Ffabric_notebook.py deploy \u003Cworkspace_id> \u003Clakehouse_id> \u003Cnotebook_name> \u003Cpath_to.ipynb>\n```\n\nReturns the notebook ID on success. If the API returns 202 (provisioning), the script polls the operation URL until completion.\n\n### Run a notebook\n\nTriggers an on-demand execution via `POST \u002Fv1\u002Fworkspaces\u002F{id}\u002Fitems\u002F{notebook_id}\u002Fjobs\u002FRunNotebook\u002Finstances`. The notebook runs on the Fabric Spark cluster attached to the workspace.\n\n```bash\npython scripts\u002Ffabric_notebook.py run \u003Cworkspace_id> \u003Cnotebook_id>\n```\n\nReturns the job instance ID. The notebook runs asynchronously — use `status` to poll.\n\n### Check job status\n\nPolls the job instance to check if the run completed, failed, or is still running.\n\n```bash\npython scripts\u002Ffabric_notebook.py status \u003Cworkspace_id> \u003Cnotebook_id> \u003Cjob_instance_id>\n```\n\n### List notebooks\n\nLists all notebooks in the workspace, or checks if a specific notebook exists by display name. Returns the notebook ID if found.\n\n```bash\npython scripts\u002Ffabric_notebook.py list \u003Cworkspace_id>                # list all\npython scripts\u002Ffabric_notebook.py list \u003Cworkspace_id> \u003Cnotebook_name> # find by name\n```\n\nUse `list` to check if a notebook already exists before deploying, or to retrieve the notebook ID after a 202 deploy.\n\n### Delete a notebook\n\nRemoves the notebook from the workspace via `DELETE \u002Fv1\u002Fworkspaces\u002F{id}\u002Fnotebooks\u002F{notebook_id}`. Use this to clean up after the cleaning notebooks have been run.\n\n```bash\npython scripts\u002Ffabric_notebook.py delete \u003Cworkspace_id> \u003Cnotebook_id>\n```\n\n### Deployment gotchas\n\n- **Lakehouse binding is mandatory.** The `deploy` command requires `lakehouse_id` because it injects `metadata.dependencies.lakehouse` into the notebook JSON and includes a `.platform` file. Without this, the Spark session starts with no lakehouse context and `spark.table()` fails immediately. The lakehouse ID can be obtained from Phase 1 discovery via MCP `onelake_list_items`.\n- The create API may return **202 instead of 201** for notebooks with definitions. Always handle both — the script does this automatically.\n- Notebook `displayName` must be **unique within the workspace**. If a notebook with the same name exists, the API returns `ItemDisplayNameAlreadyInUse`. Delete the old one first or use a different name.\n- The `RunNotebook` job type is the correct one for notebooks. Do not use `DefaultJob` — that's for other item types.\n- Job status polling: the `Location` header from the run response is the poll URL. Respect the `Retry-After` header (usually 30-60s).\n- `DefaultAzureCredential` tries Azure CLI first, then managed identity, then environment variables. If `az login` is active, it just works.\n- **Idempotency — do not call deploy twice for the same notebook.** The `deploy` command is not idempotent. If a deploy returns 202 (provisioning), wait for the operation to complete — do NOT call `deploy` again with the same name. The second call will fail with `ItemDisplayNameAlreadyInUse`. Similarly, after deleting a notebook, the name may take up to 90 seconds to become available; the API returns `ItemDisplayNameNotAvailableYet` (409, retriable) during this window. The correct pattern is: deploy → wait for completion → `list` to get the ID → run. Never fire-and-forget a deploy and immediately retry.\n- **One operation at a time.** Do not deploy multiple notebooks in parallel — the Fabric API may throttle or return intermittent errors. Deploy, confirm success, then deploy the next one sequentially.\n- **Use `list` to verify state.** Before deploying, run `list \u003Cworkspace_id> \u003Cnotebook_name>` to check if the notebook already exists. After deploying, use `list` to confirm it was created and retrieve its ID. This avoids duplicate deploys and guessing IDs.\n\n## Notebook Validation\n\nUse [.\u002Fscripts\u002Fvalidate_notebooks.py](.\u002Fscripts\u002Fvalidate_notebooks.py) to verify generated notebooks before deployment. Run it against the output directory after generating customized notebooks.\n\n```bash\npython scripts\u002Fvalidate_notebooks.py \u003Cdirectory>\n```\n\nThe script checks every `.ipynb` file in the directory for:\n\n1. **Valid JSON**: `json.load()` succeeds without exceptions.\n2. **No remaining placeholders**: no `{{...}}` strings anywhere in the file.\n3. **Notebook structure**: top-level `cells` and `metadata` keys exist; every cell has `cell_type` and `source`; every code cell has `outputs`.\n\nExit code `0` means all notebooks passed. Exit code `1` means at least one check failed — fix the notebook and re-run. Exit code `2` means invalid usage (bad directory or no `.ipynb` files found).\n\n**Always validate before deploying.** A notebook with broken JSON will fail to upload; leftover placeholders will cause runtime `NameError`s in Spark.\n\n## Global Rules\n\n- **PySpark only.** Use `pyspark.sql.functions`, never pandas for processing.\n- **Non-destructive.** Fix cells write to a new `{TABLE_NAME}_cleaned` table. The original is never modified. All notebooks target the same `_cleaned` suffix.\n- **Fix cells require manual execution.** Every fix cell is preceded by a markdown warning.\n- **Self-contained.** Each notebook has its own imports, config, SparkSession, and table loading. No cross-notebook dependencies.\n",{"name":5,"description":7,"compatibility":46},"Requires the Fabric MCP Server VS Code extension (fabric.vscode-fabric-mcp-server) for workspace and table discovery. Notebooks run on Microsoft Fabric's PySpark environment.","main","a54dec65de5a1d2a62bd5cbd9e5003adca5f5a84",".github\u002Fskills\u002Ffabric-data-cleaner",1,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents","https:\u002F\u002Fgithub.com\u002Fmicrosoft","https:\u002F\u002Fopensource.microsoft.com","Open source projects and samples from Microsoft","2026-06-04 21:31:34",{"source":8,"skillName":5,"commands":57,"prompt":67,"zipAvailable":68},[58,61,64],{"tool":59,"command":60},"npx","npx skills add microsoft\u002Fgh-copilot-fabric-agents --skill fabric-data-cleaner",{"tool":62,"command":63},"bunx","bunx skills add microsoft\u002Fgh-copilot-fabric-agents --skill fabric-data-cleaner",{"tool":65,"command":66},"pnpm","pnpm dlx skills add microsoft\u002Fgh-copilot-fabric-agents --skill fabric-data-cleaner","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-data-cleaner\u002FSKILL.md",true,[70,76,80,84,88,92,96,100,104,108,114,120],{"path":71,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":74,"sourceHtmlUrl":75},"notebooks\u002Fcontact_validation.ipynb","OTHER","其他",9093,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-data-cleaner\u002Fnotebooks\u002Fcontact_validation.ipynb",{"path":77,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":78,"sourceHtmlUrl":79},"notebooks\u002Fdate_validation.ipynb",8561,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-data-cleaner\u002Fnotebooks\u002Fdate_validation.ipynb",{"path":81,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":82,"sourceHtmlUrl":83},"notebooks\u002Fdni_validation.ipynb",9428,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-data-cleaner\u002Fnotebooks\u002Fdni_validation.ipynb",{"path":85,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":86,"sourceHtmlUrl":87},"notebooks\u002Fduplicates.ipynb",6426,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-data-cleaner\u002Fnotebooks\u002Fduplicates.ipynb",{"path":89,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":90,"sourceHtmlUrl":91},"notebooks\u002Fnulls.ipynb",6712,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-data-cleaner\u002Fnotebooks\u002Fnulls.ipynb",{"path":93,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":94,"sourceHtmlUrl":95},"notebooks\u002Foutliers.ipynb",7589,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-data-cleaner\u002Fnotebooks\u002Foutliers.ipynb",{"path":97,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":98,"sourceHtmlUrl":99},"notebooks\u002Fprofiling.ipynb",4128,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-data-cleaner\u002Fnotebooks\u002Fprofiling.ipynb",{"path":101,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":102,"sourceHtmlUrl":103},"notebooks\u002Fstatistics.ipynb",4934,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-data-cleaner\u002Fnotebooks\u002Fstatistics.ipynb",{"path":105,"resourceType":72,"resourceTypeDesc":73,"sizeBytes":106,"sourceHtmlUrl":107},"notebooks\u002Ftype_validation.ipynb",7207,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-data-cleaner\u002Fnotebooks\u002Ftype_validation.ipynb",{"path":109,"resourceType":110,"resourceTypeDesc":111,"sizeBytes":112,"sourceHtmlUrl":113},"references\u002Fdni-validation-rules.md","REFERENCE","参考文档",1748,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-data-cleaner\u002Freferences\u002Fdni-validation-rules.md",{"path":115,"resourceType":116,"resourceTypeDesc":117,"sizeBytes":118,"sourceHtmlUrl":119},"scripts\u002Ffabric_notebook.py","SCRIPT","脚本",9167,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-data-cleaner\u002Fscripts\u002Ffabric_notebook.py",{"path":121,"resourceType":116,"resourceTypeDesc":117,"sizeBytes":122,"sourceHtmlUrl":123},"scripts\u002Fvalidate_notebooks.py",2548,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-data-cleaner\u002Fscripts\u002Fvalidate_notebooks.py",[125,145,165,182],{"skillNo":126,"name":127,"nameZh":6,"description":128,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":129,"categories":130,"keywords":139,"publishedAt":41},"01M69BP4V2GV4WZVHWX5A65QEW","fabric-synthetic-data","Generate and upload synthetic data to Microsoft Fabric lakehouse tables. Covers star-schema generation, realistic fake data (names, dates, IDs, transactions), configurable row counts, referential integrity between tables, Parquet export, and OneLake upload. Use when: synthetic data, test data, fake data, sample data, generate tables, populate lakehouse, seed data, mock data.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-synthetic-data\u002FSKILL.md",[131,134,135,138],{"categoryNo":132,"categoryName":133,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000010","后端与接口",{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":136,"categoryName":137,"taxonomyType":26,"taxonomyTypeDesc":27},"01JBSEEDCAT000000000000030","开发工程师",{"categoryNo":24,"categoryName":25,"taxonomyType":26,"taxonomyTypeDesc":27},[140,29,141,32,142,33,143,9,35,144,38,39],"fabric synthetic data","synthetic data","python","test","generate",{"skillNo":146,"name":147,"nameZh":6,"description":148,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":149,"categories":150,"keywords":158,"publishedAt":41},"01M69BP1F17XK0TFKMDS23QNEV","fabric-semantic-model","Infer and generate Power BI semantic models (TMDL format) from Microsoft Fabric lakehouse tables. Covers star-schema detection, table classification (fact\u002Fdimension), relationship inference, DAX measure generation, data type mapping, and deployment via REST API. Use when: create semantic model, build data model, star schema, Power BI model, TMDL, fact table, dimension table.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-semantic-model\u002FSKILL.md",[151,152,155,156,157],{"categoryNo":132,"categoryName":133,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":153,"categoryName":154,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000013","分析与可视化",{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},{"categoryNo":136,"categoryName":137,"taxonomyType":26,"taxonomyTypeDesc":27},{"categoryNo":24,"categoryName":25,"taxonomyType":26,"taxonomyTypeDesc":27},[29,159,160,161,162,35,163,9,33,164,32,144,38,39],"fabric semantic model","create","power bi","deployment","dax","inference",{"skillNo":166,"name":167,"nameZh":6,"description":168,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":169,"categories":170,"keywords":172,"publishedAt":41},"01M69BNXPJX1D29F4SAD54K1HZ","fabric-optimization-review","Review and optimize data storage, data models, and semantic models in Microsoft Fabric lakehouses. Covers Delta table optimization (V-Order, Z-Order, compaction, partitioning), data model anti-pattern detection, semantic model DAX review, relationship optimization, AI\u002Fagentic readiness (descriptions, synonyms, linguistic schema, display folders), workspace audit (Fabric best practices, dimensional modeling, Direct Lake), and actionable recommendation reports. Use when: optimization, performance, review, tuning, best practices, V-Order, Z-Order, partitioning, compaction, DAX optimization, data model review, AI readiness, agentic, NLQ, Copilot, natural language, workspace audit, Direct Lake, dimensional modeling, audit.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-optimization-review\u002FSKILL.md",[171],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[29,173,174,175,176,38,32,142,177,178,33,179,9,163,180,181,35],"fabric optimization review","review","best practices","pattern detection","agentic","performance","storage","dax optimization","copilot",{"skillNo":183,"name":184,"nameZh":6,"description":185,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":186,"categories":187,"keywords":192,"publishedAt":41},"01M69BNSNSETNDE2R5AEQAGZVC","fabric-metadata-creation","Create reviewable metadata proposals for Microsoft Fabric lakehouse tables and semantic models. Covers schema analysis, concise technical and business descriptions, glossary terms, classifications, Purview-like sensitivity label proposals, PII and sensitive data detection, relationship and lineage hints, data quality rules, CDE candidates, data products, JSON\u002FPDF output, and steward review questions. Use when: metadata creation, data catalog, Purview mapping, glossary inference, data classification, sensitivity labels, stewardship review, governance metadata.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fgh-copilot-fabric-agents\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Ffabric-metadata-creation\u002FSKILL.md",[188,191],{"categoryNo":189,"categoryName":190,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000002","文档处理",{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[193,29,38,32,174,142,164,33,9,194,35,195,196,197,198,160,39,199],"fabric metadata creation","identity","governance","purview","azure","data catalog","PDF"]